How many tools does it take to find a resource leak in Windows Server 2003?

Problem: A Windows 2003 Server that starts refusing connections after approximately 14 days of uptime, and if left alone will completely lock up and stop displaying a local console login window. The server is monitored by Nagios, and Munin. Nagios gives a couple of hours advance warning that the server is going to go south, which gives us plenty of time to reboot it. Of the server traits Munin monitors, there doesn’t appear to be anything unusual. During the melt-down phase the following error is plentiful in the event logs: “The server was unable to allocate from the system nonpaged pool because the pool was empty.”

Task Manager doesn’t display any unusual memory usage. Process Explorer didn’t yield any immediately useful clues. Quite a bit of Google searching and about the only tool I could find that was recommended for tracking down resource leaks that affect the nonpaged memory pool was a Microsoft Utility called poolmon.exe. Poolmon has to be the most primitive wacked-up tool anyone at Microsoft has developed, released, and then blatantly ignored. It’s pure ASCII, runs in a DOS-style command window, can’t log its results over time to a log file; and I kid you not, the official way to use this utility to track down leaks is to run it and copy and paste the entire screen output to a text file every fifteen minutes so you can diff the results.

Poolmon needs an overhaul: the ability to track growth of values over time, log output to a file, and a GUI would be nice since this is a Windows utility. I know, it’s a lot for which to ask. I have no interest in picking up where I left off with Visual C++ ten years ago, so I made do with a little bit of perl and Excel. Using ActiveState’s ActivePerl, I set up a scheduled task that ran every fifteen minutes and executed this bit of braindead code that just dumps the output of poolmon.exe to a text file that has the timestamp in the file name: poolmonlog.pl

I let that run for two days, and then I wacked this bit of perl together to analyze the results: poolmon_analysis.pl

I copy and paste the output into Excel, and I turn it into a line graph. As you can see, there is obviously a problem and it is quite visible with only a short period of logging:

poolmon output graphed in excel

According to the poolmon.txt file that comes with poolmon,exe, the pool label Thre is “nt!ps – Thread objects”, which is the pool name for Windows thread and handle objects. Now I know I’m not looking for a memory issue, I’m looking for thread or handle usage. Task Manager can be used to display these resources, but it is not configured to do so by default. I open up Task Manager, click the Processes tab, and then go to View…Select Columns… and I turn on Handle Count and Thread Count. Sorting by these two columns, I quickly track down the culprit as HPBPRO.EXE which is part of the HP drivers for a Color LaserJet 3500.

Since it took me this much effort to track down this problem, and because I’ve read lots and lots of message board posts written by people having an equally difficult time tracking down similar issues, I thought I’d share. If there is an easier way that I could have diagnosed this issue, I would love to hear it. If this post helps you solve a similar issue, I’d love to hear that too.

Cheers,

-Chris

[ad#adsense-horizontal]

Sprint Sucks. Again.

For the record, I KNOW I use my EVDO modem for more than 5G per month. It is my primary Internet connection, rather than cable or DSL, and I do a lot of file transfers as part of my work. To make Sprint’s suckage even worse, I can’t actually view my usage details on-line; they don’t appear to support that for data cards.

It was fun while it lasted, but I’ll be forced to cancel before my 60 day window closes on July 23rd.

Maybe iPhone 2.0 will finally have the ability to tether to my laptop so I can use it as a network connection? Why don’t I see that on any of the wishlists that make it to Digg?

-Chris

Dear Chris,

Thank you for your reply.

I sincerely apologize for the inconvenience you have experienced in this regard.

I would like to share with you that Sprint is changing the terms and conditions for Mobile Broadband Connection Plans, Data Access Plans, and Phone as Modem Plans to include usage limitations. This change goes into effect for all customers as of 7/13/08.

The specific change adds the following limitation:

We have reserves the right to limit throughput speeds or amount of data transferred, and to deny, terminate, modify, disconnect or suspend service if usage either exceeds 5GB/month in total, or 300MB/month while roaming. If a customer exceeds the data amounts that Sprint has specified, we may modify usage behaviors, terminate, suspend, or disconnect the customer?s service.

We believes that all of our customers should get the best service possible from their broadband. That means speedy downloads for all users, not just a few.

When someone is downloading a particularly large amount of information over a long period of time, repeatedly, it can slow down speeds for other users who might just be checking their e-mail or browsing online.

So to make sure our service is fair for everybody, we are encouraging the right kind of use on the network.

We are notifying customers of this change in usage limitations via a bill message, it allows customers the opportunity to review their data. You may opt out of their Sprint service with no penalty on the line of service impacted by this change if you are not in agreement with the change. You could opt out of their Sprint contract within 60 days based on the day you receive their bill message regarding the change in data usage

I am sure the above details have addressed your concern. If you have any other questions or concerns, please feel free to contact us. It will be my pleasure to address your concern in best possible way.

Further, I would like to inform you that the account will not be cancelled, it is forwarded to cancellation departmnet to give the explaination for your query.

Thank you again for contacting Sprint. We appreciate your business.

Ivan F.
E-Care
Sprint

[ad#adsense-horizontal]

Idiocracy in the Insect Kingdom

Stupid flies live longer: study – Yahoo! News
Wed Jun 4, 8:21 AM ET

GENEVA (AFP) – It doesn’t pay to be smart and ignorance really is bliss if you want a long life — at least if you’re a fly, according to new research by a Swiss university.

Scientists Tadeusz Kawecki and Joep Burger at the University of Lausanne said Wednesday they had discovered a “negative correlation between an improvement in a fly’s mental capacity and its longevity”.

As part of their research project, the results of which are published in the journal Evolution, they divided into two a group of flies from the Basel region of northwestern Switzerland.

One half was left in a natural state while the other had its intelligence boosted by Pavlovian methods, such as associating smell and taste with particular food or experiences.

Over 30 to 40 generations, these methods led to flies which clearly learned better and remembered things for longer.

The flipside was that the flies left in their natural state lived longer on average than their “cleverer” counterparts, with a lifespan of 80-85 days rather than the normal 50-60..

“In other terms, the more the fly becomes intelligent, the shorter its lifespan,” the scientists said.

This is most probably because the increase in neural activity weakens the fly’s life-support systems, they speculated.

“This would explain why flies, like most other animals, have hardly developed their neural capacities,” they said.

[ad#adsense-horizontal]

I use Amazon affiliate links in some of my posts. I think it is fair to say my writing is not influenced by the $0.40 I earned in 2022.