1365.aspx

How to track down performance problems in COM+ applications

Someone asked me yesterday if the COM+ Call Time Tracker can be used dig deeper into the component to see which function is causing the high call times. It is an interesting subject, so I am posting the reply on the blog.

The answer is no, COM+ only gives call information on an object level. It does not give information about internal information like function call times. This means that the COM+ objects must be split into smaller COM+ objects you want to track down the problem with COM+ call time tracking. It is an ugly solution, and only works on a "macro" level as it is not feasible to make all the functions separate COM+ objects. There are better alternatives which I cover below.

First of all; keep in mind that call time is not the same a CPU time. The call time shows how long COM+ has been waiting for a the object to return. The object may be busy:

• calculating something (using a lot of CPU),


• writing something to disk


• waiting for an external resource like a Web Service or a query to a DB


• waiting for a lock on a shared resource

- Task Manager -
This is the first think I have a look at if I have problems with high call times in COM+. It is already installed on the machine and it gives an idea what is going on.

Things to check in the "Performance Tab":

• CPU Usage. It should be lower than 50% on average, never over 75% for long periods of time


• Commit Charge must be lower than Physical Memory, otherwise the machine does not have enough memory and will swap. Everything slows down when the machine swaps so you will get high call times in all components..

Then I look at the process running the COM+ Application. The COM+ Call Time Tracker already gives you the process ID in /applications/application/processID but you can also get it from Component Services. I usually check the following information in Task Manger for the Process ID; CPU, Mem Usage, Peak Mem Usage, VM Size, Handles

Some quick rules of thumb:

• High CPU usage is pretty obvious. Improve your code or get more/faster CPUs


• VM Size > Mem Usage means a memory leak. The machine starts swapping out memory and everything slows down


• Handles:  Should be stable. An every increasing number means a handle leak in the component (usually references to external objects/files that are not closed)


• If Peak Mem Usage is a lot larger than "Mem Usage", there are periods when the component uses a lot of memory

- Performance Monitor -
Performance Monitor (perfmon.exe) is a very powerful tool if you know how to interpret the results. It can be used to monitor most resources on the machine, often on a process by process basis. It allows you to find the overall bottle neck: cpu, disk, memory, network, web worker processes etc.

- Process Explorer -
The free Process Explorer by Sysinternals is a fantastic tool. Consider it Task Manager on steroids enhanced with Performance Monitor data. The latest version tells you everything you ever wanted to know about the processes running on your PC; how many threads they have, how much CPU each thread is using, which DLLs it is using etc.

One thing to check for .NET applications: The “% time in GC” counter below shows how much of the applications CPU time is spent on garbage collection. It must be less than 4-5% on average. Higher values means .NET spends a lot of time garbage collecting objects that have not been closed or disposed properly. I have seen cases of >70% cpu usage for garbage collection due to bugs in the code.

 

- Logging -
A good log is crucial when troubleshooting any problem, including performance. I usually use log4net but any other logging framework that works in a multi threaded environment  is OK. Use the different log levels to control how much is logged as detailed logs will slow the system. You can use a trace/info level to log function entry/exit points, so you can see which function grabs the time, but log input/ouput and more verbose data at the more detailed debug log level.

- Code Profiler -
Code Profilers monitor the execution of a process and tells you afterwards how many times the different functions were called, how long it took etc. Code Profilers usually run on a development or test PC so you have to simulate the calls you make in a production environment using a dedicated tester.

There are several professional tools on the market like ANTS Profiler by Red Gate and DevPartner Studio by Compuware

So far I have never had to use one as I have managed to track down bugs with a combination of performance counters, logs and code reviews.