.NET Zone is brought to you in partnership with:

Sasha Goldshtein is a Senior Consultant for Sela Group, an Israeli company specializing in training, consulting and outsourcing to local and international customers.Sasha's work is divided across these three primary disciplines. He consults for clients on architecture, development, debugging and performance issues; he actively develops code using the latest bits of technology from Microsoft; and he conducts training classes on a variety of topics, from Windows Internals to .NET Performance. You can read more about Sasha's work and his latest ventures at his blog: http://blogs.microsoft.co.il/blogs/sasha. Sasha writes from Herzliya, Israel. Sasha is a DZone MVB and is not an employee of DZone and has posted 203 posts at DZone. You can read more from them at their website. View Full User Profile

Announcing Tracer: A Generic Way to Track Resource Usage and Leaks

09.20.2013
| 1733 views |
  • submit to reddit

Tracer is a WinDbg extension I wrote last month to diagnose a resource leak that is not covered by well-known facilities like !htrace or UMDH. Tracking any resource leak starts with understanding where you are acquiring the resource and neglecting to release it – and with Tracer, you can do this for any kind of resource.

Download Tracer and review its source code.

The basic process of hunting for resource leaks is quite simple. For example, consider what UMDH does on your behalf. UMDH enables support in the operating system (specifically, in the Heap Manager that resides in ntdll.dll) to capture stack traces of heap memory allocations and correlate the allocation requests with deallocation (free) requests. If the application exhibits a pattern of leaking memory, you will see specific call stacks reported as responsible for memory allocations that have not been freed. And – even though you don’t know for sure who was supposed to free that memory – you are given a good clue in the form of the piece of code that performed the memory allocation.

With Tracer, you can perform this process automatically for any kind of resource. Tracer exposes two extension commands for tracking resource usage – !traceopen and !traceclose, and an additional extension command for displaying reports of resource leaks – !tracedisplay. You typically call the tracking commands from breakpoints strategically positioned where you acquire and release the resource you’re tracking.

Let’s take a look at an example. Suppose that you’re tracking a database connection leak. You create database connections in the db_connect function, and close them in the db_close function. To track a database connection leak, place breakpoints in these functions and call the tracking commands from these breakpoints, as follows:

bp dbmodule!db_connect "gu; !traceopen @eax; gc" 
bp dbmodule!db_close "!traceclose poi(@esp+4); gc"

The preceding commands assume that the db_connect function returns the database connection handle, and that the db_close function takes the database connection handle as a parameter on the stack (if you have private symbols for db_close, you can of course reference the parameter name instead of relying on its stack location).

Now you can run your application and wait for the suspicious leak to accumulate. Whenever you'd like, you can break into the debugger and inspect the current status of the specific resource you're tracking using the !tracedisplay -stats command. This will dump out statistics (in sorted order) for the call stacks acquiring database connections without subsequently closing them. Here is a fictituous example:

0:000> !tracedisplay -stats
Total objects with events       : 0n34
Total events with stack traces  : 0n34

----- STACK #1 OPEN=0n30 CLOSE=0n0 OTHER=0n0 WEIGHT=0n30 -----
    MyApp!OpenDatabaseConnection+0x3c
    MyApp!LogWriterThread+0x4b
    KERNEL32!BaseThreadInitThunk+0xe
    ntdll!__RtlUserThreadStart+0x72
    ntdll!_RtlUserThreadStart+0x1b

----- STACK #2 OPEN=0n4 CLOSE=0n0 OTHER=0n0 WEIGHT=0n4 -----
    MyApp!OpenDatabaseConnection+0x3c
    MyApp!InitializationThread+0x60
    KERNEL32!BaseThreadInitThunk+0xe
    ntdll!__RtlUserThreadStart+0x72
    ntdll!_RtlUserThreadStart+0x1b

Often, the resources you acquire and release have weights associated with them - not every instance of the resource will have the same cost. This obviously applies to memory (large allocation leaks are of more concern than small ones), but might also apply to other kinds of resources. To tell Tracer that your resources have weights, use the !traceopen <object> <weight> command. For example, when tracing memory leaks through RtlAllocateHeap, you can use the following command (on x86):

bp ntdll!RtlAllocateHeap "r $t0 = poi(@esp+0n12); gu; !traceopen @eax @$t0; gc"

As a result, Tracer will display object weights and will sort the statistics output by weights so you can easily identify call stacks responsible for large resource leaks and fix them first.

It’s worth mentioning that Tracer can be used to track invalid resource usage as well as resource leaks. For example, you could use Tracer to immediately know when you are releasing a resource that was not previously acquired by your code. If that happens, Tracer will output a diagnostic message to the WinDbg console. Also, you can use !traceclose -keepstack <object> to indicate that you want to retain the close call stack even if the object's allocation-deallocation delta is now 0. This helps understand where the object was created and destroyed after the fact. Just use the !tracedisplay <object> command to view information about that particular object.

A final note about performance is due here. As you can see, Tracer relies on breakpoints that call the tracking extension commands. This is - without doubt - the most expensive part of the tracking mechanism. For memory allocation tracking in large applications, UMDH is a much better idea because it does not require a debugger and does not rely on breakpoints. Off the top of my head I'd say that if you're acquiring and releasing resources more than 100 times per second, you will probably see a considerable slowdown when tracking them with Tracer. For many kinds of resources, though, this is a reasonable limit - I have used Tracer with file handles, sockets, database connections, and database transactions.

I am posting short links and updates on Twitter as well as on this blog. You can follow me:@goldshtn

Published at DZone with permission of Sasha Goldshtein, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)