Tracking BSTR Memory Allocations to Find Memory Leaks

• Download demo project - 4.3 KB

Introduction

If you find yourself needing to track memory leaks related to memory that was allocated using SysAllocString / SysAllocStringLen and you don't have a leak detection tool that can work with your application without crashing / hanging, this tip can help you verify whether or not you found a leaked memory. The tip assumes you are familiar with windbg at least in basic level to debug a process. 

Background

I had to find a memory leak in a process that reached its maximum memory limit as the process is 32 bit and thus limited to 2 GB (can be stretched to 3 GB on 32 bit OS and to 4 GB on 64 bit OS).

I tried using any known tool for the evaluation but had no success to launch the process and monitor its allocations. Some tools just failed to launch while others crash or hang. It was clear that there is no other way of finding the leaks but to use windbg.

After finding the memory allocations that grow the most, I have noticed they are all BSTR strings and had to find the allocation points in order to understand where it was not freed.

BSTR Allocation Tracking

After setting gflags to store the stack traces for the memory allocations, I have run the process again and got the stack trace of the BSTR allocations that were marked as a potential leak.
I searched for the point where these strings were freed and noticed that although there is a call to SysFreeString, the memory is still marked as busy (using winbg command !heap -x <memory address>).
When printing the stack trace, I got a result that showed allocation point that seems like no longer relevant. After "Googling", I have found that SysAllocString may use cache for the allocated memory for BSTR and thus even when you "free" the memory, it is not really free and will probably be returned in the next call to SysAllocString assuming the size is adequate.
This caching behavior might be confusing for searching memory leaks as the stack trace being stored for the memory is the stack trace for the first time this memory was allocated which might not be relevant.

In order to get the correct stack trace of the memory allocation during debugging, you need to define an environment variable named OANOCACHE and set its value to 1.
Restart the debugger so that it will get the updated environment variable's value and run the program being traced. After every call to SysFreeString, you will now see the memory appears as free and printing the stack trace of the memory allocation will show the correct stack trace for each BSTR allocation.

If setting the environment variable doesn't affect the memory caching behavior, a reboot will be needed. 

Code that was Used to Examine the Described Behavior 

#include "stdafx.h"

void AllocateFirstBstr()
{
	BSTR firstStr = SysAllocString(L"The first string to be free");
	SysFreeString(firstStr);
}

void AllocateSecondBSTR()
{
	BSTR secondStr = SysAllocString(L"Second string to be free");
	SysFreeString(secondStr);
}


int _tmain(int argc, _TCHAR* argv[])
{

	AllocateFirstBstr();

	AllocateSecondBSTR();

	return 0;
}

Running the Code under windbg with Caching Enabled (Default Behavior)

Inside method AllocateFirstBstr after the call to SysAllocString 

0:000> dv
       firstStr = 0x004ae2f4 "The first string to be free"
    
    0:000> !heap -p -a 0x004ae2f4 
    address 004ae2f4 found in
    _HEAP @ 460000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        004ae2d8 000b 0000  [00]   004ae2f0    00040 - (busy)
        7704df72 ntdll!RtlAllocateHeap+0x00000274
        74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
        74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
        74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
        74aa4677 OLEAUT32!SysAllocString+0x0000002c
        105141b DemoAppForBSTRAllocs!AllocateFirstBstr+0x0000002b
        1051473 DemoAppForBSTRAllocs!wmain+0x00000023
        1051a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
        1051c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
        7679336a kernel32!BaseThreadInitThunk+0x0000000e
        7700bf32 ntdll!__RtlUserThreadStart+0x00000070
        7700bf05 ntdll!_RtlUserThreadStart+0x0000001b

Inside method AllocateFirstBstr after the call to SysFreeString 

0:000> !heap -x 0x004ae2f4
    Entry     User      Heap      Segment       Size  PrevSize  Unused    Flags
    ---------------------------------------------------------------------------
    004ae2d8  004ae2f0  00460000  004a3190        58      -           18  LFH;busy  stack_trace

The memory is marked as busy although a call to SysFreeString was made.

Inside method AllocateSecondBSTR after the call to SysAllocString 

0:000> dv
      secondStr = 0x004ae2f4 "Second string to be free"
0:000> !heap -p -a 0x004ae2f4 
    address 004ae2f4 found in
    _HEAP @ 460000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        004ae2d8 000b 0000  [00]   004ae2f0    00040 - (busy)
        7704df72 ntdll!RtlAllocateHeap+0x00000274
        74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
        74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
        74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
        74aa4677 OLEAUT32!SysAllocString+0x0000002c
        105141b DemoAppForBSTRAllocs!AllocateFirstBstr+0x0000002b
        1051473 DemoAppForBSTRAllocs!wmain+0x00000023
        1051a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
        1051c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
        7679336a kernel32!BaseThreadInitThunk+0x0000000e
        7700bf32 ntdll!__RtlUserThreadStart+0x00000070
        7700bf05 ntdll!_RtlUserThreadStart+0x0000001b

In the above stack trace, it can be seen that the memory allocation shows the first time the memory was allocated and not the actual allocation point which is in method AllocateSecondBSTR.

Inside method AllocateSecondBSTR after the call to SysFreeString 

0:000> !heap -x 0x004ae2f4 
Entry     User      Heap      Segment       Size  PrevSize  Unused    Flags
----------------------------------------------------------------------------
004ae2d8  004ae2f0  00460000  004a3190        58      -           18  LFH;busy  stack_trace

Running the Code under windbg Without Caching

Inside method AllocateFirstBstr after the call to SysAllocString 

0:000> dv
       firstStr = 0x004fe33c "The first string to be free"
0:000> !heap -p -a 0x004fe33c 
    address 004fe33c found in
    _HEAP @ 4b0000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        004fe320 000b 0000  [00]   004fe338    00040 - (busy)
        7704df72 ntdll!RtlAllocateHeap+0x00000274
        74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
        74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
        74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
        74aa4677 OLEAUT32!SysAllocString+0x0000002c
        126141b DemoAppForBSTRAllocs!AllocateFirstBstr+0x0000002b
        1261473 DemoAppForBSTRAllocs!wmain+0x00000023
        1261a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
        1261c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
        7679336a kernel32!BaseThreadInitThunk+0x0000000e
        7700bf32 ntdll!__RtlUserThreadStart+0x00000070
        7700bf05 ntdll!_RtlUserThreadStart+0x0000001b

Inside method AllocateFirstBstr after the call to SysFreeString 

        0:000> !heap -x 0x004fe33c 
Entry     User      Heap      Segment       Size  PrevSize  Unused    Flags
-----------------------------------------------------------------------------
004fe320  004fe328  004b0000  004f3190        58      -            0  LFH;free

When caching is disabled, the memory is marked as free after calling SysFreeString.

Inside method AllocateSecondBSTR after the call to SysAllocString 

0:000> dv
      secondStr = 0x004fe33c "Second string to be free"
    0:000> !heap -p -a 0x004fe33c 
    address 004fe33c found in
    _HEAP @ 4b0000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        004fe320 000b 0000  [00]   004fe338    00040 - (busy)
        7704df72 ntdll!RtlAllocateHeap+0x00000274
        74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
        74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
        74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
        74aa4677 OLEAUT32!SysAllocString+0x0000002c
        1261b0b DemoAppForBSTRAllocs!AllocateSecondBSTR+0x0000002b
        1261478 DemoAppForBSTRAllocs!wmain+0x00000028
        1261a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
        1261c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
        7679336a kernel32!BaseThreadInitThunk+0x0000000e
        7700bf32 ntdll!__RtlUserThreadStart+0x00000070
        7700bf05 ntdll!_RtlUserThreadStart+0x0000001b

Now, the correct stack trace is being shown so that you can find where exactly the memory was allocated.

Inside method AllocateSecondBSTR after the call to SysFreeString 

0:000> !heap -x 0x004fe33c 
Entry     User      Heap      Segment       Size  PrevSize  Unused    Flags
-----------------------------------------------------------------------------
004fe320  004fe328  004b0000  004f3190        58      -            0  LFH;free

Note: Don't forget to remove the environment variable after finding the leak as it affects the behavior for all other programs as well.