Threads and Ants

• Download source - 111.2 KB

Introduction

ThreadsAndAnts.exe; hey, that's your app that's crashing!

But what's instruction 0x74DD3E28?

After investigating, you find that it's an instruction in a third-party dll, which is useless to you.  You need to find out what code in your app is calling this dll, and causing the exception - you need the exception's call stack: the sequence of function calls that lead to this exception.

You can configure your registry to save crash dumps, or add code to your app to create mini-dumps.  But you need DbgHelp.dll, which you don't have on the embedded computers that your app is running on.  Plus, even if you installed it, your code doesn't support downloading dump files; it only supports event log downloads.

You can log the call stacks in your event log, using the StackWalk64 function, but this also requires DbgHelp.dll, and your embedded computers have limited resources, so you can't install the PDB (Program DataBase) files that DbgHelp.dll needs in order to give you a call stack's function names and offsets; you have to get the function names and offsets after you download the event logs.

After investigating some more, you find a solution that doesn't require you to use DbgHelp.dll (getting the call stack for 32-bit apps is relatively easy to do without it) or intstall PDB files.

Your app logs the following message just before it crashes:

It gives you 0x000043F0, the last instruction in your app before the 0x74DD3E28 instruction, but what function is it in?

You can get a function name and an offset for this address by looking it up in your app's MAP file, or you can create an Excel add-in to do it for you; to read your app's MAP file and replace 0x000043F0 with a function name and an offset:

It replaces 0x000043F0 with the CThreadsAndAntsDlg class, the OnThrowUnhandledExceptionButton function, and an offset of 80 (instruction) bytes (from the top of the function).

If you can't figure out what line of code in this function threw the exception, you can try throwing exceptions at different places in this function, for example, by writing to a null pointer: int* pi = NULL; *pi = 0; until you get an offset close to 80 bytes. It won't be exact, because 1 line of code can contain multiple instructions, and because you're adding instructions to throw the execption, but this will help you narrow it down.

You've found what's causing your exception.

Getting Started

1. Download this article's source code to your computer and un-zip it.
    
2. Tell Excel to load the Call Stack.xlam add-in every time it starts, by selecting Developer > Excel Add-Ins > Browse... > selecting add-in > OK > checking add-in > OK.
   
   If you do not see the Developer tab in Excel, you can show it by selecting File > Options > Customize Ribbon > checking Developer > OK.
   
   If Excel asks you to enable the add-in every time it starts, you can add its location to your trusted locations by selecting File > Options > Trust Center > Trust Center Settings... > Trusted Locations > Add new location... > selecting location > OK > OK.
   
   This add-in adds a Call Stack button to your Add-ins tab, which reads the ThreadsAndAnts.map file and adds its function names and offsets to your log file.

1. Open ThreadsAndAnts.vcxproj with Visual Studio 2015 or later, and build it.
   
   Or create a project using the ThreadsAndAnts files, and an older version of Visual Studio, or a different compiler, and build it.
   
   Or use the exe and map files provided in this article's zip file.
    
2. Run ThreadsAndAnts.exe.

1. Add CallStack.cpp and .h and HmoduleArray.cpp and .h to your app, or to a library that your app uses.
    
2. If your app encounters a problem, get the problem thread's call stack, by calling one of the CCallStack class' GetCallStack functions, and log it.
   
   For example, call SetUnhandledExceptionFilter when your app starts up, to be notified if your app throws an unhandled exception; for an example, see ThreadsAndAnts.cpp and .h.
   
   You can add SetUnhandledExceptionFilter and the other problem notification functions to a library, if all of your apps call a common function in that library when they start up; you don't have to add them to each of your apps.
    
3. Generate a map file for your app; if you're using Visual Studio, you can do this by going to your project's Configuration Properties, Linker > Debugging > Generate Map File = Yes.
   
   When you release a new version of your app, save your app's map file.
   
   And log your app's version in your event log, for example, when your app starts up or when you log a call stack, so you know what version of the map file to use with a particular event log or call stack.
    
4. Download your event log; assuming it's a csv file, open it in Excel, copy your map file (or files) to the csv file's directory, and click the Call Stack button in your Add-ins tab; it will add the function names and offsets from your map file (or files) to your event log.
   
   The Excel add-in assumes that your map file names (e.g. ThreadsAndAnts.map) are the same as your exe file names (e.g. ThreadsAndAnts.exe), and that your log message text is in Excel's column C.
   
   You can change the Excel add-in code to meet your specific needs, for example, you can change the log message text column constant - MESSAGE_COLUMN - if your log message text isn't in column C.
   
   To get to the Excel add-in code, open Excel and go to the Developer tab > Visual Basic button > VBAProject (Call Stack.xlam) > Modules > Module1.  
   
   You can also change the Call Stack button image and tool tip text.

One Thread, One Ant

Imagine that you’re an ant.

You and your ant colony are gathering food – leaves – and carrying them back home.

But a leaf it too big and heavy for you to carry all by yourself.

So you cut it into 3 smaller pieces.

You can carry each piece, one at a time.

Or you can get 2 other ants to help you, and carry all of the pieces back home faster.

A thread is like you, and like the other 2 ants that are helping you; together you are 3 workers, or 3 threads.

Only for threads, the computer does one thing for a while, then it puts it down and does another thing for a while, and so on.

If you click on the Add Thread and Ant button in the test app, it will start a new CAntThread thread, which contains a CAntDlg dialog, which plays an ant animation.

Using C++ and MFC (Microsoft Foundation Classes), you can start a new thread using AfxBeginThread.

//////////////////////////////////////////////////////////////////////////////
void CThreadsAndAntsDlg::OnAddThreadAndAntButton( void )
{
    CAntThread* pAntThread = (CAntThread*) AfxBeginThread( RUNTIME_CLASS( CAntThread ) );
    ...
}

And you can end the thread, when you no longer need it, using TerminateThread.

//////////////////////////////////////////////////////////////////////////////
CThreadsAndAntsDlg::~CThreadsAndAntsDlg( void )
{
    ...

    // For all CAntThreads (i = 0 to N):
    // Tell the thread to end, and wait for it to end.
    TerminateThread( m_aAntThreads[i]->m_hThread, 0 );
    ...

    WaitForSingleObject( m_aAntThreads[i]->m_hThread, INFINITE );
    ...
}

Using C++ and STL (Standard Template Library) you can start a new thread using std::thread, and end it using join.

You may create as many threads as you like in your app (within reason; your computer doesn't have infinite resources).

Mindfulness

Being a mindful ant, with an excellent memory, you’re aware of both yourself and your surroundings, and you don't get lost; you can retrace your steps all the way back home.

And you always take the safe way home, by travelling exactly the reverse path that got you to where you’re at.

No shortcuts through the dark woods for this little ant!

A thread does the same.

Within its current function, it knows:

1. The instruction address that it’s at.
    
2. The parameters that were passed to the function, and
    
3. The function's local variable values.

Together, this information is called a frame.

A thread’s call stack contains one frame for each function that it’s been to but hasn’t yet returned from.

If one function calls a second function then the thread keeps the first function's frame (the instruction address that it jumped from, the parameters that were passed to the function, and the function's local variable values) in the thread’s call stack, so it can get back to the function later, and continue where it left off.

And it adds a frame for the function that was just called.

The current function knows how to get back to the function that called it, and that function to the function that called it, and so on, all the way back to the thread's starting function.

The call stack is a chain, a linked list of function calls (and function parameters, and local variable values).

A thread’s context always points to the thread's current frame, the frame of the function it's in right now, and this can be used to follow the linked list of frames all the way back to the thread's starting frame.

Using this article's code, you can get a thread's call stack by calling one of the CCallStack class' GetCallStack functions; they return the call stack as a string which you can then log to your event log.

If you click on the Log this App's Call Stacks button.

//////////////////////////////////////////////////////////////////////////////
// Called when the 'Log This App's Call Stacks' button is clicked.
void CThreadsAndAntsDlg::OnLogThisAppCallStacksButton( void )
{
    ...

    // Get a handle to the currently executing thread.
    HANDLE hThread = AfxGetThread()->m_hThread;

    // Get the thread's call stack.
    CCallStack cs;
    CString sCallStack = cs.GetCallStack( hThread );
    ...
}

//////////////////////////////////////////////////////////////////////////////
// Returns the call stack of a thread within this app.
CString CCallStack::GetCallStack( HANDLE hThread )
{
    m_hProcess = ::GetCurrentProcess();

    // Duplicate hThread and save the result in m_hThread.
    BOOL bDuplicateHandle = DuplicateHandle(
        m_hProcess,
        hThread,
        m_hProcess,
        &m_hThread,
        0,
        FALSE,
        DUPLICATE_SAME_ACCESS );

    if ( bDuplicateHandle == TRUE )
    {
        GetCallStack();
    }

    return m_sCallStack;
}

If you click on the Open Log File button, and wait for Excel to open the log file, and go to Excel's Add-ins tab and click on the Call Stack button, to fill in the class and function names and offsets, you'll see something like this:

What does this call stack tell you?

1. The list of functions that were called in your app, starting with _wWinMainCRTStartup.
    
2. CThreadsAndAntsDlg::OnLogThisAppCallStacksButton got this call stack, at its 226th (instruction) byte (from the top of the function).
    
3. The dlls and dll address offsets that your app called.
   
   In some cases you may be able to open the dll in Dependency Walker and find the function names at these address offsets.  But if it's not your dll, and you don't have its source code, this probably won't be helpful.

If you click on the Log Other App's Call Stacks button, select one or more processes in the list, and click OK.

In this article's app, we use EnumProcesses to get the process IDs of all of the processes running on your computer, and we use these process IDs to get handles to the processes and their threads.

//////////////////////////////////////////////////////////////////////////////
void CProcessListDlg::GetCallStacks( ..., DWORD dwProcessID )
{
    ...
    
    CCallStack cs;
    CString sCallStack;
    CString sMessage;
    HANDLE hThread = NULL;
    HANDLE hSnapshot = NULL;
    THREADENTRY32 te = {};

    // Get a handle to the process.
    HANDLE hProcess = OpenProcess( PROCESS_ALL_ACCESS, FALSE, dwProcessID );

    if ( hProcess != NULL )
    {
        // Take a snapshot of the process' threads.
        hSnapshot = CreateToolhelp32Snapshot( TH32CS_SNAPTHREAD, dwProcessID );

        if ( hSnapshot != INVALID_HANDLE_VALUE )
        {
            te.dwSize = sizeof( THREADENTRY32 );

            if ( Thread32First( hSnapshot, &te ) == TRUE )
            {
                do
                {
                    // If the thread belongs to this process.
                    if ( te.th32OwnerProcessID == dwProcessID )
                    {
                        // Get a handle to the thread.
                        hThread = OpenThread( THREAD_ALL_ACCESS, FALSE, te.th32ThreadID );

                        if ( hThread != NULL )
                        {
                            sCallStack = cs.GetCallStack( hProcess, hThread );
                            ...

                            CloseHandle( hThread );
                        }
                    }

                } while ( Thread32Next( hSnapshot, &te ) == TRUE );
            }

            CloseHandle( hSnapshot );
        }

        CloseHandle( hProcess );
    }
}

//////////////////////////////////////////////////////////////////////////////
// Returns the call stack of a thread within another app.
CString CCallStack::GetCallStack( HANDLE hProcess,
                                  HANDLE hThread )
{
    m_hProcess = hProcess;
    m_hThread = hThread;

    GetCallStack();

    return m_sCallStack;
}

Whether you're getting the call stack of a thread in the current app or another app, the process handle is used to read the process' memory, and the thread handle is used to get the thread's context.

But first, the thread needs to be paused so the call stack doesn't change while we're getting it.

//////////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////////
void CCallStack::DoDumpStack( void )
{
    // Pause the thread.
    SuspendThread( m_hThread );

    CONTEXT context = {};
    memset( &context, 0, sizeof( context ) );
    context.ContextFlags = CONTEXT_FULL;

    // Get the thread's context.
    GetThreadContext( m_hThread, &context );

    // Get the thread's call stack.
    DoDumpStack( &context );

    // Resume the thread.
    ResumeThread( m_hThread );
}

32-bit Apps

The thread's context contains the thread's current frame; this is where we start.

In each frame, the return address will always be at the EBP (Base Pointer) address + 4 bytes, and the EBP address' value will always contain the previous frame's EBP address.

#if defined _M_IX86
//////////////////////////////////////////////////////////////////////////////
// Gets the call stack of 32-bit apps.
void CCallStack::DoDumpStack32( const CONTEXT* pContext )
{
    BOOL bReadProcessMemory = FALSE;
    DWORD* pdwEbp = (DWORD*) pContext->Ebp;
    DWORD dwPrevEbp = 0;
    DWORD dwReturnAddress = 0;
    SIZE_T stNumberOfBytesRead = 0;
    ...

    // If we reach an EBP (Base Pointer) of NULL (0),
    // this means that we've walked off the end of the call stack and are done.
    for ( int iFrame = 0; ( iFrame < eMAX_FRAMES ) && ( pdwEbp != NULL ) ; iFrame++ )
    {
        // The return address will always be at EBP + 4 (bytes).
        // '+ 1' is the same as adding 'sizeof( DWORD* )', which is 4 bytes.

        // The ReadProcessMemory function allows us to read another app's memory,
        // so we can get the call stacks of theads in other apps (on this computer).
        bReadProcessMemory = ReadProcessMemory(
            m_hProcess,                // [in]  HANDLE  hProcess
            pdwEbp + 1,                // [in]  LPCVOID lpBaseAddress
            &dwReturnAddress,          // [out] LPVOID  lpBuffer
            sizeof( dwReturnAddress ), // [in]  SIZE_T  nSize
            &stNumberOfBytesRead );    // [out] SIZE_T* lpNumberOfBytesRead

        if (    ( bReadProcessMemory == FALSE )
             || ( stNumberOfBytesRead < sizeof( dwReturnAddress ) )
             || ( dwReturnAddress == NULL ) )
        {
            break;
        }

        m_sCallStack += GetAddressInfo( (PVOID) dwReturnAddress );

        // The Base Pointer address' value contains the previous Base Pointer address.
        bReadProcessMemory = ReadProcessMemory(
            m_hProcess,             // [in]  HANDLE  hProcess
            pdwEbp,                 // [in]  LPCVOID lpBaseAddress
            &dwPrevEbp,             // [out] LPVOID  lpBuffer
            sizeof( dwPrevEbp ),    // [in]  SIZE_T  nSize
            &stNumberOfBytesRead ); // [out] SIZE_T* lpNumberOfBytesRead

        if (    ( bReadProcessMemory == FALSE )
             || ( stNumberOfBytesRead < sizeof( dwPrevEbp ) ) )
        {
            break;
        }

        pdwEbp = (DWORD*) dwPrevEbp;
    }
    ...
}
#endif

To find the module (exe or dll) that the return address is in, and its offset from the module's base address, we use the CHmoduleArray class (instead of DbgHelp.dll's SymGetModuleInfo64 function).  CHmoduleArray uses the GetModuleHandleEx function to get the module handles (base addresses) for the current process (app), and the EnumProcessModulesEx function to get the module handles for other processes.

64-bit Apps

Gettting the call stack of a thread within a 64-bit app isn't as simple as getting it for a thread within a 32-bit app; but you can get it using one of the following methods.

1. The DbgHelp.dll's StackWalk64 function.
    
2. The DIA SDK (Debug Interface Access Software Development Kit).
   
   To use the DIA SDK, you have to obtain the IDiaStackWalker interface, use the IDiaEnumStackFrames interface to get the call stack frames, and create your own IDiaStackWalkHelper derived class.
   
   There's an example IDiaStackWalkHelper derived class that you can use, but it's for 32-bit apps, so you have to update its get_registerValue function to return your thread context's 64-bit registers, like Rip.
   
   And you have to update its pdataForVA function to return an IMAGE_RUNTIME_FUNCTION_ENTRY_VA structure - this may be difficult - I wasn't able to find any example code for it.
    
3. The RtlLookupFunctionEntry and RtlVirtualUnwind functions; but you can only get call stacks for threads within the current app.
    

The CCallStack class uses the StackWalk64 function to get the call stacks of threads within 64-bit apps.

Insects and Bugs

You can set a function to be called if your app throws an unhandled exception.

//////////////////////////////////////////////////////////////////////////////
CThreadsAndAntsApp::CThreadsAndAntsApp( void )
...

{
    // Sets the function to be called when the the unhandled exception filter
    // detects an unhandled exception.
    // Exception: the function is not called if this app is being debugged.
    SetUnhandledExceptionFilter( MyUnhandledExceptionFilter );
    ...
    
}    

Unhandled exceptions will still crash your app, but first your MyUnhandledExceptionFilter function will be called, and it will log a message containing the exception's call stack; the funtion calls that caused the exception.

//////////////////////////////////////////////////////////////////////////////
LONG WINAPI CThreadsAndAntsApp::MyUnhandledExceptionFilter( struct _EXCEPTION_POINTERS* pExceptionInfo )
{
    PEXCEPTION_RECORD pExceptionRecord = pExceptionInfo->ExceptionRecord;
    CCallStack cs;
    CString sMessage;

    CString sCallStack = cs.GetCallStack( pExceptionInfo->ContextRecord );

    // pExceptionInfo->ContextRecord doesn't include the frame that caused
    // the exception - pExceptionRecord->ExceptionAddress - so add it.

    CString sExceptionAddress = cs.GetAddressInfo( pExceptionRecord->ExceptionAddress );

    if ( pExceptionRecord->ExceptionCode == EXCEPTION_ACCESS_VIOLATION )
    {
        sMessage.Format( _T( "Application '%s' instruction at address '0x%p' 
            caused an exception while trying to '%s' memory at address '0x%p'; call stack : %s." ),
            GetAppName(),
            pExceptionRecord->ExceptionAddress,
            aExceptionInformationStrings[pExceptionRecord->ExceptionInformation[0]],
            pExceptionRecord->ExceptionInformation[1],
            sExceptionAddress + sCallStack );
        GetLogFile()->LogMessage( sMessage );
    }
    else
    {
        sMessage.Format( _T( "Application '%s' instruction at address '0x%p' 
            caused an exception of type '0x%X'; call stack : %s." ),
            GetAppName(),
            pExceptionRecord->ExceptionAddress,
            pExceptionRecord->ExceptionCode,
            sExceptionAddress + sCallStack );
        GetLogFile()->LogMessage( sMessage );
    }

    return EXCEPTION_EXECUTE_HANDLER;
}

The _EXCEPTION_POINTERS parameter that is passed to your MyUnhandledExceptionFilter function contains information about the exception.

For example, it contains the context of the thread that threw the unhandled exception, and the exception type (code).

The exception information may be different for different types of exceptions.

For example, for an exeption of type EXCEPTION_ACCESS_VIOLATION, the first element of the ExceptionInformation array contains a read-write flag that indicates the type of operation that caused the access violation.

You can throw an unhandled exception by clicking on the Throw Unhandled Exception button.

It will crash the app, but first it will call MyUnhandledExceptionFilter and log the call stack of the thread that threw the unhandled exception.

//////////////////////////////////////////////////////////////////////////////
// Called when the 'Throw Unhandled Exception' button is clicked.
void CThreadsAndAntsDlg::OnThrowUnhandledExceptionButton( void )
{
    // Write to a NULL pointer.
    int* pi = NULL;
    *pi = 0;
}

The following are a few of the functions you can use to be notified when your app encounters a problem.

There are examples of each of these functions in CThreadsAndAntsApp, and there are buttons in the test app's dialog that trigger them.

You can log call stacks for more than just exceptions, and the other problems mentioned above.

For example, you may have a process monitoring app that starts and watches your other apps (and a watchdog that watches your process monitoring app).

Your process monitoring app may, for example, "watch" another app by sending it a message, periodically, requesting its status.

Your process monitoring app may log the call stacks of all of the app's threads, if the app doesn't respond; to help you find out what's wrong with it.

And when your app gets a status request from the process monitoring app, it may check that all of the threads within it are still running properly.

Status requests and responses, and checking that a thread is running properly, can be implemented in many different ways, depending on your needs.