Click here to Skip to main content
15,860,861 members
Articles / Programming Languages / ASM

64-bit Structured Exception Handling (SEH) in ASM

Rate me:
Please Sign up or sign in to vote.
5.00/5 (28 votes)
1 Nov 2017CPOL7 min read 32.9K   1.1K   22   12
Rolling up a practical solution

Introduction

Windows Operating Systems provide a Structured Exception Handling (SEH) infrastructure.
On their side, some high-level languages provide internal support for it, namely a runtime library to easily deal with the OS implementation.

When an Exception is thrown, an automatic unwinding takes place, which translates to a backward search through the stack of function calls until an Exception Handler is found.

This process is totally transparent (and the inner workings vastly undocumented) for the developer, who normally needs only be concerned with delimiting the scope for the Exception Handler. The scope is defined with __try/__except or similar clauses.

When programming in C/C++ or .NET, this works for what the developers are expected to do, i.e., insert the __try/__except clauses and forget about it.

The Situation for Assembly Language

ASM programmers have no runtime library at their disposal to deal with SEH. Specifically with MASM (let's concentrate on the MASM model from here on, although things work more or less the same for other assemblers, namely those mentioned below), all they get is a set of Raw Pseudo Operations (there is also a set of MACROS, but almost mimic what the Raw Pseudo Operations do) that create unwinding information in the .pdata and .xdata sections of the PE Coff (see Ref 1, below).

There is a basic sample supplied with the description of the Raw Pseudo Operations, but it implicitly assumes the PROC (indeed, this means PROCEDURE) will be called from a C/C++ program, otherwise will not work as expected.

Rolling Our Own Runtime

1. Hooking the Callback

When there is an exception, the Operating System provides a Callback function, which in MASM can be prototyped as simply as:

ASM
_except_handler PROTO :PTR, :PTR, :PTR, :PTR

MASM provides a simple way to hook this Callback function, just add the name of some Handler after the FRAME attribute of the PROC.

While in C/C++, the compiler hooks the Callback behind the scenes when you establish __try/__except Guarded Block scopes - in MASM, things are not so simple. That is, you have to find as well a way to lay out the Guarded Block scopes within each PROC (this is a bit hard too, although can be automated as I have done). The end result is that the correct Guarded Block have to be found within the Callback Handler itself because they have been independently layed out (more about this later and illustrated in our source code as well).

2. Understanding the Callback

From its prototype, we see that the Callback provides four parameter pointers (three of them pointing to structures).

  • The first parameter is a pointer to an EXCEPTION_RECORD structure, which is defined as:
    ASM
    EXCEPTION_RECORD STRUCT
        ExceptionCode DWORD ?
        ExceptionFlags DWORD ?
        ExceptionRecord LPVOID ?
        ExceptionAddress LPVOID ?
        NumberParameters DWORD ?
        ExceptionInformation QWORD EXCEPTION_MAXIMUM_PARAMETERS dup (?)
    EXCEPTION_RECORD ENDS

    Two fields of this structure have particular interest, the ExceptionCode and the ExceptionAddress fields.

  • The second parameter is a pointer to the Establisher Frame. Although, may be important for debugging purposes (not so much in ASM, actually), we will not use it because our approach is handle the exception trying to restore execution rather than debug it.
  • The third parameter is a pointer to a CONTEXT structure, which represents the CPU register values for the thread at the time of the Exception. This is a large structure, too large to display here, but is available in the attached source code.
  • The fourth parameter is a pointer to a DISPATCHER_CONTEXT structure, defined as:
    ASM
    DISPATCHER_CONTEXT STRUCT
        ControlPc QWORD ? 
        ImageBase QWORD ?
        FunctionEntry LPVOID ?
        EstablisherFrame QWORD ?
        TargetIp QWORD ?
        ContextRecord LPVOID ?
        LanguageHandler LPVOID ?
        HandlerData LPVOID ?
        HistoryTable LPVOID ?
        ScopeIndex DWORD ?
        Fill0 DWORD ?
    DISPATCHER_CONTEXT ENDS

    I will not detail here the nitty-gritties of all these structures, they are immense and mostly undocumented.

3. The Handler of the Callback

Microsoft designates the Handler of the Callback by the name Language Specific Handler (LSH, for short). Although I would prefer the name Exception Handler, this name is already taken for the part of the program where execution resumes after the unwinding process is over.

As seen, the LSH receives a huge amount of information, which it can handle in diversified ways.
Our purpose here is nothing more than restore execution of the program (whenever possible), so we will focus the attention on what is essential and take the most reliable course of action.

From now on, our explanation will be based on what our own LSH for MASM does, but be aware that alternative ways, or even better ones, are possible, even though I am not aware of them.

So, after some preparatory errands, the first thing the LSH does is locate where the exception took place. The best bet is to rely on the ControlPc field of the DISPATCHER_CONTEXT structure. It tells us the RIP register value of the address where the Exception occurred within the PROC or where it left the PROC, if it couldn't be handled in there.

Then it calls the RtlLookupFunctionEntry API, to obtain an entry in the Function Tables that correspond to the ControlPc RIP register value. A PROC is added to the Function Tables whenever the name of some LSH is added to the FRAME attribute of that PROC. If RtlLookupFunctionEntry returns a valid value (it usually does), we will obtain from it the Begin Address and End Address of some PROC.

With that information, all the LSH has to do is look for Guarded Blocks (as mentioned previously, for MASM, they need to be established by hand).

If a Guarded Block is not found, the LSH will return a value telling the Operating System to continue the search.

If a Guarded Block is found, the LSH will call the RtlUnwindEx API to process the unwind. This function performs a massive unwinding work (see Ref. 3 below), which otherwise would be very error-prone if done by hand.

Finally, be aware that for each exception, the LSH will be called more than once, mostly catering for C/C++ cleanup needs.

Our Code

The defAsmSpecificHandler

Our source code presents a LSH, which I have baptized with the name defAsmSpecificHandler. I could have used different LSHs for different PROC, but I decided to centralize all in a single Handler.

defAsmSpecificHandler performs all tasks mentioned in the previous Chapter. In addition, it collects discretionary information to be reported when execution is resumed within the faulty program. Note that if the LSH was meant to be thread safe the collection of information should not be done this way. This is left as an exercise, I am not keen on complicating things that would deviate attention from the essential.

Here is the defAsmSpecificHandler:

ASM
; ***This is a "catch-all" Language Specific Handler for all our PROCs***

defAsmSpecificHandler PROC USES rbx rsi rdi r12 r13 r14 r15 pExceptionRecord:PTR, pEstablisherFrame:PTR, pContextRecord:PTR, pDispatcherContext:PTR
    
    LOCAL imgBase : PTR
    LOCAL targetGp : PTR
    LOCAL BeginAddress : PTR
    LOCAL EndAddress : PTR
    LOCAL catchHandler : PTR

    mov pExceptionRecord, rcx
    mov pEstablisherFrame, rdx
    mov pContextRecord, r8
    mov pDispatcherContext, r9

    ; Copy Contexts as they unwind. This serves also for reporting purposes.
    mov rdi, OFFSET originalExceptContext
    mov rax, pDispatcherContext
    mov rsi, (DISPATCHER_CONTEXT ptr [rax]).ContextRecord
    mov rcx, SIZEOF CONTEXT / 8
    rep movsq

    mov rcx,pExceptionRecord  
    cmp DWORD PTR [rcx].EXCEPTION_RECORD.ExceptionFlags, EXCEPTION_NONCONTINUABLE
    jne @F
        ; Bail out
        mov rcx,0
        call ExitProcess
@@:    
    test DWORD PTR [rcx].EXCEPTION_RECORD.ExceptionFlags, EXCEPTION_UNWIND
    jnz @F 
        ; On first pass of each exception, save data structures
        ; pointed to by arguments, so we can report if wanted, 
        ; otherwise may be skipped. 
        sub rsp, 20h
        mov rcx, pExceptionRecord
        mov rdx, pContextRecord
        call saveContextAndExceptRecs
        add rsp, 20h
@@:
    ; De-nest Catch Blocks.
    cmp blocksDenested,0 ; Previously done?
    jne @F
        sub rsp, 20h
        call denestCatchBlocks
        add rsp, 20h
        mov blocksDenested, 1
@@:
    mov rcx,pExceptionRecord 
    test DWORD PTR [rcx].EXCEPTION_RECORD.ExceptionFlags, EXCEPTION_UNWIND
    mov eax, ExceptionContinueSearch
    jnz @exit
    
    ; Search for a valid IMAGE_RUNTIME_FUNCTION_ENTRY
    ; that corresponds to the RIP value of the exception
    ; or where it left the PROCEDURE.
    mov rax, pDispatcherContext
    mov rcx, (DISPATCHER_CONTEXT PTR [rax]).ControlPc

    lea rdx, imgBase
    lea r8, targetGp
    sub rsp, 20h
    call RtlLookupFunctionEntry
    add rsp, 20h
    cmp rax, 0 ; Is return value valid?
    jnz @F
        ; We shouldn't come here (even with leaf functions).
        mov ecx, 1
        call ExitProcess 
@@:    
    mov r13, imgBase
    mov r11d, (IMAGE_RUNTIME_FUNCTION_ENTRY PTR [rax]).BeginAddress
    add r11, r13
    mov BeginAddress, r11
    mov r11d, (IMAGE_RUNTIME_FUNCTION_ENTRY PTR [rax]).EndAddress
    add r11, r13
    mov EndAddress, r11

    ; Search for the (innermost) Catch Block in range (in case of nested blocks).
    mov rsi, dataexcpStart
    mov r11, pDispatcherContext
    mov r11, (DISPATCHER_CONTEXT ptr [r11]).ControlPc
    
    mov catchHandler,0 ; Zero it, in order to know in the end if we got something.
    mov r12, 7FFFFFFFh ; Enter a big enough number in r12.
@loopStart:
    cmp QWORD PTR [rsi], 204E4445h ; End of all Blocks?
    je @loopEnd
    cmp QWORD PTR [rsi], 544F4C53h ; New slot signature?
    jne @ifEnd
        mov r14, QWORD PTR [rsi].CATCHBLOCKS.try
        cmp r14, BeginAddress
        jb @ifEnd
        cmp r14, r11
        ja @ifEnd
            mov r13, QWORD PTR [rsi].CATCHBLOCKS.catch
            cmp r13, EndAddress
            ja @ifEnd
            cmp r13, r11
            jb @ifEnd
                mov rax, r13
                sub rax, r14
                cmp r12, rax
                jbe @ifEnd
                    ; Got one
                    mov r12, rax
                    mov catchHandler, r13
                    jmp @ifEnd
@ifEnd:            
    add rsi, SIZEOF CATCHBLOCKS
    jmp @loopStart
@loopEnd:
    
    cmp catchHandler, 0 
    jne @F
        ; No Catch Block, continue searching in parent procedures.
        mov eax, ExceptionContinueSearch
        jmp @exit
@@:    
    mov rcx, pEstablisherFrame
    mov rdx, catchHandler
    mov r8, pExceptionRecord
    mov LSHretValue, 66h ; Any value. Let's test it in Proc1.
    lea r9, LSHretValue 
    
    sub rsp, 30h
    mov rax, pDispatcherContext
      mov rax, [rax].DISPATCHER_CONTEXT.HistoryTable
      mov [rsp+28h], rax    
    lea rax, originalExceptContext 
    mov [rsp+20h], rax
    call RtlUnwindEx ; Must not return. If it returns there is an error.
    add rsp, 30h ; We don't expect to come here, but anyway.
    mov ecx, 1
    call ExitProcess 
@exit:
    ; The default epilog will restore non-volatile registers.    
    ret
defAsmSpecificHandler ENDP

 

How the Guarded Blocks Are Laid Out?

I use a set of three MACROS to define the Guarded Blocks (within the code they are defined by a structure called CATCHBLOCKS) and our own PE SECTION, called dataexcp, to store the information about them.

This is the start of our PE SECTION:

ASM
dataexcp SEGMENT PARA ".data"     blocksDenested QWORD 0
    dataexcpStart LABEL near
    QWORD 204E4445h ; Signature for END of all blocks
    ORG $-8 ; Overwrite END, if there are catch blocks dataexcp ENDS

And these are the three MACROS:

ASM
_ExceptionBlock TEXTEQU <0>
__TRY MACRO
    LOCAL tryPos, level
    
    tryPos EQU $

    level TEXTEQU @SizeStr(%_ExceptionBlock)
    level TEXTEQU %(level -1)

    _ExceptionBlock CATSTR _ExceptionBlock, level
    
    dataexcp SEGMENT
        QWORD 544F4C53h ;; Signature for new slot
        QWORD level
        QWORD tryPos
    dataexcp ENDS
ENDM

__EXCEPT MACRO
    LOCAL catchPos, level
    
    catchPos EQU $+5 ;; 5 is the size of the "jmp near" instruction

    level TEXTEQU @SizeStr(%_ExceptionBlock)
    level TEXTEQU %(level -2)
    
    dataexcp SEGMENT
        QWORD level    
        QWORD catchPos
    dataexcp ENDS

    .code

     %jmp near ptr @catch&_ExceptionBlock&_end
     %@catch&_ExceptionBlock&_start:
ENDM    

__FINALLY MACRO
    LOCAL endCatchPos, count, level, temp
    endCatchPos EQU $
    
    level TEXTEQU @SizeStr(%_ExceptionBlock)
    level TEXTEQU %(level -2)
    dataexcp SEGMENT
        QWORD level    
        QWORD endCatchPos
        QWORD 204E4445h ;; Signature for end of all blocks
        ORG $-8 ;; Ready to be overwriten, if more Blocks     dataexcp ENDS
    count TEXTEQU @SizeStr(%_ExceptionBlock)
    count TEXTEQU %(count -1)

    .code
%@catch&_ExceptionBlock&_end:
    .data

    _ExceptionBlock TEXTEQU @SubStr(%_ExceptionBlock, 1, count)

    IF count EQ 1
        temp TEXTEQU %(_ExceptionBlock +1)
        BYTE temp
        IF temp EQ 9
         _ExceptionBlock TEXTEQU <0>
        ELSE 
            _ExceptionBlock TEXTEQU <temp>
        ENDIF 
    ENDIF    
    .code
ENDM

With this little arsenal of MACROS, all we have to do within a PROC in order to delimit a Guarded Block is insert the triplet of MACROS, something like this:

ASM
someProcedure PROC FRAME:defAsmSpecificHandler
    push rbp
    .pushreg rbp
    mov rbp, rsp
    .setframe rbp, 0    
    .endprolog
__TRY
    ; Guarded Section of code (level 0)
    _TRY
        ; Another Guarded Section of code (level 1)
    _EXCEPT
        ; Another Exception Handler (level 1)
    _FINALLY
__EXCEPT
    ; Exception Handler (level 0)
__FINALLY
    mov rsp, rbp
    pop rbp
    ret
someProcedure ENDP

Very easy, and it closely resembles the C/C++ semantics.

Tests Performed

I rolled up a set of six tests that must cover almost every situation faced in real life by ASM programmers. There are some marginal cases I have not dealt with. Anyway, if any reader believes they are relevant and presents code illustrating their case, I may update the article to deal with them as well.

Image 1

As mentioned, defAsmSpecificHandler is not thread safe, but changes needed to make it thread safe are few. This is left as an exercise.

The source code for the tests is in the attachments to this article.

Compatibility with C/C++

The defAsmSpecificHandler is compatible with the Visual Studio C and C++ SEH mechanism. If an Exception can't be handled within the ASM modulo, it will be passed to C/C++ where it can be handled.

I illustrate it in one of the attachments, this is the relevant C/C++ part:

C++
#include <stdio.h>
#include <excpt.h>
#include <windows.h> // for EXCEPTION_ACCESS_VIOLATION

#ifdef __cplusplus
extern "C"
{
#endif
    void proc1();
    void proc2_0();
    void proc3_0();
    void proc4();
    void proc5();
    void proc6_0();
#ifdef __cplusplus
}
#endif

int filter(unsigned int code, struct _EXCEPTION_POINTERS *ep) {
#ifdef __cplusplus
    printf("***Exception 0x%x at 0x%.8llx trapped in main***\n", 
            code, reinterpret_cast<intptr_t>(ep->ExceptionRecord->ExceptionAddress));
#else
    printf("***Exception 0x%x at 0x%.8llx trapped in main***\n", 
            code, (unsigned __int64)(ep->ExceptionRecord->ExceptionAddress));
#endif
    return EXCEPTION_EXECUTE_HANDLER;
}

int main()
{
    __try
    {
        proc1();
        proc2_0();
        proc3_0();
        proc4();
        proc5();
        proc6_0();
    }
    __except(filter(GetExceptionCode(), GetExceptionInformation()))
    {
        printf("\nExecuting Handler\n");
    }
    
    return 0;
}

Final Notes

  • While this ASM code was primarily targeted to be assembled with MASM, it can as well be assembled without changes with JWASM or UASM (version 2.43 or later). JWASM and UASM have other features, which are not supported by MASM, but are essentially downward compatible with it.
  • The code was tested in all operating system from Windows XP-64bit to Windows 10 64bit. However, for such an old operating system as Windows XP-64bit, you will need an appropriate linker (strictly, you can even use a recent one for that), you may be able to find it within the 64-bit resources provided by the time-honored MASM32.COM website.

References

  1. Unwind Helpers for MASM
  2. How to implement __try __except in ML64 (MASM)
  3. Unwind Data for Exception Handling, Debugger Support

History

  • 26th October, 2017: Initial release
  • 2nd November, 2017 : Update - Fixes in the test set (Proc1), in the C/C++ example and various typos in the text

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
AtelierWeb Software
Portugal Portugal
Jose Pascoa is the owner of AtelierWeb Software (http://www.atelierweb.com). We produce security and network software and mixed utilities since 1999. The first program I published (in a BBS) was a MS-DOS utility, had the size of 21 KB and was done in Assembly Language. Nowadays, my low level languages are more likely to be "C", "C++" and "Delphi" rather than Assembly Language but I still like it. I have nothing against more fashionable languages like C# and technologies like WPF, actually I have played with them and published software with them.

Comments and Discussions

 
QuestionDo I need _TRY {} _EXCEPT{} to use your FRAME based SEH Pin
ForNow15-Dec-17 2:57
ForNow15-Dec-17 2:57 
AnswerRe: Do I need _TRY {} _EXCEPT{} to use your FRAME based SEH Pin
Jose A Pascoa17-Dec-17 20:25
Jose A Pascoa17-Dec-17 20:25 
QuestionGreat Article Pin
ForNow10-Dec-17 6:10
ForNow10-Dec-17 6:10 
QuestionCan I use the same includelib as for C/C++ Pin
ForNow10-Dec-17 3:33
ForNow10-Dec-17 3:33 
AnswerRe: Can I use the same includelib as for C/C++ Pin
Jose A Pascoa10-Dec-17 5:24
Jose A Pascoa10-Dec-17 5:24 
GeneralRe: Can I use the same includelib as for C/C++ Pin
ForNow10-Dec-17 6:01
ForNow10-Dec-17 6:01 
GeneralRe: Can I use the same includelib as for C/C++ Pin
Jose A Pascoa10-Dec-17 9:03
Jose A Pascoa10-Dec-17 9:03 
Questionwould like to implement your code in Windows MFC Pin
ForNow9-Dec-17 13:00
ForNow9-Dec-17 13:00 
GeneralMy vote of 5 Pin
docNyie26-Oct-17 18:18
docNyie26-Oct-17 18:18 
GeneralRe: My vote of 5 Pin
Jose A Pascoa26-Oct-17 18:40
Jose A Pascoa26-Oct-17 18:40 
GeneralMy vote of 5 Pin
siekmanski26-Oct-17 5:43
siekmanski26-Oct-17 5:43 
GeneralRe: My vote of 5 Pin
Jose A Pascoa26-Oct-17 18:39
Jose A Pascoa26-Oct-17 18:39 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.