Calling Conventions for SSI Test Modules

Introduction

This paper is to understand how library routines pass data with each other, using different calling conventions.

Calling Convention Overview

A calling convention establishes a standard which determines how a program makes a call from one routine to another.

What does the convention standardize?

In order to make a program call, the caller needs to have three things:

1. The address of the routine to be called.
2. Variables, if any, from the caller, on which the called to procedure is to operate (the parameter list).
3. How the called routine returns data to the caller.

The way in which these steps are carried out (the calling model) establishes procedures or mechanisms for:

1. Obtaining the address of the routine to be called.
2. Building a structure which contains either copies of, or address references to the parameter list data. This structure must be accessible by the called routine.
3. Passing control (calling) the routine.
4. Cleaning up the data structure when the called routine has completed.

How does the caller determine the address of the routine?

If the routine to be called was part of the same binary file as the caller, the link process establishes the address as it lays out the image of the binary file. When the compiler created each object file destined to be in the binary, it inserted place holders (zeros) where the call address references should be. As the linker works its way through each of the binary modules (object files ) to be part of the final file, it discovers the starting address of each module within the file as it lays them in the binary image file one after the other. It saves each routine’s starting address in a table (exp?). That’s the first pass. The linker then runs through the binary file and replaces place holders that the compiler left with the actual addresses the routines. That’s the second pass. Library files are treated just as object files because they wind up part of the binary image. The only difference is that the LIB file structure contains a table containing the addresses of its embedded object file images. This method is called static linking.

In the case of a DLL, the image is not part of the same binary file as the caller so there needs to be a way to either give the linker a list of addresses that it can use to satisfy its call references during its build process or allow the address to stay unresolved until they are needed to make the call during runtime. The first produces the same result as the static linking method above. The second method requires services from the operating system to provide the address for the caller dynamically.

What Load Method Do We Use?

The Test Executive relies on the second method of dynamically determining the address. This method is used to load the test application DLLs and the instrument wrapper library we developed.

How do we get the routine address from a DLL?

The Test Executive uses the Win32 service LoadLibrary to bring the library which contains the routine to be called, into memory so that it can be accessed.

Once the library is in memory, the service GetProcAddress is used to find the address of the desired routine. GetProcAddress needs a name to use to look up the address of the routine to be called. This is where the calling convention begins to kick in. When the DLL containing the routine to call was created, a function name was generated to access the routine. The function name that the compiler/linker creates is not simply the routine name but instead differs based on the calling convention specified by the DLLs project settings.

The default calling convention used for MSVSC++ is the C++ calling convention. The function names created using this convention use the routine name as its root but append proprietary symbols. The adding of these symbols is called decorating. Decorating allows routines with the same name in the same file that differ only by parameter list or return type to exist because the symbols produce different names in the executable. Also encoded within the decoration, is the calling convention the function was declared with.

The decorated name can be reversed engineered by using a tool provided with MSVC called udname. Running udname on a function name ?LibStart@@YXH@Z outputs:

Undecoration of :- "?LibStart@@YAXH@Z"
is :- "void __cdecl LibStart(int)"

Notice the calling convention is decoded as __cdecl, the tools default.

The decorating of the name poses an issue for the Test Exec using the function GetProcAddress. In order to find the address of the function, you must specify its name. In order to find the name of the function void LibStart(int value) for example, the TPD file we provide would have to have a line like:

Test=…."System.DLL";" ?LibStart@@YAXH@Z";;…

And the resultant call inside the Test Executive would be:

GetProcAddress(.."?LibStart@@YAXH@Z")

Specifying the decorated name in the TPD file would work, but it is not very friendly.

Note: This works with the Test Executive because it does not specify the routine name implicitly.

The GetProcAddress function returns the entry point address of the routine to be called. The address type returned is FARPROC. FARPROC is an old definition that defines the address as a full 32 bit address. It has nothing to do with the calling convention; it is just the address of the entry point to the routine.

The next thing the Test Executive must do is to call the routine. It does this by making a call using the address returned by the GetProcAddress function.

Here is an example of what goes on the in the Test Executive (C style):

typedef int(*TSTADRS)(int);
int _tmain(int argc, _TCHAR* argv[])
{
  HINSTANCE hMod = 0;
  FARPROC fnTest;
  hMod = LoadLibrary("Test.dll");
  TSTADRS p=(TSTADRS)GetProcAddress(hMod,"?fnTest@@YAHH@Z");
  int i = (*p)(42);
  return 0;
}

If it were accessing a library, say Test.dll, made up of the following…

Here’s what’s in the library header:

#ifdef TEST_EXPORTS
#define TEST_API __declspec(dllexport)
#else
#define TEST_API __declspec(dllimport)
#endif
TEST_API int fnTest(void);

And in the library .cpp file:

// Test.cpp : Defines the entry point for the DLL application
// This is an example of an exported function.
TEST_API int fnTest(void)
{
  return 42;
}

When the Library was built, here’s what’s exported (map file):

0002:00000fe0 ?fnTest@@YAHH@Z 10011fe0 f Test.obj

Here’s the disassembly of the Test Exec (c style):

#include "stdafx.h"
#include "windows.h"
typedef int(*TSTADRS)(int,int,int);
int _tmain(int argc, _TCHAR* argv[])
{
  004135D0 push ebp 
  004135D1 mov ebp,esp 
  004135D3 sub esp,0F0h 
  004135D9 push ebx 
  004135DA push esi 
  004135DB push edi 
  004135DC lea edi,[ebp-0F0h] 
  004135E2 mov ecx,3Ch 
  004135E7 mov eax,0CCCCCCCCh 
  004135EC rep stos dword ptr [edi] 
    HINSTANCE hMod = 0;
  004135EE mov dword ptr [hMod],0 
    FARPROC fnTest;
    hMod = LoadLibrary("Test.dll");
  004135F5 mov esi,esp 
  004135F7 push offset string "Test.dll" (4240DCh) 
  004135FC call dword ptr [__imp__LoadLibraryA@4 (42B180h)] 
  00413602 cmp esi,esp 
  00413604 call @ILT+930(__RTC_CheckEsp) (4113A7h) 
  00413609 mov dword ptr [hMod],eax 
    STADRS p=(TSTADRS)GetProcAddress(hMod,"?fnTest@@YGHHHH@Z");
  0041360C mov esi,esp 
  0041360E push offset string "?fnTest@@YAHXZ" (4240C8h) 
  00413613 mov eax,dword ptr [hMod] 
  00413616 push eax 
  00413617 call dword ptr [__imp__GetProcAddress@8 (42B17Ch)] 
  0041361D cmp esi,esp 
  0041361F call @ILT+930(__RTC_CheckEsp) (4113A7h) 
  00413624 mov dword ptr [p],eax 
    int i = (*p)(2,4,8);
  00413627 mov esi,esp           //This is the stack frame
  00413629 push 8                //being created.
  0041362B push 4 
  0041362D push 2 
  0041362F call dword ptr [p]   //Call function using *p()
  00413632 add esp,0Ch 
  00413635 cmp esi,esp          //Check stack
  00413637 call @ILT+930(__RTC_CheckEsp) (4113A7h) // Save returnvalue to i 
  0041363C mov dword ptr [i],eax 
    return 0;
  0041363F xor eax,eax 
}
00413641 pop edi 
00413642 pop esi 
00413643 pop ebx 
00413644 add esp,0F0h 
0041364A cmp ebp,esp 
0041364C call @ILT+930(__RTC_CheckEsp) (4113A7h) 
00413651 mov esp,ebp 
00413653 pop ebp 
00413654 ret 

The call placed the parameters on the stack from right to left before making the call.

Here’s the called function using stdcall project setting.

TEST_API int fnTest(int x, int y, int z)
{
  10011FE0 push ebp 
  10011FE1 mov ebp,esp 
  10011FE3 sub esp,0C0h 
  10011FE9 push ebx 
  10011FEA push esi 
  10011FEB push edi 
  10011FEC lea edi,[ebp-0C0h] 
  10011FF2 mov ecx,30h 
  10011FF7 mov eax,0CCCCCCCCh 

  10011FFC rep stos dword ptr [edi] 
    return x+y-z;
  10011FFE mov eax,dword ptr [x] 
  10012001 add eax,dword ptr [y] 
  10012004 sub eax,dword ptr [z] 
}
10012007 pop edi 
10012008 pop esi 
10012009 pop ebx 
1001200A mov esp,ebp 
1001200C pop ebp 
1001200D ret 0Ch

Here’s the stack frame with the parameters list:

0x0012FDCC  dc fe 12 00 32 36 41 00 02 00<SUB>X</SUB> 00 00
   04 0<SUB>Y</SUB>0 00 00 08<SUB>Z</SUB> 00 00 00 4f 5b 91 7c 40 00 00  
Üþ..26A.............O[‘|@..

The watch window for x,y,z in the function:

x   2   int
y   4   int
z   8   int

The result of the addition and subtraction.

EAX = FFFFFFFE EBX = 7FFDF000 ECX = 00000000 EDX = 7C97C0D8 
ESI = 0012FDE0 EDI = 0012FDCC EIP = 10012007 ESP = 0012FD00 
EBP = 0012FDCC EFL = 00000293

The result FFFFFFE (-2) is what was expected: 2+4-8= -2.

So far so good. When the routine returns:

EAX = FFFFFFFE EBX = 7FFDF000 ECX = 00000000 EDX = 7C97C0D8 
ESI = 0012FDE0 EDI = 0012FEDC EIP = 00413635 ESP = 0012FDEC 
EBP = 0012FEDC EFL = 00000202 
   int i = (*p)(2,4,8);
00413627 mov esi,esp 
00413629 push 8 
0041362B push 4 
0041362D push 2 
0041362F call dword ptr [p] 
00413632 add esp,0Ch 
00413635 cmp esi,esp 
00413637 call @ILT+930(__RTC_CheckEsp) (4113A7h) 
0041363C mov dword ptr [i],eax //EAX contains the correct return value to assign to i.
   return 0;

Looks like its going to work, but when RTC_CheckEsp is called, an assertion is thrown stating that the call stack was improperly saved across function boundaries.

But what’s different between stdcall and cdecl ?

Rebuilding the test DLL using the cdecl project setting generated the following comparison.

TEST_API int fnTest(int x, int y, int z)
{
  10011FE0 push ebp 
  10011FE1 mov ebp,esp 
  10011FE3 sub esp,0C0h 
  10011FE9 push ebx 
  10011FEA push esi 
  10011FEB push edi 
  10011FEC lea edi,[ebp-0C0h] 
  10011FF2 mov ecx,30h 
  10011FF7 mov eax,0CCCCCCCCh 
  10011FFC rep stos dword ptr [edi] 
    return x+y-z;
  10011FFE mov eax,dword ptr [x] 
  10012001 add eax,dword ptr [y] 
  10012004 sub eax,dword ptr [z] 
}
10012007 pop edi 
10012008 pop esi 
10012009 pop ebx 
1001200A mov esp,ebp 
1001200C pop ebp 
1001200D ret

TEST_API int fnTest(int x, int y, int z)
{
  10011FE0 push ebp 
  10011FE1 mov ebp,esp 
  10011FE3 sub esp,0C0h 
  10011FE9 push ebx 
  10011FEA push esi 
  10011FEB push edi 
  10011FEC lea edi,[ebp-0C0h] 
  10011FF2 mov ecx,30h 
  10011FF7 mov eax,0CCCCCCCCh 
  10011FFC rep stos dword ptr [edi] 
    return x+y-z;
  10011FFE mov eax,dword ptr [x] 
  10012001 add eax,dword ptr [y] 
  10012004 sub eax,dword ptr [z] 
}
10012007 pop edi 
10012008 pop esi 
10012009 pop ebx 
1001200A mov esp,ebp 
1001200C pop ebp 
1001200D ret 0Ch

Apparently, the return is causing a stack problem. The problem stems from the offset that was added to the stack pointer when the function started. In the cdecl call, the offset is ignored because the caller is responsible for correcting it. The stdcall adds the offset back in when the routine returns.

If the function pointer in the Test Exec is changed to a stdcall pointer like:

typedef int (__stdcall *TSTADRS)(int,int,int);

Things work again.

Assumption

There is no way to tell the caller (Test Executive) what the routine it wishes to call. Calling convention is ahead of time and adjusts its call pointer accordingly. It is assumed the calling convention is stdcall because it was written in Visual Basic.

What we can do is change the calling convention of the called routine. There are two ways to change the calling convention of a routine.

1. Set the project's calling convention setting from its default to standard call. That’s what I have been using so far.

2. Leave the project setting as the default, cdecl, and selectively change the functions that you need to call from the test executive. This is done by adding the WINAPI declaration to the function prototype and body.

How do Test Modules access other DLLs.

When we build Test Modules, we use the same process as described earlier and in order for the linker to resolve the addresses of the external DLL routines, we need either a static LIB file or a definition file. We have been using the export keyword in our prototypes.

The __export keyword allows the compiler to generate the export names automatically and place them in a .LIB file. This .LIB file could then be used just like a static .LIB to link with a DLL. __declspec(dllexport) adds the export directive to the object file so that you don't need to use a .DEF file.

You can export data, functions, classes, or class member functions from a DLL using the __declspec(dllexport) keyword.

There is no standard specification for name decoration, so the name of an exported function may change between compiler versions. If you use __declspec(dllexport), recompiling the DLL and dependent .EXE files is necessary only to account for any naming convention changes.

To export functions, the __declspec(dllexport) keyword must appear to the left of the calling-convention keyword.

Using __declspec(dllexport) does two things.

1. It places the function name in the .LIB file as explained above.
2. It also places the function name in the DLL exports table.

The major requirement for stdcall is, interfacing with the Test Executive.

• Each test module needs an entry point that the Test Executive can access. Those functions need to be stdcall.
• The system library needs exported stdcall functions only where the Test Executive calls it. The Test Executive does not call any library functions.
• When the Test Module or Test Libraries make calls to the Test Executive library (texdll32.dll), the calling convention is declared in the header of texdll32, so this adjustment is made automatically.

So why not to make everything stdcall?

Setting the project setting to stdcall exports everything as stdcall. Doing so eliminates the possibility to use classes in either the Test Modules or the Test Libraries.

To allow more flexibility and to provide growth into object programming, I suggest the following:

• Export the functions that needed to be called by the test executive as extern “C”, WINAPI (stdcall) and leave the pragmas to simplify the names that must be entered in the TPD file (strip of the “-“ and “@xx)).
• Set the calling convention project setting to cdecl (the default).

The result will be that, all functions exported by default will have the cdecl calling convention. Only the functions required to interface with the Test Executive will be exported as stdcall.

• This will reduce the number of functions that require the pragma definitions to just two (StartLib and StopLib).
• Since none of the instrument driver’s calls are directly accessed by the Test Executive, there are no functions that need to be exported as stdcall.
• The Test Executive Library Texdll32.dll has in its header declaration the stdcall declaration so the compiler/linker will handle it automatically.
• The Test Modules don’t need to declare only the functions that it wishes called by the Test Executive stdcall (currently only a single entry point).

Click here for Information on Visual Basic DLL integration.