Cross Process Subclassing

• Download source files - 7 Kb
• Download demo project - 20 Kb

Introduction

With the code below I will explain how to subclass a Notepad application window using hooks & subclassing techniques. This technique can be used to build custom dll-based engines for any application. In this example I will subclass the notepad application ,create custom menus and react when the menu is clicked.

Background

• API hooking revealed-By Ivo Ivanov
• Windows Message Handling - Part 4 -By Daniel Kopitchinski

Using the code

Before I start describing about cross process subclassing ,I presume you already know what the terms hooking and window subclassing mean. Also I presume you have some experience working with hooks and subclassing in particular and the windows SDK in general. I will just briefly go over these 2 techniques and explain how you could combine both these techniques to achieve cross-process subclassing.

Windows Hooking

This is what MSDN has to say about hooks

"In the Microsoft® Windows® operating system, a hook is a mechanism by which a function can intercept events (messages, mouse actions, keystrokes) before they reach an application. The function can act on events and, in some cases, modify or discard them. Functions that receive events are called filter functions and are classified according to the type of event they intercept."

You can install different types of hooks like a keyboard hook, a message hook or a mouse hook depending upon your requirements .Hooks can be categorized in 2 ways,

1. Global Hooks
2. Thread Specific Hooks

Global hooks are, as the name suggests Global in nature . A global hook is installed in each and every thread that is running in the system. Global hooks when not properly used tend to be bulky and also slows down the system in many cases.

Thread Specific Hooks are installed in only one process. This hook can be installed in either the same thread as the calling function (to be explained later) or in a thread running in a different process.

Hooks are installed using the SetWindowsHookEx Api and un-installed using the UnhookWindowsHookEx Api.

Window Subclassing

Subclassing allows you to change the behavior of an existing window, typically a control, by inserting a message map to intercept the window's messages. Subclassing is process specific, you cannot subclass a window which is not running in the same process as your application.

Although the Windows OS does not allow us to subclass a window which is not in the same process as our application, we can work around this by using hooks and getting into the process space of the window that we want to subclass.

Now let's get to building our application.

Our application will consist of 2 modules, the hooking dll which would install/un-install the hook and also subclass the notepad application and an exe which would load the hooking dll.

The Hooking Dll

The hooking dll will be used to install/uninstall windows hook and also subclass the notepad application window. Before getting into the code for installing/uninstalling and subclassing let's look at the following piece of code

//Initialized Data to be shared with all instance of the dll
#pragma data_seg("Shared")
HWND hTarget=NULL;
HWND hApp = NULL;
int num=0 ;// Number of the subclassed window handle ,for use in the dll
bool done=FALSE;
HINSTANCE hInstance=NULL;
#pragma data_seg()
// Initialised data End of data share

The #pragma data_seg compiler directive asks the compiler to create a data segment which can be shared by all instances of the dll. The reason we need this is because the dll we will be loading using our exe application will be in one process and the dll which would eventually hook the notepad application will be in notepad application's process. So we need a common data segment which can be shared by different instances of the dll .

The code for installing the windows hook looks something like this.

// Get the handles of the Targetwindow and of the Our application
// and set the hook
int WINAPI SetHandle(HWND HandleofTarget ,HWND HandleofApp)
{

    hTarget=HandleofTarget;
    hApp=HandleofApp;
    hWinHook=SetWindowsHookEx(WH_CBT,(HOOKPROC)CBTProc,hInstance,
         GetWindowThreadProcessId(hTarget,NULL));
    if(hWinHook==NULL)
        return 0;
    else
        return 1;

}//End this function

We would be installing a CBT hook. A CBT (Computer Based Training) hook is used when we want to be notified when a window is created/destroyed/activated/minimized etc. The hook callback procedure looks something like this.

//The CBT hook Proc(Computer Based Training Hook)
LRESULT CALLBACK CBTProc(int nCode,WPARAM wParam,LPARAM lParam)
{


    if (nCode==HCBT_ACTIVATE)  //Called when the application window 
                               //is activated
    {
    
        if((HWND)(wParam)==hTarget)  //check if the window activated 
                                      //is Our Targer App
        {   
                    
            int count;
            for (count=0;count<num;count++)
            {
                if (blnsubclassed[count]==FALSE)
                {    
                    if(((int)hndll[count])>1)
                    {
                        OldWndHndl[count]=SetWindowLong(hndll[count],
                           GWL_WNDPROC,(long)WindowProc);  //Subclass !!!!
                    }
                                        
                    blnsubclassed[count]=TRUE;    // Set state as subclassed
                }
            }

        }        
    }
    if (nCode==HCBT_DESTROYWND) //Called when the application 
                                 //window is destroyed
    {

        if((HWND)wParam==hTarget)
            SendNotifyMessage(hApp,WM_APP +1024,(WPARAM)wParam,
               (LPARAM)lParam);// Send the message to our app
    }
    return CallNextHookEx(NULL, nCode, wParam, lParam);
}//End of the hook procedure

We subclass the notepad application as soon as it is activated the first time. The SetWindowLong Api is used to subclass the notepad application. The subclassed window procedure looks like this

//Window Procedures of the subclassed windows
LRESULT CALLBACK WindowProc(
  HWND hwnd,
  UINT uMsg,
  WPARAM wParam,
  LPARAM lParam
)
{    
    long val;
    int count;
    for(count=0;count<num;count++)
    {
        if(hndll[count]==hwnd)
        {
            val=count;   // this gets us the exact position of 
                          // this window procedure in the array
        }
    }
    
    long result;
    if(uMsg==273) //Message Implying Menu Clicks
        if(HIWORD(wParam)==0)
                result=SendNotifyMessage(hApp,WM_APP +1024,
                  (WPARAM)(LOWORD(wParam)),
                  (LPARAM)uMsg);// Send the message  to the vb app

    return CallWindowProc((WNDPROC)OldWndHndl[val],hwnd,uMsg,wParam,lParam);
}//End Procedure

Whenever a user clicks on a menu in notepad our hooking exe is notified and depending on which menu was clicked our application reacts to the click.

The Hooking Exe

The hooking exe will create the menus and load the hooking dll. The code for doing this is as follows

hHookedWindow=FindWindow(NULL,"Untitled - Notepad");
if(hHookedWindow==NULL)
{
    MessageBox(0,
          "Could Not find a running instance of Notepad.\r\n"
          "Please Start Notepad and try again","Error",0);
    break;
}
hAppMenu=GetMenu(hHookedWindow);
hAppendMenu=CreateMenu(); //Create the menu
AppendMenu(hAppMenu,MF_STRING + MF_POPUP,
    (unsigned int)hAppendMenu,"HTML");  //add new menu
AppendMenu(hAppendMenu,MF_STRING,125,"Make HTML");     
AppendMenu(hAppendMenu,MF_STRING,126,"Add Line Break");
hLib = LoadLibrary("hooks.dll"); //load the hooking dll
hMenuWnd = GetWindow(hHookedWindow, 5); //get the menu window
hThread=GetWindowThreadProcessId(hHookedWindow,NULL);
SetHandle = (sthndl)GetProcAddress(hLib, "SetHandle");
UnSubClass = (unsub)GetProcAddress(hLib, "UnSubclass");
SetHandle(hHookedWindow,hwnd);
FillHandleArray = (filhndl)GetProcAddress(hLib, "FillHandleArray");
FillHandleArray(hHookedWindow,1);
FillHandleArray(hMenuWnd,1);
ShowWindow(hHookedWindow, SW_MINIMIZE);

Thus by using hooks we can subclass a window running in any process. Cross Process Subclassing should be used very carefully , unsafe hooking/subclassing generally results in crashing the application which was subclassed/hooked.

History

• 22 Oct 2003 -Article Added