Creating Windows Touch Control Frameworks in C++ (PhotoStrip)

At PDC 2009, Reed Townsend presented some very exciting multitouch samples in a presentation: Windows Touch Deep Dive. The following video shows the presentation:

Many customers have been asking about how to properly handle multiple window objects and the first half of his presentation does a great job explaining the best practices. It's been a while since Reed presented this sample but I have the source code and did a short write up interpreting how everything is working. You can download the Windows Touch Photostrip sample from my server and the following notes explain the various pieces.

Summary

The PhotoStrip control created for this demo is similar to the one that is included in the Touch Pack and on the Microsoft surface. For this demo, not only is the control created, but an entire framework for having multiple reusable controls that can be mixed and placed into applications.

Typical goals for a control such as the demo PhotoStrip control could be to:

• Respond to touch messages
• Add constrained manipulations for panning
• Add support for dragging in / out of the control
• Add an AIP surface
• Flesh out the overlay and gallery

The following image shows the PhotoStrip control in the demo application.

There are PhotoStrip controls on the top and the bottom of the screen. Sliding along the control will move the control. Dragging images from the bottom of the control up will cause them to appear in the Gallery.

Demo Overview

The PhotoStrip control is encapsulated into its own HWND. This is a problem if you want to have many controls working well together because Windows Touch messages are only sent to a single window, the first window that receives the touch down message. To work around this, an overlay window takes all the input and maps it to the correct control as appropriate based on hit detection. The following diagram outlines this hierarchy of Windows.

Given that window hierarchy with the overlay, the following shows how data would flow through the control.

In the message flow, a WM_TOUCH message is generated by the input hardware. This message will be received by the overlay HWND. The overlay HWND then performs hit detection and sends the message to the correct child HWND (the PhotoStrip window, for example). If the user were dragging an image to the gallery, the PhotoStrip would then send a message to the application which tells it that the user dragged a photo to the gallery. The application then tells the gallery to display the photo in the gallery.

The following image illustrates the controls of the demo application in a screenshot of the demo running.

There are PhotoStrip controls on the top and bottom of the application and the Gallery control is in the center of the application layout.

Programming Details

Handling Messaging to the Overlay Control

First off, the application creates the overlay window and then creates the controls for the gallery and the PhotoStrips. These controls are then registered with the overlay.

The following code shows how this is done in code.

HRESULT InitWindow( HINSTANCE hInstance ) 
{ 
(…) 
    if( SUCCEEDED( hr ) ) 
    { 
        // Create window   
        hWnd = CreateWindowEx(0, WINDOW_CLASS_NAME, L"CollageSample", 
            WS_OVERLAPPEDWINDOW, CW_USEDEFAULT, CW_USEDEFAULT, 
                                CW_USEDEFAULT, CW_USEDEFAULT, NULL, NULL, hInstance, 0); 
        if (!(hWnd)) 
        { 
            hr = E_FAIL; 
        } 
    }
    if( SUCCEEDED( hr ) ) 
    { 
        ShowWindow( hWnd, SW_SHOWMAXIMIZED ); 
        DisableTabFeedback(hWnd); 
    }
    if (SUCCEEDED( hr )) 
    { 
        GetClientRect((hWnd), &rClient); 
        iWidth = rClient.right - rClient.left; 
        iHeight = rClient.bottom - rClient.top; 
        iTopHeight = (INT)(iHeight * TOP_HEIGHT_PERCENTAGE); 
        iBottomHeight = (INT)(iHeight * BOTTOM_HEIGHT_PERCENTAGE);
        hr = Overlay::CreateOverlay(hInstance, hWnd, 0, 0, 
                iWidth, iHeight, &g_pOverlay); 
    } 
    if (SUCCEEDED( hr )) 
    { 
        hr = Photostrip::CreatePhotostrip(hInstance, (hWnd), 
            0, 0, 
            iWidth, iTopHeight, 
            TOP_PHOTOSTRIP_DIRECTORY, 
            Photostrip::Top, 
            &g_pStripTop); 
    } 
    if (SUCCEEDED( hr )) 
    { 
        hr = Photostrip::CreatePhotostrip(hInstance, (hWnd), 
            0, rClient.bottom - iBottomHeight, 
            iWidth, iBottomHeight, 
            BOTTOM_PHOTOSTRIP_DIRECTORY, 
            Photostrip::Bottom, 
            &g_pStripBottom); 
    } 
    if (SUCCEEDED( hr )) 
    { 
        hr = Gallery::CreateGallery(hInstance, hWnd, 0, iTopHeight, 
                iWidth, (iHeight - iTopHeight - iBottomHeight), &g_pGallery); 
    } 
    if (SUCCEEDED( hr )) 
    { 
        g_pOverlay->RegisterTarget((ITouchTarget*)g_pGallery); 
        g_pOverlay->RegisterTarget((ITouchTarget*)g_pStripBottom); 
        g_pOverlay->RegisterTarget((ITouchTarget*)g_pStripTop); 
    }

When WM_TOUCH messages are generated, the overlay window will receive them in its WndProc method. The overlay converts the message coordinates from screen coordinates to client coordinates and keeps track of the messages. The overlay keeps track of particular inputs and then gives capture to a particular control.

The following code shows how the overlay maps inputs to the appropriate child control.

LRESULT CALLBACK Overlay::S_OverlayWndProc(HWND hWnd, UINT msg, WPARAM wParam, 
    LPARAM lParam) 
{ 
    //reroute WM_TOUCH and mouse messages 
    //if it's a down event, rehit test and set capture, 
    //else, route the event to the target currently capturing the cursor 
    LRESULT result = 0; 
    INT iNumContacts; 
    PTOUCHINPUT pInputs; 
    HTOUCHINPUT hInput; 
    POINT ptInputs; 
    ITouchTarget *pTarget = NULL; 
    Overlay* pOverlay;
    pOverlay = (Overlay*)GetWindowLongPtr(hWnd, 0); 
    if (pOverlay != NULL) 
    { 
        switch(msg) 
        { 
        case WM_TOUCH: 
            iNumContacts = LOWORD(wParam); 
            hInput = (HTOUCHINPUT)lParam;
            pInputs = new (std::nothrow) TOUCHINPUT[iNumContacts]; 
            if(pInputs != NULL) 
            { 
                if(GetTouchInputInfo(hInput, iNumContacts, pInputs, sizeof(TOUCHINPUT))) 
                { 
                    for(int i = 0; i < iNumContacts; i++) 
                    { 
                        //track the number of contacts 
                        if (pInputs[i].dwFlags & TOUCHEVENTF_DOWN) 
                        { 
                            pOverlay->m_ucContacts++; 
                        } 
                        else if (pInputs[i].dwFlags & TOUCHEVENTF_UP) 
                        { 
                            pOverlay->m_ucContacts--; 
                        }
                        // Bring touch input info into client coordinates 
                        ptInputs.x = pInputs[i].x/100;    
                        ptInputs.y = pInputs[i].y/100; 
                        ScreenToClient(hWnd, &ptInputs); 
                        pInputs[i].x = (LONG)(ptInputs.x); 
                        pInputs[i].y = (LONG)(ptInputs.y);
                        //hit test and set capture on down 
                        if (pInputs[i].dwFlags & TOUCHEVENTF_DOWN) 
                        { 
                           pOverlay-> m_pDispatch->CaptureCursor(pInputs[i].dwID, 
                                    (FLOAT)pInputs[i].x, (FLOAT)pInputs[i].y, &pTarget); 
                        } 
                        
                        //if anything has capture set for this cursor, route the event to it 
                        if (pOverlay->m_pDispatch->GetCapturingTarget(pInputs[i].dwID, &pTarget)) 
                        { 
                            pTarget->ProcessTouchInput(&pInputs[i]); 
                        } 
                    } 
                } 
                delete [] pInputs; 
            }
            CloseTouchInputHandle(hInput); 
            break; 
         
        case WM_LBUTTONUP: 
            ReleaseCapture(); 
            pOverlay->m_fIsMouseDown = FALSE; 
            pOverlay->ProcessMouse(msg, wParam, lParam); 
            break; 
        case WM_MOUSEMOVE: 
            pOverlay->ProcessMouse(msg, wParam, lParam); 
            break; 
        case WM_LBUTTONDOWN: 
            SetCapture(hWnd); 
            pOverlay->m_fIsMouseDown = TRUE; 
            pOverlay->ProcessMouse(msg, wParam, lParam); 
            break; 
        case WM_DESTROY: 
            SetWindowLongPtr(hWnd, 0, 0); 
            PostQuitMessage(0); 
            delete pOverlay; 
            break; 
        default: 
            result = DefWindowProc(hWnd, msg, wParam, lParam); 
        } 
    } 
    else 
    { 
        result = DefWindowProc(hWnd, msg, wParam, lParam); 
    } 
    return result; 
}

Each of the controls handles the touch messages in a similar manner in the ProcessTouchInput method which will cause _ManipulationEvents to be raised on the control.

Handling Touch Input for the PhotoStrip

The PhotoStrip itself implements the InertiaObj utility interface to enable inertia features. The following diagram illustrates the WM_TOUCH input mapping from the overlay window to the PhotoStrip control and its photos.

When capture goes to the control, the WM_TOUCH messages are sent to a ManipulationEvents interface which will raise manipulation events on that interface. Horizontal movement within the control will manipulate all of the photos simultaneously. Hit testing within the PhotoStrip control is used to send input to the appropriate photo because the photos themselves can be manipulated vertically within the control. Once the PhotoStrip control has finished interpreting input, that control releases capture for the control.

The following code shows how input is mapped to manipulations on the ManipulationDelta handler.

HRESULT STDMETHODCALLTYPE Photostrip::ManipulationDelta( 
    FLOAT /*x*/, 
    FLOAT /*y*/, 
    FLOAT translationDeltaX, 
    FLOAT /*translationDeltaY*/, 
    FLOAT /*scaleDelta*/, 
    FLOAT /*expansionDelta*/, 
    FLOAT /*rotationDelta*/, 
    FLOAT /*cumulativeTranslationX*/, 
    FLOAT /*cumulativeTranslationY*/, 
    FLOAT /*cumulativeScale*/, 
    FLOAT /*cumulativeExpansion*/, 
    FLOAT /*cumulativeRotation*/) 
{ 
    HRESULT hr = S_OK;
    //bound the move to the area around our photos 
    FLOAT fpTotalWidth = -INTERNAL_MARGIN, fpNewXOffset; 
    std::list<Photo*>::iterator it; 
    for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++) 
    { 
        fpTotalWidth += (*it)->GetWidth() + INTERNAL_MARGIN; 
    } 
    fpTotalWidth = max(0, fpTotalWidth); //handle no photos 
    fpNewXOffset = m_fpXOffset + translationDeltaX; 
    FLOAT fpXLowerBound = -fpTotalWidth + m_nWidth * (1-SIDE_MARGIN_PERCENTAGE); 
    FLOAT fpXUpperBound = m_nWidth * SIDE_MARGIN_PERCENTAGE; 
    fpNewXOffset = min(fpXUpperBound, fpNewXOffset); 
    fpNewXOffset = max(fpXLowerBound, fpNewXOffset); 
    translationDeltaX = fpNewXOffset - m_fpXOffset;
    //move the photos 
    m_fpXOffset += translationDeltaX; 
    for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++) 
    { 
        (*it)->Translate(translationDeltaX, 0); 
    } 
    return hr; 
}

The following code shows how the PhotoStrip uses manipulations to move photos horizontally along the control in the ManipulationDelta event handler.

//move the photos 
m_fpXOffset += translationDeltaX; 
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++) 
{ 
    (*it)->Translate(translationDeltaX, 0); 
}

The following code shows how photos in the PhotoStrip control handle manipulations and are constrained to vertical manipulations.

HRESULT STDMETHODCALLTYPE ConstrainedPhoto::ManipulationDelta( 
    FLOAT /*x*/, 
    FLOAT /*y*/, 
    FLOAT /*translationDeltaX*/, 
    FLOAT translationDeltaY, 
    FLOAT /*scaleDelta*/, 
    FLOAT /*expansionDelta*/, 
    FLOAT /*rotationDelta*/, 
    FLOAT /*cumulativeTranslationX*/, 
    FLOAT /*cumulativeTranslationY*/, 
    FLOAT /*cumulativeScale*/, 
    FLOAT /*cumulativeExpansion*/, 
    FLOAT /*cumulativeRotation*/) 
{ 
    HRESULT hr = S_OK; 
    
    Translate(0.0f, translationDeltaY);
    return hr; 
}

Note how the boundary checks from the photo object will notify the application that the photo has reached the boundary. A fake touch up message is sent up and capture from the PhotoStrip is released.

The following code shows how photos release capture.

case PS_PHOTO_BOUNDARY: 
    if (pStrip != NULL) 
    { 
        Photo *pPhoto = (Photo*)lParam; 
        TOUCHINPUT tUp = pPhoto->GetLastTouchInput();
        pStrip->m_pDispatch->ReleaseCapture(tUp.dwID); 
        
        //since we are removing the photo from the event stream by releasing 
        //capture, it will never see an up for this cursor 
        //fake an up to the photo so it's view of the event stream stays consistent 
        tUp.dwFlags &= ~TOUCHEVENTF_DOWN; 
        tUp.dwFlags &= ~TOUCHEVENTF_MOVE; 
        tUp.dwFlags |= TOUCHEVENTF_UP; 
        pPhoto->ProcessTouchInput(&tUp);
        //let out parent know that the user dragged a photo off 
        SendMessage(GetParent(hWnd), PS_PHOTOSTRIP_BOUNDARY, (WPARAM)pStrip, lParam); 
    } 
    break;

The message that is generated to the parent window contains a pointer to the photo object which can then be used to render the photo to the gallery control.

The Collage Control

When a photo is moved to the collage window, the collage uses the message to create an image that it displays. The following code shows how this is done.

case PS_PHOTOSTRIP_BOUNDARY: 
    //if a photo was dragged off a photostrip, we want to - 
    //      -remove the photostrip's capture of the cursor that dragged the photo off 
    //      -set the gallery to capture that cursor, so the photo can be dragged around 
    //          immediately 
    //      -load the photo that was dragged off the stip into the gallery
    pPhoto = (Photo*)lParam; 
    pStrip = (Photostrip*)wParam;
    tInput = pPhoto->GetLastTouchInput(); 
    g_pOverlay->ReleaseCapturedCursor(tInput.dwID);
    //since we released capture, the strip won't ever get an up for this event stream 
    //fake an up to the target so it stays consistent 
    tUp = tInput; 
    tUp.dwFlags &= ~TOUCHEVENTF_DOWN; 
    tUp.dwFlags &= ~TOUCHEVENTF_MOVE; 
    tUp.dwFlags |= TOUCHEVENTF_UP; 
    pStrip->ProcessTouchInput(&tUp);
    //translate into the parent window coordinates 
    GetWindowRect(pPhoto->GetHWnd(), &stripWnd); 
    GetWindowRect(g_pGallery->GetHWnd(), &galleryWnd); 
    tInput.x += stripWnd.left; 
    tInput.y += stripWnd.top;
    //calculate the photo size and position 
    fpPhotoHeight = (FLOAT)(stripWnd.bottom - stripWnd.top) 
        * (1 - INTERNAL_Y_MARGIN_PERCENTAGE) 
        * PHOTO_SIZE_MULTIPLIER; 
    fpPhotoYPos = (FLOAT)(tInput.y - galleryWnd.top); 
    fpPhotoYPos -= ((stripWnd.bottom - stripWnd.top) * INTERNAL_Y_MARGIN_PERCENTAGE);
    //load the photo into the gallery 
    g_pGallery->LoadPhoto(pPhoto->GetPhotoURI(), 
        (FLOAT)tInput.x, fpPhotoYPos, fpPhotoHeight);
    //direct the input to the new photo 
    g_pOverlay->SetCursorCapture(g_pGallery, tInput); 
    break;

When a photo reaches a boundary in the collage window, the collage will then remove the photo from the list of objects that it is tracking and will release capture of touch input. The following code shows how this is done in the case of photos hitting a boundary.

case PS_PHOTO_BOUNDARY_INERTIA: 
case PS_PHOTO_BOUNDARY: 
    if (pCollage != NULL) 
    { 
        //mark the photo for deletion, it will be removed the next time the timer fires 
        pPhoto = (Photo*)lParam; 
        if (pPhoto != NULL) 
        { 
            pCollage->m_lPhotosToDelete.remove(pPhoto); //make sure we don't add it twice 
            pCollage->m_lPhotosToDelete.push_front(pPhoto); 
        } 
    } 
    break;

The following code shows how this is done in the case of a boundary being passed during inertia.

case WM_TIMER: 
        if (pCollage != NULL) 
        { 
            //process inertia 
            for (it = pCollage->m_lPhotos.begin(); it != pCollage->m_lPhotos.end(); it++) 
            { 
                (*it)->ProcessTimer(); 
            }
            //processing the timers might have caused a boundary event, in 
            //which case we would need to clean up photos that passed the boundary 
            for (it = pCollage->m_lPhotosToDelete.begin(); 
                it != pCollage->m_lPhotosToDelete.end(); it++) 
            { 
                //only remove the photo if it's still past the bounds. as we have capturing, 
                //the user could have dragged the photo off the window then back on 
                if ((*it)->IsPastBounds()) 
                { 
                    //release any current mapping 
                    DWORD dwID = 0; 
                    while(pCollage->m_pDispatch->GetCursor(*it, &dwID)) 
                    { 
                        pCollage->m_pDispatch->ReleaseCapture(dwID); 
                    }
                    pCollage->m_lPhotos.remove((Photo*)(*it)); 
                    pCollage->m_pDispatch->UnregisterTouchTarget((*it)); 
                    (*it)->CleanUp(); 
                } 
            } 
            pCollage->m_lPhotosToDelete.clear();
            pCollage->Render(); 
        } 
        break;

Conclusion

Creating a control framework for complex Windows Touch applications is the best way to deliver an experience with reusable components. A newer version of this document will be released online at http://code.msdn.microsoft.com/WinTouchPhotostrip later this month (February) so keep an eye on the @WinDevs twitter account if you are looking forward to the update. Apologies for no colorization in the C++ code, I’m working on potential solutions that will work with this blog platform.