An HTML5 Remote Desktop - Part I

• Download source code - 157.84 KB
• Download binaries - 618.03 KB

Introduction

VNC and Remote Desktop software were around for a long while now. Undoubtedly, they work great for most purposes but, I wanted to bring the old VNC / Remote Desktop concept over the web, to allow full web-application integration and full cross-browser cross-platform support out-of-the-box.

Despite its name, ThinVNC is not a traditional VNC, as it does not implement the AT&T RFB protocol. Instead, it is built on top of today's web standards: AJAX, JSON and HTML5.

The Code (Part I)

In this first part, we'll see the code to do the screen capture. The standard approach is to capture the whole desktop. However, in this case, we'll capture every window individually, applying clipping regions and saving the individual bitmap for later comparison and difference extracting.

Firstly, we need to enumerate all visible top windows:

TWin = class(TObject)
private
  Wnd : Hwnd;
  Rect : TRect;
  Pid : Cardinal;
public
  constructor Create(AWnd:HWND;ARect:TRect;APid:Cardinal);
end;

function EnumWindowsProc(Wnd: HWnd; const obj:TList<TWin>): Bool; export; stdcall;
var ProcessId : Cardinal;
  R,R1 : TRect;
  Win : TWin;
begin
  Result:=True;
  GetWindowThreadProcessId(Wnd,ProcessId);
  if IsWindowVisible(Wnd) and not IsIconic(wnd)then begin
    GetWindowRect(Wnd,R);
    IntersectRect(R1,R,Screen.DesktopRect);
    if not IsRectEmpty(R1) then begin
      win := TWin.Create(Wnd,R,ProcessId);
      obj.Add(win);
    end;
  end;
end;

procedure GetProcessWindowList(WinList:TList<TWin>);
begin
  WinList.Clear;
  EnumWindows(@EnumWindowsProc, Longint(WinList));
end;

We want to keep a list of windows, with their basic attributes and their bitmaps, so we can compare with the new ones and send the differences to the client. Here we merge the window list into a list of TWindowMirror:

TWindowMirror = class
private
  FIndex : Integer;
  FRgn : HRGN;
  FHandle : THandle;
  FBoundsRect : TRect;
  FProcessId : Integer;
  FImage : TBitmap;
  FDiffStreamList : TList<TImagePart>;
  ...
  ...
end;

procedure TMirrorManager.RefreshMirrorList(out OneMoved:Boolean);
  procedure GetProcessWindowList(WinList:TList<TWin>);
  begin
    WinList.Clear;
    EnumWindows(@EnumWindowsProc, Longint(WinList));
  end;

var
  wl : TList<TWin>;
  n : Integer;
  wm : TWindowMirror;
begin
  OneMoved:=False;

  wl := TList<TWin>.Create;
  try
    // Enumerates top windows
    GetProcessWindowList(wl);
    try
      for n := wl.Count - 1 downto 0 do begin
        // Looks for a cached window
        wm:=GetWindowMirror(FMirrorList,wl[n].Wnd);
       if assigned(wm) then begin
        if IsIconic(wl[n].Wnd) then
           wm.SetBoundsRect(Rect(0,0,0,0))
       else wm.SetBoundsRect(wl[n].Rect);

          // Returns true when at least one window moved
       OneMoved:=OneMoved or (DateTimeToTimeStamp(Now-wm.FMoved).time<MOVE_TIME);
        end else begin
         // Do not create a TWindowMirror for invisible windows
         if IsIconic(wl[n].Wnd) then Continue;

       wm:=TWindowMirror.Create(Self,wl[n].Wnd,wl[n].Rect, wl[n].pid);
       FMirrorList.Add(wm);
        end;
        // Saves the zIndex
        wm.FIndex:=wl.Count-n;
        // Generates clipping regions
        wm.GenRegions(wl,n);
      end;
    finally
      ClearList(wl);
    end;
    // Sorts the mirror list by zIndex
    FMirrorList.Sort;
  finally
    wl.free;
  end;
end;

Now, we do the capture:

function TWindowMirror.Capture(ANewImage:TBitmap): Boolean;
  function BitBlt(DestDC: HDC; X, Y, Width, Height: Integer; SrcDC: HDC;
                   XSrc, YSrc: Integer; Rop: DWORD): BOOL;
  begin
    // Capture only visible regions
    SelectClipRgn(DestDC,FRgn);
    result:=Windows.BitBlt(DestDC, X, Y, Width, Height, SrcDC,
                           XSrc, YSrc, Rop);
    SelectClipRgn(DestDC,0);
  end;

var
  DC : HDC;
  RasterOp,ExStyle: DWORD;
begin
  RasterOp := SRCCOPY;
  ExStyle:=GetWindowLong(FHandle, GWL_EXSTYLE);
  if (ExStyle and WS_EX_LAYERED) = WS_EX_LAYERED then
  RasterOp := SRCCOPY or CAPTUREBLT;

  DC := GetDCEx(FHandle,0,DCX_WINDOW or DCX_NORESETATTRS or DCX_CACHE);
  try
    Result:=BitBlt(ANewImage.Canvas.Handle,0,0,
    Width(FBoundsRect),Height(FBoundsRect),DC,0,0, RasterOp)
  finally
    ReleaseDC(FHandle,DC);
  end;

end;		

Now that we have captured all visible regions, we need to get the bitmap differences against the previous capture. We do this by looping through the windows, then their visible regions and finally calculating the regions where we find bitmap differences:

function TWindowMirror.CaptureDifferences(reset:boolean=false): Boolean;
  ....
begin
 ...
 result:=Capture(TmpImage);
 if result then begin
   ...
   ra:=ExtractClippingRegions(Rect(0,0,TmpImage.Width,TmpImage.Height));
   for n := 0 to Length(ra) - 1 do begin
     ra2:=GetDiffRects(FImage,TmpImage,ra[n]);
     for m := 0 to Length(ra2) - 1 do begin
       Jpg := TJpegImage.Create;
       ...
       CopyBmpToJpg(Jpg,TmpImage,ra2[m]);
       FDiffStreamList.Add(TImagePart.Create(rbmp,'jpeg'));
       Jpg.SaveToStream(FDiffStreamList[FDiffStreamList.Count-1].FStream);
       ... 
       Bitblt(FImage.Canvas.Handle,
       ra2[m].Left,ra2[m].Top,Width(ra2[m]),Height(ra2[m]),
       TmpImage.Canvas.handle, rbmp.Left,ra2[m].Top,SRCCOPY);
     end;
   end;
   ...
end;

In the next post, we'll focus on the protocol with the client and the client code. A post with updated source code can be found here.

History

ThinVNC is currently in alpha stage and this version must be taken as a technology preview. Remote mouse input is partially implemented, remote keyboard input is still missing.