Click here to Skip to main content
15,126,853 members
Articles / Programming Languages / C++
Posted 22 Nov 2001


194 bookmarked

Mouse gestures recognition

Rate me:
Please Sign up or sign in to vote.
4.88/5 (56 votes)
22 Nov 20013 min read
Feedforward multilayer neural network and mouse gesture recognition

Sample Image


Recently I installed Opera 5 and was impressed on a Gesture UI. Moreover several weeks ago I noticed a discussion on CodeProject's Lounge about it. To all appearances people want to know about it too :). IMHO, the neural network most suitable for this purpose. As I a little know neural network I tried to implement such feature themselves.

Neural Network

What is Neural Network ? Hm it's not easy to say. A rephrased definition Zurada, J.M.:

"Neural network software is a software which can acquire, store, and utilize experiential knowledge."

I think I can point any person concerned to theory directly to several neural network sites. Here is small list of web resources about Neural networks:


Let's return to mouse gestures. After some research I have chosen a multilayer perceptron and standard back-propagation algorithm for training. The main problem was in the representation of an input data for neural network. The best result I found was in the transformation of a mouse path into a vector of cosines and sines.

For example:

path   {170:82 172:83 175:85 177:86 ...} 
transformed into 
vector {0.45 0.55 0.45 0.71 ... 0.89 0.83 0.89 0.71 ...}

Recognition algorithm.

  1. record a mouse path
  2. smooth a path to a base points
  3. transform points to angles' vector
  4. compute sines and cosines
  5. pass values (cosines and sines) to network's inputs
  6. apply softmax function on an output network vector
  7. find and verify a winner

Neural network architecture.

  • input layers : 32 sinapses
  • hidded layer : 32 neurons
  • output layer : 29 axons (one for each gesture)
  • fully connected layers
  • transfer function : log-sigmoid
  • incremental training algorithm, standard back-propagation method
  • momentum, variable learning rate (slowly reduced)
  • input noise



Sample Image

Before testing the recognition ability you must train the network (or you can load an file image of trained net). You can customize the parameters of the training process, namely: maximum number of cycles, a momentum value, a learning rate, a minimum value of mean square error (in other words "target error"). The training process will stop after achieving either of the conditions: maximum number of cycles or target error. During the training process you can keep an eye on a error's graph, a current gesture (with noise) and 2D network presentation.


As soon as you have a trained net, you can test it. Select the patterns (or test all of them), a speed value and a noise level. Besides, you can familiarize oneself with ideal presentation of gestures via setting minimal noise and minimal speed.


For recognition of mouse gestures you must press right mouse button during moving a mouse. For example for recognition "left" gesture, press right mouse button and move a mouse to the left. If a neural network can recognize the gesture, then you will see the name, probability and ideal presentation of winner. Because of freeware nature of GestureApp the mouse path must have at least 16 points :(. Sorry I didn't implemented a "stretch a path" feature so far.

Note: the direction is very important.

The network is trained to recognize the gestures but not 2D images. Hence, you can draw the "circle" gesture a thousand different ways, but the only valid way is: press mouse button and move a mouse to the right and down and so on. Once more: it's gesture, not 2D image.

Mouse gestures

Image 4 Image 5 Image 6 Image 7
Image 8 Image 9 Image 10 Image 11
Image 12 Image 13 Image 14 Image 15
Image 16 Image 17 Image 18 Image 19
Image 20 Image 21 Image 22 Image 23
Image 24 Image 25 Image 26 Image 27
Image 28 Image 29 Image 30
Image 31 Image 32


Compatible with Win2k, WinXP, Win98, WinMe. Unfortunately doesn't work on WinNT because of the need for the AlphaBlend API.


Special Thanks:
My wife Julia for her nice artwork ;)

And thanks to:
Pedro Pombeiro for Selection slider control


This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


About the Author

Konstantin Boukreev
Web Developer
Russian Federation Russian Federation
I am freelance programmer. About 3 years of experience in C++ and I would rather use ATL, STL, WTL but not MFC Smile | :) . Main backgrounds are Win32 API, COM and Networking. Now I am interested about AI (Neural Network, Fuzzy Logic and GA). Currently based in Vladivostok, Russia.

Comments and Discussions

QuestionMLP? Pin
Am3320115-Aug-12 12:32
MemberAm3320115-Aug-12 12:32 
GeneralMy vote of 2 Pin
CliffordRoche25-Feb-11 8:53
MemberCliffordRoche25-Feb-11 8:53 
GeneralStill very nice but needs a few changes Pin
baylorw17-Nov-08 16:21
Memberbaylorw17-Nov-08 16:21 
Generaldigitizer or pen tablet processing Pin
HOYAM11-Jul-07 11:50
MemberHOYAM11-Jul-07 11:50 
Generalge processing I also need document about image processing and code Pin
Member 38795531-Mar-07 16:54
MemberMember 38795531-Mar-07 16:54 
GeneralIntegrate the project in a console opengl application? HELP! Pin
ninkata1-Mar-07 0:46
Memberninkata1-Mar-07 0:46 
QuestionMean square least to fit a circle Pin
Michae_geffen7-Jan-07 11:00
MemberMichae_geffen7-Jan-07 11:00 
Generalneed algorithm for VB6. anyone could help? PLEASE... Pin
Smart Canix25-Jun-04 4:46
MemberSmart Canix25-Jun-04 4:46 
GeneralContinous Monitoring Pin
adsjk29-Dec-03 12:13
Memberadsjk29-Dec-03 12:13 
GeneralInterpolating < 16 points. Code included Pin
adamtegen11-Nov-03 12:09
Memberadamtegen11-Nov-03 12:09 
GeneralRe: Interpolating < 16 points. Code included Pin
Apuhjee19-Sep-08 6:45
MemberApuhjee19-Sep-08 6:45 
GeneralCompiling with .NET Pin
quorm26-Aug-03 14:35
Memberquorm26-Aug-03 14:35 
GeneralRe: Compiling with .NET Pin
John M. Drescher26-Aug-03 15:01
MemberJohn M. Drescher26-Aug-03 15:01 
GeneralRe: Compiling with .NET Pin
quorm26-Aug-03 15:41
Memberquorm26-Aug-03 15:41 
GeneralRe: Compiling with .NET Pin
John M. Drescher26-Aug-03 15:50
MemberJohn M. Drescher26-Aug-03 15:50 
GeneralNot really know how to use your app! Pin
Member 160351-Jul-03 8:27
MemberMember 160351-Jul-03 8:27 
GeneralRe: Not really know how to use your app! Pin
matlin12326-Jul-03 9:51
Membermatlin12326-Jul-03 9:51 
GeneralNeed help in image recognition Pin
alexpop19-Jun-03 4:36
Memberalexpop19-Jun-03 4:36 
GeneralNeed help in speech recognition Pin
Mahesh Bhargava30-May-06 20:36
MemberMahesh Bhargava30-May-06 20:36 
GeneralNeed help in image recognition Pin
alexpop19-Jun-03 4:29
Memberalexpop19-Jun-03 4:29 
GeneralA great try, however ... Pin
TeleStar22-Oct-02 19:37
MemberTeleStar22-Oct-02 19:37 
Questionwhat is this proggy good for ? Pin
Anonymous19-Jul-02 6:39
MemberAnonymous19-Jul-02 6:39 
GeneralReally Cool Pin
Tom Mason21-Mar-02 6:12
MemberTom Mason21-Mar-02 6:12 
GeneralYou may consider a little improvements... Pin
4-Jan-02 22:49
suss4-Jan-02 22:49 
GeneralRe: You may consider a little improvements... Pin
Konstantin Boukreev13-Jan-02 22:58
MemberKonstantin Boukreev13-Jan-02 22:58 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.