Click here to Skip to main content
15,907,231 members
Articles / Multimedia / GDI+

C# Application to Create and Recognize Mouse Gestures (.NET)

Rate me:
Please Sign up or sign in to vote.
4.82/5 (39 votes)
17 Mar 2008CPOL5 min read 222.9K   8.1K   144   54
This program can create and recognize mouse gestures.
Sample Image - CMGBlade.jpg


It was about time to update the content of this article.
The original article was written in 2004, and the code provided was updated one year later, even if it was not available to download from this page. That said, this software is able to recognize mouse gestures, and was originally inspired by this article.

Even though I took inspiration from the article posted by Konstantin Boukreev, this application is quite different (different programming language, gesture management and neural network at least), even though it uses a similar graphic interface (I found it amusing!) and the same logic behind the computation of gesture features.

This update contains a lot of changes (to be honest, the code has been completely rewritten), and provides an easier integration and a more flexible and extensible implementation of the neural network, moreover the solution file is for Visual Studio 2008.
More information will be provided below.

NOTE: If you are not familiar with artificial neural networks (ANN), please have a look at this page, since explaining such subject in detail goes beyond the scope of this article.


The complete solution is divided into four projects:

  1. NeuralNetworks: A library allowing neural networks creation and manipulation
  2. MouseGestures: A library allowing mouse gestures definition
  3. GestureRecognizer: An executable, providing a possible implementation and integration of previous libraries
  4. Test: A console application used for testing purposes

Regarding the first project, it contains a feed forward neural network implementation (composed of layers of neurons), some typical activation functions to be assigned to the neurons, an implementation of the Backpropagation algorithm and the PerformanceMonitor class, used to compute statistics about the recognizing performances of the system.
The network implementation assumes that input features are already normalized (in other words there is no input layer dedicated to such a task).

The second library contains the implementation of the Gesture class, which is mainly a sorted collection of bidimensional points, and the GestureSet class (a collection of gestures, used to store training, test and validation sets used by the neural network), both providing load/save capabilities (files are saved in XML format).

Other than the above classes, there are some helper functions used to manipulate the geometry of the gestures.

The main difference between this version and the previous one is the ability to extract a variable number of features given the same number of gesture points; such a task is achieved by adding or removing points from the gesture, trying to minimize the deformation of the gesture (refer to the ExtractFeatures(int count) function in the Gesture.cs file).

The executable project is probably the most interesting, since it gives an immediate visual feedback to the user.

Just like the previous version, the classificator is composed of a set of competitive neural networks, each one used to recognize just one gesture; the main difference lies in the ability to customize the morphology of the net, choosing the number of inputs, layers, neurons per layer and type of activation functions (refer to menu entry Neural net -> Neural Network).

To use the application properly, first define the gestures you want to recognize (Settings -> Manage gestures -> Create gesture), then create the training set (Settings -> Manage gestures -> Create training set) and finally create the test set (Settings -> Manage gestures -> Create test set).

In this version, the gesture sets creation can be performed manually or automatically, moreover it is possible to choose different 'seeds' during automatic creation.
Once gesture sets are created, it is possible to start the training phase (Neural net -> Training and, at the end of that, try out the recognizing capabilities of the system by hand (NeuralNet -> Verify) or by using the performance monitor (NeuralNet -> Performances).
Another news is the possibility to set a confidence threshold for each neural network, which could improve recognizing abilities of the system.

Gestures, gesture sets and neural networks can be saved and loaded as XML files in the APP_PATH\Gestures\ folder; the application uses the following convention (GESTURE is a placeholder for the gesture name):

  1. APP_PATH\Gestures\GESTURE.xml: File used to store the path of the gesture, training set, test set and neural network files
  2. APP_PATH\Gestures\GESTURE\: Folder containing gesture, training set, test set and neural network XML files
  3. APP_PATH\Gestures\GESTURE_G.xml: File containing gesture points
  4. APP_PATH\Gestures\GESTURE_S.xml: File containing gestures used as training set
  5. APP_PATH\Gestures\GESTURE_VS.xml: File containing gestures used as test set
  6. APP_PATH\Gestures\GESTURE_NN.xml: File containing neural network data

About the Neural Network

In the previous version of this article, I wrote that it could be interesting to test out the performances of a classificator based on Autoassociators, compared to a classificator based on Multilayer Perceptrons; well, the good news is that the current version can deal with this: when the neural network input number is the same as the output number, training and verify phases behave differently, allowing the identification of which classificator performs better.


This article introduced a method to recognize mouse gestures using feedforward artificial neural networks, for educational purposes.

Two libraries were provided, which could be used to implement a custom gesture recognizing system; an application, contained in the solution, shows not only a possible implementation, but even a simple program which can be used to develop and test custom recognizing systems.


  • 21st October, 2004: Article posted
  • 14th March, 2008: Article updated (code has been completely rewritten)


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
Software Developer (Senior) Apex s.r.l.
Italy Italy
I got my Computer Science (Engineering) Master's Degree at the Siena University (Italy), but I'm from Rieti (a small town next to Rome).
My hobbies are RPG, MMORGP, programming and 3D graphics.
At the moment I'm employed at Apex s.r.l. (Modena, Italy) as a senior software developer, working for a WPF/WCF project in Rome.

Comments and Discussions

GeneralRe: english comments Pin
Daniele Mancini12-Nov-04 17:05
Daniele Mancini12-Nov-04 17:05 
GeneralRe: english comments Pin
Daniele Mancini27-Nov-04 4:12
Daniele Mancini27-Nov-04 4:12 
GeneralRe: english comments Pin
DavidNohejl29-Nov-04 1:51
DavidNohejl29-Nov-04 1:51 
GeneralRe: english comments Pin
Daniele Mancini29-Nov-04 2:19
Daniele Mancini29-Nov-04 2:19 
GeneralRe: english comments Pin
Daniele Mancini26-Dec-04 14:52
Daniele Mancini26-Dec-04 14:52 
GeneralGold! Pin
User 9137327-Oct-04 15:06
User 9137327-Oct-04 15:06 
GeneralRe: Gold! Pin
Daniele Mancini27-Oct-04 16:17
Daniele Mancini27-Oct-04 16:17 
GeneralNice job Pin
Patrik Svensson22-Oct-04 2:25
Patrik Svensson22-Oct-04 2:25 
I'm also attending Computer engineering and we're also learning about neural networks at the moment. Nice to see some kind of implementation of it Smile | :)

"If knowledge can create problems, it is not through ignorance that we can solve them."
Isaac Asimov
GeneralRe: Nice job Pin
Daniele Mancini22-Oct-04 2:34
Daniele Mancini22-Oct-04 2:34 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.