Kinect Recording Interface

• Download source code - 1.4 MB

Introduction

Since I have started to develop desktop applications using Kinect, I had to test my code by standing-up in front of the sensor. At the beginning, it was funny, but as soon as I had to do it tens of times within a day, it was starting to be annoying. So, I decided to create an application that could record and playback any moves!

Background

In this article, it is assumed that you have basic knowledge of using Kinect sensor & Kinect SDK (not OpenNI) and you are familiar with C#. The application is written for Kinect Beta 2 SDK driver version 1.0.0.45.

If your are not familiar with Kinect, take a look at a great series of tutorials!

Prerequisites

• Kinect Sensor
• Kinect Beta 2 SDK
• .NET Framework 4.0
• Visual Studio
• Json.Net (included in project)

Using KinectRecorderInterface

Using KinectRecorderInterface class is pretty much easy. First of all, let's take a look at the public members/methods/events of the class.

Its use is intended to be as simple as possible!

Declaring creating the class as most of the classes.

 KinectRecorderInterface recorder = new KinectRecorderInterface();

Initializing open the Kinect sensor through the recorder and select an appropriate ImageResolution (DepthImage can be up 640x480).
Notice: Only one application at a time can use the Kinect!!

Use to open only the DepthStream:

recorder.InitializeSensor(ImageResolution.Resolution320x240);

Use to open both of DepthStream & VideoStream:

recorder.InitializeSensor(ImageResolution.Resolution320x240, ImageResolution.Resolution640x480);

Whole Initialization code:

private KinectRecorderInterface recorder;
public void InitializeKinectRecorder()
{
    recorder = new KinectRecorderInterface();
 
    //Initialize Sensor through recorder
    recorder.InitializeSensor(ImageResolution.Resolution640x480, ImageResolution.Resolution640x480);
 
    //Set all the event handlers
    recorder.PlaybackSkeletonDataReady += 
       new KinectRecorderInterface.PlaybackFrameHandler(recorder_PlaybackSkeletonDataReady);
    recorder.SkeletonFrameReady += 
       new KinectRecorderInterface.SkeletonFrameHandler(recorder_SkeletonFrameReady);
    recorder.ImageFrameReady += 
       new KinectRecorderInterface.ImageFrameHandler(recorder_ImageFrameReady);
    recorder.DepthFrameReady += 
       new KinectRecorderInterface.DepthFrameHandler(recorder_DepthFrameReady);
} 

That's all!

Assigning event handlers: Now you can create proper event handlers to take data from Kinect. Notice: EventHandlers and the EventArgs are the same as using the Kinect SDK, except for the playback!!

recorder.SkeletonFrameReady += 
    new KinectRecorderInterface.SkeletonFrameHandler(recorder_SkeletonFrameReady);
recorder.ImageFrameReady += 
    new KinectRecorderInterface.ImageFrameHandler(recorder_ImageFrameReady);
recorder.DepthFrameReady += 
    new KinectRecorderInterface.DepthFrameHandler(recorder_DepthFrameReady);

Playback EventHandler is pretty much the same, but it differentiates its EventArgs (see later on 'SkeletonData VS SerializableSkeletonData').

recorder.PlaybackSkeletonDataReady += 
    new KinectRecorderInterface.PlaybackFrameHandler(recorder_PlaybackSkeletonDataReady); 

Now, you can retrieve Kinect's data as usually (in realtime) from SkeletonTracking, DepthStream and VideoStream, but there is no recording or playback at the time.

Starting / Stop Recording Anytime you can start StartRecording() method, in order to start capturing only available (Tracked) SkeletonData. Use StopRecording() method, in order to stop capturing and save any stored SkeletonData. Saving data are stored by default in "Recordings" folder, unless you change it from RecordDirectory.
Notice: If recording has no SkeletonData, saving will be aborted.

recorder.RecordDirectory = "./StoreData"; 

recorder.StartRecording();
//Wait as much you want
recorder.StopRecording();

Now, by default, a new folder "Recordings" has been created along with the project's executable; and a file called "recording_00x.dat" has been created within the folder.

# Start / Stop Playback

After you have recorded some moves, you are able to playback them any time you want to. Use StartPlayback() to start playback of the last available recording. Also, you can use StartPlayback(recording_xxx.dat), in order to playback a recording of your choice.
Notice: Each time the application is starting/recording/saving an updated catalog (List<string>) of the recordings is available through the ExistingRecords attribute, to let you choose any recording.

//Play the last available recording
recorder.StartPlayback();  

//Play a record of your choice
List<string> records = recorder.ExistingRecords;
recorder.StartPlayback(records[0]);

SkeletonData VS SerializableSkeletonData

As soon as I reached the point that I had to store the data, I realized that I couldn't serialize SkeletonData because it was not marked as Serializable. For that reason, I implemented Json.Net which worked great! But, unfortunately, when I tried to deserialize and cast (see below) the stored data, it failed again as SkeletonData does not have any public constructor (its constructor is set to internal)!! A better solution is welcome!!!

<Code Not Working>

//This code gave me an error because there is no public constructor for SkeletonData
var json = System.IO.File.ReadAllText(filepath);
var skeletonData = Newtonsoft.Json.JsonConvert.DeserializeObject<List<SkeletonData>>(json);

</Code Not Working>

Finally, I didn't come up with any other solution rather than create a new class almost identical to SkeletonData as shown below:

At the only point that you will get SerializableSkeletonData, will be at the time that you will get SkeletonDataArgs from PlaybackFrameHandler. SkeletonDataArgs holds SerializableSkeletonData.

Using SerializableSkeletonData: It's almost the same process as using SkeletonData!

For example, let's say that we have the following code for SkeletonData:

private void recorder_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs skeletonFrameArgs)
{
    //Get the first skeleton from SkeletonFrame
    SkeletonData skeleton = skeletonFrameArgs.SkeletonFrame.Skeletons[0];
 
    if (skeleton != null)
    {
        //Do something with your data
        setEclipsePos(HeadEclipse, skeleton.Joints[JointID.Head]);
        setEclipsePos(LeftHandEclipse, skeleton.Joints[JointID.HandLeft]);
        setEclipsePos(RightHandEclipse, skeleton.Joints[JointID.HandRight]);
    }
}

The same function through Playback using SerializableSkeletonData:

private void recorder_PlaybackSkeletonDataReady(object sender, SkeletonDataArgs skeletonDataArgs)
{
    SerializableSkeletonData skeleton = skeletonDataArgs.serializableSkeletonData;
 
    if (skeleton != null)
    {
        //Do something with your data
        setEclipsePos(HeadEclipse, skeleton.Joints[(int)JointID.Head]);
        setEclipsePos(LeftHandEclipse, skeleton.Joints[(int)JointID.HandLeft]);
        setEclipsePos(RightHandEclipse, skeleton.Joints[(int)JointID.HandRight]);
    }
}

Extra Properties

KinectRecorderInterface has a PlayPause method while you are in recording or playback mode.

Use PlayPauseRecording() or PlayPausePlayback() each time you want to pause your recording or playback. Use the same method again to continue..!

//Pause or Resume recording
recorder.PlayPauseRecording(); 

//Pause or Resume playback
recorder.PlayPausePlayback();

Use Status to check or see the current status of the recording interface!

public enum KinectRecorderStatus
{
        Idle = 0,
        Initializing = 1,
        Recording = 2,
        Playing = 3,
        Saving = 4,
        Loading = 5,
        NotConnected = 6,
        PlayingPaused = 7,
        RecordingPaused = 8
}
//Print the current status
statusLabel.Text = recorder.Status.ToString();
//Check 'Status'
if (recorder.Status == KinectRecorderStatus.Playing)
{
    //Do something
}

Application already has HotKey identification running in the background! You can set, up to 4 HotKeys to handle:

• Start / Stop Recording: RecordHotKey
• Start / Stop Playback: PlaybackHotKey
• Pause / Resume Recording: PauseRecordingHotKey
• Pause / Resume Playback: PausePlaybackHotKey

It is not necessary to enable HotKeys, but you have to assign them a Key value.

//Set 'F9' as HotKey for recording
recorder.RecordHotKey = System.Windows.Forms.Keys.F9;

Application's Class Diagram

Including only the public attributes: