Click here to Skip to main content
15,397,385 members
Articles / Multimedia / DirectX
Posted 21 Mar 2007


28 bookmarked

Why Play Guitar Hero When An Android Can Do It For You?

Rate me:
Please Sign up or sign in to vote.
4.77/5 (17 votes)
21 Mar 2007CPOL3 min read
An android that plays the Guitar Hero game. A DirectX DirectShow filters the video feed from the PlayStation2, detects where, when and how to play and moves the fingers accordingly.
Screenshot - gse_multipart62304.jpg


This DirectShow filter is used to detect the notes inside a GuitarHero game video.

The brain of the Robot was developed by Rafael Mizrahi, a GarageGeeks made-man and research manager at Feng-GUI, the artificial vision lab.

You can look at the GuitarHeroNoid source code home page at Google code service.


In the GuitarHero game, each song is presented on a set of five columns, resembling a real guitar fret board that scrolls constantly towards the player. The five columns correspond to the five fret buttons and appropriately colored notes appear in these columns.

Image 2

Detecting the notes could be accomplished by using several approaches to detect moving objects; most of them might not be fast enough to implement and use. So we came up with this idea: looking at the game, you can quickly realize that the important information such as the plates is brighter than the rest of the image. Set a detection area which contains 5 trapezoids at the bottom of the picture.

f = 0; // f helps to create a trapezoid.
for (int i=0;i<m_plates_height;i++) {
  for (int j=0;j<m_plates_width-f*2;j++) {
    prgb[j+i*cxImage+(int)f].rgbtRed   = (BYTE) 0;
    prgb[j+i*cxImage+(int)f].rgbtGreen = (BYTE) 0;
    prgb[j+i*cxImage+(int)f].rgbtBlue = (BYTE) 255;
  f = f + 1;

Image 3

Representing the color of the pixels in each trapezoid from RGB as HSV (Hue, Saturation, Value), also known as HSB (Hue, Saturation, Brightness), and defining a threshold (Brightness Threshold at the properties dialog), somewhere in the middle of the Brightness value, gives you a binary representation of the pixels inside the trapezoid. Now, when the trapezoid is filled with enough white pixels, there is probably a plate over there.

// get current pixel RGB
R = (int)prgb[j+i*cxImage+(int)f].rgbtRed;
G = (int)prgb[j+i*cxImage+(int)f].rgbtGreen;
B = (int)prgb[j+i*cxImage+(int)f].rgbtBlue;

// HSV calculation

// YUV: doesn't seem as better as HSV
//Y = (( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16);
//V = Y;

// threshold V to binary, and sum the white pixels
byte b = (byte)(((V >= min) && (V <= max)) ? 255 : 0);
if(b == (BYTE)255)


Note: Different PlayStations (PAL NTSC) have different brightness.

Image 4

We connect the game video output using a capture device into a computer. A
live video streaming filter captures the video frames as images and sends each
image into the image processing part of the program that detects the notes.

Image 5

Send to Robot

The information is sent to the robot. There are three options for sending: File, Port to the parallel output, and Socket. In the socket option, you can divide the process into two computers and by that, perform a Remote surgery.

// play the 5 without a strum hit. infact, release the strum up.
OutputToRobot(((play_plate & PLATE_1) == PLATE_1),
  ((play_plate & PLATE_2) == PLATE_2),
  ((play_plate & PLATE_3) == PLATE_3),
  ((play_plate & PLATE_4) == PLATE_4),
  ((play_plate & PLATE_5) == PLATE_5),

Delays, Delays, Delays

To play a note, the player must hold the correct fret button and press the strum bar. After playing and watching the game, you find out that the PlayStation adds another delay to the equation. Pressing the frets buttons is recognized by the PlayStation game within some 100 milliseconds or so. Having this delay along with the delay it takes for the strum solenoids to go up and down, I realized that sending fret notes together with the strum action is not possible. I divided the protocol into two main actions:

  • The first, at the area of detection, pressing one or more of the fret notes lifts up the strum. The fret notes information is added into a FIFO queue.
  • The second, after about 250 milliseconds (Strum Delay at the properties dialog), pops the fret notes from the queue and sends them to the guitar with a strum down.

A nice TODO is to detect the BPM of the song during the first few seconds of the song, and adjust all those delays according to that.

Screenshot - guitar_hero_multiplayer1_small.jpg

Multiplayer is split-screen. In a "duelling guitars" fashion, two players tackle segments of the selected song. Unlike other modes, it is not possible to fail a song in multiplayer, but scoring dictates that one player will generally win. Just by moving the area of detection to the side of the screen, using smaller trapezoids, all parameters in a different configurations file (guitarhero.ini), and there you go, you can play with or against the GuitarHeronoid.

That's all, guys. Keep on rocking in C++.
You can find more information and videos here.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Rafael Mizrahi
Chief Technology Officer Feng-GUI
United States United States
Rafael Mizrahi is a CTO at Feng-GUI

He is also a hacker contributor at the mono project, and a GarageGeeks member.

Rafael Mizrahi personal blog -

Comments and Discussions

GeneralCool project Pin
Scope23-Mar-07 6:07
MemberScope23-Mar-07 6:07 
GeneralRe: Cool project Pin
Rafael Mizrahi23-Mar-07 8:24
MemberRafael Mizrahi23-Mar-07 8:24 
GeneralEasily underestimated in coolness! Pin
Shawn Poulson22-Mar-07 1:47
MemberShawn Poulson22-Mar-07 1:47 
GeneralRe: Easily underestimated in coolness! Pin
Rafael Mizrahi22-Mar-07 23:09
MemberRafael Mizrahi22-Mar-07 23:09 
GeneralCool article Pin
ETA21-Mar-07 20:27
MemberETA21-Mar-07 20:27 
GeneralRe: Cool article Pin
Rafael Mizrahi21-Mar-07 21:05
MemberRafael Mizrahi21-Mar-07 21:05 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.