Introduction

Hi guys, this is my first article and in this article i am going to show you how to count fingertips using convexity defects funtion in opencv.I hope this article would be very helpfull to those who want to learn or beginners who want to learn opencv.I will try to keep this article simple for beginners and any problem I would eager to help you. OpenCV was designed for computational efficiency and with a strong focus on real-time applications. Written in optimized C/C++, the library can take advantage of multi-core processing.Find out more about OpenCV from here.

OpenCV stands for Open Source Computer Vision and developed by Intel

Background

Experience in working with C++ and Visual Studio and general knowledge about OpenCV would be great.

Using the code

Before start with the code we must configure OpenCV with Visual Studio and below video shows the step by step approach of configuring OpenCV.

Configuring OpenCV

After configuring include below header files to make sure you have configured OpenCV with visual studio correctly. And don't forget to use namespace cv.

#include <opencv\cv.h>
#include <opencv\highgui.h>
#include <opencv2\imgproc\imgproc.hpp>
#include <opencv2\highgui\highgui.hpp>
#include <opencv2\video\background_segm.hpp>
#include <opencv2\opencv.hpp>

using namespace cv;

first we are going to create a trackbar which is later use to change the HSV color values in order to match with user's skin color.

const string trackbarWindowName = "Trackbars";

Below variable initialization is for holding minimum and maximum values of Hue, Saturation and value.

int H_MIN = 0; // minimum Hue
int H_MAX = 180; // maximum Hue
int S_MIN = 0; // minimum Saturation
int S_MAX = 255; // maximum Saturation
int V_MIN = 0; // minimum Value
int V_MAX = 255; //maximum Value

Callback funtion for the trackbar.This function gets called whenever a trackbar position is changed

void on_trackbar( int, void* )
{

// Doing nothing here since we are going to handle the changes in some other place

}

Below is the way of how to create trackbars in opencv.

void createTrackbars(){

    namedWindow(trackbarWindowName,0);
    //create memory to store trackbar name on window
    char TrackbarName[50];
    sprintf( TrackbarName, "H_MIN", H_MIN);
    sprintf( TrackbarName, "H_MAX", H_MAX);
    sprintf( TrackbarName, "S_MIN", S_MIN);
    sprintf( TrackbarName, "S_MAX", S_MAX);
    sprintf( TrackbarName, "V_MIN", V_MIN);
    sprintf( TrackbarName, "V_MAX", V_MAX);
 
    createTrackbar( "H_MIN", trackbarWindowName, &H_MIN, H_MAX, on_trackbar );
    createTrackbar( "H_MAX", trackbarWindowName, &H_MAX, H_MAX, on_trackbar );
    createTrackbar( "S_MIN", trackbarWindowName, &S_MIN, S_MAX, on_trackbar );
    createTrackbar( "S_MAX", trackbarWindowName, &S_MAX, S_MAX, on_trackbar );
    createTrackbar( "V_MIN", trackbarWindowName, &V_MIN, V_MAX, on_trackbar );
    createTrackbar( "V_MAX", trackbarWindowName, &V_MAX, V_MAX, on_trackbar );

}

Trackbar Preview

In our main function we are going to make two, three channel, empty images and frame is going to hold our original video which is captured using webcamera and frame2 will be used to show the output after some changes we do to frame image. The advantage of using Mat class is you don't want to worry about the manually allocate its memory and release it like in C structure called IplImage.

Mat frame(Size(640, 420),CV_8UC3);
Mat frame2(Size(640, 420),CV_8UC3);

int _tmain(int argc, _TCHAR* argv[])
{
    VideoCapture cap(0);  // open the default camera
    Mat frame(Size(640, 420),CV_8UC3);
    Mat frame2(Size(640, 420),CV_8UC3);
    createTrackbars(); // create trackbars
    if(!cap.isOpened())   
        return -1;

    while(true){
        cap>>frame; // new frame from camera
        imshow("Original Video", frame);
          if( cvWaitKey( 15 )==27 )  break; 
    }
    return 0;
}

If there is no error in your code you can see a window named "Original Video" which holds our video.If you want to change the camera other than the default one,just set the cap(0) to cap(1).It will open the externel camera plugged into your computer.

Now we are going to do some noise reduction before proceed further with captured video.Here i used gaussian blur to reduce image noise destination frame will be same as the source frame. kSize means the guassian kerenel size which should be positive and odd. You can find more details about GaussianBlur function and its parameters by this link

Size kSize;
kSize.height = 3;
kSize.width = 3;
double sigma = 0.3*(3/2 - 1) + 0.8;
GaussianBlur(frame,frame,kSize,sigma,0.0,4);

Below we are converting our captured frames to HSV color format.

Mat hsv(Size(640, 420),CV_8UC3);
cvtColor(frame,hsv,CV_RGB2HSV);

In range funtion takes four arguments which are,

void inRange(InputArray src, InputArray lowerb, InputArray upperb, OutputArray dst)

src - Input array (hsv mat image)

lowerb - lower boundary array or a scalar.

upperb - upper boundary array or a scalar.

dst - Binary image output with the same size as input array. White pixels represents the pixels are in-range and black pixels where they were out of the range

Mat bw(Size(640, 420),CV_8UC1);
inRange(hsv,Scalar(H_MIN,S_MIN,V_MIN),Scalar(H_MAX,S_MAX,V_MAX),bw);

Dilation and Erosion are two morphological operations. Dilation adds pixels to the bounderies of objects in an image and erosion removes pixels on object boundaries.You can read more about morphological operations here.

Mat Erode(Size(640, 420),CV_8UC1);
cv::erode(bw, Erode, cv::Mat(), cv::Point(-1,-1));

Mat Dialate(Size(640, 420),CV_8UC1);
cv::dilate(Erode, Dialate, cv::Mat(), cv::Point(-1,-1),2);

Image after erosion Image after Dialation

Eroded image Dialated image

Ok. now we are going to find contours of our input video. Iam using Dialated image as the input image for the findContours funtion since it requires a binary image. As mentioned in the opencv documentation you can use compare() , inRange() , threshold() , adaptiveThreshold() , Canny() and others to create binary image from a grayscale or colored image.

vector<Vec4i> hierarchy;
vector<vector<Point> > contours_hull;

findContours(Dialate.clone(), contours_hull, hierarchy, CV_RETR_TREE , CV_CLOCKWISE, Point(0, 0) );

In below code segment we are getting convex hull points and defects points using our contour and convexhull. After that we are drawing the contour and the hull only for our biggest contour. In here we are also drawing bounding box and rotated rectangle for our biggest contour. i have add some comments for the drawing parts then you can understand the use of those codes.

for( int i = 0; i < contours_hull.size(); i++ )
     { 
         convexHull( Mat(contours_hull[i]), hull[i], false );
         convexHull( Mat(contours_hull[i]), hullsI[i], false );
         convexityDefects(Mat(contours_hull[i]),hullsI[i], defects[i]);

            if(IndexOfBiggestContour == i)
               {
                  minRect[i] = minAreaRect( Mat(contours_hull[i]) );
                    
                 //draw contour of biggest object
                  drawContours( frame2, contours_hull,IndexOfBiggestContour, CV_RGB(255,255,255), 2, 8, hierarchy,0, Point() );
                //draw hull of biggesr object
                  drawContours( frame2, hull, IndexOfBiggestContour, CV_RGB(255,0,0), 2, 8, hierarchy, 0, Point() );

                  
                  approxPolyDP( Mat(contours_hull[i]), contours_poly[i], 3, true );
                  boundRect[i] = boundingRect( Mat(contours_poly[i]) );
                 
                  //draw rectangle for the biggest contour
                  rectangle( frame2, boundRect[i].tl(), boundRect[i].br(), CV_RGB(0,0,0), 2, 8, 0 );

                  Point2f rect_points[4];
                  minRect[i].points( rect_points );

                  for( int j = 0; j < 4; j++ )
                      {
                        //draw rotated rectangle for the biggest contour
                        line( frame2, rect_points[j], rect_points[(j+1)%4], CV_RGB(255,255,0), 2, 8 );
                      }

               }
     }

Drawing convexhull and the biggest contour

hull image

Drawing bounding box for the biggest contour

Drawing Rotated rectangle for the biggest contour

rotated rectangle

Below code segment is for drawing the defects points of the biggest contour.You can avoid unwanted defect points using the depth value. Depth value refer to the distance between the farthest point and the convex hull. In my case i have removed some defects points using depth value.It might not work properly in your case.So you can further add some conditions to avoid unwanted points.

                size_t count = contours_hull[i].size();
                std::cout<<"Count : "<<count<<std::endl;
                if( count < 300 )
                    continue;

                vector<Vec4i>::iterator d =defects[i].begin();

                while( d!=defects[i].end() ) {
                    Vec4i& v=(*d);
                    if(IndexOfBiggestContour == i){

                        int startidx=v[0]; 
                        Point ptStart( contours_hull[i][startidx] ); // point of the contour where the defect begins
                        int endidx=v[1]; 
                        Point ptEnd( contours_hull[i][endidx] ); // point of the contour where the defect ends
                        int faridx=v[2]; 
                        Point ptFar( contours_hull[i][faridx] );// the farthest from the convex hull point within the defect
                        float depth = v[3] / 256; // distance between the farthest point and the convex hull

                        if(depth > 20 && depth < 80)
                        {
                        line( frame2, ptStart, ptFar, CV_RGB(0,255,0), 2 );
                        line( frame2, ptEnd, ptFar, CV_RGB(0,255,0), 2 );
                        circle( frame2, ptStart,   4, Scalar(100,0,255), 2 );
                        }
                    }
                    d++;
                }

Drawing starting and end points of defects

defects points

Final output of the program

final output

Conclusion

Perpose of this OpenCV development was to aimed at real time-time computer vision.It focuses mainly on real-time image processing.By learning OpenCV you can develop Facial recognition system, Gesture recognition, HCI, Mobile robotics, Motion tracking, Augmented reality and so on.So hope you can see the importance of learing this and I hope to countinue more about OpenCV with my next article.

Happy Programming!

Beginner's guide to understand Fingertips counting using convexity defects in OpenCV