An application to track index finger and control mouse pointer. Clicks are recognized by a thumb gesture
I did the finger mouse project using openCV on windows. The code was compiled using MS visual C++. If you want to do it on linux, the only thing that you will need to change is how to pass mouse click events to the OS. In windows (using visual C++) that was easy.
The core system depended on openCV's CAMSHIFT algorithm. If you download openCV, it has a camshift example in examples/camshift.c You can run it and get a lot of insights from it.
Using CAMSHIFT you can track a particular color in a video sequence (or video stream coming from webcam). However, in this case, you want to track your hand. Hand tracking is not a solved problem (in the sense that the solutions work only in constrained environment).
I made the following assumptions-
1) The only skin color present in the frame is my hand 2) The background is very different from my skin color.
I will suggest, you start with this constrained environment and later you can work to remove these constraints.
So to track hand, all I needed was to track the skin color in the frame. Now skin color varies with person to person, also with different lighting, the recorded skin color varies. Again you can feel that tracking skin color is not a fully solved problem.
To track skin color, there are a lot of possiblities-
1) You can collect samples of lot of skin colors under various lighting conditions and build a database. Feasible but not full proof.
2) Track person's face using opencv's face detector (it does nto rely on skin color but uses certain features- google on Viola jones face detection). Once you get a bounding box around person's face, you can fetch the color distribution and that will be the person's skin color. And then you can track the hand.
3) Simply ask the person to place his/her hand at a specified place (you can show a rectangle in the center of screen and ask person to place hand there). Then collect the skin color from there and start tracking. This is the easiest way and I did the same.
Once you are able to track your hand, the next step is to track hand gesture - pointed finger as mouse, thumb gesture for click etc.
There are 2 ways to do it-
1) From the tracked region, separate the background and convert into binary - the pixels constituting your hand/skin gets value 255 (white) and remaining pixels get value 0( black). This is called blob of your hand. Now you can analyze this blob for various shapes, like if your thumb is out, then there will be more white pixels at one end of the hand etc.
2) Use the blob created above and train opencv's machine learning algo to recognize various shapes. This is better way.
Don't forget to normalize the image to some fixed size in thsi case as well as previous case. You can also apply smooth filter (gaussian filter) to smooth out the blob image obtained.
Once you get shape, you can send mouse down/up events to OS (on windows, it was fairly easy. I forgot the API, but I remember a simple google got me the correct API)