Finger Mouse

http://tarandeep.googlepages.com/fingermouse

An application to track index finger and control mouse pointer. Clicks are recognized by a thumb gesture

I did the finger mouse project using openCV on windows. The code was compiled using MS visual C++. If you want to do it on linux, the only thing that you will need to change is how to pass mouse click events to the OS. In windows (using visual C++) that was easy.

The core system depended on openCV's CAMSHIFT algorithm. If you download openCV, it has a camshift example in examples/camshift.c You can run it and get a lot of insights from it.

Using CAMSHIFT you can track a particular color in a video sequence (or video stream coming from webcam). However, in this case, you want to track your hand. Hand tracking is not a solved problem (in the sense that the solutions work only in constrained environment).

I made the following assumptions-

1) The only skin color present in the frame is my hand
2) The background is very different from my skin color.

I will suggest, you start with this constrained environment and later you can work to remove these constraints.

So to track hand, all I needed was to track the skin color in the frame. Now skin color varies with person to person, also with different lighting, the recorded skin color varies. Again you can feel that tracking skin color is not a fully solved problem.

To track skin color, there are a lot of possiblities-

1) You can collect samples of lot of skin colors under various lighting conditions and build a database. Feasible but not full proof.

2) Track person's face using opencv's face detector (it does nto rely on skin color but uses certain features- google on Viola jones face detection). Once you get a bounding box around person's face, you can fetch the color distribution and that will be the person's skin color. And then you can track the hand.

3) Simply ask the person to place his/her hand at a specified place (you can show a rectangle in the center of screen and ask person to place hand there). Then collect the skin color from there and start tracking. This is the easiest way and I did the same.

Once you are able to track your hand, the next step is to track hand gesture - pointed finger as mouse, thumb gesture for click etc.

There are 2 ways to do it-

1) From the tracked region, separate the background and convert into binary - the pixels constituting your hand/skin gets value 255 (white) and remaining pixels get value 0( black). This is called blob of your hand. Now you can analyze this blob for various shapes, like if your thumb is out, then there will be more white pixels at one end of the hand etc.

2) Use the blob created above and train opencv's machine learning algo to recognize various shapes. This is better way.

Don't forget to normalize the image to some fixed size in thsi case as well as previous case. You can also apply smooth filter (gaussian filter) to smooth out the blob image obtained.

Once you get shape, you can send mouse down/up events to OS (on windows, it was fairly easy. I forgot the API, but I remember a simple google got me the correct API)

Hope this helps,
Tarandeep

Category:

Entertainment

Tags:

• finger mosue
• computer vision
• visual interface

[NeoCortex3]

The source is missing cv.h and highgui.h

[tasingh]

@NeoCortex3 those files are part of opencv. You need to include opencv libraries while compiling.

[rizi619]

can you please upload .net edition ??

[tasingh]

@rizi619 @rizi619 I don't have .net edition code. What ever I have, I have uploaded it here-

Also, if you understand the algorithm description (CAMSHIFT etc), then you can easily port it to .net.

[rizi619]

@tasingh Sir i have a project idea on text recongnition via web cam is there this kind of project you know about then please tell me abt it..

[tasingh]

@rizi619 I don't know of any such project, but technically it looks more like a machine learning project- you train a classifier to recognize digits/characters etc and given any new text that you capture via webcam, you output the result of the classifier. You can also take a look into other learning techniques esp. Hidden Markov models.

[GTALMStarLeo]

Skype - psnlmstar

Programm give me plz

[NeoCortex3]

@tasingh ah okay, thanks.

[sbilal807]

how to compile it plz make a totdorial

[accaunt4o]

@rizi619 Try google-ing OCR Software

[TRUPAC12]

Is it possible to use this with led-s?

[Msgrace155]

so sad=( i followed the link above.. I was not able to view the codes...

would you mind to share the codes to me??

plzz.:D

[Msgrace155]

so sad =( i followed the link above... i cant view the codes anymore....

could you share me the codes?