Face Features Detection System - with OpenCV

[heguner]

hi, how could you detect the eyebrows? could you please give a hint?

[GladPL]

@heguner Generally it's based on contrast difference between brows and skin..

[fadillaazeaza]

wow.. i wanna  make a program about image processing but still confuse what must i do

[GladPL]

@fadillaazeaza I suggest looking into OpenCV's samples, where you can find some code written in C or Python (e.g. face detection, contour extraction, shape fitting, ...).

It also depends what you want to implement? Object detection, letter recognition, motion detection, ...

[Judegl]

Hello ! I'm trying to make a smile detector using opencv. I want to start with pictures and then real time with video streaming. Can you please recommend me some fast algorithms to use (classifiers etc) and a path i should follow. I'm new in computer vision and i wanted an advice. I "googled" it first but i am asking u because you have practical experience and I think you will be more specific. I would appreciate it. Thanq you a lot ! Sorry if i double posted it.

[GladPL]

@Judegl You need to detect mouth region. You can do it in OpenCV by using cvHaarDetectObjects() which implements Viola-Jones algorithm. For more information check OpenCV samples where you can find face detection demo using still images and camera stream. Then you have to analyze extracted region and e.g. segment lips pixels using color filtering, detect mouth corners positions and make a decision based on collected information.

[LittlePokeDigi]

Hi. I have a Sample of a Face Detection Program .

When it detects a face , it draw a Square box of the face it detects.

This is what I want to do.

When it detects a face , it paste a picture over it.

Do you have any idea how I can do it? Thanks

[GladPL]

@LittlePokeDigi You mean like some overlay? It finds face boundary and replaces this area with other picture?

[LittlePokeDigi]

@GladPL Yes something like overlay.

[GladPL]

@LittlePokeDigi You can access image pixels directly (I'll send you link how you can do this) so you can copy data from one image to another in obvious way. I suppose you can also do something like: set ROI on your input image (where ROI correspond to detected face rectangle), then copy your picture using cvCopy() and reset image ROI. If your picture and detected face rectangle have different dimensions you may need to resize your picture e.g. with cvResize().

[superharryboy]

@GladPL I'm kind of doing this too. I want to overlay images in detected zones. But right know I haven't even managed to detect multiple parts (eyes + face + ... etc) at the same time, like you managed to do in this video. I tried using a captured frame to first detect face, then with this frame with face detected, send it to another function which will detect eyes, and so on. Yep, it doesn't work. Any help?

[GladPL]

@superharryboy Maybe you have to adjust some cvHaarDetectObjects() parameters, like scale, min. neighbours, min/max size? What values did you used? Also try setting some ROI's before eyes/mouth/nose detection, e.g. detect left eye in top-left face quarter, etc.

[superharryboy]

@GladPL I found a way to detect multiple parts in a frame, but it is very slow. Too slow. The output video doesn't even look like a video, looks like a picture slideshow. I haven't tried ROI's yet, neither i know how to use them. Your video seems pretty smooth. Are ROI's important to make it faster?

[GladPL]

@superharryboy Yes, quite important, you can process smaller image region faster than a whole image. I use also parallel detection of multiple face parts, so everything scales very well on 4-6 core machines. You can also take a look on your camera settings - the higher exposure = the slower framerate. Furthermore, standard Haar object detection is not too fast, but quite.

[superharryboy]

@GladPL Up to now I've found a way to overlay images, but needs more work to track recognized/detected areas as rectangles do. I'm using a dual core machine, camera settings checked. Right now i'm have to stick with Haar object detection. So ROIs will be of help. By parallel detection you mean paralell processes? I was wondering if this was possible.

[GladPL]

@superharryboy Right, you can detect face and then find e.g. mouth, nose, eyes in parallel threads because they don't depend on each other. After that you have to synchronize.

[superharryboy]

@LittlePokeDigi There are many ways to do this. What I do is use Get2D() and Set2D() to accomplish this task. I haven't try other methods yet. This one i'm using may not be the fastest method,but it does the job. If you want more info about how to implement this method, google aishack. There is a post about this. The author has the code in c++, so if you use python you'll have to translate the code as I did.

[mhaydar1000]

Hi, really nice work mate, i have been using openCV 2.2 with python 2.7 recently. I have been trying all sort of samples but to no avail. Any codes in python that you may suggest me to proceed with? infact i will focus more on eye detection mainly. I would be very grateful. Thank you.

[GladPL]

@mhaydar1000 I don't use Python, sorry. But I will send you some links I found on pm.

[kiyotewolf]

Hello Mr. Geometry Face

[NickQueenSNSD]

Hello GladPL!

Can I ask you something about the opencv what you had used?

1. What function of opencv did you do for detecting the location of the eyes, mouth, nose,etc.

2. The opencv can help me by return the value of the location of them(eyes, mouth,etc). (maybe, it will return the structure of the face).

3. And that function, can use with the images.

Thanks for your help!

Because, the first, I will work on the Image. I try to get the distance between them.

[GladPL]

@NickQueenSNSD Hi there.

Ad 1. cvHaarDetectObjects() - check out OpenCV face detection example for some sample code.

Ad 2. Yes, it returns only bounding rectangle of the region containing searched object.

Ad 3. If I understood correctly, your question is: does this functionality can be used with both video stream and static images? The answer is yes.

Finding distance between eyes is totally possible using cvHaarDetectObjects(). Again, check example code I mentioned, it will guide you.

[NickQueenSNSD]

@GladPL : what function do you use to get the size( or the wide, the lengh) of eyes nose, mouth.?

I use haarlike and adaboost to detect part of them; but I dont know how to get the true side eyes, nose, mouth.

My purpose is get the true size of them, then calculate the relationship of them.

[GladPL]

@NickQueenSNSD If you want to detect precise positions then you have to use some image processing techniques (e.g. algorithms from OpenCV) to process pixels inside the area of these pre-detected regions. Afaik there's no cascades for features like brow, lid positions, iris center or width of the nose..

[MapkAK]

Hi!

I am looking forward do learn opencv for motion analysis and tracking, and I am interested in learning some facial features recognition as you did.

Could you please suggest material and books you read when learning about CV and IP?

I have already started reading the OpenCV2 book (the first one, the Learning OpenCV seems a little bit deprecated), also the Algorithms for Image Processing and Computer Vision by Parker.

Thank you in advance!

[GladPL]

@MapkAK When I learned OpenCV, I read Learning OpenCV from O'Reilly Media. This may be a bit obsolete, so I'm not sure what is the best book now. Maybe someone else can recommend good lecture?

Beside that, you might want to check out some OpenCV samples, or find decent course with code examples.

I can recommend some classic image processing books, but maybe you don't need to look into each algorithm and find out how it's implemented - many basic and more advanced algorithms are already in OpenCV

[powguma]

Hi I've been looking all over for cascades you used, but I can't find a single one. Can you pinpoint to where I can find them on internet please? You said there were a lot on the internet two years ago, but I can't even find anything similar. Maybe I'm missing something here... But I'm not being lazy, I really tried looking for them for many days :( please help!

[GladPL]

@powguma Dude, it took me 10 seconds to find something that looks interesting.. Try something like "haar cascades" in google..

[powguma]

@GladPL Argh I still can't find like forehead or eyebrows, but thanks for your comments!

[GladPL]

@powguma Hi. I haven't used cascades for brows and forehead. Those files may be hard to obtain. I used more common files like (frontal) face, mouth, eyes, [nose] detectors. I've seen also cascades for ears (haven't used them) and for eye-pair detection.

[powguma]

@GladPL Thank you again for your well-put comments with cares, and for helping out the vision community as a whole. The world needs more people like you!

[GladPL]

@powguma Thanks, it's always nice to help!

[lediond]

nice work.Well done.How did u do to keep tracking of mouth corners?

[GladPL]

@lediond Thanks. I'm not tracking mouth corners. I detect them on each frame.

[lediond]

@GladPL well. i could do that too. can i ask how did u do that? what did u do for detecting the mouth CORNERS in each frame??

[GladPL]

@lediond I explained it earlier. Check previous comments.

[jeetendrag10acc2]

sheldon? is that you?why the goatee?? :-)

[GladPL]

@jeetendrag10acc2 Of course that's me! Why goatee you asking? Oh, it has came into fashion lately, you didn't know that? Haha, Bazinga!! :D

[jeetendrag10acc2]

@GladPL  0:32

if (emotion_detected==surprise)

{

add_current_www_link_to_banned­_porn_website_database(webbrow­ser1.active_window.current_add­ress);

}

:-)

[GladPL]

@jeetendrag10acc2 Really nice example of practical application for this kind of software ;-)

[Nolansykinsley]

nice summary.



[GladPL]

@Nolansykinsley nice comment.

[gad2103]

@GladPL thanks for your response! so i understand that i can use haar classifiers to get a region that contains a facial feature, but i'm still not sure how to get details about that feature. for example in your video the mouth contour is perfect. can you hint at what algorithm (FindContours, Canny, Laplace?) might work that well? did you use native open cv functions or write custom functions? thanks!

[sira1rc]

like this..

I am a beginner learn openCV. Could u send to me this programing?? I will to try make program how to detect object..

[GladPL]

@sira1rc If you are beginner why would you mess with so complicated code ;-) ?

Try to work with some samples. If you want to detect objects check out "facedetect" example from OpenCV.

[gad2103]

wow! this is something I really need to replicate. how did you get the nose width? i am working with static images and i can't seem to get the HaarClassifier for nose to actually get the correct width (it gets an area that is much too big.) can you offer any advice?

[GladPL]

@gad2103 I don't think there's any haar cascade to detect width of nose. This kind of detection is used for object detection, so you can find face/mouth/nose/etc. regions. More sophisticated recognition you have to perform other way. You can find nose region with that Haar method and then detect width of nostrils inside that region using edge analysis (like I did) or something else you consider perform best in this case. Greets.

[gad2103]

@GladPL thanks for responding! i am also curious if you could give me some hints on how you get the mouth contour so well. is there a book that has a good algo for that? or did you use built in Open CV methods for that? please advise!

[gad2103]

@GladPL thanks for responding! i am also curious if you could give me some hints on how you get the mouth contour so well. is there a book that has a good algo for that? or did you use built in Open CV methods for that? please advise!

[gad2103]

@GladPL thanks for your response! so i understand that i can use haar classifiers to get a region that contains a facial feature, but i'm still not sure how to get details about that feature. for example in your video the mouth contour is perfect. can you hint at what algorithm (FindContours, Canny, Laplace?) might work that well? did you use native open cv functions or write custom functions? thanks!

[GladPL]

@gad2103 Algorithm is based on lips color segmentation and it's written using both, native and custom functions.

[kosomonova]

@GladPL How does this work in low-level lighting conditions when the colour of lips might not be apparent?

[GladPL]

@kosomonova Algorithm is based on detecting a color disparity between those two regions (lips and skin). When the lighting is very low it may not work. It's also refer to situation when image is overexposed.

When saturation and/or value is low, color information is hard (or even impossible) to determine.

[Artelis]

Which haar cascade file did you use for the mouth? I have the one with this header:

25x15 Mouth detector computed with 7000 positive samples

It's not working very well, maybe my POI isn't set up quite properly. Here is the POI I'm using:

cvSetImageROI(small_image, cvRect(face->x, face->y + (face->height / 2.0), face->width, face->height/2.0));

basically just setting it to the bottom half of the face.

[GladPL]

@ALL: There is a lot of cascade files over the internet, I used first or second I found (> 2 years ago !!), so for those people who asking for cascades I used - you should find some files, test them all and choose this one which gives you best results, and I'm sure now you can find much more than 2 years ago..

@Artelis: The lower part of the face is a good place to set up ROI for mouth detection. What's the size of an image? You're using standard procedure from "facedetect" sample?

[Chummy27th]

Hello. If you don't mind me asking, how do you detect corners? I'm making a project on beard removal and I was wondering how I can actually detect the corners or boundaries of the bearded part. :)

[GladPL]

@Chummy27th

I search for two extreme (left and right) points of mouth lips pixel distribution.

I suppose it may be some problems with beard, but I haven't tested it much on bearded faces examples.

[TheCrazydionne]

HI, could u tell me pls u use open cv wif which wrapper? opencvdotnet or emgu or aforge? then is it wif C# or C++? 

[GladPL]

@TheCrazydionne I wrote it in C/C++. There's no direct support for C# in OpenCV, but you can use some third party libraries - like wrappers you mentioned. Wrappers could be incomplete and full of bugs, so you have to read some forums and check what's the users opinion about each library. I think better way will be AForge, which isn't an OpenCV wrapper. I use AForge.NET Image Processing Lab and it's pretty cool to take some first look on what you can do with specific image. Greets.

[russellwmy86]

good :d

[Papavon4]

This is great! Have you thought about using overlays to "enhance" faces? You could add a moustach, a red nose, or bunny ears in real-time. This would be a great "fun" app in the Intel AppUp.

[GladPL]

@Papavon4 It sounds like an augmented reality application which wasn't my goal. But I thought that, maybe I'll do some stuff like this in the future ;-) Thanks.

[gmrkhan]

Hi, Like everyone else in the comments first I have to say incredible use of the OpenCV library. I am trying to gt a project underway for smile detection using the same library and was wondering if you had created any sort of documentation or if you can give me any pointers in what you used for the detection of finer features.

[GladPL]

@gmrkhan There's no manual or documentation in English.

Program is based on image analysis using image processing algorithms and OpenCV object detection.

If you want to talk about project you are working on - write me a pm.

[ShwetsD06]

Hi, excellent emotion capturing..I am working on a vry similar area for my PG research proj..Can you pls provide me some guidance in implemnting similar stuff ? or resources / links wer i can learn on facial feature detection jus like ur video? do u hv an opensource code link tht i can study ?

[GladPL]

@ShwetsD06 Thanks. As I said, I don't provide source code for this application. I can give you some guidance, but you have to specify what do you want to know, I can't describe you the whole detection system, because it's quite complex..

[ShwetsD06]

@GladPL ThankU so much for ur reply!! m lookin @ developin a system whr identification of facial emotions is a part of it. M plannin 2 use it fr mobile platforms & m studyin openCV for iPhone.I wntd 2 knw hw 2 Capture emotions like happiness, anger,etc fr which mapping of eyes,smile n face detection wud be involvd.Also m lookin at blink detection.OpenCV seems to be a good technology 2 capture tis..Can u provide me resources in form of books / links ,etc which u may hv used 2 achieve the same?

[GladPL]

@ShwetsD06 I know one book about OpenCV, but the best resource is OpenCV user group on Yahoo, especially if you need do deal with some OpenCV bugs. For emotion analysis I recommend books about Ekman FACS system. If you are really, really interested in this subject check "The Expression of the Emotions in Man and Animals" by Charles Darwin (1872) :D

For image processing there's a lot of publications. I'd prefer "Robot Vision" by Horn & Klaus or "Digital Image Processing" by Gonzalez & Woods.

[turtlesoupandbones]

i dont know how this stuff works but it looks cool

[angelferreiras]

hi, great job.. i am working a similar project. i can detect the face then, the eyes and the mouth with region of inters(rio). i need to dectect emotions like you do. i thought about making my own haarcades(.xml) file for diferent emotions, but i think that the tracking is gonna be slow. how you can get the tow points in the mouth?

[GladPL]

@angelferreiras Hi, you can try, I'm also curious how it will work..

These points are mouth corners - extreme left and right points of mouth contour.

[angelferreiras]

@GladPL Hi, can you give-me a link with contour information. i use histograms for object identifications,but contour i dont know how to use that. i saw some source code in the folder OpenCv\samples. i learn who to identify the extreme, but i dont how to get information from thats area????.. sorry for my english!!!

[GladPL]

@angelferreiras I'm not sure I understood you clearly, but you can check OpenCV manual for functions description if you're not sure about function input/output.

If you won't find it - write me a pm, I'll send you a link (I can't paste it here..)

[GladPL]

@all asking for source code:

I am not interested in making this code accessible to download, sorry..

[anilas12]

Hi, I'm doing a project to identify face land marks. Is it possible thai i get your program to guide me.

[megamanexp]

do you have site you uploaod code on the opencv i need for a project for face detection too filter on the box that do the focus

[pakhtana101]

Great Job.. wonderful work.. can i have code??

[death2ourenemy]

Do you have a place where I could download and use your code? I have OpenCV installed with my Microsoft Visual Studio and I'm wondering if anyone has code that I can exicute as a test.

[roidroid]

you should analyse 2girls1cup reactions with this.

[akshatc22]

hi do u know how to blink detection also???

[GladPL]

@akshatc22: Hi, there's no blink detection in this application.

[eoliene76]

hello ! What happened at 0:08 ? All boxes diseapear ?

Nice work dude !

[GladPL]

Hi. Thanks.

About your question: face detection with Haar classifier failed. Face was to close to the camera. When face is not found no other detection can be executed.

[eoliene76]

@GladPL

i see. thanks for replying.

[cakec9]

Awesome job man. very impressive.

[alpsayin]

great job with opencv, I am also working on a project that requires eyeball recognition, was it easy for you?

[GladPL]

@alpsayin In this case you can use haar object detection with cascade files for eyes. Worked fine in my project ;)

[dsa4fun]

hello! good work!!! :D I am working on a similar project.. What are the algorithms that you have used in this video? I have some problems to find the mouth contour and corners..!!

thank you!

salvo

[GladPL]

@dsa4fun For mouth contour & corners I used some mixed image processing methods based on color segmentation (you can try HSV filtration) to find mouth contour and then edge detection to find mouth corners.

[chamychamy]

Hey! You're great. I'm working for facial feature extraction using opencv. I want to ask if there is already a function in opencv that detect facial features that you used in your software? I am working on this now for our thesis. I hope you can help me. Thank you. :)

[GladPL]

@chamychamy I don't think that there are such functions in OpenCV (< v1.1). Maybe they added something interesting to the newest version.

[catchewer]

This must have taken forever to get all the images required :)

Good job, and especially good effort!

[feedthebear51]

You are like a transformer

[evaoliveira]

Hi! Is there any possibility of getting this software? I have send you an email explainning my propose.

Thanks :)

Eva

[Nuubxp]

How do you get this software?

[GladPL]

I created it myself ;)

[ram666ram]

well good work can you tell me how you find mouth corners and upper and lower parts of the lips? thx

[xwyangjshb2]

good job.

[YouMockMe]

Very nice. Actually I am impressed, I like how you showed the points to reflect these emotions within the encapsulated scanning areas. Of course I am talking out of my azz because I have only just found out about OpenCV yesterday. Regardless, pretty cool man.

[GladPL]

Thanks man, glad to hear that!

[abolfazls2]

hi

that was great can please say me what was your classifier?

[GladPL]

Can you define precisely what do you mean?

As I mentioned I used Haar classifier for face/eyeballs detection. Other features I detected using various combination of image processing methods.

[zenvendofzeno]

Hi there, cracyfacedjenkins, kindly send me the links you send to GladPL? Thanks :)

[GladPL]

Thanks for info & link.

[djneo2000]

great video man!!! its realy cool!!!! i would also know which cascade you use for the feature deteciton

thx in advance

djneo

[cuahuatzin]

which haarcascade files did you use?

[GladPL]

I can't paste url/links here in comments, but if you are interested I can send you pm with it.

[cuahuatzin]

can you please? thank you.

[1DanShow]

Your work looks great. Could I also get a copy? I'm working on an application for Texas A&M's Sketch Recognition Lab and need good face feature detection as a first part of the process. I've requested an evaluation copy of Luxand's SDK, but if you'd like to share what you have, that'd be great.

[GladPL]

Sorry, currently I don't intend to share this code. If you have some other questions - send me pm.

Greetings for all and thanks for comments.

[alanp23]

Very impressive. What method of detection & recognition are you using for this? Did you Haar train each part of your face?

[GladPL]

Hi. I used Haar classifier only for face and eyeballs/iris detection (for eyeballs I have also used template matching which is almost as good as Haar). Detection of other features based on ideas from some articles or was invented by myself (and implemented using various image processing algorithms).

[kerningham2]

Dude, that's awsome