 As you know, for the project we are going to do a unique ID kind of project for IIT campus for the student community and we are going to base our unique ID on fingerprints. So, I have brought today a demonstration of a sample application that has been constructed, the like of which every group has to build over the project period. This application is built around a small device, this is the fingerprint capture device called Lumidine, let me show you here. So, this is the device, it looks a very ordinary gadget but it has a fairly powerful multi-spectral technology. In fact, the computing power inside this machine in the form of a computer chip is roughly as much as the power of your PC processor. It's called an embedded system. The multi-spectral technology means that when I put my finger here, the light actually sends the active cells in the hand and therefore even if there is moisture or there is dirt, the fingerprints are captured correctly. The fingerprints are captured as a form of an image and that image can be sent back to the connected computer. Additionally, it also calculates a representation of that image in the form of a template. That template is subsequently used for comparison because image is too big. Using this base technology, the template computation by the way is a proprietary algorithm that algorithm is not available. So, what is available is a programmer interface by which images can be captured and put inside your machine. Those images come as bitmap pictures or bitmap files, BMP files. We shall later on see the BMP format and other things for you to look at things. Now, the demonstration application is about something similar to what we propose to do in the project. First, the administrator logs in into the system and the administrator is also authenticated using fingerprints. So, his or her fingerprints must have been captured earlier so that person is authenticated as an administrator. After that, there is a registration process. So, individual students, actually this sample application is meant for general Indian public. So, information like father's name, address, etc., etc., are corrected. We will of course have to collect different kind of information during registration. But during registration, fingerprints are taken of the participant and those fingerprints are recorded. There is a provision to record all 10 fingerprints. For the demo, we will use just two fingerprints. One after another, people could be registered. That is the registration process. Subsequently, there is an application that application is called attendance application. So, people who have been already registered come with their fingers, keep their fingers there. And if they are registered, then automatically the time of their capture is recorded in a database. Similarly, while going out of the class, you just put your finger on the scanner, the out time is recorded. This could be used for lab attendance, factory attendance or any such purpose. There is also a fingerprint matching algorithm inside the sample application which will be demonstrated by showing that if somebody who is already registered comes again with the same finger but claims to have a different identity, different name. That is same person trying to register as two different individuals. How the system says the fingerprint is already registered cannot record any other. So, quickly have this demo and then go ahead with the additional details. There is a large team of people who has worked even in IIT, apart from the supplier of this equipment. As I mentioned, a small company in Hyderabad which has developed the backend software. And the device itself comes from an American company. So, my friend Mansoor here will demonstrate this. He is the administrator. So, why don't you log in Mansoor? So, when he logs in, he has to use his index finger because that is what is registered. If any one of us would have tried to register as the administrator, it would have failed. Now, if we start the registration process, what we call enrollment. Can I request, just send you, can I request two of the students to come over here please? We will just register them. Criminals should be fingerprinted, right? So, any two criminals. Come on, come on. I am just joking. No criminal here. Your first collect. So, first the information will be put in. Give some. What is your name? Gaurav Tripati. Gaurav G-A-U-R-A-V. Blank Tripati. Okay Gaurav, fine. Just say F student. Yes. IIT Bombay. So, now he is scanning Gaurav's fingerprints. Yes, Gaurav, go ahead. Just one second. Right thumb is fine. One second. Right thumb. No, no, no. Don't describe. Let him choose. Right thumb. What's your birthday? 28. So, let's not change it. So, left and thumb. One second. Take it out when the light goes off. That's okay. Just register with one. So, I will now put another person. What's your name? What's your rate of birth? Sorry? Put your left thumb. After the light lights. One second. You can take out. We will just demonstrate if what happens if the same person tries to register again. Under a different name. This is actually the matching part. Okay. This is not really matching part. This is verification part. There are three aspects. One is the capture and registration. The other is verification that the new person who is registered does not already exist in my database. And the third one of course is usage. So, what we'll do is we'll just put new and why don't you come back again? Let's give him a different name. Suppose you try to register as a different person. Sorry? JR. Okay. So, he is trying to become somebody called JR. Well, for the purposes of this demonstration we'll assume that JR was born on 11 2009. So, I don't have to type in the date of birth. Now, I will ask him to scan the same finger. Left hand finger. Yes. He has seemingly registered. He has given his thumb impression. I go back. And now when I say register so when we fill in the details for registering it has remembered the earlier fingerprint and it says that the user already exists with this fingerprint. So, you cannot actually register again with the fingerprint. Now, you will appreciate why for this registration process all 10 fingerprints must be captured because otherwise Achyothan could have tried to register with his index finger or right hand thumb etc. Anyway, let's look at the attendance application now. I will exit from this registration application and we'll see the demonstration of the attendance application. So, here when I start the attendance process the device goes into the on mode. It remains perpetually on. Anybody can come in and do that. So, when I say start so why don't you come and try to attend the class. Yes. Some imposter comes in. Let's say there is an exam. He claims to be some rule number and tries to attend. So, this is the fingerprint but the attendance is not marked because the person does not exist in the database. Whereas, if a legitimate student come Achyothan, why don't you attend the class? Same thumbprint. So, Achyothan has come in. Let me also see if I can attend. I am not cheating. I had registered myself earlier. So, the system knows me. Now, let us say the exam is over and Achyothan is going back. So, if he again scans his fingerprint he is walking my dear friend. So, the Achyothan timeout is also shown. So, he goes out at this point. And after the exam is over I suppose I can also walk off. Now, this is the simple application. Thank you, Mansoor. So, you can see what we have to do in this project. We have to build an application to register people. That means you have to create an interface by which we will capture the information about every student. You have to decide what information you would like to capture. You have to capture all 10 fingerprints and you have to store them in some place which you send to the second project which is the consolidation and what should I say, database creation process. That consolidation process will collect all the data of all the students. There are 800 students but don't forget that ultimately this whole thing can be scaled up by the entire Indian population. That is what is the intention, in fact. So, you have to design your entire data structures, database, files, etc., etc. such that larger and larger amount of data can be stored. But it is not just storing as is because images, as we shall see, are not matched directly by comparing one image to the other. That's an extremely hard problem for computers. Human beings do that very easily. But it is understood that human beings also convert an image into some comparable parameters which are yet unknown. So, the second part of the project of consolidation is of computing some template from the image and storing those templates at the back end. That is the consolidation phase. The third phase is verification phase. You remember when Achyuthan came again and tried to register as someone else but his fingerprint was already there. Now, how was that fingerprint compared? That fingerprint was not compared directly with an image. As I said, image comparison is a tough problem. His fingerprint's template was calculated and that template was matched against all available templates in the database. And if the template existed, the system shouted saying already registered. So, that is the third project. Now, the third project when it is completed will be merged with the first project because this has to happen during registration. That is the third project. But the process of registration is separate because it is a fairly complex process of handling data, handling images, putting that in a database, etc. And finally, there is an application. Now, this application, demonstration application was an attendance application. But the group doing the application part could conceive of any other application. It could be an examination attendance. It could be online examination. A quiz, for example, online quiz. Or it could be any other application that is the purpose of this project. To help you perceive a larger application broken into four pieces. And then finally, so we just went through the demo of the new gadgets. Only one gadget was shown. There is another gadget which is ready. Actually, I wanted to combine this gadget with the fingerprint scanner for the purposes of this project. But this gadget came in too late. Remember, I told you about the clicker that we are designing. This is the IIT Bombay Design Clicker. About 1000 units will be coming shortly. In fact, you will all be using these for answering quizzes. So, there will be a sort of one page manual of usage and so on. And most probably either immediately after Diwali or a week after that, we shall use this at least for two weeks to conduct online quizzes. Here, there are answer options A, B, C, D, E, F. There are some reset buttons and some simple LEDs. And we have, we will demonstrate this separately to you. The idea was that if I can somehow imagine combining this with the fingerprint. For example, imagine that if this clicker itself had a scanner and I have to put my finger here. The point is that clicker has a side of the chip and that unique ID will be recorded against a roll number when a clicker is issued to you. But suppose I have a roll number, I record my roll number and I get this clicker ID. Now, I know that I am not good at answering questions. So what I will do is my neighbor who is the top or I will say, Arayalajka Kvisthu Banaade but press this answer here. Now he will, so how do you make sure that the roll number is the same person who claims to have that roll? So if we had this sensor here I could press my finger here and only as long as the finger is pressed and I press an answer the answer will be recorded otherwise not. Now here the fingerprint matching has to be done with the clicker. So inside the clicker I will store the template of that roll numbers person and put here the recording will be done. Of course somebody pointed out my colleague that I can easily have my thumb here and my neighbor's fingers pressing A or B. That is rather difficult to guess. That is why we depend upon the good behavior of everybody in life. But anyway this is just a small recap. In many national UID project 10 groups will do the same project independently. The groups will compete with each other so there could be relative grading. So group 0, group 1, group 2, group 3 of course relative grading does not mean that the best performer necessarily gets all the marks and worst performance necessarily gets 0. Relative grading is only to sort of qualify that this group together all the four projects all the four lab groups within that group have done slightly better than others. I would not allocate too many marks for this relative grading anyway but this group will be divided into four sub projects to be done by A, B, C and D independently. So each sub project has to be done independently but there has to be sort of interaction between these sub projects in terms of standardization of common interfaces and so on. And these are the four processes that I meant. Registration, classification classification is where you convert an image into some kind of a template. Then duplicate detection that is matching of templates. Seemingly it might appear that classification and duplicate detection are one and the same thing they are not. They are in fact significantly different processes and last but not the least is usage any usage you can imagine. So all four must be covered by a group so while A, B, C and D can choose any one of them but you cannot have B and D or both doing classification or A and C both doing registration. So within A, B, C and D the lab batch coordinators of these four batches must come together very quickly and decide who will do what portion. I expect about 20 hours to be spent 20 hours of working time by every student on this project that teamwork allocation etc. will be handled by the team leaders but everyone must always record whatever time that is spent. We shall discuss this aspect later in the next lecture when I will give you a sample time sheet. We will now switch over to some issues in fingerprint matching. These slides were prepared courtesy by a friend from industry Aniket, I must tell you some story about Aniket. They used to be a small company more than a decade ago in the corporation when the then chairman thought at that time LIC branches used to operate as independent offices. So if you had a policy in some office in Borivali you had to pay your premium in Borivali only whereas most people for their office would go to South Bombay somewhere in Fort Nariman Point whatever that would be very inconvenient for them to pay their premium in Borivali. So we the chairman of LIC wanted a premium by which even if someone goes into Nariman Point branch and says I have this policy number automatically a message should go to the Borivali branch and the computer programs running in Borivali branch should automatically be able to record that such and such amount of premium has been paid in Nariman Point branch. Now this required extensive message passing considering there are lakhs and lakhs of policy holder common database of policy numbers and people and which branch they have policy in. However for every such transaction of branch transaction the centralized server has to bear the brunt of all the activities of transactions which was not feasible. So we wanted a simple messaging system at that time the persistent messaging was just emerging and it was Aniket and his team had an extremely low cost implementation which LIC used across the country for several decades still very recently when they bought a commercial product because they are integrating not just premium payment and collection but n number of other things. So an extremely competent person he had applied his mind to fingerprint analysis then when I recently told him about this project he volunteered to put together some slides which I will use. Unfortunately it is not yet clear to me whether these slides are part of open source. So until I get permission from him I will not be able to put them up on to the course website but hopefully he will say yes I will also be requesting him to share with those team leaders of those teams which are going to do consolidation and matching to come and give some hints about but today I will just describe the basics of fingerprint matching techniques. So as I told you fingerprint enrollment you just saw that captures the raw image from that raw image you actually convert the raw image into some kind of a template the template is based on minutia extraction what is a minutia? minutia is a special feature that you see we shall see some minutia basically there are singularities there are points inside your fingerprint which can be identified as special points their location their relative position to each other these are important because these are what give these are the points which give uniqueness to every fingerprint. So once you do that you put that into a fingerprint template and you have carried out this exercise for some 500 of these 800 students so you have 500 templates now the fingerprint verification process is any time a new fingerprint enrollment comes the same minutia extraction is done however there is a minutia matching that is to be done so the verification process involves matching and that is completely different from simply calculating the template by calculating the minutia so we shall see both the minutia what we are looking at today is the minutia matching problem in the process we shall also see how minutia are extracted this gives a matching score remember the matching score can never be exact 100% in fact to match two images exactly as 100% pixel by pixel is nearly impossible because there are infinite ways suppose I have a thumb impression I just move the thumb slightly lower the image will be different in terms of XY coordinates of pixel value if I twist it slightly if I turn it even by 2 degrees the image will be different so an exact match is not what you look for in an image there are two major fingerprint approaches matching approaches one is called the minutia base and other is called the image based correlation there is a mistake here it should be correlation based not correction based of course everything should be corrected so the correlation based directly deals with image this is a simple approach most probably this is what most of you would adopt but if you want to calculate some minutia you can do that but this has low accuracy and it is very slow whereas this is very high accuracy fast speed but requires high quality fingerprint image in general what is done is when you get an image you do an image in enhancement first this enhancement you do orientation computation you do image segmentation finally you detect singularities and you classify fingerprints after all this you extract the minutia and then you do the minutia matching against the minutia templates that you have this is what we named the matching score let us first look at image enhancement this is something that your project will not have to do incidentally but it will have to do something called thinning and edge detection so let us look at image normalization you remember you may not understand the terms like gaver filter or gray level normalization but I hope you remember histogram equalization histogram equalization is a very simple technique which can effectively remove the blurs from an image and sharpen it so this is a gaver filter this is a normalized image this is a gaver filter enhancement there could be a local orientation of the ridges which can reflect the ridge flow direction so what is a ridge all the lines that you see there called ridges compared with mountain ranges so your valleys and ridges so the white portion is a valley black portion is a ridge now sometimes the ridge suddenly terminates here sometimes a ridge may branch off into two points those are the singularity points the flow of these ridges around your fingerprint is what is captured here you can see a slightly specialized image that you see which gives you the ridge flow direction how do you compute the orientation so here you can see these are ridges the black ridges that you see take any one ridge now along this ridge there is a a tangent sort of thing and anything which is perpendicular to tangent this is the pixel gradient direction and this is the local ridge direction now you can do a discrete direction estimation or you can have what you call dominant local orientation computation there are techniques called fingerprint segmentation for classification this is a tough problem there is a category of machine learning where you classify objects and try to understand what each classification means there are many experts of machine learning in the institute for example or who applies machine learning to processing of natural language things he does not do any image processing but classification of words and their meaning and automatic translation is equally a hard problem similarly by the way for image and signal processing we have some stalwart researchers here in the doubly department Professor Vikram Gadre and an earlier colleague Professor Ubi Desai unfortunately he has left us he is now the director of IIT Hyderabad but these people are well versed with the notion of handling images and such things of course we are facing the fingerprint problem so this is one area we will skip further time being but here we just show how singularities can be detected I told you the ridges this is the normal flow of the lines on your fingerprint so this is called a singularity ordinary point it is just the hollow point here notice that here also there is a hollow point but around this point we are going this way so this is called a delta point here is another set of ridges but this ridge sort of occurs around and therefore this is called a core point you will notice that core point delta point ordinary point are all different there is a double core point when you have ridges of this guy now these points if you can isolate and if you can maintain the relative distances between these points and the type of these points 999999% you can simply say this is this person or not this person and that is the crux of the template computation there are classification of fingerprint images using the singularity information for what is known as course matching so you can say how many singularities are there then if the singularities are not greater than 2 okay there is one delta point then you can say it's tinted R otherwise you can say it is right group otherwise you can say left group otherwise you can say whole otherwise you can say arch these are the basic nomenclature there is enough literature in abundance on fingerprints and these types even from Wikipedia you can get these details easily the question is not about knowing these details and terminology and that is what consensus most how can I write programs that given a digital image I should be able to do all of these things so extracting minutia from input images is what is most important if we remember that minutia are nothing but rich endings or rich branches then there are two available techniques one is called binarization and the other is detection from direct grayscale images binarization is a traditional approach you first convert the grayscale image to binary image how do you convert the grayscale image into binary image binary image will have only 0 or 1 for a pixel you can simply do the following you can decide on a threshold any pixel value above that threshold it is 1 any pixel value less than that threshold is 0 then you can work only on binary images these will include many pixel operations such as grayscale to binary conversion rich thinning operation it has high complexity all of m cube however your computers are fast enough so all of m cube you will not notice here is a grayscale to binary conversion this is a 8 bit pixel you convert it into a 1 bit pixel observe that even when you convert a 8 bit grayscale image into a 1 bit pixel the ridges that you see are fairly thick ridges and that is because at this resolution they are thick it is difficult to work with these thick ridges so what you do is you take this entire thick ridge and convert it into a thin ridge that means if there are lots of neighboring ones and they are all moving in one direction you sort of sense the direction and replace all that bunch by a single one kind of thing so reduce the ridge to 1 pixel wide structure this is many pixel wide you reduce it here this is non-trivial by the way because you can't arbitrarily remove some of the ones you have to remove the ones such that the basic ridge structure is kept intact so there are techniques for this programming techniques for this some of which you can discover by common sense by looking at it at the end of it most computations involve some understanding of geometry and some trigonometric computation so there is nothing triple integral or any such thing involved here but much of it is sort of creative thinking and applying it properly to a complex image these are the possible problems which we will not be considering in this project because we shall imagine that the fingerprints captured are good quality fingerprints so if you have a wet finger you may get this as an original image and after thinning you may get this if you have a dry finger you will get something of this sort if you have a wrinkle you will get something of this sort in fact capturing and matching fingerprints of old people and infants is considered very tricky and difficult because the actual fingerprints or the fingers themselves have sort of blurred prints how do you detect minutiae suppose you have thinned out the ridges so that means you have one pixel long range in a picture you may have this pixel here this pixel here this pixel here this pixel here like that if this chain ends suddenly then it is called a reshending you are located a point notice that every other pixel will have two neighbors one here one here if you take this pixel for example it has also two neighbors one here one here this is coming where in a matrix of zeros and ones for every pixel you determine whether there are at least two adjacent ones or there is only one adjacent and if there is only one adjacent one then that becomes a region on the other hand you might have a ridge branch where a particular point has three pixels neighboring pixels which are one so fairly simple concepts after you have reduced and from binary to a thin age thing you have to remove false minutia so structural or knowledge based techniques are applied to remove the false minutia I will not spend time in discussing this you can do direct grayscale approach then you don't do any binarization so if you have a h into w image height into wealth image if gray ij is the gray level the ridge line is set of points which are local maxima along one direction you notice that the ridges will be like this this is ridge this is valley this is ridge this is valley so compare it to the mathematical function this is the maxima of the function and this is the minima there are multiple maxima so you can calculate local maxima you get points on the ridge and you draw a line along the local maxima you will get a ridge point there could be a ridge line tracing you take a starting point move n pixel forward along the ridge direction find the local maxima and stop tracing after a stop criteria after so this cross section of the ridge you can sort of locate how do you terminate where stopping criteria for ridge time tracing could be any one of these so you are moving like this there is a termination you are moving like this there is a bifurcation you are moving like this you get a false minutia you are moving like this then you get a this is excessive bending minutia so you may not detect it if you are going forward these are actually if you study some of these things any few finger prints that you take it will make several images available to you you can actually discover interesting possibilities yourself the minutia matching is to compute the similarity between two input templates of their minutia coordinates the minutia coordinates are a set of r and theta and some orientation the only point I will cover in this is to show you how difficult it would be by showing you what happens when for example your registration process or referencing process itself is not good enough these are some reference point detection techniques take this to find the accurate core point in this case but consider this is most interesting it is the same finger this is the master somebody has registered in master data store now this is the same finger exactly same finger but observe it has just gone down a little bit so you have actually the core here you have the core here and you can see that the entire thing is same but because it has come down if you just say XY coordinates no pixel will match correctly that is the reason this is what I wanted to emphasize that there is no possibility of direct image matching there is no such notion image could be twisted turn right this way that way and that is why you have to calculate some kind of template and then calculate the remaining things so there are techniques here that graph based techniques anybody who is interested in graph theory can actually work out some very very interesting application of graph theory onto template formation and template matching there are a large number of error cases that Aniket has described I will take his permission to put these slides onto the course homepage and those of you are interested in detail studies can look after this look these up the point is that although this is a programming course we are going to apply programming on one side to collect textual data about students on the other side to calculate statistics based on the application for example how many marks one scored how many classes one attended classification of images how many different kinds of templates that you can think of how many different kinds of matching techniques you can think of throughout the project is likely to be an exciting project although I do not expect you to do extremely complex programming but permit me to say that most of the programming involved even in this so called complex image processing will be limited to handling arrays of integers applying some transforms which are nothing but convolution of a large matrix with a small matrix multiplication problem and of course some coordinate geometry and some trigonometry calculating sin theta, cos theta things which have been doing in your school all your life so nothing more complicated is required in terms of mathematical background or mathematical activity but the job is interesting and you can make that system work it will be fun we will stop here