 Hello friends, today we will see how to measure the performance of information retrieval system. So, learning outcome for this session is students will be able to evaluate retrieval performance in terms of recall and precision. Before proceeding for evaluation of this information retrieval system, let us see which parameters are used for performance analysis of algorithm. Yes, space and time are the measures that we use to evaluate the performance of any retrieval system or data retrieval. Less the space and time, better is the algorithm. But for information retrieval system, apart from space and time, other measures are also used. So, why this is necessary because, here in information retrieval system, user gives a vague description and documents are retrieved based on the relevance. So, as we get exact matching in data retrieval, here we are getting relevant documents as an answer set. So, what is the evaluation that we are doing? How precise is the answer set rather than which documents are retrieved? So, evaluation is based on test reference collection and evaluation major. So, what do you mean by this test reference collection? It consists of collection of documents, set of example information request and set of relevant documents for each information request, which is going to be provided by of course experts. So, given a retrieval algorithm or strategy S, evaluation major quantifies a similarity between the set of documents retrieved by S and the set of relevant documents provided by the specialist. So, it is nothing but the estimation of goodness of the retrieval strategy. So, goodness is defined based on how many relevant documents are retrieved. Now, before considering this evaluation, we have to see that how the retrieval task is processed in which manner. So, query can be processed in batch mode, whole interactive session or combination of these two strategies. Now, batch mode means user is giving this query or entering the query and getting the set of documents whereas in whole interactive session, user feeds the information need and interacts with the system as many times till he or she is not satisfied. So, here we need to consider the different factors like GUI, how many times the session is on, how many requests has been fed, how the system is interacting and how the system is giving the results. So, here we are considering only the queries executed in batch mode. So, I is an information request, R is the set of relevant documents for I, mod R of course, the number of documents in R, A is the set of documents in answer set after processing the request I and number of documents in set A will be given by mod A. Now, consider this is going to be the collection of documents that which we have for a given information request, R is the set of relevant documents whereas this green color indicates the set of documents in answer set. So, intersection of this relevant documents and answer set is nothing, but relevant documents in the answer set. So, consider this example, assume that we are having 1000 documents in the collection, now consider any information request for this information request we are having 20 relevant documents, out of this 20 relevant documents, 5 relevant documents that we have obtained in the answer set and answer set contains 10 relevant documents. So, consider this is an example, now we need to calculate these two major as recall and precision. So, what is recall? How many relevant documents has been retrieved out of total relevant document? That is nothing, but mod R A upon R and the other is precision. So, what is precision? How many relevant documents has been retrieved in the answer set? So, precision is equal to mod R A upon mod A. So, at this moment pause a video, consider the previous example and try to calculate recall and precision. So, yes, 5 relevant documents are retrieved out of 20 relevant documents. So, for a given query there are 20 relevant documents out of which only 5 has been retrieved. So, what is recall? 5 upon 20 that is 25 percent, whereas precision, 5 relevant documents are retrieved in the answer set of 10. So, precision is 5 upon 10 that is 50 percent. So, we will consider one example, now when the documents are given as an answer set all the documents will be ranked as per their relevance. So, first document will be the most relevant document and so on. Now, consider for example, we have given a query as a network security and for this particular query we are having this set of documents containing 10 documents which is relevant. So, that is nothing, but R Q and this is the answer set. So, in the answer set one by one we will see that which is the relevant document and accordingly we will calculate recall and precision. So, we have to find the first relevant document now in this example first document itself is relevant. So, here how many documents are there in the answer set right now only one document and how many documents are relevant in that answer set is only also will be one. So, recall will be one relevant document out of total 10 relevant documents which is nothing, but 10 percent. And what will be the precision? One relevant document we have received and that itself is a one relevant document one relevant document we have received in the answer set. So, precision will be obviously 100 percent. So, 100 percent precision at 10 percent recall level. Now, the consider the second relevant document. So, second relevant document is D 56. Now, here up to D 56 we have received three documents in the answer set out of that two are relevant. So, what will be the recall two relevant documents out of 10 relevant documents which is 20 percent and two relevant documents out of three relevant three answer set. So, it is going to be 66 percent. So, 66 percent precision at 20 percent recall level. So, third relevant document is D 9. Up to this we are having six documents in the answer set out of that three are relevant. So, what will be the recall? Recall will be 30 percent and precision will be 3 by 6 that is 50 percent. So, 50 percent precision at 30 percent recall level. Now, pause the video and find out recall and precision for this fourth document. So, find out how many documents are there in answer set till D 25 and of course, this is the fourth relevant document it is going to be 4 upon 10 as in recall level. So, yes. So, total number of documents in answer set are 10. So, recall will be 4 upon 10 that is 40 percent precision will also be 4 upon 10 that is 40 percent. So, 40 percent precision at 40 percent recall level. Same that we can calculate it for the fifth relevant document. So, fifth relevant document is D 3. So, number of documents in answer set are 15. So, we have got 5 upon 15 33 percent recall level at 33 percent precision at 50 percent recall level. We are having 11 standard recall levels 0 percent, 10 percent till 100 percent. So, in this example precision at recall level higher than 50 percent drops to 0 since all the relevant documents are not written in the answer set. So, if we will plot a graph this is the graph for the given example recall precision at 11 standard recall levels. Now how to calculate the performance of an algorithm over all the test queries we have seen for a single query. Now, same algorithm can be run for n number of queries. So, here nq is a number of queries. So, for each query we will calculate precision and recall and then take we will take the average to plot the graph. So, consider this is the first query which we have executed and we have found here precision and precision for the given standard recall levels. Now, run the second query again calculate precision and write down the figures in the same manner the algorithm will be executed for n queries. Finally, the average will be calculated for every precision at recall level and after that the graph will be plot of precision versus recall. Now, it is not necessary that every time we will get standard recall levels. So, if we will not get standard recall levels we have to use interpolation formula to obtain the precision at standard recall levels. So, here the formula is that we need to calculate the maximum precision for rj or for the recall which is in between rj rj plus 1. So, using this formula we can obtain the standard precision for the standard recall levels. In the next lecture we will see how to use this interpolation formula to calculate precision for the standard recall levels, thank you.