 And good afternoon. So, my topic of MTP is Crowd Sourcing for Educational Evaluation and Large Classes. So, these are the list of content, table of content that I will go through. So, introduction. First of all crowd sourcing, what is crowd sourcing? So, crowd sourcing is an idea of outsourcing a task to a large group of people that is that is traditionally performed by employees of a company. Typically performed by an organization. Crowd sourcing is a distributed problem solving approach in a sense that a larger task is broken up into smaller task. Can I start it again? I did not. Start again. Yeah, stop and start. Yeah, stop. Yeah. It will start right. Yeah. My topic of presentation is crowd sourcing for educational evaluation in large classes. So, these are the list table of contents I will go through. So, first of all what is crowd sourcing? Crowd sourcing is idea of outsourcing a task to a large group of people typically in a form of open call that is traditionally performed by an employees of an organization. It is a distributed problem solving approach in a sense that a large task is broken up into large task is broken up into smaller task such that an individual person can perform work on it. The crowd sourcing workers then perform on this smaller sub-task and the result of the sub-task is combined to get the result of the large task. Crowd sourcing users can be classified into two categories they are requesters and workers. Requesters are those who want their work to be done from the public they can also be called as the employer workers. Workers are those who do complete the work that is that has been posted by the requester. These workers are the most important and the most powerful part of the crowd sourcing classification. So, crowd sourcing can the main class crowd sourcing system can be classified into two parts applications and performance. So, crowd sourcing applications there have been many crowd sourcing applications build developed during the recent year they can be broadly classified into four categories. So, they are voting systems, information sharing system, games and creative system. Voting systems these systems require crowd sourcing workers to select their answer from a list of choices available. Amazon Mechanical Talk is a famous crowd sourcing website and an example of voting systems information sharing systems. Websites help to share information websites help to share information among the people. Some of the crowd sourcing applications have been developed to share information among the crowd. The for example, a system called noise tube has been developed to measure the to measure the daily exposure of noise of the public. The geolocalized sensors in the mobile phone devices helps the users to send data to the websites. The websites then the websites then change sends you can share the data to the general crowd games. Crowd sourcing by taking the advantage of the fact that fact that people desire to entertain a many complex problems can be solved by the help of online game players. The an example of such games that uses the concept of crowd sourcing is ESP games that solve the problem of image tagging in the large scale. These images can be used to these tagged image help to increase the performance of the image circle on the web creative system. The role of the creativity of the human cannot be replaced by any advanced technology. The task like coding and drawing can only be done by humans creatively performance. Since the crowd sourcing since in crowd sourcing work is done from a very diverse and anonymous group of people crowd sourcing is one of the performance is one of the major factor in crowd sourcing application. The performance of the crowd sourcing system can be developed can be discussed in three terms user participation quality management and cheating detection user participation. Since the completion since the completion of the task is highly dependent on the crowd sourcing workers there is one of the important fact important criteria of the in a crowds any crowd sourcing application. Studies have shown that most of the crowd sourcing workers are attracted are motivated by the financial incentives to do a crowd sourced task. However, there have been some system that do not offer monetary rewards to the crowd sourcing worker. YouTube is an example of such crowd sourcing systems quality management. Since a crowd since in crowd sourcing a requester device that breaks down the task into several small part. It is important to decide how to break down such task such that the workers can perform on these tasks efficiently. Studies have shown that increasing the financial incentives helps the to increase the quantity of the work, but not the but not the quality cheating direction. Since crowd sourcing in crowd sourcing workers are anonymous many crowd sourcing workers try to maximize their gain try to maximize their gain by by giving generic answers to the question. Some crowd source crowd based cheating direction approaches can are majority majority decision and control group. In majority decision a task is given to a number of people from the crowd and the result of and the result that has been selected by the majority of the crowd has been selected as the correct answer to the question. In control group group of people is a group of person is responsible for checking the validity of the task. Generally a task is given to a single person and the control group validates the task. If the majority of control group members validates the task as correct the task is considered to be right otherwise a task is again given to some other worker to be completed. The main motivation behind the crowd sourcing workers to work on a certain task have have been are financial incentives, entertainment and we can create a community to bring a large number of crowd. The Amazon Mechanical Turk is an example of financial financial incentive. The ESP games are the example for is an example of crowd sourcing application. Crowd sourcing in the field of educational evaluation. So, why do we need crowd sourcing and educational evaluation? Since interest in online education is increasing by the success of MOOC courses like Coursera and recent Stanford online courses. These course have a lot of these a lot of instruments are attracted to these courses because of their flexible and self-paced method of learning. Moreover these courses are asynchronous. Hence a lot of person can log on at any point of time and complete their work. Because of these because of these advantages more and more participants are attracting towards the MOOC courses. Since these courses have a huge number of crowd the evaluation of the content submitted by the participant as a part of course work is a challenge. Traditional approaches of evaluation in which a single person is a course instructor or single person or a small group of people are responsible for the evaluation of the course content is not suitable in such in MOOC courses. Because these courses have a huge number of population and evaluation by a small number of person is not feasible. So, we can use the concept of crowd sourcing over here. The in we can outsource the task of evaluation to a open and large group of people and the evaluation can be evaluation task can be completed by the help of crowd sourcing. But the major problem with this is the motivational issue. We do not find any motivation we have already discussed the motivation for which the crowd sourcing workers attracts towards the task to attract to work on a certain task. These motivation does not fit on our case of our case of educational evaluation. So, we say we can use the concept of peer evaluation for this. In this we say that our target crowd is the same as the participants whose work is to be evaluated and these participants evaluate the work of each other peer evaluation. Peer evaluation can be defined as it is the evaluation of work by one or more people of similar competence to the creator of the work. Some of the peer evaluation system that has been built in the recent year are delft peer evaluation which has been developed by delft institute of university. The other peer evaluation system that has been developed is SPAR. SPAR stands for self and peer assessment resource kit. However, both these system does not focus have some other criteria has some different motive. These system focus on accessing the individual work in a group project or assignment. The motivation behind these we just have to close it I think. No, I stop now that is why I pause. The motivation behind these system is that many participants in a group tends tends to get higher marks even if they do not deserve the quality of work done by those participants is not sufficient. A study has been conducted in a Queensland institute of technology to develop the peer assessment for large classes. Some of the challenges that were discussed are follows. The first issue that the first major challenge that comes in the peer while doing peer evaluation in large classes is diversity in intelligence. Diversity in intelligence and domain knowledge among the participants creates an unavoidable problem of ensuring fairness in the peer evaluation. Fairness here refers to the ability of peers to their peers to evaluate them in a qualitative manner. The second problem that comes is poor judgment. In the survey many students feel that the peers are not the very good evaluator. Some are very hard markers and the others are very lenient markers. So, the quality of evaluation is not good. The third issue that is there is time consuming. The peer evaluation system should be such that the participants does not have to invest a lot of time while doing the assessment activity. Lack of motivation. The lack of motivation affects the quality of peer evaluation. Many participants feel that assessment should be the response should be the sole responsibility of the course instructor and not theirs. This problem can be resolved by giving a small percentage of the course rate to the peer assessment process itself. Problem score. The main problem that we have discussed in peer evaluation is that diversity in intelligence and domain knowledge among participants creates an unavoidable problem of ensuring fairness in the peer evaluation. Fairness here refers to the ability of participants to evaluate their peers. This problem can be resolved by the problem of ensuring fairness can be resolved by exploiting the knowledge level of the participants. The participants can be grouped in a certain number of classes on the base of this knowledge level and then the task of the task can be allocated to them on the basis of this information. So, the proposed solution to ensure fairness in peer evaluation. So, there is the overall workflow of the system. So, in the MOOC courses, once the participant submits the project and the deadline is completed, an expert who is available checks first whether the performance data for the participant is available, all the participant is available or not. If it is available, then we analyze the performance of the participant and on the base of this analysis, we group them in certain classes based on their knowledge level. If the performance data is not available for all the participants, we take a quiz and the result, we take a quiz from the participant and then analyze the result of that quiz to group the participants in certain classes. On the base of this classification, we allocate the project to participants on the basis for evaluation. Once the evaluation is completed, the final grade can be calculated and published. So, how can we measure the knowledge level of the participants? We can use the performance history of the participants. Since participants attempt quizzes and exams during the courses, the performance history, the result of these exams and quizzes can be used to check the knowledge level of the participants. The assumption here is that, all the performance data of all the participants is available, if this, but this may not be the case in the practical scenario. As some participant may not have given the quiz or exam and hence, there is not a proper solution. So, the next approach we can use is, we can directly take the quiz for the assessment, for checking their knowledge level of the participants. Here, the main thing is, we have to assume that an expert is already available, who can set a quality quiz to test the knowledge. Here, the other problem that comes is delay in the workflow. So, taking the quiz and then, doing the analysis of the result takes time, the overall process of peer evaluation gets a delayed ability. The third problem is that, the performance of the system is, in this case, highly dependent on the quality of the quiz being created. So, we can use the combination of the above two approaches to develop a new approach to ensure fairness in peer evaluation. In this, we assume that an expert is available to ensure that, expert is there, who first checks whether the performance data is available for all the participants in the quiz. If it is available, we use that data to analyze the knowledge level of the participants. But, if the data is not available for all the participants, then we take the quiz from those participants, whose data is not available. Then, we can have the performance detail of all the participants. Based on these details, we can then group the participants in the knowledge level classes. Once the classification is done for all the participants on the base of their knowledge level in classes, then we can allocate project to the participants on the base of this classification. So, classification is done in this manner. Participants are classified in classes C 1, C 2, C n based on their knowledge level, where each C 1, C i represents a certain knowledge level classes. We have to note here is that, knowledge level of class C i is greater than the knowledge level of class C i plus 1. Based on this classification, we can allocate the project for evaluation. The first approach then that we can use is, allocate the projects for evaluation among the participants of the same class. The motivation behind this approach is that, the participants having same knowledge level may have the same understanding and may be able to evaluate the participant. But, the main problem here is, since also they have the same knowledge level, they may not be able to detect the flaw from the flaw, any flaw in the assignments or projects of the participants and tend to give higher marks to their peers. The second approach we can use is, allocate the, allocate project of participants from class C i to the participants of class C i minus 1. That is, allocate the project of a participant to the other participants having higher knowledge level. In this, we know that, since the participant evaluating the project has a higher knowledge level, he will be able to evaluate the assignment or project problem in a fair manner. But, here the main problem is, since the peer evaluation should also enhance the learning of the peers, this approach does not help participants to have any significant learning advantage by the process of evaluation. So, here also we can use the combination of these approaches to evaluate the work of a participant. We say, allocate the project for evaluation of a participant P i to exactly one participant from each class C 1, C 2, C n. The benefits of these approaches, the participant will be able to evaluate fairly and also they will have the learning advantage by this strategy. Once the allocation is done, the participants then evaluate the project of the participants and then the result has to be, once the evaluation is completed, the result is to be calculated. So, to calculate the final evaluation grade, we normalize the result of each participant and take the average of these class. To normalize the grade from each class, we define a normalization factor of a class C i, C j. The normalization factor of class C j is equal to the normalization factor of class C j is the ratio of mass given by all the participants from all the classes by to the ratio of mass, to the ratio of average mass given by the participants of class C j. This has been depicted in this equation. The, once the normalize score of participant P i is calculated, the normalized score P i that which represents the score given by the, allotted by the participant P i is calculated as, score given by the participant P i multiplied by their normalization factor, the normalization factor of their class. Once the normalized score has been calculated, the final score can be, grade can be calculated by just taking the average of this normalized score. Conclusion, we have seen that the main problem in peer evaluation, the requirement of peer evaluation is here, there because in MOOC courses, there are large number of participants and the evaluation by the traditional approaches is not feasible. So, we can take the help of peer evaluation to perform the evaluation in such large classes. The main challenge that comes in such large classes is the diversity of intelligence among the peers as because of this diversity, the fairness cannot be ensured. So, we have studied the, we have seen the, that we can exploit the knowledge level of participants to distribute the task of evaluation to ensure fairness. So, these are the references.