 Խoor ḍ brow PCB171 ԱԾԱԸ, ԱԱԱ broke..? Խnesota�Խ ԾԱԱԽ .. ԱԼ [( Աbart�ԱԱԼ compar ԱԧԱ Edison holes... ԱսԱԽ ԱԾ Cinderella ԳԽ ԱԱ extracting� Աԟ ԨԼ ԟ fluids slip ption ativity that all latex related are the effects you keep under one directory all latex related artifacts you keep under one directory nd then the other two files which are separate files nd then zip all of them together that should be fine nd so can you give you any instruction the more important part of this exercise was actually peer assessment which we have not yet touched upon. You are all submitting your literature survey. That's of course a critical aspect of this communication class that you should be able to improve your written communication to an extent that it stands on its own legs. So that much would be achieved partially by now and maybe at the end of next week when you submit that. Should we say that one final chance for everyone to submit three files the red line is next Monday, right? Fourth. All right. I'll extend that date in case you are improving something. I have unfortunately no credits or grades otherwise those people who submit things in time could have been awarded a higher grade. Unfortunately there is no grade for this course. Either pass or you fail. And very limited flexibility to use the standard stick and carrot approach. There is only one stick and no carrot. Not very useful mechanism. Anyway, so how many of you have participated in any kind of automated group communication? This is something that I wish to discuss today. As an example, we would take the case of automated peer review mechanisms. So how many of you have done any kind of forced peer review? There is a part of any larger exercise. You are required to assess the quality of communication as submitted in the form of digital artifacts by several people and you get randomly assigned to assess some three, four, five submissions out of those. One, two, very few, three. Why, others did not have an occasion? You did participate? So can you explain what was the task at hand and how did you go about it as an individual? Sir, I am Amami. There I was as a faculty in the military college of telecommunication engineering at MAU. So we have a journal called Technical Journal for Four of Signals. So there are officers submit articles and journal papers to be published in that. So I was given task to review the submissions done by the officers. I used to assess whatever was submitted in terms of the originality of work, the importance related to today's current trends and the originality of work in terms of own written or is it copied from somewhere. Those things were checked and then recommendations were given for publishing or not. But that task you did as part of your instructor's duties anyway. Anybody else who has participated in a peer review? So there was a course on Coursera for coding in Python. So we used to get multiple submissions from other course-takers and we needed to make a game. So there were few questions and we needed to play that game and mark whether the question was handled correctly. So can you take an example and suggest how different people were awarded different marks by you? How different people whose work you assessed were awarded marks? Were they different from each other or was it a zero one kind of thing? Mostly most of the submissions were correct. Sometimes something went wrong. How many did you assess in this fashion, the numbers? Per submission, per assignment, 5 to 6. So apart from making your own submission, you were required to assess 5 or 6 other submissions. How much time did you have to spend on assessing those 5 or 6 the total time that you spent on making your own submission? Making the submission about 3 hours and assessing about 30 minutes. 30 minutes for all 6? Yes. So assessing was not very difficult? Yes. But did you, because these were programming assignments, did you execute that program which was submitted or you just went through it manually? Executed. We did not check the code. So here is an example of an assignment which is not very difficult to test. You can simply compile and execute a program. If it works, it works. If it doesn't work, it doesn't work. But if you were to assess an essay type answer, like our friend the instructor in Mahu had to do where an assignment was submitted in the form of a paper, not really an assignment but a research paper. Now you know how difficult it is to read a research paper. We have gone through that. So if you have to read every research paper carefully and then to assess whether that paper starts putting or not and it would invariably not be a zero on assessment, right? You would give some A, B, C grades or some such thing. That would be harder, right? Yes. So approximately how much will it take for a person to assess? Let us say a four-page, a four-size document which is submitted by someone. About two hours. Two hours. And how much will it take for you to write a four-page assignment? Well, if you take everything from conceptualization, it may take six to eight hours anyway, right? I mean jotting down your ideas, organizing them, rewriting them, then checking correctness, etc. So roughly the ratio is something like one is to four. That means if you spend four units of time in preparing your own submission, then you would require one unit of time typically to assess somebody else's. So three to five assessments to be done by each individual would amount to roughly the same amount of time that you would spend in your own assessment, in your own assignment. Is that a good thumb rule? Can we generalize it? It will of course vary from, for example, programming as we just discussed, just a question of routinely compiling and executing it. And that you can do six or seven assignment checks in less than 30 minutes in fact. Whereas a written essay, in the domain in which of course you have some understanding, because you have also submitted an assignment in a similar topic or same topic. So roughly the ratio can be regarded as, so creation of one submission can be considered approximately equivalent of, you agree that this could be a useful thumb rule? So which means that if you are participating in a course, such as on Coursera or what we do in IIT Bombay X, for example online courses, it would be reasonable if I form on the people at the beginning that your assignments will be of two types. One is you will have to submit your own assignment and the second is you will be required to randomly assess so many other assignments. Then that would be a fair game. All right. Now let us consider some squiggles in this. So we agree that this could be the mechanism. The reason I want you to think beyond just these statements is because you are all CS students and you would be expected to actually design and build systems which will permit this to happen. So in Coursera for example, what kind of system did you use? In Coursera in peer assessment, what kind of automated system did you use? It is obvious that there is no human being who can intervene to look at either your submission or your assessment. So it must be an automated system. So after the submission date, did you get a list of three or four or five submissions to be assessed by you? No. How did you come to know which submissions you have to assess automatically? What was the process of automation? Can you just explain that? I used to get, I mean, one by one I used to get that you need to check this submission. You would get one by one. Yes, I did not know which users submitted. So firstly it is an anonymous review so I don't know whose submission it is. It would be obviously a machine readable thing so I can't even figure out from handwriting or something like that. So it is anonymous truth. Secondly, he says he would get one assessment at a time. How much time did you get to complete that assessment? There was no time limit. There was no time limit. So suppose a lethargic person like me gets an assessment and does not submit for one week. I mean there was that one week deadline. So you see you need to be very exact. So there is a one week deadline. Now suppose I take maximum of one week and submit my assessment at the end of one week. Only then I will get the second assessment. No, no. In that one week you need to complete all the five. So let us be more precise in that automated system when you would get up to four or five such assignments submitted by others to be assessed. What is the maximum time in which you were given all four or five independent of how you completed or not completed? Three to four days. Three to four days. But they would arrive one at a time. Sometimes two or three. Collectively five were given. Collectively five were given. No, you said you will get one. One at a time means you can check by one at a time. You are talking about one unit of assessment to be done for five submissions. But you will get all five together. Yes. In what form did you get the message? Did you get an email? It was on the website. No, no. Website is all right. Website is a web page. So website by the way is if it just comes on website then I am not reminded whether something has come on my website or not my dashboard or not. That means do I have to constantly look for such thing or do I get a mail? Do I get SMS? Do I get something? I don't remember. Don't remember. But you managed to do that. Yes. All right. So if at all we need to automate such assessments what are the aspects that we need to keep in mind in order to build such automated system which he has let's say partly described. So let us assume that for an assignment or for something some digital artifacts of a similar kind have been submitted by thousands of people and we wish to carry out a peer assessment of the kind which our friend described and we want to build a system for which we wish to write the overall functional specs what should be the functionality of such a system. So as he says after every assignment he would get four or five assignments on his website but he does not remember a communication independently as an email or SMS or he was required to visit the website and keep looking at such assignments that is assessment assignments given to him. So what would you prefer a reminder mail or something but much before that how do you choose whom to give a particular digital artifact or assessment. There are ten thousand people who have submitted we have already agreed that every participant has agreed upfront to assess three to five other assignments other other submissions that we take for granted. Now how do you choose which three or five assignments will go to him which three or five will go to him which three or five will go to him yeah random is that correct? acceptable? random choice of peers now how do I make a choice if I have thousands of artifacts what could be the mechanism all that I have is ten thousand submissions again coded submissions so the file names don't reveal anything and given by ten thousand users I have a table associating each user with the assignment submitted by that user that information I have now every submission I have to select up to whatever three to five peers to assess that so how do I randomly select that everybody is familiar with random number generation but a random number generation of this kind where you are picking up one assignment and assigning it to three to five users you have to flag those three to five users to whom we have already given an assignment okay but each of them is supposed to get up to five assignments that is how the system works is it very clear in your mind as to how that algorithm will work because towards the end there will be absolutely no randomness left there will be a few people left to whom few assignments have to be assigned there hardly any randomness there so how do you go about it we could actually let's say if we want to assign three papers to each person we could have three different permutations of the whole set such that you do not get your own paper so such three different permutations I mean will obviously land you with three different papers at random so basically instead of allocating a person like you allocate a seat in gate or J.E. score dependent fashion he suggest that we could use a set of permutations assuming that I have to assign everything to three people or four people or whatever based on that parameter I could create permutations of all the assignments and all the users and create so let's say a possibility of using permutations good choice I believe but that is an algorithmic part we are not discussing that as computers and professionals you will be good in designing an appropriate algorithm just wanted to point out that it's not a trivial thing to do so all right now having done that our friend submits his assessment for these four things everyone does that but some people are lethargic like me they don't submit in time so how do we handle assessments which are submitted now by people so there is a deadline as he said within a week you are supposed to do that first of all how do you incentivise people like me to submit in time any suggestions in the armed forces there is no question of taking any liberties with deadlines right you better do that it's a survival instinct whereas ordinary mortars like us are indicative so how do we handle let us put it this way that we would like a decent easy way for people to work yet we would like them to work with the armed forces kind of discipline as far as deadlines is concerned now how do you enforce it gamification can you explain basically whatever the task that you have been assigned attach some incentives then later on based on those incentives they will get benefitted in the future so basically the word gamification is used primarily in the context of giving some kind of an incentive so gamification is not in terms of playing games but in terms of creating an incentive mechanism as an incentive mechanism anything else because in real human life carrot alone does not always work a stick is also useful our friend uses sticks very effectively there is no carrot I think in your case but we so what stick can we have very obvious I have submitted an assignment I expect it to be assessed by others just as I have required to assess other submissions now I do not complete the assessment of four submissions as is required let's say so what the system can do is dear Mr. Fartite you have submitted your assignment I am glad to inform you that that assignment was actually rated well by your peers however since you have not submitted the assessment of four other things which were sent to you we are unhappy to announce that you get a fail grade in this assignment pass fail always works or grades or marks will always work so this is an incentive the other incentive is I think we should not really make it so strict as to fail somebody we can only hold back your results till the time you do not submit it so our friend here is a light-hearted person like me I wish Professor Sarada was here to argue his case but I will argue as if I am Sarada he is suggesting that Sarada is the name of my colleague instantly many of you may know him he teaches geography in Thames and my guru in databases basically now what he is saying is that we need not be so strict alright so let me ask you a counter question suppose there is no peer assessment I am required to submit an assignment in time two possibilities I do not submit in time and two I do not submit at all or three I do a very bad job what will be the end result if it is submitted for an examination will I pass that exam hence on the kind of work that you have done oh so that means I keep getting an assignment I keep not submitting it throughout the year and I can still pass that's why I said it depends on the kind of work yeah so you are very clear that if I do not submit an assignment which is declared to be a required part of my total assessment then I deserve to fail here is the point I am pegging an assignment and assessment of four assignments at the same level that is part of your task declared upfront you don't like it don't register for the course no but I still think that giving a let's say if I submitted my assignment now after having submitted the assignment the mere curiosity of wanting to know my performance in that assignment would drive me to quickly submit the assessment so that I can correct I think that is a better a positive incentive than to really have a stake and say you will fail in this assignment if you do not submit the assignment that is because you are treating these two as independent components which have nothing to do with each other no I am saying so wording is very important how you word it upfront in your entire offering of the activity I think that is a good point he makes because ordinarily if I am a student let's say or a learner who is doing a course in course or IIT Bombay or wherever like I have gone through a conventional exercise of education and therefore I will presume that the only reason for me to do this courses and to get a grade is to give my assignments in time so as long as I do that I presume that I deserve to pass I need the course people need to specifically state this upfront that your passing depends upon you are completing the following tasks successfully one giving your quizzes and assignments in time two doing the peer assessment in time then what about the quality of the peer assessment that's a good point so that was the third point I was about to mention but in terms of incentivizing and disincentivizing I suppose if a proper language is used that should be adequate so an important point is declare the criteria upfront you can't change the goalpost after the game begins you declare it fine now having done that he raised the question this fail pass approach will work in courses but what about the conferences and journal peer review will it work there I don't think this fail grade etc is possible there is that correct what about the non-course act very good points are being made in fact so let me tell you how it is handled it is handled through a different incentivization scheme for the reviews and other things that we talk about where the recognition at the end of the long path is the only incentive so you remember when last time we did an exercise in finding out what are the different entities that will require what are the different entity types etc and one of the entity types which was mentioned was reviewer now who is a good reviewer this also embodies the question that our friend raised what about the quality so I might do I might assess five peers but I will do a shortage of it because I want to complete it quickly so either I will give all of them a grade or all of them D grades or whatever I need by the way an algorithm method to catch such a shortage of which I cannot do on individual basis but collectively when I do a statistical analysis I should be able to do and that is an important part of any automated system analysis of the reviews that are submitted but the point that you are making is actually handled like this in such systems in fact that is why I think I referred to Professor Mausam's work on crowd sourcing should read some of his papers he has built a system in IIT Delhi which Ganesh and I are trying to get implemented here so one of the aspects is that you cannot depend on the crowd that is the common wisdom what if the crowd does not participate or does not do a quality job the fact of life is that you can build enough incentives something like gamification something like gamification but here again you state upfront that such reviewers which come up to be consistently doing quality job and in time shall be recognized shall be recognized a very vague statement so to attach something concrete to that way shall give you one example of what I have thought of doing for the last scale teacher training program that we are going to run using this model the faculty development program that we have designed this April, May, June they will roll out I think I mentioned there is one faculty development program in education effective use of ICD and the second faculty development program is on online and blended or hybrid education where people learners learn both from online now as a part of the assignment roughly 10,000 teachers are expected to participate in each of the and the assignment that they have to submit are in the form of OERs open educational resources so lets say I am a teacher of thermodynamics in some college somebody else is a teacher of electrical engineering etc etc I will form groups through a mechanism that we will create which is a manual mechanism because teachers will be assembling at remote centers which are identified so there will be physical groups of 40 to 50 teachers assembling at a remote center across 300 places in the country that is something which is not possible on a complete online mechanism but we will have these 40 people assembling at a place and they will be forming teams and there will be team submissions which will be made now the incentivization that we are talking about is first of course contrary to what our friend believes we will be declaring this criteria after that so no teacher will get a certificate of passing that FDP unless every teacher A submits the required assignments and assesses other assignments that will be part and parcel we will probably use Moodle which has a mechanism of randomly assigning thing or we will take that Moodle algorithm and integrate it with IIT Bombay X that for a random assignment is given now comes the question of quality so how do you adjust the quality of assessment every assignment is assessed by 5 people and every person assesses 5 assignments it is not a round robin so I assess some 5 assignments submitted by 5 individuals and each of these individuals assess assignments which are as he correctly pointed out I can do a randomized permutation matrix at the beginning now I have these assessments it is easily possible to do a salt and see which are the assessments which assess something at level A, B, C, D whatever whatever whatever and who rated what so now suppose the majority of the people rate something as A and let's say there are 200 assignments which are rated A now those who rated all of these as A those who rated all D's as D's are obviously better reviewers because they seem to be consistent with the majority opinion majority opinion itself may be wrong now imagine if I do that because 4 or 5 different submissions throughout the length of the FDB and I have a unique opportunity after doing this these people in groups are going to participate in a physical interactive session at a place so suppose I say now these 40 people who are there from amongst these 40 you find out who did what and try to identify 10% that is 4 or 5 people whom in your opinion are great assessors we can make the information available to all of them so I have an advantage of an automated process followed by a manual intervention the manual filtering by peers it is extremely difficult to use this manual intervention to grade all 50 people from 1 to 50 assemble at one place but it is entirely possible to correctly predict the 5 best people amongst the group of 50 you agree so I am not doing a ranking of all 50 I want the system to set 5 now in some cases there may be even 10 there may be only 3 because this will be combining the automated result with the manual interface now suppose I say that these 5 people who are selected at each remote centre will be given a recognition special recognition certificate and a 1000 rupee award which means they can do a next fdp without paying any fees would it act that is a reasonable incentive these are some of the mechanisms that we need to think of in the case that I just described we are trying to combine the automated process with some human filtering at the end after the automated process has thrown some results but the fact remains that in large group communication unless you automate to the head you will not be able to run things successful take our courseera course for example would that course have the same meaning for you if you had only auto grader for the programming assignments rather than a human assessment so you did value intrinsically the human assessment which was coming your way and perhaps that was one of the reasons why you contributed by making your own assessment but purely voluntary things do not always work consistently over long periods of time that is the reason we have job with salaries otherwise everybody should be working purely for job satisfaction but we work partly for job satisfaction partly for salary large partly for salary so you need to have a combination and that is exactly what we propose to do that here I do not know how many of you are working on associated systems for example ranking algorithms the entire area of machine learning is replete with whole lot of mechanism that are available which ones to use for what purpose and how is not very clear in this particular case we will be attempting to do that over this summer we have some exciting summer interns I mean people like what you were 3 years ago before coming to IIT you would have done your undergraduate study somewhere so these are all third year students from over what should I say about 200 100 and 130 summer interns we have chosen also we had 3000 applications so how do we assess 3000 applications have you ever thought of actually looking at each application individually so Prof. Avinash Haute came up with a very simple standard thumb rule any student who is a topper at the end of 5th semester in a college is selected why is simple thumb rule is which college students are studying that topper is a topper is a topper then he said some of the institutions of repute such as NITs and so on he would go up to the second topper 3 years ago when he outlined the scheme I asked him but what about such large number of individuals who do not necessarily shine in academic performance but are extremely creative and want to do something 60-40 ratio 60% of interns he selects like this 70-80 interns are selected like that then for all the rest 2500 people he announces a software contest and in that contest people have to participate and winners of that contest are selected independent of their academic standing so that caters to a group of students who may not perform very well academically but are extremely creative and good and of course then we have this standard funny cases of plagiarism how do you detect plagiarism in such submissions programming assignments automated checkers well automated checkers also will require you to submit I mean like you check plagiarism in research papers you can check code but you see these students who copy from each other are really smart so they will introduce different variable names I mean in hand written exams I remember that is how we used to catch copying that they have actually changed the variable names most of the program but at one place the original variable name appears by mistake and then the fellow gets caught suffice it to say that in our case what they did was very simple they took all submissions from the same college is unlikely that I am in a Mumbai college and I copy a code submitted by a student in Jharsugudha it's quite likely that my own colleagues in my college is what I'll copy from so that is how they checked and they found some copying cases then they wrote a letter and and promptly in some cases there were three identical submissions in some cases too promptly we got messages saying sorry I copied she did not or he did not so admit him or her not me and that is how it was resolved anyway now these people why I am saying this is these people will come here anyone of you who gets an idea of using any known technique or algorithm that you are familiar with or you become familiar with which will help in building a system of this kind I would appreciate you sending a small email to me we propose to assign groups of these summer interns to actually build a prototype of this system for group communication in IIT Bombay using Mossams for for the interviews which I'll be conducting the mock interviews there are two major exercises which are still remaining by the way one is weaving TED talks and writing their summaries all of you are familiar with TED talks so how many TED talks have you seen have you viewed they are not very long talks they are generally 18 minutes as a thumb rule this is a good type a compromise between 5 to 7 minute course video versus a 1 hour lecture like 5 month lecture so how many such videos have you seen any numbers anybody who has seen more than 5 TED talks can you raise your hands more than 10 more than 20 fewer more than 50 1 if I continue like that it might turn out that you mostly do TED talk weaving mostly do other things I was just joking by the way these TED talks because they are talks by individuals they combine the content connect as well as emotional connect where the person is there speaking there and that is the beauty of a lecture that's why you want to attend a discourse or a talk given by someone so these are extremely useful talks I do not know whether the average it will this statistics will be generally true for the entire class or not sure that means everybody would have viewed 5 to 10 TED talks at least but which 5 or 10 that depends upon one's own now how many of you have done this exercise of weaving a TED talk and writing the summary of that TED talk in your own words that you may not have done because you try to absorb the gist of that TED talk in your mind that is the purpose of weaving anything no? video thing but a very interesting and useful exercise is to actually view a TED talk and write its summary it's almost like writing an abstract for your work based on the presentation in the seminar that you make so if you ask your examiner that I will make a presentation for 20 minutes at the end of it please write an abstract of what I said curiously that is what an examiner is doing actually while assessing you for a grade that is what the examiner is doing his or her mind an abstract is being formed of what you have stated and on that basis the assessment is made but doing that exercise is a formal exercise a very useful thing so I will be actually giving you a set of 10 TED talks we won't have any randomization there will be numbered 0 to 9 and depending upon the last digit of your roll number you are required to write an abstract but these abstracts will be published and those of you who wish can also visualize other TED talks we select them after a lot of care trying to encompass different topics and try to see whether they remain in some sense related to the activities that you do so that would be an exercise that you will have to do in the month of April and there will be mock interviews for some people so like placement interviews those of you who are already placed or something did not participate but we will select people randomly I am calling some industry associates to conduct these interviews they will be conducted in a small room but they will be recorded on camera and I will see if I can connect through AVU so while the people are being interviewed somewhere here you can see what questions are being asked and what answers are being given so that would be fun it would be useful right so that is what we propose to do it in April and that is why I don't want to cancel classes I think that is an important exercise alright thank you