 So, what I want to briefly describe today is a consolidation of what has been said already and has been put on the model, but some more details, because there seems to be a lot of confusion on what is expected to be done by every individual, by every team, by every project group. So, I describe the project groups once again. There are specific project submissions and hard deadlines that must be noted by everyone. Finally, I will describe the project evaluation. Test questions and quizzes to be said. I will describe the approach, the work to be done and the evaluation that will be made. You will recall that we have removed five marks from the lab assignments and five marks from the quizzes. So, we shall have to describe the quiz marks and the ten mark assignments, which I will describe today. This is the repeat of the yesterday's slide. I did not spend much time on it, but basically I have re-emphasized that the projects are necessarily open-ended and ill-defined to reflect exactly how you will face the real world where the problems are not as well formulated as questions in an exam. But I believe through these projects you will learn to take decisions, you will learn something about teamwork, and you will learn to discuss design and document whatever you discuss. When you write a program individually, you are not compelled to make notes before writing the program. You can have all your notes in your mind and directly start writing the program as you often do in a mid-same exam or in an assignment. However, when you conduct a group project, your thinking must be visible to other members of the group. And the only way you can do that is by documentation. So whenever you discuss something, the discussion results must be documented and must be circulated to all members of the batch, all members of the team, etc. So please spend time on deciding what are the naming conventions, what are the file structures, what are the memory data structures, identify functions and write stops as I mentioned. That's the only way you can proceed quickly on completing a large project of this kind. Please note, however clever and experienced an individual may be in programming, one individual cannot ever do a large complex project. And that is why it is important that all members of the team for a team assignment and all teams in a project lab batch, plus all four lab batches working together in a group must work in a composite fashion. To recapitulate, there are 10 groups, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. Two of these groups assemble every day. So that is how our lab batches are formed. Each group will together work on four projects R, C, D and A, which have been already described. Each group will have four lab batches and one of these projects has to be assigned to a lab batch. I presume this assignment has already been done. If it has not been done, it should be completed preferably by tonight. Then the four lab batch coordinators shall take decisions related to any common data structures, file naming conventions or whatever common interfaces that are required. This is vital. You must take these decisions as early as possible so that individual lab batches can start writing their programs or designing their programs at least. In exactly the same way, within a lab batch, if five teams are to work on five different activities of the project, then the coordinator for the batch, in consultation with other team leaders, must allocate first of all activities to individual teams and for these teams also it is important that any common interface or any common functionality be decided upfront. Now each team will work on one activity, but the team will also contribute to the general common definitions for the entire lab batch. Similarly, while a lab batch concentrates on one project which is assigned to that lab batch, they must also contribute to the work which is required to be done by all four batches together. Consequently, there is a huge amount of non-programming work which is required and that's a fact of life. Although this is a programming course and similarly you will be doing various other engineering and science courses, while you study an individual course, you attempt to acquire proficiency in that course, but when you want to apply the knowledge gained in that course in real life, almost 80% of additional work is required to be done which is not related to that course at all. Exactly same as the situation here, the non-programming work in the project will be significant. Now, as mentioned here, since the work involves not just programming, but also testing, documentation, data and file management, schedules and meetings, reporting, etc., the allocation of work to each individual student within the team and each student within the batch and each student within the group will have a lot of flexibility. What is expected minimally is written here. Every student should write at least one program or function for his or her team and additionally should carry out at least one common task for the batch or team. This is very, very important. So, the project is not just about programming, but is about doing variety of other things which are required to successfully complete the project. This assignment, of course, should be done by the team leaders and by the batch coordinators. But please ensure that every student, because this is ultimately a programming course, every student writes at least one program. However small or big, it does not matter. Of course, some students may end up writing four, five, six programs. There is perfectly fine. But every student must minimally write a program and every student must minimally participate in any common activity that is defined for the batch. This is important. If you have not already done so, please pay attention to this. Keeping records of how much time you are spent on which activity is important, not only from the purpose for serving the purpose of your own counting, how much time you are spent on the project and doing what, but this also defines totally how much time was spent by a team or by a project lab batch on the project. And please note this is not related only to program development. As a matter of fact, when you submit all your programs, tested and documented, the total lines of code that you may write may vary between 2000 to 10,000. But the total time spent on the project will be some total of the time spent by all students working on that project together. And that time will include time for example, for discussions, time for designing, time for actual programming, time for testing, documentation or report writing, time for logistics, just maintaining manual records of fingerprints collected, uploading files, CVS work, etc. Additionally, there could be a whole lot of miscellaneous work. This kind of categorization is suggested. Individual groups may add to these categories or may subtract from these categories. But ideally every student should maintain a daily diary. It is quite possible that on some day a student does not spend any time on the project, which is perfectly fine. The entry for that day should say simply nil work. But on any day when you do a work, you have to classify what is the nature of that work and the start time and end time, the total number of hours and minutes that you have spent. So suppose I spend one hour 15 minutes on a task, somebody has spent 35 minutes on a task, somebody else from the team spends 15 minutes on a task. All these time spent on a daily basis should be accumulated on a weekly basis. And this accumulated time is what must be reflected in your report. So this is important. Daily diary must be maintained. Now each student must submit the diary entries in the form of a common electronic document on a weekly basis. This is a must. Without if any single student's diary is missing, the whole project may even get disqualified. It is that important. Maintaining your own record is that important. I repeat this because in India it is not traditional to write diaries. We generally don't write our diaries. But every professional always writes professional diary. When you work, whether it is in a mechanical engineering factory or in a software engineering firm, you will be required to maintain daily time slips. Particularly in the software development activity, a daily time slip is a must. You cannot leave your desk in the office unless you are filled in the timesheet for the day. Now that rigorous routine is attempted to be implemented here in this project. And that is why what we ask is not a daily submission, but a weekly submission. So the format for this is to be decided by the group. What I am suggesting is a simple text format which can be edited using g-edit. So every student will prepare a g-edit document which will be a growing document on a daily basis. So every day you add the day's entry to it. At the end of the week, all such documents are consolidated. So they can be put together. Student roll number so and so, diary entries, roll number so and so, diary entries. All the teams' diary entries or all the lab batch diary entries could be put together in a single document. And that consolidated summary should be written on the top of that document. This consolidated summary will count the weekly total of person hours spent by each team of the lab batch and by the batch as a whole on the project. So this is absolutely vital. This is not really meant for evaluation, but this is meant for each one of you and the batch to know exactly how much time people have spent on various activities. Ideally, the consolidation should describe not just the total time spent, but the time spent on different activities as you have classified. So how much total time was spent on discussions, how much total time was spent on programming, how much total time was spent on testing, and so on. And you will suddenly discover at the end of the project that if somebody has spent 100 units of time, some lab batch has spent 100 units of time, the actual time spent on programming could be as little as 10 or 15 units. There's nothing unnatural about it. That is exactly how large projects consume time. A whole lot of other activities in what we call program management or project management consume a large amount of time there. So that's okay. Of course, what is required is that the programs that you write are properly tested, properly documented and submitted in the final form. Now, every week this submission should be made for the consolidated document, as I said, in the form of a single consolidated document. This is to be submitted to the batch JTA. All submissions are due midnight of each Sunday. So the first Sunday will be the coming Sunday when you will make this submission. The submission need not be perfect. So suppose you have written a program only halfway through or you have written only comments of a program or you have written only the title of a program in to main and you have not completed a program. It's perfectly fine. But this consolidated document gives you and to us who are evaluating you a test of how progress has been made. So please do not hesitate to submit partial programs. In fact, you are required to do so. Ordinarily, I would not expect a final working program suddenly to pop up from the sky. That does not happen as all of you know. You will have programs which will have bugs. You will have programs which are half return. You will have documentation which has half return. It's no problem. But on a weekly basis, all such documents, all such program files, source code of course, do not submit executables and such things because these are reports which are to be visible, which are to be seen and read. So these reports will have a large number of files. An important file amongst the submission would be the consolidated diary file. Additionally, any programs that people have written partially, any design document people have written, any document on standardization of the file formats or function interfaces, etc., even though they are in the preliminary form, not necessarily in the final form. In fact, whatever has been done during that week, all the documents, all the files should be collected together and a single tar should be submitted. So to recapitulate, each weekly submission will include consolidated diary document for the week, any design documentation, partial program code, testing report that has been prepared. All files and directory should be put together in a single tar and uploaded on the batch model. So instead of having different directories of different students, I suggest that the submission material should be collated by all students and put together in a single file. I was told yesterday that there is a problem in CVS implementation. I have a meeting with Dr. Subhadra today and we are trying to set up for 10 groups and for four lab batches in these groups, 40 CVS route directories. We shall be keeping these on some of the machines in the lab itself. We'll identify which are those lab machines. So the CVS route permission will be available or those CVS directories will be accessible to the students of a batch. The lab batch coordinators and the team leaders together should work to decide on the excess control and so on. For this purpose, I propose to conduct a special lab session, a demonstration session where one member from every lab batch should be present, either the coordinator or any other designated person. I will announce that time. It may happen on this Sunday. So these 40 people will be told how exactly the CVS is to be operated by all the students of the batch. Subsequently, in the next two weeks, you will be able to use the CVS to manage your files. This is absolutely essential. You please remember that the weekly submission must include these things. So I would not like any final sort of program or project report to pop up from the sky. This must be a growing document. Please note that in the next week that you submit, you may be submitting a program which was partially submitted earlier. That is perfectly fine. So each week's submission is not independent of the previous week. It's actually a growing document. So if I do a difference between the second week submission and first week submission, I should be able to find out how much work has been actually done on the ground. The diary entries, however, will be mutually exclusive. The diary entries will be for one week each. But the other files that you submit would very well, in fact, they will almost always contain the documents which have been submitted earlier, perhaps grown to a better level of perfection. That is exactly how the project document grows. So submission and evaluation, the dates for which midnight is the hard deadline, the first submission on 1st November, second submission on 8th November, and final submission, midnight of 15th November. We have a lecture on 16th November. I have not yet decided what I will do because it is too close to the final exam. I believe the exam will start on 18th or 19th. I will take a call in the last week depending upon how much material we are able to cover. In any case, we will have the lecture on 16th, but most probably I will end up discussing only the format of the end semester paper on that day. The project evaluation has been clarified earlier. There are 15 marks for the group project. That is, each lab batch will get marks out of 15. Each student gets the same marks. So if a lab batch gets, let's say, 12 out of 15, every student of the lab batch gets 12. That is because he or she has participated in that group project. These group projects will be evaluated jointly by the back JTA and the group seniority. So based on your submissions, and this will be a growing evaluation. It will not be done only for the final project report. So what work you do every week or what submission you do every week is equally important to decide how many marks out of 15 you will get. 10 marks for each student based on peer review. I have explained the process. I will mention it briefly once again because you have never done it earlier. It is important that you understand what is the peer review. I mentioned this yesterday. It has never been attempted. But you have to award marks out of 10 based on the quantitative and qualitative contribution by each student. These words are important. There could be a few students who will make significant qualitative contributions in terms of design ideas or in terms of approach or in terms of meticulous planning and project monitoring. There will be some other people who will contribute significantly in quantitative terms. They will just do a whole lot of documentation. They will consolidate all documents. Now it is important for a lab batch. The lab batch consists of about 20 to 25 people. Maybe 18 in some cases. 18 to about 22 people is my guess. This batch forms a coherent group and within this batch the peer review process must ascertain that marks are allocated out of 10 to each student of the batch. As I mentioned yesterday, additional incentive for special extra work for best creativity, for best teamwork, etc. will be awarded by the course. And these marks will be in addition to 15 plus 10. It does not matter if the total exceeds 100. I'll be very glad if some of you get more than 100 marks in this course because we still have a provision for what we call an A plus grade, right? There's an AP grade there. I hope at least some of you will get that grade. But these extra marks will not necessarily be given to individuals. These extra marks will be given to things like best creativity, best teamwork, etc., etc. And these could be awarded either at the level of lab batch or at the level of team. Again, depending upon the reports that are submitted and I hope that in the weekly reports that you submit not only you put the diary for each individual student and the total work done, but the team leaders and the lab coordinator together should take time to write down what exactly the whole lab batch has achieved. May be a paragraph in that week. And for each team, what exactly the team has achieved during that week. So there are five teams. I would expect five paragraphs team-wise. Team one, team two, team three, team four, team five and one paragraph of the lab batch will be part of the final submission. Formulating these five paragraphs plus one paragraph will take at least a couple of hours. So please account for those two hours at the level of batch coordinators and the team leaders. Now I will revert back to paper to describe the peer review process. This is for allocating 10 marks. Actually, this is all it takes to award marks out of 10. The meeting of about 30 minutes of the entire lab batch. You can meet in the lab itself. You can meet outside. You can meet under the tree. You can meet in hostel. I can make some classroom in crescent building available for this purpose in the last week. In fact, for such meetings, I am proposing that there are some small rooms which can accommodate 25 students. I will reserve two rooms in the evenings and nights for such meetings if you wish to conduct them. What do you do in these meetings? Of course, 30 minutes may be the actual meeting, but a lot of homework should be done by every individual student and by every team. So basically you have to award marks out of 10. Ordinarily, since every student would have contributed in some way or the other, the quantitative contribution can be ascertained by looking at the diary of that student. Every student would be familiar with one zone diary and the team leaders and the lab batch coordinator will be familiar with all the diaries because they will be consolidating these diaries. You can roughly scale on a quantitative scale a sort of evaluation of how much quantitative contribution each student has made to the lab group. Additionally, the qualitative judgment can be made by groups of individuals. So suppose somebody finds out that a student has made significant contributions in design ideas or writing large programs or something, that contribution can be kept in mind. Essentially, when you give these marks out of 10, there is no quantitative accurate scale. It is not possible for anyone to give marks such as 7.25 out of 10. That is nonsense. So there will be a lot of gut-filled judgment that will have to be arrived at. However, any marks that you give must be commensurate with the quantitative and qualitative contribution made by a student. Since it's a long-term activity, I do not expect anybody to get less than 5 marks. So effectively the scale is likely to be 5 to 10. However, there could be students who have not contributed to the project at all. Say somebody has just disaligned oneself from the project activity or somebody has gone away or somebody was not well. All of these could mean that an individual's contribution is very insignificant. In which case, I would expect marks to be awarded from 0 to 4. Please do not hesitate to award 0 marks to someone who has done nothing at all. Now that is the responsibility that you have. That is the accountability that you have. However, as I said, I do not expect any such thing. In only exceptional cases, there could be somebody who gets 0 mark or 1 mark or 2 mark or 3 mark. What is to happen in this final meeting is a waiting of these marks. So generally each person should be aware of how many marks he or she deserves. And that is what should be put forward and endorsed by the team leader in this meeting. The meeting should really review only those cases where there is no agreement. So for example, if I say I claim 10 marks because I have done so much, whereas others say, no, I should deserve only 8 marks, this is what needs to be discussed. In any discussion, a decision has to be arrived. Ideally, a decision should be a consensus decision. However, if there is no consensus, then the lab batch coordinator along with the other team leaders, that is the five team leaders together, must take a call. And even if there is a difference of opinion, that must be resolved at the level of team. This is the responsibility and accountability you have. Ordinarily, if in a mid-same exam, you believe that you deserve two more marks in a question and you have got two marks less, you have a CRIP session. You meet with TAs, you meet with JTA, sometimes you come to me. The CRIP session ends in this meeting. There is no appeal. And that is the responsibility, that is the accountability. Please understand, however, one or two marks difference in this scale will not make any significant impact. And therefore, suppose I claim to deserve ten marks, but I get only eight, I should not worry too much about it. I am not suggesting that marks must be reduced to eight. By all means, if you find significant contributions, by say, five or six members out of twenty that you have, do award them ten on ten. Absolutely no issue. If somebody has not done any work, do award zero marks, convince that student. If the student is not convinced, that's the accountability you have. These marks have to be submitted, roll number wise, along with the last submission. So, the Sunday submission, the final submission will also include one sheet, which will have roll numbers and marks, roll numbers and marks for all the 18, 20, 25 students of that batch. Now, ordinarily, I would have conducted those sample interviews, which I had told you before the mid-same, before the end-same begins. But unfortunately, there is not much time. If you make submissions on 14th, you will have the end-same starting, I think, within two days of the last lecture. Anybody has an idea of when the end-same begins? Nineteenth. Okay, 19th or 20th. That still gives me couple of days, but I will not be able to make up my opinion on sampling without I go through all the project reports and individual diaries. What I will do, however, is on the day of exam, I will announce certain names. Those will be the sample names, who will be separately evaluated by me and some of the lab tiers, and we will decide the mutually convenient time during the next two weeks, during the exam itself. So, for example, if some of those sample students may not have an exam on a certain date, I could conduct the interview on that date, etc. Ordinarily, I would expect the marks allocated out of ten to stay. As I had mentioned earlier, if the sample interviews, if the sample viva indicates otherwise, that means some student has been allocated, let us say ten on ten, and I determine that the mark should be seven on ten. Then three marks will be reduced for every student in that batch. It is very, very important that you understand and appreciate this. Once you appreciate this, the need will not arise. Believe me, I have tried it in the last five years in my postgraduate courses, except once in the third year of this kind of method, where I had to change the marks for all the batch. In fact, that batch lost heavily. Each student lost five marks out of ten, and several students went into negative mark range, because they had given five marks to a student who had done absolutely nothing. According to our judgment, he should have been given zero marks. He was given zero marks, but because the peer review indicated five marks, five marks were reduced from every student. This is very serious matter. So please be very careful in your allocation of marks, so that the marks stay put in spite of any external evaluation that is done through such sample Viva. So sample Viva names over and above these ten plus fifteen marks, as I mentioned, were working out an incentive scheme. I was hoping to announce that today, but I am waiting for the TA meeting, which will happen either tonight or tomorrow. So this Saturday lecture I will announce the incentive marks. That is not important. The incentive marks really are to honour those individuals or teams which show some extra activity or extra work. So this is an incentive for that extra work. So this completes the project evaluation. Anybody has any doubt on this? Do you agree that this method is likely to be more truthful in evaluation? The peer review method? Fine. Well, those who say no are only worried that they will have to convince their own colleagues about how much work they have done. That is always far more difficult than convince an extra agency like a teacher or a TA that how much marks you did. But it is important because in real life this is exactly what you will be doing all throughout your professional careers. It is important that you learn this. Now, as you will notice, there are no lab assignments for last couple of weeks and there will be no other lab assignments in the remaining weeks. You are expected to spend two hours per lab. So instead of this, there is a composite assignment that is being given. This will have to be submitted one week before the project submission, which is exactly two weeks from now. What is to be submitted here? So let me begin by describing the notion of a test question. All of you are familiar with these kinds of questions. We had asked some programming questions in the quiz. We had asked some questions in the mid-sem. You are familiar with those questions. The questions could be simple. The questions could be very complex. All the questions could have medium complexity. I describe these three questions in S, M and C categories. These are simple questions, medium questions and complex questions. Each question is a programming question. So you will have to write a question just like you would see it in a mid-sem exam or an end-sem exam or for that matter any programming question that you might see in a textbook or in any tutorial that you come across. Ideally the question should be suitably modified so that it becomes a new question. You may pick up an existing question also. It's not a problem. The questions have to be decided at the level of a team. So the submission for these questions will be from teams and the final selection will be done by the lab batch. So that means every lab batch will set how would each question be submitted. First of all the question will be decided by the lab batch coordinator along with the team leaders. So five questions will be set. The team which sets the question will not attempt to answer that question. Some other team will be asked to answer that question. This will be a regular programming question the like of which you come across. The topics can be allocated batch-wise and the lab batch-wise topics some of which will be repeated that allocation will be announced on this Saturday. So what you are required to do is to set up a question and every student in that team so every team will be given one of these five questions every student in that team will actually time and write an answer to that problem in half an hour. This will have to be timed individually. One can take this question in one's hostel room but that timing has to be exquisite. You start writing and you stop writing as if you are giving an exam. There are absolutely no impact on the marks that you get based on the correctness of the answer that you write. You are supposed to write exactly how will you write in an exam. Once you time this out this has to be entered in a readable form. The answer has to be entered in a readable form and is to be compiled. So there will be five such sample answers. It is important to understand what exactly I mean by this. Suppose I am part of a team which has four members and my team gets a particular problem on let's say handling some array or finding out maximum or finding arranging, sorting whatever. Now as a team member along with my other team members I will actually try to write down the answer to this problem in half an hour. So I will spend half an hour in attempting that answer. That answer in comments should have my roll number so that it is known who has attempted that. All these five attempts are to be put together in a single text document. In the same text document at the end there should be final answer. Now this final answer should be arrived at by the entire team working together. So that team can spend additionally up to one hour. I hope you understand the idea behind this. Once you get a question you get a real life test of attempting it in half an hour. Of course most of you will not be able to complete that answer or most of you may not be able to get the correct answer. Doesn't matter. The attempt is what is important. So in 30 minutes you attempt to answer that question and put that attempt together. All these attempts are then consolidated by the team. The team discusses what should be the correct question or correct answer. That answer is evolved out of an additional discussion of one hour. You should then try it out on the machine ensure that it runs and then submit that as a final answer. This activity gets each member of the team so five marks are awarded for setting up a question and making submissions like this. Please note that as an individual suppose my attempt is so feeble that it is not even correct, not even complete yet I will get the same marks because ultimately the judgment of these five marks will be based on the final answer that is submitted by you. The quality of question and the final answer. However there must be a genuine attempt for 30 minutes by every student. So if I get a blank submission from a student or if I get a submission from a student individual attempt roll number so and so and then another comment sorry cannot solve the problem the entire team may get zero marks because it is impossible that any student doing this course at this stage will not be able to at least attempt something or the other. So the five attempts together five or four or three depending upon how many team members you have must be included in the final submission. Is that clear to recapitulate? A lab batch decides on five questions each question is given to a team each member of the team attempts to solve that question in half an hour half an hour is deliberately low some of the questions which are of the complex type may require 45 minutes to solve even for the experienced programmers. Does not matter if in 30 minutes you are able to only understand the question and attempt a little bit doesn't matter. What is important is finally when you assemble all these submissions you will put together a final answer this is a team effort all five members of the team must make an effort in case even they cannot solve that problem and if they want somebody's help they are most welcome to take anybody's help but that must be acknowledged that in the final answer the following contribution was made by such and such student the idea is to get as good an answer as possible for that question the team may decide to include two final answers or three final answers if they believe that two or three different alternative answers are still okay this will get you out of five marks for each member of the team again I do not expect a significant diversity in the final attempts and therefore I will be highly surprised if anybody gets any team gets less than three marks in this ordinarily the gradation will be three, four or five and this will not depend upon whether the question is simple, medium or complex even a simple question very elegantly solved properly documented and correctly tested may qualify for five marks so simplicity has nothing to do with it but marks will depend upon the quality of the final answer that is produced for the question and for the formulation of the question itself these five marks are for a quiz question in exactly a similar fashion you will set up a quiz question which may take anywhere between two to four minutes to solve there is no question of individual attempts being included because the questions would be either multiple choice questions or fill in the blank kind of question or writing one line of the code kind of questions you have to formulate these questions but what is important is if there are multiple choice almost all the questions will be either multiple choice or fill in the blank or what is the output produced etc etc what is important is along with the question if you give multiple choices you have to give the right choice you have to give the reason why that choice is right and why other choices are wrong if it is filling the blank when you fill up the blank you have to say why that particular answer is right if you have to work out the output that the program produces you must give at least one paragraph full of reason as to why that output is justified and why nothing else may happen setting up this question is again a teamwork as you can see these ten marks are almost available for asking you have to do good work somebody will have to design good questions and people will have to come up with right answers the quiz questions the marks decision will be almost always on a qualitative basis the quality of quiz I will elaborate the nature of the submission that you have to make on the quiz question you have to identify what concept you are testing and you have to describe how that concept gets tested by that quiz question I will elaborate this in the next lecture thank you very much