 So, I hope you got some time to think about that question over T on the create level question in your topic. So, let us actually begin with that and what I would like you to do at this point is if you have not written it, take another 2 minutes or so to write it and then share it with your colleagues in your remote center and RC coordinators if you could please share one such question so that we get an idea of what are such questions we will then go to the next part. So, at this point you can use chat to write to share your question will take about 5 minutes for this activity write your question share it with your colleagues ok great looks like many of you are sending it looks like many of you are getting the idea and I am going to read out some which are let us say really open ended because what we want to get into now is assessment we are going to now look at a couple of examples and then we will go to the next part. So, let me read out a few one says create a workflow for hospital management system. So, even depending I mean it does not matter which field you are from, but if you see a question like this you know that there is something fairly open ended about this and students also have to use a lot of analysis and evaluation in order to answer this question because they have to make several decisions in the workflow. So, it can be workflow for any process which involves a number of different decision making points. Another example ok here is one design a website for online shopping again this is a question which requires a lot of different inputs in terms of decision making in terms of an analyzing which might be more efficient than something else and so on. What is not a good example of a create level is writing a C program for Fibonacci series and the reason I say that is really not at the create level is it is a routine algorithm and students mostly have to code it. So try to make a difference between questions where there are multiple valid correct answers multiple paths to solution and so on and others where there is a fairly straightforward algorithm that needs to be implemented. Let us look at one or two more questions. So, here there is something that says design a super elevation for a two lane national highway. So, if you are doing civil engineering transport and it also says assume all data as per IRC. So, what is IRC, but it looks like some standards so where you have to take real life constraints and standards and then design a solution for an actual real life problem that is open ended design problem. Maybe one more create level and do not write questions like write a C program to sort elements those are more routine apply level questions, calculation questions ok here is one. Develop an experimental setup for the following physical principles. So, there is some physical principle given and the students are asked not just to do the experiment, but they are told to come up with their own experimental setup and actually create the setup. So, I think you all got an idea of what is meant by what kind of problems we are talking about. So, now let us see a challenge that we have. So, the main challenge we have is how do we meaningfully assess such create level question. So, what we will do is a quick think pressure activity here. So, now at this point do a think phase where you write two challenges of assessing the above question. So, one minute for the think phase and then we will move on to the pair phase. If you have done the think phase you can move on to the pair phase where you discuss with your partner and come up with one technique that to meaningfully assess the question that you wrote you can pick any of your questions. And make sure that the technique actually addresses the challenges that you wrote in the think phase. So, the question is if you have to assess an open-ended create level question like the examples we discussed here, what are some challenges and what is one way of assessing it which can address these challenges. And if with your partner you have done that share your technique and let us hear some of your techniques. Challenges are something I think all of us are aware of. So, that was just to get you warmed up to answer the actual question as to what kind of a technique will you use to address some of these challenges. Most of you are saying that one of the main challenges is that there is no unique answer and you are all absolutely right that is one of the biggest challenges that there is no unique answer. But yet as teachers we have an idea of poor answers and better answers. So, how will you meaningfully assess various solutions that you get. So, some of you are saying you will do a mini project evaluation that is ok, but what is your basis for evaluation. So, some of you one person has said that he uses a grading system based on milestones that he can define and these milestones can be at different cognitive levels. So, it is like a roadmap that if you have got until one particular milestone you may get some credit. If you get to the higher level milestone you may get some more. So, this is one way it can definitely be implemented. Some of you are saying developing rubrics. So, that is what we are going to talk about, but tell me what you think what you mean by rubrics. So, those of you who are saying rubrics let us hear what you mean. Another challenge that some of you are sending is that they often this is group work. How do you assign points and so on ok. So, let us actually talk a little bit about rubrics since many of you have mentioned rubrics. This word is used for a number of different things, but to effectively assess open-ended solutions and to meaningfully assess them the rubric needs to have certain characteristics and that is what we will explore in this session. So, if you want to use rubrics as an as an assessment tool let us actually start with some of the challenges which I am putting under goals and constraints. Our solution must make sure that it addresses all these goals and constraints. So, multiple valid approaches, multiple valid answers and solution approaches. This one many of you mentioned. We as teachers want to evaluate not just the final product, but also the process. That is important to us because that also tells us what the student is learning. No single measure exists. In the sense the assessment has multiple diverse criteria of to say whether something is let me put in quotes good or poor. Often the products or the artifacts that you get from students they are not text, they are not equations or numbers. So, you cannot slot it into right or wrong. Often they are very descriptive or they may be diagrams or they may be actual products. How do you look at these? So, for all of these traditional assessments are really not very useful because they will fail on one or multiple of these goals and constraints. So, let us look at what rubrics do. Rubrics are one solution. Now, this is sort of the open blank template of the rubric and I will go to I will go step by step. So, a rubric actually finally it will look like a table. This is only one row of the table. So, first we will understand the columns then we look at different rows. So, the first column this is the most important column it defines the criteria a very specific performance criteria like a learning objective. It is also called a competency sometimes. So, this says the student is able to structure open ended using specification. So, let me explain what this is. Here one of my colleagues Madhuri that you saw she is a PhD student in our group. She is coming up with assessment rubrics for design problems in electronic circuits. So, what she did is first listed various competencies or criteria that we need to see from students when they are designing electronic circuits. And one of the top ones there was to take an open problem and structure it into a more well defined problem. So, if I take your example of designing a two-lane super elevated highway that is a very broad problem. So, in order to do that what are the more structured domain specific problems that one has to answer in order to make this large highway. So, the structured problem in the highway example could be that should be able to take a load of so much should be at this certain height. It should have materials which have these properties we still do not know what those materials are. But the first step is to convert this broad problem which says design a two-lane super elevated highway into saying something design a highway with which can take a certain load and can be used for certain time and so on. So, the first one of the early competencies that she was trying to measure was that student is able to structure the open problem using specifications that are given. So, load in the highway problem can be one of the specifications. So, in a rubric for an open ended problem you will have multiple such criteria and each criterion or each competency is fairly specific. You can have as many criteria as you want, but try to make each of them concentrate only on one specific aspect of the problem and the reason for that is if you looked at some of the constraints there are multiple criteria of assessment for open ended problems which makes it challenging. So, we are trying to address the challenge by writing each of these criteria in different rows. So, we have to address each of the multiple criteria, but not together. What happens after this is we write down a description of the desired or target performance of a student that if we want a student to do this what should his or her performance actually look like. So, let us look at an example. All specifications are used to take decisions to structure the problem all interconnections of the system are specified. So, this is for lack of space on the slide I did not put in more text, but in the target performance you can write down all the things that you wish to see that you desire to see from a student's performance to be able to judge if they have achieved this competency or criteria. Let us look at why this is actually defining a performance. You still have to translate it or apply it to your own problem, but several specifications are given like the load and may be the number of vehicles which travel, may be the amount of rainfall, may be the temperature variations during the day. So, there are several specifications for the problem of designing the highway and the student must show demonstration that the decisions that they take must be connected to all the specifications. These specifications may be connected to each other they are not independent often. So, has the student made sure that the connections between the various specifications actually have been identified and used that is the target performance. As I said like this you will have multiple rows. What happens next? Once we write the target performance you as a teacher write down some other levels of performance. The lower levels of performance we will see how to do this and why this is important. In this particular example three the target performance plus three levels are written. So, we will just walk through them one by one. A student who has made an attempt to use specifications but has left out some minor specifications in the decision making process. That student will fall under the category of needs improvement. So, let us say there were five specifications and the student has used four of them and not the fifth which may have been somewhat important for decision making or they have used all specifications but have incorrectly used them that can also be come under needs improvement. On the other hand if a student has made an attempt to use the specifications but they are entirely wrongly applied or some specifications which are really needed to make the decisions have not been used at all have not been even thought of that student can fall under the category of inadequate. And finally if a student has made no attempt to use the specification we will say that this person has not done anything missing. So, because it is an open ended problem there is some subjectivity again at the boundaries but at the same time it is very useful to know that there is a boundary between most of it has been used reasonably with some minor mistakes. It does not matter if there are one or three minor mistakes so long as we can say that overall holistically it is a minor mistake versus inadequate where most where some important specifications have not been taken into account in the decision making process as a result of which the final design does not work very well. So, your rubric for assessing an open design problem has columns usually other than the criteria and target performance. So, this has what is called a 0 to 3 scale if you want to give numbers and the numbers are not really grades if you notice there is not a single number in this rubric they are descriptive. We give these to students we tell them give it to them at two places one is right at the beginning when they are solving the problem or during their learning process we give it to them saying that these are the expectations. And then after they have done their product or after they have finished their problem we give it to them and we circle that look you are here right now you need some improvement. If you have multiple criteria I will try to show you a slide for that one you can have maybe one student falls under needs improvement in one criterion and in some other criterion they are inadequate. So, we are not saying that a student is good or poor we are not simply saying 5 out of 10, but we are specifically saying that in the ability of structuring problems they need improvement, but in the ability of implementing their solution they have reached the target. So, they can have they can fall under different categories in different criteria. This gives students lots of specific feedback on where they are and how they can improve. So, formally rubrics are descriptive rating schemes that define performance levels. Each word is important here they have to be descriptive, but yet there is a rating system and what they define is performance levels just like we explained. What is in a rubric? It has a set of scales which are the ratings we had a scale of 4 missing inadequate needs improvement and target performance. The top level is always the target and the bottom level is always missing. Some people sometimes there is a 3-point rating, sometimes there is a 7-point rating it is really up to you how much you pick there are pros and cons. If you pick a rating scale that has 7 different columns you can get more specific, but then the boundaries become even blurrier. If you have fewer ratings it is easy to slot people, but then the precision gets lowered. If it is really your call there, please note that this is not a yes or no it is not whether they got it or they did not get it, but it is to what extent they got it. So, it is a continue. It has target levels of performance and intermediate levels as we saw each level is highly descriptive and specific you can put examples and each examine this each element of the complex performance has a different scale. What this means is that there are multiple rows to the rubric. Right now we have only seen one row. So, what you can do at this point is take a few minutes to write your own rubric. Take the question that you wrote and we will look at multiple rows also. The question that you wrote in the beginning of the session and if you want to do the think and pair together that is okay, but first write one rubric item actually do the things separately it makes more sense. Write a single rubric item to assess the question that you wrote. What does it mean to write a rubric item? You have to write the competency, the target performance and all these levels. So, this entire table for one row please write it. Once you have written one item then share your criterion and performance level with your partner. So, that you will be able to get more than one criteria for the same problem and then you can do the share. I am going to make a comment here. Some of you are sending criteria which are still too broad for a rubrics. So, if you say complete procedure for designing something you cannot measure all that in a single criteria. So, try to break up your criteria into very precise specific points. It is okay if you have 20 such criteria, but write one or two very precise criteria and this is not about students should be able to understand something, but what students should be able to do in order to solve the open-ended problem. Okay, since many of you are asking about examples, let me try to let me do one thing. I will show you one example and then you can go back to this problem, but I think now we are getting some good examples that students should be able to use appropriate data to solve the problem that is a reasonably good rubric item. But you are only looking at whether they have chosen the appropriate data or not. You can have another rubric item after that which says students have done appropriate calculations using this data. Implementing the data analysis and choosing the data, either keep it separate or together it does not matter, but it is all about choosing the data and using it in the problem. Let us look at one more which is at a right level, I will give you some examples which are at a right level and some which are not. See, you are not writing learning objectives even though it looks very similar to them. What you are trying to do is to try to understand how to assess these projects of students. Okay, here is one, students should be able to justify why they use the principles to design a particular system. That is a good rubric item because there you are trying to look at the process. So let me do one thing, I will show you one full example of a rubric and then you can try to work on this particular question even later. There are two separate issues here. One is to come up with an open-ended question itself. So when you say students should be able to design something or they should be able to create a solution for this problem, those are all your actual assessment questions. Now we are talking about how to evaluate those questions or how to assess those questions. These are rubrics to assess engineering design competencies. Please look at the examples on the slides. The various criteria have been called as sub-competencies here. So I will just read out some of these criteria. Is able to construct representations of the problem, is able to maintain consistencies between different representations, is able to use representations to solve the problem. So these criteria came under the ability of using representations. So this entire thing called engineering design has been broken up into several sub-pieces. This is about representations. In the next block below, it is about structuring problem that we just saw, is able to extract required relevant specifications in detail from the given open problem, is able to actually structure the problem, is able to sequence the problem based on specifications and so on. So this particular rubric for assessing an engineering design circuit had close to 20 or so of such rubric items. Let us go back to seeing where and how we use these rubrics. So when is a rubric effective? If all important elements or criteria of the performance are included. That is why in the rubric that I showed you, there were some 20 odd rubric elements. So each row is either called an item or an element or a criterion or a competency. You can use any of these terms. Important that each element or each row must have a single dimension, otherwise the student will get confused on what exactly they got wrong. The purpose of a rubric is to tell the student what parts they got right and what parts they got wrong, but still maintain a precision. The ratings must be descriptive and comprehensive. So after you have written your rubrics, what you can see is that you can go and check if your rubrics has these elements inside it. And finally where and how to use these rubrics. So as we mentioned earlier, you use them for assessment of open ended problems, projects, presentations, portfolios and so on. To give you one example, if you want to assess students presentations, you can have some elements or some criteria related to their oral presentation, some related to their slides, some related to flow and logic, but each of them must be accompanied by what is the target level that we expect and what are the lower levels. That will help the student understand where exactly they need to improve and where they were fine. So when you come back January 19th, 20th and 21st, we will do a whole session on group projects. At that point we will come back to these rubrics. So today's was more of an introduction to what are rubrics and how you can use them. And it was relevant to talk about these today because we started with the open ended create level problems. We recommend that rubrics should be shared with students up front so that the evaluation criteria are made transparent to everybody. These are, we are setting the expectations and you can give it to them not just at the end but also during the learning process. It's also a good idea to ask them to self assess their work at the end of their work. That's also a very powerful way of helping students learn. Teachers can also use it to give feedback and it's very useful to measure student progress over time. Because you see that a particular student today is falling under the category of inadequate in some criteria but after 6 weeks maybe they have moved from inadequate to target performance. So you can track students progress over specific criteria. So let me also share some personal experiences of what were the challenges of using rubrics and how we overcame them. So I am assuming that you have written rubrics or sometimes you can in fact find rubrics for certain goals. Initially it may seem that there is a lot of reading you need to do in order to give some assessment feedback. It seems so much easier to give them marks like 7 out of 10 or 3 out of 10 but the 3 out of 10 or 7 out of 10 is not at all transparent because it doesn't tell students what exactly they got wrong and why. Usually when they have complex performances like open design problems or presentations. After you mark a few papers let's say 5 to at most 10, the time it takes to use rubrics we found was very similar to giving marks. Giving marks, numerical marks are easy when you have very defined steps but when you have these multiple solutions and open ended problems giving marks also takes time. So there was a short learning curve for us let's say the first 10 papers take time and then it becomes quicker. The second question which people usually ask and it is a very valid question is how consistent can we be with rubrics in the sense if there is so much subjectivity. See what one teacher gives is different from what another teacher gives. So what we found again is that when teachers are trained in using these rubrics and train means they use the rubrics to give scores to some student answers and the experts do the same and then we compare. So you can do it with a colleague. This iterative training takes I would say 2 to 3 sessions but after that there was a very high correlation between a traditional teacher giving marks and a teacher using rubrics to give these open ended feedback and the reliability of these rubrics was also high once a teacher got trained. Reliability means will this rubric give similar answers when different people use it. So it is called Interator Reliability. So after training Interator Reliability of rubrics was high and it correlated well with marks plus as we said it has the advantages of giving some specific feedback for students. There is one last point where here there is a debate should I use rubrics and convert it to marks later or should I just leave it as it is. So what the question is is that rubrics are very open ended and I cannot maintain a spreadsheet for all these open ended things. So how do I convert rubrics to marks? Educational researchers say it is not a good idea because the purpose of the rubric is to give very specific feedback to students over multiple criteria. But teachers still use them and if you must do it what you can do is the following and this is not to give students but just to maintain a log for yourselves. You can call this as 0, 1, 2, 3 if you would like let me get the pointer. If you want you can call this as a 0, this is a 1, this is a 2 and this is a 3 and maintain logs of how different students perform on different criteria and how it changes as time. Going to marks again defeats the purpose of seeing which criteria students are good at and which criteria they need improvement in but sometimes we do need to maintain a record and in that case you can give scores. We call them scores, we actually do not call them marks. We give scores to each of these and then that is how we track the students. So let us end this session here.