 So, let us actually continue the same theme. We saw another yet another example of how to set up a research study. And there are a few things I would like to point out from that study right now. One is the interesting way in which Anita made use of the pretest. In the previous session I had mentioned that pretest is required when you want to try to control the effect of the prior knowledge of the students in the study. But what she has done in fact, is gone one step further. She is used that the pretest course itself as a variable in her study, which means she wants to focus on low performers versus high performers. So, in case you have a situation where you want to try to study these two sets of students in a different manner, that is one more place where a pretest would be useful. Of course, once you do the pretest and the post test you still have to look at the differences. But she is not only using the pretest as a control, she is also using it as a variable in the study itself. What we need to make sure of is that any variable that we do not want to play, we do not want coming in the way of our research study, stays either it stays out of the way by doing a control or we make it into a variable within our study itself. So, we know that lots of variables can affect our study. What do we do about all of it? We try to make sure that other than the treatment the others do not play much of a role. That is what we mean by control. Or we say that well let us include one more variable in our study and since we know that the variable has a different effect for different values, let us just study what it does at the different values. So, that is the kind of design that Anita has used. Couple of comments about the questions, we will post all the slides and the videos and the supporting materials. So, you do not have to keep reminding us or you do not have to keep sending chat messages. It is ok if you do so it will just clog up the chat window and by the end of the day or at least by tomorrow we are going to make sure that we post everything. If you are not able to see something it will be there on Moodle so that you can download it and view it later. So, if you remember our research design from the previous session, we said that there are a number of decisions that we have to make starting from how do you select the sample, how do you make the group assignment if there are different groups with strategy to give and so on. So, one of the important points as we are going along that list is what data to collect and how to collect it. So, far you have seen two examples and now let us try to formalize this a little bit more. The first question you have to ask is what is the goal of my study? What is it that I actually want to try to determine by doing this study? And typically your goals will fall into one of these two categories there may be some others, but two important categories are you may want to measure whether the learning effectiveness of your material or your strategy that how well have students done in terms of a performance on a test or in terms of some performance related to the learning of the topic or learning of the skills. The other category of items or metrics you would you might be interested in is whether students are engaged. So, what falls under engagement is their motivation, their interest, their satisfaction, the perception of their the perception of the course of the strategy that they underwent and so on. So, you have to decide what is the goal and see if you fit into one or the other category and most of you I think will fit into one of these two, because what we will talk about next in terms of which data to collect and how to collect it will depend on which category you belong to. So, now let us look at the first one learning effectiveness typically again if you want to measure learning effectiveness what you need to measure is performance on some test and we will see what we mean by test related to the central idea or the concept in the study. This test could be a paper and pencil test, it could be a performance test, it could be many things we will look at the specific examples down below, but what you want to see is their performance on some some test and the test has to be consistent with the goal that you have started with, which means if you want to measure the domain understanding of convolution your test has to have concepts on convolution. If you want to measure their acquisition of design skills then this test should measure students performance in terms of design skills. The next question we need to answer is how do we measure this? So, we know what we want to measure how do we measure it and just like we do in engineering and science what we need is an instrument or a tool that measures what we want to measure. So, an instrument or a tool is essentially a measurement instrument, it may look different in engineering your instrument may be an ammeter or it may be a thermometer or it may be some digital oscilloscope. In educational research an instrument may be a questionnaire, but the purpose that is being served by both these instruments are that it is measuring what your goal is. So, there are several ways of using several different types of instruments and ways of doing this measurement and let us look at this from maybe I would say ok let us just go through the bullets. So, one easy thing to do is if there already exists a standardized instrument that can measure what you want to measure use that find it and use that. What we mean by a standardized instrument is let us say you want to measure students understanding in convolution. In your literature survey you should be able to find out that there exists something called a concept inventory for signals and systems. A concept inventory is a set of about 20 or 30 questions that is targeting a specific concept and somebody has already created this test and they have validated it and they have made sure that what they have in front of them is a test is an instrument which is robust, it is reliable and valid and we will come to these words in a moment. But just like you want to deal with an ammeter that measures the same thing whether you measure the current today or 5 days later. Similarly a standardized test should be the results from the standardized test should be replicable. You may find that there exist tests for specific abilities. In an example we saw last Saturday there was a test to measure the spatial ability of mental rotation. So, it is a standardized test because somebody has created and validated this test. So, if you want to measure spatial ability simply use the same standardized test. You may find rubrics the type that Madhuri told you a couple of in the session earlier that somebody has already developed and published. So, these are instruments and tests which have already been published and you will be able to find this out from literature. The advantage of using a standardized test is that you can go ahead and use it without inventing it on your own or without having to validate them too much all over again and so on. It is like each time you want to measure current you would not start creating or inventing or putting together your own ammeter. You find a standardized ammeter make sure that it measures the range of current that you want to measure and you connect it in your circuit. So, that is the analogy here that slightly challenging part might be to try to find some of these and literature will help you in this regard. If you are continuing in terms of paper writing our mentors will be able to help you to try to find a standardized test if you if they know what your goal is ok. What to do if you do not have a standardized test that exactly addresses your goal. What you do in that case is that you will have to create an instrument and very often you will be in this category. Anita showed you an example of creating her own instrument to measure some specific aspect of programming. So, an instrument again is similar to a test that you are familiar with as teachers. It may have conceptual questions or it may have problems, but the questions and the problems and the various parts of the instrument are very specific to the research objective that you are trying to address in your study. So, final exam may not be the best instrument to use in your research study because it may have questions that address several different topics or several different objectives that are not part of your research study. So, one tip or guideline here is that once you know that you want to measure something very specific such as concepts on a particular topic you can think of creating a small number of questions by small I mean anywhere from say 5 to 20 or 5 to 15 that specifically target that question that research objective and this point is something that is important throughout that what your measurement instrument must measure what is relevant. So, if you are trying to measure programming skills you can try to analyze the number of errors that are there in the program. So, you can write down a you can create an instrument where you ask students to write a program and what you can analyze is the number of errors in that program. So, the measurement as well as the analysis must be consistent again with the research objective. Given all this marks in a regular or quiz are fine if they address the research objective but if you are stuck with your university final exam it might be a weak measurement instrument for your research study. If you have control to tweak the quiz you give so that it is consistent with your research objective in fact, you can go ahead that is the same as creating an instrument that we talked about earlier, but if you say that I am going to use the same university final exam that all other students are going to get and use it for my research study you have to be careful to make sure that in fact it does address the learning effectiveness of the type you want to measure. Let us go on to the next category, how do you measure student engagement? So, what you can measure here is students perception of their own learning. In fact, students perception of their own learning you can either call it as you can either put it in the category of learning effectiveness or student engagement because it is a student perception but it is of their learning so this fits in both. But what fits completely in student engagement is a student satisfaction, their interest in the course format or course topic. You can also for example, if you are doing a study that runs for an entire course or entire semester you can look at the attendance rates and see if there you can find some pattern in terms of the attendance rates. How do you measure student engagement? Usually they are measured by using questionnaires that measure perception or satisfaction or interest. Again, if you find a standardized instrument that measures student satisfaction in a specific course you can use that instrument otherwise you have to create this questionnaire. Sometimes you also can measure using carefully structured interviews and a carefully structured interview is not simply a conversation. There are guidelines, there are very strict ways of creating the interview questions of implementing the interview and so on. So for this again those of you who stay on beyond the final assignment and if our mentors feel that carefully structured interview might be the best instrument they will give you guidelines for this. Some do's and don'ts for creation of an instrument and this instrument is the questionnaire to measure satisfaction or interest a lot of you will fall in this category. It is not a good idea to ask do you like it or dislike it because that question is too broad it won't tell you anything specific. Instead I would suggest that ask questions that are very specific and related to what you want to measure. So if you want to measure satisfaction ask a bunch of questions related to satisfaction. Similarly asking a single question on is this method interesting is not useful. Ask many questions related to what you want to measure. So for example in this workshop we do want to find out your satisfaction levels. What we are going to ask you is specific questions related to how useful the voting activities were. How interesting was it to discuss with your partner. So we are going to break up the different parts of the workshop and ask you specific questions and ask you whether you were whether it was useful or whether it was interesting to do that particular activity. And we are going to ask you more than a single question maybe three or four questions related to a single idea that you want to find out the answers to. It is possible but difficult to analyze open descriptive questions. So suppose I ask you to write why you like the workshop. You will write a paragraph it is not this is not a bad thing to do all I am saying here is that the analysis is hard. So instead it will be easy for you to analyze if you either use a scale or a ranking scale or a rating scale. So what I might want to ask you is rate on a scale of 1 through 5 how you useful you found the polling activity in the research methods workshop or I can ask you rank which of the following activities in the workshop you found the most interesting and which you found the least interesting and I can give you four activities ask you to rank it. So either a scale either a rating or a ranking scale is it is going to make your analysis much easier. So the takeaway from what to measure and how to measure is you need to follow guidelines to systematically create the instrument that is takeaway number one we are going to come to a few other takeaways in a few minutes. How exactly to do it and what these guidelines are we will post links to some readings and you will get a lot more detailed feedback if you in fact finish the final assignment and are one among the top 200 and you stay on for the rest of the course. So if you start actually planning your paper our mentors will be able to help you with this part but we will post some general readings on Moodle in any case. Let us come to another point another important concept that the previous presenters have mentioned a few times. There are two questions that we have to ask after we create the instrument and even if you borrow somebody else's standardized instrument you still have to ask the same questions. One question is is the test or questionnaire accurate which means is it actually measuring what it is supposed to measure. To give you an analogy from circuits if you want to measure current let us say you want to measure voltage you should not put an ammeter there it is a wrong instrument to measure what you are supposed to measure even though they both operate on the same principle. So does your instrument measure what it is supposed to measure if yes if you are able to show you have to actually show this you have to prove this. If yes then what we say is that your instrument is valid. So validity is the technical term which whose meaning is does the instrument measure what it is supposed to measure. The second question you need to ask again this will be familiar to you from engineering is your test or questionnaire is your instrument precise and will it give the same results if you operate it under the same conditions. So there is a notion of replicability here of reliability and the term exactly is reliable. So if your instrument gives similar results under similar conditions then the instrument is actually reliable and a robust instrument is one where you can show both validity and reliability. So let us actually explore this notion of validity and reliability a little bit more because the notions exist beyond instruments and I will just walk you through a bunch of pictures here intuitive pictures. So this is a dartboard a picture of a dartboard I hope you can see the red little red dots what you are doing is shooting arrows or darts and where you want to get at is the center. So this is what you want to target right at the center and you throw a bunch of darts and you find two things one is that they are all away from the center firstly. So they are not measuring or they are not addressing what they are supposed to address which means that it is not valid. Secondly there is a large spread one of them reached down here the other reached up here the third one was to the right and so on. So even though the conditions were similar it was your hand that threw the dart it was the same dart or a similar dart same dart board same session of play and so on but each dart landed in a fairly different place. So there is no reliability in this example. In the next example you are again throwing the same darts while they are going all over the place you know one is down here one is up here. So there is no reliability on the other hand you can see that compared to the previous version they are all somehow on an average aiming at the center. So it looks like this person is getting in some average way is still trying to hit the center compared to this example. So this example is considered to be valid it is still approximately the center but it is very far away but it is very distributed. So it is not reliable and some of you are talking about accuracy versus reliability that is exactly the point we are trying to get at validity is accuracy and reliability is the same as what you call as reliability. A third situation is when all the attempts you make in fact are very close to each other. So the instrument is precise and reliable but it is far away from the center. So such an example is reliable but not valid and the fourth example that you will see is both reliable and valid because all the attempts you are making are close to each other. So it is reliable and it is also really getting close to the center where you want to get at. So the notion of validity and reliability that you intuitively understand through these pictures is what you have to try to establish for your instrument. Let us go to a takeaway here you need to show that your instrument is both valid and reliable. How to do so is going to come in the moodle part of the workshop because there are some statistical techniques to do so there are some other techniques such as the content analysis that we talked about in the previous session that is one way to show validity where you go and ask domain experts. So there are several different measures that you need to perform in order to prove validity and reliability and some of the links will be posted on moodle the others will be given by our mentors when you start writing your papers. The next concept we want to talk about is again we are talking about data collection and measurement is how to strengthen your study claims. That means let us say you want to try to find out how whether your strategy helped students learn effectively what we want need to do is triangulate our data which means we need to use more or two or more data collection methods. And this notion again comes in fact the historically it comes from sailors at sea and when they wanted to locate or try to find out some location that they would reach in a few days they would use the stars but they would use more than one measurement there is at least two measurements to try to reach the same point. So triangulation is when you use both more than one method to get to this to try to arrive at the same claim and the same conclusion and if you do so your study claims are going to be stronger because the multiple methods are being used to establish the same claim or the same conclusion that is the reason why you should be thinking in terms of triangulation. What so now the question is how do you triangulate you know what kind what do we mean by toward more data collection methods. So again I will show you one example here what you need to do is multiple data sources. So you can have a test of conceptual understanding it can be a standardized test and you can combine it with a small survey few questions on student perception of learning. This survey of student perception of learning is should be constructed based on the dos and don'ts slide that we saw earlier or let us say you only want to measure student engagement you want to measure if students were satisfied with the strategy that you used. What you can have is a quantitative questionnaire which you again construct using careful guidelines. The quantitative questionnaire will help you understand the what of your question which means it will help you understand whether students were satisfied or not. But to be able to understand why students were satisfied you have to follow it up with a few qualitative interviews. You saw an example like this in the presentation on design skills where the numerical quantitative rubrics were used to measure whether students learn design skills. But why they learnt what they learnt were this was determined by interviews. So in your study again try to think in terms of at least two data sources do one very well. So just because you need I am saying you should do two you should know you should you should still make sure that all the guidelines for reliability and validity and careful construction of the instrument they should still hold. But if you are able to have one main method and a supporting method your conclusion study conclusions will be stronger. So the evidence here is that the take away here is that use two or more data collection methods to establish evidence. So if you this is just summarizing the other ET jargon that we talked about I think the main two new ones we saw in this session are validity and reliability and we saw precise definitions of each of these. If we look at if we go back and look at our research design slide we saw that we have to make all these decisions. We have talked about a lot of these. So how to select the sample, how many groups, what strategies, what measures is something we just talked about. What we will not be talking too much about are how data are analyzed because what happens here is that every study is going to have its own different analysis technique. We will briefly talk about this in one of the afternoon sessions that you have to show some descriptive measures of analysis and some other measures that show that the two groups are different. So some inferential measures also have to be analyzed. This slide really gives you a full outline of what all you are supposed to do when you implement your study. When we come back we will see one final example of a research study that takes all this into account. In fact that follows a two group pre-post test design. So it will draw from here, it will draw from here, it will draw from all of this and you will see a detailed example with the technical jargon. So we will come back at 5 minutes to 2 o'clock, it is we are breaking 5 minutes before 1. But 1.55 will we can mean back for after lunch.