 Welcome back and I hope all of you had some time over tea to reflect on the presentation that you just heard. What Madhuri presented was a very good example of how you start with an idea. She started with her teacher's idea and in a step-by-step systematic manner, how she planned her strategy, how she developed her material, how she validated that her material indeed does make sense and then finally how she tested her material. So, you saw a full example of this process. These slides will be available on Moodle by tomorrow with the other slides of today's sessions and what we will also do, we saw this on the chat message a few moments ago. We will try to provide a glossary of some of the educational technology related terms, some of the jargon that she used. So, that will happen in a couple of days. The other thing which I would like to point out from Madhuri's presentation is how she used literature very effectively in the beginning to position her work and to show the novelty. If you notice she did not do, it was not a single time when she did the literature survey and said ok, I am done with my literature survey and now I will move on. She went back to literature multiple times wherever she needed to look, wherever she needed some evidence either for why her content is sound or what theories to base on or what other people have done and so on. So, this is a very effective use of literature spiraling to it multiple times and using it for the reason that you need. The other thing which she spent a lot of time on was how she validated her content and her instrument. So, these we will discuss in greater detail and from the research methods perspective in this session and in the next session today. So, we will continue the session on delving deeper into the research methods in educational technology and let us just quickly recall our goals as researchers that we want to conduct systematic studies to get data, to see how our ideas are working and we want to provide evidence to support our conclusions. The reason we want to do this is because we need to establish the soundness of our procedure and of our evaluation and we just saw one example of how to do so. So, what we want to do in this session is to look at these methods in greater detail and see which of those you can use for your own studies because at this point all of you are at the stage where you have your ideas and you are ready to in fact plan the research design and the technical and the study to implement your idea. The other thing we want to just recall is the fact that if you have a single group and if you only do one test at the end, the single group post test only, remember that we said it was not sound due to several reasons. Now in terms of the format of these slides, you will see some words written in blue and italics or purple and italics. These are the terms the technical jargon from research methods that we will start using and I will define them as we go along some of them you already have an intuitive idea about. So, the question that we saw at the beginning at the end of two sessions ago was how do we know that it was a treatment that is the strategy that is being implemented. How do we know it was a treatment that produce the outcome and nothing else? And we listed a few problems, two problems from there were that there is no comparison to a group that did not receive the treatment and the second problem could have been that the outcome could have existed even before the treatment happened. What we will try to do now is to suggest solutions for each of these problems and the title of the session is how to design an ET research study. So, let us take the first problem that there is no comparison to a group that did not receive the treatment if I use a single group post only research design and a potential solution is that have another group and compare the group that got my strategy with a group that did not get my strategy. Details about how do you know that the two groups are equivalent or how do we know what content to give each group will do in the next 20 minutes. So, you can hold on to your questions till then. The second problem was that the outcome the final outcome from the post test could have existed even before the treatment. So, in order to prevent that or in order to make sure that this situation did not happen what we can do is compare performance of the group before and after the treatment. So, we do not make a single measurement instead we make two measurements and then we check how much the result or the outcome has changed as a result of the treatment. So, in today's in this particular session we look at these two designs in greater detail. Before we do that let us just take a look at let us just summarize what we have that we do not want to do a single group post test only there may be a few times when it is ok, but at this point let us say that we will avoid such a design due to all the problems. Instead we can either do a two group post test or a single group pre and post or perhaps combine both of them. So, the next two examples you will be seeing are from here and from here. You already saw an example for the two group two group post test only design ok. So, let us look at the details of the two group post test only design that Madhuri showed an example about. So, some more technical words here that we have been using this word research design often and what do we exactly mean by research design? While planning your research study you will need to make several decisions and for example, how do you select the students or the subjects for your study? Now this is known as the sample of your study. How many groups to have and as one of you asked how do you make sure that the groups are similar? Which strategy or which treatment should I give to different groups to each group? Which data to collect? How data are analyzed? So, there are several of these questions for which you have to make decisions during various phases of planning your research study. You still have not begun executing it you are still at the planning stage. Now research design is what we mean by the overall scheme of the study it is an outline that helps you answer all these questions it gives structure to the study. The design tells us how these elements the sample the groups the strategy it tells us how all these elements relate to each other so that together you are able to answer the research question or the research objective that you have stated at the beginning. So, you can think of research design as your blueprint or an outline or a scheme which will help you answer all these questions ok. So now let us look at the first type of research design this is the two group post-test only design and the overview of this design is that we assign our students or we first make two groups and assign students into one of these two groups. So you have two groups with some number of students in each groups. We then implement our strategy which we call as a treatment to one of the groups and do not implement the strategy with the other group. Again these are the terms used for the two groups the group to which I implement my strategy we will call as the experimental group and the group which does not get my strategy we will call as the control group because it is acting as a control for all the other variables to the experimental group. There is a slightly different version of this which might work for all of you that is why I have written it right below it. You may have one group getting one strategy and the second group getting a slightly different strategy. So, this is this also falls under the design of two group post-test. The two strategies can be strategy one and strategy two or it can be strategy one and not my strategy one which is also some other strategy. So essentially the second the point here strategy one and strategy two is a more generalized version of give a strategy and do not give a strategy. In Madhuri's case she had her visualizations as strategy one which she called as a experimental group and she had plain PowerPoint slides as strategy two which she called as a control group. What we do then is after we implement the strategies we measure the same variable. If you recall she emphasized a few times that I am giving the same test after the strategy to both the groups and then we check if the outcome is different for the two groups. So this is your overview or the outline of a two group post-test only design. Now let us see why this is better than the single group and why this works when we should use it and so on. If you remember one of the problems with the single group only design was that the outcome in the single group design could have happened due to a reason other than the strategy that is being implemented or other than the treatment. Remember if you find in the two group design that the group which got the treatment performed well or better than the group which did not get the treatment. You have clear evidence that it was actually the treatment which did the difference. So what we are trying to do here is isolate the effect of the treatment and try to keep all the other the effect of all the other variables the same between the two groups. It is very similar to conducting a regular science or engineering experiment in the lab. So at the end of our study we want to be able to make a claim saying that it was due to the strategy of peer discussion that students conceptual understanding improved. We want to be able to attribute our result to the treatment and one way to do it is to have two groups with the design as we saw in the last slide. But I am already seeing some questions on the chat and all of you are thinking in the right manner that this is true or this will be true provided these two groups actually are the same along all other counts along all other measures. It is only the treatment which must be different for the two groups and everything else must be equivalent. So there are several things that we need to worry about even in the two group post stress design. It is not a magic scheme or magic process that is going to take care of all the problems. If we do not worry about these things the referee would complain that these are the problems in the design and your paper is going to get rejected. So before the referee points out these things let us worry about them. So the main thing one of the first things really is are these two groups equivalent? What we mean by equivalent is are all conditions for the two groups same apart from the treatment of my strategy. Let us see what these conditions could be. For example the average achievement level of the two groups must be equivalent. Madhuri cautioned us against having one group which had very high grades in the previous semester. It might be their ability or their high achievement levels which might cause a difference so we do not want to go there. We want to make sure that the two groups have on an average the same achievement level for the subject. We want to make sure that their motivation to learn is the same. These are quite difficult. I mean I have not yet told you how we do all of this but the point is we have to worry about all of this. We want to also make sure of other attribute variables like the on an average the age groups of the two groups must be similar and this is not a problem if you are doing it in the second year engineering among the second year engineering students. Roughly speaking you will have the same age groups but let us say you are doing some other study where you had some children in one and much older people in the other then the equivalence of the two groups would not be established in terms of age. You want to make sure that the gender distribution in the two groups are equivalent on an average so roughly same distribution of males and females in one group as in the other group and there are several other such counts on which the two groups have to be established that the two groups have to be equivalent. At this point these are only things that you know we have to caution ourselves against the next slide will tell us how to do it. Some other things we have to worry about are do I have large enough numbers in my study and some of one of you asked this question last time how large is large enough. Again I would say that if you have numbers large enough so that your data analysis techniques are valid and so that you can show equivalence. When you have large numbers then you can say that the average performance in one is similar to the average performance in the other. When you have small numbers averaging causes a lot of problem so large numbers are important to show equivalence and they are also important for some statistical tests. Some other things you have to worry about how do we told us a lot about this is the strategy that I implemented valid which means is it consistent with what was intended. Her example was that the visualization should teach us design skill and should not teach the students problem solving skill because her research objective was to teach design skill. And finally in the next session we will see something about the test itself how you ensure whether the post test measures what it really should. So, let us go through some of these one by one and let us look at equivalent equivalence and we will see how to ensure equivalence. So, there are several methods and I would say these are from strong to weak. So, the one at the top is the strongest method. If you are able to randomly assign students to each group it is considered to be a strong method to establish equivalence. So, what do we mean by random and how does it ensure equivalence? Random means that every student has an equal chance of being in one group or the other and every group has an equal chance of either getting the strategy or not getting the strategy. So, there is a random assignment of students to groups and random assignment of strategies to groups. What randomization does is because the probability of being in one group or the other is the same if you take an average of say the performance of each student and average it over the entire group it will turn out that the averages are roughly the same provided you have large numbers. However, it might happen many times that you are not able to do the random assignment. So, for example, let us say you teach one you teach two sections of a course in your college and the only way you can implement two groups is if you give one group one section treatment and if you give the other section a different treatment. So, here there is no possibility of mixing up and doing random assignment. What you need to do in that case is make sure that the two groups are matched and you have to check the equivalence. So, you try to place the student into one group or the other and then check that the average of the performance of each group is the same the average number of males and females is the same and so on. But you have to show some evidence in your study that the groups are equivalent. You have to write a whole paragraph about how you establish the equivalence using either of these methods. Is the strategy implemented for each group valid? So, typically what is done here similar to what Madhuri did that first you enter that the content or the treatment of the strategy is addressing your research objectives and then you validate with other peers or experts who may be teaching the same subject. So, this goes by the name of content validity. You check with a peer or an expert get their feedback revise your material and then repeat this process until you are sure that the content is valid and it has passed the check of a few people and there are no more suggestions that are made. We talked a lot about the random assignment why this is desirable. So, I will just pause at the slide you can take a few minutes ago. And here is simply a list it is not a glossary yet, but here is a list of the new terms that we introduced right now and you are going to have to start using these terms. Okay, there is a lot of questions on size. So, let us say try not to go below 30 and 30 that is a rule of thumb. Of course now you will start asking me questions like what if I have 28 is that okay and so on. So a rule of thumb is try not to go below 30 and 30 if you can do 60 and 60 it is great anything more is not a problem. If you have 5 and 7 in the two groups it is a problem. We were discussing is that the takeaway from this entire discussion was that if you need to for a strong research study you need to have two groups try your best to randomly assign students to the two groups else try to match and check equivalence and make sure that your strategy is valid. How do you ensure that the strategy is valid? One possible method is to take your research objectives and your content and show it to appear such as another teacher who is teaching the same course or an expert content domain expert and ask them whether the strategy actually addresses the research objective. Some of these steps in the process will be present in the slides of the previous session the example and that will be posted. How many experts should you ask? Well ask them till the next expert does not suggest any more changes. So it might be possible that the first peer you show to suggest six changes you might go make those six changes in your content and you show it to the show it to another expert and they might now suggest only two changes. So you know that something is working in your validation process. It takes a few rounds sometimes two sometimes five few rounds of checking with experts or peers to make sure that the content is in fact valid. So let us now quickly move on to another type of research design and there are times when you would not be able to do a two group post-test design or there may be some times where one group in fact is not a bad idea. But in those cases what is very important is that you have a measurement before you implement the strategy and that is what we mean by the pre-test. So let us look at that design. The goal or the way this design works is that you are comparing the performance of a group to itself before the treatment and after the treatment. If you recall the problems with a single group post-test only the main problem was there was no comparison. In research design one we saw a comparison with a different group and in research design two we will see a comparison with the same group itself but at a different time. So what we do here is measure the variable of interest before implementing the strategy we call that the pre-test. Then we administer the treatment that is we implement our strategy and then we measure the variable the same variable after implementing the strategy we call it the post-test. What we check is if there is a difference between the pre and the post if there is a difference we have to argue that it is due to the treatment. Of course there are several things we have to worry about to attribute it to the treatment. So I will leave the slide on for a moment take a look at it carefully and then we will look at the things that we have to worry about. Let us see when we should do a pre-test then when is a pre-test really needed as some of you have been mentioning it is possible that students come to us with a variety of prior knowledge and prior backgrounds. There may be a large difference a variation in their achievement levels in terms of maybe their maths capability or maybe in terms of their linguistic capabilities or there may be a difference in terms of the depth to which they have learnt a particular topic the previous year and so on. So essentially when you have a large variation in students prior knowledge we do not want that variation to become the cause of the result. So we want to remove the effects of the variation in students prior knowledge and we are talking of the prior knowledge that is of importance to our research objective. So to give you an example let us say we are trying to develop a treatment to improve students programming knowledge and abilities and I take this example because the next presentation is going to be on first year programming skills. Now students come to us with a variety of programming backgrounds some have taken C++ in their 12th standard and some have used a computer only for email. The treatment that I give to improve students programming skills could be largely influenced by which type of background students come from. In that case we want a pre-test to see to actually measure the difference in the students prior knowledge of programming and then we will subtract the pre and post tests we will only concentrate on the difference between them and the difference will take care of the fact that their prior knowledge is so varied. Note that we are mainly concentrating on students prior knowledge related to programming because our research objective is to help them improve their programming skills. So the pre-test and the post test should be consistent or should address the research objective similarly when you are trying to see what other variables can affect my study you have to have the research objective in mind things to worry about in a pre-post test design. So the biggest problem that might happen here is even though there is a pre-test and there is a treatment that comes after that something else might be happening between the pre and post test other than the treatment you have to argue that it was only the treatment that happened between the two and there really was not anything else which could have caused the difference between the pre and post other than the treatment. One way of doing it is to have not too much time between the pre and post test. So let us say your study is of the duration of three hours it might actually work if you conduct the pre-test do the study and finish off the post test that same day rather than waiting for the final exam to do the post test. So don't wait too long after the treatment to do the post test that is one way to try to make sure that it was nothing else that came in the way and you have to do a lot of other argument also here. Finally as we saw last time is the strategy implemented valid is something that we already discussed and it is the same idea that we discussed a few minutes ago. There is a good question here which we will postpone to a little later that how do we ensure that the pre-test itself does not influence the results of the post test. So you are asking a good question here it is a difficult question to answer we will take this question say after lunch during a session after lunch we will note this question down this is at 120226 seconds. Take away from here use a pre-test when there is a when you know that there is a variation in students prior knowledge with respect to the research objective ensure equivalence between the pre and post test and do not spend too much do not have too much time between the two tests to make sure that there is really nothing else in between which can cause the effect. So what we will do next is look at an example of a single group pre and post design. So I would like to introduce our colleague Anita Devakar.