 Good afternoon to everybody and welcome back to the research methods in educational technology workshop. I think that with that we should actually get started with the with the technical part of the workshop. So, if we all want to become ET researchers, one of the first things we have to learn is what is an ET research paper, what is in it, what should not be in it and so on. And I do not only mean the structure of the paper, but what should be the features in it, what do referees look for and so on. So, to explore this question more, what we will do here is, so we will look at a few examples and I would like all of you to vote by raising your hand, your real hand, not your virtual hand, inside your classrooms, inside your centres and the centre coordinator here can facilitate the vote and once the majority vote of the centre is known, the centre coordinator could please just reply to us through chat whether your answer from the centre is a yes or a no. So, we will do one trial first, please read what is there in the italics here. This is an example of a research paper, of a research, it is an example of a research idea and what you need to do is vote whether this looks like a research paper or not. Please take your time, do not start voting, this is, please take your time and is this really a research paper or is this a teacher, a faculty member doing something. So, take a minute, please read it and raise your hand in, so that your coordinator can see what is going on. Let us move on, what I see in the chat window is that there were a lot of yeses to begin with and then there are a few noes which are going up. The question is, is the paragraph written here, is it really a research paper or not and the correct answer is that no, it is not a research paper. So, all of you who voted yes, please stop voting for this particular question now, we will do some discussion of this question right now. So, all of you who think that this is a research paper, I would urge you very strongly to go back and read this. What is being written in this paragraph is an instructor who uses colored chalk or markers in order to draw pictures and waveforms. The instructor thinks that the usage of such methods will make the lecture clear to students. The instructor is correct in saying that the usage of such methods might make the lecture clear to students. However, a solution like this is tried by many instructors, it looks like an obvious or a known solution and simply compiling known solutions is not a research study, it will not help you write a research paper, even though the idea might have value as a teaching strategy. So, in your practice as a faculty member, you may have tried several effective solutions to improve teaching, but not all of them can be converted to a research study and what is required is that the solution is something unique and it is novel. If it is not a novel solution, if it is an obvious solution like the one we just saw, then it is very difficult to convert it into a research study. So, let us move on to the next example and again I would like you to read it first, think whether it is really a research study or is it a teacher thinking about the different tasks that he or she would do in the class. This is example 2, please take a moment to read it. All participants in all centers including IIT Bombay can just raise their hand to say whether they think the answer is 1 or 2. So, that the center coordinator can see what their answer is. Please stop voting at this point, let us discuss this example. Again, I am seeing a mixes of yeses and noes and there are two things I want to mention here. Firstly, the correct answer is still a no, this is not a research paper. The second point I want to mention is those of you who are voting as soon as you see the slide, you are getting it wrong. Those of you who are waiting to think and then convey the answer on the chat, there is a higher likelihood that your answer is indeed correct. So, for this example and for the few examples I again request you to take your time, read, think and only then vote. So, please stop voting on this question, let us discuss why this is not a research study. So, let us see what is happening in this idea. Again this is an instructor who has an idea which is slightly more interesting this time because he or she is explaining the importance of the topic and its practical applications to industry. This instructor is also trying to connect the topic to global and local current scenarios that exist in industry and exist in the real world. So, there is an idea here. On the other hand, as one of you mentioned in the chat, I only saw one answer which said that this only looks like a report and you are right this is a report of the strategy you implemented, it is not a research paper even if it contains a good idea. To be considered as an acceptable research paper, what you need to show is details of why your strategy is unique, how it can solve the problem and very importantly you need to establish evidence that the idea works saying that my students are happy, my students look like they are learning is not considered as evidence in a research paper. So, to summarize again for if you write a report of what you did and say that my students seem happy, it is not a research paper. So, let us move on to the third example. Again please do not vote for at least one minute after you see the example and only then you can start voting. The third example is related to the use of moodle. So, please stop voting, let us look at what is going on in this idea. Some of you may be surprised, but this particular idea that you saw the one with moodle is still not a research paper. So, those of you who think that this is a research paper, please see why we do not consider this as a research paper and even though it looks like the instructor is trying to use educational technology tool, what the instructor is doing here is using the educational technology tool in a very routine manner mostly for administrative purposes. So, use of an ET tool in a routine manner is not considered to be a research paper. In order to be considered as an acceptable research paper, what you need to do is implement an innovative method of using this tool and the innovative method must address some teaching learning goal. So, saying that you will use moodle to upload assignments and give quizzes will not be considered as a research paper and what I like all of you to do at this point is a slightly different activity. For the same idea that you saw a moment ago, what I like the participants to do is talk to your neighbor and come up with one innovative way in which you can use moodle for teaching or learning. A single innovative method that each pair can think of and share with their coordinator. Let us stop there for a moment. We saw that this is not a research study because the use of moodle is done in an extremely routine manner. So, how do you convert this to a research study? Think of one innovative idea that you can implement with moodle. At this moment, all of you should be discussing about an idea to convert this into a research study. We do not as we had mentioned earlier, we do not want all of you to sit passively and listen, nobody is going to give you an answer here. So, this is the time we can see what you are what is happening in your centers. So, this is the time where you have to talk to each other and share your ideas with each other. I see we start seeing one idea here, two ideas have started to come in please go on. Let me read out some of the interesting ideas that have started coming in from the centers. The question was how to use moodle in a more innovative manner. Some of the ideas that came in were that spoken tutorials can be used and uploaded on moodle and students can interact with the spoken tutorials on moodle to achieve some teaching learning goal that something perhaps you could consider to as a research study, as an idea for a research study, a couple of more people have written about using the chat forum in moodle to ask something, to have students interact about something very specific. Again, that can form the basis of an idea if you structure the question properly. Some of you have said that different forums can be used to for students to work on different projects within moodle that is something you can do. So, fine. It looks like in most of the centers you do have ideas on how to use moodle in an effective manner. The point we are trying to make here is what we said earlier that in order to be considered as a acceptable research paper, you need to use an educational technology tool in a more innovative way than for routine tasks. So, let us go on to the next activity and I am glad that there is so much active participation, but we will have to move on to the next mini activity. You can keep discussing these ideas during the tea break and the lunch break and next week and also on moodle during the coming week. Okay, here is the final such question. This here, the person is using moodle and interactive multimedia content and read it again think whether the idea as it stands can be considered as a research study or not. Take your time, think about it and only then please vote, do not start voting before you read it. Okay, it looks like there is a lot more knows this time. So, it seems like you have got the hang of this game. The answer is not always no, but for this question it is still a no except that we have said that the answer is a not yet as opposed to an outright no because the idea that is there in this paragraph does seem to have some basis to be converted into a research study. The person who is thought of this idea as considering animated videos to teach a certain concept they are also thinking about how to get the students more interested in interactive and so on. But what it looks like is that what is being done here is mere development of instructional material and if you only think of developing instructional material it is not a research paper even if the material has an interesting and innovative idea. To be considered as an acceptable research paper what needs to be shown what you need to do is show that the material has resulted in some improvement in terms of student learning or engagement. So, you need to connect the development of the material to some results to some evaluation that there is indeed an improvement in student learning. For example, this is the same example that is on this page. What you can do two examples of what you can do is written at the bottom of the page. You need to show evidence that the material has resulted in improvement in student learning or engagement. You can give a quiz to test students understanding on a concept learnt through multimedia and compare it with their understanding on a similar concept through a traditional textbook. So, you need to think of some such evaluation mechanism for the instructional material. So, let us just recap and summarize and these points are extremely important that is why we are repeating it over and over again. Compiling existing or obvious solutions like I am going to use colored chalk is not going to become a research paper. Reporting the strategy you implemented is not a research paper. Using an ET tool or any software tool in a routine manner is not cannot be considered as a research paper. And finally, mere development of instructional material is not an ET research paper. So, the important question in front of us at this point is what is a research paper? So far we have seen what cannot be considered as a research paper. We saw many examples and even though there was some potential there was some idea in some of the examples, none of them yet qualifies as a research paper. So, the two questions we want to explore now are what is a research paper and how do we convert this potential in the idea into an educational technology research study. Essentially, how do we take our ideas as teachers and practitioners and then convert it into an educational technology research idea. So, in order to do that what we will first do is examine some good papers, some strong papers that have already been published in educational technology conferences and looking at these papers, examining them and analyzing the features that are there in them would give us some idea of what is a research paper. Again, there is a lot of reading for you to do. So, I will pause at each slide, let you read it and then go on to the next activity. Let us look at some different types of educational technology research papers. And these are papers related research studies related to innovative classroom or lab strategies that have been implemented to improve student learning. There is a longish paragraph here I would like you to spend some time and read this. The title of the paper is how we teach impact, how we teach impact student learning and the authors have compared two methods. One is called peer instruction or peer discussion, second is a lecture method and this is in a programming course. What exactly has been done is present in this paragraph. So, please spend a couple of minutes to read this. Let us try to analyze this paragraph in detail. So, I am going to use the zoom feature so that you can actually read it, but at your end also please make sure that the screen is set to full size, you do not need to see the image of the instructor, you can you only need the slide at this point. If you look at the first sentence, the authors say that we look at the impact on student learning of the pedagogical approach. So, the problem has been described in a very precise manner. There is an impact of the pedagogical approach on student learning and that is what they are trying to study in this paper. Secondly, if you look at how they have done it, they are using a technique called peer instruction and that is novel because it is something different than the traditional lecture approach that one uses. So, this particular research study also has the feature that the solution tried, the idea tried to improve teaching and learning has a novel approach within it and these are the features that you want to try to connect to your own idea. We will come to that later, but as we are looking at the features in these bubbles in these callouts, start thinking about how you can make your own idea novel and how you can precisely define your own idea and so on. Let us look at some other features of this particular study. If you read the second sentence, it says we compare two sections of a non-majors programming course offered in the same term by the same instructor covering the same content and utilizing the same book labs and exams. So, what these people have tried very hard is to keep the conditions of the experimentation as similar as possible and the only thing they have changed is the solution approach the new idea of peer instruction in one of the classes because if you think a scientist and if you are performing an experiment, you want to keep, you want to control for the conditions that are not part of the experiment. You only want to vary one variable at a time. That is why these authors have implemented a very sound procedure by trying to keep all the other variables not related to the study constant and have only varied whether there is a lecture or whether there is a peer instruction method. So, in your research studies again you have to think of a procedure which is which is sound and is implemented systematically. So, this is just putting all the different features of this particular idea. There was a novel solution approach, there was a precise problem definition, there was a sound procedure. There is one more at the bottom here. The last sentence says that we find that students in the peer instruction section score an average of 5.7 percent higher than in standard lecture practices in the final exam. So, the researchers have evaluated their solution, they have evaluated the two different conditions the peer instruction section and the lecture section on the final exam. And from the scores they have done an analysis, they have also told us what sort of analysis they have done. So, this research study has evaluated the solution again in a very sound manner. So, I hope that gives you some idea of what all needs to be present in a research study. And let us look at one more example to make this idea more clear. There are several other examples in this slide. I am only going to show one more example in this session. But the slides will be uploaded in Moodle and part of your homework assignment for this week is to go through the slides that we did not discuss. For example, this is an example that I will not discuss. Computerized Molecular Modeling in Chemistry. It is a very interesting study here because the authors have used three-dimensional simulations and they have looked at what is what kind of learning happens. But instead let us move on to another example and this example is of a slightly different category and that is why I wanted to discuss it here. In your in this workshop or as part of your faculty, I mean as part of your faculty class, one of the things you might be doing is developing instructional material. We saw a few examples so far. Now if there is some innovative component of the instructional material and if the evaluation of it is done in a systematic manner, you can think of considering it as a research study. So let us look at one example. Again I will put the slide up so that you can read and at your end make sure it is full screen and zoomed in and so on. Okay, let us see what is happening in this example. Again if you read the very first sentence, I am going to zoom the sentence so that all of you can read it. The first sentence says mental rotation ability is important in various fields ranging from engineering to architecture to education to technology. Which means right away the authors have established the importance of the problem. The title is the improvement of mental rotation ability using Blender 3D. This was published in T40 2012. So apart from defining the problem precisely, another feature, another aspect that needs to be there in a research study is establishing why it is important. The next point we want to see in this study is something we will come to again in the following session. But this is one place where you can see it very well. Let me read out the sentence. The authors say that most existing techniques to improve mental rotation ability require weeks of training and are based on proprietary software. So the authors have not only looked at related work, they have also analyzed what is missing, what are the gaps in the related work. Which is exactly what you need to be doing for your idea when you are converting it into a research study. First we have to analyze related work. There is a whole session we have on it today afternoon. But this is one quick example where the authors in one paragraph have not only said that there are other existing techniques, but they have clearly pointed out that they are very time consuming. That is one gap and the other gap is that they are based on proprietary software. If you read the next sentence, the authors say that their program is only 3 hours long and they have used open source software. So their solution is directly addressing the gaps. Their solution approach is also novel because they have used Blender which is a 3D game engine. It is mostly used for entertainment and for making 3D movies. But there are not that many studies that have used Blender to create material for education and that is why it is considered as a novel solution approach. Let us look at the last sentence here or towards the end. The authors say that we use Vandenberg's mental rotation test for pre and post test and we see a large effect size. So again this idea of evaluation of solution is very strongly present here. They have used an already existing test, they have done it in a pre and post, they have a sample, they have described who are their participant, who are the subjects, who is the sample. So there is a sound evaluation scheme of their idea. They did not stop at simply saying we develop the material. They also went ahead and used this test as a pre and post test and have evidence to support the solution because they see what is called as a large effect size. So again to put all of this together, some good features in published research papers are analyzing gaps, importance of problem, novel approach, sound evaluation and so on. There are some other examples. I will simply pause at this slide so that you can read it and get an idea of the different types of published research papers, different examples, different ideas that people have tried and all these have been written by faculty members who are also now doing research in educational technology, some by PhD students. The previous example I showed was by a PhD student. There are a few questions on chat asking what is mental rotation and that is a good question because since I did not have much space on the slide, I removed the first sentence which said what was mental rotation and I see that that was something that one should not do because many of you are asking what is mental rotation. So when you write about your own problem, you have to start with the definition of the problem. Mental rotation just to give you a quick answer, mental rotation is a spatial ability is the ability of a person to be able to take a 3D image without having a real life object, rotate it mentally and try to imagine what is there on the back. This is a very difficult ability to do mentally and if one has the physical 3 dimensional object it is easy without it many of us find it difficult. For example, if you have a pyramid and want to see what is on the back of the pyramid or if you have a tetrahedron which is unequal, which is not a regular polyhedron or tetrahedron then one needs to be able to rotate it and see what is on the back. So what is being done in this study is the author, the researcher has developed a 3D model of the object so that students can see what is, can rotate this object in the virtual software and see what is going on. Okay, let us move on. So here is an activity which we will now move to an assignment because we lost two hours so some of the activities we will have to move towards an assignment but let me just describe to you what the activity is. You recall this idea that you had that we had seen earlier and we said that this was not yet a research paper. The researcher was trying to create interactive multimedia content and animated videos and they wanted to upload it on Moodle so that the concept could be understandable by the students. What the assignment is going to be, what is required for this idea to be considered as an acceptable research paper. So instead of doing it as an in workshop in class activity right now, we are running slightly out of time. Let us move this, I want all of you, I would like all of you to think about this and we will post it as a Moodle activity. The question is quite simple, somebody gives you a partial idea and the, what you have to do is how do you move this into the category of an acceptable research paper. Another slide that I would like you to recall what is not a research paper, we said this is quite important that is why we are putting it over and over again. Just take a moment to read this. So we will also now summarize what is a research paper based on the examples that we saw. So what should a research paper contain or what should be there in a research paper? When you are doing a research study and subsequently when you are writing a research paper, you must explain the precise and detailed description of the problem you are working on. You have to explain why the problem is important, you have to describe how you have solved the problem, what is your solution. You have to convince the reader and the referee as to why your solution is innovative and for this you have to talk about what has been done earlier in order to establish that your work is indeed innovative. And then you also have to evaluate your solution. When you think of the answers to these five questions, a lot of the good features of a strong research paper are in place. So you start with an idea which looks more like a report of what you have done and then you see whether you have written what is the problem I have worked on, you have tried to establish that the solution is innovative and so on and you are halfway there to thinking about a research study. In addition, the person who is going to be reading your research paper is the referee, the first person and before others can read it, the referee has to be convinced that the solution is indeed innovative, the procedure is sound and the evaluation claims are valid just like we saw in the examples so far. We will do a few more examples today to get all these ideas more crystallized, but this is the first time you are seeing a list, you may have seen it earlier, but first time you are seeing examples and seeing how each of these can be implemented. So, that is really our next question. We have a list of the criteria or the features that we want to implement in a research study, but how do you do so? How does one do so? We saw examples, we saw examples of abstracts and of summaries, but how do you actually do it in the entire paper? So, what we will do in the next session and we are going to start the next session in about 2 or 3 minutes. Let me just describe to you what this session is before that. It is the same session, it is a part P of the same session. What you will do is analyze 2 papers. One is called paper A and one is called paper B, there is no, we are not going to say that this is a completely different session, we will just seamlessly move through the sessions right now. We will take a break at 3.30, 14.