 Good morning and welcome to this last day of the synchronous remote session phase of the workshop on Pedagogy for Effective Use of ICT in Engineering Education. I hope all of you have had a very productive workshop over the last few days and also last couple of weeks. Till now, what you have seen and what we have focused on most is two main things. One is instructional strategies which goes under this title of pedagogy and technology tools and how to merge effective pedagogical and instructional strategies for use of ICT tools in your courses. Today, we will try to go one step further and look at what is called as action research in educational technology. So, before we actually look into what exactly is action research, let us go back to one of the slides that we discussed long ago on day one of this SRC phase and see what are the different aspects of educational technology. So, right then we had discussed that educational technology consists of technology for education. For example, creation and use of technology tools and you have seen a number of examples in the last three weeks. Yesterday, we discussed how to use visualizations such as animations and simulations. We looked at lot deal into the use of Wiki. You created your own Wiki pages. You also used Moodle as a learner or as a participant in this workshop. The other aspect of educational technology is what we call as technology of education. And here we use the word technology to mean an applied science, how to do something. And within this aspect what we discussed in this workshop are creation and use of several active learning strategies such as peer instruction, think-pair-share, debate and so on along with some of these technology tools for facilitating students learning and engagement. So specifically in this workshop in terms of technology tools for education you have become extremely fluent which means you have been able to do create level activities in terms of visualizations because you found a visualization and created a lesson plan for your own course. In terms of concept maps, in terms of wikis you created your own wiki pages, in terms of screen casting through homework and so on. You also got familiar that means you worked on apply level activities in terms of Moodle, Avue, open educational resources where you went and tried to find videos and animations for your courses and you got some exposure that is you worked on understand level activities. In terms of tools for blogging, tools for surveying, commenting, tagging, etcetera, if you recall the digital learning tools, the digital bloom staxonomy session I think on the two or three that is where you got a lot of exposure and did some understand level activities. So what we are doing in this slide is a recap of the tools as well as strategies that you have learnt in this workshop at various cognitive levels, right from an understand level to a create level for certain types of tools and strategies. In terms of technology of education that is the instructional strategies, as a learner you worked a great deal with active learning strategies such as peer instruction, think pair share, debate, brainstorm, fastest finger, collaborative activities, flip classroom, group breaks and so on. And similarly sometimes you had to wear the hat of a teacher, you have to play the role of a teacher and you wrote your own questions for your own courses for activities such as peer instruction, think pair share, group projects, flip classrooms, group breaks, higher order assessment questions and so on. So the previous slide that we saw and this slide effectively summarizes the content of this workshop and finally what you have been moving towards throughout is an integrated use of the tools and strategies. So initially you learnt these as separate strategies or you learnt the use of independent tools and so during the course of the workshop through the home works and through the in class activities you started creating active learning strategies with visualization, collaborative strategies with Wiki, concept maps along with some concept mapping tools, flip classroom which had a screencast or a spoken tutorial part as well as in class activities. So the point we are trying to make here is that when you are trying to learn a new subject or a new topic initially you may need to attain the understanding of the concept. So you learn it in independent ways and but the ultimate goal is for the learner to integrate the different parts of the knowledge so that there is some application and real life purpose, real life use of what was learnt and I am sure all of you do this in your own courses with your students. So in this workshop when you are playing the role of a learner you move from attain level to an integrate level understanding of the concepts and the skills. So let us do a poll at this point I am going to put the next slide up and then I will also put it on the A-view poll that having written lesson plans that integrate the use of educational technology for a specific topic what do you think is the next logical step? Should you execute the lesson plan knowing that your ideas will work? Should you execute the lesson plan and find out whether your idea is really working and to what extent or should you do a scientific evaluation of your idea? Again there is no single right answer here it is more about what you think is the next logical step here. So I would like the participants to vote on this I am going to start the polling. So let us start the polling for what you think is the next logical step after having written the lesson plan. Should you execute the lesson knowing that your ideas will work that is A, execute the lesson plan and find out whether your idea is working and to what extent or do a scientific evaluation of this idea. So it looks like a majority of you close to a hundred centres have chosen the choice B which says that what you would like to do what you think is the next logical step is to execute the lesson plan and find out whether your idea works. A few of you about six centres have said that you would like to execute the lesson plan knowing your idea will work and there is nothing wrong with this approach except that you will not have a clear and precise idea as to whether the idea works for your students. You are right in the sense that you think it should work because it is based on research and we have shown several pieces of evidence that such ideas do improve students learning and engagement but unless you try to find out with your own students in your context for your course whether this idea works or not and to what extent it works it will be hard for you to tweak your lessons it will be hard for you to determine whether you should use the strategy once again or whether you should go to some other strategy and so on. And the next step if you still want to pursue further and I think about twenty centres have chosen this choice is that you can proceed to do a scientific a rigorous scientific evaluation of whether your idea and lesson plan works or not similar to the experiments that we do in engineering and science. So what we are going towards is starting from ET practitioners, practitioners are people who use some knowledge who practice the knowledge towards ET researchers. So this session is all about how you can go from an ET practitioner towards an ET researcher and of course in one session you cannot become an ET researcher but all of us are going towards it and why this is important is whether you were in whether you chose choice B or choice C when you learn the techniques of doing systematic scientific research in any domain your teaching and your practice will also improve. So what is it what is being an ET practitioner and what is an ET researcher? When we teach students, when we facilitate their learning, when we think about improving their learning, their interest, their engagement and apply the strategies that we learnt in this workshop, when we come up with new ideas for doing all of these at that time we are practitioners of educational technology, when we think about which tool to use for which learning objective along with which instructional strategy, when we integrate all of it we are practicing the systematic knowledge, the body of knowledge of educational technology. Moving further, we go towards becoming ET researchers when we scientifically investigate the worth of our ideas exactly the way we do in engineering and science, when we conduct systematic studies to get data to find evidence about whether these ideas work or not. And finally, we have to provide evidence we have to draw conclusions infer conclusions and provide evidence that our ideas actually work. So as mentioned this session is about going from being an ET practitioner towards an ET researcher and before I come to specific objectives of the session let us spend a few minutes discussing why you should bother with ET research at all, what is in it for you. And the idea is that as teachers and as ET practitioners you are anyway working on problems that arise in your class, problems related to student engagement, problems related to student learning, problems related to how to integrate ICT in the most effective way and so on. So if you are anyway trying to come up with solutions to solve the problem in your classes why not go a few extra steps required for closing it and in fact do a scientific systematic study to determine what is the solution of this problem. And some benefits of such closure is that when you do a systematic study using ET research methods which is what we will discuss in today's session you can actually write a paper which means you can write a report and publish it as a scientific paper provided it has some criteria which is what we will discuss. But some benefits are that you could get a publication to your name. We have a conference called technology for education which is primarily for such studies. Others could adopt your solutions when you write a scientific report because just like engineers and scientists want to read research studies about how techniques are working and only then adopt them other teachers could do the same. And finally, we have found that your skill in applying the scientific process and scientific methods will actually improve in other areas of research including your domain when you conduct research in for example, educational technology and the practice in your classroom. So what we are saying here is that if you systematically do research in one area and here the area is your own teaching, your own learning, then your research skills in other areas such as your domain is also likely to improve. And the faculty members who have attended such workshops in the past years and done such studies have all attested to this that they have found that their research skills have actually become sharper in their own domain areas. So for this session, the learning objectives are that at the end of this session you will be able to first evaluate what constitutes a valid ET research study and what is not a valid ET research study. Just because you implement an idea it is not an ET research study. So we will first look at several examples and then you will write research questions for your own idea for your own lesson plans. You will also be able to state some commonly used metrics for evaluation of these ET studies and identify the types of instruments. So instruments are what do the measurement. So you will be able to state not only what metrics are used but also how to measure them using different instruments to evaluate your idea. So let us go to what is an ET research study and the way we will do this is I am going to post a large number of examples in the next few slides. And what you have to decide is whether the idea in each example can be considered to be acceptable as an ET research study or ET research paper. So these are all yes no questions and you can answer through chat whether you think that the example is valid as an acceptable research study or not. And for each example we will do a discussion as to why it is or it is not a valid research study. So let us go to the first one. So your colleague says that I have used colored chalk or colored markers for blackboard and whiteboard respectively for better teaching and especially for waveforms and curves. PowerPoint presentation on blackboard and whiteboard should be equally utilized and usage of such methods will make the lecture clear to students. So if a colleague uses an idea like this and gives a short summary is this considered to be a valid research study. So you can post your choices on chat, you can just say yes, no or 1, 2 and then we will discuss whether it is or it is not valid. Okay I think most of you have given the poll and let me tell you what I see from your responses. About two-thirds of you think that this is a valid research study and about one-third or one-fourth think that no this is not a valid research study. So let us see whether the referees of a paper or whether the ET research community will consider this to be a valid study or not and the answer is no they would not consider this to be a valid research study. And we see that this example here in fact is not a valid research study, why? Because what has been done in this paper essentially is a compilation of obvious and known solution. What do we mean by this? The person who did this, who wrote this summary used color chalk on blackboard or colored markers on whiteboard for showing diagrams. Now that may have value as a teaching strategy but it is really a very obvious solution. People have been using it for the past several decades. Something that you try which is obvious is in fact is not considered to be research and this is similar to engineering research or science research where there has to be some novelty in the solution before it can be considered to be an effect to be a research study. So what we are saying here is that you may have tried some effective solutions to improve teaching. We are not doubting that the solution has merit but not all such solutions and not all such ideas can be converted to a research study and especially if the solution is not novel. So we will come later in this session, we will talk about what is exactly meant by novelty and how to achieve it. But one of the main things you have to keep in mind is that not all studies is a research study and the key point, key takeaway I hope from this workshop is getting some ideas, some novel ideas for some innovative teaching strategies. Let us look at the next example. So here your colleague says that I will prepare interactive multimedia content. Using Moodle the student can access the content in order to make interactive sessions. The student will be more interested in interactive and this interactive content will help students understand the concept more easily. So is this an ET research study, yes or no? I think most of you have voted and again I see a large number of ones, larger number than earlier, think of you think that this is an acceptable research study, there are a few people who think that no this is not an acceptable research study. And I realize why you think this seems to be acceptable because compared to the previous example of using colored chalk, this seems to be more innovative. This person has not only prepared multimedia content which may be considered to be somewhat innovative but they have used Moodle, they have gotten students to access it and so on. But the paper referee will still say no this is not a valid research study. So the answer even here is that no this is not a valid research study. Why is that so? What is being done here is mostly development of instructional materials or instructional strategies and mere development is not a ET research paper. Even if the material or the strategy is based on an innovative idea. What is meant by this is to be considered as an ET research paper, the researcher needs to show that the material or strategy has actually resulted in improvement of student learning or engagement. So if you look at the previous example, you see that this person has not really made any effort except to say that the students will understand the concept. So there is no effort in terms of trying to measure what has happened to student engagement and what has happened to student learning. So without measurements of effectiveness and without evidence from data and inferences that the idea works, development of instructional materials or instructional strategies cannot be considered as ET research papers. So we will look at two more examples and I think you are getting the idea now. So try to look at the examples very critically, is there measurement, is there evidence, is there an innovative idea and so on. So here in this example, the colleague says that the purpose of the study is to use Moodle in an engineering course and study the motivation behind its use. Activities such as presenting information, managing course material and evaluating student work were done through Moodle quizzes and this person is also doing some evaluation here because instructors were asked the benefits and barriers to using Moodle. So this person used a survey. So suppose you do something like this, is this considered to be an ET research study a valid research study. So in this example again I think the majority of your large majority close to 90% of you think that this is a valid research study and what the track that you are all on is in fact on along the right lines that compared to the previous two examples, this example seems to have some evaluation. It also seems to be using Moodle so there seems to be something not very obvious in this idea. So this example clearly is better than the previous two examples that you saw. However the research community and the referee of the paper will still not pass this paper will still not accept this for a conference. And you may be wondering what is wrong in this idea now and now things are getting a little more challenging and interesting because what this person has done in this study has the person has used an ET tool which is Moodle in a routine manner. So if you use an ET tool such as Moodle in a routine manner it is not considered as a research paper. So you may ask what do you mean by routine manner versus non-routine manner because in your college maybe nobody ever uses Moodle. So this goes back to not just practitioners but also the research community. It turns out that Moodle has been around for a very long time several years maybe a decade and learning management systems similar to Moodle have been around even longer. So people have been using these systems very routinely for purposes such as presenting information, managing course materials, evaluating student work and so on. So if you use the tool for a common purpose for a routine purpose it cannot be considered as research. What you need to do to be considered as research is use the tool using an innovative method to achieve a teaching learning goal. So for example if you use Moodle to create a game that allows students to learn a concept and the teacher checks how much collaboration has happened in Moodle this is going beyond the routine use of a tool. Similarly in one of the chat sessions I believe on two days ago one of the centres said that they were using Moodle's or wikis to implement TPS through the chat forum that could be considered as moving beyond a routine use of Moodle. Even there you have to do some more work to establish that it is not routine and will come to what is called positioning towards the end of this session. But the main point we are trying to make is simply using Moodle for routine purposes cannot be considered as research even if it is a good idea and even if you have done evaluation. So what we will do is summarize at this point what is not an ET research paper. So what is not an ET research paper summarizing what all we have seen so far. Compilation of obvious solutions is not an ET research paper. Simply reporting the strategy you implemented is not an ET research paper. Use of ET tool in a routine manner is not a research paper and mere development of instructional material is not a research paper. So right now we are not even looking at the techniques of research we are simply looking at the starting point that when can you even begin to think of considering writing an ET research paper. So we are at the level of the initial idea that you propose for conducting an ET research study. So you may be wondering at this point that well there are so many points of what is not an ET research paper. So what exactly is an ET research study or when can I even think about beginning to write an ET research paper. So now what we will look at is now that you know what is not an ET research study or what should not be considered to write an ET research paper. Let us move towards what constitutes what is in an ET research study. And I think we have discussed this briefly over the previous few examples. Now let us formalize it a little bit more that one of the first things your research study and paper must have is something called novelty. What is novelty? What you need to show is that your idea is unique in the research community. And what is meant by unique is that you have to analyze prior reported literature to show that your idea is unique. The dictionary definition of novelty is that it is the quality of being of something being new, something being unique, original, innovative and so on. So you may be asking you can ask at this point that what should be new and there are several things in your study which could be new. And depending on what exactly is new in your idea in your research you have your paper is either strong or it is not as strong. So from strong to weak the problem that you are attempting to solve, the teaching learning problem that you are attempting to solve could be new, it is very rare that this happens. What do I mean by this? Let me do a very quick interactive question. I do not have an activity slide but I want all of you in your centers to simply say a few words to your RC coordinator about what is the teaching learning problem that you are trying to solve by implementing any of these strategies. So we are not asking about your solution but your teaching learning problem could be that students are not paying attention or students need to learn more. So try to quickly and precisely state what is the problem you are trying to solve and coordinators please share this answer through chat and then I will go to the next few points. So far you have thought about this because you have written lesson plans. So what problem are you really attempting to solve? Let us try to get some clarity on that. I will just post this question on chat. So I am actually seeing some interesting problems. Some problems are very common and it is right that these are important problems and that is why all of you have it. So some common problems are that students have lack of interest. Students have low prerequisite knowledge. Students are not paying attention. So most of the problems that all of you are saying seems to be in two or three different categories. One related to students lack of interest and motivation which results in students are not paying attention or students engagement is difficult for one full hour and the other category is on improvement of students learning. For example, how to improve programming skill or logic of each student? How to get students to do creative and critical thinking? Okay, here is a new category that a few of you have written that how to handle the diversity in class? I think I say about three or four about variation in the degree of student ability and student motivation. It looks like most of your problems are either in the category of improving student engagement or interest, student learning or understanding at various levels and the third category is addressing student diversity. It turns out that all these three problems are age old problems and the reason that their age old is because they are really difficult problems to solve. So unless you find a very new angle to these problems, it's a little hard to establish novelty in the problem itself. Even if you think about engineering or science research, coming up with a completely new problem to solve is a little difficult. So what most research papers do is try to come up with a new solution to solve a known problem. So your solution, your actual strategy that you can use could be new. For example, if you say that you are going to use games in Moodle to improve student engagement in a programming class, there is some novelty in that solution. And finally, what you can do is have a new context. This is the weakest. What I mean here is that maybe there is a known solution. For example, peer instruction, the method of peer instruction, it's a known method or TPS, it's known and there are several research studies about how peer instruction and TPS are actually useful and helpful. But perhaps people have not done it in your domain and not shown evidence in your domain or your context. There is some chance of establishing novelty there even though it's a little weak to do it. So some of you can have asked this question, can a non-innovative strategy which means your solution is not new? Can a non-innovative strategy be developed into a strong research paper? And as we mentioned earlier, the answer is yes, you can, but it's a little weak and you can do it provided you position this well. Let's try to understand what is meant by provided it's positioned well. So now we are at the stage where you know roughly the problem you are trying to solve. The student interest or student learning or diversity, these are all problems that have been addressed in the past. Your solution either is really novel or it's not really novel and you've simply adapted some other interesting solution. So can you write a research paper given this situation? So what all research papers need is something called positioning. What is positioning? An analysis which shows that your research study and your research work is actually required which means what are the gaps in existing work, why is your work required and how your work advances the state of the art. This is what's called positioning. So let's look at one example, we'll just explain this and then I'll show you one example. So positioning, the dictionary meaning is how you situate or how you relate your work with respect to others. How do you do positioning in a research paper and this if you've written research papers in your domain you'll be very familiar with this idea. That you have to show analysis of published related prior work, not simply internet reports but published papers from journals and peer reviewed conferences that bring out gaps in existing solutions. So you have to analyze papers that have addressed problems similar to yours as well as papers that have addressed solutions similar to yours. Let's go back to the TPS example. So suppose you want to use TPS in a data structures course in computer science in any course it can be or a mathematics course to improve students problem solving at the analyze level. You have to address, you have to find papers, a lot of papers which have implemented any strategy to improve students problem solving at the analyze level. Maybe they have implemented different strategies. At the same time you have to find papers that have a solution which is TPS similar to yours. So look at other papers which have implemented TPS for student learning maybe in other domains. And you will find that there are papers that have implemented TPS in computer science programming, in biology, in psychology and so on. And then you have to try to see what are the gaps in existing solutions and how different your paper is compared to existing papers. So this is a very important point that as the novelty of your problem or solution decreases as we went from strong to weak in the previous slide, the accuracy of your position must increase. What is meant by this? That suppose you are trying to implement TPS in a data structures course you have to be very precise as to who else has implemented TPS and in what courses and why there is need for yet another study on TPS. So let us look at a couple of examples now actually one example where novelty and positioning are addressed. So I am going to show an abstract of a paper on the next slide and this is a think pair self-assess. So there is no share phase you do not need to send anything through chat. In the think phase based on the abstract that I will show in a minute, identify one way in which the paper is different from related work. So you are being asked to analyze the positioning of the abstract of the paper that is going to show up and in the pair phase I would like you to check your neighbors answers and have they found the same gap there may be more than one gap in the abstract. And then I will show you some answers and you can self assess your work. So here is the abstract, read the abstract, identify one way in which this paper is different from related work do that individually and check your neighbors answer have they found the same gap or have they found different gaps. So you can take about 5 minutes for this activity, it will take you 2 minutes to read this. Take about 5 minutes and then I will show the slide where we do a self-assessment of whether this particular you know what are the gaps here and how this paper has tried to address the gaps. So here right now this is what you are supposed to do. In fact the answer is there in the first half of the abstract I have put the full abstract because later we will be analyzing other details of this particular abstract. But you will find the answer pretty much in the first 3 or 4 lines itself. I think you can if you have not yet moved to the pair phase please move to the pair phase and try to check your neighbors answers. I am actually going to view a few centers and I hope all of you are discussing this particular problem. Yeah please talk to each other try to find out the gaps that this research study has found in related work and how they have tried to address the gaps which means they are trying to situate their research within existing research. How have they done that? This is the time you have to be discussing with your neighbors. So it would be much better if you actually try to answer this question by discussing neighbor or you can do it within the center. So let us look at a few answers here and please do a self-assessment as to whether you found these. So as a pair or as a group check if you found the following aspects of novelty and positioning in the abstract that you just viewed. Yeah some of you in fact are sending it through chat good I think all of you have gotten somewhere that this paper has found that in existing techniques weeks of training are required and proprietary software are required. I am not asking you to analyze anything else in the abstract but just how is this particular research study related to existing work. So right towards the beginning they have said that mental rotation ability can be improved by computer based training which means that existing studies are present. Most existing technique requires weeks of training so that is a gap and are based on proprietary software. And what this research has done is found a novel solution approach by developing only a 3 hour training program using Blender which is a 3D open source software. So both the gaps are being addressed in this paper. This is actually the abstract of a paper that was published in T4E 2012 and we look at this example and a few more examples as we go on. So at the point of this short exercise is mainly to show how you should analyze existing work bring out the gaps in existing work and identify and state how your work fits into those gaps. So when you are doing positioning let us do one level of one summary. If you go back to slide 27 which I am going to show you will see that in order to do positioning you have to analyze related prior work to bring out the gaps and show that your solution addresses any of the gaps. So let us look at a summary of what is a poor way to do it and what is a much better way to do it. So the slide after this actually shows from awful to good to better on how to explain the relation to other work. So this person this summary actually picks up some hypothetical problem called the galloping problem and the source of this is down here. If you simply say that the galloping problem which is the problem you are trying to solve has attracted lot of attention and given a series of references this is a really awful way to do positioning. The referee will say that okay it looks like the author has read these papers and has identified other authors who have worked on the galloping problem but has done no analysis. On the other hand if you take one step further and say that Smith and Jones have worked on galloping which are these one of these references you are really one step better but you are still at a bad level because all you have said is that this author in this reference and this author in this reference have worked on galloping. Don't stop here. If you go one step further and say that Smith addressed galloping by some method called blitzing and Jones addressed it by a flitzing approach at least there is some evidence that you have read these papers and you know that the first reference addressed the problem using some approach and the second paper has addressed it using a different approach. Even then your effort at positioning is going to be considered as poor by the referee. You will be considered as good if you get to this level that approach one Smith's blitzing approach to galloping problem achieved 60 percent coverage and Jones approach achieved 80 percent coverage by flitzing but only for point of free cases. So if you look at what is being done here the author has talked about the problem talked about two approaches blitzing and flitzing analyzed which of these approaches have achieved how much results and also gone one step in detail and analyzed under what conditions this approach works. You should at least try to get to good and ideally you should get to better where the author says in addition to all of the above the author says what they have done. So you will be given a high grade in explaining relation to other work if you also say that we modified the blitzing approach to do something something and achieve 90 percent coverage. So if you get to this level that means you have not only analyzed gaps in the problem and in the solutions but also shown how your solution addresses the gaps. So to summarize what we have done so far we have looked at novelty and positioning discussed what they mean and identified novelty and positioning in some existing examples. So what we are going to do next is try to identify what else is important in a research study. So if you remember one of the previous slides I believe it was let me just show you which slide where we said what is not a research study. So if you recall the summary of what is not a research paper we have discussed the issues of novelty and positioning somewhere here that compilation of obvious solutions is not a research paper but you need to do novelty use of ET tool in a routine manner is not a research paper but you need to do something innovative and show positioning and so on. What we will do next is try to look at how to get rigor and evidence into your research. So for that I am going to pose a question it is going to be a think pressure activity but I will only pose the think phase first then we will break for T look at the think phase and over T think about the think phase work on the think phase and come back when we come back and rejoin after T we will do the entire activity. So what I would like you to think about is the lesson plan that you worked on yesterday where you integrated an ET tool with an with some strategy and in that lesson plan you wrote answers to what you will do and what your students will do you did this in one of the lab exercises yesterday. So what you need to now think about and this is moving towards ET research is how will you determine how will you find out if your idea actually works. So think about several statements not just one many statements precise and specific statements that you can expect to see if your idea works. So write statements such as I think my idea is successful if I find that my students are doing this my students are feeling this I am doing this and I am feeling this. So instead of three minutes let us actually take a full twenty minutes. Write this come back at 10.50 we will start the session at 10 minutes to 11 after T and then we will look at how to actually convert these ideas these these expectations of what your students are doing and thinking into precise research questions at this point. I am just going to post that on chat do not post anything on chat right now simply think about this idea over T write down as many possible statements and come.