 Good morning and welcome to this last day on the Q2P workshop on pedagogy for effective use of ICT in engineering education. So, over the past few days and weeks we have seen several ET strategies and several tools. So, today's session we will do something which takes us one step further which is to see examine effectiveness of these strategies. So, the idea here is called action research and before we get into it let us recap what are education technology aspects. So, as you would have seen over the past few weeks education technology can be broadly categorized into two parts. One part is called the technology for education which actually deals with the creation and use of technology. So, for example, you might create a visualization or you might use a technology tool such as Vicky or you might create a flip classroom video. So, all of those are technology for education and the other category is the technology of education. So, in technology of education we are we talk about educational strategies such as peer instruction, think pair share, debate and other activities that you carry out either in the classroom or in the online phase to facilitate students learning and engagement. So, broadly we are talking about technologies for education and technology of education and in this workshop we have seen both of them as well as integration of both of them. So, as technology for education you may recall that you have worked on several technologies. For example, you are now fluent and working at the create level of Bloom's taxonomy on technologies of visualization, concept maps, wikis, video casting, you are familiar or working at the apply level of technologies such as Moodle and use of A-View. And you have exposure or understand level of Bloom's taxonomy for technologies such as web quest tools for blogging and tagging, posting you know you may recall various technologies that were mentioned that you yourself also mentioned during the digital learning tools session on two days ago. As technology of education again you are familiar with tools such as peer instruction, TPS, debate, collaborative strategies and classroom as a learner as in you have experienced these tools while we have executed these activities as part of the workshop. And an important idea was that we did all the lectures in 20 minute chunks so that was also an important idea that you have experienced as a learner and which you can now go ahead and implement as a teacher yourself when you are creating activities. So as a teacher you have also written peer instruction activities for your own class and topics, you have written TPS activities, you have created group projects and you have also created flip classroom videos and activities for your own use as a teacher. In addition to that you have also integrated the use of tools and technologies you have not just simply done them separately as technologies for education versus technologies of education you have also integrated both of them as integrating active learning strategies with visualizations, integrating collaborative strategies with Wikis, concept maps with concept mapping tools, group projects with various tools, flip classrooms with creation of the spoken tutorial or the video casting with face to face active learning activities. So all of this you have seen. So now the question is having written lesson plans to integrate the use of ET in your topic which is what you did in yesterday's session what is the next logical step that you think you should take. So there are three choices here one is execute the lesson plan knowing that your idea will work being confident that you know it is going to work. Second is execute the lesson plan and find out whether your idea is working and to what extent and the third is to do a scientific evaluation of the idea. So we will just post this as a modal poll and participants in the meantime can think about these questions and choose their vote and coordinators can convey it as a modal poll. So the poll has been started and coordinators can simply send their majority answer of their center through the view chat, a view poll. So what we find is six users are saying that six RCs are saying that find out whether it is working some other four are saying that do a scientific study. We have twenty answers so far thirty seven centers. So now the majority answer here seems to be that we want to find out whether it is working majority of the people would like to find out whether it is working very few people a fair number of people have said that they would like to do a scientific study or we should do a scientific study on it and few people three people have said that we will simply execute in the class. So all of these three are valid answers except that in the first case if you simply execute the lesson plan in the class you really do not know you only have a feeling that maybe it is working and sometimes that is ok but for the main part we would want to find out whether the idea is working or not and in the process move towards doing a scientific evaluation of the idea. So how do we go about that? So that is called doing research in education technology. So we are E.T. practitioners or E.T. users as teachers when we teach students and facilitate their learning when we think about improving students learning students interest in the subject engagement in the class and so on and so forth and when we come up with ideas such as various E.T. strategies and various E.T. tools use of various E.T. tools for doing this. So when we simply do these such as the lesson plan that you wrote at that point we are at an E.T. practitioners level. So we become E.T. researchers when we scientifically investigate the worth of our ideas. When we conduct systematic studies to get data about whether our ideas are working or not and when we provide evidence to support our conclusions. Now these studies can be carried out at various levels depending upon the depth of your interest towards becoming an E.T. researcher. So at the simplest level you could carry out studies to find out whether the idea is working or not. So this session is about going from being an E.T. practitioner towards becoming an E.T. researcher. So the important keyword here is towards because at the end of this session you would have taken a step towards becoming an E.T. researcher. There is an entire different workshop which will provide training on actually performing the E.T. research and becoming a competent E.T. researcher. So we will take one step towards becoming an E.T. researcher as part of this session. So before we get into the details of the session itself many of you may have the question why should I bother to do this? I am not going to get any extra credit from my institution for performing this E.T. research. So the answer to that and performing this E.T. research is definitely going to take some more effort on your part and some more time on your part. So the answer to that question is that you are anyway working on the problems arising in your class. You have come for this workshop which itself means that you have a lot of interest in student learning and student engagement and you are actively seeking ideas to implement to improve these facets of your students. And you are coming up with solutions for the problems that you see in your class or to improve some aspect of teaching learning in your class. So the question here is there is just one more step left to close this loop of identifying the problem coming up with the solution and evaluating the solution. Then that is one small loop of a scientific research cycle. So some benefits of doing this closure such as systematic study using E.T. research methods followed by perhaps writing a paper are you get a publication to your name. There are many conferences which we will be listing out later to which which are targeted towards publication of use of E.T. tools and strategies by teachers. You can also disseminate your findings so that others can adopt your solution. So instead of simply saying that look I tried this in my class and students seem to be happy. You can now provide data which your colleagues can use to see whether they want to adopt your solution or not. And as a side effect your skill in applying the scientific method in other areas of research will also improve. So irrespective of whether you are a CAC researcher or a doubly researcher as your core area the fact that we are doing this scientific cycle here in the E.T. domain will also help you to apply the same method in other areas of research. So hopefully most of you are convinced that it is worth taking this step to towards becoming an E.T. researcher. So the learning objectives of the session are that at the end of it you will be able to write some research questions for your own lesson plan idea which you did yesterday. You will be able to state the commonly used metrics for evaluation of such studies on technology enabled learning. You will be able to identify the type of instruments required to evaluate your idea. So here instruments is being used as a technical term in the education technology domain. It does not mean instruments such as hardware instruments which we typically think of. What it means is instruments for determining effectiveness of your study so such as tests and surveys. In the lab you will identify the key components in an E.T. research paper, flesh out your own idea and you will find papers and topics similar to your lesson plan ideas. So now there is an activity think-pair share activity. So recall the lesson plan that you wrote yesterday and what answers you gave for what you will do and what your students will do. So now you need to think about this question. How will you find out if your idea is working? So here are some hints to help you think. So instead of simply saying I will know that my idea is working when I see smiles on my students faces you can actually write down more precise answers such as I think my idea is successful if I find that my students are doing and filling the blanks. My students are doing better in exams for example or my students are feeling more engaged with the subject or something like I am doing less of lecturing in my class or I am feeling more motivated to engage with my class. So these are just some examples of what you can do in the fill in the blanks. So you can take 3 minutes to do this activity of coming up with statements for how you will find out if your idea is working. So in many centers I find that people are already started talking to each other. It is not yet the pair phase but since many of you have already gone on and in some centers I see that people are still trooping in. We will wait here for one more minute before we go on to the pair phase activity. So as you have seen in the think-pair-share activity it is always useful to do the think phase individually so that you have some clarity on what your own idea is before you start borrowing ideas from your partner. So some of your centers are already sending answers to the chat. This is not yet time to send answers to the chat. Let us go to the pair phase activity. So here what you want to do is suggest one, suggest more statements that your neighbor could add to his or her answer that may indicate success of the idea. So that is the simpler part of the question. So if you have already added points like I am doing and feeling successful or my students are doing discussion. So these are some of the answers that I am seeing on chat. These are still at a elementary level. I am feeling comfortable in lecturing. So these are simpler statements that you could add. Now the important thing that you want to do as a pair before you keep sending answers on chat, the important thing that you want to do is select any two statements and determine how you will measure the extent of the success. So for the ideas which people have already said my students are feeling satisfaction, what you have to now do is identify how you are going to measure that satisfaction. So if you are saying that I am doing and feeling successful, how are you going to measure the extent of your success? If you are saying that my students are feeling comfortable in solving numericals, how are you going to measure the extent of comfort in solving numericals? So these are the things that you need to now work on together. So the focus here is how will you measure the extent of the success? So you can take some time to do this. If most of you have finished with identifying how you will measure the extent of the success, let us move on to the share phase. So share your answer with your colleagues within your RC. Mention both what is to be measured and how it is to be measured. For example some answers that I am seeing is like I will measure the scoring of marks or performing good in the group discussion, analyzing the results and ability to solve problem. So mention what you are going to measure which is the performance in the exam and how you are going to measure that is through marks. So for each of your answers share it with your colleagues in your class in your remote center and coordinators can share the most common answer through the A-view chat. So some people are still saying I am feeling happy after seeing the students confidence. So here there are two things that you have to measure. One is you have to identify how you are going to measure your own happiness, how it has increased and you also have to measure how the students confidence has increased. What are you going to say to make a claim that the students confidence has actually increased. So those are the things that you want to think about. Another example is somebody has said students doing the assignment with more enthusiasm. So that is still what is to be measured. You are saying the students enthusiasm for assignments is to be measured. What you also need to say is how are you going to measure, how are you going to identify that they are doing it with more enthusiasm now. So there are some good answers which have started coming in. So one of the answers is my students are solving 90% of the peer instruction questions within the specified time limit. So once again for those who are writing answers like my students are doing as I suppose it to get done. What you want to say is how are you going to measure that they are doing as you want them to do. So some of you are saying that students are more confident in articulating their ideas and this can be measured by arranging a presentation. Now you need to take one more step and say how are you going to measure the presentation. You have already seen some of it yesterday in terms of rubrics. So you can say that I will have my students make a presentation. I will use such and such rubric in order to measure how well they are doing in the presentation. Some are saying that I feel satisfied as the students feedback says that my lectures are interesting. So while this is valid, this is the 0th level of feedback. So what we want to do is get a little more detailed than students say that my lectures are interesting or fun. Many of them are saying the number of time a student participates in discussions and debates. Previously identified week students are now answering the question with interest. These are good ideas. Student activities in class and out of class is enhanced to 80 percent. My students are doing assignments at higher cognitive level and the measurement is the success rate after the implementation. Success rate I assume is the final exam. So once again for those who are writing answers like students are motivated to learn, you need to also identify how you are going to measure that motivation. We will measure student engagement versus time graph. We will measure student ability to explain the concept by presentation and polling. I will wait for a few more answers. I will measure students improvement by comparing the previous and now responses to the question before and after the activity. Take an exam before and after the activity. That is a good idea. That is technically called a pre-test and a post-test. Students are solving with lesser time and more numericals they are able to solve correctly. So all these are fairly measurable ideas that you are proposing and what you would have observed from the answers that I have read out, from your own answers that I have read out is that most commonly we are talking about two different ways of measuring. One is how much learning has increased and the other is how much is the student interest or student engagement has increased. So we will see what these are shortly and how to formalize them shortly. So the first step here is to identify what are called research questions of your study. So research study contains research questions and its answers accompanied by the evidence. So there is a difference between problem and research question. So the problem may be something that is happening in your class that you want to solve and the idea teaching learning idea that you have come up with may be the solution. For example you may find that the back benches in your class are not being engaged. So that is the problem and you may come up with the mechanism of continuously rotating the student's benches so that that problem is solved. So that is your idea. Now after you have the problem and the idea that is when you want to construct a research question. So now when you do a research study you have to construct research questions about this idea and the study contains answers to the research questions accompanied by evidence. So let us look at some examples. So a research question expresses relation between variables in a specified context. So a question of the type R animations effective is not a research question. So there is no context and there is no relation that is being specified here. So if you see the bottom there is the research question is R animations more effective than still visuals for conceptual understanding of electromagnetic fields. So what is happening here? So when we rephrase or when we expand this non research question into a research question what we find is that this has the variables. So what are the variables here? The variables here are conceptual understanding and animation and still visuals and we are saying that we are doing a comparison of animations versus still visuals for conceptual understanding in the context of electromagnetic fields. So anytime you are tempted to write a research question of this form R animations effective you could easily take the next step to write a more specific research question saying that effective for what and in which context as compared to what. So the second thing is that a research question should be stated in the form of a question not as a statement. So for example for the same study if we say that ok my purpose of my study is to gather support for my idea that animations are more useful than visuals then it is not a research question. So only when you take that and phrase it as a question saying that R animations more effective than still visuals for conceptual understanding of electromagnetic fields only then it becomes a research question. And the third point to keep in mind is that the RQ must imply possibilities of empirical testing. So what we mean by empirical testing is that we should be able to administer a test or a survey to students and be able to determine the extent of this more effective that we are writing. So for example should one use animations in the primary school classroom is not an RQ. Yeah should basically gives the feeling of judgment and you might simply come up with a yes or a no answer which is again not useful to anybody else who is looking at that study. So the animations should one use animations in primary school classrooms for teaching this topic for that purpose. So when you expand it and remove the should when we only focus on R the animations useful or not. So that is when it becomes a research question. So to summarize this point research question has to express relation between variables in a context. It should be stated in a question form and it must imply possibilities of empirical testing these are the three main things that you want to keep in mind when you write a research question for your own teaching learning idea. So what we will do is see a few examples of these research questions. So these are examples all these examples are from what our students in the education technology department at IIT Bombay have worked on and written papers about. So the first one is from a study of effectiveness of think pair share in a programming course. So the research questions are of the type how much student engagement occurs in the think pair share activity. So there is a notion of quantifying how much student engagement occurs and that is measured using some observation protocol and some answer quantifiable answer is found for this research question. Another research question is how does the amount of engagement change as the activity progresses. So for example you may find that there is 80 percent engagement occurs in the activity. So the next research question can answer that how much of this 80 percent happens in the think phase, how much happens in the pair phase, how does it change, does it increase, does it decrease. And the third question could be do these TPS activities lead to increase conceptual understanding and application of the programming concepts. So this can be measured by tests before and after the TPS activities or it can be measured by two groups taking the test one group which has done the TPS activity while another group which has not done the TPS activity and seeing what is the difference in their marks in those tests. So the next study is on prediction activities with visualization. So this again many of us will know that you are familiar with visualizations and you have identified visualizations for use in your course. You will start using visualizations in your course and so now the question is is it sufficient to simply show the visualization. We have already seen that in the visualization session that you do not want to simply show the visualizations there. So the question that you can ask is does prediction activity with the visualization where you pause the visualization and ask students to predict what is going to happen next. Does it lead to higher levels of learning than simply viewing the visualization and clicking through the visualization. Another question that you could ask is what are the students perceptions about learning from visualization with the strategy that you have used. Do students feel that it is making them work more? Do students feel that it is benefiting them even though it is making them work more? So these are student perceptions that you can measure by means of surveys and interviews for answering this RQ. The third RQ can be what are the difference in learning and perceptions in high achievers as compared to low achievers. So does it happen that the high achievers in your class feel that the prediction activity is required or not required? Do the low achievers also feel the same? Do they actually learn more if we ask them to do the prediction activity? So these are some RQ that you can explore in a study on prediction activity with visualization. Some more examples. So here is a study with student use of Blender which is a 3D animation software in engineering courses. So computer graphics and engineering design courses. So one question is does a 3 hour Blender training module improve the mental rotation ability of first year undergraduate engineering students. So what is meant by mental rotation here is that given an object a 3D object and given one view of that object are the students able to mentally rotate the object and draw the view from another perspective or another angle ok. So does this Blender 3D software which helps the student to visualize the object does this training improve the mental rotation ability of these students. So how would you answer this RQ? You might actually give some objects give some test to the students before the training and see how well they are able to do the mental rotation then do the training and then again give another test to the students to see whether this mental rotation ability has improved in the subsequent test. So that is how you would go about answering the RQ. Another RQ that you could ask here is does a 3 hour Blender 3D module training module motivates students to learn computer graphics concept does it increase their motivation. So you can check whether what do students say before the session about their fear of computer graphics what they think whether it is a easy subject or a difficult subject whether they can do that whether they are interested in that and after the Blender training module you can see whether there is any change in that perception or motivation towards that subject. Some other studies which can which are on training programs or use of mode of teaching learning. So one of them one RQ is what was the improvement in the participants knowledge of ET research methods. So this is from a similar workshop that we carried out last year using the same SRC mode that we are using currently. So the workshop was on ET research. So the current workshop is on ET tools and strategies. So what we measured was what was the improvement of the participants knowledge both measured by means of their assignment submission and perceived by means of what they answered as their surveys. Another RQ is how do the participants perception of the usefulness of active learning strategies in SRC mode affect their overall satisfaction. So this could be an RQ that you can directly relate to. So the way we are interacting right now is what we call the SRC mode because we are interacting synchronously and you are all remote and in a classroom. So it is a synchronous remote classroom and you have used lot of active learning strategies in the sessions for example polling and think-pair share. So how does your perception of the usefulness of these activities affect your overall satisfaction of the workshop? Would you have preferred that we not do these activities and simply go on transmitting information from our end or do you feel that conducting these activities has actually helped you to learn more from the course and to stay more engaged with the course. So this is one example of an RQ and a third example of an RQ in such a forum is how effective is the Moodle forum for collection of muddy points from participants. So muddy points are basically queries and doubts at various levels. They could be clarification, they could be conceptual, they could be deep queries. So for different types of queries, how effective is the Moodle forum for collection of queries from participants and how effective was it to address the queries? How satisfied were the participants that their queries have been addressed? So for example, many of your queries we are addressing over chat. So you are sending us the queries over chat and we are addressing it back through the video. So the RQ could be how effective is this form of getting the queries and addressing the queries. So these are some examples of RQ. So now what we will do is we will do an activity which we call fastest finger first. You must have seen that in many TV programs. So the idea here is you have to quickly write an RQ about your own research study. So revisit your answer to the ThinkPairShare activity that you just did some time ago and write one RQ for your ET research study. So the key thing that you want to keep in mind is that this should be a question and not a statement. Quickly pass on your answer to the coordinator and coordinator can share the first valid RQ that they get through the view chat. I have got the first RQ from Don Mosco College of Engineering and Technology Gohati. Do use of peer instruction lead to improving improvement in answering the in class questions correctly? UIET Punjab University. Does the use of multimedia animations enhance the students performance in understanding algorithms? Slightly incomplete because enhance when you say enhance you have to say as compared to what but still it is a good RQ to investigate. How to evaluate performance of moodle in teaching learning process? This is still a very broad RQ. Does the increased use of ICT in teaching distract the brighter students? So once again if you are able to define what is distract and how you are going to compare it is worth investigating. Is video more effective than PPT is not really a valid RQ because many of the items that we specified such as the context is missing. So you need to be more specific than that. How many number of students can solve the problem within a limited time after using ET? Once again you need to specify what is the topic. Yeah that's just a measurement it's not an RQ. Can each subject be taught through animation? This is again a very broad question it's not an RQ. You want to be specific about what exactly you want to investigate. So let me do one thing. Let me go back two slides and see whether you can take your own RQ and examine whether your RQ expresses a relationship between variables. So try to identify whether your RQ is in this category at the level of simply saying are animations effective or try to identify whether your RQ is at this level where it says should one use animations in a classroom and see what you can do to bring your RQ to a specific level like this. For example is Adobe Reader a better tool than MS Word to pass comments to students about an article. So at least if you add one more aspect where we talk about the type of article it becomes a little more specific. How many students secured more than 80 percent marks in an examination who attended the sessions on TPS and PI activity. Are simulations useful in network is again an RQ at the level of our animations effective. You want to be a little more specific than that. Simulations as compared to what and for what purpose. So the comparison is important and the purpose of use is important in addition to the context that we are talking about. Does the flipping classroom improve the learning capacity of weak students than the traditional chalk and board method. It is a medium level ok. It is see one more thing that you want to keep in mind is you want to investigate RQs which are beyond the obvious slightly beyond the obvious. How to effectively implement visuals in ICT in the class is not an RQ. That is a question that you are trying to find an answer to. It is not a research question which you can answer using some measurements. So what you are trying to do there is to come up with a process or a mechanism of implementation. So that is a problem or a question. So the difference between a research question and a question is that the research question has to have possibility of empirical testing. Here is a question. Does fastest finger first activity motivate the participants to answer better than the PI activity. It is a very good question which has come from Government College of Technology Coimbatore which is actually a good play on our active learning strategies itself. Well done Government College of Technology Coimbatore. Ok we will move on. So now let us go on to the next point here. So we have talked about RQ and we know that some things we need to measure. So let us try to formalize our understanding of what it is that we want to measure. So luckily for us these are nicely categorized into four categories. So the tell actually stands for technology enhanced learning. So if you see in some earlier slide I would have given the expansion of tell technology enabled learning or technology enhanced learning. So tell is basically wherever you are using technology for the teaching learning process. So the metrics can be broadly classified into four categories. The first one is called effectiveness. So any time you are measuring or your metric is about improving learning of concepts or skills of students you are in the effectiveness metric. You are trying to do something about the effectiveness. You are doing a effectiveness study. The second metric is called attractiveness which includes all the ideas related to enhancing student engagement, student motivation, student confidence, affective states. Effective states are emotional states such as boredom, frustration. So anything that is dealing with getting students to be more engaged or more interested all those RQ's and metrics and measures come under this attractiveness category. The third one is called accessibility. So accessibility is making the solution accessible to larger groups. So scaling the solution to larger groups, different types of learners, different contexts. The fourth one is called efficiency where what we are measuring is how much time is saved. So some of you had questions like are students able to do this faster for an individual or for a group. You may also measure things like does it save instructor effort or money or other resources in the system. So broadly speaking there are these four categories of telemetrics. Effectiveness which deals with learning. Attractiveness which deals with engagement. Accessibility which deals with scaling. And efficiency which deals with saving time or resources. So the important thing is in any study you do not want to attempt all the four metrics. Focus on one or at most two metrics in a study. So now let's do an activity of revisiting the RQ that you wrote in the previous activity. So what you did was you did a think-pair share and then you refined your RQ in the fastest finger first round. Revisit that RQ and identify which telemetric does it best correspond to. Is it effectiveness? Is it attractiveness? Is it accessibility? Is it efficiency? So vote on the question. Identify your own and vote on the question above for your own RQ. And coordinators can simply convey that majority of my RC people are working on either effectiveness or you can say that okay most of the people RQs at my RC are dealing with attractiveness. So far we have one RC that has said that the majority of their learners are effectiveness. So surely different RQs will come under different telemetrics. All that the coordinators need to do is see what is the majority. Majority of the people in your RC are they working on effectiveness or on attractiveness or on accessibility or on efficiency. Each individual participant should simply say what is the category of your RQ. Okay we have most of the RCs have submitted and an overwhelming majority 16 are in the effectiveness metric, improving of learning of concepts or skills in some form. Four are in the attractiveness metric of enhancing engagement. Another four are talking about accessibility scaling to different scenarios okay. I am surprised because when the RQs were to be shared I did not see too many of these which we are talking about scaling to different scenarios and three are talking about efficiency or saving of time or resources. So do revisit the definitions of accessibility and efficiency and see whether you are doing accessibility or whether you are doing attractiveness in some different form. So what we find in any case is that the majority of you are focusing on effectiveness with attractiveness being the second or the runner up in terms of the majority. So what we will do is we will look at a few examples of how to measure these two metrics effectiveness and attractiveness. So how to measure so as I said earlier these are called instruments okay so and these are instruments is a technical term in ET which stands for the instrument that you use to measure the effectiveness of your study. So in this case it could be tests, it could be surveys, it could be interviews, it could be observations and many other instruments. So we will only talk about tests and surveys. Tests are commonly used instruments for measuring the telemetric of effectiveness. So since most of you are RQs are on the metric of effectiveness you will be using some form of tests okay. So improvement in learning of some concepts or skills and in the next slide we will see some do's and don'ts for setting up of these tests or use of these tests for measuring the telemetric of effectiveness. The other commonly used instrument is our surveys which are for the telemetric of attractiveness where what you are measuring is enhancement in student engagement, motivation, confidence etc. So surveys once again we will see some examples of how to construct these surveys. So by and large these would be your most commonly used instruments some form of testing for improvement in learning and some form of survey for enhancement of some engagement or motivation and then these are other techniques which go a little deeper than tests and surveys for identifying what is going on in the classroom or in your course and getting into details of these other instruments is beyond the scope of this particular workshop. So we will just look at these two instruments tests and surveys. So for example when we want to measure the learning of a concept so we want to measure effectiveness of learning. So your telemetric that is being chosen is the effectiveness of learning that's your chosen metric. So when you want to measure effectiveness of learning the evidence that you need is the improvement of learning of a specific concept. You want to find out how much improvement has happened in the learning of a specific concept and what is the data that you will collect in order to determine this improvement is the performance on a test related to the concept before and after the treatment. So if you only collect data after the treatment you can only know the level of the learner. You can only say that they are 80% of my students are able to solve this problem. You won't be able to say what is the improvement. So if you want to know what is the improvement you have to have some quantification for the before the treatment also. You want to say that look in a normal traditional classroom the performance has always been up to 60% whereas after I did this wiki base activity I found that the performance on the same concept for a different group of learners which are equivalent is now 80%. So that improvement you want to be able to compare before and after the treatment or between two groups of treatment and in this case what is the instrument that you have to choose? The instrument that you choose is a test and the important thing to note here is that it's some kind of a standardized test. It's not just a set of questions that we write up because we have to ensure that these questions are appropriate for this improvement that we are measuring. These questions are also aligned to the concept that is being talked about. So it's important to design this test carefully and not simply have for example you don't want to say that I am measuring improvement of programming ability and simply give some recall questions in the test which in which case the test is actually not measuring that improvement of the programming ability but it's measuring something else. So it's important to keep in mind that these have to be in coherence with each other aligned to each other. So the standardized test having conceptual questions related to the concept that's important and as a result using a typical final exam question containing only recall or described questions is not useful when you want to measure this improvement of learning in a specific of a concept. So you want to create a test which ensures that the test question is actually measuring this concept and you can get data about improvement using that test. Another example of measuring learning of a skill. So once again here the chosen metric is effectiveness of learning and since it's a skill you want to find evidence of improvement of the skill. So in this case let's say the skill is programming ability. Others have been teaching programming in one way you have come up with a different way of teaching programming and you want to find out whether what is the improvement of the programming skill as a result of your treatment. So some data that you could collect is performance on a programming or a debugging question. How you will measure this is by asking students to write a program to solve a specific problem or you could give an erroneous program and ask students to debug the code till it gives a desired output. So these are standard trace the code debug the code or write code type of exercises which can be used to generate data on how are students performing on this question. And what you could analyze now are the number of errors in the program classify them as syntax errors or logical errors could also analyze how many errors was the student able to find for the debugging whether they were valid errors or whether they were not actually errors in the code. So what are called false positives and the last thing to keep in mind once again is that what you ask students to the test that you create has to be aligned with what you want to measure. So asking recall or understand level questions such as what is the variable is not the correct test for measuring the improvement of a skill such as programming ability. So if you say I want to measure improvement of programming ability your questions have to deal with actual programming not the theoretical understanding of a program. Coming to one more example suppose you want to measure student engagement. So engagement we know now that the chosen metric is attractiveness and so you want to generate evidence of student engagement some improvement of their engagement or the quantify the engagement at the very least you want to be able to say that how much engagement is happening. So the data that you want to collect are students perception of their interest in the course. For example the question that many of you are thinking about does do these active learning strategies improve their interest in the course and other measures which some of you already mentioned are participation rates how much time do students spend on task and so on. And these can be measured using a questionnaire. So questionnaire is a form of a survey. So you can use a questionnaire to measure this for example on a Likert scale. So what we mean by a Likert scale is where we have questions to which the respondent has to answer from strongly agree, agree, neutral, disagree and strongly disagree. So you give questions so that the participants or the students have to select between these things these four or five values and then count the number of students who have said strongly agree versus strongly disagree or strongly agree and agree and so on. So this form of creating questions is called a Likert scale and you could use a questionnaire with Likert scale questions to determine the degree of student engagement for your particular chosen idea. It is one more thing to be kept in mind while constructing these perception questionnaires. So often we are tempted to ask questions like did you like it and put a rating scale along it saying that strongly agree, agree, disagree and so on. So now this simply asking questions of the type did you like it or dislike it is not very useful information as far as a research study is concerned. What you could do is simply take one more step to make it a little more precise and ask questions related to what exactly you want to measure. For example, instead of saying did you like it you can say that did you find that your engagement or is has increased as a result of this TPS activity or you could ask a question like do you feel that you have learnt more as a result of being asked to submit these assignments. Similarly, do not ask a question like did you find TPS interesting. Again you can ask many specific questions related to what you want to measure. So you could ask for example what did you do, how much did you participate in the think phase, how often did you talk to your neighbor in the pair phase. So these could be specific questions that you could ask instead of simply asking did you find TPS interesting. And the last thing to keep in mind is that do not want to include only open descriptive questions such as what is your impression of this workshop and have the participant write one paragraph on their what they feel were the benefits and disadvantages of this workshop. So these descriptive questions are hard to analyze, they are useful but they are hard to analyze. So it is better for a simpler questionnaire to use a scale or a rating or a ranking such as the Likert scale that we talked earlier. So now we will just do a few of these activities where you can decide whether the paragraph that is described whether it is an ET research study or not. So in the first activity your colleague is saying that I will prepare interactive media content and animated videos using Moodle the student can access the content in order to make the interactive session, the student will be more interested in interactive. So the animated videos will persist in their mind and the concept will be easily understandable. So the question to you is that is this an ET research study what your colleague is saying. So please vote in your RCs yes or no and the RC coordinator can convey the majority vote through the poll. We have most of the responses in and an overwhelming majority has answered no while a small minority about 6 have answered yes. So let us look at what is happening here. The point to keep in mind is that near development of instructional material or a strategy is not an ET research paper even if the material is based on an innovative idea. So to be considered as a research paper you need to show that the material has resulted in improvement of student learning or engagement. So if you go back to this what your colleague is saying your colleague is preparing some new material interesting material may be also innovative but the colleague is not saying how it will the persistence in the mind will be measured or how the concept understanding will be measured what will it be compared against. So there is no clear RQ for what your colleague is saying here hence it is not a ET research study. So it is important to keep in mind that you have to show that the material or strategy has resulted in improvement. So there have to be research questions which are dealing with how much learning has improved or how much engagement has improved. Simply developing the instructional material is not a paper that still an useful effort but only if you take this additional step of measuring the improvement does it become a ET research paper. Go on to the second activity now your colleague is saying that the purpose of the study is to use Moodle and study the motivation behind its use by participants. So activities such as presenting information managing course material and evaluating student work through Moodle quizzes were all done using Moodle like we have done in this workshop and then instructors were asked the benefits and barriers to using Moodle. Is this an ET research study? Once again participants have to vote yes or no and coordinators have to convey your RCs majority vote through this poll. So we have about 28 responses and the overwhelming majority of responses is yes for this particular question and there are few no responses. Now if some of you have simply responded saying that ok this guy is asking two questions if the answer to the first question was no then the answer to the second question must be yes then you are in for a surprise. So the answer to this question is also no. I will give you a second to think about why is it no? It is almost yes. It can be made into a yes if some more work can be done but right now the answer to this question is also no. So even though it seems that this your colleague has done something new because the colleague is saying that I am going to study the motivation behind its use I am going to ask instructors about the benefits and barriers and you can imagine that there are research questions here which the your colleague is going to have instruments survey instruments to measure this motivation and this benefit. Why is it no? The answer is no because of this reason use of a ET tool in a routine manner is not a research. So this is again a routine use of Moodle. So we routinely use Moodle for uploading our slides assignments and students also are routinely using Moodle in order to submit the assignments take the quizzes and so on and so forth. So use of any tool in a routine manner. So for example it might make sense if I were to substitute Moodle with power point or any other elementary or commonly used technology and were to ask the same questions you would easily be able to say no no that is not a research study because everybody is doing that or a large number of people have already seen what the benefits of doing such a study are. So and the second thing is that there are no strategies that are being used there. So in order to be considered an acceptable research paper you need to implement an innovative method of using the tool. So this is important so if you were to use Moodle in an innovative manner beyond the routine manner so you have to go this one more step from a routine use go to us identify and implement an innovative use then it becomes a research paper to achieve a teaching level. For example you can use Moodle to create a game that allows students to learn a concept and then the teacher can check how much collaboration occurs. Now this is much beyond what your colleague has said that I am going to upload the information and evaluate the students work through quizzes. So going from routine use to innovative use for a specific teaching learning goal which many of you are already halfway towards in terms of the strategies that you have proposed in your lesson plans is what makes a study or a report into a research paper. This is the last activity your colleague says that I use peer instruction in my class I post questions students debated enthusiastically I saw that they were engaged I also think that they learned since they gave correct answers is this a ET research study. So once again vote yes or no and coordinators convey the majority vote through this poll. So again we have about 28 responses in where we find that the no is slightly more than the yes we have about 16 no's and 12 yeses 30 responses we have 17 no's and 13 yeses. So let's go back to looking at the question and see whether what could be the point here I saw that they were engaged I also think that they learned so these are the key I saw and I think. So what's the point here the point is a report of a strategy that you implemented is not a research paper even though it may contain a good idea. So this is again one of those which is currently a no but which can be made into a yes. So you need to show details of why your strategy is different from existing research on that strategy right. So peer instruction for example is a well-known strategy many people have already published on this strategy saying that peer instruction allows question encourages students to debate it ensures that students are engaged and it also leads to better learning. So there are lot of studies which are already published on this topic. So your colleagues work currently is in a no and it can become a yes only if there is some novelty in what in the way your colleague has used the strategy. So you need to you need details to show why your strategy is different from the existing research on the strategy and you need to establish systematic evidence that it works beyond saying that my students seem happier or learning. So there has to be a lot more effort to quantify what do you mean by I saw that they were engaged or I think that they learned that's one thing and the second thing is there has to be also comparison with what have other people already said about peer instruction. Am I the first person to say this or is this result already known in the literature that's an important thing that one needs to do and you will be doing some of these for your own idea in the lab that follows okay. So let me do an overall summary before going on to what is expected in the lab session. So the key points of the session are what is not ET research. Near development of instructional material is not research even though keep in mind that development of instructional material is a very valuable activity that we undertake as teachers. So it is a very useful activity as something to share with other ET practitioners, something to put up on your website, something to release under creative commons, something to make available to the world at large. However the development of the instructional material is not a research idea it's a research practice. Similarly use of an ET tool in a routine manner is not ET research. So what you need to do is go beyond the routine use and come up with an innovative use and then conduct a study about it in order to make it research. And the third example that we saw report of an application of an ET strategy is also not ET research. So suppose you apply TPS in your classroom and you find that TPS help your students to be more engaged or improve their learning for those students who use TPS. Again this would come under the routine category of routine application of an ET strategy. So it is already known that TPS does improve student engagement and student learning. So simply repeating that as an application of an ET strategy is not ET research. You need to go one extra step to identify what is something different that you have done with that strategy. So some features of ET research are you want to identify research questions that we saw as examples, many examples we saw. You want to use establish tell metrics for evaluating your study and you want to gather data using appropriate instruments such as test surveys, tests and surveys are what we have seen so far. So the key point here is you want to go at least one step beyond routine use. So how do we do that? So one method of doing this is to read papers on what others have already done. So when you read papers on what others have already done you will come to know what the level of whether what you are saying is routine use or whether what you are saying is innovative use. Another method is to think of slightly different uses. So if most people are using Moodle for let us say purposes A and B and if you come up with an exploration of can I use Moodle for purpose C it may be again an innovative or a beyond the routine use. Similarly if most people have done studies on effectiveness of TPS in a classroom and if you are now going to see that is TPS effective in this SRC mode where you are I am here and you are all in different centers and does TPS happen as effectively in the center as it would happen if I were in the class in front of you that again would be beyond the routine use. So the key thing for going one step beyond the routine is to look at what other people have already done and that is what we will do as part of the lab. So what you will do in the lab is to refine your study. So there is one research paper that is uploaded on Moodle there is not two there is one research paper that is uploaded on Moodle and there are a few simple questions about that paper that is to help you to get started thinking about what are RQ's be a little more clear about identifying metrics and instruments and so on and once you have done that part you will go on to identifying the tail matrix for your study. So many of this you have already done right now but you will refine that and then you will end by identifying two papers which are similar to this idea that you are proposing so that you can identify whether your idea is a routine idea or whether there is some novelty associated with the way you are intending to implement your lesson plan. So that is what we will be doing in the lab and this assignment is called refine your EET research study which is what we will do immediately after the tea break. So what happens next in this suppose you are interested. So it is not essential that all of you be interested in carrying forward your lesson plan idea into an EET research study. So as I said in the beginning it is perfectly okay if you are simply going to execute your idea in your class and be happy with informal perceptions of improvement in learning and engagement that is perfectly acceptable to stop at that point. So especially if the idea is a routine implementation of a known strategy or a known tool. So for those who are interested in executing their idea and pushing it a little bit more and conducting the corresponding EET research study there is a forum that we have created on the QEE model that is called mission 2015 going from EET practitioners to EET researchers. Another thing that you could do is to attend the conference on technology for education. So if you Google T4E 2014 it will take you to the conference website. The current year's conference is being held at Amrita University in Colombe December 16th 18th to 20th December 18th to 20th. So even though you do not have a paper in this conference or you may not be able to complete your study in time you could still come to the conference and talk to us at the conference which will help you to refine your study and submit it for the next year's conference which is T4E 2015. So simply attending this conference itself will be of great use to you both from the learning more EET strategies perspective as well as from the how to conduct an EET research study perspective. And for those of you who persevere with registration and attending the conference what we will do is we will provide mentoring for executing the research study in your classroom. We will provide mentoring for how you should go about data collection and analysis and we will help you write a research paper on your idea for the next T4E which is T4E 2015. So we have already created lot of templates to help you with this. Right now I am not going to talk about any of these templates. If you are curious you could download and use them from this website. So what we will do now is to break for tea and go to the lab for doing the lab assignment called refine your EET research study.