 Let us reassemble back. I hope all of you have finished tea and come back into your centres and let us now continue with this activity, where we had stopped before tea is we were exploring this question what is in an research study. Before that we had looked at what are not some features of a research or what are some features that should not be there or what is not a research study and now we are looking at what exactly should be in a research study and what you were going to do is recall the lesson plan that you wrote yesterday specifically what you would do and what your students would do and you were going to write down what you expect to see in terms of statements like I think my idea will be successful if I find that my students are doing something, feeling something or if I am doing something and I am feeling something and so on. So, I will just give you one more minute to write this if you have not done so yet and then we will move to the pair phase, only write these answers in your own notebook individually. Okay, now what you can do is pair up with your neighbour and look at your neighbour's answers and do two things. One is suggest more statements that your neighbour could add to her or his answer that might indicate success of the idea and most importantly together select any two statements so you should have four or more than four statements your neighbour will have an equal number of statements so select any two statements and determine how you will measure the extent of success. If some idea that you wrote on the previous slide if these are not measurable you can go back and change these but we are now getting towards a more systematic scientific data collection and evidence procedure. I will give you example of what is not measurable if you said something that my students are feeling happy it is that is really not a measurable quantity if you say something that my students are doing well it is not precise enough to be measurable so first write down statements and then select any two statements which have some measureableness and write down determine how you will measure the extent of the success so take some time this is the most important part of trying to write down your research questions so do this carefully and then we will go to the next phase so let us take about five minutes five to seven minutes to do this phase. I am just going to put the previous slide because some people have asked to see this slide so this was the think phase this was the pair phase and in the share phase what you can do when you are ready is share your answer with your colleagues so I would request all the RC coordinators to actually do an in class discussion at this point and only then share the most common answer through a view chat so in your RCs discuss what is to be measured and how it is to be measured let us look at a range of answers so please do this discussion for three or four minutes and only then post your answers not just what is to be measured but what and how if you send answers like my idea works if I feel good that is not measurable so remember this is the most important part select any two statements that are measurable and determine how you will measure the extent of success it looks like many of you are sending reasonable answers about what is to be measured but the how is still not coming through precisely so let me give you a few examples one of you say that I think my idea will work if I am able to lead the discussion to a higher cognitive level now okay you may be able to do it but how will you measure and report that in a research study so when you think about how to measure what you have to do is try to see what data you will collect what will you measure how will you actually go about doing the measurement so some of you have said that I am going to do pre-test and post-test you we want measurement metrics like students at least 80 percent of the students should do the assignment with an average of say 80 percent or greater similarly some of you have said you'll measure students attendance and performance in midterm that's also okay what will you measure let me read a few others so many of you are looking at students performance on test that is definitely one possible metric how many students are able to solve problems in different levels of looms taxonomy that is definitely one measurement that you can use and what their metric for what to measure was students ability to solve higher-order problems so try to be specific don't just say I'm going to look at their final exam marks let me look at a couple of other interesting points if you say that you want to you you know your idea works if students are confident you have to find a way to determine how to measure the confidence yeah I would say the most common answer is students performance in tests and exams but you need to get more specific so I think let's move on and try to look at specific research questions that you can write okay so if you can stop sending your responses and let's look at how researchers have actually attempted this problem you will get a chance to come back and refine these metrics as we go on in this session so a research study must contain what is called research questions after you do the analysis of related work and find out gaps and try to identify where your study fits the gaps you now have to start writing research questions for your study which are actually issues that you will investigate so a research study contains the research questions and its answers accompanied by evidence for example if you have identified a specific teaching learning problem such as students ability to solve higher-order cognitive tasks then the research question should be about how and does your solution address the problem so let's look at some examples at this point so please stop sending your responses through chat because we'll now go beyond exams we'll go beyond students performance in exams and try to write down research questions which you can actually put in a paper which you can use as a research question for a study so a research question should have certain characteristics it should express a relation between variables two or more in specific context so our animations effective is not a valid research question because even though it talks about animations and effectiveness it's not at all specific so we'll see what are not research questions and then we'll see how to write a good research question the second point of a research question is that it should be stated clearly in the form of a question not as a statement so you should not say that the purpose of my study is to gather data to support my idea of showing animations that is not a research question that is one of the steps you will do in your research study and finally a research question this is sort of the most important point that we also talked about in the think-per-share activity a research question should imply possibilities of empirical testing there should be a possibility that you do an experiment gather data analyze the data make inferences and draw conclusions if that possibility doesn't exist then the question is not worded correctly so should one use animations in primary schools is not a research question because it's not possible to answer this word this question should using empirical data should usually imply something very moral and very opinionated and judgmental so a good research question would be something like are animations more effective than still visuals for conceptual understanding of electromagnetic fields this is just one example so let's see if it satisfies all these criteria does this question express a relation between variables so there is a variable of the mode of visuals animations and visual still visuals are two values of this variable that's one variable and effectiveness in terms of conceptual understanding is another variable is there a specific context in this question yes its conceptual understanding of electromagnetic fields is it stated clearly in question form definitely it poses a question which of the two methods is better for that context and finally is it possible to do a study an experiment an empirical study to test this well it is possible if you think about it some students can be can work with animations others can learn using still visuals and you can do some testing in terms of conceptual understanding of electromagnetic fields so the question itself should tell you how to perform your empirical study so once you have seen the features of a research question let's now look at three different examples of actual research questions in published papers in educational technology and once we see the examples we'll go back to your example and you'll rewrite your think per share question but first let's see three examples of our cues and let's actually try to look at whether these characteristics of a research questions are present so this is the same abstract that you saw earlier when you were looking at novelty and positioning if you want you can read it again and then look at the research question this is there were one or two other questions but I've just included one question here it's the same example that you worked on the TPS the question is does a three-hour blender training which was the novel solution does a three-hour blender training improve the mental rotation ability of first year undergraduate engineering students so if you notice the teaching learning problem that this paper was trying to solve was the improvement of mental rotation ability in fact they even say in the first line why that ability is important then what they do is they did the positioning they talked about existing approaches and their approach we developed a three-hour training model the research question they ask at that point is does this intervention of ours does the solution idea of ours indeed improve mental rotation ability for a very specific population so your research questions really have to get as detailed and specific like the one here so I'll show you one more example that's related to think per share and since you know the technique you might appreciate it again there is an abstract and there are two research questions so I'll give you three or four minutes to read this so this was another research paper about think per share in a large cs1 programming class and what the researchers were trying to investigate is does learning really happen using think per share so take a minute read these and then we'll look at the research questions so in this study the authors ask two research questions one is do tps activities lead to increase conceptual understanding and application of programming concepts so again there are two variables which are being measured one is the implementation of tps and the other is the conceptual understanding as well as application they pose it as a question and this definitely can be experimentally investigated so that's the point you want to try to see how is it possible to do an experiment here the abstract tells you so what they did is what's called a quasi experimental study what that means is that there were two groups one group were was taught using tps and one group was taught without tps and what they found is that students who learned via tps performed statistically significantly better on a post test than students who learned the same concept via lecture so they did an actual experiment tps versus no tps and measured through a post test conceptual understanding and programming they had one more research question here they wanted to find out what are students own perceptions of learning how did students feel about learning via tps so for that they designed a survey and a focus group interview to understand student perceptions and they found that majority students agreed that tps activities helped improve their understanding they did not ask the students did you like tps but they they specifically asked the students did you find that tps helped you that tps helped improve your conceptual understanding and application one last example here I am not showing you an abstract but I just want to show you three research questions and the study that these people did was about showing visualization so this is a topic we dealt with in great detail yesterday about simply demonstrating visualizations and doing some activity along with visualizations so these researchers actually performed a study where one group learned only by viewing the visualizations this was also a computer programming course another group viewed the visualizations and did a predict observe uh correct kind of activity exactly the kinds that you did yesterday and the questions they asked was does prediction activity with visualization lead to higher levels of learning than simply viewing the visualization so they are contrasting two conditions one is prediction plus visualization and one is simply viewing they also ask about student perceptions about learning and now they get a little bit more into detail they ask about differences in learning and perceptions between high achieving students compared to low achieving students so what you can do in your own research studies is ask broad questions like did the method work for the class did it help improve problem solving ability and in addition you can also try to investigate specifically for whom it worked better did it help the high achievers more or did it help the low achievers more did it help the more motivated students or the less motivated students these are all important specific things that you can actually investigate in a research research study so there's a chat question at this time which says how to measure students perceptions so we'll come to that in a few slides we are still looking at the questions themselves how to write meaningful valid questions we will discuss how to measure perceptions and we will also discuss how to measure these different levels of learning and exams are not they're not the best way to measure learning okay so here is a very interesting question from Walchin Institute that if many people are using TPS in CS or other domains will it be considered to be novel and I think they are asking about this particular question this is an important question because now if you when this paper was written they found that there was a lack of research based evidence in CS education on the benefits of TPS but since then about three papers have been published on using TPS in a large in computer science domain classes so if you want to apply TPS to a computer science class you have to find a new angle to your research study for example you can ask the question does TPS help low achievers more or high achievers more because these people have not looked at it from that angle but if you simply replicate TPS in another programming or data structures course you will find that there already exists research so it's not going to be considered as novel so this is the game because once people start thinking and doing studies on a particular topic a lot of people will work on it so to establish novelty you have to find a new angle to the same problem that is valid you can perhaps ask how long should be each phase so that the phases are effective you can ask what happens in the pair phase to make it effective and so on so look for angles that are not there in the existing papers okay so let's now go back to your own activity and let's do this as a fastest finger first or a brain race what we called so participants revisit your answer to the TPS activity which you did about 15 minutes ago and write one research question for your research study pass along your answer to coordinator and coordinators please do two things if it is not phrased as a question if it's a goal statement or if it's a should I do this statement disqualify that question if you send us RQs that are not phrased as questions or that are phrased as should I do this questions then we'll disqualify the question at our end and the first valid RQ please convey through the view chat so go back to your study research study and the TPS activity that you did at the beginning of this session and rewrite it as an RQ I'm going to put back the slide as to just for 30 seconds what is an RQ it has relationships between variables in specific context stated in question format and must imply possibilities of empirical testing this was a valid RQ here so now go to your question your study you can go back to your lesson plan idea and write one research question from your lesson plan idea please do this activity now we will come to how to measure student perception and all after some time look at your lesson plan and write RQs based on your own lesson plan okay so let's look at a few questions that you have sent I'm going to comment on some of these questions because I think you're all getting to some point so let's look at one question it says does flip classroom activity enable conceptual understanding of equilibrium of force system among students so again it may be a valid question in terms of measurement but what you have to be sure of is in what way this approach is novel and why flip classroom is so important for forces for the topic that you're thinking let me look at a few other questions some of you have asked does visualization with prediction of output in programming language improves students ability to write programs that's actually a valid question it can be measured it's in fact you will see other studies like this so you have to be a little careful in terms of positioning but it's specific enough to be a valid RQ because you are actually giving the idea is very precise visualization with prediction of output in a programming class and what is being measured is students programming ability on the other hand if you ask a question such as do rubrics improve student performance in computer science it's too broad you have to say rubrics for what it's not computer science domain is too vast and just rubrics in of itself you won't be able to do a study which encompasses all of computer science domains here is a question which may be valid what is the impact of TPS on low achieving students actually you can try to investigate that within a specific topic then it will become a valid question okay here is a question what is the impact on student engagement of combining animation with TPS so this is something we haven't seen yet among the answers that you've already given the researchers want to study student engagement from two perspectives animation combined with TPS and they'll have to find ways to measure engagement we'll come to that in a minute how to measure things but you want to now start thinking in terms of what are the various things you can measure okay let me take a couple of doubts here so I'm going to stop looking at your own research questions that you're sending and instead I'm going to look at a couple of doubts one doubt that was asked in the beginning of the session was can a teaching strategy effect vary from subject to subject from student to student and if so how can it be a research question can we compare one student against the other so your point is valid that a teaching strategy may have a differential effect on one student then another student but what you're doing in a research study is actually looking at averages just like in a science experiment you take multiple data points but finally when you do the analysis you look at averages and medians you look at the range so it's something very similar here you can look at average over your entire class or you can divide your class into low medium high achievers based on previous and previous years marks and look at averages in each of these categories usually we don't look at one student versus another student but we look at averages over groups of similar students you can look at one section which is taught in one method average of one entire section versus average of one entire other section which is taught using a different method so most of the times you're dealing with averages over an entire set of participants the other doubt that was asked is is it mandatory for us to develop a research study in the ET field I would like to be user only and this is perfectly valid if you only want to be a user of ET strategies please do go ahead encourage you to use it this session is if you want to take things one step ahead for all the benefits that we discussed in the beginning then here are some systematic techniques to do so if you choose to be a user definitely stay there but even then it would help if you can gather some evidence using these methods that we are discussing now because the moment you have evidence you can discuss it with the head of your department or with the principal of your college saying that I implemented a new technique using ICT and I also gathered evidence to show that it works from angle of student engagement or student learning it doesn't have to become a research study or research paper but just the evidence and the inferences often make your case very strong okay let's go on a little bit more now and actually look at another metric which is important in your research study so now we have an idea of how to write research questions we have two more things to do one is how do we implement the research study how do we conduct it and the next one is what how do we actually do the measurements so in terms of sound in terms of the procedure there is a metric but there is a criterion that your research study and paper must have just like novelty and positioning it should have what we call a soundness what is soundness it's essentially steps to show that you have implemented your solution carefully because like any scientific experiment if your steps are not implemented carefully then it ceases to be valid the inferences will not be considered to be strong so let's look at one or two examples of soundness here and here is an activity quick activity just say yes or no in the chat suppose you implement the following method you or your colleague has implemented a method where the first step is you implement an innovative strategy i'm going to call this as strategy x you then conduct a test to check if students have done well after learning with strategy x and you claim that since students did well on the test assume all students many students did well on the test okay make that assumption you claim that the strategy x works so suppose you implement a research study and write a research paper like this does this method establish that strategy x really works it's a simple yes no question many of you are saying yes some are saying no okay so from the votes it seems like about 80 percent of you think that this step these steps show that method x strategy x actually works if you write a paper like this let me tell you the answer the referee decision will say that paper is rejected and they'll say that the method is not sound enough to establish the claim so if these are the only steps you implement it the method these steps are not sufficient to establish that your strategy your innovative strategy actually works this kind of a method is called a single group post test only design what it means is that you have one group of students to which you have implemented for which you have implemented strategy x you do a test at the end that's why it's called post test and that's your research design so here is the next question for you do this as a pair activity and you can send us the answers over chat why is this research design not sound not sound in the sense if you implement these steps why is this research considered to be not sufficient to establish that strategy x actually works so in pairs come up with one reason why the above steps of research are not a good method yet to establish that strategy x is effective what's wrong in this assume that you give a really good test and assume that your students actually do well on the test your test does have questions on application and analysis and students actually do well but still this is not a good enough method okay some of you are getting the idea okay I think most of you are getting the idea here to summarize like one of you said there can be many other reasons other than strategy x which led to high scores on the test so let's look at what are the problems here for if you do a single group post test only research the outcome that is the result could have occurred due to a reason that's other than the treatment some completely other reason and you won't know if it's due to your strategy x that you got a good result or some other reason because there is no comparison to a group that did not receive the treatment the result could have existed even before you did your strategy x treatment means what you implement in class the intervention your solution you never tested the students before you implemented the strategy so maybe they already knew the results and you have no control over all other influences maybe they went and watched a youtube video maybe they talked to each other so you don't know why exactly you are seeing positive results so if you do something like implement an innovative strategy do one test at the end that is not sufficient to establish that your innovative strategy works so in order to have a sound method you have to do one of two things the problem that there is no comparative comparison to a group that did not receive the statement can be solved if you can compare results of your method your strategy to some group of students that did not work with your strategy this is a two group comparison it's sometimes called a controlled group experiment this is what is called as a quasi experiment that we saw we saw those words a few slides ago so if possible in your class if you can compare one group of students let's say you have two sections let one group work with visualization plus prediction and let the other group work with demo of visualization for one topic for another topic you can reverse the order so that you are being fair so see if you can do a comparison to some other group suppose that's not possible there is one more thing you can do you can if you only have one group you can compare the performance of the group before and after the treatment this is called a pre-post test design it's not as strong as the previous one but it it's still better than the post only design so after a single test you can't claim that the strategy was successful because of all these problems and the solutions you would implement possible solutions are here okay so this is one way of ensuring that you have soundness in your procedure of research there are some other ways if you cannot get two groups at all one possible thing you can do is implement your solution for one topic let's say you only have one section and you can't do two groups what do you do then you implement your new innovative solution for one topic and in a completely unrelated topic for the same group of students you can implement the other method and repeat this a couple of times so that is called an ABAB design so there are several ways to implement research and these go under the names of research designs today we'll only talk briefly about it if you choose to continue doing research in this field then we'll show you will give you more resources and more references as to how you can implement these different research designs the final thing that you want to be careful about that your research study and papers should have is something that all of you have already discussed you need evidence to support your claims claims are the conclusions that you draw from your research so you need data to show that your solution works as claimed which means what you have to do is decide what to measure in your study which you'll show as evidence and the data that you gather should really be aligned with the goal of your study so if your initial goal was to improve student learning the data you gather should be related to student learning if you gather data about did students like my strategy then your goal should have been something like student interest match your research goal which was there in the research questions with the type of data that you gather so let's go back to our example of mental rotation and see if that example has these two points soundness of procedure and evidence to support claims so I'm going to put back that abstract and let's see where in the abstract we see that the researchers have implemented a sound research procedure and shown evidence to support the claims let's take a look at those two so in this abstract you can see that towards the bottom they said that they use something called a van der Berg's mental rotation test for pre-test and post test so they have implemented a pre-post test design not only that they've chosen to measure mental rotation using an already established test that also shows soundness then they did an analysis a detailed analysis and found the results to be statistically significant there are tests statistical tests that you have to do you can't simply say that my students learned well so let's say your students average was 75 percent how will you conclude that 75 percent means good you have to do some statistical tests between the pre and post some of these are you may have heard the terms like t test or chi square test those are all different statistical tests that you have to do and your results must be significant on statistical samples of the averages and then you can also measure something called effect sizes so all you can see from here I know you may not understand exactly what these mean but the purpose of this example is to show you that you have to actually come up with precise quantitative analysis to show evidence in support of your claim only after these evidences do the do these authors claim their method of blender 3d training works for improving mental rotation okay so in your case now let's look at what to measure so we'll come to the final part of this session and many of you have been asking what should I measure and how how should I measure different things so let's take a look here so typically technology enhanced learning metrics can have different goals and you should focus on one or two of these please don't attempt all these metrics and these are in order of usually what teachers are interested in you can choose to measure students improvement of learning of concepts of skills or skills this usually goes under the name of effectiveness this is the learning outcomes that we have been discussing so far or you can choose to measure what's called attractiveness which is students engagement motivation interest confidence and so on you can also choose to measure something like how your solution scales to larger contexts or different types of learners I'm not very sure if in any of your many of you are in this category but there is a possibility that your research study might simply be to show that the technology enhanced learning solution that you have developed the goal might be to scale to larger numbers so if you are working in online education scaling is one of the important metrics if you're mostly doing face-to-face then most likely your metrics are going to be the first or the second and finally for universities and institutes and colleges they may want to measure efficiency in terms of saving time or money or resources as individual faculty members this is perhaps not the most admissible metric so I've put these on the slide to show what are different metrics that people use to measure effectiveness but I would suggest that you concentrate on the first and second don't attempt too many of it so one quick polling activity at this point remember the research question that you wrote for your lesson plan idea some time ago so revisit your research question and simply say which technology enhanced learning metric did it correspond to best improving learning enhancing engagement scaling or saving time so just vote on your question is your question really related to learning effectiveness or engagement and let's look at what percentage of people had which category actually let me do one thing let me just create a quick poll it will be interesting to see I'm just going to create a poll send it on poll let's take a look at the percentages okay out of 110 centers it looks like about 70 centers have learning effectiveness as your major metric there's no right or wrong answer this is more to see what you're doing about 20 centers say that engagement interest etc is the major is the main metric and a similar number says scaling I'm not sure because I didn't see any research question that you said related to scaling so let me just quickly explain what this is let's say that the reason for you to implement some technology based solution is to reach out to larger numbers so for example if you use chat forums in Moodle in order to get larger number of students to participate that you can consider perhaps you can consider to be scaling but that also indicates engagement so if you say that your answer is scaling I would ask you to revisit your question a little more carefully I'll give you one example of scaling this workshop that you're seeing ET for ET which is being done on under the teach 10000 teachers platform the major goal for this workshop is scaling we've done a number of face-to-face workshops like this but in a face-to-face workshop we can address maybe 50 or not no more than 100 participants but because of our technology enhanced environment AVU and poll and interactive chat and and the two-way audio video we are able to reach out to 4000 participants so our goal here is scaling it's also learning and engagement but in a classroom most likely your goal is going to be either learning or engagement or some mixture of the two okay so let's go back to the slides and look at the metrics a little bit more and now ask the question how do we measure what do how do we measure learning and how do we measure engagement so for learning tests are commonly used instrument for measuring effectiveness and learning and let's look at some do's and don'ts for tests and then we'll come back to survey questionnaires so if your goal is learning effectiveness you need evidence this is an important slide request all of you to pay attention here if your metric is effectiveness of learning then the evidence that you need is improvement of learning of a specific concept or skill the data you need to collect is performance on a test related to that concept either before and after or in a two-group manner but you have to make sure that the performance that the test should be related to the concept the specific concept who's learning you're trying to improve what you should not do is give a typical final exam question which has only recall and describe questions what you can do is sometimes there are standardized tests available for a particular concept so for example for C programming or for forces and physics there exist tests these are called instruments and concept inventories which actually have conceptual and reasoning questions related to that concept if your improvement is understanding of forces then the questions in your test should have forces maybe make students draw diagrams and so on on the other hand suppose your metric is improvement of learning of a specific skill such as programming ability then what data you should collect is performance on an actual programming or debugging question you have to make students write a program or debug a program in order to gather evidence that their programming ability improved if you only give a question about programming concepts such as what is a variable or is this a valid pointer it doesn't really doesn't precisely test programming ability so how to measure this ask students to write a program to solve a problem or give a buggy program and ask students to debug the code till it gives their desired output so you have to measure specifically what the type of data you want to collect do not ask recall or understand level questions asking what's a variable this is not sound research for improving skill in your analysis what you can do here is analyze the number of errors that students make in the program and classify them as syntax and logical errors that will give you an idea of students programming ability so going back to what to measure tests are commonly used instruments for measuring learning of skill or concepts but you have to make sure that you measure that your tests actually specifically ask questions related to that concept or skill let's look at the next category if you want to measure students engagement motivation confidence and so on typically we give questionnaires or surveys and again there there are some do's and don'ts so if you want to measure student engagement you need to collect data of students perception of their interest and for that you can give questionnaires should you develop the questionnaire well if you can find existing validated questionnaires please use those why I'm stressing this is let's say you're doing an engineering experiment you try to find instruments that already exist and that have already been calibrated only if you can't find existing instruments do you do your own instrumentation that's the analogy here if you have to do your own instrumentation you have to first validate it you have to test that it's robust and reliable you have to calibrate it and there are several steps to building an instrument before you can gather data so see if you can find existing surveys and questionnaires for student engagement if so use those there exist many on student interest student engagement students attitudes confidence and so on if you cannot find and if you have to build your own then we'll see what to do other data you can collect for student engagement is their attendance their participation rates in different assignments how much time do they spend on task how many of them participate in extra credit activities all those are valid ways to measure student engagement going back to surveys if you have to construct your own perception surveys and questionnaires don't simply ask did you like or dislike it because that is not going to give you fine-grained enough evidence instead ask questions related to what you really want to measure so you can ask students how much did you learn on a scale of 1 to 5 very much not so much or not at all use scales or ratings like the one that I just mentioned don't ask a single question is the method interesting because that that's not very reliable instead ask many specific questions relate related to what you want to measure so going back to how to measure different things we have discussed a little bit about how how do you measure learning and how do you measure engagement there are several other methods they go by the name of focus group interviews observation protocols and other instruments these are beyond the scope of this workshop but if you are still interested in it I will discuss what to do in a few minutes so what we can do at this point is summarize what we have seen so far so in this session the key points we first discussed a lot about what is not ET research and then we looked at some specific features of ET research identifying research questions using established technology enhanced metrics gathering data using appropriate instruments novelty positioning soundness evidence so all those are features of ET research and to help you go beyond routine implementation many times novice researchers in ET ask us so how do I do all this you know how do I learn about all this one of the first steps is you have to start reading research papers and what you can do for that is we will upload an assignment in moodle so what next now we are going to upload an assignment by tomorrow this is a voluntary assignment if you want to go ahead with ET research we will post a research paper and we will ask you questions related related to that research paper and related to your own research study so doing this assignment reading as well as reviewing your own research study is one way to gather more expertise in doing the research this is a voluntary assignment but we encourage a lot of you to do it because if you do it you can also gather evidence for your own idea even if you don't want to write a research paper what next part two if you want to go one step further in executing the idea and if you want to complete your research study gather data from your own class from your own students what I would ask you all to do is register on this forum we are going to create a forum called mission 2015 here you will have other participants who are doing research in this area and you will also have mentors present who can help you with this research idea eventually if you want you can also submit a research paper to the IEEE conference called technology for education I mentioned this at the beginning at least you should attend it and see what are the other kinds of research ideas that people submit and publish what we will do from our end if you decide to go all this way is provide mentoring for executing the research study data collection analysis and so on but first you have to if you want to get the mentoring you have to do the homework and then you have to register on this forum to make your life to help you through this entire process what we have created is a set of templates on our research website so your research study actually has several phases planning conducting reporting and so on so for each stage we have created templates called idea proposal template study planning template and then when you get to the reporting there are other templates called paper planning and paper writing you are nowhere along here yet right now you are really somewhere near the idea proposal and we hope you will get to the study planning these templates are available on our website ET resources there are some other guidelines if you want to do these studies and do these resources use these resources so eventually whether you choose to do a research study and whether you choose to go all the way here or you just want to gather some evidence I would encourage you to go to this web page look at the guidelines look at the resources so that you can bring closure to your research study so what we will do right now is end the session formally here and if you have questions in your from your centers you can do a hand raise and we will start taking questions one by one for the next 25 minutes center 1166 is there a question I had one research question in my mind can we go for predicting the students course for their online for their ongoing semesters and through these newer teaching learning pedagogies because at times we try to find out the ones who are not able to grasp much and from at least from my end I think sometimes it's better to go for makeup classes and remedial actions for these students okay can you can you suggest some methods okay so the question here it's actually an interesting question and let's discuss whether you want to do it as a research study or more in order to improve your own practice the question can I predict students performance based on how they do in these activities and before you get to prediction what I would suggest is you try to gather data and start looking for patterns and correlations so people do research of this kind and some examples are let's say you give peer instruction questions multiple choice peer instruction questions where you have very carefully carefully worded choices if you can find a way to keep data of which students are answering with choice and let's say the same group of students let's say there are two students who are always giving the wrong conceptual answer you know that they are having some difficulty in terms of that particular concept or maybe through the entire course if you're able to identify such students you can definitely conduct additional classes for them or talk to them give them some extra work and so on so definitely you can use these in-class active learning strategies as feedback for you as an instructor on who's doing well and who's not now where it becomes a challenge is how do you collect this data and how do you track these students how do you gather data so there if you have access to technology that's one place you can do it the other thing you can do is let's say you have a think pair share activity get students to write on worksheet think phase and pair phase and give it back to you so based on the quick analysis you're not really grading them in great detail but you're trying to filter who's really having trouble and these quick worksheets that you do in class or predictions that students can do and write it on a piece of paper and hand it in those can help you track who's understanding and who's not so definitely this is a very important interesting point you bring up that you use these active learning strategies as feedback for yourself and right in the beginning of the session we use the words action research this is action research where you gather data and improve your own practice in order to improve the total teaching learning experience there is one more question from my colleague madam ma'am is think pair share is more effective than visualization in a classroom for numerical problems and numerical subjects okay so let me repeat the question and try to answer a class of questions so the specific question here is is think pair share a better method than visualization for numerical problems anytime you ask a question is method a better than method b you have to really go much more specific the answer is I don't know and there is no answer for this so how will you decide look at the purpose for which you want to use either think pair share or visualization look at the learning objectives look at what you want your students to be able to do for every active learning strategy and for every technology tool we discussed in great detail the purpose of using the technology what are its benefits and if those benefits match the needs that you have then it is a good method perhaps you choose to merge think pair share with a visualization we had one colleague who did this she had students view visualizations on convolution of signals this was in a signals and systems course then she gave a think pair share activity where students actually convolve the signals under the signal transformations or you can merge a visualization with a pair instruction but don't ask blanket questions about is method a better than method b because it really depends on what you choose what you want to achieve okay so let's go back and take a question from Amal Jyoti college of engineering yeah mom this is a really the classes you are giving through the session a view is very good and useful and informative and we have already started this eps in our current going semester and it's a response is very good from our students and the idea of writing a research paper based on the methodology you suggested that's really great and I would like to ask a question a clarifying regarding writing a research paper because the current semester if we are implementing any of the methods like a tps wiki or a screenshot or a flipped classroom method can we make a research paper by taking the feedback from students and also conducting a collecting their perceptions based on the performance in assignments as well as test papers and they perform overall performance okay so the question is if we do tps or use wiki and all that can we write a research paper based on students based on feedback related to student perceptions so we discussed this today and the idea the quick answer is yes but be careful what what do I mean by be careful you have to think about what exactly is the research question you want to ask don't think of the measurement and metric right at the beginning but ask yourself why am I for what purpose am I implementing tps what do I hope to improve by using a wiki once you get that point clear then you can write your research questions which are aligned to the purpose that you want to improve so if you want to improve students learning then think about various measurements you can do that show evidence of students learning and students own perception of their own learning is one metric but it's not the strongest metric so if you can show there is one point we didn't discuss today and that point is called triangulation it means you have multiple data sources to triangulate the same claim that you make so suppose you make a claim that tps help improve students conceptual understanding in a particular topic and you show data from both students pre-post tests as well as students perceptions then your claim is stronger because you have two sources of evidence so think about using multiple sources of evidence for the same claim if you only show standard course feedback saying that students like this technique that is not evidence of improvement of students learning so if you choose to do this further we I what I'd encourage you to do is look at the templates firstly look at the paper called guidelines for writing reporting and planning ET research that's available on the website that I mentioned earlier and it shows it says there what to measure and how to measure and what is valid and what is not valid or kid college of engineering my question is related to 21st century teaching and learning what sir we are learning the pedagology is the what is the difference between this learning pedagology and 21st century teaching and learning pedagology my first answer is we are in the 21st century and we are doing teaching learning methods that improve student learning and teachers teaching so there is really it's really one and the same thing so I that was really a joke but you I think what you are trying to ask is you have seen this word 21st century learning you may have seen this phrase in certain places and what modern day educators and researchers ET researchers are trying to do is identify certain specific skills and items that is really important for students to learn today so to give you one example about 30 years ago what was not very common what was scarce was information the internet was not there 30 years ago and students didn't have access to information the only people who had access to information were experts and teachers so information was what we wanted our students to get to know better knowledge and information but today all students have ready access to content and knowledge and information but what they need to learn is how to sift through this information how to make sense of this how to apply this information so today the focus should be really on how to help students acquire these skills given that there is all such information so if you see the phrase 21st century learning or 21st century skills it's coming from this perspective and even as domain based teachers engineering and science teachers in a classroom what we want to help our students do is apply the information that we are teaching in the classes and there and the the information that students are learning in order to solve problems okay center 1080 GRIET actually I have some previous experience in education technology and also I would like to do let us say I would like to do research basically we are located at Hyderabad so can we register a phd in ET technology in IIT Hyderabad or do you have any provision madam actually yeah it is very difficult because let us say I have done my phd in electrical engineering so we can only do research in either a guide or do research or register in only electrical engineering this is a new terminology let us say somebody want to do research in ET technology maybe with IIT so okay let me answer your question so the question here is what are options to do research in educational technology and especially at a phd level so right now at IIT Bombay we do have a department it's called an interdisciplinary program but it's a department of educational technology where we do have phd research scholars they have to go through the standard procedures of taking gait writing an entrance test doing an interview and so on but these people have masters in either in some discipline like science or engineering or other disciplines like psychology are also welcome but their phd research topic is in educational technology so we do have a formal program here at IIT Bombay similarly I believe IIT Kharagpur also has a similar program SNDT university in Bombay has a program in education technology and so on so there are a few but not too many institutes in India which offer phd programs in education technology Amrita university has a master's program in e-learning at the master's level and so on so this is one answer a formal answer to the question how to do I mean what are options for research but suppose you are a faculty member who's already doing research in electrical engineering and let's say you already have a phd in chemistry or engineering and so on what are options so here I would like to repeat what I mentioned earlier start reading et research papers and start participating in et conferences we have an IEEE international conference within India it happens in different different cities each year and in 2015 it's in December so you have a full year to participate in it and this year it's in NIT Varangal yeah in 2015 look at the proceedings of the previous t4s and try to read them definitely and try to come up with a research study plan it and implement it during this year that will at least get you a flavor of what is this kind of research so again this is to all engineering teachers and science teachers also to other people who are interested in doing such research read the research papers and participate in the conferences that is one first step let's go to Nyanamani college of engineering my question while explaining about novelty when he raised the question how to find the solution for addressing diversity finally give solution for addressing diversity mom okay so again this is a very challenging question we have picked it up I believe in previous chats also how do we address diversity I can only make a few suggestions because this is not an easy question to solve at all suppose so one one solution is based on the techniques that we discussed in the workshop in the past few days suppose you implement something like peer instruction based on a comment that came at the beginning of this chat let's say you're able to track with students are actually floundering which ones are having trouble with the concept if you're you will be able to track using a formal technique like peer instruction because the technique itself will bring out students who are having difficulty then you can have extra classes or a medial classes for them you can firstly just talk to them and try to find out why they're having difficulty there is another technique called muddy points and that is specifically at targeting the issue of diversity so what's done in muddy points is at the end of a class you can you can do this either face to face or in moodle so in a face to face version what you do is at the end of a class you take a box and you have students write down on a chit of paper give them very small chits of paper not a full page to write down one muddy point that they had muddy point is something that's not clear to them and all students it can be anonymous if you want or you can have them write their names students write their muddy point and fold the chit and put it in the box so what you will need to do you or with the help of some of your colleagues is go through the muddy points and classify them into different categories of questions some may be very advanced questions some may be really basic questions and you already now have a categorization of the diverse types of questions in the following class you actually address each category of questions there is a paper related to muddy points in T4 2013 and if you look at the IEEE proceedings maybe we can what we can do is upload this paper on moodle so how do you implement muddy points in a classroom we'll upload this paper on moodle and that's one way to address diversity since this is a question which bothers a lot of people and many of you would have tried it we'll also post this question in moodle forum and all other participants and other centers are welcome to respond how they address the question of diversity that might give ideas yeah my question is related to the selection of the various tools which we have discussed for a teaching learning environment is there any scientific rules formed already to select a suitable tool for the teaching learning environment so a quick answer to that is see first of all the purpose of each task so for example peer instruction works well when there is a conceptual level understanding problem between students and it gets elicited out of discussions with peers think pair share is useful when you want to discuss do problem solving or discuss a larger problem where they are building up building up open-ended problems similarly in case of tools you can look at the learning activity that you want to use the tool for and use the digital bloom staxonomy to map the pedagogical purpose the cognitive level of the task and the features of the tool to use it appropriately for the learning activity that you want your students to do so these are these are scientific decisions I mean at the end these are all scientific decisions for each of the activities for higher-order level learning or scientific tools recommended by these ones for higher-order learning it is recommended for all orders so you see lower order have some tasks or learning activities which you can do even with a simple recall or understand level questions the only thing is what we have taught throughout this workshop are activities which are meant for higher-order learning so in this workshop you got familiarized with activities meant for higher-order learning okay thank you center 1118 I have two questions the first one is in a large classroom how do you take attendance quickly are there any methods you suggest so it depends on how much technology you have access to the standard methods without technology the ones that are used if attendance is compulsory is passing around a sheet of paper or doing roll call I mean these are the two age-old standard methods which people have been using but if you have access to technology such as clickers and all that is another possibility I mean that is not something that we have worked on too much here the second question is what is your comment on using a question bank to prepare the final exam question paper okay so let us try to look at this question a little bit more on should we use a question bank to prepare a final exam paper and if so what should be inside the question bank and so on so it could work in principle provided the question bank itself is large enough and has categories of questions so one thing we are trying to do through this workshop is actually get all of you to create a question bank of higher-order blooms level questions in different topics of peer instruction questions and so on so if you have a question bank which is populated well as in it has a large number of questions and let us say the questions are tagged according to different blooms levels in a particular topic that would be a useful resource for an instructor I have a suggestion madam I am trying to prepare a question bank in the wiki space where you have provided what I would like to do is based on the level that is a thinking level I will have a set of questions and indicate approximate time taken by the student to answer each question is it better to do like that or do you suggest any other method of creating the question bank no this is something what this is exactly what we were suggesting that if your questions are tagged with the cognitive level the topic the subtopic the purpose whether it is a group activity or individual time taken you can make up various tags the more tags it has the more useful the information would be okay center 1036 we are very fortunate to attend in this workshop it is very effective and it is we are getting hands-on experience and we are sharing lot of things in this and now I am coming to my question ma'am how to know the workload of a student if all subjects are balanced they can be more productive I will have any metrics in this regard ma'am okay so the question is how can we determine the workload of a student and we do not have metrics like we did for technology enhanced learning effectiveness do not recall that there is something like that as a teacher if you are concerned about this I mean the one thing you can do is ask your students and how do you ask them again you can use something like a survey question or a peer instruction question in class give percentages and you can ask whether the particular course you are teaching how balanced it is with respect to other questions questions like this are meaningful to ask only if some action can be taken based on them so if you think you can do some tweaks to your course based on your knowledge then ask students diagnostic questions at the beginning either handwritten surveys or PI surveys and so on center 1 2 4 7 I need to know if suppose we are dealing with first year undergraduate students who have been traditionally taught in their school I mean in their intermediate school are offering resistance are not lending themselves their interest towards new technology which we are we have learned here I mean think pair share activity and all what would be the step for us to motivate them towards this thing this is a very common problem among students also that means you are not used to these kind of activities and you feel uncomfortable while you are doing collaborative activities so quick suggestion make sure that there is sufficient incentive for the student to participate in the activities so make sure you have I believe that most of you have some amount of internal marks given to students so let's say at the start of the session you say that five marks out of the total or three marks out of the total internals will be given for the participation and engagement that you show in the class so this is one kind of incentive another is make sure you practice you do such activities continuously so you may not get desired effect in the first or second tps you may have to do five or six tps to actually see students collaborating and see that their levels are improving so one thing that as your instructor you should have is patience and perseverance on that the activities that you are doing what we have found from many instructors is these active learning strategies take a little bit of time initially to catch on but it's usually within the third or fourth time you implement it students get used to it and you will start seeing results in terms of large number of discussions and more people being engaged so typically it's three to four times the other thing you can do as a teacher is explain to students why you're doing them that if they know that there is a reason which is related to their own improvement of learning and engagement they're more likely to do to doing so and if a larger number of colleagues of yours try it then students will start seeing the same techniques being applied in multiple classes so that also helps students feel that you know this is something routine we would do talk to them a lot if you see the resistance explain to them why it's happening and it's yeah instead of fighting it we should try to get them on board as to why we are doing this again this is one of those open forum questions that we can post and see what other people have tried that has worked center 1146 I have three questions regarding sample size question number one is there any guideline for selecting the sample size question number two is this sample size context dependent question number three is a sample size has any relation for different parameters like effectiveness attractiveness accessibility and effectiveness I mean can we use the same sample size for all these parameters or should we use different parameters okay so this question is related to how large should be the number of participants that you have in your research studies so we discussed several examples of research studies today morning and we talked about measurement both the metrics what to measure and how to measure and one important choice you have to make or sometimes you have constraints as to how many participants do a must I have so one of the specific questions that was asked just now was does it depend on context and the answer is yeah it does depend not just on context but also on the type of research you're doing what I mean here is suppose you are doing in-depth rich interviews you only may able to do you may only be able to do interviews of say five or ten participants and that's acceptable because you have a large volume of rich data on the other hand if you're doing multiple choice surveys typically it's expected that you have a larger sample so the focus is on the amount of data totally whether it's in terms of richness of data different types of data from the same participant or you have lots of participants and you have more precise multiple choice type of data that's one thing that will help you the other thing is suppose you're doing an experimental study with two or multiple groups in order for statistical tests to be valid there are some rules of thumb most statistical tests are not valid things like t-test and chi-square tests and all they're only valid beyond a certain sample size which is somewhere around 25 30 35 40 and so on so unless I mean this is not the only constraint but each statistical test comes with its set of rules and constraints on when you can apply it so as guidelines what we use is suppose you have a smaller sample size let's say you have only 15 people and you have a own in one class don't do a two group study do a pre-post study take multiple sources of data so that you have larger amounts of data for the 15 people try to analyze what are the limitations of your study because you have a smaller sample size be upfront about it in your research paper and if you've done this systematically the referee is will be likely to accept it on the other hand if you have a large access to a larger number of students let's say you have two sections and each student has 40 students do think of a controlled group study in that case center 1075. First of all I am very thankful that you have given a nice information on this educational technology ma'am my question is that if any of the faculty from any domain if you're interested in doing the research on this educational technology whether it is it will be accepted by any recognized in RCT okay so this is a thorny issue is educational technology research accepted by universities by AICT and boards and so on so largely right currently AICT does not accept ET PhDs as relevant for the domain so this is something you have to know if you're considering doing PhD in this on the other hand if you publish a research paper in an IEEE conference for it such as T4E some universities many universities are inclined to accept the paper so you have if this is going to be a problem for you do check with your university what we can say is many of these conferences and journals are highly ranked they are part of IEEE explore and so on so being an engineering college you can make a case that since it's part of IEEE explore it should be considered on the other hand for PhD currently the answer is most universities won't consider it seriously so please be careful if you're considering this we do have engineering college instructors registered for PhD but they are aware of this issue today Perumal College Center 1061 then you have told about a lot of ICT tools for the past six days and we have an query about the ICT tools so if you can able to use the priority of the tools from basic to advanced because if we use the advanced tool the initial stage it will be tougher for the average students to adapt the tools so if you can able to get a preference of using those tools it will be better for the faculty to use those tools in the classroom to answer this question first of all every tool has a purpose for which it has to be used and the digital blooms taxonomy clearly says that you have to expose your learner to the tools first before actually using it in the classroom so take your case as an example Vicky from the initial survey we found that many of you were not aware of what a Vicky is so all the activities in the Vicky was geared towards exposing you to the concept of Vicky and making you practice in Vicky so that in the next so the next activity that we are going to give you you are going to design use Vicky as a teacher so the purpose of and why we use this the purpose of Vicky is that it can help you in collaboration and collaboration is something that we have stressed throughout this workshop and the content repository that we create the creation of TPS questions or PI questions require collaboration from each and every participant not just in submitting the PI and TPS questions but also in reviewing the PI and TPS questions so individually at our place will not be able to review 1000 4000 TPS or PI questions and hence all of you were assigned some set of subset of these questions and you were doing reviewing of these questions at your end and then posting them in the Vicky so this is how we have matched the purpose with the technology so the same logic applies when you are using technology tools or ICT tools in your classroom be it a simple one or an advanced one rc10582 here their question my question is related to students collection of data from students now when we collect this data and publish should we inform student beforehand that we will be collecting this information and will be publishing the data what are the ethical issues in this so okay the question is while you are collecting data from students should we inform the students that we are we'll be using them for a research purpose so the answer is yes an informed consent is required I mean it is highly preferred that you do informed consent right before the start of the study so ethical issues actually there are number of subtle points here because it depends on whether in your research whether you are going to only publish an average of all the students data or you are going to publish individual data points or you are going to try to identify a student with a data point and so on so the rules are fairly involved but as a general guideline what we should try to do is inform students that we are going to use the data to analyze what is happening and especially if you're doing something like videotaping or interviewing and audio taping then informed consent is a must if you're using class averages which are anyway going to be available it's fine to it's fine to tell them but you may not need informed consent from each and every student at that point when we do surveys and questionnaires again we do ask for informed consent so what we will do is again different universities have different rules different journals might have different rules it's always better to err on the safer side if you are not sure we can post a sample informed consent form on moody if you want to go ahead with this seriously but the moment you have identifying data video data or if you're working with minors and children then things you have to be much more careful in those cases another answer to the question is we will be taking informed consent from all of you to use your data I mean the assignments the survey questions that you have answered towards the end of the workshop so you can actually look at the informed consent template that you will be getting at the end of this workshop because we use this data to first of all improve the workshops that we are conducting and secondly it actually feeds into our own research on faculty professional development so we will be specifically using the assignment data that you are having and also the wiki data so you can shortly see in the next session the survey question being put up and informed you will be given up an informed consent what do you say a page on informed consent right at the start of the survey okay so let's end the session now it's about 10 minutes to one we'll reassemble at two o'clock other questions do post on moodle or chat we'll try to get to all of you by the end of the workshop and if we can't do live then we'll definitely answer them on moodle and chat moodle is going to be open for quite some more time few more weeks so we'll try to help you with all the questions so let's reassemble at two o'clock