 So, what we will do in this session, what you will do in this session is an activity where you will evaluate two papers. The goal of the activity is that two papers A and B will be given, they are on similar topics and the broad goal of the activity is to critically analyze and evaluate the two papers. So, this is the broad goal, so let us see how to do it. Firstly, you need to form groups of two, so you can just turn to your neighbor right now, pick a partner and shake hands with them, introduce yourself to them if you do not know who they are. Take a minute to do it, find a partner and with your partner, you will be writing the answers to the questions in the worksheet, you can write the answers in your notebook. There is no submission here right now, but I will be asking a lot of questions from each of the remote centers, so please answer each of the questions enthusiastically and sincerely. Where to find these files, so there are two options for you, these files have been uploaded on Moodle and an email was sent to all the RC coordinators, so one possibility is that you can open the files on a PC or on your laptop, they are available on Moodle. The second option is that in some RCs printouts are available, so if you have a printout of the worksheet, you need three files, you need paper A, you need paper B and you need the worksheet. So let me just read out the instructions and then I will let you figure out how to actually open these files. First you need to read paper A, each paper is six pages long, paper A is the one on effectiveness of learning simulations of electronics labs. The file name is paper A.pdf if you are opening it on the PC and paper B is innovative lab experiences for introductory electrical engineering students. So let us read paper A, then answer parts of question one related to paper A. So I will show you what I mean by this, this is just the first part of the worksheet. You will see a table, there is an item in the table, in the table you will have to answer a yes or a no and there are about ten such items in question one. Similarly, there are questions two, three and four where again you have to analyze the papers and the worksheet tells you precisely what to do with the paper. So first read paper A and answer all questions related to paper A, then read paper B and answer all parts of question related to paper B. In order to answer the question, you and your partner can discuss the answers and write it in a notebook, so this is a pair activity. Once you have finished entire question one where you are comparing the two papers in terms of about ten features, you need to go to question two. And in question two, again there are three parts, you must at least try to finish question two till the end of the activity. We will spend about 45 minutes on this activity, you may need more time because it will take you about 15 minutes or so to read the paper. You cannot simply skim through the paper, you have to really read it. The reason you have to read it carefully is if you look at question one, it says locate the following items in the paper. When you find the required item, circle it and make notes on a paper, on the paper itself if you have printouts or if you have, if you are doing it on a PC, you can locate it on the PDF file on the screen and write it in your notebooks where which section, which page number, which paragraph did you find this item. So the items, some of the items are not easily, you cannot easily find them. So you really have to read it line by line and paragraph by paragraph in order to locate these items. So read the paper, look at the items, locate them, write them, fill out the table and the worksheet as a pair and we will get back, we will assemble again here at 245. Assemble again in the sense, we will start interacting again at 245. In the meantime, you have to do this activity and we can actually see what you are doing. So please take this time to fill out the worksheet and do the activity. So what we will do in this session, what you will do in this session is an activity where you will evaluate two papers. The goal of the activity is that two papers A and B will be given. They are on similar topics and the broad goal of the activity is to critically analyze and evaluate the two papers. So this is the broad goal. So let's see how to do it. Firstly, you need to form groups of two. So you can just turn to your neighbor right now, pick a partner and check hands with them, introduce yourself to them if you don't know who they are. Take a minute to do it, find a partner. And with your partner, you will be writing the answers to the questions in the worksheet. You can write the answers in your notebook. There is no submission here right now, but I will be asking a lot of questions from each of the remote centers. So please answer each of the questions enthusiastically and sincerely. Where to find these files? So there are two options for you. These files have been uploaded on Moodle and an email was sent to all the RC coordinators. So one possibility is that you can open the files on a PC or on your laptop. They're available on Moodle. The second option is that in some RCs, printouts are available. So if you have a printout of the worksheet, you need three files. You need paper A, you need paper B, and you need the worksheet. So let me just read out the instructions and then I'll let you figure out how to actually open these files. First, you need to read paper A. Paper A is six pages long. Paper A is the one on effectiveness of learning simulations of electronics labs. The file name is paper A.pdf if you're opening it on the PC. And paper B is innovative lab experiences for introductory electrical engineering students. First read paper A. Then answer parts of question one related to paper A. So I'll show you what I mean by this. This is just the first part of the worksheet. You will see a table. There is an item in the table. In the table, you will have to answer a yes or a no. And there are about ten such items in question one. Similarly, there's questions two, three, and four where again you have to analyze the papers and the worksheet tells you precisely what to do with the paper. So first read paper A and answer all questions related to paper A. Then read paper B and answer all parts of question related to paper B. In order to answer the question, you and your partner can discuss the answers and write it in a notebook. So this is a pair activity. Once you've finished question one where you're comparing the two papers in terms of about ten features, you need to go to question two. And in question two, again there are three parts. You must at least try to finish question two till the end of the activity. We'll spend about 45 minutes on this activity. You may need more time because it will take you about 15 minutes or so to read the paper. You can't simply skim through the paper. You have to really read it. The reason you have to read it carefully is if you look at question one, it says locate the following items in the paper. When you find the required item, circle it and make notes on a paper, on the paper itself if you have printouts or if you're doing it on a PC, you can locate it on the PDF file on the screen and write it in your notebooks which section, which page number, which paragraph did you find this item. Some of the items are not easily, you can't easily find them. So you really have to read it line by line and paragraph by paragraph in order to locate these items. So read the paper, look at the items, locate them, write them, fill out the table and the worksheet as a pair and we'll get back, we'll assemble again here at 245. Assemble again in the sense, we'll start interacting again at 245. In the meantime, you have to do this activity and we can actually see what you're doing. So please take this time to fill out the worksheet and do the activity. Let's begin our session again. I hope all of you have got a chance to go through the papers and do the worksheet, read the papers and so on. If you are still in the computer lab, please make sure that everybody is back where a view can be seen. What we'll do in this session is the continuation of the analysis of paper activity. We'll go through the worksheet, some of the questions of the worksheet and also see how to locate, what to look for in a paper when you're trying to critically evaluate it. Let's begin with just summarizing what the activity was about. You had two papers. A was about the effectiveness of learning simulations for electronic labs and B was a similar paper. It was about innovative lab experiences, is what they call. There was a lot of lab-related material in the paper for introductory electrical engineering students. So before we look at the worksheet solutions, worksheet answers, let's just reflect for a moment on what was the learning objective of this activity. What did you learn by doing this activity or what was our objective in assigning this question for you as an activity? And what we would like you to be able to do after doing this activity are three things. One is you learn how to critically read a paper. And in order to critically read, there is a lot of analysis of the paper that was required and you use that to also compare papers. The way you did it is by looking at the features in the table in question one and asking questions which were there in questions to ABC, question three and so on. So the point I'm trying to make here is that whenever you need to read a paper, for example, for your lit survey section in your own research study, or if you want to analyze a paper and try to evaluate whether it's a strong paper or weak paper, you can go back to these features and these questions. So use the worksheet as a template to be able to analyze a paper or to compare papers. Let's say you're refereeing a paper and you have to review a paper for a conference and you have to decide whether to accept it or not. You can use similar features because this is exactly the checklist that referees get on how to evaluate a paper. So let's move on to the actual papers right now, the actual questions from the worksheet. These items are related to the problem and I've picked out three rows. I believe these are the first three rows or the first two rows and one from the bottom. All are about whether the problem, what problem is being addressed by the paper, whether the importance of the solution is located somewhere in the paper and whether specific research questions or objectives are present in the paper or not. So if you recall you have to try to identify the location of each of these items. You have to pick out whether the papers contain these items or not. So here is that table summary that each of these items was present in paper A. This is the one about electronics lab simulations and where they were present is also given and we see that these items are only partially present in paper B. So what you can do at this time is take your filled out worksheets and do a self-assessment. We'll also look at reasons why we have put yeses and nos here. You also had your reasons when you discussed with your partner. So use those reasons and the worksheet that you filled up to do a self-assessment. So let's go through each row one by one. In a paper that we call as a strong paper or the desirable features in a strong paper we saw this earlier that the problem being addressed has to be there up front. It has to be very clear which teaching learning issue is being addressed in the research study. What problem the authors are trying to solve and so on. So if you look at paper A, the very first sentence in the abstract right below the title says that the work investigates the efficacy of software simulations of electronic circuit labs to support doubly students. So they set the stage right away and if you just look a little later in section 2 they repeat this in a slightly more detailed and precise manner. They say that the studies reported here investigate the extent to which lab simulations may replace electronic physical labs. So the reader knows what the researchers have done and they also know what to expect in the rest of the parts of the paper. So this is one of the starting points when you start thinking of a research study and that's what is the problem that's being addressed. Paper A does that quite clearly. Paper B does it to some extent. Let's look at the next point. Fairly early on the researchers have to establish why the problem is important. And if you look at paper A in the introduction section there was a sentence which says which talks about the benefits of electronic simulations. What are the benefits to students? What are the benefits in terms of cost and in terms of scheduling and so on? So why are you trying to solve the problem that you have described is another point that you have to describe quite soon and in detail. The third point about problems is that after stating the broad problem at some point you have to state the specific research objectives preferably in the form of questions because then they become very easy to answer. Again in paper A if you look at section 4 the first paragraph of section 4 the two questions are precisely stated. To what extent will subjects who use the simulations with physical labs improve their performance? A similar question. On the other hand a weak paper might have a problem. Most papers, most research studies do think of a problem but it's not mentioned as concretely precisely and as much in detail as we saw in the strong paper. So weak paper might mention a problem in a broad and fuzzy manner even if the authors have an intended goal in mind. What was done was mentioned in a very, it was mentioned but the importance of the problem was not established. I think some small part of the slide seems to have gotten cut off here but there is a sentence in paper B in the background section which tries to describe the problem. It says the paper describes new experiments and open-ended design projects and it tries to make a connection to the ABIT criteria. But it's not clear why this problem is important and it's not clear what exactly is the research objective of the paper. So to summarize let's just go back a few slides. The three points about the problem have to be present in the manner described in a strong paper. By now I am sure all of you have guessed and even maybe before while you were doing your own worksheet you could have guessed which is a strong paper and which is not the strong paper. Let's go to one more point now. If you look at the next few rows of the table it asks whether the paper that you read had described prior work that had been done to solve the problem. It also asks about the gaps in the prior work whether they were established or not. So again you can look at the filled out worksheet. Your filled out worksheet compare with what's written here. I'll just spend maybe few seconds for you to look at this. But then we'll look at each one in detail. In a strong paper the related work is stated and analysed and what we mean by this is two points. If you look at paper A section 2A the section is called related work. Section 2 is called related work and section 2A is about the electronic simulations. The first two paragraphs of that section talk about other papers and other simulators which employ a solution approach very similar to what this paper has done. So here the authors are talking about prior work that's related to the type of solution that they have implemented. If you go on to section 2B it's called learning issues. The researchers talk about a number of motivations and number of theoretical aspects, educational theory aspects on which the simulations are based. So they talk about what are the features in their simulators which are based on different learning theories. That's also considered as related work. Moreover, if we go back to the other solutions other similar solutions we see that section 2A in paper A has a sentence which says that both circuit tutor and electronic workbench these are the previous prior work, the previous solutions. Both of these may reduce the cost and time of lab experiences and then they have a but, this but is very important because that's where they are trying to establish the gap. But the efficacy of these simulations compared to that of physical equipment labs is not well understood. So they have analyzed what has been done and they have synthesized what's missing in the work that's already been done. Paper B on the other hand doesn't have any sentence or any section where analysis and synthesis of previous work has been written about. And if you remember what we talked about in the previous session we kept saying that your solution to your paper or your solution in your research study must be innovative. We emphasized that in all the examples and at that point there was one point that was made that one way of establishing that the solution is innovative is to describe what else has been done and see how your work compares or how it fits what all has been done so far. To bridge the gap between what's been done and where you are you have to analyze what's missing in the related work that will establish that your solution is indeed innovative and it also establishes the fact that you are trying to address the gap. So the relevance of your solution is also addressed here. Let's go on to the next point when we are comparing the papers. There were two items about in the table and these were about the solution, the description of the solution. One item talked about the outline of how the authors solved the problem and the other item was about the details of implementation of the procedure. So here again I am not sure if you were able to locate this because it is really buried deep into paper A. But if you look at section 4 in paper A, let me go to the next slide. The very first sentence in section 4, let me zoom this a little bit. Very first sentence gives an overview of all the experiments that were done. It says that two experiments were performed and experiment one compared something, experiment two was something else. Before going into the details of what was done in each experiment these two summary sentences are present. That's what we mean by broad overall solution is explained first and then the details. If you look at the next several paragraphs on that page and even on the next page, each of these topics, methods, subjects, procedure and many others, each of these is described in great detail in one or more paragraphs. So a strong paper does both. It first gives a broad outline of how the authors solved the problem because people need to know what you are doing before getting into the details and then it also does the details. On the other hand, if you look at paper B, there are a lot of details, but there are two problems here. One is that all the details are related only to the development of the material. It talks in a great length about the types of labs that were developed, which circuits were used there, what components were used. But it doesn't make a connection to the problem that was stated initially and secondly, it does not give the overall solution. So we will come back to this point in a few minutes. What this paper B does not have is the thought process behind the solution. So why did the researchers come up with the solution? Why do they think the solution should work? Why should it address the problem? So even before going into the details, the reader or the referee of the paper has to know why you as a person who is doing the research study came up with the solution to solve the problem. That's another way of talking about the broad outline of the paper. So the weak paper usually has this part missing. On the other hand, the strong paper, when you read a strong paper, this process is easily visible to the reader. So when you are writing your papers, you have some idea behind why you are doing the research study and why you are implementing the specific solution. That has to come out when you are talking of the solution approach. Let's look at one more point that was there. There was an item which said, do the papers have the educational theories on which the solution is based. And we will talk a lot more about this both a little bit today and a lot more next week. You will have to do some assignments related to educational theories because this part might be somewhat new to faculty members in engineering and computer science and the sciences. But we will start off here. If you look at paper A, they do talk about the learning theories and the learning issues on which their solution is based, whereas this point is completely missing in paper B. We will come back to this point a few more times. So let's now come to the point about evaluation of the solution that is proposed by the authors. Again this is a feature that we talked quite at length in the previous session when we were looking at the different examples. That one important feature that must be present is that the solution must show some evaluation. This and I think all of you would have immediately seen that paper A does this extensively because there are about four pages regarding the evaluation of the effectiveness. And I am not sure, take a moment and think about if you haven't done this yet. See if this was present in paper B. Something is written here but if you haven't read it yet, go back to the last paragraph, take a moment, and see if the last paragraph in paper B indeed talks about the evaluation of the solution. Did you all find it? Little bit. Let's see what all must be present in the evaluation. Again we will go back to paper A which does this extensively and see how they evaluate the solution. So in order to evaluate and defend the solution, essentially what you are trying to do here is to establish that your solution indeed works. You are trying to convince the referee and the readers that your solution is valid. So firstly you need to show data and results. You would have to have done some experiment. And secondly you have to show that the results are actually solving the problem. What we mean here is that the data that you collect and analyze must be understandable and typically in scientific papers results are in the forms of tables and graphs, tables or graphs and some explanations that go along with it. Writing long paragraphs of text is not the most optimal way. It's not the best way to explain results. You would need text but tables and graphs are highly recommended as methods to present results of scientific experiments. And educational and technology experiments are also scientific experiments. That's why I am using that term. So let's just zoom into the stable for a moment and see what's there and let's compare what the authors write in the text right above it. And educational and technology experiments are also scientific experiments. That's why I am using that term. So let's just zoom into the stable for a moment and see what's there and let's compare what the authors write in the text right above it. These are the data and what the authors say is results of the written post test indicate a significant difference in scores of the two groups after the treatment in favor of the combined group. So what they are saying is that the combined group did better which you can see from the numbers. So the numbers are consistent with the text and they are supporting each other. The text you can think of as explaining the numbers in the table. So showing results in the form of tables and graphs is highly recommended and explanations to make them more clearly understandable. This is necessary but it's not sufficient because what you have to do after you show results in this manner is okay, let's just for a moment compare what the weak paper does and I think again this point would be clear. A weak paper might have a tense to evaluate the solution but the evaluation is either not systematic or it's not thorough enough and so on. The sentence here is from the last paragraph of paper B. They say that outcome I was easier to measure because it was less subjective students who received the highest scores reference journal articles and conference papers the students who got lower scores merely served the web. That's the only part written in the entire paper about evaluation. So as the reader you don't know what experiment was done. You don't know how they got these results. There's no numerical results present and so on. So even though there seems to be some attempt at an evaluation this is not sufficient for a strong research paper. The second part about the results is that they should connect back to the research questions. They should be consistent. What we mean here is let's look at paper A. Somewhere on page 1 I believe there was a sentence which is the objective of their study. They say that our goal was to establish that the electronic lab simulation could produce results comparable to physical labs. Later when they have analyzed the results they have shown a table. In fact in this case the result is not very favorable because they did get an improvement but they are connecting it back and they are saying that the improvement cannot be attributed to the lab simulation only. So they are actually making some connection here. Whether you get favorable results or not whether you can say that your solution caused the results or not you have to connect it back to the research questions and say what it is. Did it work or did it not work? Finally let's look at an important item and this is called the contribution of the paper that the authors claim. So you may have a good idea you may have a problem that you started solving you may have a good idea and you did something you may have done the evaluation but at the end what is it that others should take away from the paper? What is the important contribution to the entire field that you made? Again paper A does that the last paragraph says tells the reader that after you read these studies you will have an idea so that is implied here that these studies provide an indication that there exists a simulation that yields learning at least equivalent to physical labs that was essentially their objective. So what is important is that you should be stating this very clearly and you shouldn't make the reader search for this. Weak paper on the other hand does not have a clearly stated research contribution even though there may be a contribution in terms of teaching learning materials or in terms of effective teaching strategies all these may be there on the other hand the research contribution may not be there very clearly. So to summarize paper A and B we go back to the very first session this morning and the goal was to go from ET practitioner to ET researcher. What we see is that paper B is still at the ET practitioner is still talking from the ET practitioner's perspective because they have developed a lot of lab materials they may be effective but they don't have any of the features that need to be present in a research paper. B on the other hand has taken a teaching learning idea and has implemented the further steps to that will take it to ET researcher perspective. I see a lot of chat messages which say that this last point seems to have been understood by all participants and that you got it before while solving the worksheet. So that's very good to see. So let us talk about a few other things that must be there in a well written paper. This wasn't there in any of the items but let's just go through them that a well written paper should have a consistent flow. We will talk a lot more about this in the next few sessions. Each of the pieces each of the aspects of the study must be connected to each other and this is the final summary of the features that must be there in a strong paper. Again we are repeating this often because it's very important what you need to keep in mind when you do your own research study. So until now we looked at the question one of the worksheet and your task was to self assess whether your answers matched and to see if your reasons were similar to the ones given and so on. If you have done so then it looks like you are on your way to becoming an ET researcher. You have taken the first step. There were some other questions in the worksheet which paper did a better job of analyzing prior work which paper and I see that all of you are already writing in the chat window that paper A was better in all of these. So by now the answer should be very clear but what's important what you need to do is to think about the reasons because of which you gave the answer that paper A was better in terms of analyzing prior work in terms of describing procedure and so on. So when you get a new paper to analyze you have to be able to answer these questions by looking at the features in the table. So let's actually this was supposed to be an activity which got moved to homework assignment and then it got moved back to an activity. So this is an activity that you will do right now. You will start this activity right now and you will continue it over deep because we have from now until 4 p.m. to do this activity. You can continue your activity while drinking tea. It's an individual activity to begin with. So by now you know all the features that must be present in a strong paper. The goal of this activity is to assess your own idea that you have submitted in the pre-workshop assignment and check if your idea contains all the features in the activity. So let me repeat this again because what we are now asking is take what you learnt while analyzing others papers and apply that those knowledge and skills to your own idea. The idea that you have already submitted in the pre-workshop assignment. So I will put up the table on this slide in a moment but what there is one note I would like to make here. Some of you at this point might have decided or might have come to a conclusion that you don't want to work with the same idea that you did in the pre-workshop assignment. So this is a right time to change your idea if you'd like. If you thought of a better idea while talking to your partner or if you got a better idea by looking at the examples in the morning, you can change your idea. So if you think you want to work with a different or a better idea than what you submitted in the pre-workshop assignment write your new idea first. If you want to work with the same idea that you did in the pre-workshop assignment that's also fine. The reason I am repeating this is that the assignment that you will do from today evening onwards for the next coming week is also is an extension of the pre-workshop assignment. So you need to have your idea concretely in place before you leave the workshop today. Once you have your idea you can just do this in your notebooks and it's an individual activity. Self-assess which of the features below is present in the idea that you submitted in the pre-workshop assignment. The criteria for self-assessment are given here and at this stage there's a there's a point that we need to make here. Your idea may not contain all the required features because last week when you uploaded the pre-workshop assignment you had to just write an idea. This is okay it's not because it and the reason it's okay is that you will work on incorporating all of these in the rest of the sessions today as well as over the coming week and on next Saturday. So what you need to do is think of your idea you may have it written in your notebooks or you might remember it and simply note down first whether the idea is present or not. If it's not present think of what else you need to do in order to incorporate this feature because each of these features now you know is essential to have in your idea.