 morning. You would be surprised to see me here, because the schedule says Professor Sahana Murthy is going to talk to you. She is unfortunately away. So, what we have done is we have recorded her lecture and she had coached me on where to stop in the lecture and how to interject. So, I will be conducting the lecture on her behalf. Of course, you will be seeing her speaking to you on a recorded video. We have uploaded the slides on the AVU interface, so that you will be able to see the slides very, very crisply. I will keep on moving the slides personally as the video slides advance. With this, we will now start with the lecture by Professor Sahana Murthy. Welcome to the workshop on Introduction to Research Methodologies. This is the session on how to read a research paper. By now, you might have undergone several sessions in which you may have heard of the scientific process, the various parts of the research process, the scientific method, the skills required while doing research and maybe some of the knowledge, additional knowledge that you need to develop while embarking on a research project and so on. You may even encounter some of these topics in some of the future sessions. Now, the title of the session might sound unusual. That is, why should one need to learn specifically in a separate session how to read? After all, we have been reading right from kindergarten, from primary school. So, what is so special about learning how to read? Those of you who are experienced in the research process will immediately recognize the need for developing the skill because how to read a research paper effectively and make use of it in one's research process is indeed a skill that one needs to develop and it does come as one does more and more research projects. But there are certain tips and certain guidelines which are useful to follow and that is what we will be discussing today. First, let us be clear on what type of papers we are discussing. Of course, we are discussing research papers. But what I mean here is that these are not chapters of a textbook from a course. For example, if there is a fundamental course on introduction to computer programming and let us say there is a standard textbook for it, a chapter in that textbook would not be in the scope of what we are discussing today. Some of these research papers do appear in books, but those are special kinds of books and we will come to that in a moment. Neither are these opinion pieces of individuals. For example, individual X might have an idea about a certain field and that person might write an opinion piece and maybe even post it on his or her website. We are not talking of such articles. What we are talking about here are engineering and science research papers that have been written in a scientific manner. We will try very hard to restrict ourselves to published engineering research papers. What we mean by published here is those articles that have been published in peer reviewed journals, in the so called established journals which are known in a community of researchers or peer reviewed conferences. Sometimes what happens is some of certain research papers in a specific topic are collected and edited in the form of a book. So, these are papers written by different people, but they are edited and collected together. So, some of these papers you might find in such a book too. So, let us restrict ourselves to papers of these kind and the guidelines that we will be discussing today are general guidelines since most of you come from a fairly varied heterogeneous background. You are from engineering or science background I agree, but the kind of guidelines we are talking about are not specific to any one discipline. They are mostly reasonably applicable to computer science, electrical engineering, chemical, mechanical engineering, even certain science subjects like physics, chemistry, mathematics and so on. So, these are general guidelines and you have to at some point determine what exactly works in your discipline, but at a broad level what we will discuss today works for all these disciplines. And many of the guidelines we will discuss today will or may not apply to survey or review papers which are a slightly special kind of papers. Survey or review papers are in between a textbook and a research paper on a current research problem. They usually contain an overview or a summary, a very critical analytical summary of all the research that has been done in a certain field up to a certain let us say within a given time frame. So, they are a very good starting point when you are trying to do literature survey and that comes in one more session in this workshop, but when we are talking about how to read a research paper let us leave that aside for a moment. There are a few assumptions that we will work with that the main thing is that before you begin reading a technical research paper on a current research problem or maybe even something that is a few years old, the assumption is that you are somewhat familiar with the broad area and the broad idea that is being discussed. What I mean is you may have read some related paper or you may have taken a course in the field and so on and it is if one is not familiar because when one begins a research project one is never necessarily familiar with the exact field. In that case, what one can do is become familiar and the ways to do it is to either read a textbook on fundamental concepts or take a course as I mentioned earlier or go through certain tutorials on the topic or read a survey or review paper which is in fact would be an easier starting point to dive into a new field. There is one more assumption here I am making that there was a paper uploaded on Moodle that you are supposed to read as a pre homework to the session. It is called design and deployment of clickers in distance education. I am assuming that you have read it because a lot of examples that we will be doing today are based on that paper and it would be good if you have a print out of the paper in front of you may be one for every two or three participants and if you do not perhaps the coordinators can help at this point. So, let us actually look at the process of or the approach of how to read a research paper and we have called it the three plus stage approach here. Now, there is a lot of references and lot of people have given good advice on how to read a research paper. So, what we will talk about today is really a it is been borrowed from some things that people have already said and you can I will tell you sometime later towards the end of this talk on where to find some of these good references and most of them talk about the approach to reading a research paper in a process which looks like what you see on this slide here. The zero at stage. So, this is really before you actually begin the first stage is what I call get a feel for the paper and each of these boxes we are going to expand in the next few minutes. Feel is something very quick it is literally a feel you may just pick up a paper like this and you may flip through it. Then the first important stage is to get the big picture. Then one gets the details from the paper, one evaluates the details and to get a good idea of the paper one has to at least go through stage three and if you are a research student or a researcher who is using the paper for your current work you often have to go to stage three plus where you have to synthesize the details. So, let us get a look at what we mean by get a feel for the paper and how to do it. By the way each stage will require one reading of the paper. So, when you read a research paper it is a multi or you will have to read it multiple times. In fact, that is one of the questions we explore later. So, how do you get a feel? First thing is read the simplest thing, the largest thing on the paper and the largest font is usually the title. So, read the title, ponder upon it if you would like for few seconds and then as I mentioned earlier, pick up the paper and see how long it is and papers vary in length a lot. Typical and the variation is really huge it can be as small as four papers and sometimes you see reflection papers as small as two pages and sometimes they might go to thirty or forty pages. So, you need to get an idea of how large the paper is and typically the length of the paper also is depends on the type or where it is been published. For example, conference papers for technical research conferences are usually between four and eight pages. Journal articles are longer and these can be small like five or six pages, it can actually even be four, but it can go up to fifteen or twenty pages typically. Review or survey papers are usually much longer. Second, the next thing you need to know when you are just getting a feel for the paper is where is this paper published? The bibliographic details as we call it and one needs to know how to find this information and where to find this information. Typically, what happens is it is available if you have downloaded the paper from the journal website or if you have photocopied it from a specific journal, you should look at the header of the footer of the page and you will find that information. If you have downloaded it from a website from let us say somebody's personal page, see if they give bibliographic details and what we mean by these details are the year, the title, the journal, where it is been published and so on. Look at the figures, just glance at them, see what you think and read the headings. So, when we say get a feel for the paper this is all you do and this can take as little as one minute sometimes. We will come to how long should I spend on research paper, we will come to this question little later, but this is really getting a quick feel. In fact, you would be able to get a feel of a paper much quicker than it took me to go through this slide. So, we will have to look at before we go on to getting the details of a paper is that the scientific research paper is actually a very peculiar piece of writing. It is not a story, but it is not a story as in fiction, but it has to tell a story and it is very highly structured. In fact, the moment you get a little experience you will be able to predict what are the headings in the paper, most papers follow specific structure. Every item in the paper is there for a reason. There is nothing even the figures are not there to make the paper look pretty or make it look good aesthetically, but there is a scientific reason why each item is there and that leads to the next point that each part of the paper is connected to the other part and this is true at the sentence level, this is true from section to section, from paragraph to paragraph and so on. So, when we are reading we have to be able to follow the links, then there is always a very high correlation between the figure and the text. For example, there is a place in the text which refers to the figure and the figure will refer to the text, some other piece of text back again. So, one needs to be able to easily be able to locate the information one is looking for either in a figure or in the corresponding text. So, research paper does all this, but the author is severely constrained by space because often there is a page limit on the paper. And because it is so peculiar, by the way if you are familiar with research papers this is not anything new, but if you are not familiar you would require some time to become familiar. It takes a few papers before you become familiar with the peculiar structure of a research paper. So, the next slide is something which is going to be old information for those of you who are experienced, but if you have not read too many research papers then we will just go through a typical structure of a scientific research paper. And if you look at the slide in front of you, you see that the first two words the title and the abstract seem to be in a group, then there is a long list of words in one more group and right at the bottom there is a references. And the reason the slide is shown in such a manner is that the title and the abstract is always shown right at the beginning with and it is very easy for you to locate it. The abstract usually has the word abstract in front of it. The title may not say title, but there is no way you can mistake a title of the paper at least a well written paper because it is in the center, it is the first thing you encounter, it is bold, it is in larger size and so on. Similarly, the references at the end are something which you can easily locate it is always there at the end. The pieces in the middle are will be there, but they may not exactly follow this sequence and they may not you may not find headings for each of these, but some of these some a large subset of these pieces will be there. And just to read this the introduction is usually it is almost always the first paragraph whether it says so or not and very often it will say so. You will find some background motivation for the work perhaps in the introduction itself. Then comes something which is very important but often hard to find and that is the contribution of the paper. That is what is the authors actually, what are the authors claiming to do by writing this paper? What is the central take away from this paper? Sometimes it is found at towards the end of the introduction or somewhere in the middle of the introduction, sometimes it is found in the conclusion and so on. So, similarly there are many other pieces which one should look for in a research paper, but most of these pieces are going to be there, the structure is a fairly well defined structure for scientific research papers. So, now I would like all of you to do one activity and this is based on the uploaded paper that I talked about at the beginning the design and deployment of clickers. So, what you need to do is on the paper with a pen or pencil mark all the pieces that have been mentioned here, the title abstract introduction simply mark them. I presume that all the participants have got a copy of the paper printed which is being used for this exercise. I will not spend too much time here just for 3 minutes for you to identify these particular segments. At this stage you are not required to critically comment on them, but go through the paper very quickly which you would have already read and just in the context of these titers that you see here. So, find out whether the title is abstract is introduction is, where the background motivation is, where the contribution of paper is, where the related work is defined, where the problem definition is and so on. So, do that just identify by circling those paragraphs or writing in the margin about which portion corresponds to which and once you do that please share it with your neighbor. So, I presume that you would have completed this activity because it was rather a simple activity particularly if you have read the paper. Let us go ahead with this even if some of you have not finished the activity does not matter if you have done it partially there is good enough. The idea was that not only you apply your mind in this fashion, but you also check with what your neighbor has done. So, basically 2 different people or 3 different people discuss with each other and figure out who had identified something rightly or wrongly and so on. But now we will go back to Prasasana Murthy and find out what she has to say about all of these. And as soon as you do it discuss your answers with your neighbor or if you want to work with one more partner that is also fine. So, this should not take more than 1 or 2 minutes it is ok if you do not find all of them, but I just like you to mark some of these pieces. Hopefully you would have been able to locate most of these pieces in the paper provided. So, let us in fact discuss some of these we will discuss how to locate and or where to locate some of these pieces. So, what I have in front of me is really a copy of the paper and it is ok if you cannot read the paper on the screen, but what I want you to do is look at the colors I think you should be able to see at least that. Title I am sure all of you found it is what is being shown in yellow. Abstract in fact here it says it is the abstract and it is usually the very first paragraph it is either there in different font or it is in usually it is in smaller font or bold font or some such thing. So, now let us go through the paper a little bit more let us go down and what you see here is the title introduction. So, there is a section called introduction in the paper. As you are reading the introduction the second and third paragraphs on the introduction you should be able to find you should be able to realize that the background and the motivation for the paper has been written in these two paragraphs. So, these two paragraphs tell you what why this paper this research work was done roughly what problems it was based on and so on. As you go further in the paper. So, I will go back and forth. So, you will see that the order is not necessarily the same as the order you saw on the list. The contribution to the paper is what is marked in blue it says let me just read it a little bit. So, that you will be able to locate it. In this paper we describe a solution to facilitate real time interaction between instructor and remote classroom students through the use of a student response system using clickers. We designed and developed a distributed architecture for a student response system. So, that is the key contribution that they are mentioning here. So, as we mentioned as we discussed earlier this is present within the introduction, but you may have to look a little hard to be able to locate it. Let us look at a few more. There are few sentences above the contribution what is hopefully showing up as green are the research questions that this paper will address. And here it says for example, one of the questions here is what modifications need to be done both in terms of development of new technologies and in the pedagogy used. So, that is a research question that will be explored by this paper. Then right in the introduction they even talk about the solution outline the last paragraph on page 1 what is given as purple here. The solution outline is just an overview of what is going to come in the rest of the paper. It is like the preview of a movie it is not giving the details. If we go further in the paper we will just see a couple of others. Page 2 has something a little bit more on related work where the paragraph that starts with a similar situation was implemented by University of Oklahoma. But you should also be able to find related work in the section on background on page 1. So, what this really tells us is that if we are trying to locate the various parts of a paper they may not be in sequence. They may occur in multiple places and so on. So, when you are trying to read a research paper in the beginning it may be hard to try to find them, but these are exactly the pieces or the key points that you should be looking out for and we will now see how to in fact locate them. So, now let us come to the central part of the of reading a paper and this is to get a big picture of this is to get the big picture till now we were still talking about the field. So, on the left column there is a series of questions and this is slightly what are you looking for? The right side says where to find it? So, let us just go through a few questions. In fact, the first question you should think about is what research area or topic or subtopic does the paper fall under? Starting from a broad area you can say that it is an electrical engineering paper and this one layer one level of subtopic might be communications and then you can say well it is on digital communications and so on. You can go for you have to go much further. Then this is something by reading the paper you will be able to this is a question that you should be able to find. Then you want to ask the question what problem does the paper attempt to solve? So, that is the big question here because every piece of research work is attempting to solve some problem. So, you are now trying to or you are beginning to identify what this problem is. You are also trying to get a big picture of what is the related work and importantly if there is work on it beforehand why is it not sufficient? What was the need for the authors to write a new paper? That really means that what you as the reader of the paper have to has to identify is the gaps in the related work. The authors would have written it if they have written a good paper. Again you are looking for the key contribution we tried to locate it earlier and then you are trying to look for broadly how does the paper solve the problem? This is the solution outline. Again you tried you did locate it. Then you have to come to the meat of the paper the central part of the paper. How do the authors defend the solution? And I will come to the last question in a few minutes. So, now what I would like you to do is look at the right column where to find it and we are only looking at the big picture right now. Take a moment or so and just read aloud each of these cells in the right column. This is title abstract, title abstract introduction, introduction title abstract introduction conclusion introduction figures and what you see is that the word title abstract introduction has appeared many many times. In addition figures have appeared oops excuse me in addition figures have appeared multiple times and the conclusion has appeared somewhere and the word headings has appeared somewhere. So, the point we are trying to make here is that when you are trying to get the big picture you should mainly concentrate on these topics. By the way the if a word is in parentheses what it means is that you may or may not find it in the title. So, sometimes you may find it and sometimes you may not, but you will definitely find or you should definitely find the contribution of the paper in the introduction itself. So, how you get the big picture is really by reading the title abstract introduction and conclusion not the entire paper and the reading this much comes to roughly two pages may be sometimes even a little less than that unless the introduction is very long, but introductions are never very long. Then you should go through the section and sub section headings and then look at the figures diagrams illustrations and so on. This is all you need to do at this pass at this stage. Once you have read these or simultaneous along with reading these what you have to do is write the answer to the following questions the exactly the same questions that we mentioned in the in the second column what research area what sub topic etcetera. Now, where do you write these answers that is another question that is a skill again these are small tips that might help you. A good idea is to keep making notes when you are reading the paper and some people make notes in the margins of a paper. Some underline some use a highlighter on the paper itself on the hard copy. Some people keep a separate notebook of file it really does not matter which one you do, but I know that a lot of people do annotate in the margins or use a highlighter because you want to have the paper as well as your notes and your perceptions and your ideas together. So, making notes on the paper itself while you are reading is in fact, a very useful idea that our teachers told us and their teachers have told us and so on. So, let us come to the next activity and this is a slightly longer activity. This activity will work well if you have read the paper and if you have not read the paper I would suggest that you pair up with somebody who has already read the paper before. And what I would like you to do is on the printout I am assuming you have a printout you make notes to answer the following questions. Somewhere on the margins on the side and if you want you can take an extra sheet of paper also. It is exactly the same set of 708 questions that we have been talking about so far. What problem is the paper trying to solve? What is the related work? Why is it not sufficient and so on. One point we have not discussed too much is how do the authors defend the solution. What I mean here is the authors have after giving the background they would have talked about what problem they are trying to solve. They would have written about what problem they are trying to solve and then they would have given their some outline of their solution approach. That is still at that point it is an outline or an idea. It is an approach. They have to now convince the reader. They have to convince you as to why that approach is reasonable, why it works, why it is sound and so on, why it is logical. So, the defense of solution ideas usually are in the form of proofs or an experiment which has been conducted and its analysis and results or it may be an implementation, it may be the building of a system and so on. So, what I would like you to do at this point is a pair up and work on this activity. I think it could take about may be 10 to 15 minutes. So, this is slightly a longer exercise. You are looking at the same paper which I have read. You are looking at the paper in which you have already identified some segments through a quick perusal or through a quick flipping through of the paper as was suggested earlier. In fact, Prasanna Murthy also discussed after that previous activity as to which in her opinion were the portion that were identified. Now, this is something more serious. In this case I hope you can all see the slide. So, the slide says on this paper here you have a print out make notes to answer the following question. I would like to mention and she would repeat later that it is important to have a paper and pencil with you or paper and a pen with you or the printed paper itself you can use as a paper on which you should write. But there is absolutely no sense in reading or research paper without making some kind of notes and in this particular case the activity which is a pair activity. So, two of you should do it together, but you must write down something and what is to be written is also mentioned here. So, answer each question and write one, two or three lines as answers to each question. The questions are in front of you. What research area subtopic does the paper fall under? What problem does the paper attempt to solve? What is the motivation for this problem? What is the related work and why is it not sufficient? Why key contribution does the paper claim? What key contribution does the paper claim etcetera? So, we have already gone through this story, but this time it is not adequate for you to identify which portion of the paper contains what? It is important for you to read that portion and write one or three lines as answers to the following question. Now, writing these lines is not very easy and that is why she has suggested that we should provide at least ten minutes for you. Since you are pairing on to this activity, you can quickly discuss with each other. We will now not break the session. I will just sit here idly doing this activity myself. I would expect you to do this activity and we will revert back in next ten minutes. I think we have spent sufficient time slightly more than ten minutes. So, based on the activity that you have done and this is an important activity, you would have understood that now whatever paper reading that you have done yesterday and whatever reading that you have done in the last ten minutes would have helped you to identify these important aspects of the paper. We will go back to Professor Sarnamurti to listen to her as to what are her comments on the whole of it. The activity where you were trying to answer the questions posed on this slide based on the paper that was provided. What we will do now is discuss some of these questions. So, we will try to what you can do is compare your answers while we do the discussion. For example, what research area or subtopic does this paper fall under and this paper actually is an engineering education research paper. The authors who were doing the research did develop a system, but the goal or the reason they developed it is to improve engineering education. There was not anything that was much new in terms of the technical details of the system itself. They did have to do some work, but it was not a distributed architecture research or it was not networking research and so on. The research part there was to build a system for to improve some goal within education. What problem does this paper attempt to solve and if you have read the paper if you have tried to answer this question, you would have realized that the authors are trying to the researchers are trying to implement clickers in a distance education scenario. So, you can state the problem as how to embrace the implementation and benefits of clickers for distance education. They did not invent clickers that should have been clear from the paper. They have used the known benefits of clickers or they have used the idea of the known benefits of clickers and then done the necessary modifications and implemented it in distance education. What is the motivation for this problem? This is stated very clearly in the introduction of the paper that distance education lacks interactivity and in face-to-face classes there are many studies which say that interactivity improves due to clickers. So, what the authors are trying to do is take a problem, take a solution from a different domain and apply the solution from a different domain that is the face-to-face classrooms in the domain in which there is a problem. So, this is one typical way of doing research where you adapt a solution known somewhere else into the problem at hand. What is related work and why is it not sufficient? So, again the paper says that there is a lot of work on the benefits of clickers, but all of them are for face-to-face classrooms. There is very little work on the use of clickers in distance education. So, what they have done is identified the gap. They have shown us why it is not sufficient, why the related work is not sufficient to solve the problem. And if you go back, what is the problem? It is improving interactivity and embracing clickers in distance education. What key contribution or contributions does a paper claim? So, I think they are doing two things here. One is that they are telling us, they are providing us the architecture of implementing this distributed clickers or student response system in multiple remote locations. So, they are showing us how to do it and then they are showing us proof of concept that they actually did it. Now, it gets bigger and bigger. Broadly speaking, how does the paper solve the problem? And this you would find on the first page of the paper towards the bottom. I will just quickly pull it out for you. I think we have seen it once earlier. It is what is shown at the bottom in the purple solution outline. The instructor delivers lectures from a central location. This is the broad solution and this is transmitted to remote classrooms which were equipped with receivers. The receivers communicate to a server in the central classroom over the internet. Participants respond to the questions presented by the instructor through clickers. In fact, another place where you will find how the paper solves the problem is a block diagram. So, let us, we did not see that earlier. So, let us take a quick look at that. If you look at page 2 bottom, there is a block diagram. So, the introduction and the figures are a useful place to get a broad idea of what is happening in the paper. Let us look at one more question that we have to analyze the last one. How do the authors defend the solution? And here in this paper, they towards the end of the paper, you will see that data of clicker responses were collected. You will see figures and instructor and participant feedback on the clicker system were analyzed. So, this is the proof for the defense of their solution. Let us try to find those. As you see, I am skipping the entire central part which talks about the details of the actual implementation. This is on page 5 at the top. There is a graph. Then there is a table right there. There is some text which tells you what the findings were and how they defended the solution. Similarly, I believe on page 6, there are more graphs. So, when you are trying to look at defense of the solution, try to look for experiments, try to look for descriptions of the results, the graphs and so on. Let us go on. I will just go back to one of the earlier slides and we will see where we are so far in the three-stage process. We looked at how to get a feel and how to get the big picture. And now, let us see how to get the details. In fact, the type of questions that you will see in how to get the details are not very different than the type of questions you will see in getting the big picture. It is just in the amount of information that one needs to look for. There is much bigger difference. So, this slide will quickly go through because if you look at the left hand column, the questions will look very familiar to you. What problem does the paper try to solve? What is related work? What is the contribution? How does the paper solve the problem and how do the authors defend the solution? These questions should be very familiar to you by now because you answer the same questions just to get the big picture. But the difference is in the column on where to find this information. At this stage of reading the paper, you will have to look at sections beyond the introduction. For example, on related work, you have to try to find the literature survey or the related work section. Most of the times, you will find an entire section either labeled literature work or, excuse me, literature survey or related work. You may sometimes find it as part of the introduction also. How does the paper solve the problem? This is the central part of the paper that we have skipped so far. This section will either be, can have one of several names, solution, experiment, system, implementation. It does not, it is not very or different papers call it by different names based on the context. But most of the times, it comes in the middle part of the paper after the introduction and before the conclusion. How do the authors defend the solution? You will get all the details again in the middle part where the paper talks about the experiment, the methodology, the results and so on. So, while the questions we are asking so far are very similar to the previous stage, we are now looking for more details by reading more of the paper. But we have to look at, we have to answer more questions. These are not the only questions that we need to answer. So, let us look at a few other questions at this stage. What is the precise research question address? So far, we were looking at the broad problem or the motivation. Now, we are looking at a question that says research question and the difference in this paper is so on, is the following. The broad problem could be that the researchers want to improve interactivity in a distance education scenario. And the precise research question is how to implement clickers in a distributed system, how to firstly build this distributed system to implement clickers and how to implement them, so that the clickers retain their effectiveness. A precise research question is needed because that is how the solution can be developed more effectively. Some of the other questions that you, the reader has to try to answer are why is it, why you will have to try to find out why the authors believe the solution works. For example, why do the authors think that this solution is better than other solutions. So, the reader has to try to get into the researcher's mind and try to answer these questions, not exactly the researcher's mind, but in to the researcher's writing. Then comes a very important question as to what is, what are the assumptions the researchers are making? What are the boundaries in which they are working? The boundaries or the scope? What are the limitations? So, for example, in this paper, what I would, what we can do is we will have to go back and read where the authors state their assumptions. Then comes the details of the actual solution. And as I mentioned earlier, this is in the form of an argument or a proof or an implementation or an experiment depending on the type of paper that you are reading. What type of evidence is provided? We have gone through it a little bit, but now you have to look at the figures in more great detail. You have to critically evaluate the results and so on. For example, what you need to look at in the figures is the following. I am looking at this part. Look carefully at the figures, especially the graphs. See if the axes are labeled. See if error bars are put on the data points. And see if together, all this is making sense and all this is substantiating the claims of the author. If you see a graph without any labels on the axes or without units, then it is a shoddy piece of work. So, when you are reading good research or a good research paper actually pays attention to all these details. And the overall paper should give you an idea of what is the take away message of the paper. We will not do an activity right now on this because that is going to take a lot of time, but this is going to be a homework activity for you. And later we will try to, we will see how to upload the solutions and so on. So, now we come to the final stage of paper reading. And this is where you will have to take a much more active role. So far, you are the reader and you are just trying to locate the answers to some questions. The researchers and the authors were the ones who had done most of the work. But now you have to think more, you have to do more work and so on. The goal of the stage is really the question at the bottom. You have to try to decide what is the paper trying to convince you of and you also have to decide does it succeed in convincing you of that question. To be able to do so, there is a long list of questions that you can go through. And if you have submitted a paper to a conference or a journal, you may have seen reports that the referee has given you. And the referee reports usually contain answers, contains answers to these questions. Is the research problem significant? Is it important enough? Is it worth writing a paper? That is something you have to think about. Is the problem or the solution novel or new? Now, how would you know that? And here again, your experience and your ability as a researcher comes into play because you will know if a problem or solution is novel, if you are up to date with the current research in that given field. If this is the first paper you are reading in a field, you may not be able to judge if a problem is novel or not. So, your experience and your knowledge and your skills play an important role in trying to evaluate a paper. These skills do develop, they definitely improve as you read more and more papers critically and let us say you try to read every new paper and answer these questions, your skills will definitely go up. Similarly, let us take a few others. Are the assumptions valid? Most research is done within some boundary. For example, the paper on distance education, maybe the people tested it in remote locations which had a certain kind of technology and only that kind of technology. So, their scope is limited to locations where a certain level of technological sophistication is available. Perhaps the solution works in other places, but the researchers have not tested it out. So, the assumptions have to be clearly stated and another assumption of this paper, somewhere towards the end if you have read it, you will see that there were improved results of student satisfaction and the teacher was also very happy with the level of interactivity. So, one assumption here was that the teacher had some training in using clickers beforehand. If it is the first time that the teacher got clickers to use, perhaps he or she would not know how to use it and their experience may not have been so successful. So, what are these assumptions? Are they identified and are they valid? It is something that you as a reader has to keep checking. Another thing which you can try to do is see if the different parts of the paper are consistent. So, the solution is it actually solving the problem that has been posed and the results is it really giving results to the solution approach that has been implemented. The reason I mention this is when we get papers from novice writers, we often see that the problem is something and the solution is something else. So, just you have to make sure that they are consistent and then other things like does the paper flow logically and so on. Finally, we come to the most difficult, but the most interesting part of reading a paper and perhaps the most useful because when you are doing research ultimately you want to further your own research and you want to use the paper that you have read for your own research work. So, how do you do that? That is why this has been labeled as synthesize. Synthesize is when you read a lot of different or you read or you yeah, you can read a lot of different things and then you have to analyze them. When you were doing the evaluation, you were doing analysis. Here, you are doing something more. You are putting the pieces together after analyzing them and trying to create something new. So, this is a much higher order skill that will develop. So, questions you can ask are, let us say the authors have solved a certain research problem. You can ask are there any other ways to solve the problem because if you can think of a different way to solve the problem that means that gives you an idea for a research problem. In the current example, the researchers were trying to solve the problem of interactivity in distance education. They used clickers, but can you use something else. For example, put a two way audio video chat be a better or an alternate solution that addresses the same problem or is there some completely new software or hardware technology that you can develop to solve this problem of interactivity in distance education. So, asking the question is there any other way to solve the problem is a good starting point to coming up with your own research problem. Another way to come up with your own research problem is the second point here. Let us say the authors have made a claim and they have done one experiment to justify their claim. You have to ask is there a different experiment I can perform or is there a different proof I can give. It is not that you are coming up with a completely new solution, but just a new way of defending the solution that itself can give some research ideas. You can ask you can go the opposite direction. Let us look at the third question that is also an interesting one. Are there counter examples or arguments against the paper's claims? So, let us say you find that the authors what the authors have done is correct, but it is correct within the situation that they have tested and in a new situation you have come up with an argument where their results may not hold. This happens a lot in engineering and science. So, for example, the researchers might have developed might have come up with a result in one range of temperatures and you may have an argument or you may have a principle by which you can say that well in the other range of temperatures the result is something completely different. So, that gives you an idea for a new research problem to extend or explore the same problem in a different set of conditions where the original results of the paper you have read may not hold. You can ask questions like how can the results be improved or how can the results very importantly the last but one. How can the results be generalized either in a different context like the temperature range example we talked about or can they be extended and finally, the question we are going to leave with is really what are the new ideas and open problems suggested by this work? Because that is one of the main reasons why you read the paper to get ideas for open problems and new ideas. It is a difficult question and you cannot solve it by reading the paper once or may you may not even be able to solve answer this last question by reading the paper five times. But it is a question you have to keep in mind and discuss with your colleagues and your guide and so on. So, some homework activity now the same paper that we have read so far write a two or three page review which contains a summary and essentially what you have to do is we have posed a lot of questions in this presentation. You have to use those questions as guidelines and critically evaluate the paper first and once you have done it try to come to stage 3 plus where you are trying to ask the creative questions what are the new ideas what are the open problems and so on. This again this can be done only after you have read the paper a few times feel free to discuss it with your other colleagues and people in the workshop. So, let us come to I would like to interject here to emphasize how first of all the research papers ought to be read methodically. By the way let me admit to you that I have been reading research papers all my life I have even evaluate PhD thesis but when I went through process Sarnamurti's lectures I also found a few important pointers particularly about the systematic approach. I am sure many of you who would have read research papers in the past would not have read the papers in this particular fashion asking these questions to yourself and trying to find answers from the papers that you read. It makes a lot of difference when you read papers systematically when you read papers with a pen or pencil in your hand make notes and more particularly when you ask these kind of critical questions to yourself your understanding of the paper both in an overall sense as well as in greater depth on various points will be far more significant for the same time that you would have otherwise. It is possible for example to read the paper three or four or five times without properly getting an appreciation of everything on the other hand if you apply this systematic approach the same time could result in a better understanding. One last point I would like to make before continuing with Sarnamurti's lecture is that many of you who would like to write papers and some of you who would be required to review papers could keep these guidelines in mind. So for example when I am writing a paper after all I want to share my ideas my work with others but I should be aware of how people will evaluate that paper and if I am aware that the evaluators or the reviewers are going to look for these things then it is much easier for me to ensure that while I write my paper I make it very easy for a reviewer and therefore for a reader to understand that way and I can therefore make my style extremely lucid and very clear to the readers of the paper. So this lecture in my opinion has a double advantage one it tells us very clearly as to how systematically read a paper analyze it and understand it emphasizing three important points first there must be multiple reading there is no research paper that ordinarily can be understood in a single reading. Second you must ask these questions first shorter questions to be answered in a short while next of more detailed analysis of the paper and more detailed questions to be answered all the while you must have a paper in pencil range and the last stage three of the synthesization that is asking creative questions and finding answers to them will require much longer work. This is the amount of work that needs to be done on every research paper that you read that is the only way in which you can assimilate the exact thing that the paper wishes to convey to you and then you can take that work some concerns and some questions that are frequently asked one is how many times do I read a paper and by now it must be clear to you that the answer really depends on your purpose what why are you reading the paper what's the goal if your purpose sometimes your purpose is just to decide if this paper is relevant to your work or perhaps somebody has found a paper for you and is asking you is this really a valid research paper so at that point you only want to get the big picture details go up to stage one and that's sufficient to help you decide if the work is relevant and interesting and worth reading further this may be one or two readings once you get experience this is just a single reading if your purpose is to grasp the contents of the paper and to summarize it for example let's say you're writing a literature review section for your thesis you have to be able to grasp the contents then you should go up to stage three where you are evaluating the paper you are critically looking at whether the figures are consistent and whether the claims are correct whether the solution is defended well and so on so you are looking at every piece of the paper and asking is this correct is this consistent is this sensible and so on if your purpose is to come up with new research on the basis of this paper is to base your future work on this paper then what you should be able to do is actually recreate this paper in your mind you should be able to work out every single step and that should help you get to the next step in that case you have to go up to the stage which we called as three plus synthesize the paper you will find papers that belong to all three purposes all three categories as you read not all can be used to synthesize and you won't stop reading only the big only the introduction in all papers so some papers will belong to the first category some will belong to the third a lot of papers that you will use for your literature review section either in a paper or in your thesis will belong to the second purpose next question how much time should I spend reading the paper by the way there is a big disclaimer here these are very approximate times it may be more or less but these are typical times and it varies depending on whether you are experienced or whether you are a novice in this practice of paper reading to get a feel as I said earlier experienced people just take maybe 30 seconds to a minute as a beginner you might need 5 minutes because all you are doing there is reading the title seeing how long it is trying to see what category it belongs to and so on to get the big picture I think a beginner would require about an hour and an experienced person beginner may even require 30 minutes maybe 30 minutes to an hour experienced person might be able to do it in 5 minutes then to get the details and to critically evaluate here you need a lot of details and you need to be able to think critically and analytically I would say that a beginner requires about 2 hours here and by now the paper you may have read the paper 2 to 3 times finally to creatively synthesize how long does it take it could be a solid 6 hours or it could be a whole week if you are thinking about it consistently one can never say at this point if it is a very important paper and interesting to you then you once your experience you may be able to get this done in an hour or two what should I do if I still do not understand the paper because sometimes you may have read it a few times 3, 4, 5 times and you still do not understand so what do you do then so first before going to what to do let us look at what are the possible reasons maybe the topic of the paper is entirely new the subject matter is new or maybe the paper contains jargon unfamiliar terminology it may contain a lot of acronyms you are not familiar with that may be one reason or you may be familiar with the broad area but you do not understand the technical details of the experiment this happens quite a few times because you may know the principles and the theory but the kind of experiment that the researchers have done or the new system that the researchers have built you may not know what is how it works so because of that you may not understand sometimes you are just tired because you have read the paper many times or the paper is very intense or heavy as colloquially some people say or maybe it is not anything to do with you and the paper is so poorly written that it is very hard to understand it even for an experienced researcher so then there could be many other such reasons so what to do really depends on what which of these reasons is applicable some of the things you can do is first thing I have said is do nothing so if the paper is not important for you and you think it is a struggle to read it do not bother if it is not important you have to somehow make that judgment for example if it is in an area that you will not be working in at all if the reason is that you are mentally tired let the paper aside sleep for the night and pick it up again the next day one thing that helps a lot is to discuss a paper with a colleague maybe a fellow student who is working in the same area let's say you are a PhD student or a fellow colleague who is in that research area so discussion or your guide for example if you are in an MTech or PhD program discussing with colleagues is a very good way of trying to make sense of some parts of the paper that you do not understand one more thing you could do which you can do on your own is read something else let's say a fundamental exposition of the topic either in a textbook or a survey paper that talks about this research topic and come back to it that is another way to try to make sense of this paper okay last few points most papers have a long list of references so what do you do about them should you ignore it completely or should you read through every piece of it and again the answer lies somewhere in between suppose you are not familiar with the area at all then the references may not mean much to you on the other hand it the references are a good way for you to become familiar with the area so if you want to understand the paper more clearly or more deeply look at the corresponding reference if you want to build upon the current paper let's say you want to get a broader idea of the current paper or if you want to get a broader knowledge of the research area the references are a great place to look at so what you would do is not look at every single paper in the reference but let's say the part of the paper that talks about the conditions in which the experiment works let's say that's the part which is interesting there may be some reference related to it look at those references and that would broaden your knowledge of the research area so that's why there is a star there the references are very important or there is a useful way to broaden the knowledge of the research area okay thank you so what we will do next this is just a list of references that you can read in order to learn how to read a research paper so most of this presentation has been adapted from some of these references and there will be one more session on what to do after you have read one research paper so typically one reason for reading a research paper is to write the literature review section so how do you write it how do you select which papers to read where do you find these papers and so on so this session was on reading a single research paper and now we have to go from one research paper to reading multiple research paper and what to do then will be another session thank you I would like to add a few remarks in conclusion particularly to point out one great difficulty that most of you would face in adopting one particular important suggestion which the Salaamurthy has made if we go back a few slides so what if I don't understand the paper she says do nothing if the paper is not directly relevant to my research area of course I can do nothing I should do nothing by the way there are hundreds and thousands of papers that have been published and there could be several which may appear to be relevant to me but on the first reading itself you will be able to see they are not telling so this is okay sleep and read again if you are fortunate enough to get time to sleep you should sleep because that patience you have and you can read things a fresh mind if you don't understand it after multiple readings reading a textbook or a survey article is a good idea and then you return to the paper the point I want to spend a couple of minutes on discussing the paper with Kali notice that in today's session itself you were required to spend some time with a colleague of yours whom you probably did not know before you came for the workshop but hopefully you would find something meaningful coming out of those few minutes that you spent talking to each other finding out what the other person has located sharing with that other person what you have done and so on I will tell you that discussing a paper and in general discussing your ideas with colleagues is an extremely powerful mechanism to speed up your understanding and to speed up your work why I took this point to state particularly that is while I state emphatically that is important unfortunately in the given situation in most of our engineering colleges and perhaps even in science colleges it is not easy to find sufficient number of colleagues who are either competent in that area or who have worked in that area and have an understanding of that area so that we can have a discussion with good wavelength match that is one thing where environment in places like IITs or NITs or some of the better institutions are significantly that situation is significantly different than in smaller colleges basically as you would have heard Professor Karmalkar that I am working alone in a lab full of large number of people who are all working alone so this situation is sadly prevalent even in institutions like IIT now this is one difference that you find in the research culture of the developed country they have indeed colleagues who are ready to discuss with each other for willing to spend time in understanding somebody else's work and offer critical comment and believe me wherever this happens the research progresses more meaningfully for all the colleagues who participate in such discussions faster and better now I would I do not want to stop at this stage by observing that is a problem is a big problem but we need to solve it here is something that I would like you to think about we will come back in the concluding session I will comment upon how if perhaps an idea that I am putting forward might be further expanded to work proper it is given that the number of colleagues in my own college may be very few or none with whom I can discuss things meaning but surely across the country there are a large number of people who are riding in the same boat many working in the same area as I am working in and might want to discuss something with since it is not physically possible for me to locate them go to them and discuss can I discuss with them in a distance mode to begin with how do I identify who are the people who are working in a similar area so the suggestion that I have is suppose we establish a very large research forum through these workshops remember they are more than 8200 college teachers are actually attending this workshop out of the 9000 who were enrolled that is a fairly large number most of you would be engaged in some kind of the research now suppose we as IIT hub decide to facilitate the following activity a we will put up a sort of form on the moodle where towards the end you would all put in some specific topic in which either you are already engaged in research at the level of PhD or even at the level of ME or mtech and you will write the title of the research topic that you are working on and the broad field in which you work and we will ask you whether you are interested in pairing up or joining up with other people across the country who would like to share you know their their questions with you and so on it appears to me that it is feasible to build a subservient activity to these workshops so we would in the website that we will set up soon we could keep discussion forums which could be created by such groups of people and where the discussions could happen I will tell you I took 2 or 3 extra minutes to state this point very specifically and to make this suggestion because I will tell you that in India the biggest problem is lack of a forum of like minded people working on similar problems who can discuss with each other the issues that they come to them so let us not be I am working alone in a lab full of large number of people who are all working alone but I may be working alone in my place but I work in a country which is full of similarly minded researchers with whom I can freely interact and exchange my questions and let them ask their questions and I could respond I will close with this observation thank you so much over and out