 Today, we have a presentation by Professor Sahana Murthy on how to read research papers. Professor Sahana Murthy is a graduate of physics, then did her post graduation from the U.S. I worked in MIT in physics teaching and from there I graduated to educational technology, teaching technology, educational methodology and research methodology. IIT is a unit, an academic unit called Center for Distance Engineering Education program, C D it is known as. Part of that is the educational technology cell and all these things are also, all workshops are related to that. You would remember the NPTEL project in which many IIT and other faculty members have formally recorded their lectures and these are available from either the NPTEL site or the many of them in low resolution have already been uploaded on to YouTube and may be there will be on some other site. Apart from videos, NPTEL other half is web based instruction. You take heat transfer, I think there is a heat transfer course recorded on video also there is a heat transfer course available as web based instruction. Today we have Professor Sahana Murthy on how to read, appreciate and analyze research paper. Now the illustration she is taking is a research paper given to you on clicker and that has been taken as an illustration because all of you will appreciate that. However may be that is one illustration in the main workshop there will be illustrations from may be other engineering disciplines but remember that one should not say that we should provide a mechanical engineering illustration to mechanical people, a CS illustration to CS people, a civil illustration to civil people that is not the issue and as good research worker at least the structure the content and the main theme and intent of a research paper one should be able to appreciate almost in whether the paper is from any field of science or technology. So, let us start off. Welcome to the IST workshop on introduction to research methodologies. This is the session on how to read a research paper. By now you might have undergone several sessions in which you may have heard of the scientific process, the various parts of the research process, the scientific method, the skills required while doing research and may be some of the knowledge additional knowledge that you need to develop while embarking on a research project and so on. You may even encounter some of these topics in some of the future sessions. Now the title of the session might sound unusual that is why should one need to learn specifically in a separate session how to read. After all we have been reading right from kindergarten, from primary school. So, what is so special about learning how to read? Those of you who are experienced in the research process will immediately recognize the need for developing the skill because how to read a research paper effectively and make use of it in one's research process is indeed a skill that one needs to develop and it does come as one does more and more research projects. But there are certain tips and certain guidelines which are useful to follow and that is what we will be discussing today. I should add here that although here the title and the content Professor Murthy will be saying research papers, but to a large extent what she will be discussing is also applicable to research reports, your students submission and also to some extent to other research submissions like or even unrelated to research, other educational submission like assignments, chapters in a book or a whole treatise itself. So, many of these ideas will be common, the scale and extent will be different. First let us be clear on what type of papers we are discussing. Of course, we are discussing research papers, but what I mean here is that these are not chapters of a textbook from a course. For example, if there is a fundamental course on introduction to computer programming and let us say there is a standard textbook for it, a chapter in that textbook would not be in the scope of what we are discussing today. Some of these research papers do appear in books, but those are special kinds of books and we will come to that in a moment. Neither are these opinion pieces of individuals. For example, individual X might have an idea about a certain field and that person might write an opinion piece and maybe even post it on his or her website. We are not talking of such articles. What we are talking about here are engineering and science research papers that have been written in a scientific manner. And we will try very hard to restrict ourselves to published engineering research papers. And what we mean by published here is those articles that have been published in peer reviewed journals, in the so-called established journals which are known in a community of researchers or peer reviewed conferences. Sometimes what happens is some of certain research papers in a specific topic are collected and edited in the form of a book. So, these are papers written by different people, but they are edited and collected together. So, some of these papers you might find in such a book too. So, let us restrict ourselves to papers of these kind and the guidelines that we will be discussing today are general guidelines since most of you come from a fairly varied heterogeneous background. You are from engineering or science background I agree, but the kind of guidelines we are talking about are not specific to any one discipline. They are mostly reasonably applicable to computer science, electrical engineering, chemical, mechanical engineering, even certain science subjects like physics, chemistry, mathematics and so on. So, these are general guidelines and you have to at some point determine what exactly works in your discipline, but at a broad level what we will discuss today works for all these disciplines. And many of the guidelines we will discuss today will or may not apply to survey or review papers which are a slightly special kind of papers. Survey or review papers are in between a textbook and a research paper on a current research problem. They usually contain an overview or a summary, a very critical analytical summary of all the research that has been done in a certain field up to a certain let us say within a given time frame. So, they are a very good starting point when you are trying to do literature survey and that comes in one more session in this workshop, but when we are talking about how to read a research paper, let us leave that aside for a moment. There are a few assumptions that we will work with that the main thing is that before you begin reading a technical research paper on a current research problem or maybe even something that is a few years old, the assumption is that you are somewhat familiar with the broad area and the broad idea that is being discussed. What I mean is you may have read some related paper or you may have taken a course in the field and so on. And it is if one is not familiar because when one begins a research project, one is never necessarily familiar with the exact field. In that case, what one can do is become familiar and the ways to do it is to either read a textbook on fundamental concepts or take a course as I mentioned earlier or go through certain tutorials on the topic or read a survey or review paper which is in fact would be an easier starting point to dive into a new field. There is one more assumption here I am making that there was a paper uploaded on Moodle that you were supposed to read as a pre homework to this session. It is called design and deployment of clickers in distance education. I am assuming that you have read it because a lot of examples that we will be doing today are based on that paper and it would be good if you have a print out of the paper in front of you, maybe one for every two or three participants and if you do not perhaps the coordinators can help at this point. So, let us actually look at the process of or the approach of how to read a research paper and we have called it the three plus stage approach here. Now, there is a lot of references and lot of people have given good advice on how to read a research paper. So, what we will talk about today is really a it is been borrowed from some things that people have already said and you can I will tell you sometime later towards the end of this talk on where to find some of these good references and most of them talk about the approach to reading a research paper in a process which looks like what you see on this slide here. Zero at stage. So, this is really before you actually begin the first stage is what I call get a feel for the paper and each of these boxes we are going to expand in the next few minutes. Feel is something very quick it is literally a feel you may just pick up a paper like this and you may flip through it. Then the first important stage is to get the big picture. Then one gets the details from the paper, one evaluates the details and to get a good idea of the paper one has to at least go through stage 3 and if you are a research student or a researcher who is using the paper for your current work you often have to go to stage 3 plus where you have to synthesize the details. So, let us get a look at what we mean by get a feel for the paper and how to do it. By the way each stage will require one reading of the paper. So, when you read a research paper it is a multi or you will have to read it multiple times. In fact, that is one of the questions we explore later. So, how do you get a feel? First thing is read the simplest thing the largest thing on the paper and the largest font is usually the title. So, read the title ponder upon it if you would like for a few seconds and then as I mentioned earlier pick up the paper and see how long it is and papers vary in length a lot. Typical and the variation is really huge it can be as small as 4 papers and sometimes you see reflection papers as small as 2 pages and sometimes they might go to 30 or 40 pages. So, you need to get an idea of how large the paper is and typically the length of the paper also is depends on the type or where it is been published. For example, conference papers for technical research conferences are usually between 4 and 8 pages. Journal articles are longer and these can be small like 5 or 6 pages it can actually even be 4, but it can go up to 15 or 20 pages typically. Review or survey papers are usually much longer. Second, the next thing you need to know when you are just getting a feel for the paper is where is this paper published? The bibliographic details as we call it and one needs to know how to find this information and where to find this information. Typically what happens is it is available if you have downloaded the paper from the journal website or if you have photocopied it from a specific journal, you should look at the header or the footer of the page and you will find that information. If you have downloaded it from a website from let us say somebody's personal page, see if they give bibliographic details and what we mean by these details are the year, the title, the journal, where it has been published and so on. Look at the figures, just glance at them, see what you think and read the headings. So, when we say get a feel for the paper this is all you do and this can take as little as one minute sometimes. We will come to how long should I spend on a research paper, we will come to this question little later, but this is really getting a quick feel. In fact, you would be able to get a feel of a paper much quicker than it took me to go through this slide. So, one thing that we will have to look at before we go on to getting the details of a paper is that the scientific research paper is actually a very peculiar piece of writing. It is not a story, but it is not a story as in fiction, but it has to tell a story and it is very highly structured. In fact, the moment you get a little experience, you will be able to predict what are the headings in the paper. Most papers follow specific structure. Every item in the paper is there for a reason. There is nothing even the figures are not there to make the paper look pretty or make it look good aesthetically, but there is a scientific reason why each item is there and that leads to the next point that each part of the paper is connected to the other part. And this is true at the sentence level. This is true from section to section, from paragraph to paragraph and so on. So, when we are reading, we have to be able to follow the links. Then, there is always a very high correlation between the figure and the text. For example, there is a place in the text which refers to the figure and the figure will refer to the text, some other piece of text back again. So, one needs to be able to easily be able to locate the information one is looking for either in a figure or in the corresponding text. So, research paper does all this, but the author is severely constrained by space because often there is a page limit on the page limit on the paper. And because it is so peculiar, by the way, if you are familiar with research papers, this is not anything new, but if you are not familiar, you would require some time to become familiar. It takes a few papers before you become familiar with the peculiar structure of a research paper. So, the next slide is something which is going to be old information for those of you who are experienced, but if you have not read too many research papers, then we will just go through a typical structure of a scientific research paper. And if you look at the slide in front of you, you see that the first two words, the title and the abstract seem to be in a group. Then there is a long list of words in one more group and right at the bottom there is a references. And the reason the slide is shown in such a manner is that the title and the abstract is always shown right at the beginning and it is very easy for you to locate it. The abstract usually has the word abstract in front of it. The title may not say title, but there is no way you can mistake a title of the paper, at least a well written paper, because it is in the center, it is the first thing you encounter, it is bold, it is in larger size and so on. Similarly, the references at the end are something which you can easily locate, it is always there at the end. The pieces in the middle are will be there, but they may not exactly follow this sequence and they may not, you may not find headings for each of these, but some of these, some a large subset of these pieces will be there. And just to read this, the introduction is usually, it is almost always the first paragraph, whether it says so or not and very often it will say so. You will find some background and motivation for the work, perhaps in the introduction itself. Then comes something which is very important, but often hard to find and that is the contribution of the paper. That is what is the authors actually, what are the authors claiming to do by writing this paper? What is the central take away from this paper? Sometimes it is found at towards the end of the introduction or somewhere in the middle of the introduction, sometimes it is found in the conclusion and so on. So, similarly there are many other pieces which one should look for in a research paper, but most of these pieces are going to be there. The structure is a fairly well defined structure for scientific research papers. Before we continue, let me comment on something. Professor Murthy has emphasized that you have to assimilate a paper and while doing that, you may have to read it multiple times. It is multi scanning. In any branch of engineering, when we appreciate some work pertaining to that branch, we have to do or we do multi scanning. For example, let me take mechanical, civil or that type of engineering, the traditional ones, where you look at a drawing. You do not look at the detail, first you look at the whole sheet, you find out okay, here is the plan, here is the elevation, here are the general layout, here are the sub assemblies, then there are some part drawings and then that is the first reading appreciation. Then in the second reading, you find out what each one of them is about and then in the third one, you go to the detailing. The same thing must be true of electric circuits and the same thing must be true of say in computer science, you have, you rarely have single pass compilers, you have multi pass compilers. Each pass does something in a more and more detailed way. In a similar way, when you appreciate and assimilate a paper, you will have to do multi passing. I remember an essay of Francis Bacon, I think of studies or of books, all his essays of something and he says that some books have to be tested, some to be swallowed and still a few others to be chewed and digested. Well, that was maybe 150 or 200 years old, but nowadays our literature, you have to chew and digest if you really want to appreciate it. Okay, so now I would like all of you to do one activity and this is based on the uploaded paper that I talked about at the beginning, the design and deployment of clickers. So, what you need to do is on the paper with a pen or pencil, mark all the pieces that have been mentioned here, the title, abstract, introduction, simply mark them and as soon as you do it, discuss your answers with your neighbor or if you want to work with one more partner, that is also fine. So, this should not take more than one or two minutes, it is okay if you do not find all of them, but I just like you to mark some of these pieces. Hopefully, you would have been able to locate most of these pieces in the paper provided. So, let us in fact discuss some of these, we will discuss how to locate or where to locate some of these pieces. So, what I have in front of me is really a copy of the paper and it is okay if you cannot read the paper on the screen, but what I want you to do is look at the colors, I think you should be able to see at least that. Title, I am sure all of you found, it is what is being shown in yellow. Abstract, in fact here it says, it is the abstract and it is usually the very first paragraph, it is either there in different font or it is usually it is in smaller font, a bold font or some such thing. So, now let us go through the paper a little bit more, let us go down and what you see here is the title introduction. So, there is a section called introduction in the paper. As you are reading the introduction, the second and third paragraphs on the introduction, you should be able to find, you should be able to realize that the background and the motivation for the paper has been written in these two paragraphs. So, these two paragraphs tell you why this paper, this research work was done, roughly what problems it was based on and so on. As you go further in the paper, so I will go back and forth. So, you will see that the order is not necessarily the same as the order you saw on the list. The contribution to the paper is what is marked in blue. It says, let me just read it a little bit, so that you will be able to locate it. In this paper, we describe a solution to facilitate real time interaction between instructor and remote classroom students through the use of a student response system using clickers. We designed and developed a distributed architecture for a student response system. So, that is the key contribution that they are mentioning here. So, as we mentioned, as we discussed earlier, this is present within the introduction, but you may have to look a little hard to be able to locate it. Let us look at a few more. There are few sentences above the contribution, what is hopefully showing up as green. Are the research questions that this paper will address? And here it says, for example, one of the questions here is, what modifications need to be done both in terms of development of new technologies and in the pedagogy used. So, that is a research question that will be explored by this paper. Then, right in the introduction, they even talk about the solution outline, the last paragraph on page 1, what is given as purple here. The solution outline is just an overview of what is going to come in the rest of the paper. It is like the preview of a movie. It is not giving the details. If we go further in the paper, we will just see a couple of others. Page 2 has something, a little bit more on related work where the paragraph that starts with a similar situation was implemented by University of Oklahoma, but you should also be able to find related work in the section on background on page 1. So, what this really tells us is that if we are trying to locate the various parts of a paper, they may not be in sequence. They may occur in multiple places and so on. So, when you are trying to read a research paper, in the beginning it may be hard to try to find them, but these are exactly the pieces or the key points that you should be looking out for and we will now see how to, in fact, locate them. Sorry for interrupting, but when we are discussing this another paper, you observe that she points out that some information you get here, some information you get here, etcetera. I think this discussion also constitutes a good set of hints for prospective paper writers. So, when you write a paper, it will be reviewed by someone and reviewer will be invariably looking for these things, many of you would have reviewed papers and you would be slightly upset in your mind if you find some relevant information here, some relevant information there, some relevant information there. So, I think it will be a good idea to stress over colleague teachers when they come that understand this although this is a well written paper, reasonably well written paper because it was accepted in a conference, in a reviewer conference, but still there are lacuni which can be correct. So, just as reading papers multiple times helps you to understand things better and better, rewriting paper multiple times before submitting also makes the paper better and it would be good idea to keep these things in mind. I think that is an additional input that we can give to our colleague teachers, yeah. Mr. Rathod is saying that in this paper one thing is missing that is the authors and affiliations are missing, there are two reasons for doing that. First of all whenever a paper is sent for peer review, the authors and affiliations are removed, that is in order to remove the buyer. Second, when we are circulating a paper to a larger audience, then there is a possible problem of plagiarism, legal plagiarism. Ordinarily circulation of papers for academic work is considered an accepted norm, you do not require any permission, but if I distribute the paper with a permission to redistribute, which is what open source is all about, there is a thin border line saying whether this is legally right or wrong. Academically I do not see any reason, nobody is going to make any money on it, anyway the copyright on the clicker and all the algorithms belongs to IIT Bombay which will be released in open source, but technically this paper copyright have been given to whatever I think I triple, whichever place we publish. Now is it alright to distribute that paper? So we are proceeding with caution, we are removing the authors this, yes, yes. In general a paper would have authors and affiliations, but in general when we are discussing a paper, any paper particularly some of the papers which we plan to give are ME dissertation reports. Now we will obviously choose M.Tech dissertation from IIT Bombay itself, but for the sake of self esteem of that individual where we are going to circulate the first draft which is very shoddy believe me, then we I mean that person would not like his or her name to be known to people that is what I have written. So that is the only reason, otherwise you are very right, normally papers should have it. When you consider legal aspects of these situations, things become very hazy because nobody has gone to court yet and no court has defined and unless there is a court case and there is a judgment which then becomes a beacon for the subsequent behavior, we cannot say one way or the other. It is just a safety net, sorry. Column, there is a series of questions and this is title what are you looking for? The right side says where to find it. So let us just go through a few questions. In fact the first question you should think about is what research area or topic or subtopic does a paper fall under? Starting from a broad area you can say that it is an electrical engineering paper and this one layer, one level of subtopic might be communications and then you can say well it is on digital communications and so on. You can go for, you have to go much further. Then this is something by reading the paper, you will be able to, this is a question that you should be able to find. Ask the question what problem does the paper attempt to solve? So that is the big question here because every piece of research work is attempting to solve some problem. So you are now trying to or you are beginning to identify what this problem is. You are also trying to get a big picture of what is the related work and importantly if there is work on it beforehand, why is it not sufficient? What was the need for the authors to write a new paper? That really means that what you as the reader of the paper have to, has to identify is the gaps in the related work. The authors would have written it if they have written a good paper. Again you are looking for the key contribution. We tried to locate it earlier and then you are trying to look for broadly how does the paper solve the problem. This is the solution outline. Again you tried, you did locate it. Then you have to come to the meet of the paper, the central part of the paper. How do the authors defend the solution? And I will come to the last question in a few minutes. So now what I would like you to do is look at the right column where to find it and we are only looking at the big picture right now. Take a moment or so and just read aloud each of these cells in the right column. This is title abstract, title abstract introduction, introduction, title abstract, introduction, conclusion, introduction figures and what you see is that the word title, abstract, introduction has appeared many, many times. In addition figures have appeared, oops excuse me, in addition figures have appeared multiple times and the conclusion has appeared somewhere and the word headings has appeared somewhere. So the point we are trying to make here is that when you are trying to get the big picture you should mainly concentrate on these topics. By the way if a word is in parentheses what it means is that you may or may not find it in the title. So sometimes you may find it and sometimes you may not, but you will definitely find or you should definitely find the contribution of the paper in the introduction itself. So how you get the big picture is really by reading the title, abstract, introduction and conclusion not the entire paper and the reading this much comes to roughly two pages, maybe sometimes even a little less than that unless the introduction is very long, but introductions are never very long. Then you should go through the section and subsection headings and then look at the figures, diagrams, illustrations and so on. This is all you need to do at this pass, at this stage. Then once you have read these or simultaneous along with reading these what you have to do is write the answer to the following questions. The exactly the same questions that we mentioned in the in the second column here, what research area, what subject, etc. Now where do you write these answers? That is another question. That is a skill. Again these are small tips that might help you. A good idea is to keep making notes when you are reading the paper and some people make notes in the margins of a paper. Some underline, some use a highlighter on the paper itself on the hard copy. Some people keep a separate notebook of file. It really does not matter which one you do, but I know that a lot of people do annotate in the margins or use a highlighter because you want to have the paper as well as your notes and your perceptions and your ideas together. So, making notes on the paper itself while you are reading is in fact, a very useful idea that our teachers told us and their teachers have told us and so on. This is one more important point that used to this phase, particularly whenever colleague teachers are used to looking at material mostly on internet, very few people would actually be perusing printed research papers from journals. For example, many colleges may not have journals as we discussed yesterday. Now, one more point is that while you peruse material on a screen, you have less of a willingness to simultaneously write down something. First of all, you cannot write down on the margins of the paper because paper is not in a printed form. We in IIT typically use PDF files as a tablet PC so that you can make remarks on that tablet PC itself, but that is a very costly device. If you recall in the old days, many of you would have pursued research papers in the library when there was no internet or something. We would carry those index cards and for every paper that we read, we write down the title, abstract, some additional information. In fact, I learnt it from my own guide and guru Prasad K. K. K. who said that unless you write while you are reading, there is no objective self because he mentioned that you are reading automatically re-emphasizes your retention in the mind of whatever you have read. Now, this is not about articulation. You are not writing fantastic English to please somebody else, but you are trying to write in your own words something there. How do we re-establish this habit amongst our fellow teachers? You would agree that this is a important habit? In fact, many of us miss it when we more and more frequently peruse papers on internet. You look at the paper, we do not write anything because we do not necessarily carry it. So, a simple discipline that even if you are reading on the net, you will not start reading a paper unless you have a notebook and pen rate for example. And secondly, you will not consider having read the paper, the activity being completed unless you have written half a page or one page about it, whether you copy, abstract or whatever it is. First of all, do you think it is a good idea? Then we should emphasize it. And I will even go to the extent of suggesting that in the afternoon lab, when you take them to a machine, we might have either preloaded paper or we will ask people to surf internet, search a paper in team, but every person must write down something about that paper in this fashion. Do you think that will be a good exercise sort of because if some people start forming a habit out of this, that will be useful. Yeah, yeah, yeah, they can write down on this paper while doing this exercise, but I want to additionally emphasize this. You remember we said that we will be forming teams of teachers there and if these teachers will be given some task to be completed afterwards, but to form a team just by naming them is not good enough. That team should do some team activity while they are at your remote center. Then only the team formation takes place. Team is not just by name, but what people want. One of the activities that we can give that team is for let us say five members to peruse the same paper of their choice in their field, but independently and take down notes independently. And these five notes are then compiled, commented upon in a 15-20 minute discussion amongst the team. And then the team leader consolidates the final writing about that paper from all the five people and submits it to the center corner. So, that could be a very useful exercise and a sort of assignment submission as a part of it. You agree with this? Okay, so let us come to the next activity and this is a slightly longer activity. This activity will work well if you have read the paper and if you haven't read the paper, I would suggest that you pair up with somebody who has already read the paper before. This is the exercise that we are talking about and I would, although it might sound mundane, but I would like each one of you to do it at least once for five, seven minutes because writing one to three lines like this is not very straightforward. We will do the same exercise with the participants. You will have to give them printouts at least one or two days ahead and then they should come having read that paper and do the same exercise. But if you do it once, it will be useful for you to supervise the activity. So as Professor Rathod, I was telling you that this is an activity which we will do during the lecture hours. But what I was talking about is a similar activity to be done by them on some other paper of their choice during this. So please carry on. We will take a, not interval, but we will allocate about seven minutes should be good enough because five minutes. He says five minutes. Anyway, five to seven minutes doesn't matter, but I would like each one of you to just look at that paper. And what I would like you to do is on the printout, I am assuming you have a printout, you make notes to answer the following questions. Somewhere on the margins on the side and if you want, you can take an extra sheet of paper also. It is exactly the same set of seven or eight questions that we have been talking about so far. What problem is the paper trying to solve? What is the related work? Why is it not sufficient? And so on. One point we haven't discussed too much is how do the authors defend the solution? And what I mean here is the authors have, after giving the background, they would have talked about what problem they are trying to solve. They would have written about what problem they are trying to solve. And then they would have given their, some outline of their solution approach. That is still at that point, it is an outline or an idea, it is an approach. They have to now convince the reader, they have to convince you as to why that approach is reasonable, why it works, why it is sound and so on, why it is logical. So, the defense of solution ideas usually are in the form of proofs or an experiment which has been conducted and its analysis and results or it may be an implementation, it may be the building of a system and so on. This name was not even known to me when I started designing the affordable clicker in the lab. I found this name used in several papers pertaining to high school education where these devices were used. And this name clicker has emerged as some kind of a what should I say favorite slang rather than a technical name. So, a device on which you click, but today we do much more than clicking. So, technically I clicker is not the most appropriate name, but then that is so with many things. But this is a good point that the whole paper assumes clicker to be well known. Then we will have a demo tomorrow of the Akash tablet being used as a clicker. So, at that time I will explain more about the clicker. This device you would have seen in the well known serial Karnbanega Karodpati. So, where you collect the audience pole, now the device which each person has in front of him or her is actually a clicker because they are clicking A, B, C or D. So, this applies to the collection of response for multiple question. This primarily for your own personal benefit because when you supervise this activity in the workshop you would have a good idea of roughly how long people might take it. And there will be some very slow thinkers and writers and then there will be some who are street smart who can finish it off very quickly. You have to do all authors. Only when you refer to, but when you refer to a paper in the text of your paper, when you refer to somebody else's paper you can say Gaitonde et al. But references must contain. Oh my God, 90 percent of the fights over credit come out of not only the names, but in which order they are and so on. That is incorrect. The guide should remove that or the guide should remove his or her name from the author's list. If you have provoked me, let me comment on the inclusion of names in the author's list. In IIT, Bombay there are several faculty members who if they have not contributed either to the main idea or the writing of the paper, specifically insist that their name should not be included in the author's. That is the academic standard that we ought to maintain. On the other hand, some 25, 30 years ago I said in Amdabad in an image processing conference I had gone along with Satsrabuddhe. And I was quite thrilled to find about four different papers, each one of which had multiple authors, but the first author was same. And I was looking for that person, the great man, because he had a topic. I mean he had a, of course there are people who have a broad bandwidth. They can write on them. But I could not locate that gentleman. So I attended one session. Then I thought he will present somebody as one of those other three presented. Then I went to the next paper at the end. I could not resist the temptation of asking the presenter saying, I wanted to meet him. He's not there. Oh, he has not come. The first author is, no, no, he's too busy. He's the director of the institute. Then I got bit worried because the directors of the institute do not get so much time. After all, they should either direct the institute or do research. They can't do both. So I attended the third session where there was a young sort of brash person and I thought he might open up. So over a cup of tea I asked him and he said it was a either CSI or Defense Lab. And any paper that you publish required clearance from the director of the institute that the paper can be sent for publication. And it seems some four or five years ago people had discovered that if the director's name is there as one of the authors, the paper gets cleared faster. And if his name is the first author, it always gets clear. This is, I mean, this is terrible. This is terrible. If somebody should have guts to question that director, but given the hierarchical society that we have, where we are very good at always criticizing our juniors but never raise our voice against seniors. Now that is contraindicated to good academic progress. Exactly the opposite should happen. It's another thing we are very proud of in IIT. I mean, the youngest faculty members are most vociferous in criticizing the elder. I mean, just to tell you one instance, when President J.R. Isaac was telling people, younger people were objecting to the teaching law being too much. And President Isaac said, what is this? You know, you should be able to spend time in correcting papers quickly. You should do this. You should do this. You are all young people. And President Venkatesh was the youngest faculty member who had just joined. He says, President Isaac, I have done all of that in the last two years. Have you? And President Isaac was speechless because he had never written and corrected answer books in time and all his life. So, you see, it is important that someone points out my mistake to me. This does not happen. So, what I'd like you to do at this point is pair up and work on this activity. I think it should take about maybe 10 to 15 minutes. Hopefully, most of you would have worked on the activity where you were trying to answer the questions posed on this slide based on the paper that was provided. What we will do now is discuss some of these questions. So, we will try to, what you can do is compare your answers while we do the discussion. For example, what research area or subtopic does this paper fall under? And this paper actually is an engineering education research paper. The authors who were doing the research did develop a system, but the goal or the reason they developed it is to improve engineering education. There was not anything that was much new in terms of the technical details of the system itself. They did have to do some work, but it was not a distributed architecture research or it was not networking research and so on. The research part there was to build the system for to improve some goal within education. What problem does this paper attempt to solve? And if you have read the paper, if you have tried to answer this question, you would have realized that the authors are trying to, the researchers are trying to implement clickers in a distance education scenario. So, you can state the problem as how to embrace the implementation and benefits of clickers for distance education. They did not invent clickers that should have been clear from the paper. They have used the known benefits of clickers or they have used the idea of the known benefits of clickers and then done the necessary modifications and implemented it in distance education. What is the motivation for this problem? This is stated very clearly in the introduction of the paper that distance education lacks interactivity and in face-to-face classes, there are many studies which say that interactivity improves due to clickers. So, what the authors are trying to do is take a problem, take a solution from a different domain and apply the solution from a different domain that is the face-to-face classrooms in the domain in which there is a problem. So, this is one typical way of doing research, where you adapt a solution known somewhere else into the problem at hand. What is related work and why is it not sufficient? So, again the paper says that there is a lot of work on the benefits of clickers, but all of them are for face-to-face classrooms. There is very little work on the use of clickers in distance education. So, what they have done is identified the gap. They have shown us why it is not sufficient, why the related work is not sufficient to solve the problem. And if you go back what is the problem? It is improving interactivity and embracing clickers in distance education. What key contribution or contributions does a paper claim? So, I think they are doing two things here. One is that they are telling us, they are providing us the architecture of implementing this distributed clickers or student response system in multiple remote locations. So, they are showing us how to do it and then they are showing us proof of concept that they actually did it. Now, it gets bigger and bigger. Broadly speaking, how does the paper solve the problem? And this you would find on the first page of the paper towards the bottom. It is what is shown at the bottom in the purple solution outline. The instructor delivers lectures from a central location. This is the broad solution and this is transmitted to remote classrooms which were equipped with receivers. The receivers communicate to a server in the central classroom over the internet. Participants respond to the questions presented by the instructor through clickers. In fact, another place where you will find how the paper solves the problem is a block diagram. So, let us we did not see that earlier. So, let us take a quick look at that. If you look at page 2 bottom, there is a block diagram. So, the introduction and the figures are a useful place to get a broad idea of what is happening in the paper. Let us look at one more question that we have to analyze. The last one, how do the authors defend the solution? And here in this paper they towards the end of the paper you will see that data of clicker responses were collected. You will see figures and instructor and participant feedback on the clicker system were analyzed. So, this is the proof for the defense of their solution. Let us take a try to find those. As you see I am skipping the entire central part which talks about the details of the actual implementation. This is on page 5 at the top. There is a graph, then there is a table right there. There is some text which tells you what the findings were and how they defended the solution. Similarly, I believe on page 6 there are more graphs. So, when you are trying to look at defense of the solution, try to look for experiments, try to look for descriptions of the results, the graphs and so on. Let us go on. I will just go back to one of the earlier slides and we will see where we are so far in the three stage process. We looked at how to get a feel and how to get the big picture and now let us see how to get the details. In fact, the type of questions that you will see in how to get the details are not very different than the type of questions you will see in getting the big picture. It is just in the amount of information that one needs to look for. So, this slide will quickly go through because if you look at the left hand column, the questions will look very familiar to you. What problem does the paper try to solve? What is related work? What is the contribution? How does the paper solve the problem and how do the authors defend the solution? These questions should be very familiar to you by now because you answer the same questions just to get the big picture, but the difference is in the column on where to find this information. At this stage of reading the paper, you will have to look at sections beyond the introduction. For example, on related work, you have to try to find the literature survey or the related work section. Most of the times you will find an entire section either labeled literature survey or related work. You may sometimes find it as part of the introduction also. How does the paper solve the problem? This is the central part of the paper that we have skipped so far. This section will either we can have one of several names, solution, experiment, system, implementation. It is not very different papers call it by different names based on the context, but most of the times it comes in the middle part of the paper after the introduction and before the conclusion. How do the authors defend the solution? You will get all the details again in the middle part where the paper talks about the experiment, the methodology, the results and so on. So, while the questions we are asking so far are very similar to the previous stage, we are now looking for more details by reading more of the paper. But we have to look at, we have to answer more questions. These are not the only questions that we need to answer. So, let us look at a few other questions at this stage. What is the precise research question address? So far, we were looking at the broad problem or the motivation. Now, we are looking at a question that says research question and the difference in this paper is so on, is the following. The broad problem could be that the researchers want to improve interactivity in a distance education scenario. And the precise research question is how to implement clickers in a distributed system, how to firstly build this distributed system to implement clickers and how to implement them so that the clickers retain their effectiveness. A precise research question is needed because that is how the solution can be developed more effectively. Some of the other questions that you, the reader has to try to answer are why is it, why you will have to try to find out why the authors believe the solution works. For example, why do the authors think that the solution is better than other solutions? So, the reader has to try to get into the researcher's mind and try to answer these questions, not exactly the researcher's mind, but into the researcher's writing. Then comes a very important question as to what are the assumptions the researchers are making? What are the boundaries in which they are working, the boundaries or the scope? What are the limitations? So, for example, in this paper, what I would, what I would write what we can do is we will have to go back and read where the authors state their assumptions. Then comes the details of the actual solution and as I mentioned earlier, this is in the form of an argument or a proof or an implementation or an experiment depending on the type of paper that you are reading. What type of evidence is provided? We have gone through it a little bit, but now you have to look at the figures in more great detail. You have to critically evaluate the results and so on. For example, what you need to look at in the figures is the following, I am looking at this part. Look carefully at the figures, especially the graphs, see if the axis are labeled, see if error bars are put on the data points and see if together all this is making sense and all this is substantiating the claims of the author. If you see a graph without any labels on the axis or without units, then it is a shoddy piece of work. So, when you are reading good research or a good research paper actually pays attention to all these details and the overall paper should give you an idea of what is the takeaway message of the paper. We would not do an activity right now on this because that is going to take a lot of time, but this is going to be a homework activity for you and later we will try to, we will see how to upload the solutions and so on. So, now we come to the final stage of paper reading and this is where you will have to take a much more active role. So far you were the reader and you were just trying to locate the answers to some questions. The researchers and the authors were the ones who had done most of the work, but now you have to think more, you have to do more work and so on. The goal of the stage is really the question at the bottom. You have to try to decide what is the paper trying to convince you of and you also have to decide does it succeed in convincing you of that question. To be able to do so, there is a long list of questions that you can go through and if you have submitted a paper to a conference or a journal, you may have seen reports that the referee has given you and the referee reports usually contain answers to these questions. Is the research problem significant? Is it important enough? Is it worth writing a paper? That is something you have to think about. Is the problem or the solution novel or new? Now how would you know that? And here again your experience and your ability as a researcher comes into play because you will know if a problem or solution is novel, if you are up to date with the current research in that given field. If this is the first paper you are reading in a field, you may not be able to judge if a problem is novel or not. So, your experience and your knowledge and your skills play an important role in trying to evaluate a paper. These skills do develop, they definitely improve as you read more and more papers critically and let us say you try to read every new paper and answer these questions, your skills will definitely go up. Similarly, let us take a few others. Are the assumptions valid? Most research is done within some boundary. For example, the paper on distance education may be the people tested it in remote locations which had a certain kind of technology and only that kind of technology. So, their scope is limited to locations where a certain level of technical technological sophistication is available. Perhaps the solution works in other places, but the researchers have not tested it out. So, the assumptions have to be clearly stated and another assumption of this paper, somewhere towards the end if you have read it, you will see that there were improved results of student satisfaction and the teacher was also very happy with the level of interactivity. So, one assumption here was that the teacher had some training in using clickers beforehand. If it is the first time that the teacher got clickers to use, perhaps he or she would not know how to use it and their experience may not have been so successful. So, what are these assumptions? Are they identified and are they valid? Is something that you as a reader has to keep checking. Another thing which you can try to do is see if the different parts of the paper are consistent. So, the solution is it actually solving the problem that has been posed and the results is it really giving results to the solution approach that has been implemented. The reason I mentioned this is when we get papers from novice writers, we often see that the problem is something and the solution is something else. So, just you have to make sure that the two are consistent and then other things like does the paper flow logically and so on. Finally, we come to the most difficult, but the most interesting part of reading a paper and perhaps the most useful because when you are doing research ultimately you want to further your own research and you want to use the paper that you have read for your own research work. So, how do you do that? That is why this has been labeled as synthesize. Synthesize is when you read a lot of different or you read or you yeah you can read a lot of different things and then you have to analyze them. When you are doing the evaluation, you are doing analysis. Here you are doing something more, you are putting the pieces together after analyzing them and trying to create something new. So, this is a much higher order skill that will develop. So, questions you can ask are let us say the authors have solved a certain research problem. You can ask are there any other ways to solve the problem because if you can think of a different way to solve the problem that means that gives you an idea for a research problem. In the current example the researchers were trying to solve the problem of interactivity in distance education. They used clickers, but can you use something else? For example, would a two way audio video chat be a better or an alternate solution that addresses the same problem or is there some completely new software or hardware technology that you can develop to solve this problem of interactivity in distance education? So, asking the question is there any other way to solve the problem is a good starting point to coming up with your own research problem. Another way to come up with your own research problem is the second point here. Let us say the authors have made a claim and they have done one experiment to justify their claim. You have to ask is there a different experiment I can perform or is there a different proof I can give? It is not that you are coming up with a new completely new solution, but just a new way of defending the solution that itself can give some research ideas. You can ask you can go the opposite direction. Let us look at the third question that is also an interesting one. Are there counter examples or arguments against the paper claims? So, let us say you find that the authors what the authors have done is correct, but it is correct within the situation that they have tested. And in the new situation you have come up with an argument where their results may not hold. This happens a lot in engineering and science. So, for example, the researchers might have developed might have come up with a result in one range of temperatures. And you may have an argument or you may have a principle by which you can say that well in the other range of temperatures the result is something completely different. So, that gives you an idea for a new research problem to extend or explore the same problem in a different set of conditions where the original results of the paper you have read may not hold. You can ask questions like how can the results be improved or how can the results very importantly the last but one. How can the results be generalized either in a different context like the temperature range example we talked about or can they be extended. And finally, the question we are going to leave with is really what are the new ideas and open problems suggested by this work. Because that is one of the main reasons why you read the paper to get ideas for open problems and new ideas. It is a difficult question and you cannot solve it by reading the paper once or you may not even be able to solve answer this last question by reading the paper five times. But it is a question you have to keep in mind and discuss with your colleagues and your guide and so on. So, some homework activity now the same paper that we have read so far write a two or three page review which contains a summary and essentially what you have to do is we have posed a lot of questions in this presentation. You have to use those questions as guidelines and critically evaluate the paper first. And once you have done it try to come to stage three plus where you are trying to ask the creative questions what are the new ideas what are the open problems and so on. This again this can be done only after you have read the paper a few times feel free to discuss it with your other colleagues and people in the workshop. So, let us come to some concerns and some questions that are frequently asked. One is how many times do I read a paper and by now it must be clear to you that the answer really depends on your purpose. What why are you reading the paper what is the goal? If your purpose sometimes your purpose is just to decide if this paper is relevant to your work or perhaps somebody has found a paper for you and is asking you is this really a valid research paper. So, at that point you only want to get the big picture details go up to stage one and that is sufficient to help you decide if the work is relevant and interesting and worth reading further. This may be one or two readings. Once you get experience this is just a single reading. If your purpose is to grasp the contents of the paper and to summarize it for example, let us say you are writing a literature review section for your thesis. You have to be able to grasp the contents then you should go up to stage three where you are evaluating the paper. You are critically looking at whether the figures are consistent and whether the claims are correct whether the solution is defended well and so on. So, you are looking at every piece of the paper and asking is this correct is this consistent is this sensible and so on. If your purpose is to come up with new research on the basis of this paper is to base your future work on this paper then what you should be able to do is actually recreate this paper in your mind. You should be able to work out every single step and that should help you get to the next step. In that case you have to go up to the stage which we called as 3 plus synthesize the paper. You will find papers that belong to all three purposes all three categories as you read. Not all can be used to synthesize and you will not stop reading only the introduction in all papers. So, some papers will belong to the first category some will belong to the third. A lot of papers that you will use for your literature review section either in a paper or in your thesis will belong to the second purpose to grasp the content of the paper. Next question how much time should I spend reading the paper? By the way there is a big disclaimer here these are very approximate times it may be more or less, but these are typical times and it varies depending on whether you are experienced or whether you are a novice in this practice of paper reading. To get a feel as I said earlier experienced people just take maybe 30 seconds to a minute as a beginner you might need 5 minutes because all you are doing there is reading the title seeing how long it is trying to see what category it belongs to and so on. To get the big picture I think a beginner would require about an hour and an experienced person beginner may even require 30 minutes maybe 30 minutes to an hour experienced person might be able to do it in 5 minutes. Then to get the details and to critically evaluate here you need a lot of details and you need to be able to think critically and analytically. I would say that a beginner requires about 2 hours here and by now you may have read the paper 2 to 3 times. Finally, to creatively synthesize how long does it take? It could be a solid 6 hours or it could be a whole week if you are thinking about it continuously one can never say at this point. If it is a very important paper and interesting to you then you once you are experienced you may be able to get this done in an hour or 2. What should I do if I still do not understand the paper? Because sometimes you may have read it a few times 3, 4, 5 times and you still do not understand. So, what do you do then? So, first before going to what to do let us look at what are the possible reasons. Maybe the topic of the paper is entirely new, the subject matter is new or maybe the paper contains jargon unfamiliar terminology. It may contain a lot of acronyms you are not familiar with that may be one reason or you may be familiar with the broad area, but you do not understand the technical details of the experiment. This happens quite a few times because you may know the principles and the theory, but the kind of experiment that the researchers have done or the new system that the researchers have built you may not know what is how it works. So, because of that you may not understand. Sometimes you are just tired because you have read the paper many times or the paper is very intense or heavy as colloquially some people say or maybe it is not anything to do with you and the paper is so poorly written that it is very hard to understand it even for an experienced researcher. So, there could be many other such reasons. So, what to do really depends on what which of these reasons is applicable. Some of the things you can do is first thing I have said is do nothing. So, if the paper is not important for you and you think it is a struggle to read it do not bother. If it is not important you have to somehow make that judgment. For example, if it is an area that you will not be working in at all. If the reason is that you are mentally tired, let the paper aside sleep for the night and pick it up again the next day. One thing that helps a lot is to discuss a paper with a colleague may be a fellow student who is working in the same area. Let us say you are a PhD student or a fellow colleague who is in that research area. So, discussion or your guide for example if you are in an M.Tech or PhD program. Discussing with colleagues is a very good way of trying to make sense of some parts of the paper that you do not understand. One more thing you could do which you can do on your own is read something else. Let us say a fundamental exposition of the topic either in a textbook or a survey paper that talks about this research topic and come back to it. That is another way to try to make sense of this paper. Last few points, most papers have a long list of references. So, what do you do about them? Should you ignore it completely or should you read through every piece of it? And again the answer lies somewhere in between. Suppose you are not familiar with the area at all then the references may not mean much to you. On the other hand the references are a good way for you to become familiar with the area. So, if you want to understand the paper more clearly or more deeply look at the corresponding reference. If you want to build upon the current paper let us say you want to get a broader idea of the current paper or if you want to get a broader knowledge of the research area the references are a great place to look at. So, what you would do is not look at every single paper in the reference, but let us say the part of the paper that talks about the conditions in which the experiment works. Let us say that is the part which is interesting. There may be some reference related to it. Look at those references and that would broaden your knowledge of the research area. So, that is why there is a star there. The references are a very important or they are a useful way to broaden the knowledge of your research area. Thank you. So, what we will do next this is just a list of references that you can read in order to learn how to read a research paper. So, most of this presentation has been adapted from some of these references and there will be one more session on what to do after you have read one research paper. So, typically one reason for reading a research paper is to write the literature review section. So, how do you write it? How do you select which papers to read? Where do you find these papers and so on. So, this session was on reading a single research paper and now we have to go from one research paper to reading multiple research paper and what to do then will be another session. Thank you.