 Welcome back to learning analytics tools course. In this week, we will just overview what we learned in this whole semester. There will not be any new topics taught in this class, but we will revisit what we discussed in an abstract view. So, in this course so far we talked about data collection, that is what data to collect and permissioned environment. Then we talked about how to represent the data that is very key and important aspects, because we can collect data in a different learning environment from a different resources. But representing data is very key. I suggested a simple method that is, I suggested the method that is you always have a timestamp, timestamp and user ID. If you allow the students to do multiple sessions, your classroom environment where students have to do it two, three times come to the class or they have to interact with the system for multiple times. Or in online, they have to interact for multiple times, collect the session ID and the action they are doing. What is the action? Are they doing reading or they are doing some uploading a task or assignment, something like that. And from the actions, you also should collect the contextual data. The contextual data is where this action is taking place, where and what are they doing like if they are and if they are reading some page, what page they are reading and why they are doing it. They are simulator, they are interacting with the uploading assignment, they are uploading for what, for performance or some other indicator why. Also, if they are answering some questions in a quiz or in online assignments, please think about what is the option they select and what was the correct option. So, if the action is answering to a question in MCQ, so you need both correct option, what option is selected. So, think about all these things, you know, current option, what option is selected. So, this is the action and the contextual information. So, how do we come up with the actions in a three different environments? That is a very key important point, right. So, I might have in interacting with the tele technology enhanced learning environment, I might have interaction like navigating from one window to other window clicking some buttons, every clicks I am captioned, right. That is why I suggested you capture all the clicks. But can I consider everything as an action or not necessarily because if you consider everything as action, there will be too many features to come up, you know. So, if you are talking about action, think of the major actions they do, it is only the major actions, you know. For example, in a modal or in a MOOC systems or in a technology enhanced learning environment, what is the major actions they do, you might consider login, you know, login as the login into a system or system responses. That is maybe start of the action that is not necessarily important for us when you do the, you know, analysis in a sequential manner. But you consider login is important, just to know when it is started, the login is associated with the session ID. That also gives you know, are the session is continued or they logged in multiple times. And by login ID, you can construct features like how many times the login happened in a vehicle in a day or within hours and how long they spent, average time they spent on each login time. After logging in, what are the main actions they do in the environment? For example, if you use some LMS, they might be reading something, watching videos, right. So, reading, watching videos or, you know, answering questions. So, you can talk it as a quiz, you can mark it as a quiz, answering questions, a quiz or they might be in the discussion forum, you know, discussions, what else? Is there anything, something special with the system? Just mark it, some of the systems we saw, they are interacting with the simulator, some systems they are drag and dropping something. So, if you have any specific items, mark that as actions. Now for each action, in a read, if the student is reading, are they reading which page or which PDF or you know, what are the content they are reading? If you chunk the content into smaller concepts, it is easy to track that. Then if they are watching the video, what are they watching the video? If you want to minute data talk about, are they watching in, you know, single 1x speed or 1.5x speed and we, most of the people, you know, we do mostly, when you watch the online lectures, if it is very slow, we watch in 1.5x speed, you know, that works well. So, watching speed or the seeking the video from one place to place, you can add all this information as a contextual information, you know, this, I mean, as a contextual information. These are the major actions listed in the thing, then contextual information. The first step when you do the analysis is that you plot these actions frequency, you know, a number of times occur for each student in average, time they spend on each of these actions. And if there is some interesting aspect happens, go look into the contextual information, why this student is looking at it. A typical example is, for example, you have 100 students, you saw 50 students based on a pre-test and post-test code, they are low performance and 50 students based on pre-test code, they are high performance. Or maybe, say instead of 50, let me put it out, you know, you know, it is 40, there are some 35 and, you know, there are, there are in between, you know, some students who are not neither low nor neither high, but in between. So, we do not want to, you know, classify everyone just below 50, like 49 is a low or 51 is high, that makes no sense. So, we can give a median value or some gap. Maybe we can consider that as a medium performance or, you know, we can keep it as a buffer saying that I want to clear students who are really got less than 30, students who got above 70 is a low on high, you can make your own bifurcation, right. So, that is it. Now you have this data. First step you do is after collecting these features, you know, the features you are talking about login, read, watching, plot the graph, you know, and find out is there any difference, like in a frequency, in the average time just spent. If there is a difference, well and good, go and talk about it, why the difference, whether that difference can impact the students performance, that is hypothesis, test it out with the data, correlation or do prediction, that is good. But you see that for both students, all of them are reading time is same, you know, watching time is also same, kind of same, there is no significant difference, you know, you know, you can run a significant test on this data. Then you might want to go further, if both low and students are reading for five minutes on an average, how some students are able to get good score comes to not able to good score, maybe it lies in the what are they reading and how are they reading. Now we go look into the contextual information and see is there any difference in the context comes out, that is how we do the analysis. So, I was trying to explain this in a week two, so you might have had the idea of what is actions, what are the context, but I want to give you the picture why it is happening in the learning analytics. Next is, you know, we do analysis, so we do analysis, I was explaining that, then you do reporting. So, since there are a lot of data and you mean a lot of graphs, do you want to report everything to when you publish it, I want to report everything to a stakeholder. So, depending on what are you reporting and whom you are reporting, you have to, you know, make a graph and make a plot or make inference out of it and report it. So, make sure that you know what are reporting and who are reporting and what is their interest also, what you want to tell them, like you do not want to tell all the story to the everyone in the public. So, you might be interesting saying that, hey, we know all these other factors are already established in literature, we also found it same, but there is something new we found it which is not exist, you know, anywhere in literature, we want to talk about that, let us talk about that something new. So, that is a gap you have to identify and report it. Then after reporting, the whole idea is to, you know, to understand the learning process and improve the teaching learning strategy. So, this is kind of what is learning analytics definition I talked about in a week one lecture. If you remember what is learning analytics is to collect data, you know, measure, collect, report, analysis, in the context of students working on it and in order to improve the students learning, the primary goal is to improve students learning, it is not nothing about, you know, you create a model, you are able to predict the students performance, it is about how do you use that knowledge to improve your teaching learning strategies so that the students learn better. So, we talked about different learning environments like a face-to-face, online systems and, you know, also LMS kind of things, online systems includes LMS and tele both. So, in a face-to-face, most of the data collection is by self-reporting, this is a key actually very important. Most of the data is collected by self-reporting or human observation, there are only things you can do. Self-reporting is that you might run a survey, ask student, what are you feeling about or how many students are engaged in the class or you ask them some kind of questions, they answer something and based on that, you can identify whether they understood or not. And self-reporting can be form of, it can be known in also, but, you know, you can check a small piece of paper passed on the classroom and they will answer some set of questions. Or the second way of collecting data is human observation. You are a teacher, you are instructor, you are in the class and you are collecting students performance, students engagement, students activities or number of times the students interacted with the peer, all these things you are collecting. Human observation is one of the valid or well-accepted observation. The only key point is the researcher should not get biased to, you know, laboring each and every action of the students, the participants. So, in order to avoid bias, please create a inter-rater or inter-observable reliability. So, it is based on is it observable or later it is called inter-observable reliability. So, we use Kokenskappa. We talked about Kokenskappa also in our earlier weeks. So, we there we mentioned the kappa is used to compare two performance of two systems or to, you know, to measure the agreement between two observers. So, look at what is inter-rater reliability, how to use Kokenskappa if you are doing human observation. This is a very key part. The reason is you should not get biased to labelling the particular variable you are recording. Say for example, in the classroom I want to understand the students' engagement and you might be the observer standing in the classroom. Say there are, so there are five observers observing a class of, you know, 50. Each observer is observing 50 students, they are observing the round robin method. So, that is every five, two minutes I will observe student one, next two minutes I will observe student two, like at the end of 20th minute you observe all the 10 students and you come back to the against 20 second minute you will be observing the same student again and you are marking in a note saying that student one from time 10 to 10 to is engaged and he interacted with the PFA five minutes or all the coding. So, you have to first come up with the coding mechanism, what are all things you have to code. There are a lot of well, you know, this is called qualitative research, there are a lot of resources available in such articles. Go and look at the coding existing, you might find a new coding, find it out. Once you have a coding, you record, you know, students engagement all this thing and how do you compare whether you are doing it correctly, you are not biased. So, what you do, you compare your coding with your, your, you know, peers, others, other persons coding. So, in that when you compare inter-observer reliability, you both, both observer, both two or three or more observers has to observe the same student for a certain period of time. So, usually when we start, this never happens, this Kappa is not, may not be good, you know, Kappa may not be good, may not be really great. So, what we do, sorry, so Kappa might be low at the beginning. So, what you do is, first you will have a set of rubrics to mark this when a student moves and talks is kind of engagement, you have your own rubrics coding mechanism, you discuss with your peer and both understand the coding mechanism, both are observing. After observation, you do the Kappa calculation if the Kappa is low, if the Kappa is, you know, if the Kappa is less than, if the Kappa, if the Kappa, if the Kappa is, you know, if the Kappa is less than say 0.8, discuss with your peer who is other observer and talk where the mistakes made and why he thought that is a particular not engagement and you thought it is engagement, why you thought it is not engagement, why the particular, you know, other observer thought it is engaged, discuss and resolve the conflicts, redo the assignment, you know, again observation of a new student or same student first, new set of period of time, okay. Then you check again the Kappa score, make sure the Kappa score is more than 0.8, 0.8 or more than 0.8, that is a very, very key expert when you are doing the human observation, okay, that is a very key. Sometimes it is not that you will be observing the students in a real, you know, real live environment, you might be recorded the students facial expressions or students actions in the class and you will be looking at the video and recording it because you want more data. If you want to observe data in a real classroom, you may not able to observe all the 50 students data or you do not have much too many observers to observe the data. So, you will do not have real amount of data. So, what you might do is you might keep two or three cameras, observe, record the students actions and you might look at the student one in the camera and make down their, you know, just down their engagement. In those scenarios, again talk to your inter-observer, both watch the same amount of time, say 5 minutes or 10 minutes of video and mark down all the rubrics, make sure the Kappa is greater than 0.8. If not, again redo the assignment on a new student or a new time frame, not the same time from which you are already observed. So, make sure you make the Kappa is more than 0.8. So, even observation can happen in a real time also in the video, that is thing I want to discuss here. The second one is online systems, self-reporting. It is not just self-reporting of surveys, also the students is answering the questions, the performance, all these things can also consider. Other thing is click-stream data, you know, human observation data. You know, in general, you know, not in a, you know, LA, but in a, you know, IT or Syntagent Routing System kind of environment, what we consider is simple thing, you know, there are something called a profile. This is kind of a static information, you know, static in a sense, it is a profile of a student likes age, gender, the year of the study, the prayer knowledge, maybe if you are collecting the parents information, all these things kind of a profile information when they come to systems kind of a static. And click-stream data is other data, you observe all the interactions. And then you might, you know, the performance data is another data. So, stat profile, performance, click-stream data like that is the data we compare. By using the click-stream data and the performance by combining these two, you might be coming up with some dynamic data. What I mean dynamic data, you might be measuring the skill or you might be measuring the engagement, something, you might be measuring something for your research. That data is dynamic. Why I say dynamic? At the beginning, you might start with, you know, median value or middle value. Then based on student's click-stream and performance, you will change whether delta increases or decreases. So, a student is performing very well and he is interacting with all the, you know, artifacts in the system, you might increase the engagement more and more and more. They are not interacting with all the system, they are simply watching a video, they are not doing any performance, you might reduce some of the skills can be reduced. So, this is changed based on the student's performance on the dynamic. So, in general, the data is that the full data only. By using this dynamic data, what you actually provide a new feedback and new content, it is part of, you know, intelligent learning systems. So, that is the whole idea. So, that is the whole idea of click-stream data. Even in online systems, you do the human observation, that is why I said that if you might capture the videos or something and you might sit down and record the facial expressions and do it. For example, in online environment, you might be capturing students' facial expressions or data from eye trackers or something else. If you have facial expressions data, you may not be able to, you know, code it directly. You can ask the students to self-report about their own emotions, watching their own videos or you can sit, observers can observe the students' facial expressions and code their emotions, affect the students. So, in any case, make sure that you do the Kogan's Kappa and the Kappa score is more than 0.8. You should actually know biasing human observation. If you are not doing that, you know, the people do not consider that data is valid. That is very, very important. So, all this data collection, any environment, any data you collect, please get the participants' consent. And if you are getting a participant consent, please make sure with whom you can share the data, who has access to data, in the future what you will do, what are their rights and think about all these things when you do the participants' consent data. So, what we talked about is, you know, in environments, we might collect data from different instruments like a surveys, on the clickstream data, quizzes, you know, the test performance, all these things we talked about. And these are all raw data, right. And from this, we get a raw data like a response to questions or the clicks, they speak some utterances or the facial expressions, all their raw data. So, now think for a minute, how do we extract features, you know, from this raw data and why we are doing that? That is very, very key important. I can collect all the data, facial expressions, they are talking, you know, the utterances and the clicks, every clicks, responses, performance. Why we are doing that and why, how do you extract features from this log data? Please pass for a minute and write down your answer after writing it down, resume to continue. So, the feature extraction, why we are doing it depends on, you know, like why we are doing it, that is why you have to go and predict something and you have to extract the features to identify the student, compare the students with the other students to compare the baseline performance, everything. And what features to extract, it depends on the research goal, okay, that is why I mentioned in the first slide, you know, and the actions, you want to go, you know, context to information level or simply the actions. So, you come up with your own number of features, whether it is simply the major actions level or actions combined with the context to information. And this is where the domain knowledge is important. Not everyone will be extracting a good feature. The one who is working on the domain, say education domain will be able to extract a good feature. If you are creating your own system and you know the students interaction, the system means something. So, then you might able to tell from the experience, the experts knowledge you have in that particular system, the particular domain, you might say students clicking these three buttons or reading for five minutes and doing quiz or might be looked like the student is, you know, reading this or something. How do we know that? It is like, it is you expertise and if I talk to teacher in the classroom environment, if a student is coming for only 40% of time for attendance and he is not submitting three assignments at all, really pass the exam, the teacher will say, yeah, I will not pass the exam because I know it because based on experience, you can tell that. So, this is called, I call this a domain knowledge. That domain knowledge is the expertise knowledge you might possess in this domain. A same data collection, now machine learning method can be applied if you know the domain of no mechanical engineering, you might apply in that mechanical engineering problem. Or if you know some other domain, you can apply that. So, my focus, you know, with your expertise in the learner learning environments in the learning, please use that knowledge for the creating features. And this is important. The reason is the domain knowledge will be built up based on how many years you are working on it, how many features, how many systems you created, how many features you are extracting. The one way to start is read the research papers to understand how they extract features, what are the features extracted in a similar environment. Look at those features, list down all the features, then you might be able to come up with your own set of features or combine some features you created your own features. And frequency and time is the important. I said that, if there is a major actions, simply create a feature as a frequency of that action over a certain period. Say over last one hour, last few hours, last two days, last three days, last five days, over one session, over last five sessions, frequency of any major action. For example, if the major action is reading, what I do is number of times read, number of read in one session. I am not talking about what are they reading, which page they are reading, that is if needed you go further. But I am just saying, if you are reading, just go about how many times they are reading. So, same number of read in last five sessions or same in 40 minutes. So, you can extract multiple features from one major action. So, why do we expect multiple features? You can extract multiple features and find out which features are redundant by do correlation analysis or correlation with the dependent variable or feature extraction methods will help you to do that. So, that is a feature you might have or something. How do you might say, no, that does not make looking at the read session for last five years, you might say that is perfectly correct if you have a particular domain knowledge. You might say, no, no, this particular scenario, I do not think last five sessions of reading will not really helpful, you can remove it. That is why I say it is a domain knowledge is applicable. So, for same read action, I will create multiple features based on the time. Average read time in one session, you know, it is that. And you can talk about average read in last 20 minutes or average read in last five sessions. Or you can be specific that average read before taking quiz, something like that. So, here I am combining two actions. So, you can come up with a lot of features, that is why I am saying that go and look at the papers already which is talked about this and also the domain knowledge will help you to come up with it. You can say that if a student is reading more than three minutes before taking the quiz, then there might be, you might be doing better or if the student is, you know, reading many times, like spending five minutes and it is some other actions, then quiz this may not be good. So, you might come up with your own hypothesis. To test it, you extract these features from the raw data. So, there are two things, frequency and time. This is a very basic and you can all start with that. How do you improve this set of features? How to come up with the new features and combining multiple major actions. Or you can combine the major plus, you know, contextual information. It is all come up with your domain knowledge. That is, let us start from here. That is what I have said. So, and what we did also, we talked about basics in machine learning algorithms. We talked about pattern mining, process mining and clustering techniques, performance metrics, some of the regression, logistic engineers, NAI base, decision tree. We talked about all these things in this course. So, revisit them, what is pattern mining and process mining. Most of you might know the clustering techniques, you know, and regression. Process mining and pattern mining might be new to many of you. Check it again, it is very interesting. And check for associate mining also for the pattern mining. And importantly, this performance metric might be something new. The reason is not many people cared about what is a metric, what to look for. That is very, very important. And why this metric is needed is also I wanted to inform you guys so that you know how to compare two things. Then, how these algorithms are trained? I never talked about it. In a linear regression, I said that simply linear regression, I said that there is a magic app. There are a lot of dots. There is a one line and there is another line. I said which line to pick. I told which line to pick. The which line to pick is easy just because we know that. So, from the each dot, you have to find out the gap. Sorry, this is the one last function of this. And similarly, this last function for this is a different and you take the one which is. But how do I change? Well, it is just not a change. If I find this last function, should I change it to an upper direction? Should I move it down? I never talked about how it is trained. There is a reason for that. And I also never talked about hyperparameters in each algorithm because I face talk about training. I should be involving hyperparameters. At the beginning of the course also in the introduction, I clearly mentioned this course is for someone who is new to the machine learning and who do not require any mathematics or anything. So, I try completely avoid the calculus in this course. So, in order to avoid calculus, I did not do the talk about how it is trained and what are the hyperparameters in each algorithm. But I request all of you to go and if you are interested to more about that, go and watch videos by Professor Andrew Angini, each of this recordings and learn more about it or lot of very good resources are available in internet. If you are really interested about machine learning, there are very good books available in machine learning also. The aim or focus of this course is not that. It is not to teach every algorithm with training and hyperparameters because this course is not for only the one who is already well trained in the mathematics or will know the programming or something like that. This is for everyone. So, I just kept that part very, very low. If you are much motivated because of this is interesting, I want to know more, please read further. So, also in this course, we talked about multiple tools. ISAC, we had a video. I hope you would have done the assignment on ISAC. Use it. The ISAC tool is available. Just use it and check this code if it is useful. Use it and you can contact the developer and he will be happy and you can do that. Also, we have given small scripts to run a sequential pattern mining. It is not a tool as a ISAC or the other big tool. But we wrote a small script to extract the sequential pattern mining. If you all find some other tools available online, please go and use it. There is no harm. It might call as associate mining or something like that. So, understand what is pattern mining. It checks the frequent patterns in a fine-grained level. And what is a process mining? Process mining absorbs old process. So, process mining, we introduced program software that is available for free for academic usage, also for commercial that is one software for free. So, use that software too. Vekka is completely free for everyone. Use it. Orange is free for academic. So, use these tools. Do not stop only with these tools. If you are interested, go to Tableau, go to RapidMiner, explore more tools. I will talk about that in detail how you can expand your next steps. But these five tools we covered and make sure you learn everything well. And if the videos are not enough, go and check YouTube videos for Process Miner, Vekka and Orange. IZAT, that is simple, very simple to do. We gave a sample data, just upload it, you will see everything. We just have to click buttons. It is very simple. SPM, it is not, it may not be the complete tool. You have to run a script. If you are interested such other softwares, which is talks about associate mining. We do not find any good software for associate mining in an educational setup that can be shared freely with everyone. That is why we have to write our own scripts. Hope you guys enjoy that in tracking with the tools, because this course is about tools and this main focus is that you get to know some tools and a place on some data. So, now I want to ask something different. That is, what is the difference between 20th century and 21st century? When I said 20th century, imagine the time before 2000 or you can go up to 2003 or 4. What is the difference between 20th century teachers learners versus 21st century teachers learners? Think about it. Write down your answers. After writing it down, please let me to continue. In 20th century, we see teacher as a source of knowledge. I did not have access to most of the books. Only the library have limited books. The teacher has all the access to books, the library books. And you might have a notes from somewhere. You might have read a lot of books. And in the library also, he has a special access, you have all the access to books. And the teacher teaches everything. With examples, there is a textbook. He covers each and every syllabus. And he explains the problem and he goes off and asks us to solve the exercise solutions and problems. If you have doubts, he just might teach. And we see teacher as a guru, like he is a elixir of knowledge and he has everything. And he is the one person to go after for everything. He would ask for this. He is always there to answer my questions. And he guides, in fact, not just teaching, the good teacher, because beyond your subject, what to do, how to do, and a lot of discussions, a lot of things happened. This is 21st century teacher. Most of you might be, if you are a teacher, if you are a teacher and you are born in the 1980s, you might have seen 21st, 20th century teachers. But if you are a teacher now, you are continuing that, you are not a 20th century teacher. You are 21st century teacher. What is 21st century teacher? You are not a source of knowledge anymore. The whatever data you can access, whatever resource you have, the students are more than that. Students have access to lot of more data and lot of more resources. You are not a source of knowledge. Don't expect that you have to teach everything and you have to be there and you know everything, that is all gone. Student knows more than you in a particular subject and not just students, lot of industry experts. And everybody writes their own blog. It all changed after web 2.0 and the videos are coming in YouTube or other video, video serving platforms. So, you are a motivator to students like which course to do and the students for resources guides them. Look at this particular resources for this particular topic, but that does not work much. They are the rating agencies, you go to Quora, you check out which resources go to read about it. But you cannot pulling together saying that do this particular task, maybe you are guiding them what to do next if they have any issues. But you are not the only person of source of knowledge. That is what I am trying to say. So, you be a motivator and do that. So, I picked up that. I am not a 20th century teacher. I was a 28th century student. I was asking my teacher for everything. I go to library. Only the library has a four books. I read notes, I read it. But now I am a 21st century teacher. I did not teach everything about all these videos. What I try to do is motivate you on, make an interest on, this is linear regression, this is logistic regression. This is neighbor. This can be a player. This is decision tree. I did not talk about anything about decision tree in detail. This is a decision tree how to do it. There are two important parameters. With that knowledge, what is entropy and information gain in decision tree, you can go to any resource. Now, you can pick it up easily. So, now you know what is these two and you can watch a new video or read a paper or read a book. You might get more information and more interest. All depends on your interest. So, all my videos I always talk about, all these ML algorithms, please go to Professor Andrew NG's video if you are interested more about that or any videos. I recommend a couple of papers in the videos for you to read because that is how you know how this data, how this model is applied in a real data, how people have used it. And for the tools, if it is ProM or VECA or Orange or Rapid Minute W, go to YouTube, there are plenty of videos explaining how to apply each and every algorithm, like what data to apply, which buttons to click, everything is there. There is no need for us to teach everything which is already existing. We do not want to reinvent the wheel. So, the idea here is motivate you, there is a tool, this can be used, go and learn it, that is the whole idea. And this course is trying to pull out what data to collect, what are the learning resources, what are the different environments and how you can apply it. This pulling these three things, the data collection, the environment and the ML together and how to infer, that is what we try to do in this course, not to teach every ML or the tools or in algorithms. So, if I did go next, we will talk about that next. Thank you.