 We are the event logging team of the FRG group. So let's start with the outline. This is the outline of our tourist presentation. So move quickly, let's go through it. And let's start with the introduction. So we all know what is event logging. Event logging is basically capturing all the activities of a user and store them so that they can be usually further. So what is the basic motivation behind using the event logging? So we are using the event logging so that we can capture the event so that they can be used by the recommendation system to give the recommendations. And other reason is to provide the analytics. Using these events, we will be able to provide the analytics to the user that will assist them in the decision-making. Using that, they can make a better decision whether they want to join this community or not by seeing the top communities, top trending articles, and popularity of a community over time. So what are the technology that we have used? We have used the Django Middleware, ELK, which stands for Elasticsearch, LogStash, and Kibana, C3JS, and Django REST framework. So what is Django Middleware? In this Django app, the Django Middleware basically acts like a hook. So we are capturing the event using our customized Django Middleware. We will see this thing in detail in the coming slides. Then there's C3JS. C3JS is basically used for plotting the graphs. So this is basically used for the visualization. And then there's the Django REST framework, which is used for building the API which we have built. And then the most important thing of our project is ELK. So as we know that ELK is acronym for Elasticsearch, LogStash, and Kibana. Elasticsearch is a search and storing engine, using which we can quickly get the data on simple requests. LogStash basically acts as a pipeline. It can take the input simultaneously from multiple sources, process, or apply the filters on the inputs, and then gives the output. And Kibana. Kibana is used for visualization for the admin. So let's see the overall workflow of the R module. So basically, our project can be divided into three parts. First part deals with capturing the events, processing the log structure, and then storing them in the Elasticsearch. The second is to develop an API using which we can query the Elasticsearch and get the logs from there. And the third is the analytics part. In analytics part, we provide the visualization to the user. So first of all, there's the collaboration system main web app. We are capturing the events using the middleware. Then the event logging module process those events, and develops or builds the log structure. After building the log structure, these logs are passed through the Elasticsearch via the LogStash pipeline. There the logs are stored. The Kibana is used to provide the visualization to the admin. And this is the first part. You can say this is the first part. In the second part, we develop an API. In API, you call the API, and API uses the built-in API, and queries the Elasticsearch, and gets the data from there, process it, and provide it to the user, or the client. Basically, we have the client's recommendation system, or any other user can use that API. Then the visualization part that is not shown here, that is the third part. So how we are capturing the events. The most important thing of our module is that our module is totally separate from the collaborative system. There's just a middleware. The middleware acts as a hook. Using that, we are capturing all the HTTP requests, and we process those requests. So every request which comes to the Django, first pass to the Django middleware. So these are the default middleware, common middleware, sessionware, CRS middleware. These middlewares, by default, are provided by the Django. So the module, or the middleware, which we have developed, is our customized module. That is the event log middleware. We have put that middleware below all these middlewares. So all the requests which comes to the Django servers, passes through our event log middleware, our event log middleware captures the request, and then captures the event. So the capturing of the event, I have told you, that is done by the middleware. So what is the benefit of using this middleware? So using this middleware, our system is totally separate, and we don't have to change in the main code. And it can be used in any Django app. So our module can be used in any other Django app. So this is how the logs are captured. So now the next step is creating the logs. That will be explained by Kartik. Good afternoon, everyone. Now I will explain to you how we are actually creating the logs. So first, as the request comes, as the request comes, it passes through the event log middleware, and we capture all the information that comes with the request. It contains the request and some URL parameters that are passed along with the request. And all this is packed in a Python dictionary. And that Python dictionary is sent to the event log module by the send request data function. Now the event log module temporarily stores all these Python dictionary objects in a bucket, which is basically a Python list and is used in a FIFO manner. That is first in, first out. And then there is a process request function, which runs in a separate thread and continuously checks if any Python object is present for further processing or not. And if one is available, it fetches it from the front of the list and then deletes it from the list. And then the process request function passes this request object to the event name mapper function, which matches it against the registered patterns using the request URL and the request method present in the dictionary object. Now if it matches against the patterns that are registered, it returns the name for that event, for that pattern to the process request function. And the process request function, if it sees that the event name is not none, it passes that event name along with the Python object to the create log function. The create log function knows how to create the actual logs. It adds some common fields and event specific fields to the Python dictionary and passes it back to the process request function. Finally, the process request function passes it to the store log function, which knows how to store the logs. It can be stored in ELK, as shown in this case, or it can be stored in files, or it can be further enhanced for future. So these are some of the important points regarding how the logs are created. First of all, the size of the bucket, which temporarily holds the Python objects for logging, is not fixed. The other thing is the process request function, which actually processes the logs, runs in a separate demo thread so that it does not affect the performance of the main system in a major way. The other thing is for now we can store the logs in file or in elastic search. But here we have shown only with the elastic search, because storing the logs in a file is not useful. And the other thing is the logs that are passed to the ELK are stored in the JSON format. So here is a sample log structure that we have captured. This log structure is generated for the community view event. When anyone views some community, this event is generated. And the name of the event, as you can see in the event name field, is event.community.view. This is the format that we have chosen to name our events. The format that we follow is the event, then some resource like community or article or anything. And then after finally comes one of the CRUD events, like view or edit or something else. And then you can see there are some common fields that are common to all the events. For example, IP address, path info, time stamp host, et cetera. And there are some fields which are specific to the given event. For this example, you can see that if someone has viewed some community, the community ID is stored in the nested structure called event. So now, Rahul, we'll explain you how the logs are stored in the plastic search. So good afternoon, everybody. Now, storing the logs in elastic search. We have the logs scattered by the event log module that will be stored in the elastic search. There are some of the ways this event log module send the HTTP put request to the log status. In the log status, these logs are filtered by the indexes. And these process logs are sent to the elastic search. The elastic search produce the index if not created. And it stores the log in it. The index is actually a table in a SQL database like thing. And these logs, which are stored here, can be fetched from the elastic search by using the event log API. We have written the event log rest API through which logs can be fetched out. And we have a Qwana, which visualize the different logs in the superadmin. So we have important points here. Elastic search log status and Qwana, which runs on different ports in the Docker. The name of the index is logs.logs, which can be changed in the settings. The index is actually the table in a SQL database. The Qwana right now is available only for super admin users. And elastic search is a DSL queries like build over a partial look-in. Thank you. Grafton, everyone. I am Vishwath. And OK, so our event log system provides a risk-based API, which is used to extract the data from the elastic search. It is basically a wrapper over the already existing search API of the elastic search. Our API is secured using the token-based authentication. And the data, which will be obtained from this API, will be used by the recommendation system to train their module and also for the user analytics. So these are some of the features of the event log risk API. So first of all, pagination. So whenever the calls are made to this API, a large number of results are written, which creates a load on the system. So to avoid this, the results are paginated. And then they can be handled more effectively. Then there is sorting. Then filters can be applied on the searches. And the data can be grouped according to some spatial category. So one of the main aims of the event logging system was to provide the user analytics to show the meaningful analytics to the users. So we have created three dashboards, the article dashboard, the community view dash, community dashboard, and the user dashboard. So in the article dashboard, there is the article view timeline. In the community dashboard, there is a community view timeline, the most viewed articles in the community, and the trending articles of the community. For the user personalized dashboard, we have the recent activities done by the user. The user's is most viewed article. And the state of the article, in current state of the article, which the user has created. So these are some of the snapshots. So a biograph is used to show the most viewed articles, a pie chart to show the current status of the articles of the user, and a view timeline to identify the activity of the community. So the ELK, which we have used, runs on a separate docker. The elastic search, logstation, Kibana runs on the different containers. Here is the source link for the GitHub repo. The Selenium testing has been done to verify that the logs have stored properly in the elastic search. They are the log structure, which has been formed, is accurate, and that the REST API returns the accurate data. This testing was done by the notification team of the fundamental research group. So the problem solved, since our project uses the Django middleware, we don't have to change anything in the existing code. Also, adding of the new events is very easy. With this, the API provides the data, which can be used by the other systems. And some basic analytics are added to show the users. So future scope, since we are using the middleware, the browser events aren't captured much. Browser events such as window resize or the scroll events aren't captured. But they do provide some of the important behavior results. So they can be captured. So also the video events, such as the time at which the user skipped, the portion at which the user skipped, the speed at which the user was viewing. Those two can be logged. Some more in-depth analysis of this can also be added. And the storage of the logs in the cloud. These are the references. So have you also implemented a recommender system? No. No, right? No, no, that's fine. So from what I understand, the output of your work is being used by the recommender system team. Yes. Is that the case? Yes, sir. So how come they were able to do both their works? OK, but only after they completed, they might have. OK, OK. So what are the other benefits of event logins? Like are there any other benefits of event logging other than a recommender system, of course? It can be used for the security, continuous requests, in a security manner. So can it be also used for debugging? Means like in case if there is any problem, you can look at the, explore the. If the system runs into some type of error, then we can see the logs generated. And means what activities have been done that led to this error or something like that. That also can be viewed from the logs. So in just one more question, it's like why did you choose ELK, means like are there any other alternatives or you just asked to implement using this? We can use MongoDB. But the problem with the MongoDB is that it doesn't provide the restful API. So we have to create our own API for these things like pagination sorting and a lot of aggregation and other stuff. So Elk, Elastics has provided some search API. So we just use that API. And then we can send using the HTTP. We can send the request for storing the logs using the HTTP because MongoDB I don't think provides this kind of HTTP. So you've got everything in one package. Yeah, so that's the main reason. How many rest APIs you have written? So we have five endpoints of an API. And then there are get parameters that can be added for sorting and custom fields and other things. So I had this very general question. So you made an event logging system which can be used for, as you said, debugging and lots of applications of it. So do you know of any system like this that is already in place? There might be, right? Like, IDX is logging. AddX system is logging. No, no, as in something that anybody can plug, like your system is a plug and play system, right? Somebody can take it and implement it on the back end as a middleware and go ahead with it. So something like that. So you can read up on it. That's it. OK, thank you. Great.