 Welcome, everyone, for this session on monitoring Docker containers and Dockerized applications. I'm Meenakshi. I lead the Cloud and Network Solutions team in Bangalore, India, and with me joining my colleagues Anand, Rahul, and Satya, engineers in the Cloud and Network Solutions team. So basically, what we want to address today is a short introduction about containers. We have been listening about containers all through the day. So we'll keep it short. And mainly, we'll look upon what are the challenges in monitoring containers, then what is the approach that we have come up with monitoring containers, and a short design and a short demo on monitoring containers. As we all know, containers are becoming increasingly popular for, be it a development environment or a testing environment or production. Basically, containers virtualize the way as just like the way the hypervisors virtualizes the hardware. Containers enable any boiler to be encapsulated as a lightweight, portable, self-contained environment. And that can be manipulated using the standard operations that we are familiar with. It basically wraps up a piece of software in a complete file system. And then with whatever libraries and kernel, piece of kernel that is required, and runs it top of an operating system. Now, Docker and LXCs are some of the most popular implementations of containers today. And it can be run on any Linux server as a VM or on a VM or a physical host or directly on your OpenStack. So some of the key aspects to note as we move on is the ability for them to be moved around between missions without any modifications and ability of the containers to work together. Without short introduction on the containers, we will just start looking at some of the challenges that we face today in monitoring containers. As these become widely popular, how do you build a scalable, intelligent system for monitoring these containers? That I will hand it over to Anant to go over the challenges of monitoring these containers. Thank you. Thank you, Mach. OK, so traditionally, when we say monitoring, a lot of things comes into mind, monitoring application, monitoring your host itself, network monitoring. There's so many points where you can actually monitor. And we look at some of the challenges that you face when you run applications on containers. What are the different areas where you'll have to specifically concentrate? So one of them is you traditionally monitor network, memory, disk, and CPU utilization. Of course, they will have furthermore subcategories when you drill down. But these are the broad classifications that you would want to monitor. And at different levels, you'll have to monitor. One is your application itself and on the container in which it's running on. And the host on which those containers are running on. So you could also have clusters of applications you want to monitor. So they are at different levels you'll have to monitor. And there are a lot of tools that you have open source, proprietary, there are so many available. And they may follow their own notations, their own representation of the statistical data that are collected. So if you are going to stick to a particular tool, then you're confining to the capabilities of that tool. And so your decision making is going to be dependent on the capability of that particular tool. And other point is the container itself can get overloaded if you keep constantly bombarding with the tool. So your performance may not be accurate. So that is one more challenge. And at the end of all that, you collect a lot of statistics, raw data from different containers, host, and network, clusters, all that. But what you do with that data, that's what is most important. At the end of the day, you collect all the data and then see what does it mean to the company in terms of cost or in terms of performance of the application itself. There are so many inferences that you can draw with just the raw data. So a lot of work needs to be actually put in. And that is a very hard problem to solve. It's not a single equation which gives you all the results. So that is one of the biggest challenges here. Now, monitoring an application itself that is running in container poses a different set of problem. So container will actually abstract everything. And then you have a confined place where you run the application. That means logs related to that particular application is confined to that container. Then you have distributed deployment of that application. Then how do you identify how the control flew from one container to the other? How did your network communication between those containers are you able to measure that? Is it choked at some point? Or are there multiple containers? Probably there's a deadlock situation. One is waiting for the other to release a resource. So these are all different kind of challenges that come up when you run applications only on containers. And another one is when you're talking about user experience, a lot depends on various factors, not just about your system's performance. For example, if you want to measure user experience in terms of response time, a lot depends on your internet connection speed. It's not just about the response time at the server end. So the multiple challenges that are posed because of these varied aspects. So what we are trying to address here is try to solve some of those challenges with the approach that we are suggesting and the design that we're suggesting. So your apps could be run on a container that can be self-contained in a host itself. So your cluster of applications, they all could be on the same host itself. Or they could be distributed between different hosts. So how do we go about dealing with those? Let's see. So you need to be able to identify the different levels at which you want to monitor. One is definitely your application level and then your cluster level. Now what kind of monitoring we'll get to that in the next slide. And at host level, you see some statistics, available resources, and stuff. What it means in terms of related containers, that's the key. One container might say, OK, I have one TV available. The other container might also say one TV available. For all you know, that's a shared one TV. So how do we assess those things? That becomes the key. And so in order to be able to do a good job of the monitoring, so you'll have to monitor with different strategies. One is proactively monitor. The other one is reactively monitor. And then you monitor adaptively. We'll get to this. Let me just go over this first. So what do we mean by these three terms? Proactively is when you try to monitor and get status and identify what are all the possible failure points. One example could be you could run out of disk space. So you monitor your disk utility, maybe set a threshold. When you go about that, raise alerts to prevent disk running out of disk. So that could be a proactive monitoring. The other one is reactive monitoring. Maybe the application itself is throwing some error, but you don't realize that at the top level. So maybe you can send alerts back so that you can be aware of which particular node or application is throwing error. And you can stream logs to see which is the exact error log, what could have caused the problem. And adaptive monitoring is when you can identify containers or applications that spun up automatically without having to query individually. You find out this is the container that you want to monitor. And also, you can use multiple tools, not just a single monitoring tool. You use a combination of tools, pick the good ones from all the tools, and create a common model out of it. Memory is memory. Whatever is it, mem underscore cache or cache underscore mem, still memory. So you abstract it and put it in a model, and that gives more sense. And you can do a lot more things on that data if you can actually get the combined power of all the open source tools that you have. So now let's get back to what is it that we want to monitor. So we already talked about the different levels that we want to monitor here. So let's see at each level, some of the parameters that you can monitor are what is the total CPU usage and what per CPU core what is the usage and overall system what is it and how much is your host using and how much of it is being consumed by your application. Or comparatively between containers, what does it mean and what is the load average. And again, for memory, you can see what are the page faults and what is the cache memory that is available, what is the kernel memory, what is the user memory. Similarly, for disk as well, you can monitor synchronously and asynchronous writes. What are the different response times for it? And in terms of what are the faults, if there is any read write fault, so you could monitor those. And network again, as we said, network could be monitored outside the host, inside the host. And again, between containers also, what is the bandwidth? Between containers, what kind of traffic goes between containers? You could draw inferences like which nodes of your, let's say three tire application is heavily loaded. Probably there's a lot of communication between app and DB and not much between web and app. So those kind of inferences that we can draw. So like I said, intelligently correlating all these information and making more sense out of it and suggesting business decisions or capacity planning, a lot of that can be achieved by doing this. And let's see. OK, so we'll actually go over a detailed design. We'll have Rahul here explain about the design that we are proposing. And how each component is able to help us achieve some of the goals that we set. And how did we overcome some of those challenges that we talked about? Yeah, hi. So basically, when we were designing the whole solution that we are proposing, so we had certain objectives. We want to share that. So the first and foremost was that we did not want to overload the Docker daemon itself. All the monitoring calls, if you're doing something, getting the info from the Docker daemon. And as you scale, this could be a bottleneck. We all understand that. So different. And then we want to put a different approach for monitoring at different levels. Apart from that, the other key thing that we wanted to build was a completely modular model which would be a driver-based approach for all possible monitoring components that are there today. And also add on to this would be like you could run multiple such monitoring agents and gather info. So a single agent, a single tool could give you, would be limited. So if you use more than one tool and which completes your model, you get all those data. And from that particular host regarding the host application as well as the Docker container itself. So apart from this, we all know that to get some sense out of the Docker systems, you need to cluster them and you need stuff like Kubernetes or Swarm. There could be specific info which you want to take out from them. So this was one of them. So let's deal with the high-level component design. So if you see, we have an API which exposes all that we do. And that is a REST-based API. So you could use a REST client or a UI or CLI to access it. On the right side, you see all the hosts. So we have one agent running per host which has all your containers. It gathers information and would put it on the queue in a synchronous model. And the orange section that you see is a logical grouping of components here which would constitute to your monitoring controller. All the hosts would put the information about the application, the containers, and the host onto the queue which come to the engine. The engine does the aggregation. And after it's done the processing on that data, it stores that into a database, preferably columnar database in our case as we're using. And then comes the IQ module. So as you all know, general logs, if you just store all the logs, they are dead data. To make sense, we're using this IQ module so that you can actually derive sense out of what your log gives you. So you take a sample of the data and stuff like on what days your network is more choked and how much memory load, how is your network behaving when the load on your memory goes up or CPU goes up. So you can generate as many things that you want to. So it's all unstructured data, after all. Then we go to the functions. So all the functions that would be done by this it would collect all the data and stats and logs from your app, container and host. And then the engine would model and process that data, store it somewhere, basically sanitize it and store it at a single place. And then our IQ module can analyze it. And both can send you. The stored data can give you what are present results. And the IQ module after analysis can give you predictions as well as suggestions. Going on to the design of the agent itself, so we have a driver that would monitor the host, a driver that could monitor your apps, and a driver that could get log and stats from your container itself. And there could be multiple drivers which are doing that. And all the agent would do is that it would check whether it is getting all the params that are required. So basically your sanity of your model inside the engine does not break. Once it is done that, it will package all that data and dump it to your queue. And the queue would take it to the engine, wherever the engine is riding. So more about the agent. It is basically one agent per host. And agent monitors the host, containers, and applications, as we just saw. Agent sends and receives to the engine in a sync model. So basically all of your calls, which are happening, would not be a blocking call. Your engine tells that go ahead, monitor this particular host. It takes its time, fetches all the data, packages it, and in an asynchronous fashion, sends it back to your engine. Then you have a driver-based log and stat collection. Driver-based tool, you can use the tool of your choice. So basically the agent is not restricting you that, hey, use this particular tool only. So there are so many tools today in the market, using which you could monitor the Docker container, or the host, or the application. So you can choose the monitoring application of your choice. And then this would just plug it into the system. So more than one driver can run in parallel and takes care of the sanity of the data to conform to the data model of the engine. The monitoring controller, as we saw over there, was a logical grouping of the components. The components need not reside on a single host itself or a single container. All of those components could be individual containers as well. So this is basically a logical grouping. And there is a REST-based API that connects to CLI UI and REST Client. Driver-based storage module that uses any columnar database. So that is, again, your choice. There is an IQ module that we talk. We'll see what more it does in the following slide. So apart from that, the engine does aggregation of stats and logs from different Docker hosts, integration with identity providers like Keystone. So basically that would provide multi-tenancy and segregation of the logs on basis of that. Communication from the agents via asynchronous queues, grouping and processing of data based upon your use case. The IQ module, basically the IQ module is, as I said, if you just store your logs, they would mean nothing. So they're all unstructured data. And I know I can see so many smart minds here. And when I speak of unstructured data, you understand that it opens a whole new domain. You can find what could relate to what and build a lot of meaningful stuff out of it. So that's what you can derive meaningful inferences out of this data, and which would actually provide value to the user. Then for doing this, we would use analytic tools like this library called Pandas. I'm sure you guys know it, SciPy. This is under implementation. We haven't actually implemented this. And we're looking for different modules which would help us do that. Apart from that, this module could help you. The IQ module could help you to detect the error before it actually happens. It could tell you a usage and load pattern over a long time. Even like, you know, Wednesdays, your data center or this particular node gets hit by so much. Basically, it would tell you the history of how your container has been used by going through all the logs. And that day, you can understand that I can put a load balance and maybe multiple containers for that. The suggestions, basically, the output of this module would be suggestions to the user. And maybe if we go a little further, we could develop a lot of automated systems which could take care of that. Now we have the demo, a short fraction of what we just saw in the design has been implemented. And Anant will just run through the demo for that. So for just like a POC, we wanted to implement some part of the design that we talked about. So we ran a Ubuntu VM on a laptop, and we installed some of the related packages. A lot of those components that we saw earlier are more an abstraction of the type of tool it can be. So this is not really tied to any particular tool as such. So we've been trying to make it as generic as possible and as pluggable and driver-based at every level possible. One is the agent tool itself, which collects statistics, is a driver-based one. And even the database that you plug into is driver-based. So for demo sake, we've used in FluxDB. And again, you can get graphs out of your time series data that gets stored in a columnar database. So for demo, we've used a Grafana as a tool. And driver, we've actually written a wrapper over Docker's own library. That's Dockerpy. It's a Python plug-in, Python library available from Docker, which connects to your Docker daemon and gives you some statistics. And we've written a wrapper over it. So basically, the Docker daemon's stat command is a blocking command. It's like your VM stat or top kind of command, which shows you running statistics on screen. So if anybody of you I'm sure you would have tried, if you write an API for the stat command, it gets hung there. So what we found actually in many of the documentation that is not available is there is a next method. And stat.next will actually take you to the next container and it'll keep running. So we found it the hard way, so I'm just sharing so that if you guys don't know, you can pick it up. All right. So before we see, I'll show you some part of the plugability that I talked about. So this is the agent that we are running on this machine, which is collecting statistics. And this is a place where you can configure a driver. This is very similar to the NOAA's compute driver kind of architecture. Of course, this means that there is a wrapper available from the agent to handle this particular tool. If, for example, there are other tools available, you could write your own implementation, connect to the C groups, instead of connecting to the Docker Pi or connecting to the Docker daemon, or you could use some other tools that are available like CAdvisor or there's Ruxet. There are so many tools available. So you could write your own plug-in. Oh, I'm so sorry. So also for a log collection and log streaming, you could write your own plug-in, or you could use any of your available tools like log streamer and then have a log collector in your server, which can collect that, and then load it into your database. So and some of the other configurations that you see here are related to the agent itself. And if you want to monitor just the agent itself, that is throwing an error, you could enable logs and stuff. OK. So let's go. So we were actually running one or two containers already, and the agent is already running in this machine. So we'll see what are the containers that are running and how we are collecting statistics, and what are the kind of statistics that we are collecting. We'll take a look at some of those. OK. So we see that Dockerpia is showing already three containers are running. And we've used influxdb, as I said. So let's go and log into influxdb and see what stats are being collected, how we have logically grouped some of those. So as I said, time series data that is already being logged for as a broad classification disk memory and CPU. Let's go and spin up a new container now and see that automatically that is detected and stats monitoring and stats entry gets pulled into the database. So we've created a new container called Test. So we see that appearing here. OK. So the rate at which you want to be able to monitor the stats at which you want to pull, all those can be configured. And influxdb also gives a nice UI representation of the data. So we can go and take a look at it. But it's not really tied to influxdb. Any columnar database would make sense. Since they are all time-based statistics that get updated, a columnar database really helps in that case. So we can see the different time series that are there. So the grouping here can be based on, I mean, there is no limit to their imagination. I've just grouped it as memory disk and CPU. You can group much more. You can even go further down. And depending on your, you can create your own interpretations of the data. And even business suggestions could come in from setting filters. There's so many other things that we can do here. So we'll look at what are the parameters that each one of these time series provides us. So for now, the parameters that you see here are limited to those that are provided by the DockerPy library. But as I said, the interpretation could vary between the tools that you use. So the idea is to model them completely and make it generic so that any tool will give you the similar kind of statistics. Let's move forward. OK. All right, so we can see some of the memory-related statistics here. Or OK, yeah. So what is the cache memory? And what is your system memory? And what is the response time when it comes to a network statistic? Those things are all logged here. The column that you see here is what is returned by the DockerPy library. But as I said, the model that we are building is going to be more generic. And irrespective of whichever tool, we classify them. That's where the intelligence of the engine comes in. And I just wanted to show you an example of how you can integrate it here. So let me stop it here and show you actually on the machine that is running. So this is a Grafana tool that was integrated with the InfluxDB instance that was running on this VM. I'll show you where you can go and configure the database. This is the record for memcache. Based on the time series-based input, Grafana generates this kind of tools for you. So the empty spaces show that the VM was actually powered off at that time. And you can infer so many things from this. If you can look at your response time here, then you can make out when do you see the spikes, on which date. You would probably see by a month end, everybody would connect to the employee cell service to see salary. So that is a point where that server could be choked. Or when Monday mornings you come in, there would be a lot of emails. So you mail servers in the target there, right? So okay, let's look at some of the configurations that we have here. So I've configured my InfluxDB database settings here. So this is my table name, or rather the database name. So there's no concept of table here. It's more of a series. And the InfluxDB is also running on the same machine. This is a place you can configure your credentials and stuff. And there are a lot of tools available to design your own dashboard. What are the kind of output that you want to see? What are the kind of graphs that you want to see? There are a lot of builders that are available based on the data that is available. You can go and pick and select different types of graphs. You can have different kinds of histograms generated for the same kind of data. So let's see what else can we configure here. So here you can edit. And you can give a different kind of query. Currently, I'm just showing you a cache data from the series called memory. You can drop down, and you see all the parameters that we're collecting here. And you can look at some of those. And you can check max usage or inactive. Or you can perform more operations on it. What is the mean response time? What is your high response time? What is your max? So there are a lot of functions that you can perform here to draw inferences. And you can have API calls do these, and then automate some of the operations that you want to do. For example, based on your load on a particular server, if you want to replicate it and have a load balancer replicated on another container, you could automate all that with an API where your engine goes and does that, does this analytics. And so those are some of the business decisions that you can make. And of course, sorry. This one was tried on a single container. This particular demo was run on a single container. Understanding the communications between containers and what it means overall to the application in an overall perspective, that is a totally different thing that intelligence would have to come from the engine side. This is only the agent code that I'm showing you. It's only the agent's output that I'm showing you. So as I said, all the raw data that gets collected from individual agents have to be put together, correlated, and then you try to make sense. As I said, what it means from one container starts for the other container. So that input has to come from the engine side. We don't have the engine's implementation for the demo. That's something we're still working on. But this is only from the agent's perspective. And you can set refresh time here. You can make it auto-refresh, or you can set based on time, depending on the type of graph that you can set all this. As I said, again, this is only an example. You could use Grafana, or you could use so many other graphing tools that are available, and that gets plugged into your database. The choice is completely based on what data that it gives. In terms of graph, what kind of graphs that we can generate, and how easy it is to use. So that's about it from the demo side. We can take any questions. Yes. For the demo sake, we did everything in standalone. In the production, actually, we are not confining to influxDB at all. We're just suggesting any storage component which can store time series information. It could be any column in the database. And of course, you can run it in a clustering mode. Even if you run it in a clustering mode, the engine is only going to a particular IP endpoint that it connects to. So even if the data has a replication part, your engine would still connect to only one endpoint to fetch the data. So the plugability is one of the main features of the design that we propose, rather than confining to any particular tool. The advantage is even if tomorrow any new tool comes, you could still plug it in here. And very easy to, you can incorporate that in the design. And one more thing I would like to add here is, this is something that we are proposing. I'm sure I've addressed some of those people who are just beginning to understand containers and Docker. I'm sure you had something to learn. And I'm sure there are a lot of experts here from whom I would like to also learn. So I would definitely like to probably discuss offline. If you guys can suggest any improvements to our design, to make it even robust, we would be more than happy to learn from you guys and also improve this design. And also, we're also planning to open it up. And if anybody wants to contribute to this, please ping us back, we'll share the details. Maybe we can work together. As I said, the engine and the prediction system is a big science in itself. You could take it to the level of artificial intelligence or you could just do it very simple as well. So it's a big science and that involves a lot of great minds. So if you guys can jump in, awesome. Absolutely, yes, yes. I'll answer your question. Definitely, we are going to put a blueprint on it. Definitely, we are planning to put a blueprint on it. Just let me just repeat his questions for everybody. This question was that, are we planning to integrate it with OpenStack? Or how do we do it? Is it already done? So Satya, I will explain more about that. So definitely, we are not confined to any type of driver and also any type of UI. So anyone can use our UI, our middleware and he can put his driver and UI and use it. Even it will be great. We'll be putting a blueprint on OpenStack and if it is accepted, definitely we'll like to share it with OpenStack. Yeah, sure. So the tool itself, if you see, does not restrict you to use anywhere. It could be on a standalone Docker setup or in OpenStack as well. So we've tried to keep it as modular as possible and pluggable and even we've given a northbound API. So if Cilometer can fetch that data and use it, nothing like it. And I see one of the things is when you more look at it from an application perspective and if you have a multi-tenant application, that currently if you deploy applications on OpenStack and the applications login and authentication itself may not be currently directly tied to the multi-tenancy of OpenStack itself. It could be different. So what we are also trying to achieve here is have our multi-tenancy plugged into OpenStack itself and you log into OpenStack and provision a container. So we can show statistics and utilization confined to that tenant only. And what that means to that tenant and with respect to the quota, is he going to exceed the quota or is he going to run issues? Probably the host still has resources but the tenant doesn't have quota. A lot of things can be integrated and a lot of scope for a pluggability there. I don't do this. There currently there is a plan to add a keystone piece of it where you can use multi-tenancy on the monitoring tool and also in the Docker so that you'll have a complete solution ready with multi-tenancy and multi-user setup. You would like to contribute? Sure, sure. We would share details. Currently it is in a private repository. We could give access to it and we would love to have more discussions and so we could work together. Thank you for that. So those are our Twitter handles. Maybe we can just get in touch just after the conversation. Sure. We could discuss offline more on this and what plans we have. Thank you for showing interest. Oh yeah, one more question. I've not actually worked with Monaska. So if anyone wants to. Yeah, definitely. Monaska is somewhat similar here but Monaska is like, it is a addition to this. Currently we are only, we have done for single Docker containers but Monaska is like more towards open stack side of it and we have not done it yet. Definitely we'll write a blueprint on Monaska only and we'll contribute to that. Thank you everyone. Thank you. Yeah, please bring us back at our Twitter handles in case of any further questions or discussions. Thank you so much for the interest. Or even now. Yeah, we could still discuss now if anyone has any more questions.