 Anyway Okay in this session we'll talk about monitoring of Kubernetes with OpenStack monaska So I'm Christoph. I'm working for Jitsu and I'm Kubernetes contributor and I will talk about the Kubernetes part in the session My name is Vitek Vedic I also work for Fujitsu and my focus is on a monaska project and We'll talk about monaska in this part of the session Yes, so what is monaska? Who can tell? No one don't be shy So These two guys know for sure because they have to care about their OpenStack they want to efficiently monitor their cloud system and monaska is the OpenStack monitoring and logging service So it can store retrieve Metrics you can It supports a thresholding and notifications You can also store logs retrieve logs and create alarms on logs monaska is High-performance scalable and fault-tolerance system. Here you can see the overview of the architecture of monaska So you can see it is Microservice-based architecture. You have all these small components as microservices They can always separately scaled So it's a good design for the for the cloud application at the top. You can see the apis In the middle the central component the Kafka queue, which is Apache storm And below all these small components are actually working can work independently of each other. You can also add some more components like for example lock metrics or Which is not depicted here monaska analytics or monaska transformer Lock metrics is the the component we added It is there for generating the metrics Based on the entries in the logs so you can combine these two parts logging and metrics and Combine in alerting the information from two sources or kind of complex event processing Yeah, okay Let me give you a very quick recap on kubernetes Kubernetes is a system to run and manage containerized applications in a distributed environment And basically what it is doing it's abstracting For you the complexity of scheduling containers On nodes and routing the traffic between containers And even in in case of node failures or horizontal scalings Our Kubernetes is quite flexible. You can run it on a 9 euro PC or on a full data center in this case It's open stack and we want to make sure that We utilize all the available services Um, I also want to have a quick look on the architecture. We will need it later for the integration So kubernetes cluster consists of a master node. This master node has an API Component and some controller components And then we have many worker nodes each worker node as a kubelet It's an agent of the master received commands from the master And we have a proxy that routes Traffic between the containers and finally we have the containers the actual workload So in our It's in our we assume that you run kubernetes On top of open stack And monaska is available to you As a service from your Open stack provider With this integration Basically, what you can do is something like google's container offering gke How they integrate with their cloud monitoring offerings? um Before I continue I would like to elaborate a little more on kubernetes monitoring So you better understand the value of the integration So this is kubernetes dashboard Um, this is in large parts developed by me and my team Um Actually, we do not call this a monitoring user interface. We call it A management and troubleshooting using the face Yeah, small difference The reason is that The dashboards we we focus on use cases. So we look at users and problems And then think about what kind of information do they need? To solve these problems and then we arrange all this information It's linked and so on and arrange the information around these use cases And the result is it's Very easy to use But it's limited to these use cases. So it's not a general purpose monitoring system So getting a bit deeper into architecture Um, again, we have in this diagram have a kubernetes cluster On the left hand side, we have a couple of nodes that run containers as a workload Now containers are basically processes You can monitor them and um kubernetes Has a metric agent. It's called cadvisor and cadvisor is compiled into kubernetes cadvisor Takes these metrics and then exposes them be a rest interface Um, it's completely stateless itself So the next component is hipster. Hipster is a singleton and um It carries the api server to get a list Of all nodes then iterates over each node and scrapes the metrics process them And then again exposes the the the collected metrics Be a rest interface Hipster itself has no persistence It keeps all data in memory for 15 minutes So and finally dashboard accesses hipster And as a result, uh, dashboard has no historic data and also no custom metrics So next part logs, um, again similar scenario We have a couple of nodes with containers and these containers write log messages on standard out And um the container runtime persists these log messages into files And now if you in dashboard access Logs then actually dashboard access the api server api server Goes to the corresponding qplat and qplat retrieves The log files and all the way back to dashboard So there's no processing involved Dashboard just presents these log files So you don't have search capabilities Transformation or anything like that So to summarize, um The good parts on the nice parts on kubernetes dashboard is It just works. It's typically installed by default And it's also easy to use Works effective But it also has some limitations and These limitations, uh You have again, uh, you have no historic data. You have no search You cannot customize. You have no custom metrics And you have no alerts or no triggers So when You integrate with monaska Of course, all these limitations are actual strengths of monaska And you get some additional benefits if you use monaska as a service Rather than installing yourself And first of all, of course, you don't have to worry yourself about operating a monitoring system And second also The monitoring system is outside of your cluster. You know, it's outside of your own system and Yeah, it's often many don't like it if the monitoring target is part of the Of the monitoring system is part of the monitoring target. Let's say And yeah, and finally there are also still a lot of, um Users out there that have mixed environments So they run part of the application one in kubernetes, but still they have Lots of other parts of the application running outside in virtual machines And with monaska, you can monitor both Um So the actual integration um Yeah, we forward Metrics and logs from kubernetes to monaska monaska api And one thing that is important to understand is that we don't only Modify kubernetes or we just you you configure kubernetes, but you do not modify the applications So you just can deploy any application to kubernetes and the metrics and the logs from from these applications will automatically appear at monaska so Same diagram as before it just extended it with the monaska integration So heaps that has an extension mechanism. It's called sync and we have implemented monaska sync This monaska sync it's upstream of course. It's part of heaps So it's part of every kubernetes installation If you happen to run a kubernetes cluster you have it You just have to enable it So this is your first task when you come home Enable monaska sync. Yeah, you don't even have to Rebuild your cluster. Yeah, it's really just an api call to heaps. Yeah very simple, yeah, and for logs we We simply thought the best thing to do is just take the monaska log agent and Deploy it as a kubernetes demon set Now kubernetes demon set have Some very nice characteristics kubernetes takes care that Always one instance of the lock Lock agent is deployed on every node And this works even if you later on Put additional nodes to your cluster when kubernetes takes care and then also deploys a lock agent to this node So you always have one agent deployed on every node Yeah, so The data we have reached monaska and i pass over to vitik So uh In monaska, we have the concept of dimensions And they are there to uh describe the the metrics where the metrics come from What what they what they are about so the The the the set of metric keys and metric values together with a tenant id Is the set which uniquely identifies the the metric And of course hipster and monaska Describe the data a little bit different so In the integration uh step we had to consider Carefully, how do we map the the data which we have in kubernetes? To the dimensions in in uh in monaska so that we can use that information Meaningful and also consistent with the with the rest of Measurements in monaska The first advantage of using monaska for monitoring kubernetes christoph already mentioned are custom visualizations and also historical data Searching the data so here we can see the screenshots of the of the dashboards from grafana and kibana For visualizing uh metrics and and blocks They are uh highly customizable You can create on dashboards you can work interactively Look for the historical data search for the historical data interactively I will show it In a few minutes in the in the demo The next advantage of using monaska is its alerting capability. So You in in monaska we have the concept of alarm definitions With a simple syntax you can you can define the conditions When the alarm gets triggered When the when the threshold is exceeded the alarm is the Changes the state to the alarmed state And the notification to the user is sent Monaska supports the pluggable notification mechanism So apart from from the standard notification methods like email Webhook and pager duty at the moment. There are also three other plugins for hipchat slack and jira And here on the slide I have listed two examples So what what what the alarms could look for the first one is the alarm on the on the locks We are looking for the error messages for wordpress application with a code error 404 And another one is the compound alarm where you have the metrics Coming from from hipster and the Lock metrics coming from the from lock entries Errors from in in the lock messages and you can combine these two together All right, let's come to the demo But it can be complicated now because I don't have the display here One second You have to find one one way that works for you both here I'll let let me let's close this one. All right, but then I can't see anything Let's go back to the no, this is not right. This is not good. Let's make it like this. I will just yeah, you have to All right, I don't know why it's working different to any other monitor. Can you see this? Is it okay? Right. Okay. So here is the horizon dashboard with a monitoring UI It's empty Zoom in you said Cannot speak German. I don't know how can we I Yeah, we can translate maybe they use a mini one I cannot see it here I don't think we don't there are not so many words means overview Simultaneous translation. All right, the dashboard is empty because I haven't created any alarm definitions Alarm definitions, it's English, right? Oh nice, okay, let's create An alarm definition I want to have an alarm which Looks after errors in the log messages of our example application guest book So I will just look for The maximum of the metric log error And if it is greater than null I want to launch the alarm. I will also limit this alarm to Container guest book and I want to the alarm to be deterministic. So it does not Transit transition to undetermined state only okay or alarm And I will add the Hipchat notification in the In the event details Of our presentation I have added a link to the hip chat room if you like you can go there And you should see the notification if everything is going Okay, so we have the alarm definition There should be no alarm yet because we don't have any errors And now I go to the kubernetes Dashboard we we can see here the metrics and some Kubernetes container with the cpu usage memory Now let's go to the default namespace and Deploy our example application guest book Upload And the application should be there. Let's check it out Guest book that's just a simple Guest book application You can write some entries And if you repeat the entry you will get an error You can See that error in the in the kubernetes dashboard Here it is But it's just plain text right so what what we want Is uh, we want to look after this error now in the in the Ubersicht In kibana dashboard So I will look for our error I have saved here a dashboard for kubernetes what we can see in the dashboard Is the histogram with all the law countries and here the container we have You can see the number of Container number of ports Here are filtered the error messages only Through different container we can see the errors are in the kubernetes master And here are the lock levels in the outer Ring and container in the inner ring. So all right and here at the bottom just The messages themselves so um, I see that we are flooded with With error messages from kubernetes master. So I will filter this out Yeah, now it's more meaningful And I can see that here in the guest book container. I have the error message here No, let's This I don't need so I will filter for guest book application lock level error apply And I can see these are the error messages. I was looking for right, isn't it? Okay Um, we have the log matrix component which generates Uh matrix from the logs. Let's see if I can find them in the matrix dashboard in grafana. I have Again saved dashboard for kubernetes Here at the top are the system matrix about cpu and memory usage. Here are the cpu and memory matrix for for Several container And the matrix for our guest book application cpu memory The health check and here are the Here is the error Metric we have generated Okay, so if if there was an error there should also be an alarm, right? Let's go. Yeah kubernetes alarm critical That's what we wanted And let's go to the hipchat And I cannot see anything Why There was actually an affusion Is it the one 12 I cannot see Is it the one let's assume it's the one All right It's actually all right. So that's the notification I wanted great Yeah, that's it. Okay. Thank you very much konday any questions question. Yes, if you have any questions There is a yeah, so the question was Grafana and kibana do they have plugins for for monaska? So grafana has its data source from monaska data source which allows to read the matrix from monaska Grafana 3 integration it is and For for logs we use standard elk with our own plugins. So We have the plugin for for kibana which Gives us authentication. So you you write your data Per project and we are also in We're working on a multi-tenancy plugin for kibana So the after you authenticate you can see only the data for a for a given project But the the communication between kibana and elastic search is is the standard Right elastic search is the data story for for a box. Yes Yeah Please louder. I cannot hear Yes So, uh, the only communication Which you need is between agent and the api and it supports SSL Yes Yes, so in this case, it's monaska service. So you just use this service from your open stack provider Of course, you also can deploy elk Um in your kibana this cluster But then you have to manage yourself Also, you don't have this kind of advanced processing and and transformation and what's The anomaly detection and so on all these kind of things that monaska has but elk does not have But it Yeah No, no, it's part of monaska. So a message Yeah, so the question was about kafka if it is in Installed in kubernetes. No kafka is a part of part of monaska so I would if you want to learn more about kubernetes if you want to learn more about monaska If you want to learn something about fujitsu k5, which is our open stack cloud offering Please come to our booth To give you more reason to come to our booth. We have great giveaways parrot drone And also we have a really fascinating Oculus rift You can try out So I hope see you later. Thank you. Thank you