 Okay, so let's get started Thank you for coming everyone My name is Carlos Pérez Aradros. I'm a software engineer. I work for Elastic And work on bits specifically so if you were in the previous talk We have been working on that So probably you already know the elastic stack You know elastic search lost us on Kibana Uh We used to call this the elk stack, but then bits came into a position So now I we usually prefer to call it the elastic stack Because you know belk or lkb sounds weird And we just released 6.0 a couple of weeks ago and I really love this this animation because somehow it's It it shows how the elastic stack is made of many pieces But when you put all of them together they work really great And I like that really much So today we'll be mostly speaking about bits and Kubernetes, of course So let's get started with that. So what's bits bits a family of lightweight seepers that collect and see all kinds of operational data to elastic search What do we mean by lightweight? Well, this may have many meanings, but for us is Everything is good in in go We see everything in a single binary that Doesn't require any external library And yeah, the cpu and memory consumption. It's usually low depending on your needs So meets the bit family the bits family This is it packet bit metric bit. I will be speaking about all of them This is the list of bits that are maintained by the bits team are officially supported by elastic But just for you to know there is a community of bits out there all over github so The official bits doesn't fit your needs. Maybe there is a community bit Supporting your use case I'm going to go quickly over all of them a step for window bit, which is for windows event logs But as I'm doing a demo later on I'm going to try to do this quick Okay, so logs. How do we get logs? With we have something called file bit. It tells some chips logs for you It does a lot of things like backpressure Control this means that if it's ingesting your logs into elastic search for logstars It's able to detect that system is overloaded and stop or throttle So it waits to the system to cope up with the load We can do a structure logging that means we can first parse the log lines and extract metrics from them for you Also multi-line for for instance if you logs contain, I don't know stack traces. We are able to Compact all the different log lines and put them together And we do many other things For metrics and events we have many different bits The main one to me will be metric bit It pulls api for the api of different services. It's a common use case for a monitoring agent So it talks normally to the service you want to monitor and takes the metrics from it for you It stores things in elastic search in an efficient way And actually we're still improving this and with six zero it went down by a lot And it also supports application metrics from from other protocols or standards like Getting metrics from java application with gmx or promissus for instance A packet bit you can think of it as a Like a distributed wire shark it taps into the network speaks different protocols and can extract what's going on for you Basically it it understands Protocols like http mysql whatever It reads for all the packets in the network and then extract like a transaction from them So creates an event in elastic search recently we added tls handshake parsing. So This is that's really cool for security use cases. So you can get what's going on when your users are using ssl Her bit is like uptime monitoring. It can talk to your services from time to time You can schedule checks and check that everything is available and running An audit bit. This was the last one added to the list Basically connects to the linux audit framework. It can work with audit d as sidecar or or standalone and Something cool that it does is You know linux audit framework gives you a lot of events on on what's going on with the security on With on your system What audit bit does is Read all that events and correlate the ones that are related And create a single event in your elastic search and it also does file integrity monitoring So you can check if someone changed some of your files for instance binaries Um for traces Recently a op bit A company that was working on apm joined forces with elastic search it was it it went acquired by elastic and And since then they have been working on open sourcing A lot a lot of their code It's in alpha state as of now i'm showing later on but just Taking the quantities in alpha, but yeah, they are doing really great stuff on on apm By the moment there are no jay some python agents Which support for things like flask or jango and of course more cunning So today i come here to talk about Kubernetes and and docker in this case This year we have been working a lot on supporting this well using bits I want to thank you especially bj Samuel He worked for ebay and they contributed a lot of code here. They use bits internally And yeah, we are really grateful for that. So bj if you're watching thank you So what's all about With containers infrastructure with microservices Basically when you are using this kind of architecture everything is a moving target You static Static configurations that are not valid anymore. I mean you can you can expect everything to be in place and not moving So we need the specific specific tools to track things down And we have implemented a few primitives that you can use to monitor what's going on with containers First of all, we added docker. We have we had it already actually and we recently added the kubernetes one So you can monitor the overall the overall state of Of your of your kubernetes. So I think we saw that in previous talk Uh, we also added metadata processors That helped you and reach your events anything that is coming out of your bits can be enriched with information that gives you context On where it came from And that's really useful when then you want to navigate That data that metrics that logs whatever are even correlate them So we have something to add information about what cloud you are running on something to add information from docker And something to add information from kubernetes. I will show all this in the demo Uh, this is an example of how an event that comes from file bit So this is a log Looks when you are using the the different processors and of course you can use all of them or any or only the ones that you are interested on So what you are seeing here is a log line from file bit Actually, it's telling that it's connected to elastic search But then you get a lot of metadata of where this line this line came from So I know the pod name the container within that pod the name space and even the labels And also I get information from the cloud it's running on so in this case is google cloud as you can see And this is how it works. Basically, we are pulling the kubernetes api constantly And every time that a new pod is launched we get the event And we create metadata for it and put it in our internal catch So when we read a log line from docker or whatever We parse that get the container ID and then use that container ID to check on this table And normally you find a match and getting the metadata information for for that for that container Then we get that metadata put it with the event and seep it to elastic search logstash or kafka ui Let's talk now about auto discover This is not yet released, but it will be released really soon. It's already In the release branch, so you can expect it in the following weeks We are getting auto discover What is this about it's we watch for docker events and react to changes so allow you to define a set of Prefine templates to apply when something goes on for instance in this case We have a condition which is if the image the docker image is etcd We want you to launch this configuration on metric bit launch this module to monitor it And as you can see we get the host the ip address of the of that container from the Auto discover, so we just put that in the template and it will automatically spawn the the right ip So this is how it works More or less bits watching for the docker events api When you launch a new container or Kubernetes does you get an event With all and we create like an internal instructor With all the information about what happened. What's the event about Then we take your configuration and do Condition matching in this case the condition of etcd matches. So this spawned a new module We take the template that you gave us in the configuration and do the bar expansion so from the auto discover event with the bar Bar expansion to the configuration that you gave us and you end up with the final module that we launched for you In the same way the container goes out We do the same thing, but the other way around we detect that and stop the module Let me talk about how to deploy this and deployment strategies how we normally deploy all the different bits So when you are running bits on a docker host we normally run them from a container You could do that also from the host itself, so it's up to you But we provide both documentation and docker images on how to do it And depending on what you are doing We would like to mount Things from the host for example on metric bit. We mounted the proc file system. So we got get access to all the host Proc information so we can monitor that with the system module. We will see that in a while And for the file bit case to strap the logs we do it in the Similar way we mount barley docker containers and read the logs from there And as I said when you are reading the log then Enrich that with information from kubernetes or docker Then we send everything to elastic shirts and you can see it in kibana And if you are deploying in kubernetes, you will do the same thing but We provide demo sets files manifest example manifest so you can use them or modify them to whatever you need So, yeah, let's jump into a demo Basically, this is the demo in sewing that url you can visit already It's a really simple application written in python with jango Wsgi with engines front ends Just be careful. It's not elastic. It's like an internal url, so it's el stc So the idea of the application is that you can submit questions out of this talk If we get the right questions, I can answer from there. You can also vote them So the questions you want to get answered you cannot vote them And this is how the application is running We have a mysql pod single one. This is really simple deploy. It's just for testing just for showcasing this And a couple of pods. I think it's three Running the application itself with an engine front ends So what we are doing and of course a lot balancer on the public IP What we are doing here is getting the logs getting metrics metrics from this using auto discover to For instance detect that mysql that engines and monitor them also doing network analysis Using packet width and application performance metrics using the elastic apm Okay, so this is the Demo application. Let's see if you are doing anything with it. Okay, so some people is submitting questions. That's good. Thank you And let's see how we can monitor this how this is going I don't have file bit deployed yet, so I don't have logs here, but I'm about to deploy it so I can start Getting the logs into my elastic search instance. This one is running in elastic cloud But elastic cloud runs the same elastic search that you will download from Our page is open source. So it could be running anywhere So I'm opening this repo with information on how to do this demo So you will get access to all these jammel files Let me deploy File bit Okay, so this is now creating the file bit demo set and doing some setups It it takes care of set up in anything from dashboards to anything and Logs are flowing in already. Let me put some reference times here Okay, so as you can see I'm I'm getting logs from All my different containers Let me sort this Okay, and after bootstrapping everything it's able to Tag everything with the proper metadata So as you can see here, I can get information from the pod name Or the container name for everything and maybe you just saw that it backfill The whole last 15 minutes. This is because we are extracting the timestamp from the docker Logs so we'll get that timestamp and submit the event with the correct timestamp So now I got logs from everything that happened before I deploy this which is cool And the important thing about metadata is that allows you to navigate everything On your infrastructure. So if I'm watching logs here I maybe want to watch logs for all the engines that I have deployed and as you can see in the pod name I have different pod names around maybe in this case. Yeah, I have different pod names here Uh, so I'm seeing the logs for all of them And maybe I want to focus in only one of them That's something you can do too. You can see how the log lines change And I'm getting now the logs only for that specific pod, which I think is cool Forgot to say that we are also using auto discover here. Can you read the text? Okay, so if you see these lines I'm detecting the engines container and I'm passing the engines module for file bit What the engines module does is do all the handling of the logs for you and extracting information So you have metrics stuff instead of of the log lines So this way I'm getting both I'm getting the logs, but also getting Metrics from file from file bit for for engines So I can go and open the engines dashboard for file bit And this gives you an overview of all the visits I'm having and We have people from other sites probably in the streaming. Hello And a lot of information on you know, user agents what you are visiting what you are voting actually and Volume data. So this is really cool. This is coming from all the logs And now we can jump to do metrics So metric bit can monitor both your host system Also the kubernetes layer and everything that you are running on top of them So I enabled system module here kubernetes module here And I'm using auto discover to monitor the different cities that I know that will be in the system And for instance my school I'm matching the condition here So if you see a my school module launch this, uh, sorry a my school container launch this module to monitor it You can do a lot of tricks here because At the end we get an auto discover module. Sorry an auto discover event And can react to that and do whatever we want. So in this case we could be doing Something based on annotations on that on that pod So if I see this annotation like I don't know promissus scrape I can launch the promissus module So you don't need to hard call things here. This is just an example My school And also engines. So I'm launching this now So Probably we'll see logs from metric bit here. Well, if I remove the filter And it will start CP information about What's going on in the host? So let's open the dashboard So I'm opening first the system overview that gives me An overview of the hosts that are running this kubernetes instance as you can see I have three hosts Sorry, this is because of the resolution Uh, and I get an idea of the cpu on the different Host as you say from the name I'm running them on world cloud And I get an overall idea of the cpu memory disk network. So And as you can see here the histogram on host cpu We don't have metrics only for like the last 30 seconds But this will be full of the information of what happened in the past minutes So I will say this is the host that is running my school Only one of them is running my school and because of the memory usage I will say this is the one So I let me jump into it and we can see Information about that host and you get more detailed information here like Everything we show before plus more detailed cpu memory profiles And of course the top memory and cpu usage processes and there it is my school In the top Uh, okay So this is for the system module on metric bit. Let me open now the kubernetes Overview dashboard as I say this is directly set up by A metric bit. So when you deploy the manifest all these dashboards come come with it metric bit takes care of deploying them So here we are seeing that we have three nodes a cluster of three nodes with nine deployments going on Everything looks good. We don't have any unavailable pod And you can see also the cpu usage per node And so on and also top cpu and top memory intensive processes Taking into account that we are running all the different bits here So we can see five bit around here for instance or my school okay We are also monitoring my school Thank you to the auto discover feature. So we detected that my school is running and launching the my school a module for it And there it is we get an idea of The different queries that are happening now on my school active connections or this kind of metrics that you may be interested in As we say I just launched this so we don't have much history on this, but you will see the full graph for it Let's now take a look to packet bit as I say packet bit listen taps into the network and Understands what's going on on the different protocols. I configure it this one to understand dns htp on my school And I tell it's the ports. I'm interested on because It's too expensive to listen in too many ports. So I'm listening only in the ports. I'm interested So this is deploying packet bit for me And we should start having some information from it and I really like this because It gets you a lot of insights From what you normally don't see It may be cpu intensive depending what you are doing But I think it's really cool for some cases For instance, this is what I get using packet bit for my school It's able to understand the my school protocol and and tell you what are the queries that are being run So these are the queries that are happening as we speak probably Or the traffic generated by you And yeah, I get a lot of information that I can also get with metric bits. So you too It's depending of your needs, of course And We also have with packet bit the htp person so This is how it looks. I mean packet bit doesn't know anything about these web services. It's agnostic But it's able to understand what's going on. So actually I get information not only for my My application demo application, but for other services, but This is a cool use case Actually because as I said everything gets stuck with metadata. So I can filter To the name space I'm interested on so I'm getting all the metrics from packet bit from all the For the whole cluster and I can decide To to Listen only or to watch only the ones for my default name space I can add this filter here. Ah, sorry. So It already gives me the possible values from from the document the existing documents But I can add that filter and then I'm only watching the The request the htp request for for my name space where the application is running So this is thank you to having all these processors with metadata information We think they are really useful and They are available in all the different bits And let's check apm. I I did a trick with this one I let it run for the whole for the whole demo. So it's already running. I'm not deploying it now As I said, this is just in alpha status, but it's already working and it's really great So it's open source. So you can have a look to it In this case I instrumented my Django application. You can see that once I open the repo But it's really easy to instrument And it's getting information about all the views that are available and that you are hitting in the web page As it speaks Django, I get the exact name of the view from Django And as for now, I can see the traces using this So you say you see the response times and and the results of the htp request in the dashboard But I can also see the traces for any specific view. So in this case I get information on time, you know, about the MySQL queries and so on Alpha state, it's already released and really cool. So get that at right um I created a dashboard I created a dashboard for for kubecon So what I'm trying to show is the power of having everything in one place and having the code metadata to navigate all that and correlate all that I think it's really cool when you can do that So this dashboard was created for this demo Basically, the only filter I have here is using this covenanted metadata and instructing Kibana to filter only to the default name space, but this could be everything I mean, I can't even disable this and I get information for my whole cluster, which is probably too much But for this demo, as you can see, I can see as you can see we we have All the running let me do this bigger maybe all the running pots with the containers inside. I can I have here this So I can see that there is a mysql with a mysql container on it this is a the application pot with a Both the application container and the engines container and these other two are also pots I get the logs for everything here. So everything in this name space logs are here I get cpu usage per both I get memory usage per both Also, I took the file bit View from the logs from the engines logs as you can see I I can focus for instance to see what's going on in these errors So the whole dashboard Changed for it. So I went here and then I can check maybe the logs for what's going on and I can see here Something's going on we'll check later Let me go back to the view of Last 15 minutes for instance So I think that's really cool. I mean I can have everything in one place and correlate all that all this data. So That's really useful. The same goes for apm there. It is an information also how mysql is answering and Yeah, the same way I drill down to some time range. I can do it for An specific pot so for instance I have all the logs here for all my applications in this name space But I may want to watch only the ones from mysql Well, in this case mysql doesn't have any logs because it's not logging anything So let me focus on if you want. So maybe only this pot I get the logs for this pot only and in this pot. I have both engines and application So these are the hdp records that are being answered by this pot and only this pot And the same for apm. So if only one of the pots is that what you are interested in This is a good way to Using metadata to to navigate that And within this pot, I have two containers. So engines and the application itself. I can focus in one of them So logs for engines These are so this will be the exact log that we will be seeing with Watching docker logs for that container and I can watch all the system usage for it So I think this is everything I wanted to show today Thank you for coming Maybe you have some interesting questions here. I might can also take questions from the mic if you want World Joe. Thank you Okay Yeah, I'm publishing the slides. So yeah, give me Couple of I don't know. I will publish them during today. So if you go back to the cube con It said these slides will be published there I don't know what I'm having for lunch. Thank you Non-standard log formats Hi vj so, yeah, we are working on some things but Always you can use the ingest pipeline or logs task to do the heavy lifting there if you need to process Some format of of your logs. You can use both elastic search and and logs task to process them and We just see them Is beats an alternative to promissues well I'm not sure I don't think so maybe depending on your use case, but we can work together with promissues We can get metrics from promissues. We speak the promissues protocol and I think we are doing different things We are sending metrics for later analysis. So Can you compare file bit versus 24? Well, this I guess this This is a long comparison. I don't know I invite you to Test it and see what you think and maybe someone can do the comparison for me So, thank you and you have any other question Yeah So for packet bit we are Listening to packets we use pickup. So it's not over the wire, but we are getting them from the kernel And then we are parsing the packets from from the post Yeah, yeah, it may be expensive in cpu depending on what you are doing on your loads Normally, you may want to prefer to use logs and metric bit packet bit maybe something more specific for something Depending on the use case Yeah, of course if it's encrypted. Yeah, you don't see that worries Awesome Yeah, I think it's on right now. Yeah So, uh, ebay works very closely with the beats team to contribute a lot of stuff We recently also open sourced the collect beat Which is one of the community plugin to make it easier for I think our vision is that as a developer and the vt developer when they deploy the application on top of Kubernetes, right? So all they need to do is that define. Where is my metrics? What type from the s drop wizard? Where is my log file standard out different things and We wanted to be able to make it easier so that we can automatically script the log and metric and send it there, right? So we've been working with your team very closely on that one One of the things that I'd like to you know, please is that So there is this thing a community about using from tears. They use kibana, right? And then they use logs analytics. They use kibana So one of the thing that ebay has been trying to do is that can we actually try to unify those two so that we have like one Um, you know kind of open source console to actually look at both and to be honest with you We actually invested heavily in kibana. So we actually We added a lot of plugins Even stuff that were not natively supported by kibana. We actually intercepted the call to kibana We created the cube watch so that we can actually get all the aps servers data We somebody in my team actually presented that yesterday, right? But I think one of the things that we could work together as part of the community is that can that be one potentially, right? Let's say kibana, right? So because you have the timeline support as well that you can actually pull the you know metrics One of the things that I feel kibana is much more adopted on the operational You know team You know, but this I don't know the answer But this is something that we could work together potentially Making kibana can support a much better visualization on the time series data as well Yeah, for sure. This is something we are targeting all the time like improving kibana to to see to watch time series information And we actually have something called the time series visual builder Which is an easy interface to watch for time series metrics And yeah, uh, totally agree with you I mean you have been doing great work and it has been Really cool journey working together and putting everything in bits And yeah, there's still many things we want to do on that regard Thank you. So thank you everyone