 Well, hello and welcome everybody again to another OpenShift Commons briefing this time Brian Brazil who was one of the core contributors to the Prometheus project and the founder of robust perceptions Are the a new member of the OpenShift Commons is joining us today and he's going to give us an overview of Prometheus So the format of this is he's going to take it away Give a bit of a presentation a little bit of a demo of working with it in context of OpenShift And then we'll have an open Q&A afterwards. You'll notice because we're using blue jeans There is a chat so you can ask questions in the chat I will try and repeat them all and in the Q&A and open it up for a conversation afterwards So with that Brian take it away and thank you very much for joining us today No, thank you So We're not an echo stream of error So I'm one of the four core developers of Prometheus found robust perception She does, you know trying to do consulting in support route for supporting open source I've contributed to many open source projects over the years and they've worked in Google for a while as well here in Dublin So Prometheus was founded in 2012 by Mac Proud and Julius Balls who were located in Berlin at the time and like in 2013 it started as a side project and 2013 it took it into sound head where you're working for at the time Expanded to support Bazooka, which you can consider to be kind of like OpenShift and And they added instrumentation clients for Go, Java and Ruby, which was what was used inside sound head at the time In 2014 other companies started to use it including myself and we had Johannes in Docker So another component of OpenShift and we have a new storage system the V2 storage and a new text format Which we're still using today and then in 2015 we publicly released Because previously everything was on GitHub and it was there But we didn't really tell a lot of people But since then we've seen quite an uptick in usage So to give you an example and there are about 300 contributors to core repositories We have it's actually over a hundred and fifty chart party integrations and we've got several hundred people on the middle list and there are C and It's always hard to tell with open source software But we figured there's over 500 companies using Prometheus in production and there are several companies My own funding Prometheus development In fact, there was a recent stat that came out as well that Prometheus is 56th in the world for open source contributions, so like 56th the biggest project, which is a bit of a surprise On top is Linux itself So previous what is it and so it's a metrics monitoring system. It doesn't do logs. It doesn't do tracing It's not a provider It's at its core time series database with a query language It has clarion libraries to get the data in and a general ecosystem and in general It's looking for a cloud native approach to monitoring services The basic architecture is we've got previous in the middle It's talking to Kubernetes or OpenShift to figure out where everything is going out for applications or exporters Putting in all the data storing it locally sending out alerts and offering that up for graphing If you look over the last 20 years service management It used to be the case your systems were manually configuring everything on the machines and then we went to things like Chef and Then we went to Kubernetes, which is to step up again Where we're just moving ourselves further and further away from the individual machine the individual process And we need to do the same thing for monitoring We need to not look at every single machine and oh that CPU is high But actually care about hey end users care about latency and error rates Let's look at those rather than distract ourselves with alerts for things which fundamentally don't matter to the end user We have to look at what's the ROI of looking at a particular alert So in terms of integration into Kubernetes or to say OpenShift agrees can discover all the positive services everything automatically from the Kubernetes cluster and put in all the labels of annotations Because both Kubernetes and Prometheus have a label based data model So that measures nicely and Prometheus will automatically pick up the changes So you can just set it up once in principle and have everything happen via magic from there on in Then to add your actual application Each instrument your code to capture the metrics that matter to you Because sure it's great to know that the CPU works because we can do that out of the box But you care about hey how many times is this cache being hit or how many times is it a paying versus a non-paying order? And that sort of thing and the great thing is if anyone instruments and you've using the libraries which are instrumented Well, you get that for free get those instrumentations for things that aren't instrumented because well, it's not likely we'll get our code inside my SQL There's export it's looking to work between them So lots of integrations there SNMP for your network stuff console JMX HAProxy and of course Minecraft and Factorio if you're having a little bit of fun and Incidentally Minecraft is started by a friend of mine. I should start a new server last night and Just figure out how well his Minecraft server is performing because he uses a lot of mods So as well if you look across the cloud native computing foundation Cooper to give you an idea of the sort of integrations that are out there and how prevalent integrations of Prometheus are Kubernetes itself is instrumented with Prometheus. So we integrate with each other as you can monitor the health of the cluster Linkerd has metrics in our format and is interceptors for a GRPC and plugins for fluency So now we've got all your applications either directly or via an exporter providing data We've discovered them all from the Kubernetes services covering. What do we do with all this data? We have the promql theory language Which can do basically any math you want on time series data. It can aggregate. It can join series together. It can slice things And it can do some predictions like how close you are to your quota Like I'm a bit run out of space in four hours is more interesting than how I get 95% of my quota because you could get 95% for years Or aggregating across an entire data center for latency Another key point is that there's no distinction in Prometheus between graphing and alerting. It's the same expression language So if you can graph something you can learn so much So here is an example of a more complicated query that you would have with Prometheus so this is taking example of a CPU usage in Docker containers and Getting up per container Something that up by Docker image Not by machine not by service not by user not by data center but Docker image and Getting the top five so you could figure out which containers might want to use some better optimization flags for As an example Then okay, you've got your alerts What's thing is that not whenever we alert should necessarily result in notification You can imagine if you had a rack failure with a hundred machines in it You don't want to receive a hundred pages because that's just silly And so you can actually group alerts together that are related and only get one alert for that rack failure Or maybe one alert for every machine that happens to be currently failing in that data center with you know Hundreds thousands tens of thousands machines root them to the right team and then shot the notifications You're not spamming person every minute that hey This is still firing because that really doesn't help a human who's just gotten a page about a wrecking down It's also designed to work reliably during narrow predictions So the whole architecture is around reliability such that even if your network is falling apart We will do our utmost to make sure you get your notifications You might get some duplicates, but that's better than getting nothing So to summarize then Prometheus, it's a metrics monitoring system a time series database a local hurry language The client library for your data and an ecosystem of exporters and so on Because we're trying to promote here a cloud native approach to monitoring services because caring about individual pods or individual machines Is not sane in a highly dynamic infrastructure like OpenShift There is some links there as well the slides will be up later so you can look at it So I'd like to just show you through a bit of a demo That you may wonder how easy is it to instrument things with Prometheus We're going to use an example from Python. So this is included in client Python at the top So I'm going to copy and paste this off-screen And run that then okay, that's running. So this is just importing the basics Tracking requests time just a dummy function that's sleeping for a while Server for exposition and just some random requests So very simple example, but the core lines of code are here And the decorator here and to go there there are a kind of metrics So we got a whole pot of metrics for free from Python like Also cpu when it starts follow scriptures and then here are two metrics So we've had 39 queries so far which took a total of 18.7 seconds worth of time So that those three lines of code here are just how easy are these two lines Is how easy it is to instrument your code and you can then sprinkle that liberally across your code base So now we can look at OpenShift. So here we set up cAdvisor Which is what you should be using to get all your container stats And as a Prometheus as well running here Now what's interesting if you look at the config file, so this is the Just changes so it's a little longer than usual But this here has not been hard coded to know where things are It's actually dynamically pulling in everything from Kubernetes And then just putting out a node. It's finding all the cAdvisors and using them So you end up also discovering your cAdvisors also discovering all your machines You're knowing where to get the machine metrics So as machines are added and removed it'll automatically show up now here. I've just got oc Running on my desktop So there's only one node But this approach would work no matter how many machines I had And it automatically show up as machines are added and removed. So there's no management beyond the initial setup They've got some graphs here. So here's some Container stats stop putting out too much Here we have the node exporter cities and machine stats We can see how much CPU there is how much memory is being used the disk utilization on my Tree disks and the cD drive which is empty and all the file systems And it's also predicting here as well when my file systems will fill up As it turns out my home desktop nothing's going to fill up. So everything's reducing in size Just kind of handy for once And as well for me is itself also exposed as a whole ton of metrics Now, this is a very deep dashboard that I actually use for benchmarking from ETS But this is a sort of insights you can start to get off an application You can see all the key metrics about the from ETS to no less performance And this is a pretty well instrumented binary And so that's the main things I wanted to show you and so are there any questions There haven't been any questions in chat yet. So I'm going to ask a couple myself So And we were talking a little bit about this beforehand as well The how does the open tracing project surface in Prometheus and how is that related to that this project? Uh, so open tracing and Prometheus are separate because they're kind of different approaches to monitoring So from ETS, it's all about metrics. So time series. We're tracking individual pieces of information over time Open tracing by contrast is both tracing. Uh, so what that means is we're tracking a single request as a passes through a distributed system because if you're getting Into a tail latency isn't that sort of thing or hey a cashless hit here which causes all these knock-on behaviors Things can get very complex So when you've got something like open tracing you could produce a graph that looks something like this and actually see Oh, right Which servers we're all the slow blitz and tracing does that Whereas Prometheus would more tell you that hey 95th percentile latency is blah and then you just trace and say okay Give me an exemplar of when it's slow and I can dig down into it Then we are actually planning on using open tracing inside Prometheus Uh, largely for our queries. We want something in memory that will trace them So you can see hey, here's my slow queries and a few other components But it's kind of two different views of the same data because a metrics based approach Is going to look and say right you got all these requests coming in I'm going to throw away most of the information because it's not efficient to keep around But I'll be able to tell you for each pass any chp method What the average latency is what the bite sizes are and so on so you can get an idea of the general trend A logging based approach will be tracking every single one of those requests But it can't track, you know, every single function to goes into because that'll be too much bandwidth And then tracing is kind of a logs based approach but stitching together logs across machines to track a request for its entire life cycle that's kind of where the Kind of the tree different size of monitoring you have between tracing logging and metrics Well, the reason it comes up for me um as a someone asked me the other day about integrating You have Zipkin up here, but also integrating um Yeager into Sounds like that's not something you would do So The there are some similar coal sites You'd want like if you have a htp routing the htp router to your code You might want to both instrument for open tracing and say hey here's a span Which is what you know each of these here is a span And also at the same time increment of permitious metrics So potentially this has been done to be a little library that does vote But you wouldn't necessarily want to do that all the time But that's where the similarity is because they're fundamentally instrumentation but for different goals A lot of the coal sites might be the same Well, there's a couple of questions that are coming in um The first one is can you provide a comparative feature set for permitious compared with elk? In grafana Okay So that's kind of it's a number of different tools there So grafana is a dashboarding solution. It is what we recommend for usage with permitious and these demonstrations here. This is all grafana and um So that's what we recommend And we kind of seen them as partners. We used to actually have our own dashboarding system Called problem dashboard permitious, but we deprecated in favor of grafana because well Why develop one of those for someone else's a better one already? In terms of elk while the case cabana, which is a competitor to grafana. I believe their feature sets are relatively similar But then the bigger difference really because grafana is a different team is between permitious and elastic search and the difference is that permitious is a metric space monitoring system And the elk stack is normally for logs and these are two different views of monitoring And because with permitious instrumentation You might have inside your binary like a well instrumented binary is going to have hundreds to thousands of code points that are instrumented Uh, but it's only tracking like simple counters of how many times is this line of code executed And you can get that for basically all your system that works nicely and that's metrics based monitoring Logs for example, you couldn't have 10 000 pieces of information But every single request coming in because it would just be too much bandwidth both disk and network You might get 50 to 100 But you can get every single individual request and that's where elk shine is then the logs based approach So they're really complementary systems because the way I see is you seem like permitious or a different metric space system to Figure out roughly where the problem is and drill down and then you figure that okay I've drilled down and I can see things were slow. You've looked a bit It's all the billing end points that i'm going to go over to elk and see all right So who's been making billing requests, which ones are slow at it's these exact requests from these users go over chat with them So they're very complementary solutions. You need you need a metric system and a logging system I think you I think you just answered one of the next questions Um, do you see permitious as a solution a full monitoring solution for open shift? And I think from your previous answer is you need probably both Lodging and metrics You can also do a black box monitoring of permitious via the black box exporter So that's your very simple. Is it turned on probes like htp checks icmp checks that sort of thing But that's just something you can do permitious and works out But you might also need a tracing tool something based on open tracing You're also going to need profiling tools like your gdb your c prof and all that as well Source code so you can compare all these back to the source code to see where your performance problem is And there is no one tool that does everything you need a selection of these tools to tackle different parts of your problem We need a leatherman tool for monitor monitoring Um, there's another question here. Can all components you demoed be instantiated inside of open shift? If so Um, is there a git repo with a good read me to duplicate your efforts here? So if someone wants to okay, so all the components there is actually a blog post on the robust perception blog Which I use just to some of this And all the components can be done inside open shift. However the node exporter, which is our machine agent We recommend not running under docker The reason why is that docker is trying to hide things about the machine and the node exporter is trying to get information about the machine sort of kind of Uh conflicting there. So we recommend running the node exporter on bare metal You can run it inside Inside docker. You're just going to quite as good stats In terms of how to duplicate the efforts, you know, there's the blog post and jenice Just tweaking things to your setup. There's examples inside the primitive stock implementation directory, for example Um, shuresh is asking how can I constrain my instrumentation to class versus method level? So usually you would be instrumenting not classes or methods But like if statements are at that level because you're going in personally choosing What things are important to you from like a business or engineering level? If you're trying to instrument every method or every class that's getting into profiling and profiling because Ultimately in monitoring everything is in the bench So you've got this thing is happening at this time that this context for the context is the entire call stack the full htb request everything Logs works by taking about 50 pieces of those information and storing all of them for every event Metrics instead just temporarily compresses them Profiling just says give me everything And it's just so much data that you have to take Either the logs trade off the message trade off Or only turn it on briefly because it's going to kill your performance You can have to be careful about how much things you're monitoring Uh, but the really valuable stuff is more to do with business logic than every single method in class Because that's like cpu profiling at some point which is a different beast So there's another question um that just came in that's A great explanation of the class method one. Um Can you compare prometheus with heapster ocular? um, okay, so those are two different tools screen lock and uppermost so If you look at kubernetes itself The metric server provided by the prometheus format And the thing is though that that's not great if you're running a graphite or an infox db Although infox db now actually supports that as well capacitor But let's say you're running graphite and you want metrics out of kubernetes What heapser does it goes pulls all those metrics and takes them and Puts them out to graphite for you. So that's what heapser does. It's not a monitoring system in and of itself It's more an adapter for taking this data in the previous format, which is all fully open and putting it to a different monitoring system uh, ocular den is a time series database primarily So it's somewhere where you can store your data. So for example, kubernetes itself is only intended to be ephemeral Uh, it's just for reliability. We don't want to rely on distributed systems Which doesn't mean we're limited to the size of a single machine in terms of how much data you can store and how far back you can store it But then we'd hand off to something else Maybe hawk killer, and maybe we've cortex and maybe open tstb Maybe infox db for the longer term storage of data sort of kind of complimentary in that way Of course, hawk killer potentially you can use as well yourself. I haven't looked too much into it But I know it's meant to be a distributed monitoring system Restorage So, um, since you're one of the core contributors to prometheus, can you tell us a little bit about That what's coming in the next release for prometheus and what's on the road ahead? And where are you? Well, the big thing that's coming up like our point release like we had 1.7 came out yesterday or today And which is largely incremental changes Like with open stack services covering a slight tweak to kubernetes service discovery Um, you know, but the big thing is coming up is for me to choose zero which we're currently working on And that has a new storage engine, which is far more efficient than the current one And it's also one of the things I was working on is new stainless handling, which is a semantic thing Which is important like right now if a Target disappears like you stop a pod It'll take five minutes for the data points that prometheus suggested to stop being returned With new stainless that goes away much faster than that And so it takes only one she's scraping for so that's much better for alerting And and there'll be a few other changes as well. But prometheus two zero is the big one also the alert manager To just the zero seven zero release just out has a new ui, which has all been redone So it's much nicer to use So, um, where are you looking for? Help on the prometheus project Um So we there's lots of places to help So there's lots of integrations if you have some tool that you wish had Metrics, uh, you can add prometheus instrumentation to them or write an exporter if that's not possible for whatever reason So because we've already got 150 integrations the more we have the better And we also got the core repositories is like 25 ish of them There's lots of bugs there that needs fixing features that needs implementing in a variety of languages And as well like helps if you can help with user support because as the project becomes busier It's not always possible for prometheus team Which is about 15 people to answer all the questions and more people would help Look community questions as well the better and documentations always goes front of right a blog post and has to do something That's great because once again the core Like four core developers, but there's a 15 core like on the team who are committers and we tend to everything so It's like can you help with the ecosystem as well as the core? Well, there's a couple of folks asking more questions and gary's talking about integrating jagger with open tracing soon So hopefully he'll have something to blog about around that Um gear gear may is asking. Um, just prometheus plan to support long-term storage of metrics So as I mentioned just there a few minutes ago Previous itself is designed to be for reliability as the first goal and reliability means that your alerts will be received It does not mean that all your data is safe So really the self is more than a femoral cache. So it's not meant for long-term storage of metrics So the question is where do I store my long-term metrics like stuff from a year ago? So like new capacity planning The answer is we have apis. There is a remote write api That can just write out all the data prometheus receives as it comes in And then you can build the distributed system based on that to Do as much data as you want. For example, we've cortex is that If looks to be you're talking about supporting that as well and possibly some other companies And then the idea is that that will be there and because it's a distributed system It might be less reliable under a network partitions and so on whereas for retis itself keeps on alerting And we can also read back that that as well transparently So that's the basic plan is that we have apis for this And the apis are there now The list is experimental because you might need to make some tweaks to them But they fully work if you want to build something around that Cool Um, there's a couple more. There's lots of good questions today. Um One of them is is does prometheus just work with open shift or was there anything you had to do to integrate it with open shift? So because open shift is kubernetes, there was no work because it's already kubernetes integration for prometheus so No, we have integrations like the service discovery is the main thing which is figuring out where are all the pods where are all the endpoints Where are all the easy two instances? So we've got about 10 of those now like open stack was the most recently added one And we've got ones for baritone. Uh, we've got There's nerves nervous being added, which is an airbnb ting There is whatever the twitter equivalent that is as well as it's like 10 of these cross different systems as or Tink we've added gce by now as well But if you don't support if you don't support your team because it's a little too specialized or no one sent a pull request yet There is uh, you just basically chuck us json with a list of all your targets and that works too But everybody else is just normal. This is just another way to run applications Because service discovery is the only thing that's kind of special All right, I think that answered michael's question. Um, there's one more request for a little bit of a demo Can you demo how to to alert on an item since you have it? Okay, well, let me just Do this the easy way so we just pull up the demo Have an actual nurse here somewhere So this is demoed up robot's perception. Leo. I've done weird stuff there. Uh, a simple alert looks like this So we just go alert name that example alert always firing and just an expression and that's creates an alert And then if I look at 9091 I'm sorry 903 Which is the this is an old alert manager. I can see this alerts here And it can be routed from there To email to slack the page review or whatever, but it's two separate steps The primitius produces an alert which goes to alert manager The alert manager takes alerts from all the primitius servers And then lets you aggregate those together into a single notification that goes to page review So the principle being that if you have multiple from media servers looking at different parts of the infrastructure Even if they're in different data centers, you only get one notification for a problem And that's like a key feature for reducing your paging noise But there is I think there's some blog posts out there showing how to do it in full We don't have anything to hand right now All right I think that has answered most everybody's questions. Can you on your slide deck show a slide on how to contact you? I'm guessing you probably have one there or resources for That's all for me. But just go primitius at the most perception.io will get to me All right Because those are the official for me to just kind of like stare I just have one more question. Is it okay? It's a quickie. Ask it Um, I'm just curious how this is tested in open shifts. So I presume Prometheus has unit tests and there's whole testing infrastructure there Then it's further test into the kubernetes community And then from there presumably it goes into the open shift community and more testing occurs and then Open shift qe You know does productize, you know testing other productize bits. So is that how the testing account abilities Lay out here So actually, um integration testing is one of those things that we love help on the primitius Uh, so a lot of this stuff, uh, like it's tested by people doing canaries and so on But we are talking about how to do proper integration testing because a lot of the time we discovered bugs when there's bad code Okay, so that is something I would have helped us So just clearly it's the integration testing of primitius in the kubernetes Or in that case, yeah, but the problem is with something like azore service discovery We need to run azore with ec2 service discovery We need an ec2 with maraton. We need to set up maraton and having then all these integrations is a lot of work Figuring out how to maintain those So that's definitely something we would like assistance with for those who are you know that way inclined Okay, so I'm definitely inclined particularly on the integration testing. I mean In the open shift Yeah, yeah, but if you want to go the primitius developers list is there if you want to chime in there Because I know it's something we really want to do Because we've been bitten more than we would like by refreshments Yeah, and there's definitely Definitely focused on the the open shift team That can I can point you to michael if you drop me an email Um that have been playing with and working with and testing from ethios for use Oh Okay, so there are people that had testing this integration Yeah, I'm not I'm not sure it like we're not productizing primitius In any way at red hat, but I I know people are using it. So And have been testing with it. So it definitely drop me a note. Yeah, I'll see if I can hook you up with them And I'm just curious mentioned regressions and integration. I I mean I seek out problems. I'm curious So I'm just curious. Where is the fragility or where where are the issues? Is there like a So it's normally with well integration of the external systems is normally such a service discovery because a lot of them their api docs aren't great and semantics of some of their fields that they return out great It's not clear exactly what's going on and the error handling can be undocumented and we only discover it when it breaks Especially when you're talking like a cloud provider like, you know, it was or gc Ac2, you know, you only can only discover some problems when they happen to someone So those are very hard for us to test and also if we have to get an account with everyone, you know, we're an open source project On the other hand We've got things like non-export. It was a machine monitor. And so how do we test the free bsd integrations when none of us run free fb sd Or the open bsd integrations and all those sort of questions So figuring out how to test all this stuff like I know Because we're packaged in devian by market Ferrari that's discovered a few bugs in our locking stuff because it breaks on arm So that's not tests. We run ourselves because we're all 64-bit xe6 So being able to expand our testing infrastructure will be quite useful And especially your integrations with kubernetes and open shift because a lot of our users are on that Okay, and just one last question. I don't want to stretch this out to one I'm just curious on the relationship between prometheus and your new company or robust perception.io Just what the yeah So for me, it's an independent open source project. No one company controls it And I'm one of the core developers and then separately robust perception is my company and we do consulting and support for prometheus So that's a relationship like robust perception. We sponsored from convo years now And and you know we help the community in all sorts of ways we contribute development time And then we're going you know trying to make money off consulting and support off prometheus, which is largely working Got it got it. Thank you One more question came in and actually it's a good one maybe to end on now that prometheus is under the cncf What benefits have you found being part of the cncf? Is that helping grow the project? Yes, it does help to grow the project it gives us It's easier to talk to the other projects as well and all work towards the cloud native vision Because you know each of us are seeing Similar issues but in different ways around people getting ready for your idea of cloud native And how you have to move away from the older mindsets of when every machine was special as well like they help a lot with marketing and And just general advice on how to run an open source project because there's some very experienced people there Yeah, no, we've found that it's been pretty helpful too for lots of things and there's lots of new things coming under the cncf and they're all Part and parcel of what we're trying to integrate with it open shift and into the open shift ecosystem It's been really helpful for us to have that connectivity And will be will you be coming to the austin event in? No, no, I'm trying to cut down my travel offer doing too much last year ah But hopefully we'll get some of these demos and with jagger and other things Through the cfp process this time for kubcon in austin and we'll see some more integration Work done. I think someone from us is going to be there. Yeah, I'm sure I know at least the lexus will be there He can't get out of it So he's he's been demoing permeatious through we've for quite some time So, um, there'll be there'll definitely be a few folks So anyways, I want to say thank you very much great presentation I will be putting this and a pdf of the slide deck up on the open shift blog at blog that open shift calm As well as I will dig around for all the references for the blog posts that brian has mentioned and Put the links to those as well there. So it usually takes a day or two to edit the videos but thank you all for your questions and Take a look at the events calendar on commons.openship.org for the upcoming ones We're going to two days a week. So if you have a topic you want to hear about We're going to do Wednesdays and Thursdays at 9 a.m. Pacific. So let me know and I will course stock and bribe people to come and talk just like brian has today. So thanks again brian and We'll talk to you all soon Thank you