 So my name is Ilya. I work in London company called we've works She started by rabbit MQ co funders perhaps perhaps some of you are familiar with rabbit MQ And we've been living around since 2014 and that's around the time I started working on cloud native tools that we've worked, right? the first release day a Network for Docker that that was really the first network that actually worked out of the box For like, you know containers running on multiple hosts without having to run containers in the host namespace, etc So that's that's one of our flagship products. That's called the net. It's 2.0 now 2.1 actually and More recently been contributing to to Prometheus project and Kubernetes a road tools such as QBDM Have a few projects of my own and the contribution that we made towards to the Prometheus project was Was was done by Julius waltz who may be here. Maybe not today And this is the remote right feature and also remote read that allows us to to store data Prometheus data in In a cloud service Right, we run a cloud service that provides a remote read and remote write endpoints for for your Prometheus server It's pretty easy to enable an existing Prometheus file, but You could just use our out-of-the-box configuration So this is this is just a bit of an intro about myself and the company I work for I'll show you a little bit of the product just just by just as a means of you know visualizing the data I have in Prometheus and And doing promql examples and and stuff Yes, for those who just came in this is going to be a very introductory talk to Prometheus I hope you'll find this useful And if you expected something more advanced Well feel free to stay It's a it's an app developer oriented talk as the title suggests And well, I'll just go ahead, right? So I'm an app developer. I build apps, right? That's right. Yeah, I do microservices as as everyone right and As I do those things Run them on Kubernetes. I let Kubernetes run them for me. So I don't have to Worry about silly things anymore I let Kubernetes run my microservices for me and that's that's that's what most of you do, right? How many of you don't do Kubernetes? Brilliant. We had KubeCon after all So I let Kubernetes run my microservices for me and I left this slide blank intentionally Because that this is this is what it's going to look like if you let Kubernetes run your Microservices, but don't put any observability in Yeah, you can guess so observability is pretty essential and As a developer in the age of microservices You should embrace observability and and do your best job towards that and instrument your up with Prometheus metrics We no longer have to to try and you know, find a tool that will like Look at the pocket of system calls and tell you what the app is doing, right? Why should you do that? You can actually instrument the app You can actually tell your monitoring system about the business metrics about any protocol level metrics any app specific metrics Anything like that. Even my dog likes it Wow Yeah, so dog a Skeptical cut is thinking oh Why should I instrument my app to really have to do that? Well, perhaps a good question to ask right Why well, I think this is mostly to do with microservices being a culture and Not to be mistaken as architecture, right? Culture and not an architecture You build your app You run it team takes responsibility to run the app that they ship Ops team run infrastructure Developers run their apps So they can actually put the work in to make it more observable. They can put those Prometheus metrics in Just just to to make it easier for themselves to run the app to have decent monitoring in place If team has taken on cold responsibility the better do that right because Because otherwise what they're gonna second guess what's going on You know, it's them who are running the app now Think that's kind of one of the big culture changes with Prometheus microservices, sorry It's a big culture change with microservices and Prometheus is a an observability tool That is built for microservices in the day and age of container orchestration it has native support for Kubernetes and other orchestrators and It also has support for legacy type systems where you can just write static config follow anything like that and You can also instrument legacy Systems in their entirety through an exporter, right? say you have like a set of Databases over there or mainframe system can write an exporter for that mainframe system for that into Prometheus but it works pretty well and Doesn't require any fancy exporters if you do in microservices and you use Kubernetes and service discovery and such things and the important aspect of Prometheus is that it's a pool-based system, so You don't have to configure your app and tell it like hey Your monitoring thing is over there and then oh I got to change that because there is a new monitoring thing Or whatever what I'm gonna do. Oh my client library of sports one at a time So Prometheus doesn't have that problem at all And you can run multiple versions do all sorts of things you can have like separate teams running their own instances of Prometheus and like an ops team running federated or whatnot Instance of Prometheus you can have multiples of those and it has pretty efficient Memory usage and and disk storage and remote storage which which we're going to showcase today And and all sorts of things like that that allow you that leave you so much flexibility to how you can run Prometheus so These are some some of the key aspects why you might want to care about Prometheus and What one other thing I forgot to mention it's One of these essential features of Prometheus as the the analytics facility through through the promql query language which allows you to Continuously analyze how well your apps are doing and thereby continuously improve So how do I instrument my app for Prometheus? There's actually very simple You just need to implement this metrics endpoint Which uses content type Text plane so Nothing really complicated. Oops Metrics endpoint plain text easy, right? Can do that in C perhaps I wouldn't and you you need to write a string value like key value pair for a line Name of a metric and floating point value and here's one example Metric new users count value 250 you could add labels in curly braces like that and Here is an example of This New users count metric broken down to more specific values where we use locale as as a label so here's an example of Sort of different metrics that that a Metrics endpoint of my app may output Brought us up time seconds. There's a that's that's a platform specific metric HTTP request duration seconds count. That's a that's a middleware type metric your HTTP framework web app framework whatever you call that thing and New users count that's the sort of business metric So you can put all of those in This is a very simplified version. You'd probably have labels all over the place and Like as we will see an example later on And here's a more specific case of an HTTP Middleware metric where where we have get type requests and put type requests and error codes associated with those and Yeah, this particular app has only seen one of each of those You can find out more about a Few more advanced metric types such as histograms summaries and such things on the Prometheus website documentation These things are called metric types and they're kind of like there are there are quite a few more advanced topics there So how do I use to Prometheus to improve my app as I mentioned there's this prom key or language Which allows you to curie the metric data you stored over time and You can do some pretty advanced curies I'm still learning myself. So Bear with me here. So here's a very simple one You just say a metric name and you get all the all the things there are for process uptime seconds Easy. Okay. Well, that may be not quite useful. You actually want to look at a specific job and And here Pierre would be looking at my app and this book will give me the epi uptime of all instances of my app for the HTTP stuff you'd look at speeder duration seconds count and you can do Something more specific where you'd look at my app method get path your eye slash, right? And you can sum those up To sort of aggregate all over all of the instances of that my app thing and You could do other things such as raids And we'll look at some of this momentarily in the tutorial part and There's there's another Nice metric to be aware of as the app metric This is the metric that Prometheus sets internally when it finds a given job So it looks in Kubernetes and it finds a service and it goes it looks at that so at all the pods behind that service and If it manages to scrape from from all of those it stores one in the app metric if it fails to scrape from from any of those Perhaps they're not instrumented or something like that it will store a zero so you could look at Whether whether all of your so that essentially indicates the health of your app if the app is instrumented So if you if you look at my app like this you could you tell whether whether all of instances of your app are Healthy and able to report Prometheus metrics at the very least and you can do a sum over those This will give you the number of instances actually So we'll look at some of this in more detail tutorial now How we're doing with time Plenty of time brilliant Okay, so I have a tutorial Where I'm going to use a No JS app Gonna run it on Kubernetes on GKE. I'm going to use leave cloud for Prometheus remote storage and I'm going to use leave cloud notebook for curing the data that I have there and I'm going to use tool called draft for deploying the code that as I make changes to it so Here's a So I'm gonna check out a branch to prepare a couple of branches here. I'm not very good at live coding. Oh, yes. Thank you Be one start Yeah, you can see this Tonight Mirror display Okay Okay, cool, I'll probably should zoom in We let's V zero start Okay, so So this would be you will see in the read me if you go to that repo I'll show the link once again Afterwards if anybody didn't capture that So I've done all these steps earlier So we'll prepare it I installed draft I installed the beef cloud add-ons if I go back to go to beef cloud now I Can see that? For example, if I if I go to explore you I can see that here's my cluster It's it's got bunch of things running including my Node.js demo over here and The beef Cortex agent is a Prometheus server that is scraping for from here so I mean I could show you more of this if you want so it's the end if you have time left But I can see that that the cloud agents are connected to this cluster So we're good to go and let's check out the first version of the app Well, let me tell you about this app first, right? So we launching a Pretty awesome API I'm sure how can you read as will be all over it as Soon as we launch it. It's called string Lee. It's pretty amazing It does one thing and does it well traverses strings Very useful, isn't it? so Okay, let's look at the code first So So this is this is our plain text front front page And here's our million dollar methods And Yeah, it's pretty simple. It's you use a standard library Keeping it super simple, right? So if I run draft up I Should see this running on my Kubernetes cluster now. It takes a little while. I Could show you a little deep cloud explore perhaps in the meantime. So What can I do here? Oh, this is the previous version of the thing running right now So that's kind of that's so interesting. I guess I'm gonna narrow it down into a default namespace. There's only One thing that I expect to to leave there to see Disexpected to be there right now. They're no GS out. Okay. Well Ooh conference Wi-Fi That's it's always So delightful Okay, so anyway, the app is ready now To double-check whether I have external IP here Conference Wi-Fi Didn't pray the demo gods No, am I still connected? Hmm What do I have to ping? Like oh No, could really this promise you say it today Okay Hmm interesting. Mm-hmm This is great. I don't even have to download anything in this demo Surely this should work, right? Yeah, yep, let's do that Let's do that All right Now we shall pray the demo gods. I Drop the mic. Yeah. Yeah. Yeah. Oh, yeah Maybe just a few pink pockets It's working. Okay. It's from ethios. I work That's good Okay, great. So, um, we were gonna say, um, you know, you could all get service. No just demo get its External IP. Yeah, that's been assigned. That's good. We can we can save the external IP as a variable so we can use that later So, yeah, let's let's check out Stringly up right get everybody see the console Yeah, it's pretty good. Okay. I could oh No, that's that's pretty big, but maybe One bump. Okay, that that that'd be okay. I guess So, yeah, a string Lee up is like this tells me what I can do is this And yeah, it traverses string that does one thing and does it well, that's great. We love the Unix philosophy, right? So Yeah, I've done that already So I'm gonna run Apache Bench as you do, right? We're gonna like you finally don't how can you so you're gonna Have fun whether these guys can actually stand If this up still stands after up to you like Apache Bench for a little while run it again Yeah, it's still running. Well but like back back back back of the startup at the The coffee shop we we thinking well, we're gonna have to you know make sure that we can withstand the the hockey stick growth and and we gonna We're gonna go and do some more benchmarking and We figured as I like to keep things very simple and minimal We figured we're gonna gonna write little Low-test Thing well, we're gonna use Apache Bench and we're gonna we're gonna add this hits endpoint Which is nothing really clever at all so we essentially just gonna Have a little counter here for anybody who accesses the API and we're gonna render that slash kids, right? Which is gonna render that number so if I go run Draft up again, it's gonna remain in foreground. I need to switch to another show and If I If I do kubectl Yeah pods I should see Is it the new version? Okay, draft is still building and making the release. Okay, great So now now we should see that have a new pod that is coming up Brilliant. Okay. So now if we if we just curl slash hits we can see that nobody tried to use the app Yeah See if I if I do No, this is wrong command. Oh, yeah, I have a little script here actually that That you know reads that Slash hits and stores that in the CSV file that I could potentially plot later That's that's that's pretty awesome. Isn't it? very very simple load testing framework as I'd call it and And yeah framework without the painting. It's just a frame and it's Yeah, we're gonna we're gonna run that But first we're gonna run Apache bench But I have to actually modify this a bit I need to actually pass some payload to it light right reverse Question mark string equals Hello, KubeCon and Well, let me just double check whether this is this even works, so I forgot Yeah, okay, brilliant That's an interesting word Yeah, okay, and I'm just gonna double check whether hits endpoint is working. Oh, yeah, cool, right We saw two hits on the API so we're gonna We're gonna run AB 300 C 100 and we we run that and we open another shell collect samples here and In the very first one I'm gonna run Apache bench a bunch times and Yeah, well Yeah, I think this this is a pretty cool Thing I've got this samples file now I've got like all these numbers here and as I as I'd be looking at it I'll think oh well hold on a moment Is this looking right? Yeah, it does look okay. Okay. How about I scale up my up, right? I go and scale this up And keep good old scale deployment What's called no JS demo replicas to Okay a couple replicas so keep good So run the party bench again, and I'm gonna look at my results Some polls so now I'm looking here and Okay, that's oh This is strange Why is it? Why is it zero here? So like I have no idea right then my my CSV file doesn't doesn't let me do much It doesn't scale does it Okay, so I go and Google and I go to a conference and find out like a meet-up a meet-up You know I gotta launch it tomorrow. I get to a local meet-up and I hear about Prometheus It's a thing so I go I get home, and I I figure well I'm an I'm a minimalist type person. I'm gonna I'm gonna try and read the docs about Prometheus and I write this this version was really basic Prometheus metrics and And what do I do here? Well, actually this this is all I can we need to do to do Prometheus metrics, right? so I changed from just hits to HTTP request total and Yeah, that's it Cool. Okay, let's try that. So I still use I still have draft running here in another shell So it should deploy this new version that's cool and Keep good little Get pods Yeah, so I got new new pods running. So if I if I call on Just gonna copy this from here, obviously I think I need this save the AP address and Curl metrics. Oh, cool. All right. Yeah, that works. Um So now I could run Apache bench a few more times. I think I still have my awesome Collect samples script writing. I don't need that anymore. It's rubbish Doesn't scale. Um Yeah So draft is good and my shell here cool. So now I go back to slash metrics and I can see This and that's great, but I also have configured bringing these two to store data in the cloud So I should be able to use a pre-t UI To To see what's going on. So I can see like some of the systems metric and that's great But perhaps I don't care so much about CPU usage right now It's a very simple up I shouldn't shouldn't have to care about that. Oh, great. Okay, so Yeah, so if I look at it as it oh That's that's that's not just my app. I gotta look at the Node.js demo And so these are two instances of my app. I can see that Each of them has seen Quite a few requests already. I'd rather look at it as a graph. Look, here's here's the hockey stick growth, right? brilliant Yeah, well, this is kind of cool, but you know, I also want to make sure that All the users get pretty decent user experience. So whenever they want to reverse the string 80 string they will get You know fairly quick response will Well, we'll reverse those strings and we'll do that very fast That's you know, that's what what the guy who wrote the million dollar method promised. So So, yeah, okay Well, I thought about it for a while and I thought well actually you know, there are a few reasons I could actually improve this up a little bit more and and I Figured I could use a Prometheus client that integrates where it was one of Node.js web framework like app frameworks for rest API's and My favorite one is get the minimalist one the rest of I and I'm using this this Prometheus client here and and instead of doing all of that Stuff by by hand like I did before I kind of have nice framework situation going on here, right? So I got like My roots defined pretty clearly. Here's the API route for reverse. I Already have a to-do line here, but that's okay I keep my million dollar algorithm going on here. Where is that? It's here still same. I think I never understood what that line does, but they said it's it's good So, yeah, and I don't have to do the metrics manually anymore and if draft is still running on conference Wi-Fi that looks good. I should be able to See the The metrics that Prometheus client library Has given me and I don't have this idea just I need to run this again and Yeah, so I should be able to to see all the Matrix that oh a lot Cool. All right. So, um, I'm not sure what I'm gonna do with all of these like I don't really care too much right now I kind of care about the user experience and these key requests, right? so I'm gonna look at some of these things and I'm going to look for oh, yeah, that looks cool. I can I can actually see what 95 percent of people are Seeing great. So I can I can go and check How well most of my users Are served so if I if I look at this notebook that I saved earlier To reload that And I can see here that oh Hold on. I need to deploy a load test. So I'll stop abusing conference Wi-Fi with Apache Bench, and I'm gonna like deploy a load test in the cluster. You could all create if Have a load test at deployment here Which is also instrumented with Prometheus and Just shortly see some some graph going on over here. Yeah, so my my load test deployment is up It's also using Apache Bench, but it's translate in the cluster. I don't care And Yeah, cool. Well, let's let's look at the the graph I'll keep this as a table over here It's kind of useful and I look at it as a graph over here. So I'm gonna do it an actual sum for like I'm looking at all the get requests and Actually, I want to look at Path Reverse So I'm gonna add an extra label here path reverse and Yeah, right cool. Is that does that work? No Yeah It's my load test doing the right thing So deployment Is it hitting the reverse URL? Yeah, so it should it should be that okay So that's not just a most able hmm somewhat odd I'm just gonna remove all of these labels and look at the oh This is a little unexpected Okay, I have to look at other metrics. So one of the things we have in beef cloud is that we can actually Use all the completion as we don't know what metrics are we're gonna look at So the other metrics here That's not the one Okay, well, I'm not too sure so maybe I have a bug in this code So let me have a look at the metrics endpoint once again And a good thing. It's like, you know, it doesn't require me to To know any particular format. It's kind of human readable and Oh It's not trucking any of the request and pass reverse That's odd. Is anything heating that should be a whole IP reverse String Who who okay good And if I look at the metrics again, am I seeing the reverse ones or not? Okay, oh dear have to debug that That's not showing them I'm gonna just want Apache bench once again from here because I'm getting a little confused over there So I got another patchy bench. Yeah, I'm just gonna run this again. Yeah That should definitely show up because maybe maybe that low test is broken Okay, demo gods not loving me today, I think we are close to to finish so I'll have to go and fix that take a look at the repo For about that Yeah, well that happens it's a demo as we know so if you A few more things so for those who who want to take a look at the the demo code Take a photo of the URL at the bottom I'm already well upon github and Twitter and You could try Veef cloud that cloud that people at works and Feel free to email me and join our slack. We also have a booth Come by and ask any questions and do we have time for questions now? We have a time we have a little time for for a few questions But if you guys want to go and join happy hour or whatever is going on feel free to do so Anybody has questions. I have a mic over here or just come over and And talk to me if you Thanks so much