 Hello, everybody. Good morning. Well, it's morning for me. It's 712 the morning for me for my guest. It is much later in the day, but My name is Kat Cosgrove and this is the Road to KubeCon In the run up to KubeCon, we're doing a few special shows interviews with people talking about some of the Kolo events Talking about CNCF projects in a little bit more detail than we might otherwise and on and on and on But first I have to say that this is an official CNCF live stream Which means all of us including you in the twitch chat are beholden to the CNCF code of conduct Which essentially boils down to just be nice to each other Please Anyway, again, I'm Kat and I'm joined today by Richard Hartman Richard good morning. Good afternoon. How's it going? I'm all right So we're here to talk about Prometheus observability and specifically we're here to talk about promcon, but first of all Usually when I host these shows, it's a very like 101 level thing where I either actually don't know what a project does Or I pretend that I don't know what a project does to try to get a more like Broad kind of junior level. I don't have any surrounding context understanding of what a tool is So can you tell me a little bit about Prometheus? Yeah Prometheus is a time series database Basically you record values which change over time It might be temperature at your home It might be the number of times you opened your door and both of those are not the cloud native examples, but they're rather easy to to to visualize Basically Prometheus stores event no, then we start It's a time series database which stores only numeric data So if you have events you can collapse them into numbers like for example counting things or or similar But it's all numbers which means you can do math with everything which is stored within Prometheus as Prometheus has its own language with which you can query the data and visualize the data and Export it via API and create generate alerts and everything you have this one functional language to do actual vector math on your data And it's pretty quick and pretty successful Other things know. Yeah, well, I mean it has the same label system as Kubernetes, which is kind of convenient Almost as if Borg and Borg one had been designed for each other Yeah, sorry for the one-to-one content There are two projects within Google Borg and Borg Mon Borg is the orchestrator and Borg Mon is the monitoring thing and The open source equivalents are basically Kubernetes and Prometheus. They weren't started alongside each other but From the initial design They have been basically written for each other which you can still feel in in In today like also those were the two first projects to join what is now CNCF Yeah Cool actually was a very thorough interlevel Explanation thank you for that. I appreciate it. I also think that Prometheus has one of the coolest logos in the CNCF landscape It's just it's very pretty it stands out. So props to whoever designed that logo, but You have a colo event for kubecon kubecon is coming up. I'm flying down there next week. I assume you will not be there Right now. Will you be there virtually? Yeah, yes Like at prom con and I have two talks at kubecon Cool good. Well congratulations on two talks at kubecon. That's geez. Good job. It's the limit Yeah, but you got them both accepted No there maintain a trick and such Okay My talk got turned down but it's probably for the best because it was gonna be like an insufferable anime thing with Celeste Horgan and Maybe maybe the CNCF does not want to put up with our like weird Anime stuff, which totally reasonable. We're insufferable, but Looking at looking at ccc looking at looking at debcon those kind of talks coming nice Oh, we'll definitely get accept get it accepted somewhere like we're we're not gonna let that one die on the cutting room floor That's that's gonna happen. So anybody watching this later wants to listen to me and Celeste Horgan talk about communications and docs writing for Small open-source projects from the perspective of an anime Hit us up. Anyway I'm not the subject at hand you are so in advance of kubecon There are a bunch of co-located events with different subjects different projects involved or running them and prometheus has one prom con so What's uh, what's up with prom con? Who are the the heavy hitters? What do you want to? Bragg about and uh, where can people register what's involved? So as it's a co-located event, um, basically through through the normal kubecon channels, um, you Must register for kubecon to be able to register for prom con both virtually and and in person Um, obviously same as kubecon itself. We have more virtual than in-person attendees From the numbers which i've seen which kind of stands reason We hope to flip it. Um, from what I know, it's it's slated to be yet again the largest pre-event as per usual Is it always the largest really? Um, well When we do it. Yes, like initially in 2015 or 16 in Seattle 16 It it was a co-located event. Uh, then we split it out into its own thing um Now for the last few kubecon last two kubcons. We've been co-located again. Um, because it made a lot of things easier to plan with Sure With covet basically, um, yeah But yeah, we we have the largest room which we will definitely not fill Anyway the actual content The actual conference Yeah, the the bragging is also important. I mean, we are on no brag away brag away I'm I'm here to give you a platform to to brag and to Hustle for prometheus and prom con. So like if you want to brag like by all means be my guest Yeah, um, I let's go to the to the to the heart I'll get to the break later. Um, so, um I mean prom con is basically designed as as the family meeting of Of the premises community and and that's like Emily thing we deliberately designed for just Not That get in pretty much any other conferences It's usually meant to be a mixture of 101 content and in depth We have a little bit of intermediate But usually we tend we tend to like we try to have it equal but we tend to optimize towards 101 I love that so much I love that so much. I think 101 content kind of gets like Ignored a lot, um in in tech, especially by like open source projects that may not have a ton of resources I know prometheus has a ton of resources now, but it's well In some projects, right? But it's uh, it's a thing that I think gets ignored a lot and that I think that's a huge mistake To to ignore it because even an experienced engineer who's completely new to Your tool or this type of tool does kind of need to be treated like a beginner and it can kind of It can make projects seem uh artificially more difficult to use at first than they they really are so Um mad mad respect to actually focusing on 101 content. I appreciate that But continue. I'm sorry I mean as to resources if anyone wants to get involved with prometheus, please do we have so much work And also with documentation with tutorials like it doesn't need to be cold. Um, we have more than enough work and we have Not a lot of people who who do the work So anyone who is listening and wants to get involved, um, absolutely like open doors all the way um And just for anyone who who wants to explore this one-on-one thing The the way I usually structure the schedule. Um, and that has worked nicely for all types of conferences If you start with 101 you ramp up through intermediate and go to to in-depth and then you you fall off again with more intermediate and And one one content at the end and that's how you structure the day. Um Of course, it basically allows everyone to To get a little bit up to speed Before before the really heavy stuff comes. Um, and then you have a A mellow end of the day We couldn't do this this time. Of course, we have this hybrid event and and the av staff needs I think it was 15 to 20 minutes to switch between Virtual and and onsite presentation Which means we're basically having one large cut over in the middle of the us day Yeah, which is how all the other events are structured as well as far as i'm aware, right? Yeah It is so yeah, it didn't work out this time, but usually it works out um, yeah for the for the more in-depth things, um, the most important one is sparse histograms. Um, which is one of those Things which have been asked for for ages, but which are super hard to get right um And the thing is if you do it wrongly You're basically stuck supporting the thing even if you deprecated blah blah blah you you cannot really get rid of it Oh, right. I'm just because Historically, even when something is marked as experimentally we treat it as basically, um rock stable And are super careful about not taking it away cause someone might literally depend on this with their alerting right And that's one beta one Sorry the v1 beta one apis and uh in kubernetes. We have had you know A similar similar issue. Um Yeah, it's painful Yes, I I see the reminder slash begging uh that this isn't that thing will be deprecated kubernetes Please oh, they're removed now as of this version. They are removed. So They've been deprecated the v1 beta one apis have been removed in this version. So If you upgrade to uh the the current stable kubernetes, yeah, you're not going to have access to a bunch of those apis anymore so I'm glad to see y'all are um Keeping that in mind and avoiding Making similar decisions even though uh, it may be may make things a little bit Slower to release or or harder to build, but it's I don't know monitoring. It's not something that you want to really, um gamble on In my opinion, um, we've had some some rather dramatic events in the last couple of days with the Facebook outage and um the twitch leak this morning. So Um care about your tooling People but Yeah, the virtual hybrid thing is making events difficult, but it seems like y'all are handling it pretty uh pretty graciously Is there uh, are there highlights that? like big talks at prom con that you want to Shout out uh things that you think are particularly interesting give us maybe some hints about announcements That might happen around kubecon Yeah, um, let's let those two so for for the prom con side, um The most interesting one this time is sparse histograms and how they have been implemented what this actually means for the users um That's one of those things which are it's just super experimental, but still it is extremely impactful and also it has A huge impact beyond just permethias. Um, of course, um Nobody will be will be shocked and surprised to hear that for example open metrics and open telemetry are also looking at high resolution histograms and Ideally, they are not incompatible, but actually work the same um, so this um, this will actually echo throughout basically the complete uh cncf ecosystem for Quite some time to come um, that that's the thing which i'm absolutely most, um Excited about um, we have quite a few use case slash Case study type talks this time which uh, is nice that it worked out this way um, there there are several companies and and people are just talking about how they made it work in their own Specific scenarios. We're really trying to basically, um No approach should be repeated within like say two to three years So it's always new to to people who who rejoin or who join the community or who watch the things that's not getting like You shouldn't have the same thing every year. Um, we're usually splitting those between cortex and thanos which are long-term storage Based on primates and have overlap in the maintainers between those three projects. Um So we have a few of those um yeah And we also have other stuff like we have timescale to him We have 3dp and such which are also not primates itself because one of the things we try really really hard is to be truly open and not artificially block anyone from from Basically using what primates is built and and building their own business or or their own solutions on this As long as it's compatible, which is the perfect segue to kubecom That was that was a sick segue. Are you sure you're not a like a pro streamer or like a developer advocate or something? That was that was Primo chef's kiss I have been running conferences for Over decade mc them for ages like At some point practice and raw talent. Yeah, well done. That's a mistake, but thank you. You're welcome So I did submit something to to prom con but that was rejected um course cnc f stuff pulled it into kubecon, um That's a talk about um the primates conformance program Where we are So basically we are a victim of a victim of our own success for for a victim of your own Success, I love it Yeah, no, but it's it's it's true, which is a good problem to have and a really bad problem to have both at the same time For the more engineer minded people it might be non obvious, but the market which Which we are basically dominating with the primates project is billions and billions It's substantial Which means there is a lot of incentive to To play Nice with that with that ecosystem. There's a lot of incentive for companies to To be perceived as primates compatible There's two Main pathways to do this one is to invest a ton of work And several companies and projects have done this the other is to invest a ton in marketing um They're not exactly equal in outcome those two approaches and Basically, we we realized through support questions through discussions with various users with various other projects with various vendors um That we need to actually level set and that we need to Introduce a mechanism where any project or vendor can prove to their users to their customers that yes, they're actually compatible two primates Which is unfortunate on the one side Of course, I would much prefer if we didn't have to do this, but I mean it's similar in kubernetes world On the other hand one of the benefits of this is it has forced us to write a lot of things down We're not nearly done but we have Written a lot of specifications and such which we didn't do before of course We didn't really need to but now that we need to test against something We need to write the specs down once we have written the spec down We can actually version that spec which allows us to to have a good upgrade path which Unblocks us in this we need to support certain things forever more um Because if I declare this 1.0, then I can just make a 2.0 of the permissive remote write interface Or what have you and I can say okay? This will be supported for x amount of time and after that you have to change and all those things are are the positive um unintended consequences of the of the conformance program And I will talk about this on october 14th, which is basically the shim off of a talk And then I optimize towards q&a time Of course, we had to record this early enough that we didn't know what the results would be and I still don't know What all the results will be Yeah, it's uh, it's it's a little bit hard to predict with the like I mean in general I have like public speaking is my job and in general I still have like some difficulty predicting like how much time I think Will need to be spent on q&a, but it's like extra hard Uh with things virtual and very weird with the the hybrid model because like I I don't know I I don't know I'm not speaking at kubecon this year Like I said earlier and honestly it's kind of a relief that I'm not speaking at kubecon this year because I will actually be there in person And uh, it's going to take some It's getting used to the whole people thing you know, but um That's exciting. You'll have a lot going on like in a very short period of time a lot to announce and Geez Are you are you good? Are you getting enough like sleep? Um I I used to have a slide in in the opening for prom cotton. Uh sleep is optional for the week Um, I mean anyone who's listening to this in particular junior people Don't listen to all the people who tell you you must hustle 24 seven and blah blah blah blah blah You should actually get some rest and I I I pay the price for this Um, so yeah, but there are more intense and less intense periods of time This is currently a more intense one Yeah, it's uh, that that is a thing worth noting. Uh, if you're attending kubecon, whether you're doing it virtually or uh in person I know that it's like very tempting to just like go go go and then party at night and Do that a couple times but but please get some sleep like don't don't run yourself ragged for a conference I'm not going to like you're you're not going to see me out until like one in the morning Every night of kubecon. I'm going to get some sleep and you should too and also drink water Please you're gonna be wearing masks all day. So you're not gonna be able to like sip water like you normally would So make make sure you drink drink water, please But let me pull up the prom con schedule and see who some of these Speakers are uh Ooh an ebpf talk Ebbf was super hot uh last kubecon there was a lot of ebpf on the schedule And it looks like that trend hasn't gone away for sure Uh, so the schedule is absolutely stacked. I'm gonna put this in uh, the overlay So y'all can see it real quick, but you can get to it from promcon.io add a caption There we go. So y'all can check out the schedule and uh You do again need a kubecon ticket to be able to actually attend promcon, but uh When is Oh, it's it's october 11th. It's monday. It's the day I get there Oh, hey, it's the Minus one I think or maybe it's now called day zero last time we were called day minus one Which was a little bit weird. I don't know what the name is. Um, I think day zero is the 12th Then we're still minus one. Yeah, I think you're minus One. Yeah, the main conference starts the 13th. So 12 is day zero. You're you are minus one so Yeah, well I'll stop by uh because I can I'll get there that day uh there's a lot of competition for colo events, but um, this one seems pretty stacked and I actually like genuinely not putting on a show Don't don't really know that much about prometheus. So, um The fact that there is some intro level content on the schedule is Extra appealing to me personally. I can actually learn from that. So that's that's rad But, um See me Register you can also just a moment. I'm dropping a link um There if you want a little bit more of one-on-one stuff for prometheus observability, blah, blah, blah You can find basically all my talks. Um Or almost, um There for anyone interested And I include you if you Oh, yeah I will uh pop those up on the overlay too. So people can see them Uh rad So is there anything else you want to highlight about prometheus where the project's going What you're doing at cube con what's happening at prom con? so From the prometheus as a project perspective, we've been putting a lot of effort into into Appearing more open and i'm choosing this phrasing extremely deliberately We've always been very open I I I've been doing open source for two decades. I I I I strongly believe that we've always been extremely open in all ways of participation That being said it was often a little bit opaque on how to actually get involved um There are also some some cyclic discussions and such but by and large those stores were open. Um yet we were often perceived as not being easy to engage with um, which is unfortunate basically And we've been putting a ton of work into into fixing that perception Um Which means for example that all the dev summits are fully open. Uh, everyone can join anyone can Suggest stuff for the agenda They're on youtube. Um, anyone can also join during the thing and I strongly encourage to speak up With whatever they have like those kinds of things. Um, we have office hours. We have um, we have Working groups now like for tstv or for the documentation and such Which are fully open to join We're aggressively trying to increase the contact surface for people to to come and join because You were saying we have a lot of resources. Honestly, that's not the case. Um, we Given the size like when I compare us to for example kubernetes, it's um, yeah, right difference Well, yeah kubernetes. Oh for sure and like no, but Basically we we have half a dozen people who who really consistently chip away um Between half a dozen and 10 like we have I think 17 or 16 maintainers as of right now or team members with a few more maintainers for subsystems um and None of them are full-time on primitives. So um, absolutely. We have more than enough work and we are really really trying to To get this across that those stores are really really open Like wide open like you're begging people to walk through them Yes I get it It could be like It can be overwhelming. Um Trying to figure out how to contribute to an open source project and it can be it can be like, um, it can feel really daunting Right, you feel sometimes especially new developers Will feel like maybe they're stepping on people's toes. What if I do something wrong and embarrass myself in public? I don't understand the get flow For this. Oh, they don't really need my help because I'm not an expert. So how could I possibly contribute anything? but Y'all please um Speaking as like somebody who has once felt like that um, usually the maintainers are like really friendly and you you can just reach out and and ask if something is unclear about the Contribution process and it doesn't have to be code contributing contributing docs or like Samples or something is also super helpful and often like that is a thing that open source projects are super lacking in Um, they don't they can't often hire technical writers So like we we have to write the docs ourselves um, which You know, I wish we could all have technical writers and pay them what they're worth But the reality is that like we can't so um Well, can people reach out to you if they have like concerns or like their trepidations about contributing to promiss? absolutely like Yeah, absolutely rad Uh, you want to tell people where they can find you online? uh, yeah, um at twitchy h on twitter is is how most people find me these days Richie eight at richie h.org if you do email, but I I sometimes have a little bit much email in my private account Yeah, that's relatable We have the primitives um developers and users mailing lists. We also have an overview just a moment I'm going to link it to you in a second. Um where you can see the various community channels. Um Just a moment here And speaking in particular of documentation Like website updates or or or website tech on on the back end Writing documentation is is one of those things where you can really really Use some help. Um, in particular, of course, we are currently refocusing the documentation from from just a reference documentation As a first and foremost novice to intermediate documentation Um, we've been trying for half a year to to to build up both at the same time But basically for no one can really invest the time to to change the website as deeply as we needed to And the people who created it left to some part. Um, basically we're stuck Yeah Now just changing the main documentation uh to to be really user friendly because that's one of those things which Which we are not doing very well historically to be honest Sure, of course, we keep having those waves of adoption. Um, and from from the inside you can actually really see those waves coming from different areas Um, just arriving in in the Prometheus ecosystem and too often it's Not as easy to onboard yourself onto this whole thing as as it should be I mean, I'm biased But I would say that Prometheus is actually the easiest to use tool which you have in the monitoring slash observability space by far Again, I'm biased but you are but that's okay. You're you're here to brag It's not even bragging like I I've I've my I come from the operations side. Uh, my my sanity Really depends on on point alerting Uh, and on on being able to resolve incidents which are popping up left and right quickly and without losing too much sleep it's um Yeah, and having played with or having depended on a lot of different systems I can honestly say I've never seen anything which is as easy to use and as perform and as overall good as Prometheus But that's not reflected in documentation. Um, and in particular if you have like I said we have a a Functional language to do vector math on your on your on your data And that's based on label sets. You can make n-dimensional matrices instead of having tree like structures And all of this is super nice But if I say it it sounds super scary and you won't know what that actually means And there needs to be an easy way to actually onboard onto this thing to to get familiar with it Yeah, like if you're already doing Kubernetes, it's it's kind of easier because at least the label sets and everything are already clear but The thing that you have literally one single binary which you start everything is already running And then you just have one two ten different binaries or you point it to your kubernetes cluster or your zone transfer in dns or what have you and all of a sudden the data starts coming in Honestly, that's not reflected in in in our documentation that ease of stuff and we need to change this or we are changing So you really help with the docs. Okay, noted. I will be happy to hustle for that on twitter because good good docs are a thing I am deeply passionate about And bad docs are something that I Spend a lot of time complaining about We do have a question from twitch If you've we've probably we've got time to get through this question Do the observability talks that you mentioned go over how to get started creating useful metrics dashboards and whatnot No, they don't but um, I'm currently collecting Um Stuff for a choose your own observability adventure type of thing I want to create a lot of small like One to maybe three minute Pieces Which are a directed graph of learning so you can say I'm I'm starting here and I want to get to that place in understanding what that space actually is And as part of that I want to go through a few of those I have very strong opinions on on what proper dashboards are and and what you should do and what you should not do I mean we can touch do we have more questions than if not we can just No, you can you can go on that if you want go for it assuming grafana But it's it's equivalent for or it's applicable to pretty much all systems First If you have filled graph like with with the where you have that pretty line and you you fill with color underneath Please don't do this unless you have cumulatives where you signal that you're actually Adding something versus just visualizing a line. They're super pretty, but if you overuse them It gets really confusing and part of making good dashboards Open parent thesis. This is not perfect for accessibility if you overload the visual system But it's useful for for humans as a general rule But you still need to take that you need to have more more not basic but More High contrast and such dashboards also that being said That's a super quick way to visualize if something is just a line or if it's just if it's something which is cumulative If you have something which is a finite resource with which has an upper limit a natural upper limit But also goes up and down the thing which I like doing is that on the left y axis I have whatever the actual count is and I come from the networking space. So that's Super common that use case Where you have x amount of traffic on an interface, but at a glance It's really hard to see if that's 10 megabyte or megabit or 10 gigabit or just 10 kilobit If it's just a line going like so so on the right hand side I like to do the percentage so I see the absolute usage of the thing at the same time So even if I have a super flat graph on the left side y axis, which I want because this increases the vertical resolution dynamically to to whatever level of Of changes I have at that time or in that time frame I can still at a glance see what what the absolute absolute usage is But you need to discern the two so you make this left one thick and the right one thin and With by doing that you have already like you see it once and you already know what's happening If you have anything which goes both ways again, I come from networking you flip one side. So incoming goes up outgoing goes down and with with just those three things For anything which is is basically receiving and sending anything. You already have a super nice Super nice basic graph which trans transports a ton of information In a super quick way, which humans are basically Evolved to to assess really quickly and and understand really quickly Which is why I like doing this and don't overload it like Dashboards with 20 things and they're sparkly and and colorful and everything. They're beautiful But they're so hard to read. I get overwhelmed looking at them immediately and it just I don't know My eyes tend to just like glaze over Everything at that point once I don't really want more than like four to six graphs on a dashboard at any given time Before like beyond that. It's not it's just too much for me to look at Um, we have another question from twitch. Well, if you've got time for it Is there an abstraction available or in the works for Prometheus since I think it's pretty hard to get started What does the person who asked what what do you mean by abstraction if you can just follow up on on that question? Uh, well, let's see they can they can hear us with just a few seconds of delay. So, um, hopefully that person will reply and follow up with a little bit more More context, um On the pretty hard to get started. Um, that that is an issue that can be fixed by People contributing to improve the documentation uh Which is like if you're new to Prometheus you have one Ability property superpower which which I and everyone else on Prometheus team like We have the curse of the expert We know how the thing is supposed to work So we don't hit those walls like we we can deliberately find other walls to hit and corner cases and such But as we already have a mental model of how that thing should be working We tend to not hit those walls and we tend to not think about all the things which someone who's new to the space might be thinking of right I've For over two decades. I've always begged people if they come new Make notes of what is hard and then deliver those notes ideally already with a pr, but even just the list So super useful and if it seems basic, it's still super useful of course Things invisible to us Yeah, they do. Um, it's it's a huge thing. It's why whenever I'm building like a workshop or writing a new tutorial um, I for if it's if it's geared towards beginners the first thing I do is um, Borrow some students from a local boot camp and I give the workshop or the tutorial to them And take note of like where they run into problems like where I forgot to explain something because I'm an expert I I Just forget to that not everybody has context and this is a thing that's inherent in human nature. It's not like It's it's hard to avoid. It's something we all do. Um, they did follow up Uh a like a GUI configuration that converts to code think of the abstractions currently built for kubernetes like tilt okay, um Yes and no, um There are bits and pieces or there are large pieces in in on the grafana side Um, there are very certain things can can be done in UI. Um, I think quite a few of those are paid closed source features But I don't actually know. Um, I don't tend to use them. I I prefer the other way. So, um, I don't really know Um Yeah, it depends on what you want to configure because usually if you want a guided user interface that means you are not very strong on automation and That might be enticing when you get started, but it's almost always the wrong choice long term So I would suggest Trying to just write those things out as snippets in in files as you go and literally write them by hand So you can deploy them automatically and then as you go migrate them to to actually fully automating And that might mean that you realize you don't have a cmdb or you don't know where this and that service List is living or it's not up to date or what have you But those kinds of things are actually a positive Of course, if you if you consider observability not as just alerting But as actually being able to understand that complete thing, which is your Whatever system, um Those are required anyway And the fact that you don't have them and you would like to have an easy way to to basically plaster over not having them is understandable and I I totally get why you want this But you're usually Basically just avoiding doing work, which you should be doing anyway or someone else should be doing anyway Um, of course the overall system will become more automated which actually frees you up from doing that Toilet you can invest in more tooling and more automation over time. Um, which is basically what premises is designed for It's it's designed to be fully automated So if somebody is trying to get started and they're they're struggling Um, because they're not very strong on automation or they're just they're just very very new in general Can they go through the prometheus community for help? Yes. Yeah We have we have like, um, I don't know if you if you showed that that overview. Um There you go. We have slack. We have. Yeah, perfect. Thank you. Uh, we have mailing lists We have matrix. We have slack. We have irc. We have arguably too many things. Uh, we have those github discussions um We have discourse So there's more than enough ways to uh to get in touch. We also have office hours Which are in different time zones or happen at different times. So people from different time zones can can attend Um, where we just walk through whatever someone might be having trouble with There's tons and tons and tons of ways to to get this feedback and or this this help And the nice thing is we have a lot of of people who who also help out. It's not just us um I mean ideally once you learn it stick around help others pay it forward, but even if not, um Getting contact and and tell us what you have issues issues with Yeah, make note make a note every time you run into a wall I do this every time i'm like working with a new product in general like anytime i'm using a new open source tool Uh, when i'm first trying to like stand it up or run through the the tutorials the quick start guides whatever I make a note of everywhere I run into a problem or everywhere I think that something wasn't clear enough in the documentation or sometimes the documentation is just straight up wrong That that happens less frequently, but it does happen with some tools I make a note of it and then I either contribute the changes back or if I know the maintainer I will like reach out to them personally and hand it over as an assist It's it really is appreciated Stuff like that. So If you're new and you're running into trouble like open source project maintainers do want their tools to be easy to use Like we we we want that It's not always achievable, but we want it. So, you know Tell us when things are a problem. Um We are running up on time. Is there are there any closing words other than, uh Register for prom con follow cloud native tv on twitch follow you on twitter No, be excellent to each other and hopefully see you in person next year. Hey, uh, I hope so. Um cube con the next cube con eu is in, um Valencia spain, yes, right So, uh, hopefully I'll see you in valencia. I'll be I'll be in valencia. Will you be in valencia? Maybe if covet allows it I I didn't miss a single Cube con no, I missed I missed two chinese one other than that. I don't think I missed a single one. Um Other than covet reasons Yeah, well Hopefully things are a little bit more controlled by then and we'll all feel safer Going to conferences. I'm still pretty. Um Questionable about it, but I will be cube con na. So, um Y'all I'll see you in a In a week Um virtually and live you I will be seeing virtually not live but um, thank you so much for coming on here and answering my questions and talking about prometheus and prom con and monitoring and um shouting out the community and Y'all uh contribute to open source Thanks for having me support the tools you use. Um, yeah And get vaccinated and get vaccinated. Please get vaccinated Please get vaccinated Dude seriously if someone is uncertain about the underlying science and such while i'm not an expert I will if you're honest about it. I will take the time and and Try and answer questions if yeah Good All right, well everyone Thanks, everyone. Thanks. If you did get vaccinated. Thank you Uh, if you didn't consider it, please please do it Um Thanks for everything. Thanks for talking to me. You've been fantastic. I hope prom con is a huge success And I hope your announcements and your and your talks go well. It's a lot of talks you got to give um Twitch thanks for thanks for watching and I guess twitter i'm i'm assuming a bunch of y'all came from twitter too Um, any closing words for you? No, the excellent to each other Bye everybody