 This on ah, that's better. Okay. Hi morning This I expected like six people So thank you all for coming Do we yeah, okay? so I Got a lot to get through The slides are on the the sked app So if I skip over something you can download and look things up Okay, who who who programs in go? Oh At least half. Okay, that's pretty good. There's gonna be there's gonna be some go code Who likes Yammel? Wow, okay, so I I work at Grafana labs for the first year and That's that's not particularly relevant to the talk But they do pay me More importantly, you should follow me on Twitter So I am gonna talk about adding a command to kubectl The command is called kubectl events or right now kubectl alpha events So start with How did I get here? I I basically have no idea or I had no idea how to add a command to kubectl Nor did I ever think I would But I was working on this thing caseband Who's seen that a? Couple of people so it's pretty cool. I won't talk about it But go go look it up. It it it gives you a graphical view of events of your Kubernetes system So let's just clarify what it what do I mean, what am I talking about with events? Can I do lasers? Yeah, not really. I need a like a more powerful laser So Events are little little messages emitted by different parts of your Kubernetes system maybe from your own code or maybe from part of the system like kubelet or the scheduler or Really any part of the system and they they go into the API server and And Let's take a look this this is the sort of typical user experience of events You run kubectl get events and you get You get some events out you Each one has a type which is either normal or warning Each one has a short form reason It has an object here that it relates to and a sort of human readable message So Okay, so far Okay, we've seen this right, okay, so I just want to adjust this slightly There we go So this is what an event looks like if you were to print one out in yaml And it's kind of kind of the same stuff, right? We got them. We got the message we got the reason We got a few more things like where did it come from? And the events are Kubernetes objects So in the sense that they are the same thing to the API server as a pod or a deployment or PV or you know any any of the other Objects within the kubernetes system, so they have metadata. They have a name. They have a namespace a kind and a version So That's gonna be important later So I Became aware when when working on caseband that there are some issues with With events particularly with kubectl get events and The the most important one I think for most people is that they don't come out in the right order so if I just go back to I Don't know. I don't know if anyone noticed this but they They're print out and like latest first would be the obvious way to do this, right? And they they don't they they don't come out in the right order In fact they come out in order by name Which is like who even who even knows the name of an event so So there are a few more issues with events which which we'll maybe talk about in a minute But I also want to note that that that issue was opened in 2016 So So I like a challenge An issue that has been open for I Guess five years at the time I was working on it That's you know, that should be easy right just Knock that off fix it Yeah, there were there were some comments on the On the issues as well That lots of other people thought it could be fixed in a in an afternoon probably So Never mind Yeah, well, so Why is this why is this a hard problem? Why is it not just a question of sorting the events? And the answer is There are two kinds of event Who knew that? Yeah, like nobody I mean You just came to heck all right Three if you count the beta for the the second one So the They have a kind like like I mentioned and an API version so the API version of the the first one is is v1 And it lives in the core library And I've linked all the code if you download the PDF. These are all clickable links If you if you want to kind of look in the details and Then this new one was created and we didn't want such a boring name as v1 So we called it events dot kates dot IO slash v1 I guess and So this was done so long ago And that the the cap for it is is kind of mostly about the cleanup like I yeah, we we're sort of halfway through this We better finish So it's a little bit a little bit kind of half half Like 90 percent done 90 percent to go There almost everything if it emits v1 event The scheduler is one component that definitely event emits the other one But it will it will not do that and it's there's a race condition on startup So I was kind of trying to do a demo for this and if you use kind It doesn't exhibit the problem unless you restart the scheduler. So So I'm not going to demo that Yeah, you can read the code take my word for it Let's just dive into the the code a little bit this this is the code where the events are created Which I thought made it easier to see What's going on so So the first kind has Involved object it has a first time stamp and a last time stamp So if events are coalesced if you have the same bit of software emitting the same event over and over again They they can get coalesced so so the server only stores one of them So we have a we have a first time stamp a last time stamp on an account And then Quick change So in the new one we have an event time and If there's more than one we have a we have a series data structure Which which is expresses this concept that it happens several times in a series They renamed the involved object to regarding Other things like the reason is still called reason the message is now called note So like it's a rename a thon Now Kubernetes is very big on backwards compatibility So the good news if you like is is that there is magic inside the API server that converts Won't each of these to the other one and either way you can ask for either of these and you'll get them and Even better all of the fields from the new one are also in the old one They were added so so like all of the fields all of the choices are there you can ask for event time on On the old kind of event It is null, but you it's there you can ask for it And in fact that is that is why they don't sort in the right order Because because anything emitting the new kind of event Has has a null in in the last time stamp And this used to crash kubectl So that was even more fun, but I think they fixed the crashing about a couple of years ago So the ones with null just sort at the top, okay So how do we fix this? So kubernetes is obviously very very big project and You can't just you can't just go around changing stuff. You can fix stuff. You can fix bugs without You know asking someone's permission But fundamentally changing the behavior of something needs the kubernetes enhancement process It kind of sounds grand, but but basically someone has to propose something and write a document and then it gets accepted by a SIG the the special interest groups that that's who really governs kubernetes So in the case of kubectl, it's six CLI You know the other ones are like SIG network SIG API machinery also all different SIGs you can you can look them up and And Yeah, so so the happy path it gets implemented It can also get deferred or rejected or whatever, but we won't talk about that So I found that there was already a cap approved To fix kubectl get events and So this was extremely lucky because I don't think I would have had the patience to get a get a new cap approved So it was Started the cap came into existence October 2019 Was marked as implementable. That's the kind of approved phase three months later And then time passes I own my PR February 2021 It took six months to get the PR merged And then a couple more months to get released what's it's alpha right now. It was released in 1.23. So you can use this now I didn't bring enough swag for everyone, but you you can have a command Okay, so let's talk about the command So the the the the really cool thing is the Time stamps come out in the right order That's it really okay, let's let's go into a bit of detail Okay, yeah, so so you have some options. So if you if you just run kubectl alpha events it'll it'll print them all for The current the default namespace So you can ask for all namespaces or a particular namespace like dash dash namespace I just sorry just I see some people standing at the door. There are some seats right at the front I won't I won't like insult you or anything if you want to sit near the front And and then you can do you can do events for one object and you can watch for new events So the the other thing that people complain a lot about kubectl get events is that if you if you watch if you sort of get them to come out as They happen They all they all come out as they happen and then they all print out again one hour one hour later All the same events one hour later Who knows why that is? Yeah, nobody right it's because they get deleted events events only live for one hour and kubectl get minus watch Prints out if things being created and things being deleted. Why not? Anyway, so so my one skips the deleted event So the extra value double double value. Okay Okay, so yes, right now. This is this is the this is the talk. Well, we'll start Okay, so suppose you want to add a command to kubectl What are you gonna do you are going to Create a command object. So kubectl uses the Cobra package for CLI applications and So this is the the kind of what I called step zero like you're not gonna go anywhere unless you create this command object So What I did is I I looked at the Existing commands like you like kubectl get for instance or kubectl describe or whatever you I guess that's that's kind of the Booster to get going is to find a command that does Something a little bit similar to what you want to do and copy paste that So you're gonna need your command object So the next thing if every command there's a kind of a pattern where The the work is split into two phases. So there's a phase of collecting Command line arguments And they get put in a struct basically vanilla And then the next phase we kind of decode the command line arguments and we figure out what to do with them and The reason it's split into those two parts is is for testing so that we can kind of independently test the act of of receiving CLI parameters separate from the act of actually doing some work on them and Make that easier to test so So I kind of highlighted So if you have a parameter like namespace Well, that actually goes into This guy gets kind of pre-populated So that when you talk this the rest rest is like that the API Client to the API server and and that will be populated by the machinery of kubectl to Give you a client that is talking to the right namespace And then so this one is one that I added on my command And it's a string, you know that so this that's what I mean that the first phase Is just very very plain. We're just fetching the arguments and getting them into a struct so in the in the next phase So same kind of thing why it turns out I need to know the namespace So I pulled that out from the rest client getter and then the So this thing here which object we're trying to fetch events for Gets decoded into something more complicated And we'll talk about that in a minute So and these two things are the pattern is that the first one is called flags and the second one's called options So this is a pattern within kubectl That all the existing commands follow So probably the next thing we're going to do is make a API server request and There is a really cool builder Library to To do all kinds of things with API server requests so this one is From kubectl describe because this is not actually how kubectl events works, but I thought You know this is this is the kind of the powerful library that will work for many many cases so it's It's this kind of pattern where each Piece kind of leads on from the next. It's called like a fluent API that that's the word that some people use for this kind of thing So we have a builder and we're gonna fetch unstructured data. So not go objects. Just just name value stuff And we're in a namespace Maybe all namespaces where we're selecting on some labels We're gonna flatten them. So if we get a list of things we don't want the list. We want all the objects So this is very very comprehensive and very powerful and It also covers things like like chunking. So when you When you talk to the API server and say give me all the events there might be a million of them And we do not want to serialize a million events into one buffer and send that over the network So in fact we fetch them like 500 at a time chunking or paging and all of that is Is covered by the machinery inside kubectl. So that's I Guess is one of the most important things. I wanted to mention here that there are these libraries and When I was working on this I Didn't know anything about them and it's also that I didn't find any kind of tutorials I did find the existing code, but I had to kind of work backwards and figure out what it did So this stuff really powerful really really useful now why Why could I not use it? So if you remember The event And we're gonna look we're gonna look for all Events relating to one object So we're gonna build a query that is a little bit different to the ones that the builder knows how to do and Even worse I Don't know who who spots the difficulty here. It's very very subtle. So Yeah So in my in my example here pod has a small letter Because that's how we write pod when we're just typing really fast and The thing in the database Has a capital letter. How are we gonna get past this? So so now we need to get into the concept of API resources and kind versus resource So let's let's try and do that at a high level So you can get a kind of a hint of this if you run the command kubectl API resources So this queries the API server brings down a ton of metadata and prints out this table Let me move my mouse. It's not getting in everyone's way Yeah, so like so here here's why we say pod And in fact it it accepts the singular even though the thing that's in its table is that is the plural There's an abbreviation so you can save one more character if you like you can type Pope Which is a bit more useful for something like config map you can type cm It knows the version it knows whether it's namespaced and this Is the thing that we need to query for in the in the database? So this is the kind Which is the the sort of official Thing that it's filed under And this is the name. This is what we get to type. I don't know I don't make the rules You know this it works, right? It's you know trust me But you do have to go through this You do have to go through this this Hoop So we need to go from a group version resource Which is the thing with the lowercase to a group version kind? So group is like API or apps or something like that version is like v1 v1 beta one And resource is the one with the lowercase and kind is the one with the uppercase Anyway, so there is a Yeah, so we indirect through the metadata which is in The API server and we we end up with the The kind So we have the kind and the name which is like pod slash name and then there's this other cool Part of the machinery where we we get to just say I'm gonna and these two things together So this equals this and this equals this so I can build up my query on The fields of the event object And That's what I'm gonna send to the events or API server Yeah, so so in order to get the kind So this is what this is the way I ask the API server for all the metadata I ask for a thing called a rest mapper. I fill in what I've got So this is a struct that can take a group a version and a resource But I I only need to fill in the the bit. I've got which is like the word pod in lowercase So I fill in the bit. I've got and I ask it to map it So this can be pod or pull or you know config map or cm or what whatever you've got you fill that in here You ask it to map it and then you extract the kind from that So that's how we go from what I type in to what I need to query on So this is this is really powerful if if you're doing this stuff And it just won't work if you don't do this. So it's important I Skipped error handling Because that's what you do in a talk Okay, and So last thing we need to do is we need to execute my query So I mentioned before the idea of chunking or paging the Things that come back. So that that's what this guy does follow continue what we'll get 500 and then another 500 then another 500 and it takes a callback or a Continuation or whatever you want to call it and we can extract my objects from here and deal with them so So this is kind of the guts all in in cubicle events Well, it turns out what the main thing we're trying to do is sort them So I actually just add them to a slice in in go I add them to the end of a big a big array of events and then sort them but So that's That's what you have to do You have to build your Cobra object Get the flags from the canline you have to Decode them into options and then you have to do your thing. Okay so Let's see Let's see how we did So these are the main issues that were Listed in the in the cap this is the reason why we need to change it So number one they don't come out in the right order. Yes Number two they print out one time and then again an hour later fix that Number three also so cubicle describe will also print events But it prints them out for exactly one object So if you want to if you want to kind of watch all the events on one deployment or one pod or something like that And that changes at all then the event stop why you just can't watch on describe. I think anyway So my one is less specific. So so I'm gonna call that one done Combining columns well this this was about the some of them have last timestamp and some of them have event time So I just do that in go code So we'll call that done And So there's there's one more which which I'm gonna say is not done a Timeline of events. So so if you read the Narrative on this issue. It's it's kind of saying if I ask for events it should kind of print out the interesting ones You know it should tell me cool stuff that happened on my cluster recently and So I haven't implemented that I'm not even sure how to implement it Okay, yeah, what's next what? What might we do? For this command it is in alpha so definitely looking for feedback and looking for Thoughts, you know bugs in it or ways to improve it or whatever So one one idea would would be to add a thing to limit how far back like, you know Maybe you just want to see events in the last five minutes or something like that. So that that would be That that's I think kind of related to the one about the time I want to see a timeline of interesting events So just like the last five minutes or the last one minute or something like that might be might be more useful This one related objects is is one that I That I like So what I mean by this is if some if I've got like a deployment and something is going wrong with it It could be going wrong on the deployment It could be going wrong on the replica set which the deployment creates it could be going wrong on a pod and they're all kind of related But you have to be a sort of Kubernetes super being to know where to look for this information. So So maybe the maybe the thing could just know that itself and Do that Yeah, and lastly, you know what it what do you want what I put it out there for the world Please file issues or I don't know send me tweets send tweet Anyway, that's the end of my talk. I'll take questions in a minute Thank you, Ryan Before everyone rush for lunch We have Q&A and for those of you who are going about the leaf, please be quiet for the rest of the Session anyone questions All right. Thank you very much for the talk was awesome Quick one will the the new events replace the old one At some time. Is that the idea? I think yeah Well, so there there's a cap for new events to replace old ones that the new events are more flexible So, yes, that's that's Yeah, maybe I should have researched that I guess that's one for SIG API machinery or something like that. I mean the certainly certainly that's the stated intention is that the Kate's dot I owe slash events should replace V1 events But in a backwards compatible way, right? So so you can always query for the for the previous one the same question But on get events, so you introduce something new an alpha We don't want again Two options, right? So when is the point in time that the get events is going to deprecated and we will have only Coup city events. Yeah, now that's that's a really good question. So So one of the reasons why I couldn't just hack on get events is is it is implemented in a very very generic way So when you say Qubectl get I'm kind of switching between Qubectl Qubectl Qubectl I don't know. Anyway, if when you do get Almost everything happens in the API server Qubectl does not know about the thing you're about to to print out And and that's really really flexible. That means you can use it on things like CRDs That that were not known about when when Qubectl was written But it makes it incredibly hard to do something like like Pick two one of two different timestamps depending on which type of object it is So My opinion is that that Qubectl get events will always do the dumb thing it does today That the Qubectl get events will will never be deprecated will never be enhanced Unfortunately because of a technical detail of the fact that it's implemented in the API server and And it's just it's just a kind of raw. It's a breaking of the layers to put that intelligence of that layer Hello. Hello. Does this command in his current state support wild cards or regex matching? No Let me think about that. So some of these things work on a prefix and I Actually don't remember whether that works for Qubectl get events, but I Mean I showed you the code it says equal So right now it doesn't have a regex matching, but file file an enhancement request on Github slash Kubernetes slash Qubectl Thank you for the top So you just mentioned that you did not find any tutorial or something to fix that you have to dig the code So what is your advice for our newbies if you want to implement or fix anything? Oh? Well to some extent that's why I did this talk I hope I gave you some pointers and and definitely go to the the notes for the talk. There's a lot of links there So if you're a total newcomer I would advise a couple of things Tests are great great pissed to start, you know You can you can look at some tests and look at the coverage and expand it a little bit And and people are gonna love you for it unit tests particularly If you're Wanting a more of a challenge look at the flaky tests and and stop them flaking or you know pick one and Figure out why it's flaking and and fix that Next thing you can listen into the SIG meetings or you can use find them on YouTube and and like follow along I Visit one or two You don't have to say anything you can just show up and and kind of listen to how it goes and get an idea And then and then you can you can show up you can say hi. I you know do you need help with something? So that's my top three tips for getting started as a newbie so you had to implement the parsing of the kind and object name in the Tube cattle command isn't Soon that will be already implemented in a more generic way because we have to cut to create a delete and get Doesn't it exist there? Yeah Yeah, so I think I agree with the spirit of what you're saying that the I mean I I tried to demonstrate there's some really powerful libraries in there, but still not exactly the thing I wanted and in some case like The thing that that specific thing that you mentioned about how to decode the name that is in cubicle describe but it's not exported and After a little bit of to and fro with the maintainers we agreed that I would Reimplement it so there's a kind of a go pro verb a little copying is better than a little dependency So it's kind of better just to copy paste it once and then maybe over time we can figure out It should be a reusable library Anyway, I think I agree with you in spirit, but it is what it is All right. Thank you everyone. Thanks Brian. Have a nice lunch. Thank you