 Hello friends and enemies, I was always told to open with that because people have to figure out who's who, it's kind of fun Congratulations, you found this room. I know it's quite the trek and in way, I yeah, I was a little lost myself Hopefully I'm really hoping this is gonna be the funnest talk you go to I was like there's a lot of information and that's Flashing is gonna be great But I really just hope to 101 intro everyone gets a better idea of how Kubernetes works fundamentals that would be bad. I kind of need that one. Oh Ha, that's because I'm not plugged in But I'll just have to keep doing that Yeah, I really hope that this is gonna be just a fun talk We we are going to build Kubernetes up here and and not build like compile like you'll see Functionally what kubernetes does when you do things? So that's me I'm on Twitter a lot if you want to DM me of DMs open or ping me and you feedback. I Am also a CNCF ambassador Nobody really knows what that means But I'm in the group of ambassadors that kind of promote cloud native computing foundations tools It's being one of them and a lot of other ones So we're here we do there's a decent-sized group. We do meet-ups all over the place But it's just kind of a fun thing to do. I also wrote a book with Chris Nova about cloud native infrastructure So if you haven't picked that up, they were giving away yesterday a little bit See and I booked that info has all the links if you actually want to go get one Really appreciating feedback on that too. That was a fun project to do this year And then I also I believe I'm probably the only speaker at the conference that has an IMDB page Which is super exciting But I'm not here representing my employer anything I'm here To help everyone just figure out what this kubernetes thing is. I mean, it's okay if you've been running it for a long time We're gonna go over how it fails just some other things in it I actually was partially inspired by the quote that Sarah had in the keynote today from Benjamin Franklin where it was tell me and I will forget teach me and I may remember and Involve me and I will learn and so that's really kind of like getting people involved in how this stuff works and Visually seen and you can look at diagrams all day, but seeing how where the failure modes aren't what actually happens I think is really powerful. And so to do that Everyone know what LARPing is? Yeah, so it's live-action role-playing and basically You dress up you go somewhere and you act out something whether it's Civil War re-enactments or corgis Doing the great battle of corgopolis whatever whenever that took place And so we're gonna we're gonna do that here live on stage. I already have some some plants in the audience Thank you for volunteering And then this is my last slide everything else is gonna be up here But this is just kind of a really basic overview of some of the components. We're gonna go over and some of it But we're not gonna go over Mainly the control plane is is The back end of Kubernetes what really involved what Kubernetes deals with to make stuff happen inside the cluster And then we're also gonna have nodes are super important anyone remember when nodes used to be called minions I used to love that. I don't know why just minions sounded so cool And the things we're not gonna go over are those add-ons there Those are all kind of implementation details a lot of times It's some of them like DNS is a critical piece of the infrastructure But a that's really hard to represent us with a person and be you can kind of switch it out You don't have to use the internal Kubernetes DNS to kind of solve some of these problems Almost every cluster will bring one for you. But again, you don't have to so with that I'm gonna go ahead and Show you how we're gonna do this actually wait. Sorry. Let me skip back Vic you want come up? So Vic has graciously volunteered to be one of my cubelets And and the cubelet is the process that runs on a node. We're not treating nodes as cattle today There's no it is Texas, but there's no guns allowed. We're gonna take care of this. We like Vic So Vic's job as a cubelet is to run pods and that is his main job He needs to whenever someone tells him here's a pod Vic makes sure that that's the thing that he runs and he makes sure that always runs And that's it he does other things when there's a cluster But as a standalone Cubelet like you can take a server and run the cubelet on it and you're fine. You can just okay. Here you go Here's your here's your stuff. I mean, it's it's not a lot different than just running docker Docker makes sure that the containers that it runs are always running So in this case with with Vic we're actually starting off with We have what we're gonna represent as pods and so I told you it's visual come on. This is gonna be good Let me tell you having it. Oh, no, it has a hole. Wait now. We're good Having a bag full of balloons like TSA was just So anyone know what what we're doing here, what is this app? Python anyone come on went through a lot of effort to get that There we go. Thank you. It's like their color. Oh, there is a hole. See. I'm not that good No, wait, I'm gonna tie another knot here or you know what I have a bag so it doesn't matter So we're good But this isn't just any Python app Yeah, I think we're good. So this is actually gonna be super special because anyone use Django and for some reason I always just imagine that Django was like a dog. I don't know why it just seemed like a good dog name So we're actually gonna have Vic Run Django and keep it running for as long as he can and if something happens to it It's your job to so the cubelet again, you keep it running. Whatever it takes in this case. This is gonna be a static pod Static pods are just defined My skills doing this while talking. We're not super practice. Just just FYI. It's like typing but on a whole nother level Wait Okay, his legs are upside down. There we go. There's the dock. There's Django. I was neck pop, but that's all right And actually Django 2.0 came out so special little So as a static pod You can define these on the node as just YAML on disk the cubelet comes up He reads from disk and says oh look I have a pod to run I'm gonna run that thing so he fetches it from the container images from the registry Which is going to be my bag here full of balloons For when other people come up So they can get their containers out of the container registry. That's even Docker branded so But it can be whatever you want you can tell it what registry to get the images from and he just runs that container and and that's his job Right, we just built Kubernetes right. This is it Kind of it's not really a cluster right like this is a server doing this thing This is no different than when people used to go people still go and Docker runs something on a box and it runs, right? It's like okay. That's not very special So the complexity comes when we now want To Vix right we want to servers to run containers and we're gonna not manually do this stuff And it's not going to be things on disk It's going to be we're going to dynamically put these pods everywhere. We're gonna do stuff with it. So that's where It's ED comes in And to represent it's ED We're gonna use a database in the cloud Right, that's it. Like that's your thing. It's a database running in the cloud right there So come up along The lawn is also going to be one of my notes He has graciously volunteered so what we first need to do here actually I need to go my notes to make sure I do this Because I forget things so much There we go. So just to make sure so we're gonna keep a database of typing Node in order to identify these nodes. We don't want to just do host names. That's not that's not really special It's not always unique. So we're actually going to assign labels It's how pretty much we do everything inside of Kubernetes. There's labels everywhere. If you don't like labels Probably use something else And then we're just going to track what pods are on there We'll pluralize it because we might want more than one pod later on so first off we have Vic and we're in the label of How about hoodie oh wait, I got and Beard beard I like it. Okay, and he's running our Shango pod right so along here He is Can you tell he's one of the new Amazon metal? Servers, so he's he's a metal server. We're just gonna say that Or bare metal whatever you want. He's not running any pods right now That's fine. He you know, we haven't scheduled anything. He didn't have any static pods. We're fine So Add a divider here. So it's a little easier to track We also need to inside this database. We need to track our pods These are the things that obviously we care about that. We're running and And last thing we need to check is where they're where they're actually running So pods also have labels and we track what nodes they run on in this case the static pod Never actually made it into this database because it was just a thing on disk that we didn't really care about So if we want to if we have these nodes now, we have a database. How did they know? How to talk to this database? How do they get their information? How do they read what I should do? Any API server they they're not talk directly don't hard code your stuff back to the database, right? Don't do that. So we have an API server that sits in between. I'm gonna play the API server in this in this place I am the only one that's allowed to write to the database They're not allowed to no other component of Kubernetes can write to the database only the API servers And so anything that they need it they they read through me and then they they can write back any component Unfortunately, you can't really see the screen, but hopefully we'll fill you in on some of this So let's go ahead and run another application Someone give me a language Shower's coming up. I'm gonna do Ruby. All right. Let's just let's just do Ruby just because I have red balloons So Ruby as a label read now what happens Nothing that they're not that they need to look at the database and know hey what pod should I run? But how do I get a pod assigned to a node? I? Need a sketch and your scheduler. I need something that actually tells me. Oh that goes here and so Brad you come up. So I'm gonna add another divider over here actually here Cubelets on the sites if you can't stand right over here. Just so keep it visually separated Yeah So go ahead and pick where should this ruby pod run? It can be one of two Okay, so go ahead and get in there's a red balloon in there go ahead and blow that up and make sure it's running and so oh API server is gonna say Hold on and and Say hey, what's assigned to me? What's assigned to me if there's nothing they do nothing if there's something then they say They look at what they're currently doing and what's assigned and they say hey is that right and they reconcile They say oh I need to do something or I don't and that's it and that's really like the function of the cubelet There are other things that they'll do like they'll have ping checks They'll kind of notify through the API server to say what their health is say what features They have there are a lot of things like that, but in this case, we're we're gonna go a little simple So again who's actually creating pods inside the cluster They don't actually like like I want this pod It's not a really good thing to do inside of Kubernetes because it doesn't if you want to scale it You got to like you manually do all these things so when you actually want to make something that dynamically scales you typically make a deployment and What deployments do under the hood is they make what's called a replica set replica sets You sell it how many it wants and then that creates more of those pods and it's it's super awesome So we're gonna add another row in this in this database of course replica sets have labels And then We tell it how many we want so I heard I heard Java before we're gonna go Java labels are big and blue How many replicas I Just I'm just gonna do one because I won't kill anyone yet So we now have someone told the API server. Hey, this is what we want running. This should always be running in this cluster But now what happens? Nothing. We we don't have anything that creates those pods. We just I stored this object as the API server It's in the database, but Chris go ahead and come up and you'll be my Controller manager you can stand there stand here. It's either way So the controller manager does a lot of unique things in the cluster. It's actually like four or five different things It's hard to name things one of the things the controller manager does is looks at replicas or replica sets And this figures out how many were desired and how many are running So he kind of keeps track of what the Actually, I forgot you're running that Ruby container now. So are the Ruby pods. So there you go. I almost forgot. Good job So he needs to actually create replicas in the cluster Don't worry about it Because we have we have he can look at how many Java Replicas we wanted and how many pods are actually used those same labels and then figure out like oh, hey I need to reconcile that state. He does nothing else. He creates a replicas and then scheduler over here We'll then assign those so get go ahead. We're gonna say Java Give me a name. What was randomly in any things? Oh Oaf like that. All right, that's cool So this is big and blue and then Scheduler which which node do you think can handle a big blue Java app? I'm gonna say metal again. All right. All right So it's now scheduled till on all the way at the bottom of the bag. You'll find them No very bottom Can't miss it So while he inflates this Java app You have to make sure they're both running And keep going keep going it's Java, you know So Replication controller created these replicas also responsible for Helping out when nodes come and go inside the cluster. So So when a node comes up Ian, you want? When a node comes Rep or controller manager is now responsible for kind of making sure that this this state of how many nodes we have and how Many are actually declared in the database matches again reconcile state Just figure out what's different and see what was desired and what's different Ian now came up. He's a another cubelet in the cluster. And so we're gonna go and add him Yes, well, we'll just give him the the Kate's Label there and he has no pods. That's fine. We don't we're not gonna automatically rebalance We're not gonna do anything else. It's just another cubelet in Wait, wait, hold on So Controller manager is then does these you know as these labels to the nodes make sure if you're in a cloud environment You also have these these cloud controllers that'll look at like your cloud AP if it's AWS You're gonna look at the AWS API Make sure that that node still exists and if not delete it out of the cluster because you get in this weird stale state Frequently if nodes are coming going if you're in an auto scaling group Things change and you need to be able to reconcile that state just for fun another name Not oh You're a controller manager. You need to tell me what you're creating these replicas Jenny Jenny Okay, so we still got big blue same labels and scheduler Obviously, okay, we're gonna go to Ian It's really hard with two So go ahead your turn you get Java. It's a big blue one. So yeah, we're just going through the cycle The other thing that the controller manager is responsible for whenever there's a namespace in kubernetes There are default Authentication and tokens don't really go over that here But it's really just how what pods can talk to the API server gives access to other things inside the cluster Controller manager is responsible for that. The other thing we also want to do is you're gonna create services services are how you kind of route because pods come and go so you don't really so service you're not going to Route to a pod because it can easily what's up? What's that? Oh, you're right. Thank you controller manager. What's up? Yeah, so that's that's a good point. Let's say nodes came up Something got out of sync on the API and you say what what happened like what where's the failure points? Okay, like all sudden though if I look at the cluster I have this many but something else is wrong You can know who to blame right you go look at the logs of the controller manager And it's easy to tell like at that point. It's not the scheduler because he's not doing that. It's not one of the cubelets It's not me Who's the blame right? They they were supposed to tell me so if there was a Reconnecting oh shoot Okay, good point what happens if it CD goes away what happens has anyone ever had that happen I did it Yeah, yeah, it's amazing. You know what happens nothing They keep running they know what they're supposed to do they had to do nothing else. It's fine. No one can talk to me I can't talk to the database. You know what I'm gonna return 404. I don't I don't know 500 errors like I can't do it right now That's okay. It's CD can go away. You can't change anything the cluster will stay up. I had it down for days It was amazing and and like no one knew no one no one deployed anything. It was it was amazing I See these back up. It's awesome. So you're right. So we need we need Java Oaf is over here and Java Jenny is here. So now back to our services, of course They also have labels Trying to connect. All right. That's that's fine. It's he's gonna take a little break and back with us shortly so In the meantime I Killed right anyone. It's like Who knew that could happen, you know, what's the cubelets job? Cubelets job Run it again, right? Go fetch that image if it is That's all you got to do as a cubit, you know, luckily I think I only have four of those so But that's it all the cubelets just gonna go back in and and run this thing because that's the responsibility of the cubelet and Make sure what's assigned to you is what you're doing if Etsy D. He doesn't come back. We're gonna have a hard time Wait, one thing I will say the Wi-Fi here has been very successful for me overall When you get the like above the snow line in this room like maybe not so much But anyway, so labels again have our services have labels One other thing that services have is they actually have what they want to match because late the service itself has a label That's how things like ingress rules and other things will match is they'll find a service which is a stable IP address But then we need to know what labels the service itself wants to match and What pods that is actually running on so we're gonna say let's say we have a Ruby service All right labels for this don't have to be the same thing as what's in the pod so we could say gem and It's gonna match the label red So we know controller manager again is responsible for this. He looks at the cluster He figures out which pods have those labels that we're trying to match and then he tells the API server Oh everything with this this these are the pods that match that label So which which pot is it? Maybe we only have one I'm sorry It's only Ruby at this point. Yeah, because we didn't create that through a replica set that was just Every time I just like glance at the Wi-Fi then it comes back. It's amazing. I thought I had this off I guess I should be offline. I should have done that. I'm sorry. We're gonna wait again at CDs down It's fine. We're totally cool. We can't create a service. That's okay. Everything else is still running Where's it going next? How about that Wi-Fi? I was just you know praising it and then this happens. It's cool We're gonna wait for that or we're just gonna move on Yes, if at CDs down and that's what happened before. Yeah, the the kubelet already knows This is my definition. He can't get a new definition You can't change what he knows But he'll look at the API server and say like well I'm just gonna keep running what I'm doing if I then start returning 500s to all these things then They'll just say look how much I can change. I'm fine Like I know what I had to run last time I knew about it And I'm just gonna reconcile that state and make sure I'm running what I'm supposed to Yeah, okay, let's do it. Vic go ahead and have a seat If you lose the node, yeah, you can't change the state of the cluster. So yeah, that is a problem Go ahead. It's just on vacation for a little bit, but so what happens because Django is out of the cluster, right? but Django wasn't a It wasn't actually in the in part of at CD because it was a static pod. So that's okay So we're not gonna rebalance anything. We're not gonna change where they are running. It's just that's fine vixen vacation We don't really care about Django in this place in this case. It's alright. Sorry. We care about you. It's not that I don't know if I can take this. Can I like switch it to offline? I know We're gonna keep going one more question. Yeah So yeah a pod is containers inside of it. So Depends depends on the implementation detail your your container runtime interface Determines that in a lot of cases defaults Docker if it's cryo, it's gonna run a different command if it's rocket whatever it is the that interface the container runtime interface tells the cuba how to interact and how to do something with this pod and so if you notice on if you're running Docker you have all these pause containers because Docker doesn't actually have a default way to make a Group of containers in one thing. So they have this pause container and they layer all the containers on top of it if you're using something like rocket or Cryo they do that by default. You're not gonna have Pause container because it just it can do that already So it looks a little bit different depending on the implementation play that comes back No, that's wrong All right, we're taking it CD offline. We're not running the cloud anymore. That was databases in the cloud. Maybe not so much So I have Excel installed. I think I do We're gonna leave it right now Defecated nodes. Yes, don't run in the cluster. So another another thought is how do you how do you have highly available? API server, what do you do? Right, so we're calling this CD Clusters over Wi-Fi bad idea So Second API server, that's all you have to do create another one. I'm just gonna Kelsey doesn't even know I'm doing this I'm gonna share with Kelsey. He's gonna be my other API server. All right, and if I share with him Any any one of these components can now talk directly to Kelsey He has right access to this database still no one else does but that's all you do and then you load balance your API servers I don't need to know Kelsey state if he's down. That's fine People people will fail when they talk to him and and succeed for me. That's totally fine Same thing with your actual components. How do you load balance a API server? Sorry the scheduler and controller manager It it's slightly different. You can add one Right in those cases, they can't they can't directly NCT is gonna handle locks for them, but they can actually figure out master election in at CD So they can say hey, I'm the leader right now Everyone else you can only read and so if you have one or five It'll get a lock in at CD and say hey, I'm the master scheduler right now No one else can schedule anything and that's fine because you only want one thing doing that scheduling You can have multiple schedulers you can assign those things with annotations and and run these things but per Schedule type or whatever. You can only have one scheduler So wait Okay, one more We're gonna do one more thing here. I want to show Michael. Can you come up? You're gonna be my There's other things that a lot of things that we're just trying to I'm just trying to give some basics here Michael's gonna actually be my initializer. Anyone know what an initializer does It initializes pods it changes them in some way There are other things called pod presets and admission controllers those are built into the API server So when someone says I want to run this application the API server will change it somehow by either saying like no You're not allowed like I don't accept that container like I don't want Java my cluster I can say no Java and the API server just won't allow it. It's just nope that you're not allowed to do that Their pod presets and and admission controllers are really cool But they're hard-coded in the API server and if you want to change them you got to reboot the API server You have to or compile your own it wasn't super flexible It was kind of a v1 of being able to modify things as they came into the cluster So they introduced recently initializers which I think are alpha and one nine But anyone go into an Istio talk. That's what it uses. It's an initializer in the cluster. So when something comes in One thing that we add to these pods Right as we add an annotation So someone I heard go earlier. So we're gonna say we got another one Let's just say we'll make it red too And we're gonna say we want two of those So we have go names Bob and go Not Bob Red red scheduler we only got two I mean it's go apps. I'll play the real-throwing job Actually, come on up. You can we can do one for Vic again. He's back Vic yeah Okay So but wait wait hold on because I have an initializer now. I have a config that says I need to initialize things literally That's what we're gonna do When I create this thing now the API server says hey, I created this thing I have a config for initializer so I need the initializer to tell me if it's ready or not You can have multiple of these you can chain them and say oh here's initializer one and two and three so in this case Michael's gonna say okay. We need to actually Put a envoy proxy on it. So we're gonna run Istio and it's gonna run an envoy proxy so He would mutate These pods and so we actually need to add one more question one more thing here We got don't worry we got this Java containers in each one, but then Michael here is gonna say it's go and it's Envoy Going on boy. So now the containers in there. So do we sign it? Yeah, you're good. Wait. Wait. Wait. Hold on Initializer then tells me is it initialized Well, I mean is it in is the object in the database initialized it's yeah Okay, we are initialized go There's a there are pink balloons in there that is envoy color coding guys This is good. If you rub it on your head really good. You can probably get this Go. Oh, we didn't begin pick color There were no gopher balloons You know how much I looked for gopher balloons and I could not find ones So this is what we're doing sure Yeah, yeah, we got two schedules so go ahead and blow those up Cluster is gonna make that happen and this is what happens when they're creating pods It's only one left So one other thing that I noticed here was see these labels on on go. We labeled it red We wanted to match red for our service, right? So what's gonna happen? Controller manager does that match this thing? Well, the the go app is labeled red and we wanted to match label red. So now we have go Bob and go But that's a problem because the service doesn't know that it's a different app the service is just matching on labels So if you label something two things the same your service will go to the wrong place It's good, I got I like it This is why this talk was really fun So really and that's that is the job of initializer and again Careful with initializers if you make an initializer config and you try to start your initializer Guess what? He can't initialize himself. So there are some of those chicken eggs You need to worry about you can tell it I don't ever want to initialize this namespace or this this pod or whatever it is, but that's literally here's our cubelets Here's initializer controller manager and API server and that's kubernetes. Thank you Any other quick questions about failures or how a component would work? That's not clear Yeah What's that? node evacuation Well, something labels the node is needing to be evacuated Yeah, you can so in that case, let's say we're gonna Evacuate a lawn here and we're gonna say hey, he's no longer schedulable. And so in that case Literally what we can do is we find any pod that was matched to the lawn The pods themselves that have replicas The pod gets deleted and then the controller manager says hey, I need two of those things They're not I don't have to it creates another one the scheduler then looks and says Oh, I have a new thing which Cubelets are available and schedules them. It's the same process. You just delete the pod it says oh as they're evacuating We're just gonna delete those things the cuba then says I don't have to do this drop some fine. I'm out Right and then if it's scheduled to Vic we recreated that thing Vic gets it scheduled and he runs it so yeah ideally so Custom resource definitions ideally would have their own scheduler or their own controller And so those would you would apply an annotation that says I want to use this is a custom resource definition and your Scheduler inside of it is only gonna look at those definitions. And so the default scheduler doesn't know anything about your CRD So it's not gonna do anything with it It's just gonna say hey do I have anything that is they actually will get an annotation That's like hey if it doesn't have an annotation here for a different scheduler then I will look at it But if it does I don't care not gonna look at it They only look at their specific thing and they do control loops on that they just reconcile All right, please give the cluster a hand and thanks for coming