 Good morning, everyone. How are you all doing? Having a good conference so far? All right, let's do this. So I thought we'll just start with something not related to Kubernetes. This is a project called Dali. I'm sure you've come across. Or Dali Mini, you provided some text and it gives you an image. As you can see, Donald Trump and Mario Kart 8. People play Mario Kart here? Yeah, you might have played. So I thought what we'll do is we'll start with maybe if you can guess what the caption was provided, what image got produced. So I know there's nothing to do with Kubernetes, but we'll become clear in a second while I'm talking about this. Can anybody guess what's happening here? Any, there's Godzilla in there. There's Cort, Godzilla on trial. So there's basically, Cort sketch of Godzilla on trial. Anybody play Connect 4? Anybody play Connect 4 with cucumbers? With AI, you can play Connect 4 with cucumbers, as you can see here. So Connect 4 with cucumbers. How about this? What is this? What do you all think? It's a toaster in a bath. It's not just any toaster. It's a happy toaster in a bath. So with this AI, we can do something like this. We've got Gandalf here doing some vape tricks. They actually didn't ask Gandalf to do this, but we can do some stuff like that. If you want to check some of these out, you can see it's Dali Mini generates that. One last one before we finish. This is a format printed by Edward Munch, the person who sketched the Scream. You can see this is quite good. Now, the reason why I mention these is because all of these topics here, all of these technologies used to do come up with something like this GP3, Dali, and Clip. They buy a company called OpenAI, I'm sure you're aware of. And they're all trained. The model is trained on Kubernetes. And Kubernetes is the point of discussion today. And Kubernetes scheduler will go into that in a second. And they train this using 7,500 worker nodes. And they've had some issues with scheduling of the pods of where they run. And they managed to fix it by understanding the details of the scheduler, how Kubernetes scheduler works. So I thought it might be a good idea to understand what it actually is, what is a Kubernetes scheduler, how does it work, and how we can influence it. My name is Salman Iqbal. And I am an MLops engineer with App Ear. I'm also a Kubernetes instructor with a company called LearnK8s. Check it out, learnk8s.io. We've got a bunch of blogs on there. And we do Kubernetes training as well. And you can follow me on Twitter. I do also have a YouTube channel. So I'll plug it in and make some more Kubernetes videos. So you can like and subscribe, whatever people do nowadays. So that's me there. Now, I thought what we'll do is do a quick recap of what containers are. And then we'll go into Kubernetes. We'll talk about the Kubernetes scheduler itself, how the Kubernetes scheduler works, how it puts the workloads on Kubernetes. So what you have is a runtime of some sort. In this case, we're saying Java virtual machine. It could be any runtime, Python or whatever it might be. And we get all the configuration files or the artifacts and assets alongside of the application. And we stick them all in together. And we put in any of the dependencies that might be, for example, Java Cryptographic extension or what we call a container, maybe a Linux container. And this is how we end up with a container. I'm sure you already know all of that stuff. So once you have a container, one container you can spin up, you can start, you can stop. And that's absolutely fine. But there's some limitations. If it crashes, you have to start it again. If you have multiple servers, how do you figure out deploying those containers on multiple servers? How are you scaling? And this is where Kubernetes comes in. And amongst other things, what Kubernetes does is schedule these containers for you or what are known as pods in Kubernetes. Pod could be one container or more than one container, one or more than one container running inside a pod. And the good thing about this is you submit the container to a cluster of machines. In this case, I have three machines. One of them is working as a control plane, the one that's got the Kubernetes logo. That's the one in charge doing whatever it needs to do. And the other are the worker nodes. So you and I can write a request and submit it to a Kubernetes cluster and it can receive the request. And then it can decide where to place these containers or these pods. We'll say pods now instead of containers. So it will decide where to place them. And there's some checks it does before it can decide where to place them. That's what we're going to discuss today. So the first node on the left is empty. Why don't I stick it on there? And then the second will come in, stick it on there because it's a replica of the same pod. It won't put it on the first node because if you lose a node, that could be a problematic. And then you can keep submitting other pods and Kubernetes decides where to put them. And maybe you have a container with significant CPU requirement. So it gets deployed next to this container and it basically, you can keep submitting requests to it and Kubernetes is going to do what we said in the beginning. We'll try and play Tetris with the containers and try and pack them together inside Kubernetes. But how does it do that? How does it decide? Let me put the first one on the first node or the yellow one on the second node. How does it do that? We might not have just two nodes. We could have hundreds of nodes. How does it figure that out? Well, this is where the Kubernetes scheduler comes in. And this is what happens. Imagine you want to create a deployment. A deployment is a recipe that says what kind of pod you want to run, what kind of application you're running, what kind of container you're running, and how many replicas of the application you'd like to run. So you write a configuration in a YAML file and you submit that to the cluster and the cluster receives the request. And when the cluster receives the request inside the control plane, the thing that is in charge of everything, it first goes to a component called API server. And the purpose of the API server is to take a request and store it inside a database called HCD. If you would like to understand a bit more about the API server, there's a video on YouTube that I've put in, so check it out. You can check it out there. So request comes in and it stores inside HCD database. Once it's stored there about what kind of deployment we would like to create, the next component that comes in is a controller manager. And the purpose of the controller manager is to look at your deployment requirement, deployment that you submitted and figure out how many replicas of an application you'd like to run. And it creates that in HCD. This is the database key value database. And then is when the next bit comes in, which is the scheduler. And the scheduler decides where to place these pods. And it's done in two phases. The scheduling phase and the binding phase. In the scheduling phase, the scheduler decides the best node for the pod. And in the binding phase, the pod is assigned to the node. And as soon as you create the pod in HCD by the controller manager creates the pod in HCD, it actually doesn't create the pod yet, but it creates a record in HCD, which is a database. The pod is added to the scheduler's queue. And the scheduler processes pods one at a time. So you could have like in a deployment, you could have like 10 replicas, or people submitting multiple deployments. To decide if a node is suitable for the pod, the scheduler scans and filters only the relevant nodes. And filtering might not be enough. As you can imagine, you could end up with a long list of suitable nodes that you could deploy your container on. So the scheduler scores them. For example, the scheduler will try and prioritize nodes, which might be empty. And we'll look at them in a few seconds. I don't know how it does that. So this is all stages it's going through. Even though it looks like just one block, there's multiple things that are happening inside. After scoring, it's pretty much deal is not done. What it does is notifies any of the parties to check if the binding needs to be delayed, which is when the pod is actually created on the node. But other than that, the pod is assigned, and it's time to create a binding. And binding is just another Kubernetes object that links the node to the pod, like deployment or service or an ingress. It's just another Kubernetes object. It's also stored in that CD. And you can do kubectl get bindings to figure out what's inside it. So the pod is scheduled, and all the objects are now updated in that CD. So as in the scheduler has decided which node that pod should run on. So what happens next is the request is then needs to be carried out to actually create the pod inside the worker node. The way that works is in each of the worker nodes, we have a couple of things. We have a component called kubelet, which talks to the control plane to figure out if there's anything that needs to be created for it. And if there is, which the scheduler has decided just now, that yeah, there's a pod that you need to create, it delegates the creation of the container to a runtime. ContainerD, DockerDemon if you're using older version. But all that does is then just run like what you and I would do, run Docker, run and actually spin up the container, spin up the pod with a container inside. And the pod is now created. We talked about the scoring filtering phase, but let's have a look at some of this stuff, scheduling and binding in practice and how does it actually work. So imagine you create a deployment. There's just a simple app, it's just a web app with red page, but I need to have one replica running. It does require GPU, it's just an example. You would never run a web app on a GPU, why would you? But this is just an example, you want to run it on a GPU and you require one CPU or one gigabyte of memory. And we write a deployment file and we submit that to the cluster. And this is the state of our cluster. We have 12 nodes, some nodes are full, as you can see. That's shown by the capacity of the sizes of the windows here, the green and yellow one. Some nodes are empty and some nodes are half full. And you can see some nodes support GPUs and others don't. So basically we have a mix of cluster in here. Now what we need to do is deploy that. What Scheduler is going to do is in the first phase, it will filter out and it will filter out any of the non-relevant nodes. Our requirement was to deploy it on a GPU, so in this case not all nodes support GPUs. So we can discard the ones that don't support GPUs, so they can go and look at that in a second in a few minutes how does it do that. Then we're left with four nodes. Which one should you pick? The Scheduler then ranks and prioritizes the one to deploy on. In this case, there is an empty one. Why not choose that one and run the container on there or run the pod on there? The pod is then scheduled on the node. But how did it do it? Well, you might be wondering how the Scheduler filters the nodes first. We saw in this case, it was quite straightforward. It needed a GPU and it could filter the rest of them out. How about in other cases? Well, the way filtering in Kubernetes works is done through what's known as predicates, which is just checks. It does several of these checks. There's about 12 of them. I'm not going to go through all of them, but just mention a few of them. For example, we'll do this pod fits host ports. It checks if the node has the free port that you've asked for. For the pod is requesting a port. Just make sure that exists before you can run the pod. It also checks stuff like, has anybody specified that this pod should run on a specific node with a name? You can do that in a YAML file. We'll see an example in a second. This will check if there's enough resources on the node for the pod to run, so things like memory and CPU. Again, this checks if somebody's asked to deploy these pods on selectors and this is about volumes, just making sure there's no conflict of volumes. There's no conflict of disks. Again, this is all to do with storage. This is checking if there's no memory pressure. Again, it's just going through a bunch of these checks to make sure to filter out things. If any of these things fail, you can take that out from the list. Check node dispression, check node conditions, make sure the node heartbeat is good, everything's working okay, file system's not full, so it checks all of that stuff. It does more checks on tolerations and we'll talk about the tolerations in the later part of this. Then it evaluates to check if there's any volume binding that needs to happen. The summary of this is that the schedule itself has a number of checks it goes through before it can filter any of these nodes. Once it's done filtering, through this long list that you just saw, if you'd like more information, you can go to the Kubernetes docs and read a bit more. Once it's done filtering, it doesn't end there. Scoring is also equally as exhaustive and it goes through again some priorities, again, some checks that it does to give it a score. Some of these are quite good, so this will make sure that it spreads things that belong to the same service, same stateful set or same replica set across the nodes. It will try and not put them on the same nodes. If you have a deployment with four replicas, belongs to the same replica set, then it will try and put them on separate nodes. This just talks about affinity and anti-finity. We'll talk about this in a second, but we wanted to list it in here. This will basically favor things which has fewer resources running and it's got a bunch of these checks. Again, it's going to go through and make sure that these checks give it a score. Again, there's some of these, node affinity priority we'll talk about them in a second, taints and toleration. This one is kind of useful, image locality priority. It favors a node that already has the container image for the pod that you're trying to deploy. Imagine if you're trying to, let's say, you have, I don't know, 50 or 60 nodes, and you're trying to do a deployment. Some of the nodes might already had that container run before, so it will check, say, oh yeah, this image exists on that node, so you'll try and put it there. Again, based on this, this is about make sure that the pods that belong to the service run on a different node. It goes through it and makes sure, this is the last one is making sure it just spreads the pods across. The scheduler is trying to do the smart things to deploy the pods across all of these nodes. To do it, to spread it out, to make sure if one of the nodes goes missing, things don't really go south. The scheduler is optimized to make the best placements. By the way, if anybody has any questions anytime, please feel free to ask. Maybe we need a mic through, but please feel free, put your hand up and we'll get a mic to you and ask you, is everybody doing okay so far? Are we all good? Yeah, excellent. We've got some thumbs up. Awesome. This is Kubernetes standard procedure. That's how it does. You submit that request, it decides, it's basically there to figure out the best placement for your pod on your behalf. As you saw, there's quite a big list there it goes through to figure out the right node the pod should be running on. It does it when you submit a deployment. But sometimes you probably know your application better than Kubernetes does, the Kubernetes scheduler does. So you might want to influence the scheduler and how can we do that? And this is the next part is about how we can influence the scheduler to place the pods where we want them to place. We've got four things in here. We have node selector, we'll come to that. We'll look at them, look at all of them in a second. We have node affinity, pod anti affinity, and taints and tolerations. So there's quite a few things in here, four things in here that it goes through. And we're going to cover them all. So node selector is a simpler strategy to decide which node pod should be assigned. The first step is to assign labels to nodes. So labels is just key value pair. And pods have selectors. This is the selectors that service is used to run pod on. So let's take an example. I'll show you an example and we'll run through a couple examples and then hopefully that'll make sense. So you have these nodes. What you can do is you can run a command. You can do kubectl nodes, label nodes, and name of the node and you can apply a label. In this case, we're applying a key and a value pair, app equals where it's just key value pair. And in this case, we assign this label to the first and the third node. And apologies for showing a deployment file this early on in the morning, but this is YAML file. We live in YAML world in Kubernetes. That's a deployment file. And in here, if you see at the bottom, you have this thing called node selector. Sorry, the box is shifted up slightly. I'm not sure what happened, but it should be the last two lines. So node selector says app red. So you can imagine and maybe you can guess with me. We've got three deployments we need to run. Where do you think it will go? First, second, third, or fourth node. Three replicas. You can put your fingers up like one, two, three, or four and I'll see. Yeah, we've got one in the back. Yeah, people got it. People got it perfect because the size of the pod is you can fit like two in each node. So we've got two in the first one and the one in the middle one on the third one. But this is basically an exact match. So it's a hard requirement. You just say this pod needs to run on that node. And the examples for this is you might have a GPU and it might have a label on it and you might want to run a machine learning workload on that node. And then you can keep doing this. You can keep increasing the replicas. Because if I go from three to five, the problem is that now I can only put in an extra pod. There's no space for the fifth one. This is a hard requirement. So the fifth one will stay pending. So you might be thinking, well, that's a bit wasteful. There's two nodes which are empty. What can we do? Well, Kubernetes provides no-definity. And there's a couple of things in here. This is a bit long, required during scheduling, no-during execution, preferred during scheduling, no-during execution. That's the property that provides. But I try to remember it as hard and soft requirement. So required is hard and preferred is soft. So hard constraint is run the pods and nodes with GPUs. Soft constraint is run a pod in UK first and then run it somewhere else. That's what it does. And again, we do the same thing. We'll take an example. We can label the nodes. So we've got the first and the third node labeled. And this time, I'm going to show you YAML file with a lot more in it, but it's actually quite similar to what's happening. The point which is highlighted here, you have to see the key where it says in the last block, match expression, key app in values read. It's doing exactly the same thing as before. And the thing is what we've got is required during scheduling. So that's a hard requirement. So we haven't really changed much from last time around. But what you can do is when you submit that to the cluster, the request, the same request you submitted to the cluster, it'll go ahead and get deployed in the first node and the third node because that's exactly what we asked for. What we can also do is then increase the number of replicas. Let's increase the number of replicas. And what we'll see is we're still in the same position. What we really want to do in this case is use the other option, which is prefer during execution, prefer during scheduling, ignore during execution. But the thing is because you can take this deployment and apply it, but the pods have already been scheduled. One is already pending. So it won't really push the other one into one of the nodes that doesn't have the label. You kind of have to delete the pod if you do it halfway through the deployment. But if you decide from the beginning, you want to use the preferred one, that's absolutely fine. It will do exactly that. It will go ahead and then you can delete one of them and it will go ahead and deploy it. But so this one, the preferred during scheduling, ignore during execution is all about deploying to the ones that you asked for. And if there's no space, then deploy to the other nodes. So the next one is pod affinity and anti-finity. Sometimes there might be some apps that you want them to live next to each other. So two pods, you want them to live next to each other. Maybe you're trying to reduce latency because a cluster could be across different regions. And if a cluster, a Kubernetes cluster is across different regions, there's some latency in there. Or sometimes there might be two pods that you don't want them to live next to each other. If they run on the same node, you might be able to see what's running in the disk. You don't want to do that, so you can perhaps put them away and we'll share another example as well. And the way it works is based on, again, labels of the pods and the pods from the same namespace are considered for scheduling. So namespace is logical separation of things in Kubernetes, but this is not cluster-wide. It's just a namespace specific. So, you know, it's the same labels as before their services use. So let's take a look at an example. We've got four nodes. I have this pod called app red. Label has got label app red. And I created deployment with three replicas. The pod definition is that it's got this metadata, three replicas, three replicas, and it has that label pod red. And they get deployed in this fashion, the first and the third node. And we wish to deploy a second app. The second time is a green app. And we want it to be next to the red app. So we want it to have some affinity. So we add these labels in here, so pod affinity, and we say, well, it needs to be next to the red app. And this time, when it tries to schedule it, I'm sure you might have guessed it by now, we'll try and put it next to the red app. There's some space on the third node. We'll go ahead and deploy it there. And anti-finity, as you can imagine, works the opposite way. It will try and put it away from the red app. We'll use that to pod anti-finity if it's not next to, it will try and find a spot, whereas not next to the red app. And we have the third application we wish to deploy. This is the yellow app. In this case, we can say, put them next to red and green. And the thing in Kubernetes scheduling in this case is there's no partial matching. The both requirements have to match. Currently, we don't have any nodes with the two pods or any pods with the two labels, the same labels, so that that will stay pending forever because that's a required requirement. But again, with this node-finity and anti-finity, we can have a hardened soft requirement, just like as node-finity, even with pods, we can have that. So if we do that, we can, with a hardened soft requirement, we can change just like before, prefer doing scheduling, ignoring execution. And if we do change that, let me just ask you all. So where do you think we'll get deployed? One, two, three, four. Any fingers in the air? Nowhere? People think nowhere. Well, I've got four in the back. To be honest, it can go anywhere because there's no partial matching. It could go to the third one. It can go to the second one. It can go to the fourth one because it's just a preferred requirement. It's not like a hard requirement. It's a soft requirement. Pod-anti-finity works in opposite way. And this will ensure that it's not scheduled next to the red and green pod, not just one of them, both of them. That's what it will do. There's a couple of things you have to be careful about. This is all customization I'll say in the end. So this is, pod-anti-finity could take time. Imagine you have thousands of pods. You have to run this calculation to figure out which pods have labels, where does go what. And it's quite useful if you want to schedule pods together, if you want to minimize latency. And maybe sometimes you don't want to schedule pods in the same node. So Ingress, for example, is a good example. You don't really want two Ingresses to end up on the same node. Ingress is the one that deals with the external traffic that comes in. So an Ingress pod, you don't want it to end up on the same node. So you can give an anti-finity to itself so it can repel itself. So this is the last one. And in terms of customization, in terms of giving hints, excuse me. And it could get a bit confusing because of the way the API works, but hopefully we'll get through it together. So what we've investigated so far is whoever writes that deployment file is in charge of what gets deployed where. So as developers, we can deploy anywhere we like and we can just say, oh, this pod should go here, it should go on that node. But in this case, whoever's in charge of a cluster, so an administrator's in charge of a cluster can decide where your pods actually end up. So in short, nodes can repel pods. That's the thing about this. Some nodes can reject pods that you're trying to deploy to. It's similar to the node selector, but the way it works is every node has a label, just like we had before, pods had labels, key value pairs, but it comes with an effect. And the effects, there's three effects. There's no schedule, prefer no schedule, and no execute. And it's better to remember as hard, soft, and evict. And the thing is it's a little bit weird, the way it works, but I'll explain to you. If, with an example, imagine I've got three nodes here and I taint one of the nodes, kind of like a label, I taint one of the nodes with app yellow, no schedule. And the way this works, bear with me for a second, is I've got the first one here, app equals yellow, no schedule. I've got the second one here, app equals green, green, prefer no schedule. And if I try and deploy this red pod, and I say, toleration app equals red. So you create a deployment with the toleration of app equals red. And this is our requirement. Now the thing is, the way you look at this, the request will come in. You will look at the first node. And the first node has app equals yellow, no schedule. And the way it gets applied is if the pod tolerates that label, the effect is ignored. But in this case it's saying, if the pod has app equals yellow, you ignore the no schedule. You can come in and live in the first node rent free. I mean not rent free, but you can live there. But if you do not tolerate that label, you cannot run here. So it goes to the first one, it rejects it, because the pod has this red label toleration. It goes to the second one, it says prefer no schedule. Well, app equals green, it doesn't have that label. It says, well, if you can't run it elsewhere, then you can come here. So it goes, OK, we'll keep that in our back pocket. The third one doesn't have any labels, so it goes in and deploys in the third one. Does, that makes sense. Some nodes, OK, good. So, this is how it works. Now, if I take the same deployment and increase it to three replicas, if I increase it to three replicas, what will happen is that two pods will run on the last node because there's no label. And then the second one, which is in our back pocket, it will basically go ahead and deploy it on the second node because it said prefer. And this is how you make the node in charge of what actually runs inside the node itself. And if we decide to scale it to five replicas, as you can imagine, the fourth one will run in the middle node, but the fifth one is not going to run anywhere because the first one is very strict. It says, unless you have app equals yellow, you cannot live in that node. And sometimes what you can do is you can decide to, let's say we, let's look at this label, app equals green and this action no execute. What happens when you apply no execute? Well, in this case, green app equals green. The app that is running inside the third node doesn't have app equals green label. It has app equals red label, toleration. So it will basically evict those pods. It doesn't tolerate that. It'll say, yeah, you can get out of here. So maybe you need to debug something or maybe you need to upgrade that node. You can use that for evicting some of the pods. If we decide to create a deployment with a green pod, again, we just say app equals green, toleration. So if you tolerate the label, we ignore the effect, the effect that happens. So in this case, the effect doesn't apply to it. It comes in and lives there happily. So it's common practice to take the control plane or the master node that sometimes is called with a no schedule because you don't want the workloads to be running on the control plane because control plane is already busy running multiple other components anywhere. And this is what happens when, you know, when you spin up a cluster, what you do is the control plane doesn't run any of the worker nodes and then just all the workload runs on the rest of the pods. Now, if you're sitting there and thinking to yourself, there's a lot of customization. There's a lot of things to consider. There's like tolerations. There's taints. There's labels. It's getting a bit confusing. And to be honest, it really is. If you think about a scheduler is doing this. If we try and use everything that Kubernetes gives us, if things go wrong at the middle of the night, 3 a.m., the pods are not getting scheduled, this is going to be, yes, at 3 a.m. in the morning, trying to figure out what's gone wrong, debugging the issue, and having some smoke. So the thing is, Kubernetes is a gift that keeps giving. It doesn't just end here, but we can still customize more, right? We can still customize more. At the beginning of the session, I said, well, scheduling is done in like two phases, pre-filter and score. But the scheduling and, sorry, binding. And in the scheduling, we had these bits. To be honest, it's actually a bit more granular. The filtering phase is broken into pre-filter and filter. A scoring phase also has multiple stages it goes through. And all the blocks in the scheduling phase are customizable and pluggable. So you can write your own logic if you wanted to. And in the binding phase, you can see this is also broken down, but they're not customizable or pluggable. So in the beginning bit, if you don't like the way it does filtering, you can write your own code and figure out and plug in your own code and do your own filtering if you wanted to. So this is just a pseudo code. All this does is if you look at the middle one, the Q-Sort plugin just takes two pods and gives you one. It returns one, tells you which one should be prioritized over the other one. So if none of the options work for you, no selector, pod affinity, anti affinity, taint tolerations, plug in customization, you can write your own scheduler from scratch. You can write your own full scheduler and you can find an example of them all over the internet. And when you define a pod or a pod template like this, you define the custom scheduler. You say this is the scheduler we would like to run. The way it works is when you define the custom scheduler, the default Kubernetes scheduler ignores the pod. But then it's our responsibility to watch out for pods with a custom scheduler and assign them to a node. And we have to come up with a logic of a scheduler, entirely up to us what logic we would like to write. Depending on our requirements, we can write whatever we want to write in that logic. This is just a pseudo code example for a custom scheduler. Again, a custom scheduler also runs as a pod inside a node anyway in the first place. So you have to give it all the permission so you can talk to the API, pull in the information that you need to figure out where it runs. For example, in this case, a bit of bash here, you get all pods, KubeCTL get pods. There's no presentation that's not complete in Kubernetes without writing down KubeCTL get pods. And then for each of the pods, you go through and pick a node, all the nodes, and you get a number of nodes, and then you pick a node at random, so you just chose one pod. And then this is the bit, the binding bit. The binding, as we were saying in the beginning, is another Kubernetes component. And you can see it in a little bit. This is just a call request that's been sent. And in the third line, you can see it says kind binding, and all it's doing is just saying the pod that we've submitted to the cluster has to run on the node that's been picked for it. And then we basically submit to the cluster, it just gets assigned to the pod. It's just a call request that it gets sent, and that is then assigned. Now, any questions so far? Is everybody okay? I think we're doing good on time. We have six minutes, so we've got time. I'll do a quick recap. And if you have any questions, feel free to ask. I'm going to be hanging around anyway. But so what we talked about is the scheduler is in charge of deciding where a pod is deployed in the cluster. And scheduling, the scheduler goes through two phases. The scheduling phase and the binding phase. And in the scheduling phase, the scheduler filters the list of nodes, which are using predicates. We saw a number of those. And then we also saw like taints and tolerations and how that worked. And then it ranks the nodes using what are known as priorities. So, you know, things like, oh, is the node empty or does the image already exist in the node and prioritizes those? And then what we saw is if that's not enough for us, we can influence the scheduler to deploy things on the node. So the simplest one we can use is node selector to force a pod to be deployed on a node. We can define it in our YAML file. It should be deployed on that node. And then node affinity is another tool at our disposal to influence the scheduler. And this is basically we can extend the node selector with affinity so we can have like the soft and the hard requirement if the node is not available. If there's an empty node available, it can go and deploy there. And if you want the pods to stay together close to each other, you can use pod affinity. And if you prefer not staying next to each other, you could use pod anti-finity. So if you want to push them away from one another. Taints and tolerations are useful to prevent pods from being deployed in specific nodes. And you can extend the scheduler by writing your own plugin. And if you prefer to schedule the pod yourself, you can write and use a custom scheduler. Now, all of these options exist in Kubernetes. But the question is, should you use them? And I get asked a question, and I tend to just show them this. So life in general, you know, there's some pitfalls, but with Kubernetes, the outcome is the same, but maybe sometimes you look cool when you're doing it. But I don't know, that's up for discussion. So things are there. I tend to just basically leave. The node selector is kind of useful, depending on your use case, to be honest. But you just have to be a bit careful. If you have labels, anti-finity, pod affinity, things are getting applied. When it goes wrong, that's the bit where you need to know, how do we debug this? If you can't do that, then you're great. There's a couple of things I'd like to share before I end. I work for a company called Appia based in the UK as an ML Ops engineer. Check out, we've got some blogs on there. More importantly, I also work as a Kubernetes trainer. So if you're interested in learning about Kubernetes for a company called LearnK8, we've got some pretty cool blogs on there. I am definitely not being biased here. And we also do training. So if you're interested in learning more about it, it's kind of similar to what we just did now. It's quite hands-on training. So check it out, LearnK8.io, but definitely there's quite a few blogs on there. One more thing, if you like comics like I do, and like beautiful drawings like many of us do. My friend Chris Nessbitt-Smith, who's sitting in the back diligently is doing a talk later today at 10 to 3 around policy as version code. It's got cool comics. I don't know what it means, but this is about policies and how you can do that. But definitely check that talk out if you're interested in making sure you run the right thing on your infrastructure and things never go wrong and how you should use them as code. Definitely check this talk out. I highly recommend that. If you have any questions, feel free to ask them now. If not, you can follow me on Twitter. You can message me there. If you have any questions, I'll be happy to answer. If you don't reply on Twitter, leave a comment on my YouTube channel. I'll definitely reply there. I'll go back to you if you have any questions. So that is all from me. I hope you all enjoyed the talk. Are there any questions or are we all good? Okay, I think everyone's got the first talk of the day is done. I hope you enjoy the rest of the day. Thank you all for coming and yeah, cheers.