 Mae'r rhan o'r rhan o'r cyfrifio. Hi! Hi! Mae'n Mat. Yn ymlaen, mae'n gweithio'r cyfrifio o'r cyfrifio'r team redhat. Mae'n meddwl o'r cyfrifio ymlaen o'r cyfrifio o'r cyfrifio, oherwydd o'r cyfrifio cyfrifio. Mae'n meddwl, mae'n meddwl o'r cyfrifio, mae'n meddwl o'r cyfrifio o'r cyfrifio o'r cyfrifio o bobl wyrd. Ond ydw ymlaen i gymroadd hanfod, neu'n may'r llwydません. Da d parasite fel'r cyfrifio sgiel Mae'n ddweud o'r oedd, mae'n golygu o'r tawch ardennig yn gwaith. Mae'n gweithio eich 10,000 yma. Mae'r holl yma eich bod yn ddifen y gallu'n gweithio, ond y gallwn eisiau gwahodd o'r ddau o'r documentaidd. A yma'r holl yma'r holl yn gyfnodydd. Mae'n holl yma'r holl yma. If, let's start right in the middle here with the API server. So the API server is the heart of Kubernetes. Absolutely everything, well, pretty much everything talks to the API server. It's kind of like RabbitMQ is in your OpenStack server, except if RabbitMQ was also user facing. So when I talk about objects in this talk, those objects are created, they're persisted, they're updated in that API server. And there's other components in Kubernetes which are called controllers. These watch the API server, they look for what's going on, and they respond to things that happen in the API server to make things happen in the real world, like in your cloud. And the controllers which are vital to Kubernetes itself, they're running over here in the control plane. But we're going to be looking at one in particular. We're going to be thinking about this one here, which is the cloud controller manager. This is the controller that talks to the cloud and makes things happen in OpenStack. Now, your cloud runs, your Kubernetes cluster runs on what's called nodes. A node in Kubernetes is a place where pods can run. And here we've got three of them. And the most important thing for us to know about a node right now is that a node runs a cubelet. We're going to go over all of this stuff again. So this is core Kubernetes. But as we're quickly going to see, this is not the whole map. So rather than just talking about the components themselves and listing them, what I'm going to try and do is talk through three different scenarios. So I'm going to talk about what happens when we scale up a cluster. So if we add a new one of those nodes to our cluster, I'm going to talk about what happens when we deploy a pod using some persistent storage. And I'm going to talk about how we can access an application that's running inside our cluster from outside of a cluster. So jumping straight in, let's scale up our cluster. So specifically, what does this mean? I mentioned this node object here. So this cluster has three nodes. We want to add a fourth. So maybe these three nodes are the control plane nodes. And we want to add a worker node. So what's that going to look like? And straightaway, we're off the Kubernetes map because Kubernetes does not define how Kubernetes is installed. It's explicitly out of scope. And as you might imagine, there are a number of different ways of doing this. So to keep it real, I'm going to talk about just one of them, which is OpenShift. And OpenShift uses something called the Machine API. But the Machine API shares a common ancestry with another project called cluster API. So while I'm going to be talking specifically about OpenShift, many of the concepts that I'm talking about work very, very similarly in cluster API. In fact, they're so similar that in Shift on Stack, we actually use the cluster API under the hood as our implementation. The common features between the two are that it is Kubernetes managed by Kubernetes. Hence, the cluster, I love this logo. It's turtles all the way down. The biggest difference between these two implementations is that in cluster API, the management is done by an external cluster, whereas OpenShift manages itself. So in cluster API, the machine API, the same things are happening, but in cluster API, they're happening in a different cluster. In OpenShift, they're happening in OpenShift. So what are we trying to do? We're trying to get a new node. A node represents a cubelet to Kubernetes. A node tells the Kubernetes cluster about the cubelet. It tells it its properties. It tells it its endpoints. It doesn't define anything at all about the infrastructure that it runs on. For that, we're going to need another object. In machine API and cluster API, that's the machine object. This is where the OpenStack properties that we expect to exist somewhere, this is where they live, in the machine object. The machine object describes a glance image. It describes a flavour. It describes what networks the server is going to be connected to. If you've got any hardware that you need to present to your workload, so if you've got SRIOV, if you've got VGPUs, they're going to be defined in the machine object. So how does this flow work? First things first, we kick off the process by creating a machine object in the API server, but that doesn't do anything. That's just a machine object in the API server. What happens next is there's another component called the machine controller. The machine controller is watching for the creation of machine objects. The machine controller is going to see the creation of our machine object. It's going to read the description of the machine that we want to exist, and it's going to create a nova instance. That nova instance is going to boot. As it boots, remember we're trying to create a node object. We need Kubernetes to be running on this server. The outcome of this server boot needs to be that we are running a cubelet, and that cubelet is bootstrapped into our cluster, and it can start to perform cluster actions. In order to do that bootstrapping, it's going to make extensive use of nova metadata. Once it's up, we now have a running cubelet. Cubelet has joined our cluster, but we still don't have a node object. The first thing that cubelet is going to do is it's going to create that node object, but it's not ready to use yet. It's tainted. A taint is something you put on a node object that basically says pods aren't allowed to run there yet. It's actually going to have a number of taints that mean a number of different things, but the one we're most interested in is the cloud provider taint. The cloud provider taint says the cloud provider has not initialised this node yet. So our cloud controller manager, which is cloud provider open stack, is watching for that. Specifically the node controller within cloud provider open stack is watching for nodes that are tainted with the cloud provider taint. The first thing it's going to do is it's going to try and work out which real nova server corresponds to this new node that came up. Once it does that, the next thing it's going to do is it's going to ask nova about the properties of that server, and then it's going to annotate the node with the properties. It's also going to annotate it most importantly with the endpoint addresses. The endpoint addresses on the node object, they are how the rest of the cluster knows how to contact this specific cubelet. If they're wrong, then the node will not be usable within the cluster. Eventually all the taints will have been removed and the node is ready for use. To recap, the machine controller, which on open shift is running in the machine API namespace in the open shift cluster itself. In cluster API, it's running on your management cluster, that's going to create your nova instance. Your nova instance is going to boot and it's going to start running cubelet. Cubelet is going to create its own node object, but the node object will not be ready for use until it has been initialised by the node controller of cloud provider open stack. Next step, we've got a new worker. What are we going to do? We're going to deploy an application on it. Eventually, any non-trivial application is going to need some storage that doesn't go away when you reboot a pod, when you restart a pod. In open stack, we've got two great options for this. We've got Cinder and we've got Manila. Unfortunately, I'm going to be talking about Cinder because tomorrow in this room at 5.20pm, there's a talk about Manila, which has different capabilities, but works much the same way. Again, we're off the map. Kubernetes doesn't do infrastructure directly. Kubernetes manages workloads which run on infrastructure, but it doesn't manage that infrastructure itself, not core Kubernetes. With storage, they decided not to reinvent the wheel. There was an existing project out there called the container storage interface that was already in use by other container orchestrators, and it was providing storage services to containers. Kubernetes decided we were going to use CSI. CSI provides a generic interface for managing storage attached to containers. Both Cinder and Manila have CSI drivers, but, as I said, we're just going to be talking about Cinder today. As a user, what do I want to end up with? I want to end up with one of these things. I want a persistent volume. This is an object that represents real storage that exists, that is backed by actual storage, and I can attach it to my pod and I can store stuff there, and it'll still be there when I restart. But this isn't where I start as a user. What I create as a user is one of these. This is a persistent volume claim. This just says I want some storage. I'm typically going to say I want 50 gigs of storage. I'm going to describe its access mode, so read, write, single attach, and I'm going to find its storage class. Its storage class is going to reference a storage class object, and this isn't created by the user, typically. This is created by the cluster admin or maybe your installer, and this describes how Kubernetes should create that storage. In this case, it's going to say we're going to create this using the CSI driver, the Cinder CSI driver, and it may also describe some properties that the specific CSI driver is interested in. A common one you might want to define is volume type, for example. How's this going to work? Let's think about what needs to happen for a regular Cinder attach without the context of Kubernetes. In order to use some Cinder storage in an open stack workload, first of all, we need to create that storage somewhere. We need to attach that storage to a server where our workload is going to run, and then on that server, something on that server needs to be able to recognise the storage that was attached and to do something useful with it. We need to do all of these things in Kubernetes. The first of those things, the creation of the storage, that's done by the provisioner, which runs in the control plane. The second one, the attach to where the workload is running, that runs in the attacher, which is a separate controller that also runs in the control plane. The last bit, the find our attached storage and do something useful with it, that runs directly on the node. Actually, that's directly orchestrated by Kubelet. But the provisioner and the attacher and Kubelet, they don't do storage themselves. Instead, they talk to the CSI driver. The CSI driver is an application that exposes its CSI API via GRPC over a local Unix socket, not Kubernetes, why we need these shims. The way we do that in the control plane is the provisioner and the attacher are running as two pods in a container, sorry, two containers in a pod, and we've got another container running in that same pod which is running the CSI driver, which means that the provisioner and the attacher can forward their CSI requests to their local CSI driver. We have something similar on every single node. We've got a CSI driver running on every single node, and when the Kubelet wants to do a storage thing locally, it doesn't do it directly, it knows where to find the local Unix socket of its local CSI driver, it sends that API request there and the CSI driver does it for it. What does that workflow look like? Where do we start? The user creates a persistent volume claim. They say, I want some storage. Next, the provisioner is going to spot that the user wants some storage and the storage class is one that this provision knows about, send the CSI. So the provisioner is going to create the sender volume that fulfills the user's request and it's going to create that PV object which the user can then use to reference. So now we have the PV object, the persistent volume, and then the user is going to run a pod and the pod is going to reference that PV. And when the pod gets scheduled to a specific node, something's going to have to attach it there. So now Kubernetes nodes, we have a workload, we have some storage, we know where it needs to be, the attacher now knows all of these things and the attacher running the control plane is going to do that sender attach. And then Cubelet then needs to do the last step. So Cubelet says, I know I've had some storage attached, so I need to ask my CSI driver to go find that storage for me wherever it is, format it on the node and mount it into my container. So lastly, we've got our application, we want to use it outside the cluster. There are a few ways that we can expose it, but we're going to expose this one using a load balancer service and in OpenStack that means Octavia. Specifically, the external connectivity is going to be from a floating IP and that floating IP is going to be backed by an Octavia load balancer and the Octavia load balancer is going to distribute that to pods within our cluster. And what does that look like from an API point of view? Well, this is fortunately extremely simple. All you need to do is create a service of type load balancer and it will do the rest. And what does that? Again, this is in the cloud controller manager. So the cloud controller manager is running several controllers. We already talked about the node controller. This is the service controller running in the cloud controller manager. And I say it's simple, but it can be simple, but you can also make it complicated if you need it to be. So you can customise the load balancer service that you create in a large number of ways. One of those ways is by putting what's called annotations on the service object or the administrator can pass the generic cluster-wide configuration via the config file that we passed to Cloud Provider OpenStack. So what does that look like? This is actually the end of a demo that unfortunately I didn't quite have time to show you the rest of. But in this demo, I've created a MySQL database and I've created a load balancer service that references that MySQL database. And the service that I created that was backed by this load balancer service is simply a very simple service that says type load balancer. And what I'm going to do here is I'm fetching back my service which I previously created, which we can see has now been annotated with an external IP. So the service controller created a floating IP. And this is, as a user, how I can retrieve that floating IP. And I can connect to it. And this does a wide variety of protocols, by the way. So you might use this if an ingress was inappropriate. So a quick recap. Cloud Provider OpenStack. We talked about the node controller used in node initialisation. We talked about the service controller creates our load balancer services. We talked about the machine and the cluster API. This defines the infrastructure that Kubernetes runs on. We talked about the container storage interface. We talked about how the provision and attached controllers, which run in the control plane, do most of the work. And how format and mount runs on every single node. And they both do their work via the CSI driver. But there were some things that we didn't have time for. We didn't have time to talk about the other controller in the Cloud Controller Manager, which is the node lifecycle controller. This is responsible for deleting nodes when the CCM notices that the underlying infrastructure has gone away. And we didn't have time to talk about Courier, which gives us OpenStack native networking for your Kubernetes cluster. And as I mentioned, we didn't have time to talk about Manila, which some other people will be talking about tomorrow in this room at 20 past five. And with that very brief overview, are there any questions? With ironic integration? So the question was the machine API creates instances. Is it possible to have it create ironic instances instead of Nova VMs? Fortunately, my colleague Emilia is sat right there. Does he know the answer to that question? Yeah. So you can technically, I think, do we test it? Yeah. So in OpenShift, I think we test it, but please don't sign any commercial contracts based on that promise. In cluster API, I think we do not test it. I can't think of any fundamental reason why it should not work in cluster API. I don't know the answer to your question. I'm sorry. Any other questions? Okay, cool. Is that happening in CAPO, cluster API provider OpenStack? Oh, okay. So there's a different cluster API provider. Okay, cool. Any other questions? In that case, thank you for bearing with me at the beginning and have a good conference.