 Hey, you're too kind, hey, thanks, can you all hear me well in the back? Seems like this is working pretty well, right? Great, I'm Michael Rivnak, I work for Red Hat, I'm based in Raleigh, been working on the operator SDK for the last half a year, more or less, and particularly around the Ansible side of the operator SDK. How to use Ansible to make your own operator, before that was working on how to make an Ansible based service broker, or how to add entries to the Kubernetes service catalog, using Ansible playbook bundles, there's a couple other teammates in here around. So I've been on that team for about last year in change, focusing one way or another on how to enable deployment and self-service provisioning and that sort of thing on Kubernetes, especially using Ansible. That's what we're gonna talk about today. So first, what is Kubernetes? How much time do we have? Not very much, so this is gonna be quick. Kubernetes, of course, is a system for managing workloads across a cluster of machines. We put work into pods, and Kubernetes schedules that out for us. Usually in a pod we have containers, but now we're starting to have virtual machines as an available thing. You can move around and schedule. But a couple of the key things in terms of the experience of using Kubernetes as a refresher. This is what it looks like to tell Kubernetes to do something for you. So here we have a pod definition on the left. This is in YAML, of course, it's declarative. If you've not seen this before, this is your first time looking at this sort of thing, don't squint too hard. But the interesting part starts at the spec, about halfway down. And that is where we tell Kubernetes, I want you to make a container, and it should look like this. Here's the image I want you to run, and here's a port that I want you to expose. And on the other side, we've got a service that looks kind of similar, right? So this is how we often think of the basic way of interacting with Kubernetes and getting something done. You write out some YAML, or maybe you use some tooling that helps you write out some YAML, and then you hand that to Kubernetes and say do this. And it's declarative because you are declaring this is what I want, and then you step away and trust that Kubernetes is going to do that for you. Of course, you trust and verify that it actually did it. But this is the experience, it's declarative, and it's YAML. So now let's talk about Ansible in the Ansible Kubernetes module. If you have ever in the past used or tried to use Ansible with Kubernetes, you may know that there's been quite the evolution of Kubernetes support in Ansible. And what we had available a year ago, or even a year and a half ago, has evolved substantially, to say the least. So starting with Ansible 2.6, we had this KAS module available. It is leaps and bounds better than what we used to have available for interacting with Kubernetes from Ansible. So if you've not tried this one yet, and especially if you've tried something in the past, strongly encourage you to check out the new KAS module. Makes it really easy and elegant to interact with Kubernetes. But here's an example, on the left, we are just creating a config map. This is, again, it's just a little bit of YAML, it's declarative. And you could, for example, save that, snip it to a file, and type kubectl create-f, and give it that filename, and it would create you a config map. On the right side, we have the same thing, but now we're doing it with Ansible. So we've added these three extra lines on top. This is what a normal Ansible task looks like. And in Ansible, the experience of writing Ansible is, you create a series of these tasks, and each task does a thing. Like you would say, make sure these RPMs are installed, and make sure these ports are open in the firewall. In this case, we're saying, make sure that this config map exists in Kubernetes. But you can see on the left and the right that this is exactly the same thing. You can inline exactly the same YAML that you're used to working with, that you're used to creating in Kubernetes, right into the KADS Ansible module. And then I've taken the liberty here of just giving an example that we can start templatizing it. So in the bottom right, you see that instead of having the word red, I've now templatized that, and it's now variable. So we can set the color, perhaps based on what variables are currently available in Ansible at the time that Ansible's running. But the key here is that it's a very natural experience. If you've done anything with Kubernetes, if you've dealt with these kind of Kubernetes manifest files, transitioning that to Ansible is just stupid easy, it's so easy. And if that's not easy enough, here's the other way you can do it. This looks a little bit ugly if you're not used to working with Ansible. But this is a one-liner that enables you to save that manifest in a file, just like you normally do, and look it up as a template. So you can now have a file or a directory full of these manifest files. You can templatize them to your heart's content. And you can just reference them just like this from an Ansible role, and use them very transparently and very simply. So now you can run Ansible, and it will create, or it will ensure that all of those resources have been created, create them if they have not been created. Or if their state is different than what you've declared, it will change them to make sure they match that state that you've declared. So high level story here is Ansible is really easy to use with Kubernetes. And why would you want to do this? So similar patterns, so we're looking at YAML on both sides. It's more or less declarative or idempotent on both sides. So it's a pretty easy, easy fit. There are a lot of ops teams out there that are already familiar with Ansible. Perhaps using Ansible already to deploy their applications in a non-containerized environment. It can be a pretty natural transition. And in many cases, you can reuse some of that same Ansible asset that you've created elsewhere when you transition things into Kubernetes. And then even if you've never used Ansible, it's very easy to learn. It's just ginger templating here that we're doing. And it's really easy to get started. We have some tooling that's gonna help you get started with it as well. Okay, so what is an operator? This is a Kubernetes controller. An operator is a special kind of Kubernetes controller. We're gonna look at an illustration in just a moment of exactly how they work. But a controller is basically a process that runs in your cluster. And watches for these kind of objects to be created through the API, and then does something in response. For example, there's a pod controller that watches for you to create pods through the API. And when it sees that happen, it goes in, makes sure that now it starts containers or makes sure the right containers are running in the right places based on that pod definition you have given it. So that's what a controller is. An operator is just a special kind of controller that knows how to deploy and manage an application. So if you want to deploy a database like Postgres, you can make your own or you could just reuse one that somebody else has already made, an operator that understands how to deploy that Postgres database on the Kubernetes and also then do day two management of it. Know how to back it up, how to restore it, how to upgrade it, how to fix it if it breaks in certain known failure modes, that sort of thing. And this gets to the third point of human operational knowledge in code. Anything that an ops team might do to manage a running application, as much as you can, you want to encode, write out encode in your operator. And this is how you scale the operation of applications beyond what a human team can do. This is how the large cloud operators are able to manage so much workload at such great scale, because they've automated a tremendous amount of that. So that's in a very basic sense what an operator is. But let's start with the beginning of how do you interact with your operator? How do you define the interface between the human who is going to use an operator to deploy something and the actual mechanics that are gonna do it? And it starts with the ability to extend the Kubernetes API. Kubernetes has this thing called custom resources. If you imagine a REST API, and a whole bunch of REST API endpoints like you've seen in many web applications, Kubernetes also has a REST-ish API. But what's unique is that Kubernetes gives you the opportunity to extend that API, you can add your own resources. I could add one called Michael and make it whatever I want. I could add a resource called Postgres and make it whatever I want. And now we're talking about how to make an operator. So you can create this custom resource, you can specify what fields are available to use in that custom resource. Here, this is a very simple example of a memcache d resource. So now, once I've created this, there is literally a new set of endpoints in the Kubernetes API for my cluster that would be available at cache.example.com, such view on alpha one, such memcache d. And I can create those resources, I can delete those resources, and I can update them, do all the normal web API kinds of things with that. And you can specify what fields you want. So now we can start thinking of this like an installer. All the same kind of input you might take in an installer for an application, here you would add those fields to this resource and collect information from the user about what things do you need to tell me about how to deploy this thing you want me to deploy, and then I will go do it. So an installer is another way to think of what an operator is. And this is what the whole picture looks like of a controller, which of course an operator is a controller. So we have this smiling face up in the top left that interacts with the Kubernetes API. And we've created some custom resource like that memcache d resource endpoint. And this smiling face tells the Kubernetes API, here is an instance of that resource I want you to create. So here's a memcache d definition, make this. That resource ends up in Kubernetes, the Kubernetes API stores it in this data store, which happens to be at CD. And we have this orange controller down here in the middle that is watching for events related to that particular resource type. This is our operator. And when it receives one of these events, its job now is to run a reconciliation loop. So it sees this object just got created or updated or something happened to this object. So I'm gonna run the reconcile function with that object. And that reconcile function is the real brains behind your operator. That is what decides how to create the resources, how to go look at what exists in the world. What has this person asked me to make the world look like and reconcile the differences between those two things. And in that reconcile loop, in this case of an operator, we're going to create some resources or ensure that those resources exist. And out-pop the pieces of your application. So this would be all the pods and services and storage and anything else you need to do or create in Kubernetes, sorry, okay. He's just dumbfounded by how easy this is. This is how you can deploy an application and make an operator. This is the basic workflow. And what I wanna point out about this workflow, this reconcile loop, every time you run this reconcile loop you should get the same results basically. Much the same way as when working with Ansible, we have a goal of you could rerun the same Ansible playbook over and over and end up with the same results at the end. So this is a really good fit here. And what we're gonna look at now is in an Ansible model, our reconcile loop down here in the middle. We have the operator SDK, something we'll talk about in a minute, which helps you to build your own operators, not only with Ansible but other ways as well. We have an operator that has this binary that we've created already running in it. And a mapping file. And this mapping file is very simple. All it says is for this particular resource type, like memcache D, run this Ansible role or run this Ansible playbook. And that's it. So your operator knows which resource type should it be watching. And every time it gets an event for one of those resources, it looks up in this mapping file, which role or playbook should I run. And then it goes and runs Ansible and feeds it that role or that playbook. And when that runs, it uses that KADS module that we looked at earlier. And does whatever that Ansible code tells it to do, and out pops on the other side the pieces of your application. So the operator SDK, this is a set of tooling and libraries that help you to create your own operator. By default, people tend to start with go. At least Kubernetes developers start with go. Because Kubernetes itself is written in go, the client libraries for Kubernetes are themselves written in go. And there are a lot of very important and helpful pieces written in go that enable you to interact with the Kubernetes API in an efficient way. There's queuing built in, there's smart caching built in and all this kind of stuff. But if you're gonna create your operator in go, you have to write in go. You need a software team that's gonna create a software project and write it, create it, maintain it over time, learn all these libraries. Very doable, very achievable, I love writing in go. We created the Ansible operator itself in go, and it's a nice experience, but it's coding. Ansible is another way that the operator SDK enables you to create an operator. We'll look at exactly how. But the basic story is you get this command line interface operator SDK that will scaffold out a new operator project for you, whether it's go, Ansible or Helm. And in the Ansible case, it gives you all the pieces you need, and you just go in there and start filling out this Ansible role that it created for you. Then there's also the Helm side of the operator SDK. If you have existing Helm charts that you would like to use and be able to quickly create an operator based on reusing those Helm charts, you can do that as well. And here's the GitHub link for how to find the operator framework in the operator's SDK. There's a lot of documentation there, and some real good materials. Let's talk a little bit more about the interface, experience of interfacing with this, and how we get data of this installer type data from the user into the operator and into Kubernetes. On the left, we've broken down a diagram of what a custom resource looks like, so the type is defined by group, version, and kind. And then within some given namespace, we uniquely define each resource by its name. So within any type, you can have one name of a resource per namespace. Now we get to the spec, this is the important part. This is where you specify what you want the world to look like. And you can put any series of key value pairs here that you want. And whatever key value pairs a user puts here to describe how they want you to, for example, deploy Postgres, those key value pairs get plumbed all the way through into Ansible. So while Ansible is running, it has this global state of variables that are available to all the Ansible templates. And each of those key value pairs get plumbed straight into Ansible all the way through and available right there in your templates. So you can, using Jinja, templatize your Kubernetes manifest, make that part of an Ansible role, and then just specify those keys and values right here in your custom resource when you create it through the Kubernetes API. And then the status is something you do not touch as a user. The status is what the operator owns. And that is its way to communicate back to you what is it done? What is the state of the world? Did it fail? Did it encounter any errors? Is it in progress of running what you've asked it to do? Has it succeeded? All those kind of things you might find in the status. This is what an Ansible operator container image looks like. We've done, again, most of the work for you and you only add two things on top. So the green section down here is the base image that we provide. It includes the binary that is the operator. It includes Ansible itself and Ansible Runner. Ansible Runner is something of a vehicle to programmatically run Ansible. Ansible Runner, I think, grew up as part of Ansible Tower. It was a way for Tower to programmatically interact with Ansible, get the results back from it, and so on. So we use Ansible Runner with Ansible. And then on top of that base image, you just provide one or more Ansible roles. And then you provide this watches.yaml file, which is just a mapping of group version kind to a path on disk to a role or a playbook. So it's pretty simple. That's wrong on time. That's better. This is what the developer experience looks like of actually creating an Ansible-based operator for the first time. So the operator SDK is this command line tool that comes as part of the Operator SDK. So you run a new command, you give it a name. So in this case, we're creating the memcache d operator. We specify our group version and kind here. So cache.example.com is our group. V1 alpha 1 is our version. Kind is memcache d. And now we're telling it the type is Ansible. The type could also be go or it could be home. But in our case, we're doing Ansible. That creates for you an empty Ansible role that you can now start filling in with your Kubernetes manifest files and turning those into templates. It creates for you a mapping file that's already set up with this resource of this name mapped to that brand new role that it created for you. It has the custom resource definition and it has all the deployment manifest that you need to deploy your operator in Kubernetes. If you'd like to know more about this, my colleague Derek here and I are running a workshop on Sunday morning at bright and early nine o'clock. For those of you who don't have too much fun on Saturday night, we will be here. There will be coffee, I'm sure. So come hang out and come, we're gonna put our hands on code and we're gonna make Ansible operators on Sunday morning. So that, I think we've got a few minutes for questions. So, lay it on me, which room, no idea. But I'm sure it's on the schedule, yeah. Yeah, we've got two days to figure that out. All right, what are the questions besides the room? A112, thank you Carol. No questions. Sunday, thank you for stalling with another easy question. Yes, part that's true, yes. The question is, is the state of the world represented to the Ansible playbook through those variables that come in from the CR? I would actually say no. The state of the world is available for inspection by Kubernetes itself. So you can use the Kubernetes module to read other objects, for example, and get state from them potentially, as that makes sense. And Ansible is quite powerful. I mean, what we've looked at here just now today is extremely the most simple basic option. But you can do advanced kind of things. You can handle upgrade paths. So you could, for example, look at, this user has asked me to deploy version 1.3, but I can look at the cluster and see that 1.2 is currently deployed. Based on that, I'm gonna run this block of Ansible tasks and facilitate that workflow. So the same kind of logic you could do in a go route. Thank you, yeah, A112 is gonna be the room on Sunday morning. So yeah, a mix of you get the user input from that CR, then you can look at other resources as well to figure out state of the world. That's exactly right. The operator gets an event and the operator should not care what type of event that was. In some cases, events will even be collapsed into one. If there's a lot of events that have happened since the last reconciled, they may be bundled into one poke that just says, you should reconcile right now. And yeah, every time reconcile runs, its goal is to look at the current state of the world, look at the requested state of the world and move the real world closer, maybe not all the way, but at least closer to what the desired state of the world is. Yeah, the basic question is here, sometimes using Ansible and even if you write an operator and go, I think this is also true. Sometimes you need to wait for something to be done before you proceed. So there are different ways of handling this. Generally, the ideal thing to do in an operator is initiate whatever that thing is you're gonna have to wait for and then return and finish your reconcile. And then reconcile again later and check did that finish. And there's different strategies for how to make sure that a new reconcile gets triggered. We probably Sunday morning would be a good time to talk about that. But yeah, that's the ideal thing to do. But you can also just have a very long running reconcile. An operator can have multiple workers running concurrently. So if you think it's acceptable for your resource to tie up that specific resource and have one reconcile that runs for minutes or however long it takes. If that's okay for your scenario, you could totally do that. And that would be fine. But you are gonna have a finite number of these reconciles that can run concurrently in your operator based on how you configure your operator. I think that's all the time we have, is it quick? Short answer is yes. Our existing roles on Ansible Galaxy, which is like Puppet Forge or Python Package Index, this is like global collection of stuff. Is that at all helpful? The answer is yes. Because sometimes after you deploy Memcache D, now you wanna interact with Memcache D. Or after you deploy Postgres, you now wanna interact with it and create a database, create a user and load tables or whatever. So that's usually where it comes in handy. But think about it in time. I'll hang out in the hall if anybody has more questions. Thanks for coming.