 Let's get this thing started. Hi everyone. Thanks for joining us today. We are going to have a wonderful webinar. I am Chris Short, Principal Technical Marketing Manager at Red Hat. I am also a cloud native ambassador. I'll be moderating today's webinar. A few housekeeping items before we get started. During the webinar, you're not able to talk as an attendee. Sorry. But there is a Q&A box at the bottom of your screen. Feel free to drop your questions in there and either myself or I will ask Tim. We'll get those answered. And without further ado, I'd like to introduce Tim Atnall, co-worker and senior product manager at Red Hat. And they'll be talking about building Kubernetes operators in an Ansible Native way. Take it away, Tim. Thanks. Great. Thanks, Chris. And thanks, everyone, for tuning in. Like Chris said, my name is Tim Atnall. I'm a senior product manager with the Ansible team. And I'm a longtime Ansible guy. Really know my way around Ansible having been a contributor and then later was a user customer and joined Ansible and came along for the ride when we were acquired by Red Hat. So I'm actually not a Kubernetes guy. I mean, I've been learning it. I really love it. But I come from this Ansible background. What I wanted to share is a way that you can use Ansible and Kubernetes together. So let's see. Am I sharing my slides, Chris? Can everyone see? Yes, you are good to go. OK, great. Excellent. So what we're going to be covering today is just a little bit of background about what are operators for those who may not be as familiar and why you'd want to use them. And then why would you want to use Ansible to build them? And how do you even do them? Why would you want to build them using Ansible? Because isn't Ansible one of those traditional IT tools? What does that have to do with cloud native applications and architectures? And then I'm going to show, do some demos of how you would develop your first operator with Ansible and actually show a basic one that we've developed and then give you some information on how you could learn more about this. So with that, I'm going to start off. This is clearly a CNCF webinar. And we all know and love Kubernetes and what it can do and all the functionality and how it's bringing web scale architectures to our industry and how you can utilize it and all the great things that it can do. So it's just worth pointing that out because what operators can do is build on all of that great functionality and all that power of Kubernetes here. And then what Ansible can do is make that more accessible to more people to be able to use and automate what happens on their Kubernetes clusters and with their cloud native complex applications. So what most a lot of people don't realize is that really Ansible and Kubernetes are quite a natural fit and have a lot of similarities in a number of ways. Both of them help make hard things easier through automation and orchestration. And they're both very active, widely used open source projects. And last year, I think we were 6 and 7 on the GitHub Octoverse report when it came out for activity. So we were Kubernetes and Ansible projects were right there neck and neck in the top 10. So we both have very vibrant communities working together to solve common problems. We have KubeCon coming up and it's going to be, from what I'm hearing, 12,000 people gathering there. I'll be there in the red hat booth for any of you who want to stop by and say hi. I'll be talking about this there. And the other thing is that they both use YAML to describe this desired state of the world. They're both desired state engines. They both use YAML. And here's an example of what I mean. On the left here, we have a definition file that you could use with KubeControl, for example. Setting up a config map. And you could feed that in. And great, you have a config map now in your Kubernetes cluster. On the right side is a single task from an Ansible playbook using our kates module that defines the exact same config map. And you can see by the parts that are in red are almost identical to the definition file you would use with KubeControl. Now the one thing I always like to point out, and one of the first advantages of using Ansible for this is if you look down there at the color, there's a Jinja 2 variable there for templatizing what you do. And this is something that hopefully I'll be able to show you in the demos that you could templatize your definition files and how you deploy and manage things in Kubernetes. If inlining the definition file, like I showed in the last slide, isn't your thing, you can also do something like this. In fact, a lot of people do this, and we actually have it set up this way in our demo operator that I'll show later. You can keep the definition file separately as a template or as a YAML file and read them right into your playbook and feed them into the kates module to get passed on to the kates API. So it's a nice little piece of syntactic sugar you can use to manage your files differently if you don't want to keep all this in a playbook. You can keep it separately as files and manage them accordingly. All right, so that was a little bit of the similarities that there are between Kubernetes and Ansible and how they can work together. Let's do a quick review of what are Kubernetes operators for those of you may not be as familiar with it but we're just to set some context here what we're talking about. So as I was saying, we all know Kubernetes, love Kubernetes, and it's incredibly powerful and brings a lot of functionality to doing things in highly available, scalable manner in a consistent way, bringing a lot of best practices and functionality there. And that's great and gets us so far, especially when we can make our applications or our services stateless because that's easy. It can be very genericized and it can be handled in such a way that if a pod dies that Kubernetes can just replace it and things continue on and they work great. But what do you do when you have to use state when your application needs this special handling and how you manage upgrades, how you manage scaling events, how you manage failures, things of that nature? Kubernetes cannot in and of itself capture all of that operational knowledge in order to manage that application. And that's where operators come in because that stapleness is hard and is unknown to Kubernetes and operators let you bring that extra smarts to what Kubernetes can do. So like I said, operators are there to manage complex Kubernetes applications. They can encode this human operational knowledge. They can handle things such as patches, upgrades, scaling events, things of that nature. And to do it in a Kubernetes native way that we can just do it right on the cluster itself. So these operators are purpose built for your specific applications and services to give Kubernetes that extra smart and to not just handle installing and setting up your application but do all the day to management of what happens, managing the entire life cycle and what happens after the install and the setup initially happens. So without operators, what happens is we get and when we have these special, these stateful applications that require this extra knowledge, extra handling, you end up in a reactive mode without operators. You have to continually check for anomalies or for events happening in your cluster and alert a human if something happens so that they can respond. So one of your operators has to deal with possibly intervening and it could be, it could take them minutes, it could take them hours, it may require them getting up in the middle of the night to deal with this stuff. So that's less than ideal. With an operator, you get to be more proactive. Because it's running on the cluster and it is continuously adjusting for that optimal state and monitoring what is the current state of what's happening on the cluster, it's able to react within seconds and make the adjustments and perform the operations needed to keep the application healthy and running as expected. So when we look at what an operator is, it really is a controller with a very specific pattern. So we have the Cades API and what makes operators go are custom resources and you create a custom resource around your particular application. Then the operator itself is just a controller and that has two special parts to it or two functions going on. One is for the controller to watch events happening on the Kubernetes cluster for the resources that your application is using. And then when something does occur to do something called a reconcile loop. And what a reconcile does is take the actions necessary to get the application in the desired state it should be in when the state changes. And so what's happening then is this controller, this operator is managing your Kubernetes application on the other according to the operational knowledge that you fed into it. So the operator, just to take a step back for a second here operators are part of come from the operator framework. This is something that CoreOS began and then when they joined the Red Hat family we've continued to invest in and advance further. There are three parts to this operator framework this toolkit to build and manage operators on any Kubernetes cluster. This is all I should have mentioned this earlier on this is not specific to Red Hat or OpenShift which is our cloud native platform which Kubernetes is very much at the center of. This can be used with any Kubernetes. In fact, I'll show this a little bit later when I do my demo I'm using MiniCube. So there are three parts to it. I'm gonna be talking mostly from here on out about the operator SDK which is what you would use to build your operators. And specifically the part of the operator SDK I'm gonna be talking about is the Ansible part that's built into that. There's also implementations for doing it in Golang of course and then also some limited stuff you can do with Helm. Also part of the operator framework is the life cycle manager for doing install updates and manage of operators and their dependencies. So it's a nice kind of package manager like tool. And then they're also for doing operator metering so that you can get metrics out of that and for customers that need to do things like charge backs and things of that nature. That's what that's there for. But we're gonna be focusing on the operator SDK part from here out, but that's what makes up the operator framework that's out there. And all that is of course, we're red hat, it is open source and can be found at this GitHub repo you see at the bottom of my slide. Okay, so that was enough about operators and some background on that there. Why would you wanna build an operator with Ansible? Well, okay, so the idea is we're taking a lot of the same principles that have gone into Ansible in general in traditional IT space, whether it's cloud or networking or bare metal or windows machines. The idea of making things easier to do. So taking this, doing powerful things in a very simple way. And so in this case here, we're trying to make it easier to deploy manage these Kubernetes applications but do it in that Ansible native way. So what we're trying to bring to that is taking, we have this traditional IT, a lot of people have invested in automating with Ansible and to really reuse a lot of those skills in this existing ecosystem and this thriving community that's out there. So using all those tried and trusted Ansible tools and to reuse that knowledge that a lot of organizations are already have and are using for managing things that aren't cloud native because as much as we all love cloud native, there's still going to be traditional IT around for the foreseeable future. So this gives you a chance to use that same, the community and to also be able to automate entire stack, whether it's cloud native or traditional IT with one simple language and one automation engine. It also allows, it lowers the barrier of entry to what type of skills and knowledge a person needs in order to automate these applications and to manage these applications. Because it's Ansible, you don't need programming skills in order to automate what is happening on your cluster with these operators that are written in Ansible. And even if you do have the programming skills to do this type of thing, this will let you iterate faster and it'll make maintenance easier. So you can focus your time on other things that require your attention, that cannot be as easily handled as what we can do here. As I already showed is it uses declarative state definitions just like Kate's itself and you can template all that which can make things much more repeatable and reusable than the standard tooling that's out there. So it can really help you do get more out of all the great things that Kubernetes and cloud native systems can do. As I mentioned earlier, there's three ways that the operator framework lets you develop operators. There's Helm, but Helm's limited to only doing what you see there is phase one, the basic installation. Ansible and Golang, those two types, you can do the entire lifecycle of operators. It's one of the things that I get a lot as I talk to customers and at events and conferences, a lot of people think that, well, I'm somehow gonna be limited if I use Ansible and I really need to use Golang. And what we've found is in the people that are writing operators with Ansible is that it can do pretty much everything. There is going to be that very, very, what we believe, small edge cases where you really need to get into and very closely control what's happening on the cluster in the Cates API, where you then may need to move to using Golang itself. But for the vast majority of use cases that we've seen out there, Ansible's been able to handle them all and go across and do the entire lifecycle that an operator needs to manage. All right, so let's take a little bit of a closer look at how an operator written in Ansible looks. So this is another version of the diagram I showed a little bit earlier. And what's different is that you see that we have a few extra things going on inside of this operator. When you build an operator using Ansible, you get this binary, this generic operator that's been written that gets packaged in and bundled for you. This comes for free, you don't have to touch it. We'll talk about that a little bit later. Now, the things that you as someone developing an operator with Ansible are responsible for is the watch file. And what the watch file does is define the group version kind that you, the resource that you wanna watch. And then you define what playbook or role should be run when something changes in the state of that resource. So this is how we map events happening in the Kubernetes cluster to an Ansible playbook or role that is inside of the operator itself. So taking a deeper dive into that and going through the workflow. So you have this operator SDK binary that's part of the operator that gets built. And that has a number of functions in there for you. Generic, well, I don't know if generic's the right word, but all the common plumbing that you really have to take care of and be mindful of when you are writing an operator that you don't, that you really need to take care of. So for example, it provides a reverse proxy and that proxy takes the calls coming from your Ansible automation and adds owner references to it. And that's to make sure that garbage collection works correctly. It does a fair bit of aggressive caching so that you don't pound your, a Kate's API proxy too hard and essentially create a denial of service attack on your own Kubernetes cluster. So it's doing a lot of things like that. And that's actually written in Go. So it's a very, very efficient operator going. The other thing that this is doing is that when the container initializes, it is reading in that watch's file so it knows what it is looking for. So at that point that binary, when it sees that an event happens, it will then look at what was defined in the watch's file and then run the Ansible playbook or role that's been defined for it, take the response coming back from Ansible itself and then pass that along to the custom resource to update the status of your operator. So events are happening, it's going into the operator itself, the Ansible automation runs there, it goes updates the status and manages your Kubernetes application, does its reconciliation on the cluster itself. All right, so that's how Ansible operators work. Developing them can be quite easy and we've tried to make this as frictionless as possible. So there's a few steps that you have to take here. You can get the operator SDK, said completely open source tool, it's available through a number of different ways. I'm using a MacBook here, I used Homebrew to install my copy here. So what you do to get started is use the operator SDK to do a new and hand it a few pieces of metadata, what is the operator itself, what's the API, what's the kind and then the most important part if you're going to be using Ansible here is to use that type equals Ansible that you see there in red at the end of the command. One thing I wanna stress, I've said this once, but I think it's worth repeating again here is this is a native part of the operator SDK. This is not a plugin or some extra thing that you need to add. When you install the operator SDK, you get this right out of the box. So at this point, what'll happen and I'm gonna show this to you in a moment in the demo is this will build out an entire project for you. Not only will it create the skeleton of a role to write your Ansible automation, but it will also create deploy files for you. It will create a build file for you. It creates basic tests and molecule which is a testing framework for Ansible content. It creates a whole bunch of really nice things that really get you up and running quickly to developing the operator. So at that point now you have a project ready to go. You just need to write your automation with Ansible. Whatever those steps are, often it's going to be using the Cates module which is pretty much like cube control but with some extra Ansible things to it. So anything you can do with a cube controller or even the OC command if you're an OpenShift user, you can use that module for to get things done. So you write your automation, you then have to define your watch file. In all actuality, that new command creates a watch's file for you based on the inputs you've given it. Sometimes you don't even have to touch the watch's file but if you do, there's already one started. It's a pretty straightforward small little YAML file that you have to edit, very straightforward thing to use and manage. At that point, you have your automation ready to go. You've run your tests, you feel good about what you want to do. You then run this build command. It's actually a very shallow thing that's been added for convenience but at this point, everything gets assembled. It takes your Ansible automation, it takes a lot of different, a base image that has all the necessary dependencies of Python and Ansible, Ansible runner, some things like that bundles it all up into a container that now you can deploy to your Kubernetes cluster through any means. At that point itself, it's a container ready to go. You do not need to install anything special on your cluster. You do not have to use OpenShift. It should work on any Kubernetes system at this point. It's just another controller container. So taking a step back and I've been talking about this but again, this is worth repeating. This is the anatomy of an Ansible enabled operator. What you see in the white is for all of you that are developing an operator with Ansible, that's the part that you are responsible for, creating the roles or the playbooks and the watches file and managing that. Everything you see in that Raybox, you get for free using the Ansible operator SDK when you do the build command. So those are all the things I talked to you about. So far that binary, that generic go operator, the Ansible, Ansible runner, all the Python libraries, all the things that you need, you just get and so that you can focus on just what is it I need to automate about my Kubernetes application in order to keep it healthy and running smoothly and kind of creating that public cloud native experience of managing services through a UI or whatever you're using to manage your Kubernetes services. So that's I think a really, really nice thing and really speaks to how using Ansible can make things easier and more accessible to more people in your IT organization to participate and contribute in your cloud native systems. Okay, so drum roll here. I'm going to do, try and show some examples and do some live demos of what I've been talking about to give you a sense of how this stuff works. So you're right on schedule. Oh, excellent. And I've been ignoring my clock. So that's, wow, I'm even impressed with myself right now. Yes, I'm quite impressed as well. Yeah, so there's one question in the chat that I'm going to try and answer live while you're getting set up here. Okay. Atensood said is one new controller created for each entry in a watch file with Ansible operator or just one controller for the entire watch file. My understanding one, yeah, my understanding is it's one controller to the entire watch file AKA the entire operator equals one controller. Yes, one operator can be watching multiple things. Correct, correct. Aten, if that does not answer your question, let me know. Do Ansible based operators handle concurrency in the sense of what? Yeah. Concurrency like go in the sense of like a programming language? Obviously not, because Ansible is not a programming language like go, but if you mean like executing multiple tasks at once, it is achievable. And then there's one question here. I got to reread while you're still getting set up. Okay. It's a follow-up. Go Ansible. Chris, is my terminal window displaying? Yes, it is. Okay, great. You could increase the font like a little bit, that'd be great. I'll plug in that. Looks good. If anybody can't see, let me know. Handle multiple events to reconcile. Yes, Carlton, it can handle multiple events or can watch from multiple events to reconcile. It creates a CRD. So you can have it essentially watch for, it watches for any event to happen on the group version of the kind for the CRD and then triggers as a result. So it does handle multiple events and reconciliation loops. Okay. So I'm gonna keep moving on here since I hate typing live. I've been running this through. So the first thing I'm gonna do here is show you how easy it is and what's in a project when you use the operator SDK to do a new. So this is the line that you saw in my slides just a couple of minutes ago where we're doing the new. I'm creating something called the null operator because it's not gonna have anything in it. But the point here is to show you what you get when you initialize a project. And so it's of kind null and there's that type equals Ansible at that point. So if I run that, I get this input that comes out of all the things that created. We're gonna take a look at that. Though I now have this directory here where we have a build to deploy a molecule that's the testing framework that I talked about. We have a roles directory in the watches file. So let's take a quick tour here, what's in there in the build. We get a Docker file and we also get a Docker file and a shell script that helps you run your tests. So when you're working locally and trying to work this stuff out, you have something there to speed things up. Under deploy, we have a number of manifests that can be used with Kubernetes that gets built out for doing things like setting up your role, role binding service account and also the CRD and the custom resource itself so you could just feed them right into cube control and get something up and running very quickly. See here, and then I'll show you what's under the role. So any of you that are familiar with Ansible, you see you have what looks like a starter role here of something that you could write your tasks in. You can add your default variables or internal variables or templates or whatever it is that you're doing. It's already created out a skeleton of a role for you to do your automation and so you don't have to create that yourself. So again, it's trying to make it really easy for you and then we have the watches file here that it generated for. So we see the group version kind based on the input that we gave it and then we see that it is calling the role null that it created for us. Again, you could, this is just a starter so you can add more entries to the questions that were being asked to have it watching multiple resources and calling, you could have multiple roles. You could add multiple roles to your operators or different playbooks. You have a lot of flexibility at that point, a lot of power of Ansible in that. And some of the things that we've started internally working with is you have the entire power functionality of Ansible here that you can use. One prototype that was done internally was for LCD and that can do a backup and restore to S3 because it's using the S3 module that's built into Ansible to send it off the cluster and store a backup in an S3 bucket. So something to consider. This is definitely a bonus of using Ansible and you had the templating but then you have all those modules and all that functionality that Ansible has for uses outside of cloud native there. All right, so let me see here. Let's move on to the next thing. I'm not gonna do a build here because I think moving on to some of the other material that I have here will be more interesting. So let's see, I should be back to my slides now at least I hope. So the next thing I'm gonna show here is deploy a Kubernetes application that has to be orchestrated and managing multiple pods. We've tried to keep it as simple as possible just so that you can follow along and get an idea of what's happening but also get a sense of the power of operators and how Ansible can help manage these type of things when you need to coordinate multiple pods and services and things. And what we've come up here with is a scalable caching service and there are two components to it. We're using Memcache, which has been around a really long time. It's general purpose, distributed memory caching system for database calls, APIs, like I said, it's been around a while. It's very generic in nature. It doesn't have a lot of fancy features like replication and fall over and things like that. It just so happens and I know this from experience that I was actually somewhat involved in Memcache community almost 20 years ago, Facebook makes extensive use of Memcache to this day. So they created a project called McRouter. And McRouter sits in the front and it's a Memcache protocol router and it can help scale Memcache deployments and add all type of extra distributed functionality like connection pooling, flexible routing is a whole lot of things you can do with it. So what we're doing here is we've created this caching service around these two components where we're gonna have a McRouter operator that we're going to deploy and that is going to manage a single pod running McRouter and McRouter needs to know how many Memcache nodes does it have in its pool and be configured to know that. So as things happen like scaling events or errors happen and a Memcache pod goes away or we scale back up because we're thriving as a business and we have more data and we need more caching capabilities and we add more that service McRouter knows about what is in its pool and is being adjusted for dynamically on the cluster as these events are happening. There's no need to go in and manually reconfigure McRouter for what it has to work with and this will happen within seconds of each other. So with that, I'm gonna move back to, going to move back to my terminal, sorry, struggling with my multitasking right now. Okay. Sorry, that's why I'm trying to be here and feeling in better. So I think I've got all the questions answered. If you feel like your question is not answered, please re-ask or rephrase. If you feel like I've dismissed your question unnecessarily, I'm sorry, this isn't a place to talk about GitHub issues. Yeah, so I think one of the questions that might be remaining, I understand that you can use Ansible to manage Kate's clusters like KubeCuttle. Yes, you can. Yes. The Kate's module can allow you to do that. You can, Tim showed earlier in the presentation using the Ansible Lookup plugin to deploy any kind of Kubernetes resource via AMO Lookups. But there's probably some edge cases where the module and KubeCuttle might be out of sync right version-wise. So that's always possible, but edge case again. We're not saying Ansible can replace or Ansible should replace, it's just, yes, it's there. But the case module exists to make things like the Ansible operator possible. Right. But it is, and this is something I was just working on. This is still something that we're building up. I do have some of that sort of KubeCuttle replacement to show the use of templating here and reusability. Yeah, absolutely. Like I love to use Ansible for templating like a lot of KTML, right? Like if I need to change, like I use Ansible variables to manage resource limits, like a lot just because I'm always tweaking those. So like each application will have one big group bar and I'll manage it like that. All right. Okay. Cool. I'm going to move ahead and we can come back to more questions. Hopefully I get through this without incident. So what I have running here on my MacBook is a mini cube and that's all up and running and looking great. And actually what I should have done is show you that I have nothing up my sleeve so to speak here and nothing's there right now. So great. Let's give it something to do. So what I did here is just to speed up the time is I've taken a lot of the role, role binding service account deployment files that were generated and I put them into one file here, MacRouter operator. I'm going to run that here. And now we've created the custom resource definition, the role, the role binding, all that stuff I just said here. So if I go back, actually what I should have done is this. All right. That's a little more readable. There we go. We now see that just by running that it's deployed the operator. So now our operator is out there. It's waiting for something to do. So at this point, what I can do is let me see here. I'm trying to remember where I put things on my machine. Okay. Yes. So actually I should have shown you what's in this here. So what I do is I created a file at have time. And all I did here was I added the a custom resource called MacRouter. I given it the name, my cache service default namespace. It's going to have a memcache pool size of two pool sizes replicated instead of sharded. That was something I built in so that I can control what type of pool setup I'm using. So what I'm going to do now is use that file to deploy things. And what I get when I do that is, oh, what did I just do? Where'd it go? All right. There it goes. The needed a second. All right. So just by creating that custom resource, that one API call, we now see that we have a MacRouter instance up there, which is that first my cache service there, the second line, the second pod that's listed there. And I got two memcache nodes that came up right away and they created all the necessary services and deployment and replica sets out there to make this stuff work and run. So now we have an entire caching service ready to go. Now let's see here if I can get a little fancy and to some of the questions that were put out there. What I was trying to do earlier today is I started creating a playbook to templatize that last step that you saw here. So what I did was I created a, I took that file and I put in ginger to templates variables in there so I can control things and I could roll out multiple of these cache services under different names or namespaces or give it different pool sizes. So let's, in this case, try and do increase the size of the pool. So I'm just gonna do this in the command line. You could have done this using something like Ansible Tower or there's a whole bunch of other ways I could have been feeding this in, but for simplicity, I'm using the command line. So I have Ansible running on my machine. I'm gonna call the playbook that I wrote and I'm gonna tell it to give me a pool size of three will do. Let's see here if I get this right. So our playbook just ran and now we see, I probably got the variable wrong back here. Oh, right, I know what I did. Forgot that. That's more like it. All right, there's a feedback that something changed when I ran that command. So if I do that, now we see I have a third memcache node came up and McRouter has been informed that it now has a third memcache node to work with in its pool. And this is a generic thing about, I should have mentioned about operators in general, not just with Ansible here, but it can really help you enforce policy. So if you are like, we need three nodes in memcache and someone comes along and is doing things manually and doing things that they probably shouldn't be doing, what happens is that this can help enforce that. So let's say I kill someone, not following policy, I'm an operator and I go in there and I decide I wanna take down a pod at this point. What happens is the operator is monitoring and says, hey, I'm supposed to have three memcache nodes in my pool. That's the policy here. What happens is, is that we see that the pod, the one that was in that I deleted was brought right back up by the operator that's out there. So if we, so that's an operator in action. Let me, let's see if I can take it a little bit further. I guess we have some time for this, right Chris? Yeah, it's a 48, so you got about 12 minutes left. Okay, so let's say I need to deploy another one here. So I'm gonna do one called CNCF cache and I need to quickly roll out a separate caching server for whatever reasons we decided we wanna segment this and have the one that we already put up, but we need another one. So I'm gonna create one CNCF cache at this point and this is gonna run and give it a second, did I, there we go. We now see that we are getting these CNCF services coming up and now we've been able to repeatedly very quickly rolled out a new instance of our service. So we're seeing all of the, in this case, I didn't change the pool size. So we got the default of two memcache nodes coming right up and it's created the deployment and the service for that memcache. And again, this is all just using the basic templating that Ansible allows for in order to very quickly adjust and redeploy in a repeatable way what we're doing on our cluster. And then let's see the last thing I will show is let's say, okay, we use that, we wanna get rid of it. I created a variable called state and if I change that to absent, that should then bring down the stack. So we see now you see things have already disappeared and it's terminating all the parts of that application service that we had out there. Okay, so with that, I'm going to pause and see if there are any questions out there, Chris. So I fired off a bunch of answers. I'm not sure if they're fully qualified answers at this point, so let's just run back through them real quick. So the contents of the playbook that you just ran. Yes. So think, can you show the playbook? Oh, yes, sorry, I skipped that. I didn't mean to. No, that's fine, someone just asked. That's it there. So I could have wrote this a little more elegantly. I'm just running it on my local host and I'm calling, I only have one task in this case. It could have been a lot more and there are examples of it having more. I'm calling the Cates module. There I have the state where I'm, that variable that I used in the last step. And then I'm calling the template file that I had and feeding that in as the definition that goes off to the Cates API. So it was a pretty small basic playbook. I could have done more if it necessitated it, but in this case, one task was enough. Yeah, I think I said this every time you and I have been in the same room together talking about this. One of the hardest problems we have in demonstrating just operators in general is that like we're trying to do something that manages state and potentially like status and all these other different things, but package it up into this tiny little demo and that's oftentimes very hard to do even when we did a workshop at Ansible Fest a couple of weeks ago. It was a full day workshop and we still used the same McRouter example because explaining the entire concept and then demoing it is cumbersome to package in such a way that is demonstrable. So one more follow up question here or a couple more. So how does the operator come into play in the delete pod example? The stateful set controller would recreate that pod without the operator. So I don't think I quite understand the question perfectly. Do you by chance? Yeah, well, it's that the replication set would know to bring the pod back up itself too. So I guess what I should have done is changed the, I think if I would have changed the replication set the operator would have kicked in to snap it back to what it should have been. That probably would have been the, I believe the more accurate demo. Yeah, so okay, so here's some good ones. Where are the parameters stored that you changed with the playbook, right? Like you've got your variables here. Where do you have those far stored right now in this playbook? So I think what they mean is like in this case, I'm just speeding up. I was speeding them in through the command line. I could have put them in a file that was checked into to GitHub and fed them into the playbook, for example, or if you were using something like, plug it AWX or tower, you could have built them right into the job template itself. So there are a number of different places you could have done it. In this case, I was just doing it right on the command line. And then I used a lot of the default filter that it's built into GINJA too. So let me see here. Yeah, in theory, you can store these in a groupfars file like you would in an Ansible role. You could also potentially store them as config maps, I think. Yeah, I guess you could. I could, right? I didn't think that. If you somehow expanded the config map via groupfars, maybe. Yeah, they didn't think about trying that. You should probably come up with an example of that. Yeah, I haven't either, so. I'm an Ansible guy, so I think, you know, I'm still thinking most of the most Ansible terms. Bridging both worlds right now, so I haven't thought about that, but I will definitely go back to our team and be like, hey, is this possible? Yeah. Yeah, and so what I have here is I highlighted one of the areas where I had a default. So if I don't set that variable, it'll default to my cache service. If I would have taken a little bit more time and I probably should, I could have created this as a role and used the defaults file in there or var file in there to make this template a little less verbose itself and also make that easier to edit in the future, especially if I had a lot of different defaults. This is a pretty simple definition, but you could see ones that could have dozens of parameters and then be better off in a separate file. So Lucas is asking, can you give any examples for open source operators that are using Ansible right now? Operatorhub.io, I think has one. Yeah. I'm trying to think of other ones that are out in the wild that are open source. Yeah. Well, there are, this is something that we're working on and yeah, I'm trying to think of- Right, Ansible in the operator SDK is fairly new, right? So just the presence of it is within the past year. I mean, we just started it when I joined Red Hat last June. So yeah. So yeah, I mean, this is relatively brand new stuff. So the fact that it's not in production use cases or open source use cases is not surprising because we have done like just recently started the talking about it, if that makes sense. So Lucas does point out the operator framework has an awesome operators repo. I just found that the other day. I didn't even know it existed. So that's also very good. Yeah, there are a number of examples. Just say, you remind, you know, this question reminded me the question from Lucas reminded me that I had a couple more slides here. Yeah, definitely check out operatorhub.io. This is just ready to use operators that have been put out there by the community. They're not necessarily by Ansible. I think a lot of them are actually Golang, but still it's something you should be made aware of. And also as you create your own is that you could contribute them here. And the other couple other things to mention is for those of you looking for more information, some of these links already had in there, there's ansible.com slash operators is a good jumping off page to other resources. And then also there's the operator framework getting started guide that's out there. And yeah, here's a couple more resources for you to look at. I need to update this once I publish the McRouter one that I've been using here to one of our group repos. But there is an Etsy operator that was done by our team and they were developing the operator SDK to validate how far can we push this thing as this thing as feature complete as we want it to be. And then if you're looking for another simple operator very related to what we were just doing, there's a memcache operator that you can find here under the operator framework organization. We hope to put more examples out there. It's something that I need to work on personally, all these talks. Yeah, we just started the work to contribute operator framework to CNCF as an incubator project. So there's a lot of work going into this kind of all at once. So yeah, there's much, much, much more to be done. So some questions going back to your demo. Can you delete the deployment instead of the pod so that the actual operator is recreated or re-creates the pods? Okay, let me switch your terminals back. Let's see, in Ansible, can you filter for specific events in the watch's file? Yes, I did that one live, that was easy. So yeah, you can filter for any of those things with Ansible. Yeah, there's a number of features in the watch's file. You can do like finalizers too that I didn't go over. But if you go and look at the documentation under the operator SDK in the Ansible section, you'll see more options there. All right, sorry, so we were looking to... So someone's asking for examples of upgrading Memcache versions. I would say like there's tons of examples of upgrading Memcache Ansible Galaxy, galaxy.ansible.com. That is the place for people to share roles and you can take those roles and then reuse them in your Ansible operator. And that would be, I'll just close the Q&A window. That would be the best way to like figure out how to do that with an Ansible operator because I don't have an example to do that right now. Yes, and that's done. How do Kate's secrets and Ansible vault secrets play along together? They really don't, they're kind of two different things. You can use them both in this use case, but I would recommend sticking with the Kate's secrets for now and using those inside your cluster as opposed to trying to expand Ansible vault things on the fly. All right, let me, I'm gonna give this a whirl here and try to delete the CNCF cache deployment and the operator should bring that back. Let me see here. I've never tried this before so this should be really interesting. There you go. There it is, came back. All right, can you, so I really like, we need multi-threaded Q&A in Zoom but we are at the top of the hour. Everybody, I'm very sorry if I didn't get to your questions. Please feel free to reach out to Tim or myself, via Tim Twitter, GitHub, email and we will be sharing this webinar at cncf.io slash webinars and thank you. Come see us at KubeCon. Yeah, come see us at KubeCon. I'll be in the Red Hat booth presenting. I'll also be presenting at the, was it the Cloud Native Rejects Conference the weekend before KubeCon. Yes, I will also be at KubeCon as well. We look forward to seeing all of you at the next CNCF webinar. Thanks for joining us today. Have a good one. Thanks everyone.