 Hi everybody, welcome to the Kubernetes Operator Framework Workshop. This is presented by Red Hat. My name is Matt Dorn. I am a principal engineer at Red Hat. I am doing this presentation with Michael Rivnack, senior principal software engineer at Red Hat. We also have two other Red Hatters here to help out you all. Senior engineers Rose, Crisp, and Melvin Hillsman are in the chat. Any questions you have throughout this presentation and workshop, feel free to put them into the chat. And Rose and Melvin will be happy to help you. Hopefully you're in the right place. Today we're going to learn about the Operator Framework and how it can help you build operators. Before we get started, we just want to take a little bit of a poll. If you can just put a plus one in the chat. If you have access to the chat to let us know where you're coming from here today. So how many people here use Kubernetes and OpenShift regularly? If you use Kubernetes or OpenShift regularly, go ahead and put a plus one in the chat. If you use those regularly, go ahead and put a plus one in the chat there. Just so we can get an idea so we all know where you're coming from with your experience. And I'm going to try to see if I can get the chat on my phone so I can see if I can see the responses. But that will at least give our Red Hatters an idea of where you're coming from. How about anyone developing Golang? Do you have any Golang developers here today? Any Gophers? Any Golang developers? Let's put a plus one in the chat. How about use Ansible regularly? Anyone here use Ansible? Ansible, plus one in the chat if you use Ansible. Anyone here ever attempted to build a custom Kubernetes controller from scratch? Anyone building a custom Kubernetes controller from scratch? Which can take a lot of work. We're going to learn how the operator framework makes this process easy for you. And then plus one in the chat, if anyone has tried the operator SDK, Ansible operator or Helm app operator yet. Anyone ever played around with these? If you played around with them, go ahead and put a plus one in the chat. So our goal here in this session is to help you succeed with our tools. So in this entry level session, we're going to be exploring the operator framework, the operator framework. Now, before we get started, we just want to leave you, we want to go ahead and give you, first of all, this is a workshop, you're going to be getting the environments right after the introduction, and we're going to ensure that you can all connect to those environments. Right now, we're going to do a brief intro. You're also going to be getting the slides and we're going to show you where you can get those slides. These are some of the support channels that you all can use for getting help on Kubernetes Slack, it's Kubernetes operators. We also have Google groups. And we have the open shift comments operator framework, which is every third Tuesday of the month at 9am Pacific. And again, you'll be getting a copy of all this in the slides. So don't worry if you miss it. So let's go ahead and start out this talk with what an operator actually is. So if you go ahead and look up a Kubernetes operator, you'll get kind of like this formal definition that says an operator represents human operational knowledge in software to reliably manage an application. Right. But what does that mean? What does that mean? Well, we actually go ahead and say what does that mean? I mean, this is kind of a generic, you know, definition. Like what are we actually talking about here to really know what we're talking about here? We're going to go ahead and we are going to go back a few years. Back into 2016 on the core OS blog for many of you that may not know what core OS is core OS was a startup in San Francisco that specialized in an operating system called container Linux. And container Linux's goal was to just run containers. That's all that it just ran containers and it was very secure. It had auto updates turned on by default. There was no package manager on the host because the assumption was that if you wanted to run something, you ran it via container. And as you all may not know, Red Hat acquired core OS a few years ago. So on that blog, Brandon Phillips, CTO of core OS wrote that there's this new type of software called operators, this new type of software called operators. And it says the operator built upon is built upon the basic Kubernetes resource controller and application specific knowledge concept that this thing is a it's a type of software that builds on these three concepts. So what are these three things of Kubernetes resource or specifically a custom resource. Okay. Controller controller Kubernetes controller we're going to talk about what that is in here. And then knowledge knowledge. So a Kubernetes resource. A controller and knowledge well let's talk about what each one of these are. What do we mean by resources and Kubernetes. Well, first of all, you have the concept of a pod right that's the, the single unit the lowest unit that we use for deploying our workloads is the pod what is the pod. The pod is really one container or multiple containers, right, we know that containers in a pod shared the same network stack, they should, they share the same host name, they share the same inter process communication, right. And also we have the config map config map is great for storing application data right it's great for storing information that we can attach could be a config file could be something of that sort that we attach to our pod right and thus give access our containers access to to consume. What else do we have here we have a route a route gives us access to our workloads that are running within the cluster without routes, also known as ingress for those you that made us use regular Kubernetes. Our users would not be able to get access to our apps so they allow that type of accessibility from outside the cluster these are all examples of resources and Kubernetes right now what are controllers. Well, these are also considered resources, but they're resources that are powered by something called a controller. What is a controller will think about the replica set for those you that may be unfamiliar with this, we understand that the replica set allows us to specify the number of pods that we want running right you put the number of pods you want our replicas because we know that two is one and one is none we want high availability right what the ability to scale. We know that if something goes down with the easily come up for our stateless workloads. That's a replica set right that's a replica set the controller the replica set controller ensures that that number is always running and we're going to learn about that coming up here. What about the daemon set controller what is the daemon set controller doing when a user goes and creates a daemon set and ensures that a pod is running on every node in the environment the controller ensures that right. So when a pod dies that will get respond on that node or when an administrator adds a new plus or adds a new node to that environment if you bear metal maybe a AWS instance that a pod will be spawned on there. And then we have the deployment what is the deployment the deployment is great because it allows for server side rollouts and deployments ensure that if one of our replica sets go down. They get recreated. So there's a deployment controller that is always watching the Kubernetes API and ensuring that so these are the controllers and there's many of them running simultaneously in your Kubernetes cluster right at all times. And then we have the domain or application specific knowledge the last requirement for an operator and this is very important because the whole idea of an operator is that it gets rid of all the manual. Or work or toil that you have when managing your app. All the headaches that you get in the middle of the night when you have to wake up and go do something go maybe restart a service or figure out why something is broken. Well the operator does that because you have coded right all of this information into it, not just the installation and the ensuring that it gets running but the scaling properly and what we mean by scaling properly. We're talking about the fact that if you're running a stateful workload that just coming up and down and blindly spinning up a new pod when one dies similar to a replica set may not be appropriate. Right. You may have a particular cadence or rhythm to the way that your database pods should come up. You are going to program that because only you know how your app should properly scale. You're going to determine that self healing when something goes wrong. Maybe a database becomes corrupted or something like that cleaning up when it actually goes and creates resources updating to a new version right of your application that's running. Backing up because we know that if your if your information is just stored on that cluster and that cluster gets destroyed, you're out of luck so you won't have offsite backups and of course if you're doing backups you want to do restore and all of that good stuff. So this right here those you who know and love Kubernetes I know that you've all have seen this many times right this is the resource schema. This is the object the fun this kind of fundamental unit that we use for interacting and creating things in Kubernetes right via the API. It consists of four parts right the first part is called the GBK aka the type meta. This is where you specify the group version and kind. We have a group version in this case extensions v1 beta one because it's a way to disambiguate different kinds that may be similar. Right, so in this case extensions v1 beta one is the API endpoint for the kind of replica set. We also have metadata aka object meta. This is where you put the name of your object and also the namespace where it's going to reside. We know that namespaces aka projects and the open shift world are great for doing multi tenancy right you may have a namespace for the human resources or development or production. It's a way to separate your objects. We also have another section here called the spec. This is really important because we also know this as the declaration. This is where you as a user go and declare what you want. In this case, we're saying we want five replicas that all use an engine x image and have a label that's app equals engine x and of course there's the status section as regular users. We don't really modify the status section but it's there to tell us what's going on and to tell us what the current state of the environment is you're going to learn that when you go and build an operator. It's your job to go and update the status. Okay, it's your job to go and update the status. That's when you're building a controller. A regular user is not going to update this but again the controller that we're going to build in here is going to do that to provide information to our users. So remember we said that one of the requirements of an operator is that an operator takes advantage of resources. Well, specifically it takes advantage of something called custom resource definitions, a very, very powerful feature in Kubernetes. CRDs allow us to extend the Kubernetes API. For example, this right here is an example of Grafana, kind Grafana. We know that the kind Grafana, you know Grafana, it's used to visualize metric data right. So we can get a nice visualization typically for monitoring. This is not something you get out of the box with Kubernetes. But what you can do is build your very own custom resource right. In this case someone has built one called Grafana that allows you to put in all the details for your Grafana application whatever it may be. So this is a complete customization that a user has provided that has become a resource on the API and they've done this without even needing to bring down the API source code and recompile it. Really, really powerful. So let's show you how you would do this. You can actually extend the Kubernetes API by creating our very own object resource via something called CRD or custom resource definition. So if you look at this process, it should make sense. Okay. First of all, if we actually go we say cat my new CRD.yaml. If we say cat my new CRD.yaml, we can actually go and see that we have a custom resource definition here. This is how you create a custom resource definition. This is how you create a new endpoint on the API. You're going to say I want to create a brand new endpoint called MySQL. Do you see how we're putting MySQL in here? It's probably pretty easy to guess what I'm trying to do. I want our users to be able to create a Kubernetes native object called MySQL, right? And when we do this, let's go ahead and create it. We notice that via the CLI, if we then run this command, and for those of you who don't know what OC is, OC is the client for OpenShift. It's really very comparable to Kube CTL, which is the client for Kubernetes that sends API calls for you. They're very, very similar. OC just happens to have some extra features for OpenShift, but they both do the same thing. They both send API calls to the Kubernetes API. And as you can see here, as we only do OC get CRD, this confirms that we've created the MySQL custom resource definition, okay? This really goes and creates the MySQL custom resource definition. Let's now actually go verify our new MySQL resource and object. We're going to go ahead and say OC get MySQL. And after we do that, notice how it says no resources found. This is actually a good thing. This tells us that the API endpoint exists, but now we just have to create a MySQL object. For example, MySQL example or MySQL number one. So let's go ahead and create that object. So if we actually go and we say, okay, here's our new object. Notice that we now have the ability to specify the group that we originally created and the kind MySQL. And we can put in a bunch of values, key values in our declaration. Now we haven't showed you what specific key values are where you would tell your user to put in here or what we would tell your user to put in here. But the reality is that the reality behind this is that you can actually create a man page, right? You can actually go and create a man page that will assist you using the explain command with what telling your users what to put inside here. And we can talk about that later. So let's go ahead and create this MySQL object. After we create the MySQL object and we say OC get MySQL, notice that now we have our WordPress object. And we actually can output that in the ML and see that it's persisted on the API. So we now have an object on the API. But just so you know, nothing has actually been created in our Kubernetes environment yet because we don't have a controller. So it's just the fact that we've created something on the API. That right there is a custom resource and a custom resource needs a controller to act upon its presence. So what does a Kubernetes controller do now that we have an object on the API? Well, that controller is going to sit there inside of our cluster or just so you know, a controller can also run outside the cluster. And it's going to look at the current state of the cluster. It's going to watch the API and look for a MySQL declaration or any declaration that it chooses to watch. Once it receives an alert, aka an event that that actually has been created. Okay, actually that has been created. Then what's going to happen is we're going to compare the current state to the desired state. So we're going to go ahead and right in this class, we're going to build an operator that looks at the current state of the cluster and compares that to the declaration. Okay. After that, we're going to go ahead and act. We're going to perform all the actions necessary to make the current state meet the desired state. And this controller gets triggered anytime there is an add update or I should say create update and delete event on the things that we're watching. And that's a Kubernetes controller. We're going to build one in here. We're going to get the access to the slides and the learning environments right after this introduction. We're going to make sure we understand these concepts before we dive into that. So the replica sets in action. First of all, what we want to do is you want to go ahead and we are going to actually what I'm going to do here. I'm going to interrupt this real quick just to make sure we all have this. I know that some people may want to be following along with the slides and so forth. So I just want to provide this to you now. I'm just going to interrupt this really quickly. If you open up your browser and red hatters that are in the room feel free to share out this link. It's workshop. That poor was trained on me. Workshop. poor was trained on me will give you the learning environments momentarily. But if you all want to follow along with the slides. If you go to workshop. poor was trained on me. And you go to the support tab here. Right. We have your you have the slide here slides here you can download. Just so you know that content or as you have to go somewhere right now you have to run maybe you can't make you try to make another presentation but pretty grab the slides there. Also we have a copy of an ebook that was written by some red hatters here on Kubernetes operators. We have a link to operator hub which will show you momentarily and some other links here so I just wanted to make sure we had this link for you to download the content so red hatters feel free that are in the chat feel free to hand out that link workshop that poor was trained. Not me. Thank you. So how does a replica set controller work. How does a replica set controller work. Well replica set controller. If you go and create the replica set right what does it do. Well you say that you may want one replica. So then what it does is it goes and creates one pod right and notice that it uses labels to ensure that it's keeping track of the right pod. You're selecting one app equals my first app and the pod has app equals my first app. If we then go and we scale out our number of replicas to three. If we scale out our number of replicas to three. The replica set will then spin up two more pods because our controller sees that the declaration has changed. So the left of the set is what we call the primary resource and those pods are the secondary resources the pods are the secondary resources. So what are owner references what are owner references well owner references when the replica set goes and creates something it puts an owner reference on it and the great thing about owner references is that owner references make it so when we go and we delete our replica set. The pods will also get deleted. And these are what we call our dependence or children and the way it works is that if an owner reference if references the owner in this case the replica set. If that replica set goes away and gets deleted. Guess what the pod is not allowed to exist. So if the parent goes away. The child is not allowed to exist. And this is built into Kubernetes built into Kubernetes. This is how we do garbage collection. You can imagine what it would be like if you had to delete the replica set and then go and find all the pods and control it and delete those that would be really annoying right. So how does the replica set control loop work. Well, once you create a replica set. The first thing the controller is going to do is find pods matching the label selector. Right. And then it's going to compare the match versus desired count and if there's too few pods. Then what we're going to do is create additional pods from the current template. If there's too many pods. We're going to delete the excess number. And if there's just enough. Then we're not going to do anything. It's going to be fine. Everything's going to level out. So check this out. We have a really nice visual for you seeing how exactly this is going to work. So your controller watches for two things. It's going to watch for the replica set for anytime someone creates a replica set. And it's going to watch for pods that are owned by that replica set. So it's watching for two things the replica set and pods that are owned by the replica set. Right after someone creates a replica set what happens it kicks off an event. Once the event gets created the replica set controller is going to get the current state. And it's going to list all pods by the label specified here at this case we're going to have zero we don't have any pods created yet right and zero is less than the declaration of three. So what is the pod logic right or that auto that automated human knowledge going to ask for it's going to say create a pod please. And after we create that pod, the very action of creating a pod that is owned by the replica set creates an event that the controller sees you see this because the the controller is always also watching for pod that are owned by the replica set. What happens is it will then list pods by label. So it would be one at this point, less than the number of replicas yes it is we still don't have three. So then we create another pod which creates another event. In which case we list all of ours we have two is less than three. Then finally, once we have three meeting once the actual state meets the declared state we're happy. Do you see that. What about this situation here, what about when we delete a pod, when we delete a pod. We then have to, and we're going to create another one. So controllers are always watching the API to ensure that the declared state in our object meets the actual state. Do you see that. So going back to our my SQL example, we need a controller. We need a controller because without a controller everything that's quiet in our cluster. Right, we just have an object on the API. So we need a custom controller to notice the new database object and act and do something. But think about this. I think it were as simple as just creating or reading or updating or deleting a my SQL object my SQL that you as you all may know it being a my SQL admin is very intense right there's a lot of things and tasks and activities that you need to know to maintain that a lot of operational knowledge. And that's all stuff that you're going to have to build in to your operator because you don't want to have to manually control it. You don't have to have to any manual intervention. So to recap, custom resource definitions, custom controllers and your knowledge, equal operators. Okay, and why do they matter what operators matter, because we're trying to build an as a service platform experience on Kubernetes. We want to build an ecosystem of software on Kubernetes that can be as easy and safe and reliable to use and operate as a cloud service, low touch, remotely managed one click updates. For example, for example, what we actually do here is we have. You can see from the dashboard, you can deploy the at CD operator or the Prometheus operator or the vault operator, right, you can deploy these. And this will, you can easily install these operators right from the dashboard and it kind of gives you that easy experience that you would experience on something like AWS or Azure or something like that. And it's really easy to deploy an operator in a Kubernetes environment. So how do you create your own operator well life before the operator SDK that is found in the operator framework. If only we're simple as maybe defining some resources that you wanted to watch, and then writing some like pseudo code here to get the current state and compare to the desired state. It wasn't that easy because custom operators require many building blocks and blur play code. First of all, you have to go and research and download tools to interact with the API. These are the eight, these are the client libraries, maybe some of you have used these but these are the ways of programmatically interacting with Kubernetes and go Python or Java. You have to learn how these work first, then you have to have knowledge of informers or shared informers for object caching event handling. Why do you have to know about caching, because think about this if you're building a controller and that controller is watching the API, and there's tons of controllers watching the API. You don't want to intensely tax the API and eat up its resources so we have caching that our controllers do to make it less intense on the API server. You previously had to communicate desired state and actual state via annotations, track cumulative resources, test scaffolding and repo organization, so much stuff you had to do to build a custom controller. Just an example of a custom controller that was built a while back using client go, the Kubernetes client library for go. It's a lot of work. So that's why we need an easier way to create operators, and we need an easier way to manage them. And the operator framework, which the operator SDK is found in is a great way of creating operators in a really cool way. With operator SDK, you can build an operator with go lang, right, and we set up all the go code for you so all you have to do is write specifically your logic and go. So all the surrounding code boilerplate code is already there for you for you. Ansible. So those of you who know and love Ansible, you can use your existing playbooks existing modules roles collections to actually go and create resources not only within Kubernetes, but you can use those to create resources outside the cluster as well. And you're going to learn that Ansible operator when Michael talks about that with you is powered by the Kubernetes modules for Ansible, which allow you to create tasks, the way you know and love with Ansible that actually create resources in Kubernetes really cool stuff. And then also helm, the helm operator allows you to take all your existing helm charts of those you're familiar with helm and convert those to operators as well. This is an example of the operator capability model. As you can see, if you convert an existing helm chart to an operator, it takes you to basic install or seamless upgrades both Ansible and go. It will take you all the way to the ultimate operator autopilot where you have horizontal vertical auto scaling, auto healing, abnormal detection, all these great things that you can build into your operator. Also, there's the operator lifecycle manager operator lifecycle manager runs on any Kubernetes cluster well, whether it's vanilla Kubernetes or its open ship. It enables cluster admins to manage operators because as you all know, creating a custom resource definition. That's a pretty powerful privilege right you're modifying the API with OLM. You may have a user who's only locked to a specific project or namespace right because you don't want them having access to the entire cluster. A user who may just have limitations or may just have rights to a particular project or namespace will still have the ability to go and create an operator. So this is very, very powerful stuff. OLM handles that entire lifecycle for you and allows a user who may not have cluster right privileges to actually go and install operators. We also have the operator hub.io. This is like an app store for operators. All these operators run on any vanilla Kubernetes. This has nothing to do with red hat or open shift in the context of being proprietary. This is upstream community driven. It runs on any Kubernetes cluster. I recommend you all check it out. And if you're interested in getting your operator into the operator hub, please come talk to us. I will be in the virtual red hat booth after this presentation, and I'm happy to give you all the guidance you need to actually get your operator into the operator hub. So with that said, I want us all to go ahead and check out our learn our training environments because we're all going to build an operator right now. We're going to build an operator right now. We're going to go to learn.openshift.com slash training learn.openshift.com slash training. After you go to learn.openshift.com slash training. You'll see a web page that looks like this and what you want to then go do is click on. Operator SDK and after you click on operator SDK, you're going to click on operator SDK would go now these modules just so you know these modules will be around. After the training and we're going to give you a link to access them so if you're having issues and for some reason it's not working because we are in capacity at 100 because we were originally we're just set up to have 100 support 100 individuals here 110 days. Don't worry, you will have access to these but if you can we're going to do the operator SDK would go. So you're going to click on start scenario there and then we're not going to do this right this moment I just want to show you what the objective is we're going to talk about it, but it's going to make sure you all have access. You may have a situation where it says provisioning environment or creating environment just wait for that to complete. Melvin Rose and Michael the Red Haters in the chat are happy to help you if you all are having issues. Okay, they're happy to help you with this. So we'll actually get start doing this in just a moment. I'm just going to continue on a little bit here. So we have our workshop that poros train me and we also have our learn that openshift.com slash training. And again, you can get your Kubernetes operator Red Hat certified and featured in the open shift operator hub or operator hub that I owe today. Please come talk to us. I'll be in the virtual booth for Red Hat after this session. Please come talk to me. I'm happy to get your operator into there so our users can check it out. So what are we going to do here? We're going to build an operator and this operator is called the pod set operator challenge. So let's go ahead and create a simple operator. Let's go ahead and create a simple operator. We're going to call this operator a pod set. Now, first of all, we want to let you know that when you go and create an operator. Your operator is going to when it goes and installs that application, right? It can it's going to create a lot of things. It may go and create a deployment. And then after it creates a deployment, it may go expose that deployment as a service, right? And then after it expose that as a service or maybe config maps or maybe routes and ingresses that make it get created. You can certainly do that. Why would you do that? Because the deployment in that case is a good fit for managing your pods. Think about the controllers you have built into Kubernetes. You have deployment. You have stateful sets. You have David sets. You have a variety of controllers to choose from, right? That you want to create. If none of those are a good fit for your operator, then what you can do is go and create your own pod management logic if you want to. And that's what we're going to do here. We're going to create this thing called a pod set. And the pod set is going to actually not only is it going to watch for the creation of something called a pod set. It is also going to go and create its own pod maintenance logic. We're not going to rely on a replica set. We're not going to rely on a David set. We're not going to rely on a deployment to manage our pods. We're going to build the logic into the controller. And so it's a simple controller that manages pods for us. As you can see here, the whole idea is that you're going to put in the number of pods that you want here. You're going to say that you want to have like, you know, four pods, five pods, whatever you want, and it will spin up that number of pods and it will spin up that number of pods. After we actually go and spin up that number of pods, the names of the pods that currently manages will be in the pod name section and status. So that's what we want. Very, very simple. You don't have to put anything else inside here. If you change it to one, then the pod set will automatically go and scale down to one. So these are the requirements. The requirements are the replicas and the status. That's it. Replicas and pod names. That's it. Some hints before we actually do this are that, first of all, you identify what you want to watch. We're going to be watching for pod sets. And we're going to be watching for pods that are owned by the pod set. Those two things. We also have to make sure we set controller reference or owner references. Remember, I told you about owner references and what they are. They're there to actually control what gets to control what gets cleaned up and what gets deleted. That's where owner references come in and we're going to have a special function from the controller runtime library to allow that. And then of course we have to use labels to keep track of which pods we want to manage. So let's go ahead and do this. I'm going to go ahead and open up my environment here. Now, just so you know, after we actually go and create this actually after we go and do a go based operator, Michael is going to walk through creating an Ansible based operator. So first we're going to build an operator with the operator SDK would go and then Michael is going to show you how you can build an operator with Ansible. Right. So if you hang out after the go based one, maybe you all aren't going developers. This is a little too much for you. Just stick around and Michael will talk about building one with the Ansible. So the first thing that we're going to do here in our exercises is we are going to go create a new project called my project. Now we know that in the open shift world a project is the same thing has a namespace same thing. So then after doing that, we're going to go and create a new directory in our go path. Now what's the go path for those of you that aren't go furs are not into the go programming language. That's sort of our sandbox for our go projects that we built right for our go programs. After we're inside there. We're going to go to this red hat directory and we're going to look how simple this command is to create an operator operator dash SDK new pod set dash operator dash dash type equals go look how easy that is. Right now if you wanted to build an Ansible one all you'd have to do is dash dash type equals Ansible. And what's happening right now is it's going and creating all the boilerplate code. Notice how you haven't told that what kind of operator you want to create. You haven't told it what you want your customer resource definition to be you haven't given any that information. This is just the default boilerplate code. And if you are wondering what is really happening here while waiting for this what's going on behind the scenes. What's going on is these are all your packages your dependencies that are being imported into your go code. This is going out to the net and grabbing all those packages for you. Okay. This is using depth for dependency management for our dependency management for our packages that we may depend on in our code. Okay. So now that we have our boilerplate code we're going to drop in to our directory. Let's go ahead and CD into the pod set operator directory. Now if I do LS inside that directory notice that I have some default code here and I'm sorry we're not using depth we're using go modules we're using go modules. We're okay. So what we're doing here is with our mod files here. This is a way for us to actually tie a go package to a specific branch or version right so we can actually ensure we're getting the right version of that dependency. Also you here have a deployed directory it's going to have some manifest into it. So this is just our boilerplate code right now. If we go into step two to everyone we are on step two of seven on our exercise here. Here comes the very very powerful command after we've created our boilerplate code we are now going to tell our operator something we're going to tell our operator. Please create this custom resource definition and we do that via the add API command we say operator SDK at API dash dash API version and we put in the version here at dot example dot com slash v1 alpha one dash dash pod set. So this is where we're getting customizable right we're saying we want this to be our group version and we want this to be our kind so go ahead and run that command so we can see what happens. So as you can see it's creating these go files for us automatically specifically one that we're going to look at is called the types dot go we'll look at that in just a moment. But for right now what you need to understand is that it's taking this information that we've given it and creating all the code that will reference the fact that we want a custom resource definition with these parameters again. We have not provided any other information to our operator we haven't told it what it's going to do how it's going to act any other information in the CRD this is it. So now that we've created this notice how it gives us a CRD automatically you see our customer resource definition it automatically created this for us based off the information we provided. I'll tell you right now that there's a whole bunch of really cool validation information that we can put inside here and we'll show you an easy way we can do that. But for right now this is pretty generic but it's something right it's it's something so let's go ahead and click continue and go to step three. Now the first step that we're going to have to go ahead and do is update our types dot go file. Now what the types dot go file is is an opportunity for us the operator authors to talk about what we want users to put inside the spec. Right. And what we want in the status and the reason why we're doing this is because we need to create a predictable schema. What do I mean by that we need to create a predictable schema. What I mean is that if our code is going to be creating human operational knowledge and look at the declaration of the spec and status that users are going to create if we're going to grab data and values from this declaration. We need a predictable schema to interact with those values within our code. Well how do we do that how do we do that we do that by first customizing our types dot go file. So as you can see what we're going to do and by the way you don't have to copy and paste any of this stuff in we can actually go and run this command right here and it will automatically update the file. How easy is that if only coding were that easy right well that would be very fun coding was like that in the real world but here's the idea. Right. Here's the idea. The idea here is that you are going to go and notice that we're setting in the spec section. We're doing a struct. First of all what's a struct. It's a custom type and go a custom type that we're creating we're inventing this. We're going to say we want a replicas field that's a 32 bit integer. And this right here this JSON tag is going to do the marshaling and demarshaling of the JSON that the user creates or YAML right which is the super set of JSON. And we're going to say look in the replica section and then when you grab the number let's say three put that as that value so we can manipulate it in our code. Look at the status section here memory told you how to have pod names. Let's go ahead and make sure that that's an array of strings and that we get those values and can publish to those values based off the pod names within the actual object. Right. So then after we do that we then create or run the generate case command and many of you may be thinking what is this generate case command. The generate case command runs the deep copy code generators that ensure that you can reference these these values and these keys and fields in your code. After doing that we bench this is a really really awesome command. Look at this command generate CRDs when you run this command generate CRDs. It is smart enough to go look at your types that go file and notice that you had these particular values in there and it says okay we're going to go ahead and we're going to create a CRD that tells you that replicas is required you see that so validation will start occurring on the API level. This is really really handy for just adding that API validation so that users can't put in bad values into their declarations. And then what we want to do is create our CRD in our cluster because the operator cannot start unless the CRD is running. The CRD is available in other words an operator cannot watch an API endpoint that doesn't have doesn't exist. So we want to make sure that the podset operator endpoint exists. Okay, after doing that, we're going to go ahead and make sure the CRD is available and it is we see the CRD there. Right. And then after doing that we're going to go ahead and add the new controller. We don't have a controller yet right so at this point there'd be nothing to manipulate our declaration. So how do we add a controller to our project you just run the add controller command. And what this tells our code is it says please create some boilerplate logic, right some boilerplate logic to manipulate your code, but we don't want boilerplate logic we want to customize our very own controller to actually go and create a podset. Now what I'm going to do here is I'm just going to copy what I want you to do is scroll down here. And I want you to just click this button to we make sure it gets updated. That's going to update your code and I'm going to talk about this code briefly, and then we're going to run our podset. So I'm going to take this here, this logic, and I'm just going to go ahead and open it up in Microsoft code so that we, my ID so that we can get a better idea of what it looks like here. Yeah, open a new window to make it more clear. Okay, so let's talk about what's actually happening here what's actually happening inside of our project here. So first of all, the first thing to pay attention to and look I do realize we're very short on we only have an hour and a half here so we're going to go into this quickly but again all the resources will be presented to you will be here after to answer any questions. The first thing we're going to watch for our podsets right because podset is the primary resource and we're going to watch for podsets, and we're also going to watch for pods that are owned by a podset. So we care about two things we care about watching for the creation update and delete of a podset and the creation delete and update of a pod that is owned by a podset. And that gets if one of these gets created somebody creates a podset for example it triggers what's known as the reconciler. The first thing the reconciler does is it gets a copy using the client. And you may be wondering what's the client do the client just sends restful API calls to the coup net is API. It's going to get a fresh copy of the declaration. What's the declaration having it, it has the number of replicas that I want. After getting a fresh copy of that what it's going to see is that we want three replicas let's say for example, and then it's our chance to get the actual state of the environment by listening all pods owned by the podset instance. We're going to go ahead and list all pods owned by the podset instance. How do we do that. Well it's really easy we're just going to do that by ensuring that we look for pods that meet this criteria that have this label, the name of the podset with this version. Right. And we're just going to run a list. For those of you who use Kubernetes day in and day out running this function here this method is the same thing as doing an OC get pods with a filter on it right we're passing list options and enables us to filter based off the labels. After we get the number of pods we can range and go ranging is just looping over this list of pods that come back. And we're going to only count the pods that are in pod running and pod pending. We don't want to count pods that are being deleted because those are being deleted right. If they have a deletion timestamp not equal to nil we don't count those they don't count because they're not really owned by us are going away they're going bye bye right. So we're going to put count on the number of pods using length and while we're at it we're going to grab their names just so we can actually use that for the status or we're going to range over their names. We're then going to use the deep equal package which does comparisons to compare the status in my declaration to the actual names. Right. And what we'll do is this will automatically send a status update to get the pod names into the object. Of course, here comes the most important part of all, comparing the current number of pods to the declaration. This is our comparison to the actual state of the cluster, which is here to our declared state. And if the number available is greater than the number in the replicas we're going to scale down, but the more likely scenario is that the number available is going to be less than. In the declaration, we're going to go ahead and scale up. Do you see that. And what's it going to do when it scales up. It's going to create a brand new pod notice it's just a busy box pod, but notice this right here generate name generate name ensures that every time it creates a pod it will actually put a unique hash at the end. That unique hash makes it so we can create multiple pods in the same namespace, because I know that all you Kubernetes experts out there know that you can't have multiple pods with the same name inside one namespace. So generate name will actually ensure that it puts a unique hash at the end. Maybe some of you have actually seen this before. So, with that said, let's go ahead and create this thing. What we're going to do is we're going to go ahead and click on continue, and then we're going to go ahead and run the operator inside. Okay, I'm not sure what point I went I muted. I think we lost you just about 10 seconds ago we got. Okay, cool. Okay. All right, so what I'm doing here, what I'm doing here is I am running the controller within the cluster. I am running the control I am running the controller within, excuse me, excuse me, excuse me. I'm running my controller outside of the cluster using the operator SDK run dash dash local command it's using my code config in my own directory. And it's watching me my project command and watching my project namespace, and it's, it's waiting for the creation of my kind pod set. So this is a great command to run. I'm doing development. Let's imagine you have to go and make a code change. All you have to do is run this command it will compile this is kind of like doing like a go run CMD those you that are gopher's out there, right you're just compiling the creating the command and running it real quick right. Just to test your code. So what are we waiting for here as it spins up or waiting for the creation of the kind pod set. Let's go ahead and create it we're going to open up a new terminal. What we're going to do inside of our new terminal is make sure that our custom resource has replica set. So if you click that it will make sure it has three set for the replicas. And then here comes the moment of truth. I'm going to create my custom resource and I should see my reconciler my controller getting kicked off. Okay, actually make sure I'm on the, in the right spot here for me one second. Hey Matt, quick question. So this previous one operator that is run within the cluster only right I mean I see it's within the namespace my project. Yes, this is running outside this is running outside the cluster. We happen to be inside of a cluster here we know the typical way that you would run a controller the more formal way if you will the way you would do it in production is by actually compiling and building an image right and putting that image into like something like a deployment and putting that into your cluster and operator SDK features all of the tools to actually do that for you. But we're in a arena we're trying to just kind of show you in this environment how quick it is to just run the controller outside of the cluster but typically in production, you would run it inside. Does that make sense. But I'm still confused I see here it is specifying hyphen hyphen namespace my project so I assume it is within the namespace which is within the cluster. How is it outside the cluster. Because you know when you use the COOP CTL command line are you familiar with COOP CTL when you use right so when you use COOP CTL and you say OC get pods right. We know that yours probably typically scope to a particular namespace, or you can pass a namespace flag. So this is no different that kind of running the COOP CTL outside the cluster. So in this case, so whether you're running a controller outside the cluster or inside the cluster, you're still hitting the API right as long as that API endpoint is externally accessible. Okay, got it. Okay, just the local binary it is running the local Yes, this is just this is just a command that is used for development purposes to ensure that you can quickly it's just for it's not something you would do in production. It's just for development. So if you're trying to test your code, you want to just compile your binary and have it point to an API endpoint that you can watch and ensure that your logic works. Okay, thank you. Sure, no problem. So now that we've created the custom resource. Now that we've created the custom resource notice that our controller reconciler was kicked off. So now if we actually go and do OC get podset, we can see that we have a podset. And if we say OC get podset example podset dash O YAML. Notice how we have three and this has the pod names we currently own. Let's make sure those pods are there. Yes, they are. Let's make sure those pods have an owner references on them. Yes, they do. Right. Yes, they do have owner references on them right there. Right. So if I delete my podset, it should go away. What we're going to go ahead and do is we're just going to go ahead and in this case here. We're going to go and let me just make sure let's go ahead and do this. Let's go ahead and change the number of pods to five and see if my podset responds accordingly. Yes, it does. My podset is now spinning up by pod. If I look at my controller, I can see that my controller is reacting to those changes. Do you see that now if I actually go and delete my podset completely. My expectation is that the podset should delete the pods as well. Let's go ahead and try it. Yep. Notice that by just deleting the podset. It also deletes the pods because the pods have owner references on them. And the pods are not allowed to exist if their owner doesn't exist. And at this point, the example podset owner does not exist. So the pods are going away. So I know that was very, very quick. And we want to, we definitely want to answer your questions, but we ask you to put your questions in the chat. And we will happy to assist you also will be in the virtual booth after this presentation. I'm now going to go ahead and hand it over to Michael, who will now show you how easy it is to build an operator with Ansible. Hey everyone, stand by just a moment. I'll get the right screen shared. All right. Anybody can't see my signs. Somebody shout at me. Let's get rolling. We're going to talk about Ansible and we're going to make an operator using Ansible and see how good a fit. Ansible really is for automating Kubernetes in general in for creating an operator. It's really, really wonderful way with a lot of particular advantages I think you're going to like. So let's look at the Ansible K8S module first just to get a taste quickly of what this looks like and how this is a good fit. On the left, we see just normal YAML that you would use when interacting with Kubernetes. This is defining a config map. So this is the kind of text. Of course, those of you familiar with interacting with Kubernetes from the command line, perhaps you might have that text in a YAML text file and run qctl create dash f that file or apply it. And now that resource gets created in Kubernetes. Well, on the right, you see we're doing exactly the same thing, but we're using Ansible to do this. And of course, in Ansible, Ansible also very YAML based just like Kubernetes tends to be. In Ansible, we put together a series of these kind of tasks where we declare what we want. So classically in Ansible, some Ansible code might start with we're going to install some RPMs or install some Debian packages and that that we're going to configure a firewall. These kinds of things. So here, this particular Ansible task is using that KS module in it. We have a definition that we're just inlining a definition for this config map. You can see the YAML from the left side of the right, exactly the same. The only difference that I did put in here just to kind of show off the feature is templatizing this color value. So instead of just hard coding that is read, we now are using the templating features of Ansible to make that something we can inject at the time that we run Ansible based on whatever variables we specify when running Ansible. So this is a really easy way. You can take your existing Kubernetes YAML and inline it into an Ansible task like this and immediately now Ansible becomes a Kubernetes client for you. You can use Ansible to now, every time you run this Ansible task, it will ensure that that config map exists and it will ensure that it exists in the state that you've specified here. So it's pretty cool. It makes it very easy to interact with Kubernetes resources and you can already see that this, that Ansible itself is fairly declarative just like Kubernetes is declarative. In Ansible, we want to declare what state we want to end up with and we can keep running Ansible again and again and it will always ensure that after it runs, no matter what your starting state was, you end up in the correct ending state. I'm sure that sounds familiar after listening to Matt describe how controllers work. Now, what if you don't want to inline your YAML in that way? Well, we've got an even better option for you here with the KADS module. You can keep your Kubernetes YAML in a template file and reference it in just this one liner here. So here we're using the Ansible lookup feature. This is a normal feature of Ansible that those of you who I know I saw we have a number of Ansible users attending here from our little survey at the beginning. So the lookup feature here can look up your Kubernetes template and utilize whatever variables you have at runtime that you've provided to Ansible or that it's using or looking up or deriving at runtime. And it will use all that information to render your template and then apply the result to your cluster. So let's just imagine for a moment you already have a big YAML file. It's just a big Kubernetes manifest and you figured out how to deploy something, whatever your application is. So you've got deployments and services and secrets and config maps and maybe pod disruption budgets and all the all the kind of things that you want when you're deploying a template on Kubernetes. You can take that and just save it as a template, start templatizing values and reference it from this three lines of Ansible code. And now you can use Ansible as your client to continuously apply that manifest to your cluster. And of course in a minute we're going to see how to turn that into a controller. So let's do an Ansible role for a moment. If you're not familiar with the concept of a role, this is a core Ansible concept that, again, going back to the classic like traditional operational use of Ansible, you might have a role that's something like web server. You would make a role for your environment, for your corporation or your organization, wherever you're operating that would install your favorite web server, configure it the way you want, maybe get your TLS certificate rotation set up that kind of stuff. Whatever your standard web server should look like, and you can apply that role to different machines, right? So roles are a little bit like roles in a play. You know, one person has one role, a different actor has another role. Well, here we're grouping Ansible code that's related to accomplish a certain thing, like turn a host into a web server, or in our case, deploy a Kubernetes workload onto a cluster. So a role is a way to group that stuff together. And this is the basic layout of whatever role looks like. Now this role is going to get templatized for you. It's all going to get scaffolded it out so you don't have to make this by hand. So you're going to see this after it gets created for you. And these two things highlighted in yellow are the really the only parts that you need to be familiar with to get started and really go a long way. The first one is under our tasks directory this main.yaml. So main.yaml is where you enlist each of your tasks, just like we were looking at a moment ago. So just going back a slide, here's a task that we would put it could be one of many tasks into our main.yaml file. And then templates, that's a directory where you'll just drop in whatever template files you care to reference from your tasks. And it's really that easy. Ansible Galaxy I'll just mention is essentially a global repository of Ansible roles that the people have created and shared, and that you can then reuse. And in fact, we're going to use one in just a moment when we get into the workshop exercise part of this. So why would you use Ansible Kubernetes? Well, we have very similar patterns here, right? We have yaml on both sides, and it's very declarative on both sides, and I didn't put it on both sides. So, on a controller and Kubernetes we want to be able to say that no matter what the starting state is we have declared what we want and a controller to get you what you want, regardless of where you're starting. Ansible is just the same way. So these are a really good fit. Now we have a lot of ops teams that are already familiar with Ansible. In fact, based on the questions that Matt asked at the beginning of this workshop, these results are very typical of the results we find when we ask groups these questions. Typically, we have a lot of Kubernetes users who already know Ansible, who are familiar with Ansible and even using Ansible for other things unrelated to Kubernetes. And it's a very easy and simple transition to begin using that tooling and that existing knowledge and expertise to manage Kubernetes workloads in Kubernetes itself. Ansible is really easy to learn to do if you're not familiar with it already. It's real simple, especially what you see today. It uses JINJA templating. I'm sure you're already familiar with that. Even if you're not familiar with it by name, I'm sure you've encountered it and utilized it at some point. And Ansible is fully capable of day two management. You can build a completely capable advanced operator or controller using Ansible. And look at today's very basic. We're just going to scratch the surface and try to give you a taste of what's possible and how you can get started on your own. But you can really go very deep with Ansible. It's very fully featured, both on cluster and off cluster. And that's one last point I'll raise before we move on. Ansible has this huge ecosystem of existing capabilities to interact with services, with software, with databases, storage, with all kinds of appliances. It can interact with cloud environments. It can interact with hardware. You can use Ansible to configure networking hardware. All this kind of stuff already is baked in Ansible. And you can access all of that functionality from a Kubernetes operator now by using Ansible inside your operator to implement your operator logic. So let's look at the operator side of this real quick. Now, just recapping what you've already learned or what we already know. The basic workflow of how a controller works, right, is we've got the smiling little client up here talking to the Kubernetes API and creates a custom resource. And in there, they specify what state do they want their application to be in. And we have this controller that is watching for events. It's continuously watching for events to happen in the API service related to the resource that it's watching. And when it receives that event, it runs that reconcile function, it runs some kind of reconciliation logic that evaluates the current state, evaluates what your declared state is, what's the difference between the two and makes changes to move the current state closer to your desired state. And of course, in the operator pattern, our goal is to deploy and manage some kind of workload. So your application gets deployed and gets spat out the other side. How does this work in the Ansible world? Well, this controller in the middle, we have implemented an operator for you already that is part of the operator SDK. It is this generic Ansible operator. Now, we implemented it in Go. We utilize the operator SDK to do it. And it basically handles all the logistics for you of watching the API for events. And when it sees an event that your operator cares about, so you tell it which resources you're going to want to be watching. When it sees an event, its job is to go find the correct Ansible role or Ansible playbook and run that role or playbook. And it'll run it injecting all the information from your custom resource into Ansible as variables at runtime. So when you're rendering those templates and doing other kinds of logic and Ansible, it has access to everything that was in that custom resource and can utilize that in its reconciled logic. Now, we have this interesting watch file in the middle. We're going to look at an example of this in just a moment. That's really just a mapping that tells this binary, this Ansible operator binary, when an event comes in for a particular resource, what role or playbook corresponds to that resource. And that's how the operator knows I'm going to go load this role and I'm going to run that role with this input. Now, digging just a little bit deeper into a couple of features here, some kind of interesting stuff going on. The proxy on the right side is one of the more interesting parts of this and has enabled us to provide some very handy features. So when you run Ansible in this context, we're using what's called the Ansible runner. Now the Ansible runner was a part of the Ansible tower project originally, it's just a programmatic vehicle basically for running Ansible in a programmatic way, instead of in a human command line kind of way. Ansible runner emits events that this Ansible operator binary watches for all this is basically implementation details you don't have to worry about. But Ansible runner when we run Ansible, we set up a Kubernetes API proxy. So that it talks directly to this proxy. And that proxy enables us to do some interesting things. The proxy one enables us to provide a cache. So we can have a long running cash that is interacting with Kubernetes API. I think Matt mentioned the cash and we know that the cash is a very important part of a controller. On any given Kubernetes clusters, there are so many controllers running all the time. If they weren't doing fairly aggressive caching of what they're watching, we could easily overwhelm the Kubernetes API service running on that controller at any given time. So we're running this cache for you outside of Ansible that Ansible is able to take advantage of just seamlessly, transparently even. We're also able to do some things like intelligently watch what resources is your Ansible code creating. And then based on seeing what resources your Ansible code creates, we will automatically watch those additional resources as secondary resources for you. Really neat stuff. And there are some other features that come along with that, but those are a couple of the highlights. And let's see with that. Let's just keep moving on. Okay, here's the example that watches file. Now we have here an example a memcache the resource type. And this is the example that we're going to go through in our exercise in just a couple of minutes. Memcache D. So what this says is every time you see an event for this memcache the resource run the playbook at this path on disk. And that's it. And we could have many different entries in this watches file for as many different resources as we want to to be able to watch and have a controller for. There are lots of other features you can implement here. For example, we support finalizers. Some other kinds of advanced stuff that probably don't have time to go into right now, but this is the basics. You don't have to really touch this you'll see that this gets scaffolded out for you automatically, but it's worth just knowing and understanding how this part of of your answerable based controller works. As we saw earlier, with both answerable and go you can create very feature rich controllers and operators and do everything you need. In fact, as an exercise I've implemented the pod set operator that we looked at with Matt. I implemented that in answerable. Very straightforward to do answerable is at the same time, very, very simple to use. And if you if you want a simple controller. Very easy to do, but you can also then when you want to when you're ready to go very deep in terms of functionality. Okay, quickly here in the spec part of your custom resource, whatever you specify, you want your state to be all those key value pairs you put in your spec, get directly injected into answerable as variables at runtime. So we looked at that example in the beginning color was was one that I put in that template. So we could in our spec for a resource say color and say read as the value, and then that key and value pair will be available inside answerable at runtime for your templates and any other logic to utilize. So it's a really easy pipeline here directly from the Kubernetes API all the way into your answerable code, all taken care of for you, and you can just write your templates or write your answerable logic and not have to worry about it. Status, not going to get into that too deep but suffice to say we auto generate a reasonable fairly generic status for you as part of the answerable operator. But then if you prefer to have a custom status make your own status you can absolutely do that as well. So here's the experience we're going to get into this and see this in action in a moment you can run the operators to take a man, you say type equals answerable here, and it will automatically scaffold out an answerable role. It will create that mapping file for you the watches that you know it will make your custom resource definition and make a deployment manifest so you can actually deploy your new operator. And if you want to learn more about this after we're done with this workshop of course, you can go to ansible comm slash operators anytime and find a bunch more information there that let's go into our workshop. Okay, here we go. Let me back out of here. So we're going to go to the answerable operator overview exercise. See this then cooperate with me. Okay, so we're at the answerable operator overview exercise. Luckily I'm going to fast forward you guys through a lot of this content. You'll see there are eight steps as part of this well I have good news. The first let's say four and a half steps are basically just learning material notes is valuable learning material I don't want to downplay that part. But thankfully stuff we've already mostly gone over in this talk so far. So I would recommend come back read through it. A second pass through this content is definitely going to help you really understand this better and cement your knowledge of what you're picking up here. Again with this workshop really work. We're aiming to give you a taste of what's possible and enable you to come back and dig deeper later. But for now we're on step five we're scrolling about halfway down and we're ready to run this operator SDK new command. And run that in boom just scaffolded out all the things we just talked about. So let's run this tree command and see what we have. Okay. In the build directory we have a Docker file for actually building our our operator. We have the ability to do all these manifest related to deploying the operator standard stuff that you get. We also want to deploy a careers operator molecule. This is an interesting testing set of tooling specific ansible that as it turns out is very useful even for testing non ansible based operators and controllers. Not going to go into that today but definitely recommend if you're interested in testing and how to do some pretty effective testing. Check that out. So here's our role. So we had a memcash D role get scaffolded out for us. We have the things that we recognize we have main dot em on our task directory. Remember we talked about that in the tabless directory. Now our goal here with this operators we're going to deploy memcash D we're going to make an operator that's going to deploy that workload for us. So of course we have a custom resource called memcash D to represent what is our desired state for that memcash D deployment. Let's change directories into our memcash D operator that we just created and continue on to step six time to customize logic. Now rather than opening up that main dot yaml file and typing in some stuff or pacing in some stuff. We're going to utilize ansible galaxy and one of our colleagues Dylan who also worked on the on the operator framework. Created this role for us that we're going to reuse ansible galaxy is a tool you can use to actually download and install an ansible role. So there we go and just click on that it got that role for us. We can just check out what we have in there. Yep. Okay. Now we see two roles in here and we want to get rid of that that one that we're not going to actually use. Let's click on RM. Cleaned that up. Let's take a little closer look at what Dylan gave us. Again familiar things. He's not using a templates directory but we do have a tasks file here. In fact, we can take a quick look in there. Stand by. I'm going to try to make this civilized for us. Oh gosh. There's more here than I saw it. Okay fine. There we go. That's at least a little more readable. So we can see here what Dylan did is he inline his definition. So we're using the KS module. Right. And we have a deployment that he's defined here and he's templatize a few values. So that customer resource that we're going to create in just a few minutes is available in here under the variable name meta. So meta dot name is the name of your custom resource that's being reconciled. Meta dot name space. Well that's the name space of that customer resource in size is just an attribute of the spec. That's something we're going to specify in the spec of our custom resource. So that's what we have going on here in this ansible role. Now we can look in in the defaults file. And we see we've defaulted a size of one so we've specified a default value for the size. Now we've already looked at this stuff so let's just keep going. Now because we changed the path for where our role exists, we need to fix our watches file. And we've got this handy link here you can do to W get that. And you know I'm just going to take the easy and lazy route right now and do that. Otherwise we could open up, you know, watch the watches that you have a file and just fix it by hand. But that nice little link just kind of took care of it for us. Okay, now let's continue on. And let's actually create our custom resource. So we need to tell Kubernetes about our custom resource definition. Again, it got scaffolded for us by the operators decay. So that part is ready to go. We're just going to click here. And now this is one of the most amazing things I think about Kubernetes. We just added an entire resource endpoint to the Kubernetes API and you know, unlock this whole set of features and capabilities around that from watching those resources getting events reconciling and all that stuff is now enabled. And we have a couple of ways, kind of following in the discussion that one of you is having with Matt, a little bit ago, we can run an Ansible based operator just like a go base operator, either locally, just on your laptop for development purposes, or of course, build it into a container image and run that container image in a pod inside your cluster. Now I'm a software engineer, I hack on operators, you know, all day every day. So I'm typically running these operators on my laptop. And I want very quick feedback. So if something doesn't work like I expected, I want to kill it, I'm going to make a couple changes to my code and immediately rerun it. I don't want to wait for a container image to build and then push that image to the right place and then wait for the cluster to pull the image and that's all just too much time. So that's why what we're going to show you right now is how to run this thing locally. So you can get quick iteration in your development cycle. So we're going to run this thing. Because we're running it locally, instead of inside of a container image, we need to put our roles in the correct place. Because if we remember from the watches YAML file, it's expecting that inside your container image, that your role is going to be available at this path in the file system inside your container image. Well, when I run this thing on my laptop, there's nothing there. I don't put those roles in that path on my laptop. But what we're going to do in this environment is we are in fact going to copy what we have to that path in our local environment. And now we're ready for the big moment where we actually run this thing. So let's just skip down here to we're making a new project, which is based on namespace OpenShift has put some additional features around namespaces, but suffice it to say that's what a project is which created one. And now we're going to do a very similar run command here and immediately it's running because we've already got the binder there's no piling to do. It's already running it's already watching things. We see the normal kind of log statements we would expect. And now here's an important part to not miss in a second terminal window. So we hit this plus button here. And that enables to open a new terminal window. Okay. So now we can we can just click these tabs to go back and forth to better operating our operator running in one of them. Now in this one we're going to change directories into where we're working. And we're going to create a custom resource. So there's already an example customer resource scaffolded out for you. We are just going to create it. We can look at it real quick. We see it's got a size of three. And otherwise, this is just a totally normal Kubernetes custom resource, right. And in the spec we can put any key value pairs here we want, and all those will get injected into ansible and used to render things and and have other logic. So we create that thing. That's created. And let's run it was he gets. deployment. And we see no resources found. Okay. Why is that? I don't know. There it is. Okay, maybe I'm not in the right namespace or there it is. It just took it a moment to run. Fantastic. So we see some output here we see some ansible output. It took it a moment for Ansible to wake up. We see normal ansible output here is our controller ran that Ansible code created that deployment we already have three out of three pods up and running. And now, if we want to change our size here, the exercise says we can change to four. Let's go ahead and let's change it to five. So I'm going to save that. And now we're going to skip down here and apply that change. Okay. We applied the change. We're going to wait. Okay, we see three out of five are ready. You can see some are creating. So there you go. That's basically what we have. We now we now just by taking an existing Kubernetes manifest, templatizing it a little bit and putting it into this scaffolded project for you, you now have a controller in an operator, which not only gives you this way to continuously reconcile your workloads and your resources. It gives you a way to manage your Kubernetes resources from the Kubernetes API. This is what Kubernetes native is really all about. You have a Kubernetes API endpoint called memcash D. You can use it to now manage and continuously reconcile that your desired state. And you now have access to the whole operator ecosystem. So everything around operator lifecycle manager. And all the power of that kind of management, that whole dogma of management of Kubernetes workloads and Kubernetes clusters and infrastructure, even themselves. All of that is available to you now through Ansible based on just this little bit of work that we did. So that's the overview here. I hope you've enjoyed this and see some value in this. I would encourage you to go back and do some of the other exercises in that are available for the operator framework. And with that, Matt, I'll hand it back over to you. Well, thank you. So what I'd like to do is just kind of give you all an idea of how, and I'm going to share my screen out in just a moment. I want to give you an idea of what the experience is like using and installing an operator on the dashboard with the operator lifecycle manager. So hold on one second here while I get my environment up for you. Give me one moment. Okay, so you all should be able to see that properly. What you're looking at here is the OpenShift dashboard, which this is for OpenShift 4. And you all can check out OpenShift 4 at try.OpenShift.com. Try.OpenShift.com. If you go there, this will automatically show you how easy it is to get started creating an OpenShift 4 cluster directly on AWS, Azure, GCP, or even bare metal. And what you have to realize is that OpenShift 4 is completely powered by operators. The entire infrastructure is powered by operators. And as Michael was saying, it's even Kubernetes native on the infrastructure level. What do we mean by that? We mean that literally if you, for example, deploy a cluster on a AWS environment, the actual AWS instances that are running Red Hat CoreOS, which is an immutable operating system from Red Hat. Those objects are Kubernetes native. So you can think about when, for example, you exceed capacity in the cluster or when you perhaps have to create pods and they're pending because you don't have enough capacity. It will automatically send Kubernetes API calls, which will create new objects that represent the underlying AWS instances, which then in turn send AWS API calls to spin up new instances. So it's completely Kubernetes native all the way down. What we're going to show you here is what the operator experience is like. If you want to go ahead and certify an operator and get it into operator hub, what that would be like for you. So what I'm doing here is I am just getting my credentials on the command line. And I am going to log into the OpenShift 4 dashboard. Okay. So now that I'm logged in here, what can I can actually see is on this left panel, I have all my various categories for all my Kubernetes resources. And now we have this operators tab. With this operators tab, it actually allows us to browse the operator hub. This is a place where all of our partners and customers can actually publish their operators. There's a variety of operators in here, everything from API brokers to databases to things for networking like service mesh, a lot of great stuff in here. And for example, let me go ahead and let me show you what the experience is to install Argo CD. For those of you who have never used Argo CD, here's the reality. You all should check this out because we all know that in the real world, users do not typically go and install. First of all, they don't go and install operators manually. That doesn't happen. Also, I mean, with that said, it's in the same vein as we don't go and do coop CTL apply manually, right? We know that in production scenarios. It's not like, okay, it's time to roll out the app and everyone starts, you know, getting on the command line running coop CTL apply, which you typically have are your Kubernetes objects that are checked into some sort of version control. And within that version control system, you have that connected to a cluster and you have times in which some kind of trigger or some kind of hook kicks off the application of those manifests into your environment. Argo CD is a great way to do that. It provides our get ops piece to the puzzle for continuous delivery, right to get our Kubernetes objects into our cluster. Argo CD operator was created and started by some members of our team here in the operator enablement team at Red Hat. And it's a great example to kind of show you installing an operator with that experience is like notice here how you have the install button. So if I'm a regular user and I'm locked to a particular project or namespace, I can easily go and deploy this operator. Just like Michael shows you with like, you know, creating the CRDs and we chose you with go operator. That doesn't happen here. We don't manually go and do that. It happens for us automatically and OLM, the operator lifecycle manager is kind of the missing piece to the puzzle that handles all that for us. So what I'm going to do here is I'm going to go ahead and before I even install Argo CD, I'm just going to go ahead and create a brand new project here. And I'm just going to call it Argo CD and feel free any questions you all may have feel free to drop them into the chat we have Red Hatters here that will better here to help you out. So then what we're going to do after doing that is we are going to go and click on Argo CD. And click on install. And then after clicking install, here we have Argo CD. We have alpha, we have the approval strategy. And what we're saying here, this is literally incredible about this, we're saying install this operator into the Argo CD namespace. Okay, and then thanks to OLM, you have this thing called an approval strategy where you can choose whether or not you want it to install automatically, or if you want to review what pieces that's going to install, and then approve that. This is comparable. Think about this. If you're an operator author, you have the ability with OLM to push out updates to your users, you actually have the ability to push out new versions of your operators to users. If you're a consumer, you may not want that update to happen. You know, behind the scenes without your knowledge, you may want to approve it first. This is comparable to like, you know, maybe you have an Android phone or an iPhone, and you get, you can turn automatic updates on or off. It's the same concept here. So what we're just going to go ahead and do is I'm going to do a manual strategy, and I'm going to install the Argo CD operator to the Argo CD namespace. Now you can see here what it's doing is it's actually going and looking at something called the cluster service version, which is a custom resource definition that OLM provides. The operator author writes the cluster service. I'm sorry, the cluster resource definition. And what you can see here is we have a subscription that's been created, which means that I want to get updates for this, but I have to approve them first. And then if I go to requires approval and go to preview install plan, this shows me what is going to get installed when the operator gets installed, right. So what I'm seeing here are custom resource definitions. As you can see here, I'm seeing the cluster service version. I'm seeing the service accounts. You may be wondering what what are all these are back artifacts being installed. Understand that when you go and install an operator in a production like scenario like this, you're not just going to blindly give the default service account with, you know, perhaps cluster admin privileges to a assignment to a deployment that is running within the cluster, you want to be very restrictive on what permissions you give these operators because they're running in your cluster. And you it's just kind of comparable to, you know, in the Linux operating system right we understand that we don't run everything as route. We want to use kind of like a least privileges type model. So this service account that gets created is going to have a particular role right or cluster role and that role or cluster role will control which API calls the operator is capable of making. And you can see here the custom resource definitions are going to be installed and everything like that. So let's go ahead and approve it. Now notice right when I approve this, the Argo CD operator is being installed in my cluster. If I hop on over to the command line. And I look at the Argo CD environment here. Notice that the Argo CD operator is now running. That's just the operator itself. I actually haven't gone and installed any particular custom resources yet so the app the application itself also known as the operand has not been installed yet. Okay, let's go ahead and look at our logs here just so we can see this thing spin up and what it looked like when it spun up. So here I'm looking at my operator logs. So you can see that the operator was came up online and verify that some endpoints were available and now it's watching for the creation of the Argo CD custom resources they haven't been created yet. So we can actually go and spin up a window here, and we can watch in real time for the creation of those resources. Also something else I want you to take note of is that the CRDs in my environment and by the way, when you run an OpenShift for environment, you get a variety of CRDs that actually get installed. And out of the box, right? Because everything is managed by operators, everything. So it comes included with CRDs. But I want you to notice in particular that the Argo CD CRDs were automatically installed thanks to operator life cycle manager. Do you see that? So I didn't have to necessarily have cluster admin privileges for those things to get applied. So now if I say OC get Argo CD, right? I don't have any resources created yet, but the endpoint actually exists, right? The endpoint actually exists. So what we want to do is let's go ahead and create an Argo CD. Let's go ahead and get this thing started. How do we get this thing started? Well, let's show you this. On the actual dashboard, we had a question about this actually in the chat. The question was, is there any, what's the best way to kind of have users provide information into a custom resource? Well, let me show you this on the dashboard. If you click on Argo CD, and then from Argo CD, if you actually go and click on create instance, this is our custom resource, there's a variety of custom resources that ship with Argo CD. And we see here that there's two options. One of the options is just having a custom resource that where the user can fill in any desired values. And if you're wondering what possible values, you know, could a user put in here? Well, you would want to look at the explain information, or if you wanted you could look at the instructions here in the description. But this is the really cool part to show you. Thanks to the OpenShift descriptors in OLM, if I click on edit form, I can create that cloud service like experience with forms. So I can actually customize what my user sees here when they go to fill this out, and I can actually provide some default values. Right. I can turn features on and off here with these switches. I can provide bullions, all types of really cool stuff to make that experience a lot easier. And if you scroll up in the chat, I presented a little guide on how to customize these descriptors. I put a link to that. Definitely check that out. It tells you all about how when you build an operator, you can build these things and it makes the experience really nice for our users. But right now what I'm going to do is just flip back to our YAML. And I want to show you this in real time. I'm just going to go ahead and watch our controller. I'll do a dash F. Notice that our controller is waiting for the creation of a custom resource. And let's go ahead here. And I'm going to just go create my empty custom resource, which should be acceptable. And as soon as I create that custom resource, look at the gap here. Notice we are waiting for the creation of a custom resource. And then suddenly the controller was reconciled, right? Why was it reconciled? Because remember what controllers do. Controllers wait for the creation, update, and delete of the object they're watching. So the ArgoC, the object was created, right? And the controller saw it and it reconciled it. And now it's running through its comparison of the actual state and declared state that's inside our CR. Well, let's let's see what it's doing. What could this possibly doing? What is our operator doing? What is our human operational knowledge doing for us? Well, let's take a look and see. Well, we'll say OC get deployment. And notice that this is off this particular operator. Remember, the thing about operators is they provide human operational knowledge. This particular operator is providing for us a series of Kubernetes deployments that represent Argo CD. We have our Argo CD server. We have it's using Redis as a key value store. We're using DEX, DEX as a way to use OpenShift as an authentic identity provider type, right? So you can imagine this is really awesome, by the way, because you can imagine if we already have users in our OpenShift environment, right? That we either were using GitHub for authentication or using Google or LDAP would have you. Now, we can use those existing users, have those existing users, signed into Argo CD, right? So we don't have to set up a separate group of users for authentication. So the DEX acts as a broker that allows me to specify OpenShift as a identity provider type, if you will. And now we have our application that's actually been created. Let's see here. I'm not sure exactly if we have a, I don't think it automatically creates a external route for us. No, it doesn't. It's probably something that we have to specify in our custom resource. But at least you see that this part has automatically been set up for us. Let's check out our services. Yes, there we go. Look at all this what the operator did two minutes ago. The operator also set up a service for serving metrics. Why do we care about a service for serving metrics? Because Prometheus, right? Our Prometheus operator that we deployed to this namespace, and by the way, Prometheus operator automatically comes installed in the operator hub. Okay. Notice here that if you install this to our Argo CD namespace, it will be able to easily scrape these metrics really, really easily. And of course we want to expose our server, our DEX server. So you can just see that the operator does a great deal of work in setting this all up for us. If you want to learn more about Argo CD, if you want to learn about more about how this all works, feel free to provide on my email in the chat. We also have some great resources on YouTube, just type in Red Hat Argo CD on how to actually utilize it once it's set up. But this is really giving you the overall operator experience, right? The operator kind of look and feel. Now that I have the Argo CD CR actually created and Argo CD is running. Right. If I go to my Argo CD here, I can see that this is my custom resource. Right. That represents Argo CD. Okay. And of course, if I wanted to give access externally, I would create a route or easily provide a route inside of my custom resource. And notice when if I want to actually go and fleet the operator, I'm just going to go inside here and click on uninstall. Okay. Now, this is the operator that is embedded in the operator hub. Please see us if you're interested in getting your app operator inside here. Alternatively, those of you who have no interest in OpenShift, maybe you run a vanilla Kubernetes cluster, you just go to operator hub.io. And here, these will run in any Kubernetes environment. Right. We even have an Argo CD that's been created for upstream. What does that mean? It means that it doesn't rely on any OpenShift objects to do what it does. It knows that it's going to be running in a vanilla Kubernetes environment. If you're interested in contributing to the operator hub.io or the embedded operator hub on OpenShift, please check out this link that we provided at workshop.coros-train.me if you go to support and you go to submit your operator. You'll notice here that the upstream community operators is where you want to create a pull request and please read the instructions on creating a pull request. But this is where you're going to do when to put your operator on operator hub.io. Here is where you're going to put your operator that gets featured inside OpenShift. Okay. And it is community supportive. If you're interested in having an operator that has some sort of enterprise offering that you charge your users for, you can check out the Red Hat Marketplace. Right. So the Marketplace is a place where you can actually go and offer your enterprise offering. If you're interested in any of this, please come see us. We're happy to help you utilize the power of operators and share it with all the users. Not just regular OpenShift, but vanilla Kubernetes as well. So with that said, we have a few minutes left. Were there any questions, any concerns, any questions at all? Do we have anything? Feel free to put them in the chat. Sure. Yes. Somebody asked, is there a Argo, they want more information on Argo CD. I will see if I can grab this link for you, but anyone from Red Hat, if we look at, there's a great model that a great example that I think Christian Hernandez put together, I have it right here for deploying Argo CD on OpenShift. And I have the link right here. Yes. And I'm just going to drop it right here into Mario. Here it comes. I'm sorry. Everyone, here we go. Okay, Mario, check out that link. That will show you how to install Argo CD and use it on OpenShift. Justin Miller asked, what is the best way to start learning more about OLM? Is there a book? We have a free book. If you all didn't catch it. A couple of Red Hatters have created a free book. And I'll just go ahead and just share my screen out so you can see that. The free book is available on the website. Workshop.corelesstrained.me. Go to the support tab. Click on this link right here. It's an O'Reilly book, written by Jason Dobies and Josh Wood from Red Hat. Tells you all about operators. Great book. Free download here. Definitely check that out. Mario, you can run the Argo CD operator on Vanilla Kubernetes. Yes, you can. All you'd have to do is go to operatorhub.io. And it's just a couple of one-liners to go and install it. You click on install. And you go to install on Kubernetes. This will install OLM. And then this will install your Argo CD operator. And this will run on Vanilla Kubernetes, Mario. If you have any questions about that or concerns, you can feel free to email me. No problem. Happy to help you with that. Any other questions or concerns? We have a few minutes left. Any other questions? Okay, I think we're all set up here. All right, thank you very much.