 Well, oh, yeah sure is Just want to make sure the audio is connected to it. I'm not going to use any Okay, I'll leave this as is because I see they still have their laptop. Oh, yeah. Yeah All right. Thank you You're good. All right Thanks so much for coming to this very first early session today. Can you hear me? Okay? All right, cool So this is going to be talking about automating stateful applications with Kubernetes operators To introduce myself. I'm Jan Kleiner. I got loud I am a developer advocate at Red Hat. I focus on OpenShift, which is a distribution of Kubernetes And while we're not going to be talking about just generically talking about Kubernetes today I do want to lay the groundwork before we start talking about operators To make sure that everybody in the room has at least some fundamental understanding of Kubernetes the Kubernetes API And some of the different types of primitive types and entities that are available, but Just to be sure How many folks in this room already feel like you're pretty familiar with Kubernetes? So if it's every single person no every single person, okay We're going to go through at least a little bit then just to make sure that you have a foundation So the rest of the talk makes sense So what is Kubernetes? Basically, it is an open source orchestration system for managing containerized workloads So if you are running applications services and containers and you need to do that at scale in an automated way Kubernetes is a great way of helping you do that So what are some of the types of things that will help you do? So there's a lot of stuff written up here and that's not even half of it But some of the things it can help you with deployments. It can help you Simplify the process of handling networking and routing so you can access the applications that are running from outside the cluster It can do health checks and also recovery if there's failures all sorts of things Can happen in a fairly automated way so that you don't have to manage all of this and kind of figure out your own systems for for taking care of it to get a little bit more detailed I wanted to talk about some of the Main object types in Kubernetes and the reason for this is not because I need you to know about all of them But it's so that when we talk in a little bit more detail about operators you can understand how that relates to the Primitive object types in Kubernetes. So you see here things here listed like pod and service deployment, these are some of the the core pieces And types of objects that will be in a Kubernetes cluster And any time you're interacting with basically anything in Kubernetes you're using the Kubernetes API in some form whether you're using the API Directly and running curl commands or if you're using cube CTL the command line interface Oh, I got a little I get the top of my hands a lot. So I'm going to try not to do that All of those things are using or on top of the Kubernetes API so that's a really key critical piece of the whole system so When you're working in Kubernetes one of the things that's kind of unique about it is that you are typically You're describing your desired state of the system and Kubernetes is doing All of the work kind of behind the scenes to try to bring the actual state in line with your desired state So if you're saying I want to deploy this application Here's the container image. I want to use and I want three replicas of it I want three pods running with this application and Whatever other things you want You're just going to put that information into a spec which is part of this object definition Let me go ahead and show you kind of an example. So this is like an excerpt of what? An object definition for a pod might look like you can see there's some metadata Like the name of it what kind it is at the top you can see the kind is pod in this case And the spec is what I want to make sure that we're thinking about a little bit before we talk about operators the spec is where you're defining what your desired state is and then Kubernetes is constantly going through the process of trying to reconcile the actual state with with your desired state And we'll talk about that a little bit more later, but I just want to at least introduce that concept before we start talking about operators But we're not here to talk just about Kubernetes in general We're here to talk about operators and before getting into the details of you know What operators actually are and how they work and how to use them? I wanted to first introduce Why should you care about them in the first place? What what is it that they're what problem are they solving? What is it that they're bringing to the table that that you might not have already? So that's where we're going to start As you may know scaling this stateless applications in Kubernetes is easy. There's not a whole lot to it. In fact, there's Primitive types like replica set that can handle all this for you So if you were to run this command for example Cube CTL scale and you pass in what it is that you're your application here And then how many replicas it's going to handle that for you and scale it up to three So how does that work? So here you can see kind of an example This would be like the starting state So you're using Cube CTL which is on top of the Kubernetes API you want to scale it up your application Your desired state would be you want three currently. We're over here We have one instance of that pod running when you run that command Community is going to do whatever it has to do to spin up two additional pods With your application running and so that's just happened for you. So that's pretty straightforward There's not a whole lot that you as the user have to think about or know you just say hey I want three and you get three So what about applications to store data? Scaling stateless applications is easy Applications that store data might be a little bit more complex You can think about things like a database where you might they might have their own notion of a cluster So you're running in your Kubernetes cluster, but maybe you're running something like Redis enterprise or something like that that has its own notion of a cluster as well that introduces some complexity To the system that in general in Kubernetes creating a database is easy So say you have some imaginary database here All you have to do is tell it what to run pass it the image for what you want to run And it will get it up and running for you and the real complexity Comes in when you're trying to run and maintain things like databases or stateful applications over time and That's where we're getting head in that direction now So what are some of those considerations that make this harder? So Pretty much any complex application is going to have to deal with these type of actions as you run them and manage them over time so, you know if you have something like a Redis enterprise cluster and at CD cluster or something running on Kubernetes you may need to resize it over time Maybe the demands on your system have changed For sure you're going to have to upgrade or patch if there's some sort of Vulnerability that comes out and you need to update the version that you're using you may need to reconfigure things If what you're running has stores data, you may need to back up that data for some reason Or things always go wrong. So dealing with healing You know something crashes. What do you do in that case when you've got this application running in a distributed system? How do you handle that those types of things are a little more complex than saying? create this database and There's additional complexity that comes in when you're running on a system like Kubernetes So basically when it comes down to it, what are you trying to say you may only have to install something once But you're going to have to kind of continually deal with configuration management and upgrades over time Particularly as things like patches come out and you have to find some way of getting that applied in your system Patching of course is critical to security and running secure applications is critical to any business And when it comes down to and anything that's not automated is slowing you down all of those things that we described Handling upgrades handling, you know failures and recovery and back up all of those Generally, I'm going to take some interaction from a person, right? They're going to take time from somebody's day They're going to somebody's going to have to have noticed that that problem happened in the first place and then Have the knowledge and the time and the skills to take care of the problem and do whatever is necessary And all of those things they do slow down the process of deploying and delivering your application Whatever it is that you're trying to do so What can we do about it if only kubernetes knew kubernetes knows how to automate lots of things already, right? Well, what if it knew all of that operational knowledge for how to? manage these you know, say a database or your complex application in Kubernetes already, what if it knew how to do that for you as well? So here's our imaginary co-worker John he has worked at your company for about five years and he Knows everything there is to know about running. We're going to call it production ready database. It's not just some imaginary database He knows what it takes to get it Deployed on your kubernetes clusters. He knows what it has to be done when you need to upgrade. He's got all that operational knowledge He's experienced. He's done this for a long time He's the person that everybody in your organization is going to come to if they need a database deployed or if they need Some of these operational tasks done all of that's going to come to John The problem is he's really good at what he does, but there is only one of him Everybody needs his time and some organizations aren't even going to be able to have a dedicated resource like him So they may not even be able to run applications like this because they just don't have that operational knowledge So what if we could take all of that knowledge and experience? That John has about how to run and maintain and upgrade and manage these applications this database over time And put it in a box. That's very simplistic, but just imagine that we could do that You can take his operational knowledge and put it in a box. All right Then if we could do that then we could take that that operational knowledge the software version of basically all of this this Information that he has in his brain and we could deploy that on any kubernetes cluster and any organization and any cloud And you could have that production ready database running and managed and a production Suitable manner anywhere and so what we're really talking about there is what operators are trying to provide So they are basically automated software managers that handle the installation and lifecycle for kubernetes applications And when we say kubernetes applications That can mean kind of a few different things But what what I'm trying to describe here is it's an application that runs on kubernetes It also has some like touch points into the kubernetes api or into, you know other objects running on kubernetes, so it's kind of It's not just running, but it's also like interacting with it All right, so how does this all work? How do you operate? What are operators made of? How do they work? How do you make one? That's what we'll talk about a little bit next One of the beautiful things about kubernetes is the fact that everything kind of goes through the API is that you can also extend the API and Controller, sorry got a little excited there operators are the operator pattern is basically Made up of two components. So there's controllers, which is a thing that already exists in kubernetes and custom resource definitions Was it also a thing that exists? So you are going in the case of an operator It is a custom controller that's specific to your application and then a custom resource definition And then we'll talk about this in a little bit more detail in a minute But what that custom resource is is so kubernetes knows what a pod is knows what a services It doesn't know what production ready databases. You can't say cube CTO get production ready databases It's gonna be like I don't know what that is, but we can tell it what it is by Defining these custom resources and then we can use this controller to watch Changes to those types of resources and then handle whatever events happen And that's kind of what we're looking at here, which is this operator pattern. So Over here on the far Left side you've got your kubernetes API The custom resource is going to be whatever that instance is of your your complex stateful application in our case we're going to keep calling it production ready database and then you have the controller which is doing those two Important tasks watching for events that events could be like I've asked for one of these to be created or I need to update the Version or whatever else your your operator is going to do and then it runs this reconciliation loop when one of those events happens It's going to say okay. Well, show me what the current what did you ask for? What's the current state of the system and now it has that operational knowledge built into it that logic of what to do when an Upgrade needs to happen or when a new one of these needs to get deployed. That's all built into this controller Which is deployed is just a container image And so it can do whatever's necessary and I as the user who has said I'm going to use this opportunity I don't need to know any of that knowledge the operator knows so that's pretty cool and then That handles everything for you. I'm losing my voice. This is great So here's another way of kind of visualizing it white Rounded rectangle as your Kubernetes cluster this thing down here on the bottom with the red circle and the Lightning bolt symbol is your controller. So it's sitting there. It's just watching and waiting and then We're going to have our custom resource So you can see here instead of kind being pod like we saw at the very beginning Instead of being one of those, you know built-in primitive types. It's production ready database in this case We still have some metadata, but what's unique and interesting and is specific to our application to our Production ready database is what's in the spec here? So we're telling it we want this database to have a cluster size of three We want to read replicas and we want a particular version to be running. So when we when we apply that then the Controller which is listening is going to that the arrow in the little circle. That's do whatever it takes It's going to do whatever it takes to bring up those three Instances of that production ready database. So the operator is handling all that for us So I don't like to talk about things in like an abstract way. I wanted to give a real example so we're going to look at kind of a demonstration of Deploying and using the xcd operator if you're not familiar with xcd. It's basically a distributed key value store It's also what Kubernetes is using as its primary data store. So it's used for storing And replicating like information about cluster state in Kubernetes So here's an example of what our xcd cluster Custom resource would look like we're telling it that we want to have a size of three and a version of 3.1.0 And like we already talked about this before so I don't want to go into too much detail But here's that observe analyze and act kind of loop that's going on here So I'm here's an example of what that might look like. Let's say you start it off Your xcd cluster a has two pods in it one of them is running version 3.0.9 One is version 3.1.0 if you remember we actually want three members and we want them all to be running 3.1.0 So that's what we find out. Okay. What's different than our desired state? It said, okay Well, the version needs to be different for one of them and we should have three members So then the operator is going it knows what to do to get us that additional Member brought up and make sure that everything is upgraded to 3.1.0 I as the person who just wants that CD running on my cluster don't need to know how that works or what it's going To do it's just going to happen for me because the operator has that knowledge encoded in it Okay, so this I always have a hard time Controlling this from the side, but we'll hopefully get this right I'm going to show you this video and talk you through what's happening and hopefully that's clear enough that you can see What's going on there? But I'll I'll say it out loud anyway, and I may pause at a couple points to give a little more detail on what's happening But this is an example It's also very similar to something you can find on learn that open shift calm where you can go through this in an interactive way yourself of Deploying an at CD operator and then actually using it Once you've got it up and running using at CD Clicking okay Right here, so we're going to start here By running create and we're passing in there's several YAML files already kind of queued up for us One of them here is the CRD so that we are letting Kubernetes know what an at CD cluster is Then there's some RBAC stuff that has to happen. So we need a service account We need a role and a role binding. These are just things that at CD Cluster needs in order to run. So this is kind of like housekeeping stuff That has to be done first as part of the setup every operator is going to have different requirements for what it needs To have running on the cluster before you can use it So we're getting our role in our role binding setup And then now we're going to create a deployment that has the Container image for our at CD operators. This is that controller piece that we talked about it is deployed as an image So now we've got that done We'll check that it was created I'm gonna be able to pause here when it's time Okay, so that's running and now We see that there's a pod for that is running as well. So now our controller is running now We can define an actual at CD cluster by referring to that custom resource. So we're Pause they get it. Okay, so You can do it like this What you're looking at here We were in count on this file here So this you can see that we've got a kind is at CD cluster and Kubernetes is gonna know what that is now because we've already Defined it and we're telling it what we want in the spec We want size of three and a particular version there Now that we've looked at the file we can actually create it So we're passing that file in to create And what that's going to do is create that at CD cluster on our Kubernetes cluster So we can verify that that worked because now we can Now we can actually run something like here Coups you tell get at CD clusters and that's actually going to return something for us because now our Kubernetes cluster knows what an at CD cluster is and it's telling us yet. There's one. It was created nine seconds ago and then What we'll see next is The pods getting created so there should be three of them because we said we wanted a size of three You can see it's bringing them up and eventually soon here You should see there's well, there were three in running state now Okay, so three of them came up we want to make sure that this actually worked and it's not just like magic, right so Little bit of magic because there's actually another Terminal window that you can't see here in this video I couldn't do them side-by-side but from another pod and our cluster We are Connecting to our at CD cluster Using the command line tool for that and then we are going to put something in the data store and then get it back out Just to prove to ourselves that this thing actually works. So that's what's happening now So we'll write the key in a value and then get it back out So at this point all we've really done is deployed that at CD cluster into our communities cluster using the operator and then prove to ourselves It works. So what else can we do? Well, let's say Okay, let's say that we want instead of three Three members in our at CD cluster we actually want five we need we need that for some reason all we have to do to make that happen is Change the spec tell it that we want five instead of three this command looks long But all we're doing is a patch where we basically just updating that value for spec from three to five So that's what's happening here, and then we can watch as the Whatever actions the operator needs to take to make that happen are being done So we can observe as those two additional new pods are created So you can see they're starting to come up There's the fourth one there in running state, and then it's going to bring up the next one as well So this is kind of a simplistic example, but it kind of gets the point of course of how an operator Works and what it can do We could have also gone through the process of updating the version But for the sake of not having you watch a very long video We'll stop that right there All right, so they mentioned before you can you can do almost the same exercise yourself if you go to learn that open shift calm such operator framework there Maybe eight or nine different Learning interactive learning scenarios related to operators that you can try out There's one I think on just like an overview of the Kubernetes API if you need a refresher there There's this at CD Operator tutorial, and there's ones using the operator SDK, which we'll talk about in a minute Create operators using ansible go. I think even helm and some other things as well So it's a really great resource if you want to try it yourself And get your hands dirty a little bit with with operators And you can use operators today. Let's say you don't you don't care so much about you know Getting in the weeds, but you want to see what kind of operators are out there operator hub.io Is a place where you can find a bunch of? Kubernetes operators that have been built by the community Anything you see there will also have information about how to actually deploy it on your system and There's quite a bit of stuff there, so you can check it out if you're working on things that make sense to be an operator Excuse me, and you want to make those available to others to be able to use there's information about how you can actually get your Operators that you've developed on this list as well On that site So we're going to shift gears a little bit now and talk about some of the like tools and things out there to help you if you are Building operators yourself perhaps or managing them on your cluster in a slightly more advanced way So the operator framework is basically like an umbrella over some other pieces that are available to you other projects So those projects are the operator lifecycle manager, which is you can think of it as kind of an operator for operators So when an operator is used for managing the installation and lifecycle of Kubernetes applications operator lifecycle manager Manages the installation and life manages the lifecycle of operators, so it's kind of just like a level removed Operator metering is available to let you see kind of like usage reports and and metrics and historical Information over time if that's something that's useful for you and then finally the operator SDK So if you are in a situation where it makes sense for you to be building an operator The SDK can be a useful tool for you So you don't have to use the SDK in order to build an operator. You can just do it however you want, but the SDK is kind of nice because it's going to Provide you with some scaffolding and code generation so that you don't have to do it all from scratch So it will have kind of scaffolding there for some of the common use cases for operators You know installation upgrade what have you and then you can fill in your logic for your actual application and use case there the SDK supports creating operators with Ansible with go and I believe helm as well Different slightly different capabilities for some of those than others, but all that support is there It also provides you high-level API is to kind of hook into some of the things in Kubernetes So that you don't have to basically so you don't have to do as much of this stuff yourself But of course if you wanted to create an operator using Python or Java or whatever You can do that too, even though there may not be SDK support for it It's not going to stop you from doing it if that's what you wanted to accomplish Something to note because when I talk to people about operators, I get this question a lot is kind of like Do they all have to do all of those things? No There's this whole kind of capability level or maturity level Continuum here for operators. There's kind of broken into five phases It could just that's kind of hard to read on the far in there But that phase one is it's a basic installation. So maybe you've got an operator and all it does is install Whatever your application is And it moves on in complexity from there phase two. It's it's doing upgrades Full life cycle would handle maybe backup failure recovery Insights maybe that's handling some of these more like metrics logging type of stuff And then phase five, which is listed as autopilot here is doing all of this doing everything It's not necessarily the case that your operator needs to be handling phase five to be good It's just what it would actually make sense for what you're trying to do But it's nice to know if you're on operator hub and you're using a community built operator It's nice to know kind of what which of these things is it doing? So this is kind of like a nice framework for being able to talk about What capabilities does each operator have built into it? I Really flew through that there a little bit faster than I thought I would I have some resources I want to share with you so we you can go on get hub to operator framework that'll have links out to The SDK to metering and the life cycle manager as well as some other resources as well There's a nice site here chorus has a nice like sub-site on operators It's got a lot of information particularly good if you want to share this with somebody But you don't want to send them to like a github project And get them in the weeds with the technical stuff. This is got kind of a nice friendly overview of operator concepts Operator hub that I always talked about is where you can go actually find operators if you want to try them out It's a nice blog post here, and then of course I'm learned at open shift calm those interactive tutorials are really Really good if you want to try things out and get your hands dirty So we have Lots of time for questions if anybody has questions Yes, I think they're bringing you a mic How would you populate data in the database because if you have a stateful application You usually want to have like a starting point that you're working from right so under this whole setup What would you recommend is the way to? pre-populate the data in your System yeah, that's an awesome question that I don't know the answer to you, but I will find out I can give you my card after okay, and get back to you Yeah, I'm not gonna pretend. I know the answer, but I can definitely find out Some recommendations there, I'll just keep the mic, okay Yes Is Cuban that is itself using operators to manage its at CD cluster that he uses so I Don't know if it's using to manage the at CD cluster. That's just always there. I don't know, but what I can tell you slightly different is OpenShift OpenShift 4 is using operators to basically do almost everything So it's been completely changed so that you know installation is using operators managing upgrades and things like that is using operators I think there's Well over 40 operators just running as part of OpenShift right now Just as like the guts of it basically so I know that operators are used a lot there But I nobody's ever asked me and I've never looked into it to know if I'll give you my card too, and we'll find out and I'll let you know I have one more So to create those custom resource definitions Do I need to be a cluster admin or can I create that on a per-project a Lot of things related to operators do have to be done as cluster admin but Once you have it done there's an option to Deploy an operator either cluster wide or on a per-project level so you can kind of choose that way if you make you as a cluster admin can make an operator available for non admins to actually use If that makes sense, but a lot of this the setup and getting it actually like installed and configured Many of those things need to be done by cluster admin Thanks Any other questions? Thank you so much. If anybody else does have questions I can give you my card and find those out to you. So thank you very much