 Hi, I'm Josh Wood, a developer advocate for OpenShift at Red Hat. I'm joined by my colleague, Jason Dobies, and sometimes co-author. We're going to tell you about all the exciting updates that have happened over the course of 2020. They've seen new adoption, new features, new frameworks, and new initiatives for operators and the tools that we use to support and construct them. So we'll dig into a little bit about what operators are, make sure we're all on the same page, and then we'll take a look at things like our recent induction into incubation at the CNCF with the frameworks and tools that we use to build operators, and some of the new features that that's brought into the SDK and the OLM. Before we get too deeply into that, let's take a look at what operators are and why you care about them in the first place. I usually like to start talking about this at a really high level, because the pattern that operators enable, let us do advanced automations on the Kubernetes platform, and they're really distinct in terms of what we want to accomplish with them from the implementations and the techniques that we use to accomplish those things. So what I always say is we care about operators and operators are important because any application on any system or platform has to be installed, configured, managed, and upgraded over time. We see this with web browsers and automatic updates so that we get the latest security patches. We're very used to it with app stores and the applications that run on our mobile phones, and certainly our operating system package managers have delivered automatic updating facilities and maintain our applications configurations through new versions of that software. Those same things are desirable for the applications that we deploy at scale on Kubernetes or on OpenShift clusters. As a corollary to this idea that all software has to have its lifecycle managed from installation on, is something we thought a lot about at CoreOS and certainly have continued to work on since the acquisition of CoreOS by Red Hat in 2018, which is where the sort of legacy or lineage of operators comes into Red Hat and OpenShift. That corollary is patching is critical to security. Making sure that we're running the latest released version of the software that comprises our applications is our best armament. In the race, we find ourselves with security vulnerabilities. As patches come out, as those vulnerabilities are discovered and addressed in software, making sure that we actually apply those patches to our software and run the versions of that software that contains the patches for the latest known vulnerabilities is one of the best ways we can stay on top of our first layer security defenses in the applications that we develop and provide. With that idea, any of those functions from installation to patching for security, to applying upgrades for features, and to keeping software running, any of those processes that aren't automated are slowing you down. In fact, that's what brought a lot of us to Kubernetes in the first place. The idea of how easy Kubernetes makes it to automate the lifecycle management of simple stateless applications. If you imagine a web server, all it does is listen for request over HTTP and in return file system content in response to the file named in that request to some HTTP client. It doesn't record state about the session. It doesn't store data or read and write new data in a user's record or in some persistent underlying database. It's in fact completely stateless. Any other web server can replace it. If that piece of software falls over, it's very easy for ourselves or for Kubernetes to automatically start running a new copy. If we want to scale from one copy to three or five copies to handle an increased number of users coming to our website or our service, that's also really easy for Kubernetes to do. In fact, we can scale a fungible, simple stateless app like this up and down just by adding new copies, new replicas. We could say something like kubectl scale replicas equals three to go from one deployed a stateless web server to three or five or whatever number we specify. In certain ways, once we get away from these stateless apps that make it so easy to demonstrate and model how Kubernetes automates application life cycles, it's still pretty easy to move to the next stage, to the persistent or clustered apps that generally lie at the bottom of an application stack. Stateless design is great. Managing stateless applications on Kubernetes is easy and cool. But somewhere down the line, most real-world applications have to store persistent data, maintain a sense of state, and many of those applications, if you think about databases and file system clusters, even have their own notion of what it means to be a member of a cluster. To join a cluster, a database might need authentication assets or to know about who the right leaders and read followers are in a cluster of nodes comprising a database server cluster. Deploying a database on Kubernetes is not really any more difficult than deploying our stateless web server. If we can find a database we like in a container registry somewhere, we can create a deployment with it on our cluster. But running that database over time is considerably harder. All of the usual things we rely on DBAs to know the internal application state of our persistent data in the database and of the connections between nodes that make the database cluster. When they want to be resized to scale that cluster up and down, when a node failure happens and a node needs to be replaced, when we want to back up or reconfigure that database cluster while maintaining database service, this takes detailed knowledge of the internal state of the application, and we have to connect that new pod we bring up to replace a failed member in our cluster, needs to know about how to join the cluster. As I mentioned before, there's a question of, do I need authentication assets to become a cluster member? Do I need to know what my position in the cluster is, and I need to know API and networking in points so that I can begin communicating with the remaining cluster members. That's usually up to a human operator. We call them DBAs a lot of times. If we're talking about a cluster of file system servers, we might call them a systems administrator or even an ops team, but somewhere, rather than automating these processes for complex stateful apps, there's usually human knowledge required of that application's internal state. Wouldn't it be great if we could use Kubernetes and the techniques we use to manage stateless applications, if we could apply those tools and techniques to managing these more complicated, stateful, persistently storing applications? In other words, if only Kubernetes knew all about the internal state of my idealized production ready database server that we're using in this example, operators are a way of teaching Kubernetes about how to run a particular, stateful complex application by extending the Kubernetes API. In fact, operators represent a combination of two things, an application-specific custom controller that watches custom resources that contain the information about that application's internal state and represent it in a way that our custom controller code can take action to manipulate and reconcile it like other members of the Kubernetes control plane. So for instance, we might have a CRD that describes our so-called and notional production ready database. And if you look under the spec, you'll see a couple of fields here that are not usual standard Kubernetes API fields for a standard Kubernetes resource. Instead, you're looking at fields like cluster size and read replicas and even the version of the software that we wanna deploy that are particular to our production ready database for which we're constructing an operator and for which and whose internal state is represented by these custom resources. So you can think of operators as software site reliability engineers that manage the whole life cycle of complex, stateful applications on Kubernetes clusters. So while Kubernetes is excellent at scaling your stateless web server up and down, perhaps even automatically in response to increasing or decreasing demand at given times, Kubernetes operators teach a particular cluster about managing the whole life cycle of a particular application. And in fact, that coupling is so tight that we often refer to the software that an operator manages as that operator's operand. So you can think of this combination of a custom controller with a set of custom resources that it watches, reacts to watch events and takes actions to reconcile in the real world running on the cluster as the simplest definition of what a Kubernetes operator is. Now, pictorially, it looks a little bit like this illustration of our usual process inside of the Kubernetes control plane. We have this idea of a reconciliation loop that runs in response to changes that the controller's notified of by watch events. Instead of the solutions, we provide a custom controller, a bit of executive code that runs in the Kubernetes control plane that knows how to watch, react to and reconcile our specific custom resources. Now, because that's a pattern rather than a particular product or a specific technology, it makes sense for us to develop both the pattern and the tools and distribution frameworks we've built in an open way. And so the first bit of really cool news about operators that we can talk about happening in the latter half of 2020 is the donation by Red Hat of the operator framework to the Cloud Native Computing Foundation. Now, you may know the CNCF. It's a large array of projects under its rubric at this time that include Cryo, the container networking interface, Linkerd, Rook, and a lot of other software that's thought of as Cloud Native, and you might call it Kubernetes adjacent. Now, that means that the CNCF, which is actually part of the Linux Foundation, is open, neutral governance for these products or projects that often have a lot of different vendors interested in them and leveraging them to build platform pieces or to extend platforms the way we use operators, for example, inside of Red Hat's OpenShift Kubernetes distribution as our way of extending, adding features to, and building atop the Kubernetes core of OpenShift to provide the day to enterprise features that that product is all about. So the CNCF as a neutral foundation to oversee the governance of these kinds of projects is a really good place for community-oriented development like the kind we've tried to do on the two main components of the operator framework. That is the operator SDK for building operators and the operator lifecycle manager, which is sort of an operator for operators that manages discovery, distribution, and continuing updates to the operators that are running on a cluster while those operators in turn manage their operands, the applications that they manage and keep up to date over their entire lifecycle. So what it means for the operator framework is that under the CNCF umbrella, we've got a path to graduation that makes the framework, the operator framework that we've developed at Red Hat to build and deploy and distribute operators sort of the official extension mechanism for managing staple applications on Kubernetes. This underlines the design in Kubernetes itself where the API has these modular extension points that you can create in the form of custom resource definitions and then use and manage those custom resources. And those are the benefits we see from joining that community, which makes it clear that other developers in the community who are working on operators can be involved in the management and the future planning of features and fixes in both parts of the framework, the SDK and the OLM. And my good friend, Jason Dobies, also from Red Hat, is about to give us the details of that donated version of the operator framework, especially focusing on the SDK, but with some notes about the operator lifecycle manager as well. So without too much further ado, I'd like to turn it over to Jason. Take it away, Jay. Thank you, Josh, for that completely spontaneous and unscripted transition. So he introduced the SDK and OLM, the CNCF incubation and so forth. Let's talk a little bit more about the SDK itself. So some quick background. The SDK is exactly what you would expect out of an SDK. It generates project scaffolding. It generates a fair amount of code for you and really sets you up at the point of, hey, here's the basis for an operator. Now it's time to implement your business logic. And then it has some build utilities for, you've done those pieces, you've tied in, you've tested as necessary. Now we can actually build this operator and ship it and make it usable. So this is a very high level overview of what the SDK does. But again, if you know what an SDK is in general, you have a pretty good idea of what to expect from it. Now the operator SDK itself, and again, we're talking about this particular CNCF project, which is one avenue for writing operators for facilitating writing operators more easily, supports three different ways of assembling your operators. Going from right to left, actually, the full programmatic way supported in the operator SDK is using the Go programming language. Not a surprise there. Everything in Kubernetes is written in Go. So the SDK will generate you a scaffold project with all of the necessary hooks inside of Kubernetes to register it as a controller and do all of the kind of tedious boilerplate bits and pieces that you need to do to establish your code as an operator. And largely leaves you at a point where it just says to do, implement your stuff here. And at that point, you have full access to everything that the Go programming language offers. Now, the SDK also has the ability to generate scaffolding code that will run an Ansible Playbook or a Helm chart for you. So if you've already invested in those infrastructures in your Ansible shop or you're very heavily invested in Helm, you don't necessarily have Go developers. You can absolutely still use the SDK. And the benefits there, there's a lot more than I wanna cover in this talk, but everything in terms of the delivery through Lifecycle Operator's Lifecycle Manager to the interface using custom resource definitions. There are a number of advantages for taking your existing infrastructure and then using the SDK to bundle it into the operator framework. So with that quick SDK introduction, we're really happy to announce that the 1.0 release came out a few months ago now, but still relatively new, especially in the grand scheme of about two years of really refining what CoreOS started with the operator pattern and figuring out the best way to create these tools and make it as simple as possible for developers to get started and to quickly iterate on their operator functionality. Now, the rest of these slides are gonna cover some of the new stuff in 1.0, but I wanna take a quick interlude and go back in history for a second. Now I've been working with comments for a little over four years now, and I've watched it grow from these fairly small, very intimate events to these very large, back in the days we used to get together in person, events that I would help organize and run as well. And it dawned on me recently that there's gotta be a lot of people listening to this that came in on OpenShift 3 and OpenShift is Enterprise Kubernetes and understood that messaging. Which is fine, absolutely stick with that messaging, but before that time, OpenShift 2, and I'm sure Diane is screaming right now because she would know better than me if comments was still around in OpenShift 2, but I imagine we have a fair amount of new users in the past couple of years. OpenShift 2 was our own attempt at a similar type of concept. And those of you who were around back then will remember things like cartridges and the other terms alluding me right now, but we had these concepts and looked around the industry and said, you know what? This seems an awful lot like containers and awful lot like Kubernetes. So in OpenShift 3, we pivoted to using upstream Kubernetes. And that meant investing a lot in upstream focus when we would contribute back to Kubernetes and that meant rebasing our software on a lot more standards in the industry or at least the fact of standards. Now I mentioned that because those of you who may have used the SDK in his pre-1.x version, it's gonna look a little bit different now. Again, we started out with this custom idea. We knew we had something we wanted to solve. Let's play with some code, let's see what sticks and get these ideas out there. And as things started to mature for 1.0 or I should say the road to 1.0, we looked around the industry and once again realized that we don't need to be competing with a lot of these upstream projects when we could easily be contributing to and really leveraging for the benefit of both the upstream product as well as ours. So a lot of 1.x has been revamped to use CubeBuilder in particular, but we also have some integration with Customize and Cuddle and probably a few other projects that begin with K that I haven't actually listed here. I know I don't have time to make this reference but I come from a Java background and everything began with J and 20 years into my career, now I'm seeing everything begin with K and it's just a really bizarre parallel. Point being 1.x is going to look a little bit different because we've done this rebase on top of CubeBuilder. And again, that's for a number of reasons that we don't want to put up the perception of competing with these other projects. We really are attempting to work alongside them. So let's just contribute to CubeBuilder, let's make it as good as it can be and then add in the extra features that we need out of the SDK. One of those contributions came in the form of a CLI framework where you can do plugins to the CubeBuilder. And we worked with the upstream and we got that functionality landed. And as such, the SDK 1.x has been rebased to now use plugins into CubeBuilder. So if you have an existing CubeBuilder project or if you're just getting started and you're worried about, oh, am I buying into a particular type of project structure or am I locking myself out of something in the future? You're not, this is compatible with everything CubeBuilder is going to do. So the project structure is going to look a little bit different than in the 0.x days but it's more consistent with what other projects in the industry you're doing. You'll see at that bottom that list of commands, a lot of them use make now, which is, again, I've been in this industry long enough to kind of laugh at, oh my God, make is back. So you'll see some of the workflows may have changed but all the functionality is still there. We really just kind of gutted a lot of the pieces that we were duplicating or we could better contribute to upstream. So the project scaffolding, a lot of that came from CubeBuilder, a lot of these generation pieces that you're seeing in terms of the CRDs and the other files we use for OLM are based on top of Customize. So the other thing that we've added into the SDK is the generation for a lot of the pieces necessary for OLM. Now, remember from what Josh said, operator or life cycle manager or OLM is effectively what YUM is to RPMs, OLM is to operators. Now, if you've worked with this in the past, you'll realize that there's a couple of files that you have to hand jam on your own and the CSV you see it mentioned in the middle here, the cluster service version represents your operator but it is pretty verbose and it did take quite a bit of manual effort to get created. Now, all of that is gonna be handled by the operator SDK for you. So being able to run a few commands, point it at your operator and say, hey, I want you to generate me the pieces I'm gonna need for OLM. And this is not just that because as the SDK evolves, so did OLM and their new packaging format is this notion of an operator bundle. This bundle is itself shipped as a container image and in the future that's gonna use the artifacts back but for now it's simply a container image and that contains the metadata pieces around your operator and that's what OLM uses. You can then organize those metadata bundles or those operator bundles of metadata into catalogs and make those available to your cluster. So now you have a lot better organization and versioning in terms of subscriptions coming into new versions of operators being released available to your cluster. We're bringing this back to the SDK. This stuff is now built into the SDK. There's also this new side tool called OPM that's used for manipulating these bundle images and these catalogs. So that way you can much more easily work with, I have this set of operators and I wanna organize my alpha channel, my beta channel and my stable channel for instance. Testing is a good thing. The SDK is obviously going to help with that. So we've integrated with this project called Cuddle. I might have it written down what it stands for but whatever the case, the Kubernetes testing tool, I think it is, there's a lot of weird capitals in there. Point being this tool is again, an upstream project that we've pulled into the SDK and it works very much on this. Here's what my cluster should look like after the operator runs, here's what I expect it to look like. So if that means we have a bunch of pods created or some services and all these extra pieces, this Cuddle project, we reuse all of their formats to say, hey, verify these states for me. So it's also something called scorecard and you can see the link at the bottom there. And that's a much more customized testing for your particular operator and its logic. Now, one of the cooler things is that this data is now packaged inside of this operator bundle. And remember, this is a separate image that is metadata describing your operator. So if I wanna run some kind of acceptance testing that shows yes, my operator is running, I no longer need to go to GitHub and grab the source and understand this is how I run them. This is managed by the operator bundle itself. So it's effectively packaged alongside with the operator itself. So I also wanna talk about some alternative technologies for lack of a better slide here. Again, the SDK is a way of doing things. It is a prescriptive way for Helm and Ansible and Go, but the world is much larger than that. And operators themselves are a concept. It's really a paradigm. It's how you use particular constructs inside of Kubernetes to accomplish something. And just saying that out loud, you can realize that doesn't mean we're necessarily tied to a language. So in the spirit of operators everywhere, I wanted to showcase a few of the other types of projects and frameworks out there. Now myself, I'm a Python guy. You can see the link at the bottom here, the speaker who Sergei, PolicySys last name eludes me right now, spoke at a definition event a couple of months ago about the KOPF framework. And it's the Kubernetes operators Python framework. We'll include a little snippet there. I'm obviously not expecting anyone to copy it down, but if you do know Python, you'll see this. And this feels very Pythonic, very annotation based, very decorator based. And then it's going to facilitate all of the pieces you need to in terms of connecting to Kubernetes as a controller or running your custom resource definitions and generating those. So I'll leave this up for another half a second when people take screenshots or pictures, but definitely watch this talk. It was a really cool introduction to it. And like I said, I'm more on the Python side of things. So I'm really happy to see this gaining some traction. Java is and will remain for the foreseeable future a very popular language. So no surprise, we do have a Java operator, SDK up in the community. It uses your, again, your typical Java constructs here. So your basic Java Pojas that you're already working with. And then we have ways of generating your controllers. Now you'll notice here, the interface looks a little different. And this is way too technical to go down in this talk, but this takes, this framework takes a vastly different approach than the operator or CK does. That's not saying it's necessarily better or worse, but if you've done anywhere with operators so far, you may notice this delete resource and creator update that feels a little different, which is very cool. It shows the power of this as a paradigm and it lets the language solve it as best applicable for its language. And then as usual, you pick the right tool for the right job. There's a C sharp operator framework out there. I don't know much about C sharp at all. So to prevent sounding really stupid, I'm gonna say I'm sure it's really awesome. And I'm gonna smile and move on. Don't take that as a joke. Seriously, take a look at that framework, but I know nothing about C sharp to say anything intelligent. I've even seen a bash-based operator, a coworker reached out to me and said, hey, I was messing around with something and I decided to see if I could make an operator function in bash. Because again, it is purely paradigm. It is a way of doing things on top of Kubernetes. I looked at the code and I thought better than dumping a giant shell script on the slide. So we will just leave that as redacted, but you can see the VEPA was up there and all joking aside about being bash, it's actually really cool that he was able to pull off this type of operator pattern with using just a bash script. So certainly not any kind of framework that's extensible, but it's a really neat way of showcasing that, yeah, this really is a pluggable framework and this really shows the kind of power and extensibility of Kubernetes. Back to you, Josh. Hey, Jason, that's funny, the bash thing, just because, one, it reflects probably some of the very earliest mock-ups and models of the operator idea years and years ago. And two, because early on in containers, we used to joke a lot about re-implementing Docker with just bash scripts. That's what that made me think of while you were telling us about the bash framework for Kubernetes operators. So- How was that kind of guy that had the t-shirt go away or replaced you with a very small bash script? Like, this is totally not my idea if I'm trying to write it. Right on. So, and now that we've kind of talked about a lot of different ways where if you're building operators for your applications, these are the toolkits that are available in the community and also evidence of the adoption and the way the community is working together with this pattern. And as you can see in like these variety of different frameworks that address building operators in different languages rather than go and even in bash like you were talking about. So the end result of all of this and the reason you really want operators is either for your applications to be a lot easier for your end users to consume and manage or for the foundation software that the apps you build every day rely on, your databases, your file system stores. And so one of the things we try to point to and where a lot of progress and a lot of news has happened over the course of 2020 is in the using operators today through both Red Hat Marketplaces and especially the thing we'll focus on here, the operator hubs. Now at operatorhub.io you'll find a list of operators ready to run on your Kubernetes clusters or your OpenShift clusters that have OLM management and OLM itself being open source is an out of the box component that we ship with every single OpenShift Kubernetes distribution. But it's also easily available with instructions available at operatorhub.io for adding the OLM to your vanilla or other service providers or other distributions that are running your Kubernetes clusters. So the things we're talking about are not OpenShift specific but they bring this sort of graphical discovery user experience to operators that we think is one of the big reasons why people are adopting them, building them for the apps that they distribute or adopting them to make it easier to build those apps because it makes it easier for their developers to run a corporate approved stack of databases and middleware or one way or another the underlying foundation apps that are key can be consumed along with a built in automatic lifecycle manager in the form of that operator that runs that application. So if you look at operatorhub.io or operatorhub within an OpenShift cluster you'll find a way that for administrators to make it easy for users on those clusters to self-service to get these operators, install them on the clusters and make instances of the applications they manage whether that is a cluster of database servers, file servers, a cluster of SEDs providing raft quorum for your distributed computing project that runs atop SED. Those are the kinds of things you'll find in the operator hubs and the community representation of those ready to use today operators can be found easily at operatorhub.io. So let me jump in while you took that quick break. In terms of integrating vanilla Kubernetes like it is straight up built into MiniCube. If you look at the MiniCube's add-ons there is one for OLM and just installing that it's gonna wire up a source directly to operatorhub.io. So it is as simple as the MiniCube add-ons and then you just start creating subscriptions to get those operators deployed. So just to reiterate this message that this is not an OpenShift specific concept. These are all base Kubernetes things that we're leveraging. They're leveraging the very heavily inside of OpenShift but you do not have to buy into the OpenShift infrastructure to start developing on them or start using this functionality. But since this is OpenShift Commons we mainly want you to do exactly that. Nevertheless, I do mean that you're gonna pick some of those great points Jay, actually. And one of the big driving things even from the earliest days at CoroS that I could speak to about the operator idea or pattern and especially as we started hitting on the idea of an OLM, a distribution and manager for operators, a distributor and manager for operators. That goal of making sure that it's not only open source but actually openly runnable on different instances of what amounts to platforms built around the Kubernetes core has always been a key design goal for those two projects. So it is an important thing to bring back up. So with much appreciation for the half hour or so of your time that we've already taken up and with hopes that all of that information has been useful and it'll encourage you to run out and get involved in operators whether you start using them today or building them for your apps. We do have some resources that you'll find in these slides after our session. I've pointed in particular to operatorhub.io as a distribution point for operators. If you're looking for how to build operators and more about the concepts and the documentation behind the operator framework, the SDK and the OLM, you wanna look to operatorframework.io, that first link on the slide. And obviously we can never resist one shameless plug for our book with O'Reilly. Ooh, Jay's even got a copy to hold up. If you ever get the chance- I proactively remember to keep it near me actually. Yeah, right. Oh, and with a pen ready to sign it just in case somebody walks in and asks for it. Of course. If you ever get to see us in person again, we'll be so happy to sign a copy of this book for you folks. But in the meantime, if you follow that bottom link, that bit.ly link there, that is a pointer to an electronic edition of our Kubernetes operators book available for free with compliments of Red Hat and OpenShift. There's also a few places we would point you to get in touch with the community since you're at a community event like Commons in the first place, this is probably going to be important and part of what you wanna know about how to find operator framework mailing lists, how to find the GitHub repos that contain the operator framework and OLM source code and how to join the Commons special interest group that addresses this idea of API extension and custom controllers, the operator SIG within OpenShift Commons. So with those few pointers and hoping that you'll come out, find us, ask questions and get involved in building and using operators, we've been Josh Wood and. Jason Dobies. And we thank you very much.