 Hello everyone. Thank you for attending our KubeCon EU 2021 presentation where we'll be covering the Operator Lifecycle Manager project commonly referred to as OLM. My name is Alexander Green and I'm a senior software developer working at Red Hat, primarily focusing on the Operator Lifecycle Manager project. My teammate Dan is also a developer for the OLM team. In today's presentation, Dan and I are briefly going to introduce the concept of a Kubernetes operator, discuss some of the challenges associated with managing a Kubernetes operator as it evolved over the course of its lifetime, and finally introduce some of the features that OLM includes that makes it easier to manage these resources. So typically we rely on Kubernetes for deployment, scaling, and management of applications. But at the end of the day, we really just want to have a service available to our engineers or our customers. Take, for example, the classic MySQL application on Kubernetes. MySQL is a relational database management system that consists of both stateful and stateless applications. If you take a look at the resources at the bottom of this slide, you'll see a deployment channel, a service, a persistent volume, and a persistent volume claim. As a developer with limited MySQL experience, it's fairly easy for me to get a MySQL service up and running on my cluster by just creating four Kubernetes resources. Unfortunately, there are a few shortcomings with this approach. First, each of these resources are independently managed. I don't really want to manage each of these resources on my own. And when I have a disruption to my MySQL services, I won't immediately know where to look. I might have to look if the deployment came down or the service itself was deleted. I might have to see if the persistent volume or persistent volume claim encountered an issue. It would be much better if I could look at a single resource to identify what's going on with my MySQL application. Perhaps more importantly, I don't actually know how to safely perform an upgrade to a MySQL database. Now, sir, I could devote a lot more of my time to understanding MySQL, to understanding how the upgrade should be performed and make it safely happen. But I would much rather leave it to domain area experts to codify and handle this upgrade path. And this is where operators really come into focus. Operators allow domain area experts to codify the operational knowledge and software life cycles into an operator. This operator exposes a Kubernetes API that allows me to define what type of application I want installed. In this case, it allows me to define the MySQL database that I desire. And based on that information, it's going to create all the Kubernetes primitives such as deployment, stateful sets and everything else you can imagine so that that service is available on my cluster. You may be wondering how this is done. Operators are composed of just a few components. First, you have a controller, which is just an application that runs on your cluster typically as a pod that includes a client. This client allows it to communicate with the Kubernetes API to understand the current state and the desired state, as well as creating, updating or deleting resources. Once the controller is created, you have the custom resource definition. The custom resource definitions are responsible for expanding the Kubernetes control plane by introducing a new API. These are cluster-scoped resources that define the API's name, version, scope and supportive fields. The CRDs are effectively blueprints for resources that the customer can create. The resources that the customers create that implement the custom resource definition are typically called custom resources. So with this information in mind, a typical approach to using an operator would be for a user to create the MySQL CR in our example. The operator would notice that that event occurred. If you look at the MySQL database described by the user, it would try its best to create that MySQL application. And then it would write the status of the MySQL application to the status of the custom resource I created. This gives me a single window for me to define what I want on cluster and then a single place to look to see the status of that resource. Hopefully that makes it fairly clear why operators are valuable in the Kubernetes ecosystem. It allows you to build Kubernetes native applications. But let's discuss why OLM comes into the picture for the management of these operators. So when you decide to use OLM to manage your operators, you'll automatically gain access to an extensive ecosystem designed to make expanding the Kubernetes control plane very easy. OLM immediately grants users access to a catalog of over 100 operators. And the contents of this catalog is updated in real time as new versions of the operators become available. When you install an operator using OLM, that operator can place dependencies on other operators in the ecosystem. Doing so allows operators to take advantage of services offered by these different operators. So I could write a front end website application that relies on the MySQL operator to store database, store information in the database it provides, allowing me to build a very good complex application without reinventing the MySQL database operator. Finally, once these operators are installed, users are capable of identifying which operators are running, which custom resource definitions are available to them, in each namespace. So despite these features that I introduced earlier, you may be wondering, well, why can't I just use a different tool to install and manage the operators on my cluster. This is mostly derived from the fact that the management of CRDs and operators as they evolve over time is far more complicated than that of typical Kubernetes primitives. By installing a Kubernetes operator, you are expanding the Kubernetes control plane. And as soon as the CRD is introduced, users can take advantage of the services offered by the operator by creating CRs. Changes to the custom resource definition associated with the operator could invalidate these existing resources on cluster if they don't adhere to changes in the schema, such as when a field is removed or a new required field is introduced to the schema. Perhaps more troubling is the fact that if a CRD is removed from cluster, every CR on cluster is immediately deleted, potentially causing customer data loss. This is extremely concerning when you think about our previous example where we discuss a MySQL operator. Imagine if you had workload operator installed and a user accidentally deleted the CRD causing all of the customer data to disappear. That's not a good time. So outside of the possibility of rendering existing workloads invalid and losing customer data, another big issue with using operators without OOM is the fact that Kubernetes does not offer a concept of CRD ownership. When a CRD is introduced to a cluster, it is not tied to a specific controller that consumes the resource. Rather, it simply introduces an event stream that any operator can watch and reconcile. If you introduce the CRD, there could be multiple controllers that watch for events on the CRD in the same namespace. This can cause race conditions where the two operators constantly go back and forth updating the resource and the resources they create. Finally, there are other complex scenarios that make upgrades being an operator potentially dangerous to users when you don't have a system like OOM in place. As a user of an operator, you deserve to be provided with the opportunity to upgrade an operator safely as new versions become available. In my opinion, one of OOM's greatest features is the ability for operator authors to define safe upgrade paths that are pushed to customers clusters as they become available. This approach allows operators, authors to ship bug fixes and new features in real time to customers all over the world as they become available. You may be wondering which operators are available with OOM. This list of operators can be found at operatorhub.io. These operators are developed and maintained by domain area experts that are trying to get their product out to you in a safe and controlled manner. Now that I've discussed the impact of operators and the number of features that they are supported by OOM, my teammate Dan will provide a bit more context surrounding how catalogs are built and made available in your cluster. Thank you. Thanks, Alex. So I'd like to discuss today the operator bundle format and some different ideas around packaging and delivering Kubernetes manifests and Kubernetes data onto the cluster. So to begin with, let's look a little bit about how we package and deliver content in the Kubernetes space today. Generally we have solutions that evolve around Helm charts, customize or getups, get repositories, flows or simply raw YAML. These are all the different ways that people commonly take Kubernetes manifests, template them and push them to a cluster. There's also another variation on this, which is to use container registries, which are generally available from within the cluster or easily externally via hosted registries like Koya and Docker Hub to store manifest, to store Kubernetes data. And this is a very attractive sort of system because the Kubernetes cluster already knows how to talk and pull down and push images from a registry. So we take a step in that direction and we start thinking about how do we package Kubernetes manifests in container images. We have this manifest bundle Kubernetes enhancement proposal that we merged recently and it basically describes how we can associate a set of Kubernetes YAML manifests with one or more vulnerable container images to provide a standard way to transmit data between clusters. If you think about some of the features you get for free when you tie in your Kubernetes manifests with a container image, you get things like identity. The container image is hosted in a certain repository on a certain registry so you get identification with it. You get distribution in the sense that the container registries are highly available and accessible from within the cluster. You get replication, deduplication and signing. For example, you can sign container images to give more information along the software supply chain on where that image came from, its provenance. And you also have cluster ingress generally as I said, because everything in Kubernetes essentially rolls around a pod and a pod has an image in its spec, so there's a lot of tooling around dealing with container images and runable container images in particular. So this manifest bundle cap is a way to try to standardize how to package and associate Kubernetes manifests with container images. So to make it more clear on the next slide here I have an example of an operator bundle. So this is the LCD092 operator bundle. And we see that at the top level there's two directories, slash manifest and slash metadata. The manifest directory contains the kubectl-appliable level manifests. So essentially everything that you would say as your CRDs, your core group resources like config maps and services, your OLM specific custom resources like cluster service versions, everything that you can apply to a cluster via kubectl-appli-f, that would be under your manifest directory. Whereas the metadata directory is a directory that only contains application metadata and this includes things like name, version, package information including dependencies and a whole host of information that any sort of package manager like tool would need to know to be able to install and manage your manifests in a more robust way. And so this is an example of an operator bundle and this bundle fully defines a particular version of the LCD operator package. And so conceptually what you can think about is that if you have a cluster, you may want to install an operator bundle onto it. You may want to install multiple operator bundles onto it that all associate themselves with different services like the MySQL operator that Alex mentioned earlier, the SD operator, et cetera, et cetera. So you want a way to sort of group these bundles together and expose them within the cluster. So on that note, that's where the idea of composing bundles comes in the index. For example, here on operator hub, we have an example of a couple of operators which are associated with bundles. So what we can do is we can compose these bundles into an index, which is itself a runnable container image, which has a set of bundle references inside of it that knows how to provide bundles to install via an API. You can basically package up these different bundles, like let's say the Istio and Akiali bundle into an index. And then you can ask the index, hey, can I install this Istio operator or Istio mesh inside my cluster and the index will be able to surface all those manifests, all the data that you need that's stored within those container images to actually install them onto the cluster. So what the index does is it enables one to have a content addressable store of operators and services from within the cluster, even in a disconnected environment. And where you can mirror all the content that you need ahead of time into the internal cluster registry, image registry, and then install things that way. So index is basically a composition of bundles and is very powerful tool for using and installing different operator services onto your cluster. So that was a brief introduction into the Manifest bundle format. And with that, I think I like to talk a bit about the future of OLM and some of the things that we have in store in the next year or two. So going on the same topic of the bundle format currently OLM supports certain bundle resources, several of them certainly, but not all of them. And one of the big things we want to do is we want to support bundles with arbitrary resources, including custom resources and policy security policies. So you can imagine that their bundle, let's say the operator bundle packages a certain version of an operator. If you allow the user to package a custom resource into that bundle alongside the other manifests, you can basically have an init configuration for your operator on startup. After the CRDs are installed on the cluster, there can be a CR already there, so the operator can use it to have some more advanced configuration than it would in maybe like a config map. So that provides some interesting use cases. Also, policy security policies are something that generally associated with security and overall robustness. There's a big ask for being able to lock down the workloads associated with operators and ensure that there's no privilege escalation. So we want to open up this bundle format to basically everything, so OLM will be able to gladly accept any manifest that it finds in a bundle. Another big thing that we're working on right now is around declarative indexes. So indexes today, as I described earlier as sort of a composition of bundles, they package all the bundles into a SQLite database internally and then expose the API, the serve bundles API over a GRPC connection. And that works well, but it generally produces some edge cases and pain points. And what we want is to have a more GitOps friendly design in that we have a declarative index config file, which is essentially just a big JSON blob that users can use to edit and replace and update in place the different bundles that compose the index. And basically have really nice experience in modifying an existing index or building an index from scratch is just a JSON file that an API can serve in front of it. And part of this is also using other open source projects like QLang to help validate that the index JSON file meets certain conditions and basically makes it much more easily extendable and usable by other projects. So this is another exciting initiative around declarative indexes. Lastly, we have V2 APIs. V2 OLM is being actually developed on sort of in a prototyping stage right now. I think one exciting thing is that we are planning to allow users to unpack and run any standalone bundle image without an index. So you can just say I want this version of this operator on the cluster and we can install it. No problems. Another which is very interesting is the idea of using pluggable content provider controllers. So right now all of them sort of want all the thinking that it's a series of controllers. But we could sort of make it more modular. So more similar to the existing persistent volume persistent volume claims storage class design that's already in native Kubernetes, so that different content types can be supported via different controllers. So the OLM.bundle type which is sort of akin to the bundle format that I described earlier would be described by the OLM controller. But you can imagine there'd be another Helm.chart type media type which is a Helm chart. Helm controller knows how to basically when it's season one, it knows how to interact with it, how to unpack it and how to put it on to the cluster and the manager's lifecycle through updates and through deletion as well. And so basically becoming more modular in what kind of applications media types that we accept and maybe accept more than just simply operators right now. OLM is very focused on operators, but we want to support different types of applications in the future. So this is sort of the OLM.b too. Another interesting initiative we have is around cert manager. We want to integrate with the cert manager project, which is an outstanding Kubernetes project to issue and manage certs when installing operators via OLM. OLM currently handles CA cert generation and rotation using self-signed certs for all operators web hooks and API services that it manages. And so delegating to cert manager would naturally be very attractive to users who want sort of, you know, a highly advanced and configurable cert managing when they install operators and other operator artifacts. So that's about a bit. That's a quick run through. Here is some information on the OLM community. OLM is part of the operator framework, which is a CNCF incubation project. And we have a public working group or issue charge meeting every week. Every week on Thursdays we have a meeting. Please check out the operator framework OLM Dev mailing list for more information, sign up to the mailing list and you get meeting advice to the calendar events. We also have a Slack channel on the Kubernetes Slack hashtag OLM Dev, so please stop by there and feel free to ask questions. And lastly on the bottom, there's our community page where you can find links to all the meetings and all the information. I'd like to thank everyone for coming to this meeting and learning a little bit more about OLM. And now me and Alex will be around and happy to take any questions or comments that you guys have. Thank you, everyone.