 Thanks for being here, everyone. I'm Adam. I work at DigitalOcean. I'm going to talk today about what happened to the service catalog. And before I start, I want to say I'm very happy to be here. I was supposed to be here in Amsterdam in 2020 to give a talk. And then the global pandemic started. And I had to give that talk alone in my office online, which was not as fun as being here with everybody. So thanks for coming today and really enjoying being here at KubeCon. So today, I'm going to talk about the service catalog. And service catalog was a Kubernetes SIG and a Kubernetes project. It started around 2016, and it shut down last year in 2022. And at least nominally, this talk is about why it shut down, what happened to it. But really, what I want to talk about is something a little bit broader, which is how we consume external services from applications running under Kubernetes and the way that we manage those services. The service catalog was one attempt at doing this. This is what it was for. But I think it's more constructive maybe to talk about the solutions that have replaced it, what people are using today, and maybe some directions for the future and things that we can do going forward to solve this problem a little bit better. So I don't usually put this kind of slide in my talks, because I don't think you actually care that much about me personally. You care more about the topic. But today, I am going to give some opinions. And so I think it's important that you know sort of where I'm coming from, what my biases are. So I'm going to talk a little bit about myself. Again, I'm Adam Wolf-Gordon. I work at DigitalOcean. DigitalOcean is a cloud provider. We provide all the things that a cloud provider might provide, like virtual machines and managed Kubernetes and managed databases, network things. And these days, I work in our product strategy group. So I work across all our products, helping align roadmaps and think about what our customers need and that sort of thing. But before this, I was the tech lead for our managed Kubernetes and container registry products for the last few years. And one of the things I thought about a lot in that role is the way that our customers put all the pieces together that we offer to support their applications. No application is an island. They do need other things. And our customers need to put those pieces together to be effective. So that's kind of the background of this talk, is thinking about that problem. And before I talk more about service catalog, I want to give a disclaimer, which is that I was not directly involved in service catalog when it was a project, except for as a user of it. So back in 2020, at DigitalOcean, we started to think a lot about this problem of how do our users manage other services that they want to use when they're using Kubernetes? And as a POC, I built a service broker for us, and we set up service catalog and tried it out. And we really liked that approach. The overall shape of the solution made sense to us. And we plan to go forward with it. We put it on the back burner while we worked on some other things. And when we came back to re-evaluate last year and actually start working on this, we discovered that service catalog had gone away. And so I was kind of curious why that happened and what replaced it. So like I said, I have opinions about this general problem space, and I've used the service catalog. But I wasn't in the room. I'm going to try and stay away from any sort of political pieces of maybe what happened to the project, because I just wasn't there. That said, as part of putting this talk together, I talked to some folks who were deeply involved in service catalog, and so a big thank you to Erin, Carolyn, and Gabe, and Jonathan and Constantine. I don't know how many of you were in the room. I know Jonathan is, because I met him before the talk. But they were kind enough to talk to me or at least give me a few lines on Slack about their views on what happened here. So I want to start by backing up, and let's look at the problem that we're trying to solve with service catalog and a little bit about how it works. So if you're building an application today, and you're going to host it in Kubernetes, maybe your application needs a bunch of things. It needs a database and a message queue and an object storage bucket and a way to send email. And you don't want to operate those things yourself. I've operated object storage before. It's a giant pain. I don't want to do it again. You probably don't want to operate yourself either. So you're going to use managed services for all of those things, maybe from the same provider where you host your Kubernetes, maybe from a different provider depending on your needs. And when you're developing your application, you probably provision all of those things manually. You go into your Cloud Providers GUI or CLI. You create the dependencies, and you copy and paste the credentials into your manifest, and you deploy, and off you go. But when you go to put this into production, that clicking and copying and pasting, it doesn't scale. It's not repeatable. So you want to have some automation. And you might do that using Terraform. You put all your dependencies into a Terraform config, and you run that as part of your deployment process. And then you template your manifests, and you inject your credentials in that way, or you use something like Vault to keep secrets. That works. But now you've got these two separate declarative sources of truth. You've got your Kubernetes manifests for your application, and you've got your Terraform configurations for everything else. And you have to put those pieces together yourself. It would be really nice to keep this in one ecosystem and be able to manage things in one place, have a single source of truth for all of your infrastructure and your application. And service catalog is an attempt to solve this problem. I want to look a little bit at how it worked so we can get into some of the advantages and some of the issues that it had. Service catalog was built on top of the open service broker API. The OSB API originated in the Cloud Foundry community. And the idea behind this API is that providers of cloud services, like DigitalOcean and all the other clouds and various vendors who run things in the cloud, would implement this API. And that would allow platforms like Cloud Foundry or Kubernetes or any other platform, theoretically, to manage instances of those services. So this API allows for this nice decoupling between the provider of a service and the consumer of a service, where the provider implements the API once, and they don't have to implement new code every time somebody wants to manage it from a new platform. The platforms also implement this API once, and now they don't have to write new code every time they want to consume a new service. So the basic objects in this API are called services, plans, instances, and bindings. We'll talk about these things a little bit more when we get into the service catalog side of it. A service is like a kind of service. So my SQL would be a service. A plan is a particular configuration of that service. And then an instance is an actual running instance of the service. And I hate to use the word instance to define instance, but that's what it is. So it's like an actual running database somewhere. I think the binding is the most interesting object here. It's a relationship between an application and an instance of a service. And it does two things. Number one, it sort of architecturally represents that relationship, which is interesting. But more importantly, maybe for practical purposes, is it includes the information that the application is going to need to connect to the service. So it includes your connection string, your credentials, or whatever it is that you need to actually connect. Service catalog was the Kubernetes implementation of OSB on the client side. So it used Kubernetes resources to map onto this OSB API and starts with the service broker resource, which configures which brokers the service catalog should talk to. And then we have types that map to all of those different open service broker objects that we just talked about. So we have service class, service plan, service instance, and service binding. Those map onto service plan, instance, and binding in OSB. As a user, you can list or inspect the service class and the service plan to see what's available. And then you create these service instances to actually create an instance on the provider. And you create a service binding again to get those credentials or whatever you need into your cluster. The neat thing about the service binding object is that it has a secret name as part of the spec. And with that secret name, service catalog is going to put whatever is provided by the provider in that service binding into that secret. And that's how you actually inject the credentials into your application. So that's how we're solving that copy and paste problem that we talked about. Having talked about that, let's talk about really the meat of this talk. What happened to the service catalog? Why don't we have it anymore? And to do that, I want to talk first actually about the advantages of the service catalog, some of the nice things in this model. And then we'll talk about maybe some of the reasons why it was replaced by other things. So first off, a huge feature of the service catalog, at least in my opinion, is the discoverability aspect. The OSB API tells you what services and plans are available. And they also have helpful metadata, like how much a plan costs that could be part of that API. And you also get a schema for how you create instances. So the parameters that you need and their types are there in the API. This provides a really nice user experience. You don't have to go somewhere else to look up the slug for a plan when you want to put it into your spec. It's right there in Kubernetes. And you could do things like implement auto-complete on top of this, implement validation on top of it, all of those things. I haven't really seen any other solutions that have this degree of discoverability built in as a first class feature. So I think that's a really nice feature of service catalog. Another nice feature is that decoupling that we talked about earlier. Service providers implement OSB once. Consumers implement it once. And then they're separated by this nice clean boundary that's the goal of all APIs, right, is to separate things. And as a provider, then you could at least theoretically add features, add services, add new plans to your broker. And the people consuming those services don't have to update anything in their clusters to get access to those new services and plans and features. You update it once, and now it's all of a sudden available to everybody. That's great for consumers as well, of course, because there's less operational stuff that you have to touch. And another big feature here is this idea of a service binding. Having this explicit relationship between an application and an instance of a service is powerful. It's nice architecturally. It also solves this problem of getting connection information into your applications in a consistent way across multiple providers. So that makes it theoretically easy to switch out what implementation you're using. Use a different database provider in your staging environment than you do in your production environment, but configure them the same way. That's something you don't have to learn every time you switch to a new provider. It works the same across all of them. So I'm calling downfalls. And this is really the meat of the talk of why good service catalog not work, despite all those advantages we just talked about. The first one was just execution and timing. And I don't want to say that anybody did a bad job of implementing it. It's not execution in that way. But more that just the time that it was built was maybe unfortunate. Service catalog was built before CRDs were added to Kubernetes. It was eventually converted to use CRDs. But the whole idea of extending Kubernetes with custom types wasn't really mainstream yet in the community. What that meant was the developers of service catalog had to build a lot of that plumbing themselves, had to sort of figure out what this looked like and how it worked. And it took a while for the project to really be useful and fit for purpose and do what people needed it to do. It just kind of missed its moment in terms of popularity. And I think any project has sort of a limited interest lifetime maybe before it takes off. In this case, it just didn't take off at that time. Probably is that it depended on this OSB API. And the OSB API itself never became mainstream. Providers didn't implement it en masse, which is really what was required for this to be truly useful to a lot of people. One thing that that meant is that broker implementations were typically not built by the providers of the services. And they weren't operated by the providers of the services. They were built by people who were interested in this service catalog concept. And then they had to be operated by each individual cluster operator. So that sort of defeated some of that advantage of the decoupling. You still had to operate this thing in your cluster in order to broker onto a service. And the providers couldn't take advantage of that feature I talked about of being able to update just on their side and have consumers take advantage of it right away. There was also limited interest. This is maybe part of why OSB didn't catch on. There was limited interest in the full potential of using multiple providers at the same time. I think that's maybe a more common pattern now as we see more specialized companies building individual services like databases and not just big cloud providers providing these cloud services. But at the time, people weren't as interested in that. Of course, as cloud providers, we're not interested in that as much. We want people to use our own services. So that made me value of the service catalog wasn't fully realized for a lot of users. In terms of technical features, one issue with the service catalog model is the lack of ability to validate resource definitions. I talked earlier about the OSB API providing a schema for how to create an instance. And that lets you do very basic validation of making sure the parameters are there and that they're the right types. But validity is more complex than that. Sometimes you might have certain combinations of parameters that are valid and other ones aren't. There's a little bit more business logic that goes into that. And OSB didn't provide a mechanism to do that kind of validation. What that meant was that you would create your Kubernetes manifest, apply it into your cluster, and that resource would get created successfully. And then in the background, service catalog would fail to provision your instance. As a user, that's a confusing user experience. It's very confusing to debug. It would retry infinitely and just keep failing. It's much better to fail fast and fail synchronously whenever we can. And that was one awkward piece of the user experience for service catalog. And finally, an issue that's actually common to everything I'm going to talk about today is there were multiple sources of truth. And anytime you're managing infrastructure from Kubernetes or from infrastructure as code in general, you are going to have multiple sources of truth. You have the state of your instance that you define in your Kubernetes manifest or in Terraform or wherever. And then you have the actual instance that lives on your provider and is a almost physical thing in the world. And so those two things can diverge. In particular, as a user, maybe you provision something using this service catalog or using whatever solution. And then you go directly to your provider and you modify it or you delete it. What's the right user experience there? I don't think anybody has really decided how that should work. Do we reconcile it back? Do we recreate it if it's deleted? Does that make sense for all kinds of services? I don't think there's anybody doing a great job of this. And it is going to require some more collaboration, I think, across the industry to really come up with a meaningful model for this kind of problem. So a service catalog was maybe failing to gain traction. CRDs were introduced to Kubernetes and the operator pattern really took off and became a huge thing as it is today. Operators were initially a tool for solving a slightly different problem, which is managing complex software that runs in Kubernetes. But people very quickly realized that you can use an operator to manage anything. And that includes things outside of the cluster. So that means you can apply it to this problem as well. For anyone who's not familiar with operators, an operator is a custom controller for Kubernetes. So you create a custom resource definition that represents the state of something. And then you implement a controller, which we call an operator for that CRD. And it reconciles the state of whatever it's managing to match the state that's defined in the cluster. Usually what you are managing is other resources within the cluster, things like pods or persistent volumes. This is great for operating things like distributed databases where you need some domain-specific knowledge that goes beyond what Kubernetes can do itself in order to manage it effectively, manage quorum, things like that. But in the context of what we're talking about today, operators can be used to manage stuff that is outside the cluster. So for example, what we ended up building last year at DigitalOcean was an operator for our managed databases, so that users could create and manage databases from Kubernetes. In this case, you define the properties of the database that you want to manage in your customer resource. And then the operator uses the DigitalOcean public API to manage those instances of databases and also to pull credentials into the cluster and put them in secrets for you to inject into your applications. So operators do have some advantages. There are reasons why operators took off and sort of supplanted service catalog for this use case. First off, they are totally custom for every provider and every service. So that means as a provider, I have full control over how my operator works. I can make the customer resource definitions map really well to my services model. And there's no awkward parameters field. The fields are right there. They're in the spec. It feels like a normal Kubernetes object. Relatedly, when you build an operator, you can use your existing API. There's no new API to learn, nothing new to implement on the API side. You already understand the API that you offer for your product and you don't have to understand Open Service Broker as well. Like I said, operators feel like Kubernetes. Kubernetes is a bunch of controllers. That's what it is. And operators are just more controllers. So this concept of an operator is very Kubernetes native. That's a big reason why I think it took off in the community. It feels like a natural extension to Kubernetes. And there are good tools for building operators. It is very easy to get started building an operator. There's CubeBuilder, operator SDK, controller runtime. All of these things are great tools that you can use. There were some tools for building service brokers, but they never really took off and had the same mind share that these operator tools have. There are tons of developers who know how to use these operator tools or familiar with building operators. And that's a huge advantage in terms of having this kind of thing catch on. And related to those and calling back to what we talked about in service catalog, it's very easy to do client side validation in an operator. You can use admission control web hooks for this. You can also do things like defaulting with admission control web hooks. And in our DigitalOcean database operator, we actually have a web hook that then calls out to our public API to validate that a request is valid. That means we don't have to duplicate our validation logic and also you can be pretty sure that if your manifest is accepted, it is going to actually create an instance for you as long as the API isn't down or something like that. That being said, there were a lot of things I liked about service catalog. As I talked about earlier, it was sort of our plan at DigitalOcean to go forward with it. So I want to talk about what are some of the limitations of the operator model when it's solving this problem? Number one is the coupling. We talked about how OSB decouples the provider and the platform or consumer really nicely. With operators, it's kind of the opposite. They're very, very coupled. If you want to offer a new feature in your product and you service and you plan and you have an operator, you probably need to update your operator to support that. And then all of your users have to update the actual running operator in their clusters. So there's this sort of burden of support and maintenance that's introduced and affects both the provider and the consumer. Second, there's no discoverability in operators, at least not built in. Theoretically, you could add CRDs that represent things like service classes and service plans. But I haven't actually seen anybody do that. Even I didn't do that. And I knew that we could. Theoretically, you could do this. But there's no standard model for it. And really, even if you did, it would be different for every single operator. As a user, you would still have to learn how that discoverability works for each service that you're using. And that limits the value of the discoverability in the first place. And same problem with service bindings. Operators fulfill that basic need of a service binding in terms of getting credentials into your cluster, getting connections into your cluster. But it works differently for each and every operator. There's no standard model for thinking about how bindings work in this operator world. So that means that as a user, if you're using a bunch of different providers, you've got to learn how each of those operators works. And you have to know where the secret is going to be created, how you configure it, all of that stuff. You don't have that consistency that you had with service catalog, but working the same everywhere. So the last current solution that I want to talk about today is crossplane. Crossplane is a CNCF project. It's designed for building deployment platforms on top of Kubernetes. And one of the things it lets you do, as part of that, is provision cloud resources via Kubernetes resources. So that's a lot like what an operator lets you do and a lot like what service catalog let you do. So it can definitely be used to solve this kind of problem that we're talking about today. Before I talk about how crossplane works, I want to give a disclaimer, which I don't understand crossplane very well. I actually find it a little bit hard to describe, although I did go for dinner with crossplane people last night and that helped. But if I get things wrong about crossplane and you're a crossplane person in the crowd, please come and find me after and correct me. I would love to know more about it and understand it more fully. That said, crossplane is sort of a framework for managing resources using Kubernetes APIs. Different service providers build what are called providers for crossplane. And these are a lot like operators, but they use some crossplane glue. Platform teams at individual companies or developers can then use these providers to compose resources from the same provider or from different providers into a meaningful abstraction for their use case. I think of crossplane as providing the glue to stick together resources from potentially different providers in a consistent way. So quick advantages and disadvantages here, since like I said, I'm not as familiar with crossplane, I won't spend quite as much time on it, but one advantage here is I think crossplane really fills a bit of a gap between what we had with service catalog and what we have with operators in that the provider implementations are completely specific to the service and to the provider. But there is this common resource model across all of them. So that means it can fulfill some of that original promise of the service catalog, which is having a single way to think about all the resources that you use externally across all your providers. It is also Kubernetes native. It feels like you're using Kubernetes just like with an operator. That's really helped it have strong momentum and adoption. There's a ton of providers for service crossplane, and many of them are built either by the service providers themselves or by upbound to maintain crossplane. So they are well-maintained. One reason for that is there are also really, really good tools for building providers. Not only frameworks like we have with operators, but also code generation tools. So if you have an open API spec or you've got a Terraform provider that you maintain, you can actually build a crossplane provider on top of that. And finally, it is possible to do client-side validation via web hooks with crossplane, and you have a concept of claims, which are a lot like the service binding concept. I'm gonna say I don't fully understand claims. That's one part of crossplane that's a little bit opaque to me still. But I think if you squint, they look like service bindings and that has a nice architectural feel to it. And onto downfalls. I hesitate to call these downfalls because crossplane is going strong. It's very popular. I don't wanna apply that it's falling down in any way, but I think here are some issues that I see with it. First off, it's just complexity. Crossplane is very powerful. It can do a lot of stuff, and that's great when you need it. But to get started with it as a developer and use it to sort of just provision what I need for my application, it's a lot to learn. There's a lot of terminology and stuff that you've got to sort of acquaint yourself with in order to even get started with crossplane. Next, a few things we talked about with other solutions already. The slack of discoverability, there isn't discoverability in crossplane in the same way that we had in service catalog. And that multiple sources of truth problem hasn't gone away. There is also still this coupling. If you're running crossplane yourself in your cluster, you are still gonna have to update those providers anytime you want new features from them. And as a service provider, I have to update my crossplane provider anytime I wanna offer something new in it. The last thing here is not actually something that's crossplains fault in any way, but I still wanna talk about it. I think it's very tempting to use crossplane as a generic infrastructure management tool like Terraform. And I think there's a little bit of a risk that in the community it gets thought of that way and maybe becomes more of an infrastructure code but make it Kubernetes tool rather than solving this actual core problem of how do you manage resources that you're using from Kubernetes applications. I think lots of people are using it the right way. That's certainly, you know, the big users of crossplane are using it to build these developer platforms and that's what it's for. But I do think there's a bit of a risk just in the messaging around how it is used and certain people are gonna use it as a Terraform replacement which maybe distracts from the goal of it a little bit. So finally, this talk is kind of about the past but I also do wanna talk about the future like I said at the beginning and maybe give some thoughts about where can we go in the future to solve this kind of problem. And I'll start by saying I don't have a solution here. If I had a solution I wouldn't be giving this talk, I'd be talking about the solution instead. But there's some directions I'd like to see us go and wanna talk about a few of those. One general direction that I'm really interested in is what I call application-defined infrastructure. This is the kind of model where developers declare what they need in terms of capabilities for an application and then some other layer translates that into infrastructure and service instances. One model for this is what we call infrastructure from code and there's a bunch of tools that do that like AMP to nitric and modal. There are a few that I've seen. Those aren't really in the Kubernetes and cloud native world. In the cloud native world, this is a bit of a concept in the CNAB, cloud native application bundles project and Porter, which is the implementation of that. But maybe not quite mature yet. This is definitely not a mainstream thing but I think it's a really interesting direction. I think there's room for a Kubernetes native solution that's shaped like this. One property I think is really important to keep centered as we talk about this problem is offering a low barrier to entry for service providers. A tool for managing infrastructure is only as good as the amount of infrastructure it can manage. If you can't manage the things that you need, then it's not going to be useful to you. And it's best if providers themselves can maintain the things that are going to be used here. So like we have with operators, if there's a low barrier to entry and you can create one easily, that's a huge help. One property that helps with that is flexibility, making a solution that lots of different services and lots of different providers can fit themselves into and isn't too opinionated about how they work under the covers. I think that'll be a really key thing to keep in mind. And finally, I hope that we can reflect on the past as we build future solutions for this problem. And that's really why I'm giving this talk. I think I'd love for us to revisit some of the things that Service Catalog did well, like discoverability in that user experience and think more generally about the user experience and the ergonomics of these tools, make them more usable, make them easier for people to get started with, and revisit things like service bindings and consistent ways to work across multiple providers. I don't think that anything in this talk gives us a full future direction or that looking at the past can teach us exactly what we want to do. But I do hope that this renews that conversation and that we can look at what we've lost with Service Catalog, look at what we've gained with things like operators and crossplane, and maybe build something that solves this problem more completely going forward. That's all I have for today. I would love to talk to people more about this problem. Please come and find me during the conference. I'm gonna be here all week. You can find me on either the CNCF Slack or the Kubernetes Slack, I'm AWG there. You can also send me an email, it's on the slide. And the QR code there is for you to give feedback on this talk. It's useful for me, I think it's useful for CNCF. So please feel free to do that. And I think we're out of time for questions, but I'm happy to chat with people separately. So come and find me. Thanks.