 All right we're recording now. Hi everyone and welcome to our cross-plane community day panel where we're going to talk about bringing the cloud to Kubernetes. So today we've got a panel assembled of folks who are experts in taking cloud APIs and modeling them as Kubernetes custom resources. I'm not sure what we'll talk about later. I'm not sure if there's a consistent name for what that problem or the software that does that is. That's an interesting thing in and of itself. So first I just want to introduce everyone that we have on the call here. We have Joyce Ma from the Google Config Connector team. We've got Matt Christopher from the Azure Service Operator team. Jay Pipes, not Pepez as he tried to trick me that it was before, from the ACK from the Amazon Controllers for Kubernetes team and my co-worker colleague from UpBound, Dan Mankham, a maintainer of the cross-plane project. So today we're going to sort of, I guess as I said before, think about how we could build a consistent KREM experience on top of the cloud APIs. But first I want to ask Matt, what should we call these things? Yeah, control is operators. Yeah, at least internally in my team, we talk about this as a cloud operator or an Azure operator for the cloud, something like that. We tend to prefer the operator nomenclature as opposed to something else like a plugin or add-on or something. But I think you're right that I haven't heard of industry accepted term for what this thing is. So Jay, you've used the controllers parlance but it used to be the AWS service operator, right? Was there any real thought in changing to controllers or was it just a nice acronym? Well, A, I'm terrible at naming things just generally. I'm like super flat and explicit about everything. So yeah, these are a bunch of controllers that do AWS stuff on Kubernetes. So it was AWS controllers for Kubernetes. Yeah, I actually like config connector as like the brand or like, because that's what it is, right? These are connector pieces between the Kubernetes API and resource model and the cloud provider's APIs. So it's like connecting the configuration that's stored in Kubernetes with the cloud. So I kind of like that. But yeah, operator was... These are more than just an operator, right? Like the connection of a CRD and a custom controller. They are specific types of controllers that provide that sort of integration bridge, right, between Kubernetes land and the cloud provider land. So I like the connector part of things. Yeah, me too, which brings me to... I think I'm a little hazy on the timeline, but I think config connector was one of the first projects out there doing this kind of thing. Joyce, do you have any background on what inspired the GCP team, the Google Cloud team, to start config connector? What was the background behind that project being bootstrapped? Yeah, exactly. So basically we've been heard from customers for years that they want to move to cloud and meanwhile they want a consistent environment like on-prem and in-cloud and they also prefer consistent user experience across APIs. So in that sense, imperative APIs is naturally limited providing this consistent user experience. And we started looking at this declarative world. And at that time, it was back in 2018, there were already a deployment manager in GCP that supported declarative configuration of the APIs. And also there were like Terraform Google provider. Those are good toolings, but both of them has its own shortcomings. And meanwhile, we see that there are already customers moving to the Kubernetes platform. So we think it's better to provide a more consistent user experience so they can manage their applications not only for cloud, but also for other native Kubernetes resources. So we decided that we want to build something on top of Kubernetes to manage GCP. Nice. Is that similar for you, Matt or Jay? Is that sort of the same kind of story? Yeah. I mean, very similar for us at Azure. We wanted a Kubernetes first approach, right? Even though there are other ways to do declarative APIs, if you're in the Kubernetes world, it feels kind of awkward to have to step out of that in order to interact with the cloud, especially because you're going to be deploying your core service in Kubernetes. And then your core services dependencies like a DB or something, you all of a sudden you can't deploy through Kubernetes. It's kind of awkward. So that was one of the big drivers for us at least. Yeah, it's exactly the same for us. I mean, we have had cloud formation, right, for well, what seems like forever, right? And there's also Terraform for another sort of declarative way of driving infrastructure changes. But yeah, just like Matt said, we had customers coming to us saying, hey, we prefer the Kubernetes API. We prefer the consistent Kubernetes resource model and API machinery. And especially our devs prefer that, right? So what we've struggled with really is that most of our customers, what we found, they have a lot of application development teams that prefer to use the Kubernetes model. But a lot of these customers are huge enterprises, right? And they still have these like central IT teams. And a lot of those central IT teams also like to work with cloud formation or they may use Terraform or other solutions. So it's sort of a mixed bag. And ACK, the AWS controllers for Kubernetes is primarily focused on that first set, right? The application developers that prefer to use the Kubernetes language. And we've had, yeah, some interesting discussions like how do you work in this hybrid environment where you've got a central IT team that's used to either imperative configuration changes or using cloud formation and things like that versus the app dev teams that are using and preferring the Kubernetes model. Right, right. So Dan, do you, do you, I was hesitant to, not hesitant really, but every time I asked someone from AWS, hey, why did this project start? I get the same answers because the customers wanted it sort of thing. And it's interesting. It was good to hear that. I didn't do that on purpose. I know, I know. It's kind of the best answer to the question though, to be honest, you know, it's much better than like we thought it'd be cool or whatever, which, which brings me to Dan, do you, do you know the sort of, have you done any archeological spelunking to, to find out some of the history of the cross-plane project? Do you feel like you could answer the question of like, where did cross-plane come from? Yeah, I'm, I know at least from, I was aware of the project when it was initially announced and, but wasn't yet with the company. So part of joining upbound and working full time on cross-plane was kind of like buying into the mission, I guess. So I can speak to that at least. And I know something that really motivated me to work on the project was this idea of kind of like a unified control plane, right, or a common way to address cloud providers. And it's grown to really more than just cloud providers, right, to address APIs. Kubernetes happening to be the vehicle for facilitating that. So I think, you know, a lot of the things are similar to the things that other folks on this panel have already said, but with the added layer of kind of like a common interface. And there were, at the beginning, there were some pieces of kind of the composability story, right, piecing these different things together and building higher level abstractions. But most of that has developed over time since the initial impetus. Yeah, my understanding is really mentioned the service broker stuff yet. As far as like componentization or whatever, composition, things like that. I don't know. I guess if I'm putting my archaeologist hat on, right, cross-plane and a number of other things started out sort of in the same time frame that the open service broker concept and community kind of started to become a thing. And I know that the service brokers kind of fallen out of favor and out of style, but I think it's important to bring up that topic as far as like our archaeological dig here. That's very true. I've got to admit I'm somewhat personally ignorant about the history of the sort of open service broker project initiative. I will say, for those of you who are recording this week ahead of time, but for those of you who are watching this conference now, I believe there will be a talk that relates cross-plane to the open service broker from Yarn at Accenture, I believe, if I recall correctly. So that'll be a cool one to keep an eye out for. But yes, that's a really good one to call out that they were definitely sort of forging the path with regards to sort of thinking about deploying all these apps to Kubernetes. Now, what do you do if you want infrastructure that doesn't run statefully on your Kubernetes clusters? So just to provide a little bit more color and take this with a big grain of salt, because Dan and I were, I think, maybe two of the first people working on cross-plane who were founders of the project. So there was a period before it was open sourced when it was sort of an experimental thing that folks like Luke and Ilya and Jared and Versailles were working on. And I believe that it actually was an accidental thing where they were trying to go build something else and then they found it really useful to apply something a little bit like the persisted volume claim pattern to just databases and buckets of things like that. And then they built that to support the other thing that they were building and like, oh, hey, this is useful. We should open source this and get this out there sort of thing. I believe that's where Crossplane came from. So I also just going back to Dan and I don't want to talk about crossplane too much, but I do want to ask Dan, can you give a little context, Dan? Crossplane kind of approaches things a little bit differently than the other clouds, right? Or than the clouds, I should say. So the other projects represented all this call focus on one cloud sort of thing and delivering all the APIs of that cloud, whereas crossplane scope is a little broader, right? Yeah, that's exactly right. And there is a lot of crossover in terms of representing resources as Kubernetes objects. That being said, if you want to build a layer on top of that that allows you to interchange these resources as common building blocks, you have to have some interface, right, that all of them satisfy, right? So just like if you were doing object-oriented programming or something like that, if you want to be able to change things out, like for like, you need to be able to address them in the same way. So crossplane has a common runtime as well as some common embedded types that it puts into all of its objects. So I know Jay actually mentioned earlier today that it's kind of like the ACK types have kind of the top-level spec of a resource, you know, the fields that are actually going directly to AWS. Whereas with crossplane, we kind of have a container for those sorts of fields, and then the common fields on top of that. So just as an example of some things that would be common between resources, the way they reference each other, the way they produce connection secrets, the way they report, whether they're ready or healthy, those are all things that we need to be consistent with so that we can, you know, compose them into a higher-level abstraction and aggregate up some of those components. Yeah, I think another place where sort of crossplane approach gets interesting goes back to what we were talking about before with regards to potentially integrating with like legacy systems and things like that, because we opened up the, you know, there's only so much that's implemented so far, but, you know, we just released like a vSphere provider, for example. So you can mix and match your, you know, cloud of choice with maybe, and, you know, maybe your cloud of choice is vSphere, I don't want to paint vSphere, it's purely legacy stuff. But, you know, if you do have a different environment, sort of thing, let's say if you're moving from Azure to AWS or vice versa, you can kind of do both consistently or theoretically if you were using we don't have a ton of old school bare metal stuff. We have like an equinex provider, but I don't really think they're very old school. So, but if you were on bare metal, you could hypothetically have a crossplane provider that you can sort of mix and match there. Or you could imagine, like having a, you know, a provider that just goes and runs Terraform or CloudFormation for you if you wanted to. So this kind of gets back a little bit. I think one thing that crossplane has really focused on is thinking about the separation of concerns between sort of like platform teams doing part of the job and then sort of the consumers of sort of platform teams or the customers of platform teams doing another part of the job. Is that a dynamic choice that you see for conflict connector users? Who are the main people who you see using conflict connector? Is it sort of mostly SREs or mostly sort of SREs sort of just offer conflict connector as a service so that the average application developer, the person who writes deployment might also write a cloud SQL instance or something like that? Yeah. So we identify our customer, our target customers are like platform teams and platform administrators who are like designing, building and setting up their centralized infrastructure. And I think those are our main customers and they might create a general infrastructure for the entire organization or they might create infrastructure on demand for the application teams. So basically we consider those are our target customers. On the other hand, application teams can definitely use us but I guess it really depends on like what they, how they like design their infrastructure or how they design their own workflow but from our perspective like platform and means are our target customers. Makes sense. And I think I've seen, I know that Google has some examples on GitHub of building abstractions on top of cloud, sorry on top of conflict connector types, but Google typically seems to prefer to do those abstractions client side, right? I think I've seen like this tools like KBT, the way there's some examples of how you can have some use KBT to render out some tools. Is that sort of something that maybe would be more focused to application developers rather than platform teams? Yeah. So I think like CAPT is part of, is definitely like a new thing that is used to integrate with other toolings on the like Kubernetes ecosystem. The thing about like conflict connector is it's used to mainly manage the like manage the control plan of the resources. And for a lot of the application use cases, they might be focused on more on data plan. And so that's why we consider like managing the infrastructure is the main use case. For example, we have examples of creating resource hierarchy, which is basically create the organization, create the folders, and create different folders for different teams, set up I am permissions and set up networking connections. So those are actually something we find the conflict connector will be really useful, because it's providing the consistent template consistent to user experience and similar infrastructure can be set up for different teams with several configurations by variable supported by CAPT. So also, so the entire workflow can be simplified and like, yeah. So I want to sort of ask a simple and very into this question to Matt, because one whenever I think about managing what we in Crossland called high fidelity, or maybe a more common way to put it would be sort of the granular custom resources that map to a cloud API. A lot of folks come to this problem thinking, oh, I want a database and there's just going to be an API that is my database. And in some cases, looking at UJ, the API for an S3 bucket is like 25 different API calls, because it is a branch. 19 separate API calls to update a bucket attribute. Anyway, go ahead. But so Matt, I always use it as an example here because Azure does something that I think is really good where when you create, as far as I'm aware, when you create postgres SQL server, for example, it will usually, it will not allow any traffic and then you have to go create some firewalls to either allow virtual debt or just an ID range to access it, which is, you know, secure by default, which is good. But then it changes the problem from just going to give me a database to give me a database and configure some network security and you know, potentially make a virtual network or something like that if you need to. So is this also something that the ASO uses are sort of pretty comfortable doing their sort of people who are infrastructure types who are using this? Yeah, so for the most part, we, you know, people who are using ASO are, as you say, exactly, you know, they understand what resources are being deployed and they sort of are like, if not platform administrators, at least they're bridging the gap between sort of a platform. They're like the platform admin for their team, right? So we have some customers where like they've set up ASO and they're using, you know, Kubernetes RBAC and stuff such that they're delegating platform administration to an individual team within their company and each team sort of does their own platform admin and then the ASO is maintained by the core team, but they don't concern themselves with what a particular team in their company is deploying. They just give them, you know, sort of carte blanche to deploy whatever they like, as long as it's, you know, in their namespace. And so yeah, the general case is that customers sort of understand how to configure these things. And one of the things that they like about sort of the operator pattern and ASO is, hey, like, I need to create like six of these things or whatever, as you say, like the Postgres server, a DB, some firewall rules, etc. And it's really nice in Kubernetes to be able to do that by just like applying, you know, six, six resources all at the same time. Like you don't have to worry about ordering. You don't have to worry about like, does this come before that? It's sort of just make it so. And then you wait a little bit, and then it's all there. And so that's one of like the big selling points, I think, of not just ASO, but sort of the general pattern that Kubernetes is espousing is like, hey, you don't have to concern yourself with, you know, Jay mentioned, there's these 19 APIs, but presumably like, you don't have to think about that when you're actually interacting at sort of the CRD level, which is a big win. Exactly. Change the spec and let the controller deal with all the ugliness behind the scenes, right? And which S3 update attribute API call it's going to make, you know, hide all of that implementation, that imperative implementation goop in the controller and allow the developer experience to just be one of just tell me your desired state. That's it. So Jay, speak, I'm not 100% sure about this, but I was reading a few weeks back now on the on the ACK GitHub repo, and was doing a bit of research myself to answer this, like, who is the target audience of ACK, and was I correct in thinking that ACK also has a bit of a bent toward letting people build higher order things on top of ACK stuff? Like, is there a thought there of like, hey, someone might want to build their own controller and pull in some ACK controllers, or did I today sort of not quite get that one? You mute it, Jay. Jay's experiencing technical difficulties. So let me while he figures that out, let me let me roll on back to back to Dan. So now sort of it's interesting to hear that sort of from from ASO and conflict connectors perspective, we're mostly seeing platform teams or SRE folks being the people sort of doing the end to end, spinning up the infrastructure sort of thing. Dan, Crossblade sort of has this approach of separation of concerns. Do you want to give me a bit of context around what Crossblade does there? Yeah, absolutely. And, and honestly, it's very compatible with what both Matt and Joyce have said so far in that the actual granular, as we call them, managed resources are things that we don't expect, you know, non-infrastructure admins to interact with, because they're generally pretty complex, right? And understanding how they work together requires a fair amount of background knowledge. And that really informs the composition model, which I was alluding to earlier. Another thing that is kind of a maybe the most controversial topic, I'll say, between Crossblade and some other projects is the cluster scoping of those managed resources, right? So all of these managed resources, the granular ones exist at the cluster scope, while most other projects, whether they're, you know, managing infrastructure or doing other are at the namespace scope, right, for isolation and RBAC privileges and that sort of thing. The reason why Crossblade takes that approach is because the composition model allows you to define that abstraction and expose it at the namespace. So then you're able to put the permission, permissioning, you're able to raise it to the level of abstraction, right? So you're able to say developers interact with this, you know, friendly interface or this friendly resource that has very simple fields that map to things that we want and enforce policy we want. And we can govern their ability to do that at the namespace scope. But then it renders out those resources at the cluster scope. And that's where the infrastructure admins do those kind of operations with, which config connector and ASO, you know, are providing them the ability to do as well and actually control those granular specific resources. Thanks, Dan. So we've still lost Jay. I'm just going to, oh, here he comes. Welcome back, Jay. You didn't miss too much. Zoom crashed. Sorry. All right. Well, let's keep rolling on now to some of the sort of maybe more interesting to those of us on the call, at least technical stuff. Matt, what would you say is sort of the biggest challenge of building out something like ASO, either technically or logistically organizationally? Right. I mean, I would say that these two things sort of go hand in hand. It's consistency and scaling to the number of resources that there are in the cloud, right? And the more you try to scale sort of manually or by throwing more people at it or whatever, the harder it becomes to sort of maintain that consistency, as I'm sure everybody on the call is familiar with. And so it's this constant tension between like, how fast can we go? How many things can we support? But then are we giving a uniform experience? Are we giving, I think Dan mentioned earlier, like it's the same, I want to be able to look at the right properties and see state. I want to be able to do references the same and all that. And so I would say like, that's one of the biggest challenges is that constant tension between these two things. And of course, the team that we have working on ASO is just it's such a small fraction compared to the team in Azure that's building new things and shipping new APIs, that if you look at it from sort of a person hours perspective, there's just no way that like five or six of us could ever keep up with however many, you know, like thousands of developers, there are producing new APIs in Azure. And so we've got to come up with some way to do something about that disconnect. And that's one of the big challenges. Joyce, I know I've spoken to people in the Config Connector team and heard similar things. Do you agree that that's sort of the biggest challenge or is there something else that you'll find trickier? Yeah, I completely agree. Like consistency is always a big problem or big challenge, I would say, as it comes together with scaling, because the more resource you support, the more edge cases you were running to, and the more features you need to support to cover those edge cases. So it's really hard. The lucky part for us is we are built on top of declarative libraries, which they communicate with service teams directly. And we work closely with the declarative library team to make sure like they feature or the resources they support aligns with our customers requests. And then they can probably delegate their own request to the service team. So it's like there's a layer in between us and the APIs. So Dan, what's what's this like as a maintainer of crossplane who is, you know, we have almost the same problem, but another layer up I think, right? Yeah, that's exactly right. I was going to just respond with hard and next question. But it's definitely very challenging. All the things that have already been mentioned are applicable for us as well with the added part of we we are not at these companies, right? So some some internal things we don't have access to, of course. And, you know, we also well no one can right now but theoretically in a in a future time when we're not in a pandemic, be able to walk down the hall and knock on someone's door and ask them how an API works. That being said, we definitely had awesome partnership with the folks on this call right here, and being able to share some of the generation of kind of the manual steps for creating these resources. So specifically with AWS, ACK and provider AWS for crossplane currently share a code generation pipeline, which definitely helps accelerate both projects as you know, they have the context for, you know, understanding how these API is working intimate level, and we're able to contribute back based on our experiences as well. So that's definitely a factor. And then one thing we haven't touched on, which I definitely see as a as a open source maintainer as a big component is community, right? When you get more folks working on things, you know, whatever their background, it's a huge benefit for for driving progress. So especially since crossplane has joined CNCF and had its one dot a release and that sort of thing, we've seen a lot of folks just from, you know, all of the world, all different backgrounds, all different companies come together and say, you know, I'd like to learn how to do this because it it scratches an itch that I have. And having that kind of open community definitely supports that and and you know, hopefully those folks that come along and contribute can also grow in the community and take on larger leadership roles and take something away from contributing as well. Yeah, that's that's kind of one of our big approaches to scaling is scaling the community to scale with the amount of work of building building this stuff out. The whole consistency thing is really interesting to me. I think, you know, we talked about why people like to use Kubernetes to manage their infrastructure. And I think part of sort of what we touched on there was roughly just tooling consistency, they're probably already using Kubernetes to do other stuff to manage their applications. But I think part of it is because people like the Kubernetes resource model, the KRM, and I think people like it because it's a very consistent predictable API. And I think a lot of what we've all sort of seen, you know, cross-plane, we call it the XRM or the cross-plane resource model, where we're just, you know, a super set of that, we're just saying, Hey, what if we make this more consistent and more opinionated than people like, Oh, we like that. But then there's also an interesting thing where hypothetically you've now got, you know, when we're when we're when someone comes to write a new cross-plane provider, there's different layers at which you could ask them to be consistent. You could just say, Hey, you can go write this provider in Erlang for all we care, just have a consistent API. And there are trade-offs there, right, that you need a big enough community for that provider where everyone's going to be coming in learning Erlang potentially or like using a different tool set, building new libraries, all that kind of thing, but sure. And especially if that's open source community, we might want to have that flexibility. Or you can also, you know, take the approach that we've come closer to taking is like, we actually want you to build providers mostly in a specific way, using go, using these libraries, etc. etc. I'm guessing sort of, I know Joyce, you said that you've got sort of DCL, which, which I believe is like a go library ride. Jay at Amazon is everyone, do you think you're going to get everyone sort of pretty consistently building control is just using the ACK style sort of thing, do you ever have issues with people saying, No, I really want to go do this with Java or something like that? No, we don't really have any issues with that. I think it's our, our biggest challenges are primarily people based, right? I mean, ACK is a set of service controllers, right? The one for each of the service API, so one for S3, RDS, etc. And each of those services is backed by people at AWS, right? And those service teams, some have a very long history of doing things their own way. Some have a very short history and are like very comfortable working in the open source community. Some aren't. So yeah, it's been a challenge bridging across those different service teams for me personally. I mean, it's a challenge I welcome, but it's definitely been a challenge, right? Because I have to sort of explain how Kubernetes works in addition to, you know, like, hey, what are our needs from an ACK perspective? We generate the controller, right, for the, for the different resources in that service API. And then we work with the service team to build tests and that kind of thing. So yeah, the variety of knowledge about Kubernetes and container ecosystems within the service teams at AWS has been a challenge. I'll just put it that way. Yeah. So we're pretty much at times, but just to wrap us up, I want to, you know, we've been talking about technical challenges and things like that. I want to just do a quick round the room and I'm going to ask everyone just to give me a, you know, one or two sentence answer of what are you most excited about on your project at the moment or like what's most interesting to you if it's not a feature, a technical challenge or something. So start with Joyce. What's, what's excited you the most about Confit Connector at the moment or what you're working on? Oh yeah. So I think the most exciting part is like customer can actually use Confit Connector to manage Google, manage GCP resources, like native Kubernetes resources. It's like, like I see, I've seen a lot of, like customer calls about that. And it's also amazing that we can keep full fidelity of those unaligned GCP resources. So those are something I'm really excited about because that's exactly the goal of Confit Connector. And I'm so excited to see customer actually feel it. Yeah. Yeah. It's really exciting to see this sort of vision that we've all been working on, like becoming real and being used in production environments. How about you, Matt, similar answer or do you have a different take? Well, one of the things that we're sort of incubating in ASO right now is Azure has ARM and it has specs for all of the management plane operations are in OpenAPI. And so one of the things that we're working on is, you know, I mentioned the scaling challenges and the consistency challenges. And we're sort of pretty close to an alpha now where we're planning on generating, as sort of Jay mentioned, AWS is doing as well, the controllers for all of these management plane resources from these OpenAPI specs. And so I'm really excited to sort of hopefully be able to knock down one block on that challenge of, hey, consistency, scalability, we've got this thing that we can generate using OpenAPI. And actually, you know, like 90 or 80% of the Azure resources will just be able to code generate. And then, of course, we still have to solve testing and all of these other problems. But at least that's one big step toward a consistent experience across a large number of resources that doesn't involve tons and tons of manual effort and, you know, people massaging each and every one. We're sort of lucky in Azure because there's that uniform way to deploy through ARM and all the specs are there. So it's just a question of writing the code to turn it into Kubernetes resources. Nice. And I want to give a shout out one of the things that I'm most excited about in the novel space in general is Matt and his colleagues over at New Zealand put together a little proposal on how to do versioning of Kubernetes resources relative to cloud resources, which is really well thought out, really, really great read. I recommend it if you're a note about this stuff like me. Jay, how about you? Well, I think there's two things I'm most most excited about. From a user perspective of either crossplane users or users of ACK individual service controllers, I'm really excited about bringing a consistent way of interacting with some AWS service resources using the Kubernetes API machinery and Kubernetes resource model. I think that that consistency has been asked for for a long time by application developers who prefer Kubernetes. So I'm super excited about that and just the consistency in general. The other thing I'm really excited about is actually using ACK as a mechanism to bring the world of open source development to internal AWS service team engineers. Not only the Kubernetes ecosystem and sort of like how that works and CNCF projects like crossplane, but just generally getting some service team engineers at AWS out of their shell and using sort of like more upstream open development tooling and build tool chains and just getting them comfortable with contributing to open source projects. Like personally, very important to me. So yeah, that's what makes jazz is me up, gets me in the office each day. Yeah, I can totally empathize with that. I'm not there in any way bad at open source, but I spent my first five years, six years of my career in Google and there was a bit of a bit of a culture shock to leave Google and be using the open source equivalent of all of this internal tooling sort of thing. So I totally get exposing the average engineer to like more of how stuff's done in the outside world is always fun. Dan, and last but not least, what are you most excited about with crossplane? So this is I guess a bit of a deviation from the other answers. But one of the parts of crossplane that doesn't get focused on too much, but I'm particularly kind of partial to is the package manager, which kind of has some opinionated workflows for how you can install and manage and version CRDs and update packages and roll them back and forth. And in a proposed kind of extension to this in the future is the ability to support essentially functions. So right now we have providers which are basically controllers and CRDs. We have configurations which are just YAML objects that get installed into the cluster and then functions could really lower the barrier to entry for folks authoring some of this kind of specific functionality. So you could imagine day two operations or disaster recovery or migration. Those could all be things that could be packaged into small functions. And there's a lot of innovation happening in the function space and the Kubernetes ecosystem and also more and more folks are using to moving to things like Lambda and things like that. So providing that interface to interact with this kind of ecosystem that we've built of cloud provider resources as Kubernetes objects is something that I think could be a really powerful model. Yeah, super excited about that too. All right. Thanks everyone. I think we're pretty much at time. So I'm going to stop the recording now. Thank you all again.