 So I suggest we start. Welcome everyone to our research user group meeting. I don't see any specific items in the agenda apart from our topic. So today we'll hear about the cross-plane project and we have Nick from UpBound to introduce the project and answer our questions. So I won't hold any longer. Thanks a lot again, Nick, for doing this and yeah, it's a flow. Thank you. All right. Yeah, it's breaking a bit, at least for me, the microphone, but I think if you speak, it should be good, but can you give it a go? Sounding a little bit better to me now. Can you hear me all right? Yeah, that's better. It was noise-canceling before. Every time you speak, it would cut yourself out, so it was pretty awesome. Looks good, no? Yeah, that's good. Give me just a second to try and drag my windows around so that I can share the screen without you seeing all of my speaker notes in my own thing. Okay, wait a minute. Wait, what? It's breaking up again, Nick. But the sharing is good. It's just the sound is breaking up again. Yeah, there's probably many of the media that are doing different browsers and things like that. It might be that if you keep talking, it will calm down if it's just noise-cancelation. That's good to go, because at some point it was very good. All right. It's not working. In the meantime, I see Antonio joined and I see him for the first time here. Maybe Antonio, he doesn't seem to have a microphone though, but in any case, if you want to say hi or paste in the chat, I don't know. Yeah, okay. In the meantime, we can wait for Nick to join. It's weird. It sounded like he would speak and then whatever noise-cancelation thing this is doing would go, oh wait, I've heard this before. I should cut it out. And so every other second, it was noise-canceling itself, which was pretty awesome. Yeah, let's see. We'll join again. We need to see a DJ set that actually takes the use of that and can make some cool sounds. Zoom magicians, I've seen. I have not seen a Zoom DJ yet, but never say never. Well, I keep saying that we need Jamie and his jazz jamming skills to entertain us for these breaks. The production's like improv. And then there you go. Go ahead. That's your moment. That's it. It's your moment. He's been delaying his event, but he will have to do it. It'll be exciting. All right, I'm trying to catch Nick also on the slide to see if he needs help. Is anyone using crossplane while we wait? We can already start. Let's give it to the player. He's back. Okay, let's give this another go. That's much better. I'd make a UDP joke right now, but y'all might not get it. Yeah, I recently switched from Chrome to Safari, and I like Safari a lot more, but I'm finding that a lot of the internet is built for Chrome rather than Safari. All right, let me get my presentation going again. I have to quit and reopen my browser to have security permission to share a window. Yeah, so I was about to add like we do, we've been using crossplane, but only for integrating with public clouds. And we've tried the providers for pretty much all the major providers. And we use it actually to deploy Kubernetes clusters. So we manage the clusters at the different clouds from within a non-premises Kubernetes cluster, and we just orchestrate the remote ones using crossplane. And it works pretty well. I'm excited about hearing about it. This browser shared just a window with you instead of my whole display. All right, we are now good to go. Thanks for bearing with me, folks. All right, so my name is Nick Cope. I am a steering committee member on the crossplane project. I've been working on the project for about three years. I actually started working on it right after the founders of the project, the same people who founded the Rook CNCF graduate project, shipped the 0.1 release, which was very much a tech demo, and then continued to be a tech demo for some time through to our 1.0 release, which I want to say was late 2020. All right, so what is crossplane? We think of crossplane as being a framework for building control planes or API-centric control planes specifically. So what does that mean? Think of crossplane as an interface to your platform. So your platform could be a lot of things to a lot of different people, which can mean both what is your platform for? Is it for, is it a platform for application developers to quick-wish apps, sort of a PAS or internal PAS type of system? Is it a machine learning platform? Is it sort of a mini internal cloud platform? And it can also mean what is it made of? What is it powered by? So, you know, for a lot of folks, it is a cloud provider like AWS or Azure. It could be on-prem hardware, could be other sort of supporting services like GitHub, ServiceNow, et cetera. So many of these things expose APIs, but they're kind of one-size-fits-all. So Amazon, for example, if they're designing an API, like the one you can see here for RDS, folks request and manage database instances, they, inherently, you know, they have lots and lots of customers and they want to design an API that has all the features that all those customers want and that suits everyone. So it's very powerful, but it's kind of one-size-fits-all. And in our experience, that could be somewhat overwhelming, especially if you are a platform team or an SRE team and an organization building a platform for your application developers, they may not want to think about all these fields. It might be sort of a way too much cognitive burden, way too much for them to learn, but they just want to focus on what they want to do, which is, you know, probably have somewhere to store their data, in this case, or they build and run their app. And you may not want them to have access to all those things. You know, there might be fields that you want to set as a policy, as it were, sort of thing, to control. So that's really what Crossplane does. It takes anything with an API, brings it into a control plane, which is built on the Kubernetes control plane, declaratively. But then what really, I think, makes Crossplane special is that it lets you define your own APIs on top of those, your own sort of API abstraction, so you can present things to your customers however you like. And again, when I talk about you and your customers or I'll talk about the persona of platform curator and platform consumer, I'm talking about usually you as a platform team member, or SRE, someone who's supporting application developers and then your sort of consumers and customers are application developers. So this is an example of basically what you saw on the previous slide. So in this previous slide, you're seeing an example of YAML. Everyone disses YAML in the Kubernetes and it's kind of cool thing to do these days, but you can just think of it as a REST API really. Behind the scenes, what it's doing is taking this YAML document, confirm it, transforming it into a REST call effectively. And I think REST APIs are pretty powerful, generic interface. So this next slide here is an example of what you just saw. As it regards to cross-plane concepts behind the scenes. So what cross-plane is actually doing here is taking what we would call a claim, this Postgres instance, turning it into an intermediary object called a composite resource, which in turn gets rented out into a couple of specific cloud resources like RDS and in this case a database parameter group. This is a little bit of a more detailed view of the same thing that illustrates some of those concepts a little bit more. So again, you've got the claim and the composite resource and manage resources. So again, remember that the claim would be the Postgres instance in the previous slide. The manage resources would be RDS database subnet group in this example. So some of the new things that you see introduced here are more sort of platform team or platform curator concerns. You have the composite resource definition or XRD and you have the composition. These all sort of configure what your platform can do. So composite resource definition teaches cross-plane that a particular type of composite resource now exists. In this case the X Postgres rule instance. It also teaches it that a new type of claim exists because those two objects are tightly coupled and two types go hand in hand. Cross-plane then needs to know what to do if someone creates one of those claims. If someone says, hey, I'm like a Postgres instance. It has to know, oh, that means I should go create an RDS instance or I should create a DV subnet group or maybe it should be using a different cloud. It should be using a cloud SQL instance in Google or maybe it should be using some sort of on-prem system with an API investment of a database. So that's where compositions come in. Compositions are effectively a sort of a template or policy saying, okay, pick this one and it'll tell you how to fill in the blanks. So what I mean by fill in the blanks is going back to this example you can see here that the only field that we have on the Postgres instance here is storage. You should ignore the fact that storage is actually 20 gigs over here. I realized that after I took the screenshot pretend that there was 100 gigs. But all these other fields have to come from somewhere and if they're not sort of specified here in the Postgres instance, that has to be a different resource. And you can almost think of the Postgres instance as configuration fields then being overlaid on top of the RDS instance. So that's all done with a composition. And one of the interesting things is that there is a one-to-many composite resource type to composition relationship. So that or another way of saying that is pretty particular type of composite resource for this Postgres instance. You could have many compositions. You could have the Azure one and the AWS one. You could have the big one and the small one. You could have the East Coast and the West Coast one or something like that. And that can take many fields or just fundamentally what the thing is made up of whether it's RDS or Cloud SQL or something else and effectively make that a single knob to your people consuming your platform to the application developers. At that point they're really just matching on labels similar to a Kubernetes pilot matching a node. So they could say, give me the West Coast composition please or they could add, they could say, give me the West Coast production composition please give me the West Coast production AWS composition please and by specifying labels they will get the right one. And as you can see, I think this dotted line in the middle sort of shows that everything above the line is what we think of as a consumer concern an app team concern. Everything below the line is more of the person who's sort of configuring the platform, probably the platform team. I use a platform and control plane somewhat to mean the same thing. I would typically think of your platform as having sort of one or more control planes as the interface. So this is sort of a quick slide on what crossplane is made up of. So what are the moving parts? We have core, which is if you go to github.com slash crossplane slash crossplane and sort of follow the instructions to install crossplane itself, that is what you'll end up with. And really crossplane core is two things. It is the parts of the code base that power composition. So that's this stuff, that's the feature that allows you to have composite resources and claims. And it's a package manager who lets you install extensions. So funnily enough, if you just install crossplane core and then stop there, it doesn't actually do anything. It really is kind of just a framework that you then have to go configure to do anything useful. So the package manager lets you install crossplane extensions. We use a package format called X package, which is just a simple OCI-based packaging format. And today, the things that you can extend crossplane with are providers and configurations. Providers teach crossplane about managed resources. So that teaches crossplane about these, managed resources down the bottom, which again are things like RDS, DV Summit Group, Cloud Signal instances. Providers tend to be focused or packaged based on a product or a cloud. So we have an AWS provider, we have a Google provider, we have a SQL provider for actually speaking SQL as an API to create databases, users, things like that. We have a Terraform provider, which is a little bit of an Inception thing, which is a provider for crossplane that goes and runs Terraform configurations. We have a Domino's Pizza provider that can go in one of our product managers that are bound made to order pizza. Is it something that you always have to do if you're making an infrastructure tool apparently? And then, so that's providers. We also have configurations. Configurations extend crossplane with support for new composite resources. So that is composite resources and claims. Configurations will actually typically drop in composite resource definitions. XRD. I think we lost him. No. He was sounding so good. He was brilliant. And I'm out. He's back. I might actually need to drop off somebody's requesting my time so I might see me wander away. Sorry. Thanks for understanding. Yeah. Thank you, Spike. Yeah. Glad to have you back, Nick. It was looking good. Thanks. For what it's worth, this is... This is not a common problem. This is very... I'm just looking forward. If you start doing a live demo, like, the gods are going to have a video today. No, no. All right. So where was I? I was just pointing out that providers add support for new managed resources. Configurations add support for new composite resources. So you can actually just define a composite resource directly by just authoring an XRD and applying it to your cluster and authoring compositions and applying it to your cluster. So you can sort of share the same configuration that, you know, let's say you define your Postgres instance type and then you actually have 10 control plans you want to go and ship that type to. Then packaging them up as a configuration can be a handy thing to do, to just distribute them out to everything that you can version that. Configurations can have dependencies on providers. So you can say this configuration needs the AWS provider, the Google provider, a cross-platable automatically install kind of just a convenience thing for distributing cross-plane configuration. Some of the stuff that we have in the works that we think of as cross-plane extensions that will likely be packaged up as well soon are secret stores. So that is a way to teach cross-plane when I create something like the Postgres instance in the previous example. I might need to push connection details somewhere. I might need to say this is the address that it's running at. This is a username and this is a password. We've always supported Kubernetes just writing that to a secret. We now have Alpha support for pushing to vault as well and we're working on a sort of plug-in base model so you'll be able to teach cross-plane about more secret stores there. XRM functions is something that I think is pretty cool, which is one thing that we have heard and that we're always conscious of with cross-plane is that the language used to configure composition is intentionally limited. So it's effectively a template for a bunch of resources to go create. It does not support conditionals. It does not support iteration. There's no such thing as if I put I can't put number of databases 23 on my XR and have cross-plane go and be like oh great, I have to go create 23 instances of this database that we effectively did not want to build a sort of programming language as a REST API. So what we're working on at the moment is the ability to mix and match the current version of composition with effectively a pipeline of simple functions where you can basically say cross-plane when you see a composite resource that looks like this execute this pipeline of little OCI images basically each of which is a function that tells cross-plane what to do or call that to a webhook potentially very similar to KBT or customized taking a lot of inspiration from KRM functions if you've used those. Coming soon also webhooks we're adding plugable webhook support to this some of those webhooks probably going to be built in so there should be things like validation of core cross-plane types but we're hoping to allow you to also add your own bespoke webhooks so that you can sort of add your own sort of validation admission control webhooks for your composite resources for example if there should be some extra policy applied or you want some second order effects to happen when one of those is created it's just an example of some of the managed resources that we have in the AWS provider today we have an AWS provider at this point that has I think 700 of them or something like that bit of a taste of our roadmap at the moment there is a roadmap for the github.com slash cross-plane org I mentioned the plugable secret stores that we're working on, custom compositions is another name for those XRM functions I talked about Validations relates to the webhooks also looking into just improving our support for Argo CD a lot of people use cross-plane with Argo CD and there's a few bugs at the moment that make that a less than optimal experience so we're going to work with the Argo CD folks on that the other thing that's in quite a lot of demand is observe-only resources as we call them, which is the idea that maybe you want some stuff to be represented in cross-plane that is, the cross-plane isn't authoritative isn't declarative for so you might want to refer to a piece of infrastructure pull in some data from it and use it somewhere else in a read-only and observe-only fashion I think that's kind of the end of the presentation for the moment but one other thing that I want to point out that's not worked into these slides is that in all of our examples here we we tend to talk about cross-plane as being used for infrastructure constructs like Postgres instances in this case we're increasingly seeing folks use cross-plane for and we just haven't got a lot of demos of this at the moment but people ask us, what app model should I use with cross-plane should I use cross-plane together with or something like that or some other simplified application model especially if I'm installing cross-plane as a add-on to Kubernetes rather than as a standalone control plane and being based on Kubernetes it is compatible with any of those things and then if you like one of those app models you are more than welcome to use it with cross-plane it should work really well but we also think cross-plane is a way to build your own app model because it is an open-source symposium run by some Amazon folks the other week and some of the people on stage we're talking about how it's similarly to the fact that it's really tough for anyone to design the one true Postgres API that just works for every company, every organization every lab it's tough to do that for the one true application model as well and the nice thing about cross-plane sort of composite resource system is that it is pre-open-ended you can use this to define a composite resource that represented an app we went and used our plugins for our providers for Helm or Kubernetes or EC2 instances etc to go and actually spin up workloads somewhere and potentially also spin up the databases and queues and all the things that that needs behind the scenes so that's something that is possible today and we just need to do a bit of work to document that fact and highlight it to folks and give some examples alright anyway I will stop presenting at you now and open the floor to questions and discussion awesome thanks a lot Nick I'm looking here so if anyone has questions feel free to come forward Can you just do a simple comparison with some of the other computing tools in this space like something that would come from HashiCore or from some other large organization how do they compare and contrast yeah sure I think that shooting from the hip the things that we see people comparing cross-plane with the most often our Terraform we also see people comparing cross-plane with some of the cloud provider based sort of declarative Kubernetes configuration tools so that would be Amazon's ACK, Amazon controls, Kubernetes the Azure service operator and Google's config connector Terraform, especially if we talk here just in pure open source world so I'm not considering Terraform cloud and enterprise and things like that while comparing to cross-plane because those are proprietary there's a couple of differences one is more of a community basically cross-plane is part of the CNCF we are an open governance project we do happen we were founded by UpBound which is where I work we do have a lot of UpBound folks working on it but we have a steering committee that is open to people who are not UpBound employees we have maintainers who are not UpBound employees etc etc whereas Terraform is open source but it is not open governance it is a HashiCore driven HashiCore project but putting aside how their communities run I guess the summary there is we think of organizationally we're much more like Kubernetes and all the great CNCF projects than HashiCore but technologically I would say HashiCore is an IAC tool right so it's a command line tool that is one shot cross-plane gets installed onto a Kubernetes control plane where increasingly promoting a model where you can actually just go and install a really small Kubernetes control plane that doesn't actually run your controllers it doesn't run all of your sorry it doesn't run your controllers it doesn't run your pods that is just sort of stand alone as a place to run cross-plane separately from maybe all of your app clusters that you run Kubernetes apps on but you can also install cross-plane just as an add-on to a Kubernetes cluster so the point here is that if you think about all the benefits that you get from Kubernetes you declaratively say hey this pod should be running and if the pod stops running Kubernetes try really hard to get it running again cross-plane does that for everything it manages as well so if you say hey cross-plane I need this Postgres instance and that Postgres instance stops running cross-plane will keep trying to run it Terraform does that as well but it's typically reliant on you have to go and invoke Terraform and you say that put it in a Chrome job, put it in CI CD or some way to get Terraform to so that keep checking in and correcting any drift in your infrastructure another thing is that just slight design differences and this is a sort of a trade-off thing Terraform very thoroughly creates a graph of all of your resources and computes everything and that can end up with some challenges where if you just sort of model all of your infrastructure as one Terraform configuration you might want to say hey Terraform please go update this database for me but then someone's changed your buckets out of band as well and Terraform's IY just have to fix everything so I'm going to change the buckets and I'm going to change your databases as well that happens less with cross-plane just because you really can't change things out of band with cross-plane because it's going to immediately notice and change them back but we also designed cross-plane in a somewhat lazy or Kubernetes-like fashion where it's all very decoupled and asynchronous from each other so if you want to just go change the database there's like no forced relationship on anything else that your infrastructure is doing there so that's sort of the medium depth comparison to Terraform the comparison I'd say to all of the various cloud provider Kubernetes controller projects it varies notably the Google one is not open source at all and so most of the cloud provider ones again not really community driven owned by their vendors so you're really going to go sort of work with the cloud providers to get features added or bug fixes etc etc but they all are single vendor they focus only on their cloud so if you are want a provider for managing GitHub repositories or something that falls outside of a cloud provider you kind of have to go find some other sort of controller or operator there you could let's say you are in all of the big free clouds and a few other things you could install all of those operators all of those controller sets and then install operators for all these other things but then you have a somewhat disjoint set of tools that all are kind of the same overall in the Kubernetes ecosystem there's subtle differences between them in crossplane we have what we think of as the XRM the crossplane resource model which says anything that our provider drops in should write connection details in a standard way should have a standard sort of overall shape of its API for house and the spec and status is laid out things like that should have standard status conditions so that you can tell when they're healthy and when they're not healthy have standard annotations for importing resources things like that so with crossplane you get the kind of benefit that no matter what back end API you're on things will work similarly like all the APIs will seem familiar to you and then again the real benefit I think with crossplane is that it has this abstraction layer on top of it if you go and install Amazon controller for Kubernetes you're really just taking the AWS API and putting it in Kubernetes it doesn't simplify much compared to just doing a REST call to Amazon's API you're just doing a REST call to the Kubernetes API to make a database whereas with crossplane you can define your own API on top of that as I showed in one of my earliest slides that sort of wraps that up in your opinions so if your developers I worked at Spotify for a long time and one of the projects I worked on was before Spotify was in the cloud was their machine provisioning system and when we moved to the cloud one thing that was that we were conscious of in the early days was that developers already had a mental model from our previous system that we built of what machines were they didn't think in like Amazon terms they thought in Spotify terms of like machine sizes and features and things like that so you know in that example something like crossplane would help a lot because you could just move to the cloud and just keep the same abstraction sort of thing as like as a layer and this is something we see anyone who thinks themselves as a platform team many people think of themselves as necessary as doing in the moment they're really trying to build a layer between the people that they support their customers and the sort of huge amount of infrastructure that they run so that they can kind of present it in a slightly more user friendly and controlled way that's really what crossplane does that things like ACK in my opinion don't do we do love ACK though we work with their team and we actually share a code generation pipeline with them so there is a lot of camaraderie between us and the folks that work on you know ACK so KCC etc we done talks together and we chat and swap notes every now and again yeah that's my summary of all the tools in the space let me know if I forgot any thanks for doing that thank you I had one question you kind of hit on this already in terms of like setting up that mini cluster and you know using installing control plane there and then using that to manage some of your other concepts and constructs have you seen any other patterns that have developed in using crossplane in this way like get ops you know you've seen like app of apps and all these other ways in which to manage things have you seen interesting ways or you know typical ways that people typically set this up in their cluster or within their environments probably the most emergent way is is sort of the idea of a standalone or central control plane especially in really large orgs we'll see folks wanting to use crossplane for kind of everything so they'll want to use crossplane to deploy all of their clusters in the first place so let's say they've got 20 clusters around the world for whatever reasons or they might give a cluster to each sort of department or team or something like that for running apps on and they might want those clusters to have Argo CD installed have you know maybe Prometheus installed and a few other you know service mesh or something they like crossplane because they can have crossplane go and create the cluster in the first place so go to TrinityKS cluster or GKE cluster or whatever we have support for provider wise and then they could also use crossplane without providers for Helm and Kubernetes to go and declaratively install Argo CD or any of the other tools that they want onto that cluster and then they can use crossplane the central crossplane to also install crossplane onto those clusters and have folks over there use crossplane in that cluster to sort of manage their infrastructure apps whatever they try to do with that so this is something that there isn't a one of the questions in chat was is it possible today yes it is possible there isn't much in the way of sort of special tooling for it yet but you can really think of this as like process wise you know spin up a spin up a relatively small cluster however you like to you know however your spins up clusters you know be it in the cloud or on-prem doesn't need to be particularly huge because crossplane isn't particularly resource intensive you're not going to have a lot of nodes in your clusters you are going to have a lot of CRDs instantly which is a actually one of our biggest challenges in crossplane at the moment we're hitting the scaling moments of Kubernetes CRD wise it turns out if you have more than like a hundred or two hundred CRDs the API server was not designed for that so we're working with folks in the upstream Kubernetes community to try to make that better but you know spin up a relatively small control plane and install crossplane into it and you're kind of done you know then just by convention don't use that one to go run you know random nodes and instead use it to spin up other control planes other API servers and you know use those as your Kubernetes clusters amazing amazing cool no that totally makes sense to me and kind of like what you said before to run like Terraform and everything else I know I know I know the industry loves and us first them story but I've I've been able to use Terraform with crossplane and then kind of like use Terraform to get up to the bounds of instantiation and then use crossplane and flux and Argo and all those things like to handle those downstream things too so really interesting to see how everybody's using the different tools within the space and I appreciate it Nick thank you it's fantastic yeah no worries I haven't used Terraform in a little while just you know for obvious reasons because I you know have my new tool of choice but I contributed to Terraform before I ever worked with crossplane I really love the project and it's it's it is pretty funny because I you could totally have a crossplane provided for Terraform and then you could have you know crossplane calling Terraform and make yourself a nice endless loop or something I gotta work on that that's gonna be my next hack day project right thanks I actually have a kind of a follow-up question you mentioned like we were discussing about having composition and having crossplane doing the composition in in our case we actually use crossplane mostly to deploy the low-level stuff like the clusters themselves like what you were saying having a central cluster that then manages multiple clusters in different clouds different regions and maybe the S3 buckets you need or something like that and we actually do that using Argo so we basically have Argo CD resources that are crossplane clusters using that GCP provider or AWS or whatever you mentioned there's work ongoing to integrate Argo CD with crossplane a bit better what is that exactly because one thing we had issues with is for example if we deploy those clusters using Argo CD you can't really register those clusters to then deploy additional resources in the same Argo CD instance it's not trivial because like Argo CD has some logic on how to manage multi-cluster deployments from a single instance and if you're deploying the clusters themselves using crossplane as inside the same Argo CD instance it's kind of a loop and we have a kind of hacky way of registering those is this the kind of thing that is being worked on that doesn't ring a bell it might be I will say I'm not the biggest expert in Argo CD it's not something that I personally use a ton I we have a we have an issue on crossplane slash crossplane which is titled something along the lines of crossplane should play nice with Argo CD and that is our parent issue for some of the little bugs that we've noticed so some of the things that I can say are missing at the moment if I understand correctly is that Argo CD does not have a concept of readiness for most crossplane resources and this I think and I actually just pinned one of the folks from the other day one of the Argo project leads to start talking about this I've been meaning to reach out to them for a while just what notes my understanding is that that might be best fixed in Argo CD I think Argo CD basically has a been configurable set of types that you can teach in what status conditions for those types mean they're healthy but with crossplane when we have hundreds or thousands of managed resources that's a lot of configuration to add to Argo CD and then on top of that if people are just defining their own arbitrary types they probably can't reconfigure Argo CD for all of those so I was considering asking the Argo CD folks how they feel about adopting the case status emergent spec which really just says amongst a few other things if you have a status condition called ready then consider that to nothing else consider that to be the thing being ready I think another thing was that there is label copying that we automatically do between crossplane claims and crossplane composite resources that confuses Argo CD because Argo CD will manage a claim and it will add a label to that claim to say hey I manage my Argo CD and we'll just copy that over the composite resource and then Argo CD is like I'm managing this composite resource but I didn't create it so I guess I got to delete it or whatever so some of these things I think need to be fixed the first example there I think need to be fixed in Argo CD the second example probably needs to be fixed in crossplane so step one is going to be just talking to the Argo CD folks about the issues we're seeing and getting their opinion on these things to figure out where we fix them and how interesting yeah and maybe also a related question is regarding secrets you mentioned integration with external secret stores there's like a ton of ways of managing secrets on the whole stack depending on how you look at things like Argo CD also coming back to integration with Flux or Argo CD they both have also their ways of handling secrets also external secrets so if you would use some of these tools to manage your crossplane deployments where would the secret management be at or is there like a best practice on how to manage all of this no I don't think I can say there is a best practice because it really depends from from organization to organization you know I'm one of these people who doesn't find Kubernetes secrets to be that bad sort of thing I think a lot of people are like wait they're based 64 encoded at the API they shall never use them sort of thing either there are some R back issues and whatnot but but you know a lot of folks who are much more sort of opinionated and experienced with security than me I'm sure will violently disagree and you know different different orgs are going to have different policies some people are going to be can only put it involved other people say you can only put it in AWS is I think it was called key manager or something like that so our approach in crossplane is to we would like secret stores to become somewhat similar to providers so that you know you can kind of just plug in whatever makes sense for you there with the current alpha implementation we we just we effectively refactored the the API a little bit to be a little bit more generic previously it was the API was very specific to Kubernetes secrets so that API still exists because it's part of the V1 API so we're not going to get rid of it but we've sort of added a new field there's an alpha field that sort of speaks a little bit more generically about publishing connection details that can has a new type that's similar to provider config we've used provider config before we now have store config for where to store secrets and that can be told as to in tree baited ones at the moment one is for Kubernetes ones for vault the Kubernetes one can now publish secrets to a different cluster which is convenient if you have that kind of central control plane and you use that for infrastructure you can say hey just go push Kubernetes secrets over there which is kind of neat the vault one we may take out and make that a plug-in because we think that we'd like to sort of package all of these things as you know you can drop it in and you know if you use vault something rather than having it natively built in so we'd sort of limit the native support for stuff that interacts with the Kubernetes API and everything else would be a plug-in but it was a lot quicker to build it in natively for the alpha support so it's there for the time being and I know one of the other possibly maintainers are that is actually prototyping sort of a plug-in based approach for secrets at the moment that would open up all those at once but as to you know which one you should use or how you should do it and that kind of stuff it's kind of I can't really tell you that that's up to your org to yeah I asked this because in this group like we try to come up with like best practices that will simplify research oriented deployments but sometimes even the like the basic bits also needs some best practice recommendation but as you say it really depends on the actual tooling being used it's interesting actually because my background is you know 10 or 15 years in platform and SRE right but then for the last three years I've been building this cross-plane tooling and I don't run cross-plane a lot myself to go do anything so when it comes to learning the best you know I've run it to develop cross-plane but I'm not running the infrastructure anymore so when it comes to what are the best practices when using cross-plane we really look to what is the community doing so you know when I described that pattern of using a central control plane like that's what we saw people doing you know I have co-workers at UpBound who use cross-plane to run UpBound's infrastructure so I look to them to tell me what the best practices are and the secret store support only shipped in like the most recent version of cross-plane so it's still kind of early days to get a signal on you know what the best practices about that are awesome yep any questions hi Nick thanks for the talk I'm having troubles getting my head around how the REST API is constructed I mean in my mind there's something that's normally hand-built and crafted and you have to define the different endpoints and what methods they accept so in your example of Postgres there how does the API Postgres get constructed is it a matter of the difference between provided annotations and what's missing or it's a good question and to answer I'll ask another question are you familiar with CRDs or custom resource definitions in Kubernetes at all a little bit yeah okay so we basically extend those or use those as a framework and so CRDs when you install a CRD in the Kubernetes API it kind of tells Kubernetes hey I have this new type that I would like you to expose as part of your REST API and here's kind of the open API scheme before it and then the Kubernetes API server is very clever and is capable of basically adding all those REST routes and methods and things like that in order to accept requests for that but then the API server is really just kind of a document store so if you create one of those things the API server is really just gonna go cool I'll put that in and it's there and not do anything else so cross playing what we call an XRD which is just like a super set of a CRD almost slightly more opinionated takes away some of the CRDs in my opinion were sort of really designed for a software engineer to go right like it was designed for someone who's like building software on top of Kubernetes so we tried to take away some of the weirder fields of CRD that maybe like sort of an operator doesn't care about that much make a little bit more opinionated and then cross playing has controllers that understand what an XRD is so that we'll actually do something when you create one of those documents so the API server then knows okay I should create these routes and accept documents and accept REST requests basically with a certain shape and then cross playing also knows about those and so through the composition type you can think of the composition type as kind of the Helm chart basically like the Helm templates and Helm chart or like like the the HCL in a Terraform module sort of thing it's kind of the thing that says what should cross playing do when that REST API is called and interestingly what cross playing typically does is like it sees a REST API getting called that's been defined and it just makes calls back to Kubernetes to say okay create these other things in Kubernetes in REST API and then subsequent set of controllers go and create things in the external cloud which is then nice because you kind of see the process of rendering as a series of sort of documents in the API you see the API that you made you see that someone's called that and created a thing and you can sort of query that and list it and you can sort of query and list the intermediary things that cross plays to render it out and then you know cross playing will call someone on the system to do that kind of make sense? Yeah thank you very much that chilled in a big gap for me Cool So I think we're getting to the end of the hour so yeah thanks again Nick for this really nice overview I'm sure we'll keep coming back to the project as people get more familiar with it Yeah definitely let us know if you have any more questions in future and thanks for bearing with me with those technical difficulties at the beginning No problem thanks a lot for coming and talk to us So for I think we close here we don't have anything else in the agenda the next meeting will be in two weeks the topic is on bare metal deployments so if anyone has we don't have a speaker yet so if anyone has suggestions feel free to ping Jamie or myself and yeah see you in two weeks thanks a lot everyone for attending Thank you everybody see you next time