 Hello, thank you for joining me for from tweet to bad idea creating an embeddable convert any style API server My name is Jason de Tibbers I am at equinex metal and I am working to bring cloud native infrastructure management into the data center I am also one of the co-maintenors for the cluster API project and you can generally find me on Twitter at de tibber So Going back to Twitter is how this all started I put out a call asking if anybody Thought it would be possible to go ahead and build this idea of a minimal Kubernetes style API server that supports CRDs And that took me down quite a bit of a rabbit hole But why would I want to case this rabbit hole? Well because CRDs actually give you a lot to be able to build out applications pretty quickly to provide a lot of value For example CRDs give you out-of-the-box data persistence version semantics defaulting Validation and clients and those clients don't just apart crowd applications They also give you the same list watch type semantics that a Kubernetes itself does However CRDs also come with them a pretty steep cost for certain types of applications because it brings a dependency on Kubernetes Now for most things that's not generally too bad because You're generally deploying applications on infrastructure that already exists things like that But what if you're building applications that manage that underlying infrastructure? For example in the cluster API project we work around that today By having the ability to use kind of an ephemeral bootstrap cluster to be able to stand up the cluster API manage infrastructure and then Kind of move those resources over to that cluster that you just created Now that works okay for cluster API But what if you're trying to get something even deeper down in the infrastructure? For example automating bare metal infrastructure in the data center Like the Tinkerbell project which is a CNCF inbox project Well you got a chicken and egg there that is very difficult to kind of resolve So is there a way to leverage CRDs in that type of environment? well This rabbit hole got me doing a lot of research, you know has somebody already done this You know, there's been kind of rumblings back and forth between people in the community about whether or not This is something that could be possible. You know people talking about potentially doing it Tim Hawking was talking about this back in 2018 about trying to move more Kubernetes resources themselves towards the RDS versus being built-in types and things like that So what is the current state? Well, I started prototyping it out a little bit and it took me about three days to get to the first POC of this and It worked okay. There were some challenges there, but it seems at least from this point to be possible So before we get more into that, you know, what is the goal that I'm trying to solve well To begin with, you know, like I said want to be able to build a minimal API server capable of supporting CRDs But would also like that to be able to be embedded into the application itself so that you don't have that external dependency on Another system to be running in advance of the system coming up The other thing is is I wanted to be able to avoid exposing all of the command-line arguments that the Kubernetes components themselves do because they may not make a lot of sense If you're trying to stand up an application that isn't Kubernetes itself I also wanted to try to avoid needing to do any crazy vendor management Or having to import directly from Kubernetes Kubernetes because that creates a lot of maintenance burden and you know There's a lot of version lockstepping that you have to do Additionally, I want to try to avoid where possible reinventing any new unnecessary wheels I don't want to recreate the listing functionality of Kubernetes Where it's not absolutely needed I would like to be able to leverage the actual community code wherever possible I'd also like to avoid exposing unnecessary ports, you know, there's there's no reason why An application would need to expose things that are only used internally So can we take advantage of some features to avoid having to expose many different ports and talking back to each other on open public ports and And The other thing is is I want to be able to support standalone and what I'm calling bootstrapping use cases so being able to solve that chicken and egg scenario of managing infrastructure before that infrastructure already exists and What about the existing tooling? So if you look at some of the existing tooling in the space it generally falls into several different categories of the first one's API aggregation and Generally right now the most mature project for that is a simple API server Kind of like API server the hard way you basically have to create everything from scratch and That seems very painful maintenance wise and it also doesn't actually give me the capability of being able to run CRD's using it There's also a new and upcoming project API server builder alpha, which is trying to do for API aggregation what the cubular project has done for building CRD based projects That also brings into the same kind of deal is that it's meant for building aggregated API servers and I really want to be able to leverage the functionality of CRD's rather than API aggregation and Then there's the CRD based tooling, you know cubilder and the related projects and Those are great for building CRD based applications that have an external dependency on Kubernetes Maybe not necessarily for trying to you know Do the embedded style model that I have hopefully My thought is is you would still be able to rely on these tools in addition to A new project to sit there and provide the full functionality for this So we don't expect I don't expect to be recreating the functionality that exists with the CRD based tooling And then why not just use the underlying libraries like I said, I don't want to reinvent unnecessary wheels So I started again into the cube aggregator and API extensions API server projects and Try to see, you know, whether I could just use those directly and port them in for a project and What I found is that in general, there's a lot of overly verbose and replicated configuration between different components that you have to pass long They're also right now very highly tied into Cobra and if you use the commands directly it automatically exposes all of the command line arguments from Kubernetes and Quite a bit of the functionality for actually tying together the extensions API server and the aggregation server right now actually lives in Kubernetes Kubernetes itself and Trying to import that in and use that directly not only creates that dependency on the main Kubernetes repo and all of the dependency management challenges that involves but it also brings in all of the functionality and dependencies of Cube API server and I would like to be able to expose something much more minimal and easier to configure and The other one is is that it's quite difficult to actually correctly wire them up properly Like I said, these are tied into command line arguments So there's a layer of command line config that goes into options that then needs to be chained through several different ways and and The components are kind of glued together at various different points of that configuration chain so trying to Correctly wire them all up together is actually quite error-prone today outside of the main repository So what exactly would this standalone use case look like? Well in this In this simple use case my thought is is you would have your own application Providing its own, you know services for whatever you're trying to do you'd be able to embed in kind of what's currently this bad idea project and You can leverage the existing Kubernetes clients and tooling to interact with that application how you need to So let's take a look at that in a bit more depth For the purposes of this demo, I am going to take advantage of two different repositories Both under the github organization the tire fire. I created this github organization Basically as a joke because I didn't know exactly if this idea was going to have legs or not and the core library that I'm going to be using today is under the bad idea repository and The example Application that I've started to create that actually embeds the bad idea API server is just example right now So let's go ahead and take a look at that code I'm not going to dive too much into the bad idea repository right now But let's dig into the example a little bit So if we come in here Just look at the dependencies that we're bringing in right now we can see that I am importing bad idea and Outside of that importing very little from the other Kubernetes repositories I am importing controller runtime here because I built this project out using the Q builder scaffolding for the CRD resources and the controllers And I did have to pull in API machinery and client go Right now, I did have to make sure that the controller runtime version is synced up with the API server version that bad idea is using I don't expect there's going to be too much that we can do to break that just because of the way that All of the individual Kubernetes components work today and I did run into one initial issue with Transitive dependencies that I'm having to work around not right now with this replace because pulling in controller runtime right now is forcing or trying to pull in a newer version of Nostik here and This replace is helping use the older version that's required by the API machinery right now There are some PRs in place and some work being done upstream to Resolve these issues and once that's resolved. I should be able to remove this replace So what does this application actually look like? I'm going to warn you that this is very much an initial POC, so it is a little bit rough at this point But if I come in here we can see Here in the main function it looks very much just like a normal Cubilder-based application I'm creating Basically the signal handler to be able to handle cleaning up of the resources I'm spawning off the bad idea server and This is doing a lot of legwork behind the scenes for me. This is actually going to spin up the Embedded at CD instance. It's going to stand up the API aggregator and it's also going to stand up the CRD server Once this project gets a little bit further along and implement some of the core resources It'll also stand up The necessary components to be able to handle that as well and then at that point I'm creating a Client configuration Into that bad idea server and if I go up there to that real quick all I'm doing here is basically generating kind of a Very basic rudimentary cube config The server I know it's going to be coming up on localhost port 6443 It's going to automatically generate the certs the first time you create this similar to how the Kubernetes API machinery does So because of that I'm setting insecure skip TLS verify so that we don't end up with any certificate errors and I'm also configuring things right now to use the allow all authentication policy So I do need to set a username and a password here to make the client tooling happy it's actually going to ignore that and just allow it anyway as an unauthenticated request and Then I'm going to put that together in the cube config and With that cube config I'm then generating a rest config and Here's where things get a little hacky right now But as soon as what I'm doing here is I'm basically waiting for That bad idea server that we created to Come up. So I'm creating a discovery client here and making sure that the server responds before continuing and Again a little bit hacky right here is I'm basically just shelling out to customize and cube cuddle right now to apply the CRD's that are Defined through the cube builder scaffolding and Then at that point After those resources have been applied I'm now actually starting up the controller manager For the cube builder based controller in this project now I Just really started kicking the tires on this I Didn't really have a really good Example to use so basically I just created a Very simple bar type within the within a food group for the purposes of this demo It's just a very simple spec definition it defines a color and a shape and For the status it outputs a path I'm not actually doing anything with the path right now eventually. I'll probably actually Create an image with the color and the shape and put it somewhere accessible from the server But I'm not actually doing any of that right now If I actually go into the controller We can see that all I'm doing in the recon style right now is I'm getting the request Requested resource. I have a couple of to-dos in here to actually do something with them And then I'm simply just setting the status just to show that The resource has been reconciled So I can go ahead and bring that up right now And this is going to actually go and spawn up the all of those instance All of those services that I mentioned And we're going to see some errors in here. I'm still working out Some of the intricacies around how to wire all of these things together Especially in the case that we don't have things like services and endpoints So I still need to troubleshoot what exactly still trying to talk to Trying to query the server for services and endpoints to clean that up But let me go ahead and open up another terminal here and Here what I'm going to do is I'm basically just going to create a very simple cube config similar to what we Generated inside of that application and write that to the disk real quick and export The cube config environment variable so I don't have to keep typing that And now if I go in here and cube cuddle api resources What I can actually see is now I'm talking to This example api server that's running And the only resources that this server knows about are custom resource definitions the api services for the registration And the bars type that I've defined and applied When bringing up the service And if I go even further I can cube cuddle explain Explain the bars and As we saw in the api definition We have a spec we have a color a shape and the status defines a path So with that I can go ahead and Let's create one of these resources So here we're defining a very simple bar With the name of test color blue and the shape is a circle We can see that that's being created And let's go ahead and take a look at that in more detail And here we can see that it in fact has been reconciled and we do have a path defined But remember we're not doing anything in the controller to actually instantiate that now But it does show that the resource was reconciled And the status was updated for it So now We see that we have an api server The api aggregation is working to be able to contact the api extension server And we are able to actually Create crd-based resources And interact with them But is that actually persisted so let me go ahead and kill this service off And like I said, this is a little bit rough right now. So when I killed it off ecd isn't quite cleaned up properly So that's something that I still need to Sort out here. So let's clean up those socket files there And let's go ahead and bring it back up again And if we are actually persisting this data to ecd then when I bring this back up I should expect to see the resources The resource that I just created Still there So let's see if that happens And we do we see that resource is actually there. So we've Created this minimal embedded api server. We've we've wrapped it In this example application where we've also defined these crd's and With one binary we're able to bring up a system that is persisting data with ecd And can interact with crd resources. All right, so that means that this is Completely ready to use right and you should use it in production I wouldn't go that far yet. There's still a lot of work to be done here So what exactly is missing? Well, the first thing is configurability of pretty much everything um everything from What tls certificates you want to use to being able to use an external ecd data source or even some of the different Configuration around what ports you want to listen to and things like that Also, it's missing kind of What the core core components? That tim hawken was talking about the things that you can't necessarily implement as crd's Things like namespaces and rback We're going to need support for things like config maps to be able to handle leader election for controllers possibly secrets And sure i'm missing other resources that may be needed in practical application Uh, also, we're missing the basic admission controllers things like the namespace life cycle management to Be able to create, you know ensuring that a namespace exists before you try to create resources on it things like that support for Being able to do the defaulting validation and conversion webhooks that are a core part of What really makes? the crd Implementation very valuable And this is likely actually to be the biggest challenge that we have for the project Because we're going to need to implement some type of services implementation to be able to do this and since we're not Implementing all of the core Kubernetes resources like pods. We need a way to be able to route that Service request to something that is actually running that webhook So we need to try to figure out how to handle that type of situation And then finally the ability to run All of the non-deployment based components of a non-trivial crd based application And I figured something like cert manager would be a good Kind of milestone like can we run all of the bits of cert manager outside the actual deployments? As part of this project So if you are into bad ideas or otherwise are interested in Being able to have an embeddable api server that supports crd's Please reach out via the repositories And I plan on continuing conversations with the community on how we can improve use of these components for use cases such as this Thank you very much