 All right, so I hope I can be heard. Yes. So welcome everybody to I think it's a last talk today in this track So I'm happy that so many people are here interested in a project Couple of people work on for I know one and a half years or something KCP It's complicated to understand there are many views you will see many views on this thing which is called KCP and everybody will find their own definitions their own use of that and I will try to Inspire you to think about how you can use it and How it's different to the tools we have had in the past especially Complimenting Kubernetes as we know it so about myself. I'm Stefan Schumanski. I'm looking at redhead. I Have been working long long time years on CID so I was implement of many of the foundational features and CIDs and CIDs are everywhere in KCP. So everything we see here KCP CIDs are behind that It all started with an experiment or with a Yeah, with a prototype 2014 Kubernetes added namespaces to Kubernetes Before that everything was basically cluster scoped nowadays. Everybody takes namespaces for granted There was discussion around whether they needed like whether the namespaces are necessary and yes the community concluded yes, we want them and they were added and The isolation we get from namespaces is pretty weak. Everybody knows that like you cannot run multi tenant Workloads with high isolation by our namespaces and namespaces are namespaces because they separate names Was never meant for anything more than that So the experiment we did was What if we did a different kind of partitioning of Kubernetes? What if namespaces provided more isolation than just names? The most prominent example, everybody knows CIDs are not namespaced, right? So if we want namespace CIDs, we get something like a type space Bad name maybe but not namespaces. So the experiment is let's do more isolation than just names If we do that, we get something we call it workspace and the workspace is a type space in this sense that CIDs are Independent, but basically everything else like discovery open API All that is independent even namespaces are independent if you have two workspaces You will have two default namespaces on both sides. So The result of that is basically a workspace is like a cluster. The user thinks it's a cluster But it's backed by the same Kubernetes API server if you want We put a target so we built that like in one API server partitioning Into workspaces so you can run it and you can have hundreds of workspaces Every workspace is basically as cheap as the namespace So as cheap and as quick you can create new workspaces in milliseconds And they don't cost because they're the same control claim behind it We put another goal and this is one million on One at CD one API server. You cannot have a million namespaces. Everybody knows that So we put another goal here, which is far beyond what you can have in one API server just to drive our exploration Obviously, we need something like many API servers horizontally scaling so that was a goal the idea as a prototype and The goal of course was clusters become uninteresting like you your cluster is your workspace You don't think about that there are there are other people on this cluster If you are talking about namespaces, usually you're not admin in the cluster So you think about there are other people on this cluster But if you have a workspace where you can install CRDs and everything else and you're admin you forget that there is actually a cluster API server behind All right, so if we do that and we did that those question come I mean if you talk to people about KCP is this a better Kubernetes will we replace Kubernetes with that? And now we have KCP and it's so much better Can I install operators in a namespace finally and? If KCP is its own project the question comes can we build that into cube itself? like can we do something like namespace and V2 and Forget forget about the questions for today. Those are basically not the questions. We are asking we try to step back so we have seen Turing machines in One way or another in the studies of computer science or mathematics So they're compute models. This is a physical Turing machine. It's a machine which can compute. It's very theoretical But it can compute everything basically you can compute in any programming language and there is this Yeah, this band this thing with the characters and it computes it operates like a small CPU and In computer science you ask yourself. What can this thing compute? Can it compute the square function can it compute exponential functions? And it turns out the thing can compute everything and then you ask yourself for what is it really? suitable which things you can compute on this thing in a elegant way and Let's get into this mood of thinking we build a machine You know one machine with this Kubernetes API server you can build controllers Kcp is built on on on on Kubernetes, which means you can build controllers But we will extend this thing. We don't stop at the workspace. We will add dimensions in a second We build something we call it for today the Kcp machine and we ask ourselves. How can we use that? We are not building the better Kubernetes if you look into the into the website of Kubernetes It's saying basically it's a production grade container orchestrator Kcp is as a core project is nothing has nothing to do with containers nothing to do with Orchestration of containers so it uses its technology of Kubernetes, but we try to generalize it So forget about containers for today for the next hour. It's not about containers For this talk Basically, what you get are those workspaces. Here's one. It's pretty empty. There's nothing inside They are CRDs. There are a couple of things like conflict maps secrets airbug Resource quota all those generic things, but nothing about workloads just not there So it's a generic API server and it has CRDs Alright, and whoops when we do that We want to add three dimensions, so we'll talk about three extensions now So we start with this empty API server and we extend it the first dimension I add we have seen it already It's a the red arrow here the red dimension one million We want many of those things Every single workspace is like a small Kubernetes cluster so you can't write controllers against it You have at CD behind so you have all the resource version semantics, you know from Kubernetes So basically it's it's Kubernetes at its core, but it doesn't have the workload APIs The second dimension is if I have a million workspaces What are Services between them? How can controller operate on one million workspaces? So we are thinking how can I program a controller which is basically multi workspace aware? How can I do something like I get objects? Everybody knows list and watch in Kubernetes So you have informers at the programming model of Kubernetes What is an informer against many workspaces and how does an operator do something with many workspaces? That's the second dimension and if you do that if you think about that and find a solution and I will show How it looks like later on? You can build multi tenant services Nothing you can really do in Kubernetes at least not across a million or something like that a third dimension is we At basically locality over the whole planet So you have workspaces across the planet basically in every region for example of the planet Every workspace in itself again is like Kubernetes has the guarantees of at CD for consistency Across workspaces. You don't have that especially don't have that across regions across regions You need a different model and I put those blue boxes here at the bottom They are not consistent as Or in the sense of at CD For example, they're eventually consistent. So if you need global states, you have to somehow get that in an eventually consistent way Alright, so if three dimensions horizontal scaling API service provide a provider persona. So this is Controllers which are against multi workspaces if you do that you get the service provider persona into the system and planet scale so it's really distributed over the planet and The question is again think of a compute model. What can you build with such a construction? And that's a challenge for us today. So if we translate that Imagine you have one one million generic workspaces cheap and quick as namespaces So in in one millisecond you can get another one spread over the planet and if you do the math Thousand object per workspace you get a billion objects Something like that And I have some examples here what you could build with that Who has built end-to-end tests of controllers? I guess a couple of people here So controller runtime controller tools as something like an inf test right? It starts an API server and you can do your your control loop testing if you have such a thing which has space for one million forget about an API server just use such a thing and Point your controller against one of the clusters one of the workspaces and Do stuff until it's finished and then delete the workspace again Again 100 milliseconds you have a new one super cheap super easy That's one one use case If you have a million workspaces you might think about how do they relate? One obvious thing is ork hierarchy so you could maybe model a company hierarchy in workspaces so every workspace has one parent and This means you get a tree like this yellow company and it's a green company at the bottom there So you can build something like that and we will see later on so we have done that and we have Built a user experience around that in KCP. It's not necessary like it's an optional thing But depending on the use case. This is very helpful Clusters means they are isolated in a sense. They have their own air box system So they are the whole and roll bindings completely independent and this means you can basically start as a cluster admin in every workspace and It's independent from the permissions of the next one of the neighbor basically So you can start with a model where everybody is cluster admin again. They are much smaller than clusters So sharing is not a problem If you have many of those workspaces, right? We share clusters in Kubernetes today because there's overhead like to get a new cluster. It's expensive and Just takes lots of time If it's so fast and quick and cheap to get one Everybody can be cluster admin again Multi-chain operators. I talked about that so service providers for those workspaces. It's an obvious example See these are independent so you can have different versions in different workspaces Green is we want V2 alpha one and one V1 and V2 we have here So everybody can choose basically which CID version is desired If you do that, you can also think okay multi-chain operators one can be one dot two three so an old version and people stick to it the yellow people and There are those cutting edge people They want the newest version and they can run in the same environment in parallel so you can build such a setup and Keep services running side by side a Generic control plane is super interesting if you don't talk about workloads Cross-plane is an example cross-plane is a project which talks about non workload objects and This is a very good example a very good use case how you can use a generic CID based API server Cross-plane doesn't need deployments. It doesn't need things about networking about ingress. It's just Unused by cross-plane so use something like KCP and run cross-plane on it and overhead for cross-plane is much much less than on Kubernetes because They can share infrastructure think about use cases where Yeah, in the IOT space in the edge space You have a million objects and they somehow needs a consistent workspace a consistent control plane You could use that to connect your fridges to KCP for example and last but not least Here comes I mean communities comes back into the picture If you have many clusters you might think about use cases where you think policies or where you think actually the workloads Or you have the source of tools which operators should be installed and There's the last thing I will have another slide in a second You might share API's between the KCP on the left and clusters on the right side And with that you get something like API as a service based on CIDs So if you want to understand more about the last use case There's another talk tomorrow and birds of feather on Friday about the idea to have basically Services on the KCP side, which are multi-tenant just by the system by definition and To connect many many tenant clusters Kubernetes clusters to KCP And you get a model which implements software as a service based on Kubernetes CIDs Supernative no rest API, which is non-standard, but there's normal CIDs and Yeah, there are those two Possibilities to hear about that All right, so I Short some examples. I bet the audience here has many more We start with the fresh Kubernetes Without the workload API's every cluster is generic and isolated and we add stuff And the challenge for you is to think about what could you build with that by adding by re-aiding the things you want And to have many many of those workspaces alright, so zooming in a bit into API sharing so I said a service provider can define API's and Can export them to many workspaces and built an operator operating then on the tenants So here's this example. That's the actual API about that to get a bit more concrete So you have the API resource schema imagine this looks like a CD So everybody who knows here these you have the names here the kind list kind and so on that's basically a CID spec Nothing else you define that in the system in KCP If you do that, nothing happens. It's not that KCP serves anything. It's just a definition of a resource schema And then we have built something called an API export so you can take those schemas name them here in the list in the slice Create an export of that and make your API resource schema as an API Available basically publish that to the KCP To possibly one million workspaces everybody can bind to your export and use your Controllers basically your semantic you define for your API So up to a million API bindings can exist and they point in some ways So we use a path here in this hierarchy in the prototype they point to the service provider to your export basically and In the moment the user in his workspace create this object The the API in this example at third manager is available in that workspace. Nothing is running in the workspace You are the one on the right side who operates To op who operates Those objects So you will see every object the user like certificates in the case of third manager Every object the user creates you will see in your controllers and you will do whatever is necessary to implement the semantics and again one two One million that's idea alright, and This feels like software as a service right software as a service you have this persona this one That's person basically who export something and many many consumers and you come back to I mean if you do that you have software as a service at least in KCP and This is for another talk around I did some time ago in Kubernetes we have Built something which doesn't support sauce this export thing doesn't exist in Kubernetes, right? You can have CDs You can have operators, but there's no way to get an API from somewhere into your Kubernetes So we have built something like that. We have something like services So we have CDs custom types, but there's something missing and this is basically this thing and There's a CLI for that. We have built to bind and export to your local workspace alright, so One million workspaces a billion objects cheap and fast think about what you can do This is a website. We have created to to bring Say basically what what KCP is and the main the main points here. I want to to highlight Maybe this is the most important Slide for understanding. This is not about replacing Kubernetes It's using Kubernetes at its core like we use it Basically, it's a complete API server minus the workload APIs is in KCP. It's used in KCP Everything in a workspace. So this is a set rule set and stone everything in a workspace is Kubernetes, so it's every API we offer is conformant to Kubernetes very important every workspace is like that Tooling just works if you use q-cattle, it just works in the workspace and the API is we build others just conformant They are based on CDs. You don't see CDs, but in the background CDs are at work all the time so We complement Kubernetes. This is most important here and we live and breathe Kubernetes and the community So we are all part of the community of Kubernetes All right, so isolation through workspaces is a big thing we have seen that Can be many admins in every workspace another admin and Many devices can be connected to that so as relations who workspaces we have seen up to a million massively multi-tenant API services so you can build controllers operating on many tenants Users can consume those services natively by a bind command and there's nothing running in workspaces You don't see how things are operated, but they are available and the API is just work as you would expect think about this whole thing is horizontally scaled and You have workspaces Over the whole world basically in every region you can have you can have workspaces But it's from it's forming basically a mesh So it's one system and you will see how the UX looks like in a second. It's one system one installation it's not like Kubernetes clusters which are in this region in this region in this region and you have to decide it's It's all in one system This is a walled garden. That's an important question. So we build bindings, right? We have exports and bindings It's nice. It's in KCP. But what does it mean for my Kubernetes cluster? and There's a project we have also started To write multi-tenant operators in KCP and then use something like cube bind Cube bind is an open-source community project. We have started in it's an available in Github So you can search for that and this brings back everything you build in KCP large-scale like if you're a service provider And you have thousands of customers You can use Kubernetes APIs built in those workspaces and bring them back to Kubernetes clusters and Cube bind is doing exactly that. So if you look here It's a bottom and small maybe you will find again a cube cuttle bind command But again or here it's not against an API export in KCP, but it's against HTTPS URL And tomorrow there's a talk about that to see it in more detail how this works So no walled garden KCP is the back end is a system to build multi-tenant services. That's one view of KCP So show me the code. What is KCP as code? Everything was theory now. So let's get this into more concrete context so here's a repository KCP dev KCP on Github and Yeah, you can check it out. You can you can run KCP So that's let's clone that into the to the local directory and it's all in one If you started with KCP starts, it's it's not like Kubernetes with different components You can just say KCP start and it's running If you do that There's an admin Qube config being created in dot KCP if you set your Qube config variable environment variable to that file Cube cuttle will work. It's Kubernetes. You are in a workspace. So two commands later It's running and you are in in the first workspace So we built this command W s which stands for workspace and it's abbreviation You could also write Q Prattle KCP Workspace but because this is very long and you type it often we abbreviated that to just W s and this operates more or less like CD change deer in in Unix so you can say Q Prattle W s dot and it tells you you're in root Root is the first workspace. There's nothing else in the system. It's pretty fresh fresh at CD. Just one workspace called root And you can create another Another one. So I create one which is called QCon and It's created it takes I know 100 milliseconds to do that. It's ready and we can enter it So let's enter it. I said it's like CD. So let's say Q Prattle W s QCon and we are in In the background your workspace is set is put into the QCon fix So there's a different URL pointing to the new workspace and you have a fresh Kubernetes like cluster It's empty. It's generic, but you can add everything you like via API bindings, for example, or any CD you like and Remember or kayak east was the example what this is implementing is exactly that we are now at this position in the hierarchy So we started somewhere. That's it. It's a top here and we moved one level down that's our cube con workspace and we are there and We can go back like that. We are in a different workspace and this is very much like Every workspace is like a directory in Unix same idea and again This is just one way to see and use KCP KCP is much more generic You can you can run KCP. These are working on that without any hierarchy but this is just one way to use it and For many use cases. It's pretty nice to have that All right, and every workspace again is like a Kubernetes cluster without API's Of course, you have to add API's we will see it in a second. How this works. So you can see API resources, what is there conflict maps secrets events name spaces resource quota everything which is generic Service accounts airbug and so on. So this is what you get when you check it out and start KCP start you get a Cluster with one workspace and you can create any any number you like so to sum up KCP as a repository gives you that gives you a generic control plane So Kubernetes based on Kubernetes, but we have removed disabled supports. It gives you workspaces So isolation in one in basically one chart It gives you a command to move around it gives you export and bindings and it adds Sharding we're working on sharding to run that to run that in multi multiple charts Cross the globe with eventual consistent global state. You can imagine why I'm talking about global state One global state you see here on the slide like the exports must be global right if you export something to everybody in this mesh of Workspaces on the planet. You have to to replicate your export everywhere. That's one example We have more examples of that but API export is one of those things So you will serve everybody in this mesh of workspaces and that's the fourth point here. That's sharding Alright, so we have empty workspaces and you can imagine what you can build on on top of that We have some things some projects some site projects basically some services We are working on and one is to read compute So we have removed compute Cool for many for many use cases like if you talk about cost plan, you don't want compute but compute is kind of interesting and one topic we have tried what we're working on is Building something called transparent multi cluster. So it's a compute service It uses the API set cube also provides the deployments and services and similar things So you can apply the usual API as you know, but you get something like a basically an elastic compute service based on communities API The inside here is I mean, this is just one service very important there can be many many others and What we have done here and this is a pattern. I think People like in this area. So we have re-edit deployments. We have re-edit services But we have not reused the scheduler of Kubernetes So to build elastic compute you want a different scheduler obviously. So we run workloads not in KCP We run them on real Kubernetes clusters, but we are hiding the clusters from the user So you can add more and more clusters as capacity basically and the user's workloads the deployments in the example here will be scheduled on to those capacity clusters So we have replaced something of Cube. So this thing the construction we have built here in KCP By removing things gives us lots of flexibility to read them like we can read deployments But do another kind of scheduling If we take that to the next level There are different like different kinds of compute you can imagine So many people might know V cluster for example, can you build V cluster on KCP? I bet you can Again, you get all the sharing of workspaces. So it's an interesting topic Maybe to talk about things like V cluster where you put KCP on top of one cluster of one open-shift or one Kubernetes and reuse the compute below but give everybody a workspace but share infrastructure like API server another example You want deployments, but you want to schedule to edge devices Edge devices aren't I mean if you have many edge devices like tens of thousands or millions even you will not have a million nodes in a Cluster right this doesn't work. So the schedule of Kubernetes. Maybe it scales up to five thousand but not to twenty thousand or a million So maybe for edge you want to do the same pattern the same thing you want to get back deployments and other Kinds of yeah workloads basically workload like resources But you want a specialized API for the scheduling for the placement of workloads So there are people in the community Who think about that and next week say they start something like I know maybe it's a sick Which is coming up. We will see but to talk about API is placement API is which reuse cube compute objects But build another placement Mechanism and There are many more of those things and this is a pattern that you can basically reconstruct three Compose things we know and you are much more flexible than in Kubernetes All right, this is a website. So take a look We love collaboration everything here is open. So it's KCP dev slash KCP on GitHub It's not a source project community driven. We have weekly meetings on Tuesdays 5 p.m. European time. So Six hours before what is it 11 or something? We have a YouTube channel if you want to watch discussions all community meetings, it's all there if you Google groups, so if you want to if you prefer emails, that's fine and The most important thing we have a select channel. So we are always there as a basic basically doing work hours So there's always somebody if you want to talk to us explore KCP or even help us contribute that channel is the main place for communication All right That's all I wanted to show and I think we hopefully have some time for questions Before we start questions, we have t-shirts here in the front. Somebody's interested in t-shirts Andy is here. Andy is co-leading KCP He does see with the two t-shirts. I think so come here and yeah Alright, so time for questions Great presentation. Thanks Stefan Two questions. Is there any relation to hierarchical namespaces? With KCP no technically not namespaces So we are also asked and this is a similar question. Can we just do namespaces v2? I mentioned that right something similar and namespaces v1 there in API and it is limited as it is, right? Sure, so we cannot change anything Workspaces have a hierarchy if you use this model With this model you get the hierarchy, but the namespaces inside our flat again because they are namespace v1 The pattern you see people don't care about namespaces anymore. So just stay in default and that's it so, I mean Would it be possible or? Thinkable to have the workspace be a core concept in Kubernetes API Yeah, it's feasible certainly the API server we are using it's a cube API server with modifications it adds partitioning of the key space and yes technically this is feasible It's not done at the moment. So we add we use cube API server and API extension API server and we add exactly that you touched on the idea of syncing at global scale in an eventually consistent manner What's the mechanism for that? What would you be using to have them talk to one another? At the moment we are just exploring something pretty simple. It's a Distributed cache. So the objects we know we want to share we want to replicate every chart will push them into a cache which is globally replicated and The most interesting bit here is the question What is a compute model in such a words like how does the controller work on eventually consistent? secondary data So the model we come came up with which we are implementing is basically a Controller which is aware of that right? It's a different model a little different from Kubernetes extended basically it has a local Informer so it has a local Memory cache from the local chart. So it sees everything every workspace local and If it doesn't find something there, it has a second informer, which is fed by this distributed cache That's a compute model we came up with This works very fine with those things like API exports or in compute we have soon targets for example things which are pretty static in a sense I would eventually consistent and nothing stops us from adding API is for example for Consensus, so we I mean sometimes you want that right you need consensus, which is some are global and There could be an API which is based on a different mechanism not this Distributed cache but something else which gives you consensus those things can be built in they can be even primitives It's not I mean it's an open domain basically what we need and what we built there Thank you Stefan for the presentation You mentioned earlier like that the target is to be able to create a workspace and make it as cheap as namespaces Can you tell us more on how workspaces are implemented? Like is there a dedicated API server? Process a dedicated DCD. How does it work in so? We add something I mean, you know it how containers work in in the Linux kernel, right? There's some kind of context and there's a namespace name passed around in the Linux kernel We have basically the same model here, so we pass around in the context something we call a logical cluster It's part of the API server URL the client talks to so we derive it from the URL from the path and pass it around and In the storage layer of Kubernetes we turn that into a different prefix in at CD so that's a technical trick to do that and There are some some things I don't go into details here, but you want Efficient lists cross tenants, but on the same resource so you have to put The logical cluster name at the right position in the key and we can talk about that in detail But that's the idea. So that's why it's efficient. That's why you can have inform us cross workspaces Thanks, Stefan. Great talk. I've done so question, but I'll limit it to one for now You mentioned that with KCPU and vision like developers having a single API where they can still deploy things like Qtl applied a shift deployment to yaml But the actual workload runs on one of the underlying workspace clusters. Yeah in this model of Transparent multi-class it's one service. I stress that it's one service built on top It's not necessarily the compute service. You can have many other variants that one works in a way that every Kubernetes cluster which is capacity for this service has a sinker agent which connects to KCP it watches its tenants so it watches deployments and it pulls them down and There's some transformation happening to make them work on the Kubernetes and then they're executed in Kubernetes And the status is synced back. So in the in the workspace, you see deployment dot status Whatever you are used to in deployments. I See, okay. So syncing all the status back to the KCP server for that deployment That's a model of the service. Yeah, I see. Yeah, thank you Thanks for the great talk are there already client libraries or documentation or guidance on how to write a Multi workspace control. Yeah. Yeah, so so Andy is an expert here If you want to come after the talk, he knows all about that. We have a extension of controller runtime clients and we have a Generator which is similar to client go which adds scoping to inform us and everything so you can you can watch Everything on a chart, but you can also scope it down to one workspace. So it's very client go like I think it's even derived from that I'm not sure but any is expert. So client libraries Thanks a lot question Where did you want more? So yeah, one thing that is not clear to me is isn't KCP subject to all the scalability limits of Kubernetes So when it comes to at CD as far as number of objects we store or you know the way eventual consensus here are replication works isn't KCP subject to all the scalability limits of at CD the same way you Know at CD is I mean if you scale out we have multiple at CDs. We even Researched it's not really ready and we don't use it at the moment We researched to use something like cockroach at the basis to scale even more But the point is and this is it's an intentional decision workspaces are Smallish clusters so they are usually not the size of a 5000 node Kubernetes cluster So of course for one workspace you have the scale limits of at CD obviously because this always lives on one at CD or the whole upper machinery Limits what you can do in one workspace, but because workspaces are this Consistency domain so the resource versions only makes sense with all this inward one workspace You can scale them up like you can have many many workspaces or many many charts and every controller really cares about One partition of that. That's the way how we get much higher scale I see so you're focusing only on a subset of objects like workspaces and through the export APIs We're subjectively exporting them to other clusters and that way you're managing scale So the objects always live in one workspace and the workspace has it must fit into one at CD That's that's a limit the export is cross workspace. That's completely independent But the objects live in one workspace Thank you so much. All right. Thank you