 So our talk today is entitled Mission Accomplished. Kubernetes is not a monoripo, now our work begins. Thank you for attending on one of the last slots today. I am Justin Santa Barbara. I have been contributing to Kubernetes for, I guess, longer than I care to admit now. I've done a lot of work initially on the AWS Cloud Provider. I kicked off the Kops project a number of years ago, and I've been at Google now for a number of years trying to help people with their adoption of Kubernetes. Hi, everyone. My name is Cyprian. I'm a and I'm the engineer at Microsoft these days. I am also helping companies migrate to Kubernetes. And in my spare time, I'm a maintainer and contributor to various cloud native projects. So OK, so this is what we're going to be covering today. Broadly, why did we split up the monoripo? Why did we go from one big Git repository source of truth into the many repositories that we have today? When we did that, did we lose things? Or was it all just just upside? And spoiler, the things that we did lose, how can we get those things back? That's what we're going to be covering today. OK. In the beginning, there was one big Kubernetes repo, so everything in one place. All components were versioned together, like there was no cloud provider or other things. And there was end-to-end testing multiple clouds on every PR. And that was actually pretty good. It was the right architecture for the time. It was actually very easy to make big cross-cutting changes across multiple layers. You could have one PR that could change the API. It could change the Kube API server, the controller manager, kube-cuddle, it could change the scripts that installed Kubernetes. You could do that all in one PR, and everything would just go together. It was a very productive way to move quickly. And we had good test coverage as well. We had test coverage that would test on AWS and GCP. We had some other tests on other cloud providers that were a little bit more complicated. But those, the AWS and GCP tests were kicked off on every single PR. So every time you made any change to Kubernetes, even if you just changed the docs, we would make sure that you hadn't broken AWS and you hadn't broken GCP. And we were able to test all that cloud provider functionality, all the functionality in Kubernetes on every single PR. So it was a good productive place where we were. So it was pretty good. But why did we stop doing this? There were a lot of technical issues, many related to GitHub. There were tons of notifications, so people could not keep up with them. There was a huge volume of comments and it even broke GitHub loading PRs. And the worst and most frustrating, you couldn't merge PRs. Like it could take two, three days to merge your PR because there was a huge queue right before each release. And this was when bulk merge was introduced, but still was not perfect. We also had a bunch of what I would call people issues. So it was hard to know in this growing community of people who needed to see each PR and who didn't need to see each PR and make sure that like the right people had approved the PRs. We introduced owner's files to try to route things more intelligently to the right people. But it ended up that basically we would bottleneck on a few key people. It was very hard as these people, I would say have more and more people to get everyone onto the, at the time quarterly release schedule and get everyone synchronized. And that just became harder and harder as we added more cloud providers and generally more functionality. So a lot of what I would call people issues as well. Okay, why did we stop doing this? It enabled us to do a lot of architectural improvements. This way, things are not depending that much of a single person can be split up of the Kubernetes repo. And also projects that use various pieces of Kubernetes could just source those components instead of just having a big dependency of Kubernetes and conflict with other dependencies that don't upgrade at the same time. And fundamentally, those technical people and architectural reasons were solid, but really there was an underlying reason. That was sort of a philosophical reason about how we wanted to be as a project. We did not want to be a monorepo where we were like the centralized gatekeepers for this is cloud-native and this is not cloud-native and you're in, you're out, you're hot, you're not. We didn't want to be that. We wanted to have this industry. We wanted to have a large number of tools where anyone could come and collaborate and put their ideas in. And that is what effectively became the cloud-native ecosystem that we have today. And we had a bunch of technical strategies that we used to enable this idea. So basically the core idea was to make sure that you did not have to contribute to KK. So you could get anything done without having to contribute into KK, KK being Kubernetes, Kubernetes. That is largely the case today, but at the time, in the early days, it was not. And so things like CRDs enable you to add types without having to extend the API server. That was not always the case. Things like webhooks enabled it to change the behavior of any type. We added APIs where there were not extension points before for things like CRIs, CNI, CSI. So that is some of the tactics that we used to get to this point. With the release of Kubernetes 127, even cloud providers are out of the main tree. This is a huge success. This was the last to go. So now we have broken up the Monorepo. We have CSI, CRI, CNI, cloud providers, and probably a few other things. All are developed in different GitHub repos and by the people that are actually interested in those topics and maintaining them. And so here are some vanity stats that sort of reflect where we are. So in the Kubernetes organization, there are 77 repositories. In the Kubernetes SIGS organization, there are 161 repositories. There are 240 GitHub projects tagged with Kubernetes controller, so that's a good proxy for operators. There are 27,000 plus tagged with Kubernetes, and there are more than 47,000 repositories that consume Client Go. So if you think about where we were, where all that development was happening in that Kubernetes, that second entry in that left-hand column, so now where we have 48,000 repositories on GitHub, this is a huge success in terms of de-bottlenecking ourselves and enabling activity to happen outside of just that second line there. As Justin was saying, innovation can all happen outside of KK. And we have now Carpenter, tons of operators. We have worked separately from KK on IPv6 for some time. Even Docker was removed, and everything kept going with container D and cryo. Well, now we have Docker back also. And it's much easier to contribute to individual CSIC and I and cloud provider projects. The IPv6 story was quite interesting. Technically, it was supported by Kubernetes since I think version 1.4, but it only worked in specific setups without any cloud provider support. With some work being done on dual stack, like two, three years ago, many of the blockers were fixed and inconsistencies, and enabled external cloud providers to do their own thing around IPv6. Also, the release of IPv6 prefixes by the major cloud providers helped us add it to cloud provider AWS and now we're working on GCE, for example. And it's much, much easier because you just have to talk with a bunch of people from AWS or Google and that's it. If they say it's okay, you don't have to consult any other person. And of course, it is not all roses though. There are some thorns here. And so what have we lost? Well, I think the big thing that we've lost is that the experience for Kubernetes administrators, the people whose job it is to take those 47,000 pieces and put them back together, that experience is worse than when it was preassembled for them and tested for them in the KK repo. At least in Kubernetes, Kubernetes, we don't actually test these things together anymore. You have no guarantees and there's no one putting these things together for you. So a lot of people will go and you're almost compelled to rely on distribution now, whether it's a commercial distribution like GKE or EKS or AKS or whoever it is, or an open source distribution like KOps, you're very much compelled to, or it is much easier if you, you have to get your testing from somewhere else. And the problem of doing this yourself is now so much harder, right? There are all these components now which didn't used to exist and they're all developed independently on different schedules by groups that may or may not actually talk to each other that much. And it is a harder problem than it was before. It was never easy and now it's even like a bigger problem than it ever was. And that complexity is only magnified when you have upgrades and gradual upgrades and you have to worry about all these different release schedules and you are now responsible for putting those things all back together. So we have definitely lost something. Okay, should we put everything back together for testing to make it resemble more to a monorepo? Is this practical at this moment? Choosing version is very difficult and the complexity becomes exponential with the subsets of features and versions for each component. Keops does this, but so do other distros. So does every project that wants to run E2E tests. But all of us do it separately, duplicating various efforts and we do it better or worse. I'm gonna talk through a little sort of case study of one of the problems that arose here, which was the cloud provider for GCP. We had a bug that came up on the Keops E2E testing where a particular combination was failing. The combination was Keops and external, the external version of the cloud provider GCP in a particular networking mode. It's called IP alias networking mode and that combination was failing E2E tests and we didn't really know why. And the details aren't that important, but effectively what we, the manifest that we were using to install the cloud provider GCP, there's a little bit of that manifest over there on the right, right? Yep, was a little bit different from the manifest in the cloud provider GCP repo. And it actually turned out that both sides were broken. The manifest we were using was exercising functionality that was different from the functionality that they were testing in cloud provider GCP. That functionality was actually broken on the GCP side. Arguably they were testing the wrong functionality there, but there were things broken on both sides. And the problem is that we catch these things only after cloud provider GCP has been released. Typically, Kubernetes has been released, cloud provider GCP is then does their release and so on. This is really, really late in the day to be catching these sorts of problems. We would like to catch them much, much earlier. And so what we're working with specifically for cloud provider GCP and Kops, but we're going to be doing effectively on all the components and we would hope that other distributions to the same is this idea of sort of continuous testing with production manifests. So we're adding tests to cloud provider GCP. We've added tests to cloud provider GCP so that they continuously test with Kops and they're manifest. And they publish that manifest that they intend for us to consume. Kops will consume that manifest that they are continuously testing with. So we have some test coverage there. Other tooling, cluster API, Kube-ADM, whatever it may be, they can also do the same thing. They can, obviously they can consume the tested manifest that cloud provider GCP is now publishing, which is great and better than they are today. And if they also want to, they can add additional testing to cloud provider GCP so that cloud provider GCP can test with more sort of production environments and fewer of its own test scenarios. Okay, we are asking components to publish working manifests, not just container images. If all the Kubernetes components start testing their manifest in fully assembled distributions, Kops and others, then all the distributions can consume these manifests for testing purposes at least. We should catch all single source bugs in this way where bugs are introduced by the component. Combination bugs will be caught by the distros and are hopefully much rarer. Absolutely, yeah, so like a single source bug being a bug that's introduced in cloud provider GCP should now be caught basically when they introduce that PR. The cloud provider GCP PR will run against GCP with production tooling with Kops at some relatively recent version of Kops, getting back to where we were back in the days where everything was in KK, that test will run against real GCP infrastructure with real production scripting. And so hopefully if it's a single source versus multiple source, multiple single source, single source versus combination is sort of subtle, but the idea is if it's a bug that is specific to cloud provider GCP, that should hopefully get caught there. There are bugs that happen only when two things go wrong at once as it were, so you have a linker D in combination with some psyllium configuration, those sorts of things where they both need to happen, that'll only get caught later on by the distributions, but those are much rarer than the single source bugs. So components or projects should think about how distro do upgrades or how their customers do the upgrades, because if you release manifest, you have to take into consideration immutability. There are various fields that you cannot change with an upgrade like selectors or if you change the namespace, you might end up with awkward things. There's disruption. People don't really like to apply a new manifest and then their cluster be broken because you had to run some special command or something that was, you cannot express the other manifest, so it's not quite good. And the skew because the components don't necessarily think of their manifest as production ready. They tend to use simple versions of this manifest that the distributions have to enrich with various things like anti-affinity or, and in time, this gets to a certain level in version skew for distribution compared to what the components release. And in time, it's hard to keep track of all these smaller or bigger changes around the manifest. And besides all these smaller problems, there are non-cubernetes objects that cannot be expressed in manifest like IAM policies, firewall rules, TLS certificates, which should be somehow specified in a way that the distribution can keep track of them and can set it up so that the components works as expected. Not just have some, well, a bunch of documents somewhere, they say, hey, in this version, you have to set up this, this, and this. In time, they get lost. Yeah, the dream on this second one there is that we'd be able to install a CNI, for example, that we'd never seen before in a distribution tool, that the CNI would express the firewall ports that needed to be opened. So it's not a huge problem that there's a fairly small universe today. This is surmountable and that we can sort of hard code and we know, oh, you know, Celium needs this particular port to be opened. But just when we're thinking about the next 100x growth, how are we going to enable these things to move forward and grow another 100x, as it were? So these are the things we're gonna be working on. They aren't insurmountable. Another problem is that we don't actually have a great standard today for what a manifest should look like if it isn't static. And a lot of these manifests are not static. So a lot of the components need some per cluster information. So a lot of them need the cluster name or they need the cluster ciders, some of the ciders, service ciders, pod cider. And that information, there's no standard today for how we are going to plumb that through. Once again, I suspect we'll make do on a one by one, you know, one by one interaction basis, try to generalize those sorts of things. But these are some of the challenges we're gonna be working through with the components as distros. Another one is a lot of the components have what I classify as variants. So they have whole feature sets that are enabled or disabled in different combinations. And a canonical example is you can enable or disable encryption in Cillium, for example. That's okay if you have a small number of those features, but when you have a large number of features, you're potentially talking about two to the N manifests that we're now asking them to publish. So I think there's a interesting discussion there about how the components want to publish the feature sets that they want to be consumed. Obviously, we can publish multiple manifests if that number of feature sets is small enough. But if they genuinely believe they want to publish two to the N and N is big enough, that is gonna break down pretty fast. What about the Helm charts? It's very popular these days. Is this the answer? We think that not quite. It's close, but it's not perfect. Helm charts by default allow a large number of setting combinations. Because of that, you may end up with a generated manifest that is not quite what the component expected when the Helm chart was created or what they test with. There's no standards for the parameters structure, like the name of all of them. So you may have to read all of those or they will differ for similar components. And this is kind of like an API that has to remain stable across upgrades of the Helm charts. So our proposal is this. We ask the components to publish manifests in a way that they should be used in production. And we ask of them to test with this manifest, ideally with production tooling. Doesn't matter which production tooling, but not use internal things that they cannot be reproduced by other people easily. And in return for that, this is the sort of we ask and we promise contract that we are working with the components on. We, K-Ops and hopefully other distributions will join us in this, will help those components to test their manifests with production tooling like K-Ops. So we'll work with the components to add those E2E tests. We'd like for it to be K-Ops, but it's, we'd like for it to be K-Ops and cluster API and Qubit EM and a broad ecosystem of production tooling. What matters is that the manifests are tested in ways that are the same as are going to be run in production. And in return, we won't arbitrarily patch your manifest. So the manifest that you publish will be the one that goes into production. If we have to change it in some way, we will at least talk to you. This is sort of, this is not surprising, but this is somehow different from where we have landed recently where people aren't even aware of the fact that these manifests that are actually running in production are not the same things as the projects themselves are publishing and testing. And in short, so there's, we've come a long way. We are very close. I think we have a way to achieve some of the things that we may have lost when we broke them on a repo up. It's not the most technically challenging work, I think at this point. There are definitely some technical challenges as we talked about, but it's as much organizational and making sure that we go through and set up those testing jobs and make sure that the pipelines work so that we're consuming the same manifest that the components think we're consuming. And we believe that if we do this, that we actually are now gonna be able to get back to that reliable and easy Kubernetes experience while at the same time supporting this grand ecosystem that we have deliberately created and get back to where we were. And with that, I think we'll draw our talk to a close if there are any questions we can take them. And there is a QR code if you would like to leave feedback on this session. I don't know if we have any mics, but it's a small room so you can probably just shout if you have any questions. Thank you. At the back there, a question. This is the biggest goal. Yeah, I have a question because as far as I know, Kubernetes is the biggest goal, like open source repository. And I have a question. Did you encounter any problems regarding, for example, Go modules or sharing code between the repositories and how you were tackling it? Have you left any feedback to the Go team, et cetera? Basically you're, yeah. Thank you. Yes, it's a great question. So there's definitely been a lot of back and forth with the Go team about how Go modules work, how we should structure things. Anyone that has seen the staging sub-directory knows that the Go team, we're not the canonical example of how you should write well-structured Go. A lot of it has evolved over the years and as you say, it's because we have been one of the biggest Go projects and pushing the boundaries. That communication channel is open, I think it's healthier than ever. People like Tim Hawken, for example, I've done a lot of work on trying to use Go workspaces recently. And yes, there's a lot of feedback that gets fed back to the Go team in that regard. And historically, the relationship is healthier than ever. How about that? Daniel, do you want to read Daniel then, David? Thanks, so I mentioned a scenario where let's say there are 10 providers and one component, right? And so you change to the component manifest, gets tested against these 10 providers. And let's say today, one or two providers run the test and or however that works. And one of those tests fails. Is the burden on that provider or on the component? Now, let's say then a week later, all 10 of the providers have run the test and now it's failing for eight out of the 10 of them. Now, is it, you know, that question of the burden, where, yeah, do you have any thoughts on that? Yeah, I mean, I think the tests that run in the provider, in the components repos, we would expect for them to run on every PR and if there's a breakage there, that's where we'd expect the component to pick up on that. I feel like if we somehow see that something then fails in K-Ops when we pull the latest version, you know, we'll probably look at what's going on there first because our assumption will be that it's our fault somehow. I think we'll see what happens going forwards. There's at least some expectation then now that the code, the manifest that we're getting has worked at some stage, and we can now go back and look at those tests and having those two test results and the history of those test results will enable us to hopefully get some idea of where that problem may lie, but yeah, we'll figure it out sort of as we go. David, please, thank you. A similar vein, slightly more pointed. So I'm now a component, I support 10 different platforms and I'm trying to do the most good for the most people and one out of the 10 fails. Do you believe this should prevent a repo from saying, yeah, I'm real sorry, you're broken, but this is an important fix for the other nine? And as someone with a distro, we face the issue. Honestly, I think that should be up to you. I think when you take those, you're not obliged to accept the 10 E2E tests. I think it helps you to have more E2E tests and I think if you want to say, yes, we don't care about Google, then that's up to you, right? And I think it's a fair statement. Anyone should be allowed to make that statement and say that and make that choice and we're certainly not gonna stand in the way of that. I think one of the things that has happened today is that those statements are not explicit, right? So at least now we'll be able to say, yeah, we are deliberately doing this rather than I've accidentally broken like this thing over here and I have no idea really until six months later when you're like, oh yeah, I forgot about that entirely, which is sort of where we are today. So I think we're in a better place. I don't think anyone in this room wants to centralize and say, thou shalt not break other people, but I think if trying to give any good examples, if we had a secrets operator, right, that was maintained by Google and somehow it was breaking a lot on Azure for example, right? Presumably some pointed questions, even more pointed questions would be asked at that time. But I think we are, as you say, all trying to do the best we can and to date that sort of behavior has not arisen. So we're hopefully okay if we're able to make the choices deliberately, I think. Thanks, Sarah. Was there one more question there? And then maybe, I'll explain to the last one. This work on removing the cloud provider's code from the repo, I think it can be seen like a major paradigm shift cleanup in the repo. So I'm curious, can you think of any other like major cleanup efforts that's going on right now that is kind of like worthy of to take a look or maybe even contribute to? David, do you want to, I mean, I feel like a lot of it has happened, but yeah. We've had a lot of cleanup and a lot of spaces. I think if I were looking at like, what is this closest to? I would see this as comparable to the effort like when we introduced RBAC. We went everywhere and updated everyone and like Jordan did half of them, half of them by himself, right? And it was a lot of work. I think we're going to face something similar when we do sell admission where it's going to be, we want to replace these webhooks. Someone has to do all the work. So it's not the first time we've had something like this. This is one of the most costly though, but it would have a huge benefit. This would be awesome. Yeah, another one might be that there's going to be, I think more around maybe pulling some more stuff out of staging. I think like some of the work going into workspaces is maybe going to enable that. And so like maybe we'll split KubeCuttle out. You know, it's, I don't know, we'll see. I am not saying I'm not volunteering nor am I volunteering anyone to do that work. But you know, I think there will be, we're certainly unblocking a bunch of these things. And the sell work is, as David said, super exciting. I would like to add, like we are removing entry volume plugins in favor of CSI. It was a huge effort on storage site. Here we are, storage volumes, the next next. All right. Thank you very much. Thank you. Thank you.