 I'm Giuseppe. I work for Pivotal. I'm Yulia and I work for SAP. We're here to talk to you about how garden has grown in the past few months. So first of all, what is garden? Garden is the containers engine for CloudHoundry, Conqor CI, and Bosch Lite. So you might be wondering, what is a container? A container is effectively a way to distribute applications. In the industry, we've been trying to solve this problem for quite a long time. I think the reason containers are so successful is that they provide two things. They provide isolation. So basically, you know that your software will be isolated from anything else that runs on the same machine and also encapsulation. So all the dependencies that your software requires can be shipped with the software itself. How do we do this? The Linux kernel has been providing primitives to achieve containerization for quite a long time. In particular, the Linux kernel provides namespaces and it provides C-groups. Namespaces basically allow us to limit what a process on the system can see, and C-groups can allow us to limit what the process can do on the system. In CloudHoundry, we've been using this primitive for quite a long time. We used to have this Container Engine called Warden, which was written in Ruby. Then we rewrote it in Go and we renamed it to Garden. It was all good and great, and then one day Docker happened. It was very disruptive for the whole containers community, because they were not using anything different than us. They were using exactly the same primitives as Garden. But what they did was they provided a very good user experience for them. So suddenly everyone was able to just take containers, ship them around, manage them very easily. It made containers very popular, and Docker itself became very popular. To the point where it became this de-factor standard for containers, and at some point we at CloudHoundry were like, maybe we should use Docker as well, and we did try. The problem is Docker hasn't been conceived to be reusable. So it's very opinionated, it's quite big, and it's monolithic. It's not very modular. So we did try and we basically failed. Luckily, some time passes and the Open Containers Initiative gets created then. What the Open Containers Initiative did was basically, first of all, establish a standard format for containers, which was really standard, and they also released a reference implementation for it, which is called Run-C. Run-C is exactly the opposite of what Docker is. It's a tiny binary, it takes a config file which complies to the spec, it produces our container, that's all it does. Incredibly reusable, small. So that's what we did. Basically, we decided, okay, we're going to get rid of all our containerization code and just replace it with calls to Run-C. This marks what we call the beginning of what we call the year of glue, where we basically, garden is not stopped being a complete containerization solution, and instead it's an obstruction on top of an existing containerization tool, in this case Run-C. Does this mean that garden is useless? Not really. Garden still makes a lot of sense as an easy to use obstruction on top of this standard containerization technology, which allows the consumers to just be stable, they don't have to change when the underlying containerization technology changes. Also, it allows us to provide our own security defaults, and in garden we try to turn on as many security features as possible by default. This is an example of a few that we do turn on by default, and this is like a comparison with other container products on the market, and how we tend to be a little bit more keen in enabling security features by default. Now over to you, Lea, to talk about what is actually new. Yeah. Now, thanks for just being on what garden is, but how did our garden actually grow in the last few months? We've been actively working on three main tracks, and those are CPU metrics, Continuity and Rootless, and I'm quickly going to go through the first one, which is CPU metrics. Cloud Foundry already has this functionality, when you do CF app and the application name, so you get some numbers about CPU, and you're right, we have that, but the current approach currently has a few problems. The main problems are three, the current metrics are a bit confusing. The other thing is that applications can actually affect the performance of other applications, which is not perfect, and the third thing is that life is super hard for operators, and I'll quickly go through the different problems. So first one, the confusing metric. So let's say you have an application, and you push that application to Cloud Foundry, and when you get the information about the CPU, you see, okay, this application is currently using 40 percent CPU, then you push the exact same application that's consuming the exact amount, same amount of CPU, and then all of a sudden you see, boom, 200 percent. So what's wrong here? So the current approach has this problem where actually it depends on a few factors that are not visible by the user. For example, on how big is the Diego sale, or what kind of other applications are running, or the same Diego sale, or if other applications land on it, that will affect the current approach. So yeah, when you get this information, you don't actually gain any information because you have no way of knowing how much CPU your application is currently using, or how much it actually needs, or when you need to actually scale. So users don't have the answer to the different questions like how is my app performing right now? Is my app going to perform well on prod? Is it going to perform well if it lands or a different sale, or even if it continues to run on the same sale? And it's impossible for them to autoscale on the information that they're currently getting. So the second problem is wrong direction, sorry. The second problem is that if you have high CPU applications and by high CPU applications, I mean applications that are constantly using high amounts of CPU, that could negatively affect normal applications. So let's say you have a Diego sale and you have 10 applications of the same size running on that Diego sale. So in theory, every application gets around one-tenth of the CPU, so that's their fair share. And in theory, all of them should be able to spike over that fair share when they need to, so you'll be allowed to use more than you're entitled to. But what happens if all of the applications on that Diego sale are consuming as much CPU as they could possibly get? So that means that you'll never be able to spike over that threshold, and that could lead to really poor user experience. And the other problem is that both the applications, so the one that it's constantly using high amounts of CPU and the one that's idle most of the time, they pay the same amount of money, which doesn't sound fair. In order to fix this, operators have to do something. And so they have to build this perfect world where every application is able to use their spike CPU, so all of the CPU that they need. And that's hard. So in theory, that should be super easy, because you depend on the fact that not all of the applications are using 100% of their CPU all of the time. But in practice, if you have those high CPU applications running on that Diego sale, that won't be possible. So what operators normally do is they just simply over commit, so they give enough CPU for the sale, so all of the applications can use their spike CPU like 100% of the time, and that's a lot and someone needs to pay for that. We try to solve that problem by introducing a maximum fair CPU. So what that is, you have this maximum CPU shares that you get and you can spike over that and that's relative to the memory that you're getting. But so the CPU shares, but that's a bit hard because you have to find this perfect value where it does actually something because if the value is too high, that means that all of the applications can spike a lot over their entitlement and that means that you're not doing anything. If it's too low, that means that all of the applications won't be able to spike over their entitlement and that causes problem because what problem or it doesn't give you any solution. Okay, so those are the problems. So what do we wanna achieve? The first thing that we wanna achieve is we wanna make the CPU metric understandable. So that means that when the users see a number, for example, 100%, that should mean for them, okay, right now you're using all the CPU that you're entitled to. If they see 105, that should mean, okay, now you're using a little bit more than you should be, but that's okay because there is some spare CPU in that machine. But if you depend on the fact that you will always get this CPU, then you should consider auto scaling your application because you're not guaranteed to always get this. The second thing is bad applications don't affect good applications and good applications shouldn't get a decreased performance from what they used to get until now. And the third thing is make life a bit easier for operators. So that means that they should feel really confident that they provision enough CPU for all of the applications to spike when they need to without having to provision as much as I described a few minutes ago. And that sounds really good, but how do we plan on doing this? So the thing that we did is we introduced a thing called CPU entitlement for each container. So that means this is the CPU that you should be normally getting. We actually implemented this already and there is a new metric in Cloud Foundry. We didn't change the old one. We exposed a new one that can be consumed through a plugin called CPU entitlement plugin. So the reason we didn't change the old behavior we introduced a new one is because the behavior is like really different. So the problem with the new... So if you have an application which is using 25% of the host CPU, the old plugin will always show you 25%. Doesn't really matter how much CPU you're entitled to use. But the old plugin, the behavior will be a bit different because if you're entitled to 25% and you're actually consuming 25%, you'll get 100% in our plugin. If you're consuming 25% but you're entitled to 50, then you'll see 50%. So this entitlement that I was talking about, it's currently relative to the memory as it used to be. There is a mapping that can be configurable by the user about the operators which is a Bosch property. And we exposed that property. We tried to write a really good doc describing how operators should set up that property because it's not that trivial because operators need to know information like how big the Diego sale is and stuff like that. And we also put a same default which means that if you wanna have a perfect, so an optimal system where all of the applications are guaranteed to get at least their entitlement, they will be able to do that. So we split the actual implementation in three phases. So the first one was the meaningful CPU metrics. The second one was the operations tooling. So at this phase, no throttling will actually happen. So we're only going to create the tools that operators can use to see how the system is currently behaving. So they should mark applications as under or over entitled. So operators at the end should be able to give us some feedback if our algorithms work as expected. So if you're marking the over and under entitled applications as such, the actual entitlement throttling will happen on the third step. And there are a few key points here. So no matter how you behaved in the past, you are always guaranteed to get at least your entitlement. So even if you behave like really bad in the last months or so hours or the last rolling period, you'll still be able to get at least your entitlement after that throttling will actually happen when you have multiple applications that are trying to spike over their entitlement and there is not enough CPU for all of them. So the thing is that the preference and bursting above the entitlement will be given to applications that and containers that behaved good in the past. So the thing is that you gain the ability to burst by being under entitled and you lose it by being over entitled. So for example, if you have two applications that are entitled to use 5% of the CPU and one of them was using 10% in the last few hours and the other one was like being idle most of the time, then you give privilege to the good container to burst. The next track that we were working on is container D. So as Giuseppe mentioned, garden wants to be like glue between different components and we just want to be a super thin layer around everything and we want this layer to get even thinner and thinner and container D was a nice opportunity to do that because both we and they, we use RunC for container creation and life cycle. And we also have this layer wrapper around this RunC which is common between us. So at some point we decided, okay, why don't we just get rid of this layer and just use that logic from container D because that is a repository that a lot of people are using. The community is actively contributing to it, testing everything. And we also get the nice opportunity to contribute something back. So if we found some bugs and stuff like that, we can just bring that back to the community. So the way, because of the way things are implemented right now, that wasn't hard and we were able to do this change incrementally. So for example, container life cycle operations like container creation instruction are currently happening through container D in CF deployment by default. The other step was process operation. So when you run a process in a container and this is currently experimental but it's already implemented. And the third step is to get rid of our image plugin. So the image plugin is the component which actually takes an image, for example, Docker and turns that into the root of S of your container. And we have this component which is called root of S and now we would be able in the future to just totally get rid of it and start using the snapshot or logic from container D. And now Giuseppe will tell us a bit about root list. Okay, so we've talked about security earlier and how we are enabling all these security features for every container that we create in Garden. And this is true. So our containers are actually quite solid. But as soon as you get to a very secure door I think you should start worrying about the wall because that becomes the next more likely target for attack. And in our case, the wall is called Garden server. So the Garden server is actually still running as root which means if an attacker manages to get control of the Garden server they will have basically unlimited power on the system and we don't want that to happen. So we've been putting in lots of work into being able to run Garden as a non-privileged user and this would mean that even if an attacker got control of the server they will still be a non-privileged user on the system will still have not much power to make too much damage. This is actually done and it's currently under test. The way we've done it is explained in detail into this other talk which is called the root to root list and quite a few of our team members have delivered this talk in all sorts of conferences. So I encourage you to go and check it out. Please just pronounce the root to root list. Never say the route to root list because people back in London will get very upset if you do that. So as soon as we got this ability of running as an unprivileged user another interesting opportunity was well we can run Garden into its own container and it can be a normal unprivileged container and this is exactly what we're doing. We're leveraging BPM which is this new process manager for Bosch which effectively runs every Bosch workload in a container which is by the way like it uses RunC under the hood and so if you deploy Garden with Bosch you can use BPM and run Garden into its own container. Of course as Julia mentioned we are working on a container D backend so we do want both these things so rule list mode and BPM for container D as well so we're working on that as well. And finally working on all these amazing features it just happens that we got stumble on all sorts of weird behaviors of the kernel or the operative system. Things don't work, just don't know why so you spend maybe weeks investigating and then you find out very interesting stuff about how Linux works. So we decided to share this. We've published a couple of blog posts and we definitely intend to do more of that in the future. So to recap, we've seen how Garden is effectively and easy to use and secure body for the abstraction on top of well-established standard container technology. We've seen how we're fixing the way we measure CPU for containers and the way we allocate it as well. We've seen how our glue is getting thinner and thinner through container D and we see how we're getting more and more secure by running rulelessly. And finally how we're also trying to share what we're learning through our blog. Oh yeah, this is the attribution for all the emojis you've seen in the presentation. They're from this project called Streamline UX. And that's it, thank you. Thank you very much. Mia and Yulia are gonna be hanging around if anybody has any questions. Thanks. Thank you.