 Okay, so hello, my name is Julian Friedman. I work for IBM on the garden project. I'm super excited to be here. This is really exciting. It's awesome to see so many familiar faces. I work out of the UK most of the time so it's always nice to see people who I normally just see on video screens. I'm going to talk a bit today about garden, which is trial foundry's container technology. Anyone who is at ONSY's talk, which is awesome, will already know a bit about containerisation, why we use containerisation, what it is, what Diego's looking to get out of it. So I'm not going to talk too much about that and I'm just going to apologise if there's some missing context about containers. Today I'm going to try and answer three questions. So question one, what are containers? What are we trying to achieve by containerisation? Question two, what is garden? What is docker? What are the differences and why do we use garden rather than docker? And then question three, can we use docker instead of garden? The spoiler is yes, I'm going to demo doing that. We have a back end for garden called docker. I'm going to show that working. Then I'm going to ruin the one more thing. The one more thing is should we replace docker with garden with docker? That's a much harder question. I'm going to give a little bit of my view, but most I'm just going to give you ammunition for the discussion that I think is there. I'm going to give some facts about that. So containers. Has anyone heard of containers? Have they come up at all, containers? Yes, so containers are pretty cool. Containers have got a lot of attention, right? But what actually are they? Well, the more interesting question isn't what are they, it's what we're using them for. What are containers for? I think there's about four things. First one is this idea of packaging. That's the docker use case. That's this idea that we can have reproducible images that we're going to ship over. Same image works here, as works here, and so on. The second one is reproducibility. It plays into packaging. I'm going to run my thing locally. I'm going to run it remotely. It's going to work in the same way, and I'm going to know how that occurs. Thirdly, super important, isolation security. If I'm running lots of containers on a particular machine, I want to make sure that that's secure, that each one is isolated, so I can run lots of applications so they don't interfere with each other. And there's loads more in Onzi's talk. I'm sure that will be available on the interwebs. And the fourth one is cheap ability, cheapness, something. We can run low. We get the density of them. The other thing we're trying to achieve in density. And the thing to notice is some of these bits are more important for some people than other bits. For the docker use case, what's really important is this awesome packaging workflow, three-producibility workflow. That's a developer experience. That's me on my laptop. That's what I care about. If you're building a pass, that's a multi-tenant platform environment, what I really care about is I can do this in a secure way so that I can run the images and be confident that I can run them in production, scale them, not have people, getting other people's data and so on, and do that at a good cost. And inevitably, there will always be trade-offs when you're building systems. And the question is, how do you make these trade-offs? So garden is very much built to try and emphasise the later things. But it turns out that we can actually enable the docker use cases as well. So there's some obvious follow-on questions. How hard is creating a container? What are we actually talking about here? Well, it turns out not actually that hard for certain values of hard. It's actually most of the stuff is there in the kernel. The kernel gives you, as Anzi said, isolation. The kernel gives you namespacing. So a lot of what a container technology really does is provide an API to actually use those in a consumable way. The kernel gives you the primitives. And most of what docker or garden or rocket or any of the other things are doing is providing a way of using those capabilities. So the garden way of doing that is to provide a programming API for those capabilities. Because really gardens use cases to power platforms as a service and things like that. So we have an API-driven approach. Garden doesn't have a CLI, for example. Garden has an internal API, but that's not what we treat as our external interface. We treat our external interface as the rich client. The bigger question is how much work is creating a production-ready container? Something that we can actually trust to run our applications. So using these things together, that's easy. But finding the right, finding ways to run that securely so you trust it and scalably, that's pretty hard. And even more, finding the right abstractions, how you expose this to users in a way that makes sense, is actually really Diego's problem, not garden's problem. That is super hard. Those are the actual challenges. The bit around how you actually run a container is not that much stuff. So I feel like I kind of have to put in a history lesson, even though it doesn't really matter. I'm now going to go, we were there first, basically. Warden actually predates the other two. So the obvious question is why did you build Warden when Docker was out there? The answer is Docker wasn't out there, so we built Warden. And then later Docker came out. We've now rewritten Warden. We've written a lot of components in Go. That's why it's called Garden. It's actually mostly the same code base underneath. And really, it isn't really that important, right? That this is where we are now. But it is worth saying, and you might have heard of Docker, it is worth saying that there is a difference in the use case between these two. So if we compare Docker versus Garden, the original use cases of these two, Garden was always built to run lots of containers in production in a multi-tenant environment. And actually when it was first built, his main use case was running one root FS. So each container is basically going to run the same root FS with lots of applications on top. Whereas Docker's use case was originally less about the multi-tenant environment, I think it's fair to say, but also about each container might have a completely different root FS, managing of layering and images and so forth. And there's a pretty big difference, right, between when you're trying to run a single container on a laptop, which is a hard problem, versus loads of containers and trying to orchestrate those and run those on a big system. But you have to notice, right, these use cases are coming together, right. It's pretty clear that Lattice is very interested in providing a single machine developer experience now and can really do that. And Docker is increasingly interested in how we build big systems. So these two things are coming together. So give me a high-level view of Garden, right. What does Garden look like? What's the Garden abstraction? So number one, Garden is a VM model, right. So Garden containers look like VMs. Inside a Garden container, it seems like you're running a little VM. And that's actually super useful, because we have projects like Bosch Lite, which people might have used, which enables you to simulate running a whole cloud foundry environment on a virtual machine. And that uses warden containers underneath to spin up what look like lots of virtual machines that look like a production system, right. It's also used for the build pack applications. So for build pack applications, we want to be able to throw in that compiled build pack into a container, run it, run health checks inside it, and so on, and make them appear like a VM, right. So to do that, in Garden, we have an init processor. We run PID 1, right. So in a Docker container, the user's application is the only application normally in the container, and it is the first PID. In Garden, we have the first PID, and it spawns all the other applications. And that can be used to reap child processes, and so on, so that it looks like a real environment. And Garden is built to run a pass, right. It's also a multi-platform. So Garden is an abstraction that you can build multiple backends for, right. So we have a Garden Linux back-end, which I work on. There's also a Garden Windows back-end. And the reason that works is because it's quite a well-chosen abstraction. It's quite a high-level abstraction. And it runs, it was built to run build pack applications, but it also runs Docker images. So we actually support Docker images. We use Docker's code to pull the images down and then just run them inside a Garden container. So it's kind of the best of all worlds. Give me a high-level view of Docker. Has anyone heard of Docker? So Docker, obviously an app-based model, right. It doesn't stop you running multiple processes within it, but its main model is an app-based model. One app per container. And in general, the container is packaged the root file system of the application. So a Docker image doesn't separate, here's the image, and now run this in it. The metadata includes what should I run it, who should I run it as, what's its home directory, et cetera, et cetera, right. It's quite a tightly coupled thing, which is really great for a user, right. I've got like my thing and I know how to run it. It runs the same everywhere, but it is quite tightly coupled. And it uses a layer file system to make it efficient to run whatever root files you give us, right. And because that's a big use case, which is really great for local development. So that's really great for local development. Obviously, if you manage that in a pass, it's actually pretty difficult, right, because if you think about build pack applications, they're all going to share the same root FS, which we can cache, right. As we use layered file systems to cache it, and then each build pack is just a little bit on top, right. Whereas in the Docker case, all it takes is for someone to push, instead of pushing something based on a Ubuntu, they push something based on Red Hat or anything else. And we have to pull down the bits for that file system because we don't have the base layer, right, the shared base layer. Someone can keep doing that. And now we've got lots of disk space being used and having to manage all those layers, having to patch all that stuff, right. It's a much harder problem for a multi-tenant environment. So what's so great about Garden? Well, it's got a really great abstraction, right. So because we've got quite a high level, simple abstraction, you can plug in multiple backends. Therefore, we've got a Windows backend. Therefore, we're able to do this experimentation. Secondly, it's got a Go client. The Go client is the first class citizen. It's built for building containers with, right. If you're building a container, what you want is to be able to have a bit of Go code that says, create me this, run this, right, especially if your platform's built on Go. Freedom to experiment because we can put in these multiple backends and plugable backends, right. I've said plugable backends a lot of times. Why do I keep saying plugable backends? Well, we already have Linux and Windows. And if we're talking about experimentation, then you could actually run a few other ones, right. For example, obviously Docker, right. So we've got a great abstraction. It's got a Go LAN client and it's plugable. Maybe we can take advantage of both of these. But I'll quickly talk about what's not so great to be fair, right. There's some technical debt. Garden is one of the early container technologies and it's done good service. But there's a good deal of technical debt in the code which we're having to fix. One of the big problems is it really wasn't built for this use case. So it's not built considering how do you securely run untrusted images, which is the use case that's now much more prevalent with dockerized containerized applications, right. So that's a new use case for us and securing that is pretty difficult, right. In the old world, we just ran you as the VCAP user and we never let you have roots. And now people want to run their root FS, right. So their Linux image, which we can't secure and we can't see first as roots on our machines. That's a super hard problem and it's a big change, right. So we're having to figure out how to change garden to do that. God, I love clickers. Yeah. It also has a much smaller community around it. We just have to be honest about this, right. Garden's great but it's a pretty small community of users versus docker, et cetera, right. And that has some big effects, right. It means for example that we support a couple of layered file system drivers, whereas docker supports a huge number of layered file system drivers, operating systems, et cetera. And it's less what I know. So what's so great about docker? Well, it is an amazing user experience. It's great. It's a package of applications up. There's this huge benefit of, you know, everyone uses this, right. Docker doesn't have really a standard, right. Standard is the wrong word, but it's great that everyone's running the same sorts of images and knows what to expect. It has this huge community around it and it's pretty robust, full featured, and very, very well known, right. There's a name value to it. What's not so great, well, for us, it's, I don't say this is a knock against docker, right. The tight coupling is great from a user experience point of view. But if you want to build a pass on top of it, what you really want is to be able to say to it, use this root FS, run this in it. And actually with docker, I tend to say run this root FS and then it will figure out what that root FS wants to run in it and how. And that's quite different from what you want to build something on top of, right. Because it's not like a, it's an opinionated abstraction and you really want a non-opinionated abstraction when you're building things on top of it and you want to have the freedom to experiment and change things around. The API is the standard, right. That is a problem when you want to, for example, run the same thing on windows, experiment with different things. When there isn't a standard that you can expect, that's still a problem for us. And it has a one app, the container model, which is great when you're trying to run docker apps. When you're trying to run, for example, Bosch Lite, which wants to represent real machines and be used for, like, that broader use case, that's a lot more difficult. So that brings us to part two, building a docker back end for garden. So how do we combine the advantages of garden with the huge community and familiarity of docker? Well, we write a docker back end for garden, right. So, plugable back ends, let's write a docker back end for garden. Why? Well, because we can, right, reason one. Because we want to know what the gaps really are, right. So actually, garden does most of what we need, right. It's working great right now, but there is a huge community around docker. There's lots of available resources for it. If we can take advantage of that, that would be awesome, right. That would be great to not rebuild the wheel, right. And we'd have three goals, right. We want to maintain the freedom of choice of back end container technologies, right. So one of the great things about garden is we can swap in multiple back ends. You can have a windows back end. We'd like to be able to have rocket app container, et cetera. If that later becomes available, if that becomes the right choice, it's great to keep that open, right. Well, we can do that, right. This is just a plugable back end for garden. We write it as a back end for garden. We're not getting rid of the garden API. We can do that. Secondly, we want to be able to take advantage of the ecosystem around docker. While we run it as a back end for garden, we get all the advantages of the docker code in the back end. That would be awesome. And finally, we want to make sure it works in a multi-tenant pass environment. This is harder, right. This we don't get for free by writing it as a back end. But we get the same APIs and we probably have to build some stuff on top of docker. So we probably have to run our own stuff to add the additional things for multi-tenancy. But those are things that we should push to get back into the community where we find those gaps, right. That's what working in a local source community means. So this is possible, right. So how is it going to work, right. Well, basically, we're going to start our own demon as process one, right. We need to do that because we want to have this VM-like model, right. And we don't want the user to be able to run as application one. And we want our own thing if we want to support that model. So we're just going to run our own one. And we'll tell docker that the application it should run is not the user's application, it's our container demon. And that runs as the in-it process. But actually in the back end, it's just going to shut out directly to docker because it's fast. So we just run the docker executable to run things. It has a couple of problems which I'll talk about. It gets it working pretty quickly. And then we actually just reuse lots of gondolinix libraries. So gondolinix has libraries for doing things like setting up the networking, the way that we like the networking setup, and for spawning off process so that we can reattach them later and doing all these things. And actually all the primitives at the bottom are pretty much the same, right. All kernel resources. So we just reuse those modules around the docker container. So we get the docker container to create the container with our process in it and then use our existing code to actually spawn stuff into it. And it's actually surprisingly easy. This didn't take very long to get working, which is because Gondolin is actually a pretty simple API, which tells you a lot about the management garden. Here's some pictures. This is how you spawn a garden container. This is what happens in garden, right. So a garden container, you have something called wooshdee, which creates a container, and it actually clones a copy of itself into the container which runs as the first process. That process keeps a little socket on the outside world. So it's got a socket in the outside world which not in the container. It spawns stuff into its container. We've still got a socket outside and we ask it to do stuff. And that's how garden works today. When you actually want to run a process, you've got a little thing called woosh, which is the warden shell, and it says, hey woosh, tell wooshdee to start a container over this socket. And then woosh starts a container, so that's the process inside the container, and it forwards the input back, send these to a connected. So you then got the input to an output. Standard input error just gets connected between them. It looks like that. So that's how garden containers work. We are now going to dockerize it, prepare to dockerize. Boom. And this is all that you have to do. So number one, change the name. Number two, that's now called initdee, got rid of the thing. Number three, we use a docker to spawn it instead of cloning it in, but same code, just run it inside it. Rename those to Dosh because it's not worth doing if you can't have a funny name for at least one component. And that's basically it. So the Dosh talks to initdee over the socket outside, runs the processes, great. So let's have a quick demo. I was going to do this demo live, but then I chickened out. My Wi-Fi didn't work this morning, and I was like, nah, this isn't a good plan. But if anyone would like to see this demo live on my laptop later with the code, then buy me a coffee. Do that. So this is, this is Lattis. People have not seen Lattis yet. You've got to check it out. It's amazing. It's basically a Diego deployment in a vagrant virtual machine just running. And what I've done is I've changed the Lattis deploy script so that instead of installing garden linux, it installs garden docker. But this is installed a vagrant up, and it's now running a docker demon inside vagrant with garden docker, the garden docker back end rather than garden linux back end. And I'm going to target it. So I target the, so this is my Lattis CLI, right? And I think for a while. Okay. And then, so we now like look at the logs, right? So now all the components are coming up, right? This takes a little while when you first start Lattis. So all the components have to find each other and start talking to each other. So that takes a little bit of time, but eventually they'll come up, and then we're going to create an application, right? So we're going to create, and it's going to create itself from a docker image, right? Now normally in garden linux, what happens is we use a docker code to pull down the docker image, and then we start up in a garden container. But now, and great, it's up, but now what we're going to do is use docker to do the whole thing. So we're going to say to docker, just run this, and it's going to download the root file system and get it up and running. So we say create, it thinks, create in the container. Now, this is the first time, this is a fresh VM, so there's no cache. So it's going to take a little bit of time to get the container. So it's now putting down the bits from docker hub for the docker root FS. It takes around to the second T, so the T in container, until it pulls down for the first time. But really all that's happening now is we're putting this down over the interweb. And, should say boom in a second, and yeah, that. So house check passed. So we spawn our house check, excuse me, into it, exited, and it is running. We take the URL and we forwarded the ports. It's got the proper networking, proper port forwarding, and great lattice. And just to prove that this is actually really fast when you're able to download bits from docker hub, we'll just do it again. And boom, great. Got a second container, right? Super, super fast. And actually all going through docker, right? So, great. Yeah, proof. So let's SSH in, and what we should see is if we SSH in, firstly, if we list the process, what we'll see is a docker demon is now running. As you can actually see what this looks like. Scroll up a bit, we'll see docker demon up here. Yep. And then you see our garden docker back end. And these processes are called initDs. These are actually the init process in the containers that this has spawned, right? But the really interesting thing is, so firstly, we create little directories for each container on the file system. So we're going to just cd to one of those directories. And the way the DOSH CLI works is it looks at what directory it's in to know what container to go to by default. And then you tell it what to spawn inside the container. So we're going to say DOSH PS. Sorry, I should say docker PS. And you can see the containers are listed in docker, right? So they're managed by garden, created by garden, but they appear just perfectly normally in the docker PS list, just like every other container. We go in here and I say DOSH PSAF, right? And we've spawned PS into the container with DOSH, right? Which is the docker shell for garden. But I can also just use docker exact and spawn a process. So I can use all my existing docker tooling and that, yeah, that works great as well. So that is garden docker. Now, so what doesn't work? User namespacing doesn't work. User namespacing is one of the ways that we can run multi-tenant applications properly. Unfortunately, docker doesn't support this yet, right? This is one of the examples of where this is a feature that garden has prioritised because we need to run these in multi-tenant environments. We don't have that feature in docker yet, which is kind of a biggie, right? That's actually kind of a problem, to be honest. Snatch at restore, again, one of the great architectural features of garden is that if garden needs to be updated or if the process goes down, that's fine. All the containers keep running and we reattach to them. That doesn't work unfortunately with docker. You've got this long running docker demon, which is a problem, et cetera, right? There are problems, right? So, part three, because I'm running low on time, was this a good idea, right? Well, it was fun. So, there was that. It does work, right? This actually works really nicely and you get to use all your docker tools, runs for docker ps, et cetera. So maybe, right? We have added this whole extra layer, right? We've got this whole extra layer of complexity, you know? We've got to manage docker and manage the docker demon, do all that stuff. And actually, the user experience, if you saw what I did the latter's push, is kind of identical, right? These primitives are actually, the kernel primitives are pretty much the same in either case. So, yeah. But it's nice to know that we could do it now, right? We didn't think we could do it a few months ago and now we're pretty sure we can do it. So, it is all about the use case, right? Containers are really great but it's all about what we presented to the user. The actual kernel primitives to isolate stuff are pretty similar between all these things. And some of these, you know, build packs have a lot of advantages for a pass, right? So, we do want to support both of these use cases. We don't want to lock into one use case or another. And we want to keep this plugability, right? We want to keep me out to experiment with all these different container technologies. Garden was why it was so easy to pull in a garden docker back-end. That's an awesome feature. And even with leveraging the docker technology or anything else, we still need this garden component because there's still all these things that we build around the outside to actually manage the containers and provide this abstraction. So, I recommend I have about one minute for a Q&A, maybe two. I have about two potential questions before I have to call it. So, the question was, this was useful and fun and worth doing. Thank you. Will this actually become like a full thing that the product will do? I don't know, right? I think this is an open discussion as to where this will go, right? At the moment, garden in its works, it has some features like user mapping, user namespaces that we actually really need in production. Snapshot Restore, which is a great feature for us. So, it's possible, but it's really a kind of a product question, right? Engineering-wise, we can do it. Product-wise, is this something where the benefits of introducing this extra layer actually win us enough to make it worth doing and spending that effort on? I hope so, yes. You can look it on my laptop later, but once I get approval for my manager in the front row, I'll put it on GitHub. Is the plugability enforced or implemented through the plug-ins and the CLI plug-ins, mostly? So, how is the plugability implemented? So, the way it works is that garden has two components. It has a server, which provides the API and the rich clients, and you can actually put in just as there's an interface to provide a back-end, and we have a garden linus one, a garden windows one. This is a garden docker one. It's actually quite easy to plug that in. It's quite a small interface to run the containers. Yeah, so, it turns out not that many because garden is quite a small API, so actually garden is more a subset than the other way around. So, we try and limit that abstraction, right? We don't want to have lots and lots of features, but what is more different is some of the trade-offs that are more about running a pass, right? So, for example, when containers, when we have to grade garden linus, the containers can keep running, right? We don't have a long-running demon. We could just reattach the containers, whereas with docker, you end up with a long-running demon process, and I don't think there are plans to move away from that yet. So, yeah, good because that makes a lot of sense for that use case, but for a pass use case, it's really important to be able to, like, keep your containers running even when you're doing roll-in deployments, which you do a lot. So, I think that's the end of my time, but if you'd like to hear more about this topic, there's a panel, Bill Pax Containers for Cloud Foundry, where I suspect the merits of some of these approaches may come up. So, it will be awesome to see that. Thank you and have a great night.