 All right, there's still people coming in, but I think we're going to get started. This should be fun. Okay, so welcome to Advanced Container Tricks for Java developers. The for Java developers is very much in brackets. These are mostly some Advanced Container Tricks. I want to thank everyone here who's very brave and came to a talk that actually has advanced in the title. You were warned. If you came to the wrong talk, this is how to leave, which you can do at any time. I'm going to tell a small story now. I wasn't going to say this. I'm just going to tell you. I'm just going to tell you straight up. The way this talk happened was I looked through the list of tracks for the conference. I thought, what would be a good talk for the conference? When I saw Cloud Native Java, I thought, Container Tricks for Java. That would be an amazing talk. Basically, the tip I have is before you submit a talk called Container Tricks for Java, see if you know any, because, yeah, let's see what happens. My name is Dr. Jules. I'm an IBMer. I'm sort of the first dojoer, depending on whether you're willing to disqualify Colin Humphries for being too tall, which I am. I am the Project Lead for Garden, which is Cloud Foundry's Container Engine. I'm also now an award winner, since this morning, when I got this. It's pretty. So when the live demos mess up, remember this image. By the way, before this talk started, I realized that my e-key sometimes doesn't work. Let's get a new MacBook. This could be interesting as we do the live demos. So if the e-key becomes too annoying, we will be going to video, Advanced Container Tricks. So some Advanced Container Tricks for Java developers. What I'm really trying to do with this talk is talk about what a container is, give you some tricks and tips about what that actually looks like, the underlying stuff of containers, and a few tips that you might particularly want to know if you have the misfortune of being a Java developer. What is a container? I'm going to have to go really fast, because there's a lot of progress bars later on when you push things to Docker Hub and stuff. I have no idea how long those take on conference Wi-Fi. What is a container? In the beginning, there were JAR files, also known as ghetto containers. You shipped them around. The right ones you run anywhere. It was great, right? Java solved it. But, of course, they didn't quite solve it, and there are a few things we didn't like about that, like it didn't work. So we invented isolation. We want to have primitives at the lower level in the operating system that has achieved some of these goals of right ones for anywhere. So we said to Linus, can you give us some containers, give us some isolation and sharing, please? I know it wasn't Linus for the record. It's metaphorical. But he metaphorically said, here you go, and we got namespaces, which are a low-level feature that gives us isolation. The letters pretend that a process is an isolated world, doesn't see any other processes, has its own view of the file system, and so on. And C groups, C groups, let you share resources between your processes so that one process, which is probably a Bitcoin miner, can't take all of your CPU, next to your process is doing good stuff. We wrapped some of those low-level APIs into things called LXC. That became Warden. Warden became Garden. We invented this thing called CF Push. You could give us code. Create one of these Linus container things with the namespaces and C groups, and everything was good. Of course, it wasn't all good. Something happened next. A few people may know what happened next. There was a disturbance in the force called Docker. So what did Docker do that was interesting and that wasn't in what we had before? Because suddenly there was this whole idea of containers that wasn't the same as what we were doing when you did a CF Push, if we're honest. There was two things. It was encapsulation. Encapsulation as in I could take this whole operating system and move it around. That was this containers idea, this quote-marts containers idea that you may remember revolutionizing the world two years ago before we all forgot how great it was. So you could have a registry, you could move these things around, right? And there was something else that Docker did very successfully, no? There was something else that Docker did really successfully, it created a user interface around those containers. So you could directly work with them, so you could put them into registries, so that these low-level concepts, which we used to wrap inside CF Push because they were low-level primitives that we didn't think anyone should ever see, were actually so good that people could actually and wanted to use them, yay. So what is a container then? I think it's three things. The first two are the things that we had all along and that you get in CF by default and at that standard model, that isolation resource sharing, what we call Linux containers, right? And the second thing is what we call containers. This is this shippable encapsulated unit that we can move around. So Linux containers and containers. And we're now going to dive into Linux containers and to give you an idea of what that actually means and then we'll talk about container containers after that. So TODR of Linux containers is they don't exist. They're basically, and this is quite interesting, just processes with a little bit of magic. What does that magic look like? It's really these things. It's namespaces, which are the isolation, it's C groups, which are the resource sharing and it's pivot routes, which allows us to swap a file system into a container. So this is trick one. Containers are just processes and at this point I'm going to attempt to prove what I just said. Oh, that's already looking good. So I've got this application, ha, you can't see it. That's great. How do I mirror? No. I have to look at this screen. Yeah, it's only on this one. This is going to be fun. Mirror. Arrangement. You see this is teamwork. Yes. All right. Is that visible? That I wasn't encouraging you to clap, it's okay. The claps can come later. All right, so I have a little application. Cool. It's just like, you know, got an index, JSP, whatever. It's just a simple application. And I am going to CF push, boom. We all know what CF push looks like. You may not know what it looks like on a conference Wi-Fi, but we're about to find out, oh, it's quite slow. Cool. I guess I already had it running. Cool. So we downloaded all the build packs. We're creating the container. We're now going to run the process in the container, right? Cool. Do the CF push dance. In the meantime, I am going to go into the cell. So this is where the container is running. This is the container host. I'm logged in with Bosch. Cool. So I have an ops tab and a dev tab. The ops is the SSH into the host. The dev is my CF push. So my ops hat and my dev hat. Cool. And what we're going to do is we're going to CF SSH into my app. So I'm SSH-ing into that container that I created, which Cloud Foundry makes super easy. And I can do a PS in here. And I just see a few processes. Now, of course, the host is running on. I'm going to see loads of processes. Let's show what's happening, though. Sleep 1, 2, 3, 4, 5, or some other code. Secret code. OK. So if that's sleeping, and we see that's that process, right? So what's going on? You can see that process has a pit of some huge number here, 1, 3, 3, 0, 1, 4, 6. But in the other one, oh, it's just got this pit of 1, 3, 1. So containers are just processes. In the host, it's just a regular process. In the container, we've tricked it into thinking that it's in its own little machine and it can't see all the other stuff. So let's carry on and say, what was going on there? What was happening? How did that work? It worked by namespaces. So there's a set of namespaces in the kernel. The one we were really showing there was the PID namespace. And what that does is it has a mapping. So it has a mapping from PIDs in the outside world in the host into PIDs in the container. And if you're in that mapping, I can see you in the container. And if you're not, I can't. And that's all it really is, is a mapping table of those PIDs. And the other ones work quite similarly. Cool. Including this other one called pivotroot. For example, it's called pivotroot in the mountain namespace. So in the mountain namespace, you have your own view of the file system and all the mounts. So you have your own set of mounts in the container from in the host. And you can change what slash points do. So instead of slash pointing to where it normally points, which is the root of your disk, you can point it to a directory. And that's how a lot of the magic of containers is happening, right? We've got a busy box or a bundle, whatever we want the container to look like. We put it in its own mountain namespace. And we save for processes in that process and below. Just look at this instead. So trick two, how do we actually find container PIDs in the host and see what's going on between these two? And also, how do we access container files? So we're going to play a little bit. Oops. You just saw a preview. So we're going to play around a little bit. OK, so what I can do is I still have this sleep process running. We can see that it has the PID of 131. And in the host, we can see it has this enormous huge PID. How can I see what's going on? One thing I can do is I can say proc, then this enormous number. And so if I look inside proc, I will see that for all the different processes on the host, there's a subdirectory with that thing's process ID. So in this case, it's this one. This is really good. So far, I haven't had to use the e key. So this has been working out great. And inside that, I have a variety of files. This is a fake file system from the kernel that's going to tell me stuff about that process. Or in this case, it's really a container. So I can ask for propPID status. I'm going to get all of this nice stuff. I can actually grab this value NSPID. Did I misspell that? Cool. And what I see here is case sensitive. I see it's PIDs in all its different namespaces. So you see, that's its PID in the host, and this is its PID in the container. That's kind of cool. What else can I do? I can do this. So what if I'm inside the container and I want to figure out on the host what's this process? This is a nice little trick. So if you ask for procSelf, well, I should say, so procSelf, what is procSelf? ProcSelf is a link to my own PID. So my own PID in proc. So instead of having to know I'm proc224, I can say procSelf, so that's what I'm doing. So if I ask for procSelfCGroup, and we're going to see what a CGroup is later, it's going to give me this, and it turns out this thing at the end is my container ID. So this is the actual ID of the container. So it's a bit of an information leak, but it's a nice little trick to know that's the ID of your container. And so in the host, I can, for example, I mean, there's various things I could do. I'm just going to become privileged. This is a very secure system, as you can see. And I can use Run C. Now Run C is the tool that we're using to create containers. It's also the tool you'll find in Docker or Kubernetes at the bottom. Run C. On a cloud drive, you can just find it here. So if I say... Oops. If I say that, I'm now asking Run C, tell me about that container with that ID. And actually, I can ask Run C all sorts of things. Say Run C list. To tell me the containers that are running, there's only one running on this host, and you can see that that's its ID. But in this case, I've asked for its processes, and it will actually tell me, here is the process IDs of all the various things in the host. That's a nice way of getting from inside a container to a process in the host, to all this other information because you have the process. I think I said I was going to do another thing. What else did I say I was going to do? Access container files. This is a nice little trick. The files in the container look different from the host, obviously. So if I do ls var vcap... Is that, I think, home vcap? We have the files. So if I do that in the container, I see, yeah, home vcap app is better. I'll see those things. So if I look in home vcap app in a container, I see that stuff we pushed up. But of course, as you probably expect on the host, there is no home vcap app, right? Not surprising. The container's got its own view of the file system. So how do I see those files? There's an easy way and a hard way. One way is I can just look at the disk where that container is stored. So if I go to var vcap data, this is really a cloud foundry way of doing it. If I look at var vcap data, there's a directory called garden depot. And inside there, for each container, and that's our container ID, there's this bundle. And this is an OCI bundle. This is a common interface for describing containers. And I could look in there and find the container root files system, but we don't want to do that. Instead of that, there's this really nice thing to do. So if I grab this PID, what I can actually do is I can say proc PID root. And then inside there, if I go home vcap app, that's pretty cool, right? So I can just look inside the container's root file system, because it really is just a process. Actually, what we did with that pivot root is just change what that proc PID root pointed to for that process. And that's what Slash is mounted to. So now I can see inside the files and I can do all sorts of nefarious things to my Bitcoin minus. Let's go back. So that's namespaces. That's namespaces. Resort sharing. That's a pie. Resort sharing. Resort sharing is done by C-groups. In the same way that namespaces give you isolation, C-groups are going to give you sharing between different applications. So they're not going to hide stuff, and there's a ton of different C-groups. For example, there's various CPU C-groups, the devices C-groups, memory C-groups that all deal with different resources. If you're looking for them, they look like this. So you just have this directory structure that you can mount. And in this case, my C-group would be the C-group inside the CPU hierarchy, and there's just files in there that I can look at, see what the current values are, write to to change the values, which is pretty cool. So let's go and look at what that is, and then after that we'll see how we can use this from inside the container as well, which is pretty cool. So let's get started. We have no time. It's never going to work. So let's get started. So in the host I have this... So one nice thing you can do is if I do proc C-group, it's actually up there, what you can see here is the C-group path. So for example here we go. That is the path in the host to the C-group. So in the host, if I say CD and C-groups are generally put in here, sysfscgroup, that's one of each of the C-groups that we have. And let's pick memory. Memory is good. So inside here we've got all these files and these represent what your container is set up with for all of these shares. The ones that are particularly useful is we can look at limiting bytes, for example, and we can see that our container's memory limit is a gig. Do you like how I pretended that I just translated that into a gig? I did cfpush-1g earlier. That's a little tip. And if I want to change that, I can just let's take a whole byte of memory away from my... where am I? Yeah. Done, right? It's as easy as that. So if you want to fiddle with this, you can just set that stuff. And actually if you want to reproduce locally something that's going wrong on a remote system, C-groups are just a Linux feature. You don't actually need any of the container stuff to reproduce this. Just put it in a C-group. How do you put it in a C-group? There's a file in the C-group that says C-group.prox and that's a list of all the PIDs that are in this C-group. You may recognize these PIDs. These are the PIDs that were in RunCPS. It's the same ones that are in there, right? So all the things that are in the container are in that C-group. I can actually put more processes in there by just writing their PIDs into that file if you want to put another process into the same container. How do I use that inside the container? Well, it turns out at least in recent versions of Cloud Foundry you need something reasonably recent. Then if you look at sysfsc-group in the container what you will see is, so let's look in the memory one just the C-group for your container. So you don't get any of the other containers. You get directly this container. But you can look in that and you can say let's look for my memory usage. As a Java developer I suspect you care about memory usage. And I can see there's my memory limit. If I try to try to write give myself more memory as I hope you'll expect. If this lets me we have to open a CVE on Cloud Foundry. Good. Read-only files. So it's mounted read-only inside the container. You can't use this to directly do stuff in the container but it's very useful for reading what the real situation is. But you can get to it on the host if you want to do real stuff and you have root. So that's Linux containers. Encapsulation. Containers, what are they? Well, pivot root are registry and basically caching. That's the trick that's going on here. Vorey talks about pivot root. So let's talk about the other two. The way this works is layered file systems. So what you have is an image. In Cloud Foundry, this is really a droplet. The top layer, which is a tar file, is what you think about in Cloud Foundry as a droplet. The other layers are the root FS. But what's important is you've got a collection of these layers and you can swap them. So you can swap the top layer, swap the second layer but the other layers stay in cache. And we have file systems that we can use to combine those together straight away and then as you write to them it stores those off somewhere else. So they're copy and write file systems that let you combine these layers together and that's what lets you do this fast which is pretty good. That's what makes it shippable. It's that trick of taking this isolated namespace encapsulated directory and figuring out how to put it in a registry and put it down elsewhere quickly. So with the time I have which is not tons I would like to show you how to do some of this in Cloud Foundry. So in Cloud Foundry these are still just containers but you might want to operate them in Docker or in Kubernetes or you might want to run them locally and I wanted to show you how that works. So I'm going to show you all of those things really quickly if I can which is going to depend on the progress bar. So here we go. All right I don't need to be assesation anymore. There are stop jobs. I don't care. Good. So I'm over here. Now what I'm going to try and do is there's this amazing thing called CF local. I do it by doing CF install plugin CF local. I already did it so I'm not going to do it again. So I can say a few things. So one that's really cool is I can say CF local stage my app as a Docker image. What that will do is it will run the whole staging process in a local Docker container and it will be a droplet at the end. Let's not bother with that right now though. Instead of that let's do this a little bit more quickly and I will say CF local pull my app Docker. And what that's going to do is it's going to talk to my cloud foundry and pull down of course my app pull down the droplet from cloud foundry. So I'm going to get the droplet which is just really that top layer which is just my app's files and it's going to put that locally in a file called my app dot droplet eventually always in conference Wi-Fi great. All right in the meantime then let's happen to happen yes cool forget in the meantime so I now have this thing my app dot droplet so I now have the droplet version of the app and what I can do is I can say CF local export well I can do this yeah let's do it this way first. So I can say CF local export my app take that droplet and convert it into a docker image. So convert it into an image with the root FS and that droplet and tag it so name it in docker Jules me slash my app docker cool and now we have this image I've done this a few times and you can see there it is that's the image with that char it's in my list and actually if I wanted to I can actually just do this I can do CF push and then we have a flag dash O Jules my app docker right oh I should put that in a registry first you can't just do that because the Cloud Foundry can't talk to my machine so it's going to look for that Jules slash my app docker in a docker registry so I can do docker push cool and it's going to push that up so it's taking that up now you see most of those layers are hitting cache because most of those are already in docker hub so that's fine so I've just pushed up the root FS just that extra layer at the top the droplet and then what I could do is I could just do CF push dash O that which is going to take way too long because Cloud Foundry is going to have to pull it back down so that's not going to work but I could just do that right and we'll go a little bit way through so we'll just run it and it was just skip staging start running it don't care so that's how you can take a Cloud Foundry application, convert it into a docker image if I wanted to actually I can just run that locally so I can say docker run my app docker and I'm just running that same thing with docker so I've just got that in a docker container and now I can use any of those local things you can make it even easier you can say CF local run my app and it will do all of what I just did and give you a running thing that you can then interact with so this is all containers, this is all standards you can just use it it's pretty cool, I think that's kind of useful so what is a container it's isolation, it's fair sharing and it's encapsulation we've done a few tricks already I want to give you a few tips about using containers for Java applications in particular in Cloud Foundry so number one, memory in Cloud Foundry memory in Cloud Foundry, what happens when you cfpush-m64 megabytes it's just that, right? we're just echoing that number into the C group, that's all that's happening if you want to reproduce it locally if you want to mess around with it, that's all that's happening what happens if you go over and get killed, don't do that there's a small problem Java doesn't know anything about that so Java thinks it's got all the memory on the host and we'll try and use it and it will get killed, which is not great so first tip to Java don't look at Proc Mem Info which is what Java does because that will lie to you look at sysfsc group because that's real you can in CF memory calculator which will do the math for you, it knows what your memory limit is and it will do it and give you the right flags if you're on written enough versions of Java there are nice flags that are starting to come available to look at the right place because containers are great CPU so what happens when you do cfpush-m memory 64 megabytes well actually turns out we set the CPU as well turns out in Cloud Foundry there's not a separate CPU flag to set the memory and we're going to give you CPU in proportion to your memory limit so you want a big CPU limit you need a big memory limit that's by default and by default you'll get a share and what that means is if you've got, if you're the only application on the host, you get all the host because we're generous if there's two applications on the host it will be shared in proportion to how much memory each container has so if you've got a 64 megabyte container the 64 megabyte gets 66% the other gets 33% and the Bitcoin miner I keep running has 1% but there is this opt-in feature called experimental CPU maximums in Garden which you can turn on and that will set a maximum limit as well which is also just a C group value that we set in that case there's a problem Java doesn't know anything about that it's still looking at proxy CPU info and you will think that would be that big an issue but it turns out that it's going to try and create way more threads than it should for garbage collection and things and go quite wrong so tip memory size also sets CPU shares which is good, the fair sharing stuff is really difficult because it's really hard to know exactly how much you're going to get because it will change when other applications get deployed you can suddenly have half as much CPU as you had before even though the app didn't do anything wrong so think about setting maximums and it turns out recent JVMs, I think you need JDK pens have started to look at the C groups to get your CPU so it's worth looking at recent JVMs to get support for reading the correct CPU values and don't look at proxy CPU info in a container, it's a lie so I want to close with just a few final thoughts you may have heard of Kubernetes, has anyone heard of Kubernetes? yeah final tip, CF vs. Coop what's going on, state is not about code it's about states the difference is not about whether you push code or containers it's always containers if I've showed you anything this was all in cloud foundry it's all run C, it's containers either way the difference is whether you have a platform to make this apps or stateful apps and that means although cloud foundry is going to try and hide the containers for you and let you just focus on your code if you do want to interact with the containers, they're right there you can use CF local, you can use CF push you can use all the tools you might use with run C and in all the other hosts, so most things that you find about Docker or Coop should apply if you have to go and operate your cloud foundry so cloud foundry tries to let you focus on your code so you should never need to know anything about what I just told you but I still hope it helped thank you very much