 Ydyn nhw'n gweithio. I'm Jules, this is Will, Gareth, the responsible one and we're going to talk about OCI and BuildPacks. Okay, thanks. Let's start with a problem. I don't know how many of you have tried to do this, but trying to write a BuildPack for a natively compiled language, particularly one that does dynamic linking, is a bit of a pain. If you haven't tried it, I dare you. Go and have a go. It's not fun. An example, Haskell. Haskell's integer type is arbitrary precision. That means you need libgmp installed in the compilation container, and then the binary that you get is dynamically linked to that so that you need libgmp in your runtime container, and, well, libgmp isn't in the root of s, so it's not much fun. It kind of sucks. You can do it, and people have done it, and they might say, oh, it's not so bad, so you could pull down this library and move it into the right place and set some environment variables, and ydi, ydi, ydi, ydi. It's really a pain, and what's worse is it's really, really slow. So that's no good. Wouldn't it be nice? Jules' suggested title. Wouldn't it be nice? If you just wanted a simple BuildPack, you didn't have constraints about, like, network access or that, I mean, there are good reasons why you might not want to do this, but wouldn't it be nice if you could just do this? Install the compiler, compile the thing. Happy. There are reasons that this doesn't work today, so for one thing, AppGet needs to be run as root. We have unprivileged containers now, we could start doing that, we don't today. That's not such a problem, but that leads to another problem, which is that once you're running as root and you're running things like AppGet, anywhere on the file system, and those things your app might depend on at runtime, they're not going to make it into the droplet. The droplet today is just tarred up from this one particular directory, and so that's not going to work. So you really need to, instead of taking a droplet, you need to have a delta of the whole file system. So that's one problem. Let's talk about a separate problem. What if you want to move your app having staged it to a different Cloud Foundry without restaging? Or what if you want to move it to another platform or another orchestrator, or maybe you just want to run it on your local machine? That's a bit tricky to do at the moment. Why do you want to do this? There are a bunch of reasons, maybe you just have some weird requirements about where you run your workloads, maybe you just want to debug something locally, but the most compelling for me is actually this idea of running identical bits across your Cloud Foundries, across different platforms. Maybe you've just got multiple Cloud Foundries for availability, maybe you're doing some sort of workflow where you're pushing into a staging environment and you're validating, you're running some scaling tests or load tests, and then you want to move that thing on to your production Cloud Foundry, and you don't really want to restage it because then you have this question, did staging produce identical output in both places? And the answer at the moment is probably, it's not great. If you read the continuous delivery book, testing the same artifact all the way through, not rebuilding and moving it around. If you went to the Cappy talk yesterday about the V3 API, you might be thinking, well, what about droplets? We're going to get droplet download and droplet upload, and that will kind of solve this, and that's true, it does kind of solve this. But it only solves it for Cloud Foundry, it doesn't solve it for other platforms. These tables aren't kind of much of a standard, it's just this arbitrary table full of stuff that you can run, if I want to pull it down and run it on my local machine, I've got to untar it and figure it out, and I probably don't want to do that, I probably just want to run a container, something like that. Finally, there's not really an ecosystem or tooling around these tables, these droplets. If you want to do things like vulnerability scanning, you want to archive them in some registry, you kind of a little bit out on your own, there's not a good ecosystem of tooling to do that. So some more goals. How could this be better? Well, what if we exported compiled OCI images after staging? I should probably take a quick diversion and talk about what OCI images are. The OCI Open Container Initiative, it started a year and a half ago by Docker and CoreOS and folks like that, and along with the runtime spec from which we get Run C, there's an image specification which is very similar at the moment to Docker images. So you can sort of think of this as a Docker image, but the OCI part is important, it's an open standard, something we can all collaborate on, something that's going to be good for interoperability between Cloud Foundry, other platforms, everything else, and you get tooling and an ecosystem around that. So if we can export these things, and then we can import them again, we've solved our identical bits problem, you can move them around easily and you get all that tooling and stuff that I was talking about. And then there's a lovely bonus, we kind of think that by doing this we can make it all faster. So when you're staging and you're scaling or you're doing your rolling deployments, anything like that, it's going to be a bit quicker. That'd be pretty cool. So I'm going to hand over to Jules and he's going to tell you a bit about how this would work and what it would look like. This is a cartoonist impression that I've made here. So how do we actually make this work? How do we actually make this work? Basically what it comes down to is two things. It's layered file systems, which I'll talk about and it's a shippable image standard which I'll talk about. But first we need to talk about how it works today. So today you might be familiar with the current flow, you've got your code, push it, and this is the part we call staging. You then run a build pack that converts your code to something that can run. We store that as an object called a droplet and we run it. So if we look into that staging process this is where the build pack happens. Everyone's familiar with the build pack and actually the build pack is pretty great. The build pack bit is awesome. We've got this huge library of different ways of converting your code into something's well-running thing with much more complexity and support and all these things you can do with a build pack. The problem is actually not the build pack bit. The problem is this bit at the end where we dropletise it. What dropletising actually means is we tar up a directory called slash app which probably made more sense before we'd heard of containers and container images. But less so now. We tar that up, we move that out and then we have a tar file. What we do then is at run time we take whatever the current root file system is which is pretty great because it means we can patch it and keep it up to date. Then we just copy that tar file back in and we extract it and then we run your app from inside that run directory. It's okay and it works and it does mean that we can patch the root file system independently from the app. But now we've got a tar file we have to copy the tar file around and we have all the problems and restrictions we had earlier. You have to work around this system. So what could we do instead? Well we could first describe droplets all over again like this. We could describe the fact that droplets are just tar files. There's no metadata. You extract them again and again and again in Cloud Foundry which is actually why it sometimes takes quite a long time when you start your app because we have to start the container and then copy all the bits in and then on tar and then start things up and therefore you get this one directory at the end. What would be better is layered images which maybe people are familiar with you might have used them in such little known systems as Docker they've proved quite popular they use a layered file system so they use whole file system layers but diffs on top of each one to make that efficient and what you have when you have the combined diff is an image with an ID that can be stored in a registry and moved around and what's even better now is we have open standards for how to describe those images and open formats for those images and registries so that you can pull them down and share them. That looks like this you've got an image you can have a base there, let's say this is just an empty layer that starts off the image let's say we have a buntu, this is like a big layer with all the bits from a buntu you then might have go so you've got just the bits that you would add on to a buntu to put go on top so if I run app get go inside a buntu just those diffs over a base of buntu you get stored in that layer and then I might have my app so then when you want your app you actually have the same requirements you still need go, you still need a buntu you still need a base layer but now you want your app to be put on top of that so you still want to just save the deltas but you can share those layers which makes that really efficient we don't actually have to copy around this whole massive tar file we're just copying around the bits that are different and caching all the shared bits so if we think about that in terms of the staging process what we could do is we take your code we run the build pack and instead of tearing up a directory we just export an image so what we do is we just take the deltas that's really what would have been the contents of the tar file and we store them as an extra image on top of the others at the end well it might not be there but I'll give you a couple of slides alright so you now got an image how do we actually run that well now all we do to run the image is we ask Cloud Foundry hey please run this image cos actually we already have support for running Docker images in Cloud Foundry and OCI images are basically the same format right now so we know how to do that so actually all we have to do is make staging export that OCI image and we have all the rest of the machinery because right now in Cloud Foundry you have two different mechanisms for running things you've got running an OCI image and you've got running a droplet based app but actually if you think about it all we have to do is get rid of that one and now you've got two ways two very good ways of converting your code into an image you can use a Docker file has a lot of restrictions doesn't have the big library of things that buildpacks have et cetera but it's a really nice way of doing it we can't patch your router fast for you afterwards if you go down that path but it works right that's a valid approach alternatively you can use CF push and you can push your code we will stage it we will run the build pack against it an OCI image in a registry which you can now run any way you'd like in this Cloud Foundry in another Cloud Foundry in a different orchestrator entirely if you'd like and either way you've got an image and you run it so there's no tearing you've got OCI images all the way through the stack simplifies everything as well as being faster and about you not being able to manage what router fasts you have beneath it which is pretty cool actually so incase there's a Dr Nick in the front row about to predict a problem are there any problems with this I think you should write down your problem and then see if we've covered it done love team never ask a witness if you don't know answer will be I'm worried now so the first problem I suspect people will ask is what about patching the second problem I think people might ask is what about existing build packs that might be working in the current way and the third thing is probably what about patching with maybe more exclamation marks let's talk about patching I think that's a real big thing and that's why we always talk about we need the build pack flow because of patching the router fast this is why you have a problem with patching it's because if you bake this all into one image then when you have a CVE security floor in one of the layers you've now got a problem you don't have to restage all your layers which is a big problem and now you don't have immutability in what you're going to do and the way that build pack based apps solve this is really just by ignoring it what they do is they say I'll be fine and they take a tar file and they just change the router fast underneath it and put the tar file back actually we can do that so all you need to do let's say this one's got a problem in it we need to patch it so that image had an ID you create the IDs by combining the layer IDs together that's how these IDs get created so everything's immutable we could create another we'll just patch that layer here we regenerate that and we move the layer back on top if you actually think about it that's literally just untarring that's untarring in a format it's just back we just created a new image we haven't actually done anything magic all we've actually done is created a new image and shared the layers so if you think about when you create an app normally your new Golang app all you've actually done is said well this shares all the layers with this one and I've got a new layer on top yeah why not it's just a rebase right I mean yes after a half a decade it works to build packs so why not just so there's actually all this is actually doing is untarring and what we've actually done is taking that tarfun and reapplied it and then we've just used a layered file system to do it instead of untarring it every time but it's the same exact operation that we've done and now you've got a new image and what's nice now is instead of relying on Bosch to push out the new rootFS to all of yourselves then there's a sort of side effect all the droplets will get re-extracted to these new rootFS and back up you can now manage this in the platform so now you've got a new ID corresponding to this image and we can actually just use Diego to roll that image out and we can roll it out properly know which versions are running where is your staging environment running a different version than your production environment et cetera if you want to lock it to a particular version that wasn't patched and run that locally that's got an ID and it's immutable so you can debug the problem that only happens in one version while we roll back the rootFS for example and that you just run as a container that's just a container image it's just an OCI image you can use a standard container ecosystem tools to run it it's just an image with a little bit of magic because we can rebase these images what would that mean? it would mean that writing build packs could potentially be as easy to write a lot of the time these aren't great and build packs have a lot of advantages but at the moment as soon as you get to one of these situations where you're restricted by build packs people go around the platform and what they do is they actually just go create a dockerfile create an immutable image that they run once with an app get and then use the platform to push it and now we can't patch it we can't deal with it it's lost all the benefits of the build pack flow it's gone out in the system because we made it hard so given that people are already doing this we can make it something that we can actually help them do and keep them within the system and that way we produce OCI images and we run that OCI images and things are faster and they're simpler and they're maybe a bit better as well just as a gift we've left lots of time for questions because I think there'll be lots of questions because Dr Nick is here two questions I should say that we have a document that describes this in a lot more detail and this is quite high level so there are some things that we're not diving into in the slides about how this works that we're very happy to go into if there's not enough time here but catch us later if you have any questions burning hello so to not give up build packs would be super valuable I don't think anyone wants a build pack v2 like just give up build packs so currently I just assume all build packs assume if they write something to the slash app it'll stay if you write somewhere else it'll go bosh compile pack is having the same idea we can relax some of the restrictions like slash app existing build packs might be relying on some of those restrictions so for example they might during staging write to some file that they expect won't survive into the running image and there's nothing fundamental about this that requires so what you can do is when you save that delta layer you can just save the stuff in slash app at most 50 build packs but really is probably 20 and really 75 right right exactly like in practice the build packs team can actually opt in to some of this stuff and take advantage of it and people who want to write their own build pack say a haskell build pack can opt in to some of this stuff at the moment if you've got a haskell app what you end up doing is writing a dock of fun and not writing a build pack which is awful so this way we can say you can opt in because you can say what root FS you would like your staging to run in so it doesn't have to be the standard one that way you can use a minimal root FS you don't have all the packages that we have to maintain in the current root FS in case anyone needs them so you have a small minimal package which is actually better from a squeeze point of view easier to manage and you can put whatever you like in that and then you opt in while you're doing that to saying actually we're going to run staging in the new way and save the whole thing so you have to be willing to deal with that fact bonus question because I thought of that question and I thought of Bosch and we're always looking at ways to just reimagine what Bosch could be can we take this idea and Bosch jobs being containers and containers yeah yeah this is it's a total topic and it's a totally valid topic I've maybe said similar things and had similar ideas I decided we should raise the temperature of the water one degree before draining the ocean but yes you totally could I think Bosch has amazing ideas in it but some of the pieces of Bosch were clearly designed before we'd seen containers like images and that's the same about this it was developed in the same building as Warden that Warden did not really exist in it I think a lot of these had the fundamental exact correct ideas but then other new things have happened to the side and we can actually pull some of the ideas back in and make quite big advances so one area where it was sort of designed before containers and still has lots of advantages over shippable containers but now it's suffering for some of the restrictions and you can kind of combine them and get both is this and I think another is it is a bit like whenever you first train someone with Bosch you'll find a writer package where they do an app get in it and then get super confused when the stem cell rolls and their thing is lost and nothing works any longer right and there's actually no reason for that having everything in Varvik app packages happens primarily we didn't have containers so that's how you contained things if you compiled the package made an immutable thing and then extracted it in a known directory structure and replaced it that was containers early if we actually used containers in there we could probably make big advances CR Talk at CF Summit 2017 and the idea of just moving the image across a tree is open up a lot of these ideas and not having to replace everything all the time and we're all the people ways to not have to have Bosch touch everything for 3A right, there can be some big caching advantages just on its own to some of this stuff a little bit more about how the image refacing might work so in particular one of the nice things is when you roll that just restart the app with this you have to do something to generate that new image whatever it is there's some interaction so how do you make that not terrible so basically there's two answers to this so first answer is Diego so we actually know within Diego how to roll things out which we do all the time that's how you push your new code out a staged artifact is a package hash in CF terms so we already really kind of do this with the droplet when your app changes, when you've restaged a new droplet we roll the new droplet out that's really the same process the second question is how you create the new app create the new image it's basically just git, it's a rebase an image is a set of layers with a hash where the ID of the hole is calculated by hashing the combination of all the IDs so you keep having a unique ID all we're doing is doing a rebase in git terms and you can find people who've done proof of concept tools that do rebases with existing docker images if you want to rebase a docker image those exist, they're currently kind of hacky but there are lots of them as Dr Nick says there's only like a small number of build packs in practice and they already go to a huge amount of trouble to work around this stuff in their current model if you see what I mean to make this stuff work so this isn't a huge burden for them to watch out for the minor edge cases if you rebase there are some minor edge cases if you rebase like if two layers have done the same thing then a rebase can have a conflict that's a problem but actually most of the time in actual practice I don't think you hit that very often if you isolate this whole build tool so people can build pack teams can play with it and see how it goes over and over again right, I think one of the other nice things about it is it takes a lot of the magic out of the platform potentially because instead of having this sort of push process it's really quite opaque although v3 is making some of that better you now have a staging process that produces an image and a running process that can choose that image and actually you can use any tooling to create these images potentially so this thing that's shippable and understandable in itself which is a lot less magic and kind of exists outside the platform as a shared thing that gets worked on by the platform which I think will have a lot of advantages over time for like how we can make things more understandable and simple What would the user experience be like for CVE has been discovered we now need to update our images how would you see that working because presumably at the moment it will be a new Bosch deploy and there will be a new root of s on the VMs would that workflow change or would you see a different interface for how people treat it rebasing of images so the build packs team would use a slightly different interface because it wouldn't be a Bosch deploy anymore in the build packs team we probably could make it a Bosch deploy you could automate it so that when the new stem cell rolls out we do that with admin build packs when the admin build packs you do a Bosch release it sort of updates them in the cloud controller right so it's actually if you think about it it's very close to admin build packs actually so admin build packs really they used to be just deployed on the cell with Bosch we just Bosch deploy them and they sit next to the containers but then we wanted people to be able to have an API for changing what admin build packs were there dynamically so we added an API for doing that and then we just tell the cells here are the build packs you have with the staging request and they download them it's really the same thing, we add a new admin API that said cf update build pack and then give it the new base image for that build pack and anything that's running on that we just rebase inside there's just an in memory operation and now it's got a new hash and it rolls out so there's some machinery that you have to build around that but there's quite a lot of advantages to it I think Does this imply some sort of registry that CelfCounter is not going to be running and how do you all manage to provision that? Yeah so you don't have to it could be using some external thing but I think it makes a lot of sense to have an integrated registry for droplets we have an integrated blob store this is really kind of an extension of that and especially at the moment if you think about the docker flow there's a big weakness that we haven't figured out quite how to solve in the docker flow which is if you push the docker app at staging time and you give us the privileges to pull that app at staging time at running time it's not clear how long we should hold on to that username and password because it doesn't fit into the Cloud Foundry UAA model what you really want is the permissions on that image in that registry to be CF permissions to be organ space permissions so then if you push anyone in your space can run and download the staged OCI image from the app et cetera so that's why I would see it being part of the platform properly integrated with the UAA and with this rebasing content that could have been either there was a concurrency issue left so you keep talking about just rebasing the application bits and still in your example you're using Golang so probably to be honest the thing about Golang is you don't actually need the container Golang is a terrible example because in practice it's a static binary so just run it Golang is like why we should never have invented containers just ship Golang around imagine I said Ruby makes a lot more sense nowadays we're building lots of Golang applications because it's cool so I said probably you can even include the additional staging steps in this rebase rebase the original source code then restage then move to new layer but in this case you have to create the additional layer first quite a few yeah I mean so once we're using images throughout we actually have a lot of options for doing this sort of thing right actually once we're just taking OCI images and running an OCI image and they're two separate steps rather than being integrated if you want to go and run a separate CI thing that's doing it differently that would integrate into the system in a very clean and kind of obvious way so it opens up more of that stuff we've kind of got this kind of very fixed model right now and I would see the first step as we kind of open up that we implement the existing model on top of this flow and that then maybe opens up some possibilities so all the things the build pack so if you think about a build pack what it does is it detects first to see if it should run but then the main things are one it compiles your source code into something that can run and two is it writes out some metadata which describes how to run it like what start command should be run basically right and all of those things can be described in an image spec so when I say all we do is we take the layer diff and now we've created an image that's not quite true we take the diff and that's how we create the root part of the image the metadata we get by running bin detect, bin release rather so we just use the build pack spec to say what's the start command and we put that in the image you can do some interesting things down this line so what you could do one pattern you see quite a lot in the docker ecosystem is inside the registry inside the github repository they'll have a docker file which is the docker file to run their app there's nothing that stops you having a build pack file which says what build pack you should run to run this app and maybe even locks a version you could lock the exact build pack to run your app with and the platform could detect that do the staging for you and run it for example so there's some interesting things you could do along those lines and then that build pack file could say what start command it wanted for example so there's some options that kind of open up some of this is very blurry there you could kind of do some interesting things down that path there's a separation between these and runs that does not have a difference so we're still running staging and staging is producing an OCI image time as we said I had a feeling we would run out of time with some questions left catchers like we'd love to talk about this we're blurring a lot of details if you have input and edges we've not considered then definitely talk to us disclaimer, no commitment that this happens wouldn't it be nice