 My name is Tyler, and this is Steven. And today, we are talking about BuildPacks. So what are BuildPacks? I think some of us are familiar with this scenario. Developer creates an application, runs a CF push. Maybe this is a Ruby app. Our BuildPack is going to run, install Ruby, install a version of Ruby, and we're going to get a running application on Cloud Foundry. Pretty simple. Well, we are on the BuildPacks team. We love BuildPacks. We love this model. There are a lot of reasons that we really love this. First of all, BuildPacks are an opinionated build. This means we get this really nice, consistent application environment structure across the organization. Our apps look the same. Because of this nice consistency, security is a really primary concern as a part of the BuildPack model. This model allows us to easily roll and update our applications quickly. And we also don't feel limited by this model. We still have flexibility. We can still use all of our favorite languages, tools, frameworks. And of course, we're always trying to help the operators, empower operators, and allow developers to develop. So thinking about this, let's take a look at an alternative to BuildPacks. A really popular alternative in the application build is the Dockerfile model. There are a lot of good things about Dockerfiles. Dockerfiles are really flexible. It's really easy to add dependencies, change up your application image. These Dockerfiles create container images. There's a lot of great things about those. Container images come with immutable container layers. So we know that the code that we are testing against is the same code that we are deploying. It's also a straightforward model, pretty easy to pick up. And we also can easily tell what's going inside these Dockerfiles. We're writing it out line by line. But it's not all good. Some of the issues with this model comes up when specifically when we're creating app images and also a lot in the enterprise world. Because we're creating app images, the dependencies in our environments are locked behind other Dockerfiles because we are using Dockerfiles on top of Dockerfiles and layers on top of layers. So if we want to patch a dependency at the bottom layer, we need to rebuild layers on top of those. Dockerfiles are so flexible that it's pretty hard to create a consistent structure across the organization for these apps. And because of that inconsistency, it is harder to keep those secure and update all those dependencies. Again, because of that inconsistency, it's harder for operators to control the contents of all the apps across the organization and therefore harder for them to audit those apps. So let's take a look at what this might look like. Here is a pretty typical Dockerfile application, based application. We're going to be starting at the OS layer at the bottom where we are installing any OS level packages, maybe OpenSSL, maybe like MySQL connector if we want to use that. And then our next layer is going to install Node.js. Let's say we're using a Node app here. We're going to install Node.js. We're going to pull from that previous Dockerfile. And finally, on the top layer, we're going to install our app, run MPM install. We have our running application, a nice Node app. Well, what happens when we give the developers a little bit of flexibility here? Let's say the developers are all managing their own application images. So everyone is going to choose their own base image for their app. They're going to choose their favorite operating system. They're going to pick the packages that they want. And then maybe everyone is installing their Node layer as well, differently as well. Maybe they're installing different versions of Node. And finally, they're installing their apps on top of that. So we've got a lot of divergence here already. But what happens when we look org-wide? We've got lots of applications running. Well, now we have chaos. And we have lots of different applications on top of Node layers, on top of OS packages. Well, what happens when an OS level CVE comes in? OpenSSL has a critical CVE. We need to patch this right away. Well, in this scenario, an operator could rebuild that OS image layer. But it's more likely that that operator is going to wait for the downstream package to get updated. And it'll pull that one in. So after it updates the OS layer, it's going to have to update each individual Node.js layer. And finally, the developer can rebuild their apps on top of those layers, test all of those, redeploy them. And this scenario may take something maybe like months, but in reality, it may never get patched across the org. Maybe we're not doing it exactly this inconsistently. Maybe we're doing Docker files a little bit more the right way, where we have a consistent organization image for the OS layer and the Node.js layer. We are using good practice and making sure all teams are using the same OS, the same version of Node, and they're building their apps on top of that. Well, CVE comes in, operators are going to wait for downstream to build that one OS-level image, rebuild the Node.js image, and then all the apps on top of that. This scenario is better, but still not great. Maybe this gets rolled, depending on how long these builds take and the testing and the redeploying. Maybe this takes a few days, maybe longer. So let's take a look at the BuildPack model. On Cloud Foundry, we start with our staging container, which contains BuildPack. We are going to stream in our application, our developer application. The BuildPack is going to inspect the app, figure out what dependencies it needs. Using this Node.js example, we're going to install Node. We're going to run MPM install, get the MPM packages. We're going to tar up those layers into what's called a droplet. Export that out of the staging container. And next, we are going to provision the desired number of running containers. And those launch containers are going to contain packages supplied by the platform rootFS. The droplet is going to get streamed into the launch containers. The platform will invoke the star command, and we have our running apps. So we can easily tell that with this BuildPack, we're able to take care of the inconsistencies that arise from the Dockerfile model, take care of the Node.js level, take care of the operating system level. So, sorry. So Cloud Foundry has a really interesting way of dealing with vulnerabilities when they occur in operating system packages. And it ties into the infrastructure for how Cloud Foundry runs on top of VMs. So this is a Diego cell. And you can see it has multiple apps on it. They could be different droplets. They could be the same. And it has operating system packages for the containers, for the stack, which is like the container rootFS. And it has operating system packages for the VM to support the Diego components themselves as they run. So when a vulnerability hits, you have a whole bunch of Diego cells, and they have some vulnerability in them in the operating system layers. And to patch this, the platform one by one starts a new Diego cell with duplicate apps from one of the old outdated Diego cells that has new, where the new cell has new operating system packages, both for the stack, the container rootFS, and the stem cell, the VM. And then it takes the old cell down. And so it runs through each cell and brings new cells up and takes old cells down until the whole platform is patched. And this is really nice in Cloud Foundry because it lets us take advantage of the way the infrastructure looks in order to patch security vulnerabilities in operating system packages sort of live in production for a very large platform in a couple of hours. It's sort of very consistent. And of course, we only do this for the operating system packages that have that sort of ABI compatibility contract and only with operating system packages that are patched by upstream vendors in sort of safe ways. So we've never had a report of this causing problems where app behavior changes in an undesirable way. So that's a pretty exciting advantage of BuildPacks that people don't think about too much. So again, as Tyler said, we like BuildPacks because they're opinionated, they're secure. You have nice security benefits. They provide a lot of different languages. They provide a lot of operator control, which is really important. They make developers' lives easier. But there are problems too. So this sort of droplet model involves a lot of data transfer that's probably not necessary. We build droplets, and then we transfer the same droplets to the Diego VMs many times but when we don't really need to necessarily. We rebuild a lot of stuff every time, even if we don't need to. And that sort of simple BuildPack model where we're building a brand new droplet on each rebuild. And there's some caching, but it's pretty inefficient. It's sort of a simple form of caching we have right now. So we've been working on a new BuildPack contract along with Heroku. It's sort of a collaborative engineering effort where we have contributions from Pivotal and Heroku. It's called Cloud-Native BuildPacks. It actually just entered the CNCF last week. It's a really exciting thing. This new contract has some really key changes over the previous one. It's sort of a complete breakaway from what we had before. It creates portable OCI images. It uses separate build and run images. So you can have compilers and stuff on one image, but not have that in your final container. The detection process gets a lot more advanced. So detection involves looking through the application and figuring out what dependencies need to get supplied and creating a plan for the build ahead of time, which has a lot of benefits. It also serves as a build of materials for the container afterwards. It has BuildPack groups that allow for multi-detection. So we can pick a bunch of different BuildPacks that apply to your app without you having to think about that ahead of time. And we can break the BuildPacks down into really small pieces that just do instead of a Node BuildPack, we're going to have a NPM BuildPack, and a Yarn BuildPack, and a Node BuildPack, and then individual BuildPacks for different partners or extensions that you might want to your app. And I think most importantly, this leverages some really new features of the OCI image format and new features of Docker registries to really efficiently update just individual layers on a registry that need to get updated. We don't have to rebuild lower layers in order to rebuild higher layers with OCI, which is something not many people know about. And I'll talk more about those a little bit later in the presentation. So let's take a look at some of these changes a little bit more in depth. So if you remember the droplet that we were creating during staging before, we've gotten rid of that. We're using OCI images. There's a lot of really nice benefits to this. Right off the bat, we get a layer digest for each of the different layers. And this is contractually associated with the contents of each layer. So we can easily audit our applications, make sure things haven't changed. Not only that, those OCI images are very portable. We can run these images on all over the place, a lot of different places. It really makes it nice to abstract that. So let's take a look at the actual change of the implementation of BuildPacks. So on the left, you'll see the old BuildPackvent interface. This is how we used to develop BuildPacks. We would have four separate executables. These would all run in order. And we've reduced that down to two. We think that we don't need this extra complication. And we hope that this simplification makes it easier for people to write their own custom BuildPacks. Not only that, but another interesting part is that previously the detection script was generally a pretty minimal step that would return 0 or 1 depending on whether the BuildPacks should be used for the application. Well, now the detection is going to have a little bit more responsibility. It's going to reach into that app and figure out a build plan for the subsequent steps to run based on those contents of the app. And we'll see that come up more. So because of this change, we are able to create a new way of doing multiple BuildPacks. So we have this new concept for multiple BuildPacks called a BuildPack group. And this is going to be how a lot of BuildPacks are kind of thought of in the new API. So let's take an example of a Ruby and Node.js app. Pretty common scenario. So we've got our BuildPack group, which we have defined as a Ruby BuildPack and a Node BuildPack. We have our application code. First thing that's going to happen during staging is the detection script is going to inspect the application code. It's going to say, OK, great. This is a Ruby app. You should use this BuildPack. I'm going to return 0. I'm also going to contribute a BuildPlan. I think we need to contribute a Ruby version. And maybe I might provide metadata along with that for the BuildSteps as well. We're going to move on now to the next BuildPack in the group. This BuildPack says, yes, this is also a Node.js BuildPack, our application. So we are going to return 0. And I'm also going to contribute to the same BuildPlan and let you know what versions of Node I also want to contribute to this app environment. So after that, we're going to go back to the Ruby BuildPack, which is going to run its BuildStep. It's going to read from this BuildPlan. It's going to create a Ruby layer because it knows it needs a version of Ruby. It's going to run bundle install. The tech script probably told it to do that. And then we're going to move over to the Node.js BuildPack. It's going to read from the BuildPlan during build, create the Node.js layer, run MPM install, get some Node modules. So one of the things that's really nice about this change is that we're finding that making these BuildPacks really modular is a really powerful thing because we are able to reduce the dependencies that we need only to the applications that really need them. We're able to reduce a bit of resource usage there. So in this example, we thought initially that the Node.js BuildPack would be just a single BuildPack, but really we're finding it's better implemented as separate BuildPacks where the Node.js BuildPack provides just Node and the MPM BuildPack, its detect script, tells the build process whether it should actually install MPM modules. So you could imagine that there is an application that only needed Node, didn't need modules, and we were able to moduleize like this. A really good example of this is the Java BuildPack, and the Java BuildPack team is working on splitting the Java BuildPack into a lot of separate BuildPacks. Thanks. So I want to go into a little more detail about how this works and specifically how we make rebuilds really efficient using the new model. So I think the key idea behind it is that we only rebuild and upload layers when it's necessary, and then includes uploading layers all the way to the edge. So we just update, rebuild a few layers that we need to locally or on your CI system or on a platform, and then those individual layers get transferred through Docker registries and Docker daemons all the way out to the cell or whatever you could run on Kubernetes, Kubernetes deployment out at the very end. So to do this, we use some interesting features of the OCI image specification. So the OCI image specification breaks layers. It's sort of a departure from the Docker v1 image specification where now instead of linked layers that point to each other, you have individual content addressable layers for the file system, and you can apply those in an order, but it's not like get anymore. They're not linked together. So we can rebuild those layers, whatever layers we want to replace, and just update the individual ones in the image and just send those up to the registry. So another feature we take advantage of is a new feature in modern Docker registries and the sort of Docker v2 registry where it's a feature called cross repository blob mounting. So we can have one image repo that's a source of the root FS for lots of other image repos. And then without any data transfer, or without the negligible amount of data transfer, we can point 1,000 image repos at a new base image just by going across each image and making a small metadata change. As this would let you update, this would take the Cloud Foundry Model where we have the cells and we're rolling them and replay that on top of the Docker registry to get the same benefits. That's one of the coolest things about this, I think. So the result of using these two strategies together is that we get really fast builds, a really minimal data transfer, and we can do the sort of layer rebasing thing directly on the registry. And so I'll kind of go through and compare these two ways of doing things a little more technically. So before we had just the supply and finalize build step, and so you have your application and it runs supply finalize, does everything and generates a droplet. And when you want to rebuild again, yes, there's a small cache it has access to that where it can recover some of the things that downloaded last time, but it's basically gonna do all the same things and generate a new droplet and transfer all those same droplet bits out to the edge. Where in the new model, we take those two steps, that sort of detect step, and that supply and finalize step, and we combine that supply and finalize step into one, and then we introduce two more steps that the platform is responsible for called analysis and export. And so detect selects the build packs that should run and figures out what dependencies the application needs, analysis sort of grabs information about the previous image. Build uses that information to determine what layers to rebuild and export sends those layers up to the registry, and I'll talk about that more in a little bit. So to kind of break this down really granularly, in the analyze step, we pull information from the image configuration, which is a special sort of easy to access section in a Docker image in a registry, and we write that into toml files that each build pack can read. And so you'll see in the build step they can modify those toml files, and those toml files represent the contents of the image layers. So in the first step here, we pull them from the remote image. In the build step, the build packs read those toml files to determine the contents of the existing layers and decide which layers to regenerate, and they just regenerate them by creating a directory with the new contents in the file system. It's very simple. And they also still have access to a small transparent cache. It has a little bit fewer guarantees than the cache in Cloud Foundry right now. It may go away, but it can also be used to cache things that you need to download for build, for instance. And then so you end up with new layers in your compiled app. And then in the export step, we re-upload the newly created directories as layers to replace the previous ones. We leave other layers alone, and then we combine all those toml files together into a big JSON blob and put that back on an image label to live in the image configuration so that on the next analysis step, it can be recovered and it sort of works like that. So to kind of give you an example of this, to break it down into what this looks like in first build and second build. You have your app in your first build. Analysis doesn't do anything yet because there's no image there. Build generates these Ruby or downloads Ruby in Node and installs them, installs Node modules, installs Ruby gems, export, uploads all those layers. And then in the second build, if just parts of that are updated, just say package JSON changes and your gem file changes, we read information about the previous layers. We decide, oh, we're gonna rebuild the Node modules and gems. We just update those individual layers and the app layer on the registry and everything else stays where it is and we don't have to regenerate it. And that's sort of the process. So the goals here are increased portability from what Cloud Foundry is. We don't have droplets anymore, which are kind of a Cloud Foundry-specific thing. We have OCI images, which will run anywhere. We kind of decouple that build process from Cloud Foundry so you can run it anywhere. It's sort of a more flexible model. You have lots of smaller build packs that are simpler and easier to understand and more transparent. And we can really reduce the build time and the data transfer where some builds are now milliseconds or before they were 30 seconds. It's been a kind of impressive thing to see. So if you want more information, the CNCF project is at buildpacks.io. Just to clarify a little bit, the CNCF project has the infrastructure and tools and specification for build packs now, but the Cloud Foundry build packs are still part of Cloud Foundry and they'll be that way for the foreseeable future. And that's all we got. Thanks. Anybody have questions? So we worked on this effort along with Heroku and sort of a collaborative engineering effort. The infrastructure and tools, though, like we have a CLI now that'll build build packs into OCI images. We have a formal specification for it. All that's part of the CNCF project, but the Cloud Foundry build packs are going to continue to be part of Cloud Foundry and the Cloud Foundry Foundation. The Heroku build packs are going to continue to be part of owned by Heroku in their respective places. So you're asking about compatibility. So if you build new V3 build packs on the CNCF spec, will they run on Cloud Foundry? So they won't run on Cloud Foundry right now. We have a bunch of different options for enabling that. I think the simplest thing that we plan to do really soon is make a wrapper that you can just run on a V3 build pack and actually run on a group of V3 build packs and it would create a Cloud Foundry style build pack for you. That's until Cloud Foundry has native V3 support. And there's compatibility in other directions, too, that we've been thinking about for making V2 build packs run on V3 things. And that's a kind of complicated thing. Any other questions? A little? Yeah, that's a great question. So it's not quite like a Git rebase. In a Git rebase, it's really regenerating the top layers, just gets handling that underneath. In this, we're just swapping out one layer on the registry. So the top layers don't get touched. Their IDs don't change. They're all content addressed. Excuse me. So how do you handle merge conflicts? That's an interesting question. The idea is if you construct your layers so that they don't conflict, then they just apply naturally on top of each other. So part of the difference between the Dockerfile model and the build pack model, this new V3 build pack model, is that in Dockerfiles, you can do whatever you want in each layer. In this, they're restricted to directory so that they all apply cleanly, essentially. Anybody else? Yeah, that's interesting. So in general, we make the build packs provide the dependencies in a way so that they don't need the same directory. There's a little, each build pack, it's a sandbox directory that has a POSIX style root in it with a bin and a lib and all of that. So they're all kind of isolated. And then we set up the environment to string all those together. If that makes sense. We've been thinking about some interesting things we could do with operating system packages to let us kind of take advantage of the same layer rebasing, but that gets pretty complicated. Any other questions? You're asking about Knative and build packs? Yeah. So currently, we have a build template in the Knative official build templates repo for the Cloud Foundry build packs. And that works right now. It's the same V2 stuff. Just kind of, we have a shim layer that makes V2 kind of behave like V3. You get some of those features to work on top of registries and with OCI images. There's a PR in there right now for full V3 support that should hopefully get merged soon. You can check out that PR. I think it works. We're just some wording details that need to get worked out. And that uses Knative's build CRD, which is like the part of Knative that does builds. So if you use those templates, whether you can use that to build apps on that platform. Any more questions? Last one. So Heroku started a build pack registry that they've kind of talked about making a more open community-related thing. I don't think they're ready to announce anything quite yet. But yes, we are looking to create a big community repository build packs. And with nice integrations that would let you easily pull down the build packs you want to use instead of kind of confining them to a list of build packs on each particular platform. That's definitely part of the plan. Sorry, can you say again, do we have an? Yeah, so we're currently converting the Node.js build pack to v3. And we're going to provide a wrapper to make that work on Cloud Foundry 2. The, I think, say that'll be done in a couple of weeks probably. The Java team has been working on splitting the Java build pack into a bunch of different v3 build packs, too. They've made a lot of progress on there. They have some parts that work really well now. So that's when the Cloud Foundry build packs will work on v3. The other part of that question is, when will Cloud Foundry run the v3 build packs? We think we can get that wrapper done pretty soon, but we don't have a timeline for what native support might look like yet. So I didn't hear all of that. You're asking about build pack versioning and how that plays into detection. Can you, there's a question about you have an application and you have multiple build packs of different versions in the same list? And which one gets selected? So the v3 API, the new CNCF API, supports versioning. And so you can construct your, it's now a two by two list of build packs because you have build pack groups and then you have candidate groups. It supports versioning so that inside of one of those groups you can say, always pull the latest version or pull a particular version. And you don't have to list the same build pack at a different version multiple times or if you could, but it would be obvious that you're doing something you shouldn't do if that makes sense. You can have different candidate groups that have different versions of the build pack in it if you wanted to, though. Does that sort of make sense? Does that answer your question? Cool. Anything else? I think we're just out of time here. So thanks, everybody. And feel free to grab us afterwards if you have questions.