 I'm Steven Levine, and this is Ben Hale. So we're founders of the Cloud Native BuildPacks project, along with Terence Lee and Joe Kutner at Heroku. The Cloud Native BuildPacks is kind of a next-gen, you know, BuildPack API and tooling for BuildPacks, for platforms to implement, you know, BuildPacks that follow the same sort of specification. It moves to, moves everything to container standards like the OCI image format and Docker registries. So, like I mentioned, this project is a collaboration between Pivotal through Cloud Foundry and Heroku, and it entered the CNCF about six months ago. Sorry. So the first thing I'm going to do is walk you through a case study of what it's like to patch a high CVE in OpenSSL, in sort of through the Docker file model and the Cloud Foundry model today, and then in the Cloud Native BuildPacks model. And this was a real CVE from 2016. It was, I think, a malicious ciphertext denial of service vulnerability. So if you have a Node.js app that's built using a Docker file, it's built using a series of sort of connected images or really using the layers from a series of connected images. So you might have an operating system packages, you know, based image from operating system vendor like Canonical, that someone might build a Node.js image on top of that that provides Node.js, and then somebody might install their application dependencies on top of that and creating another image. That's probably going to be the end user who's going to launch the image in production. So when there's a vulnerability in OpenSSL, that's going to be in the very sort of bottom layer of that, the operating system packages layers. And so if you're a big enterprise and you have 500 Node.js apps, it's going to take a long time to get all those patched, especially if you don't manage your Docker files very well. So if your developers all grab different base images from different places and different node versions from different places, when there's a CVE, it can take a really long time for them all to, you know, pick up the things that patch those vulnerabilities from upstream sources. So who knows how long that would take for 500 apps? If you do Docker files better and you have, your corporation has a trusted Node.js image and they're all based on Ubuntu trustee or something like that or Bionic now I guess, then you just have to wait for all your apps to rebuild on top of those images, but that can still take a really long time and it's swapping out the same bits from underneath everything anyways. So in Cloud Foundry, we have a special model for dealing with this. We keep the application bits that are generated by a build pack separate from the operating system layer. We use a contract called ABI compatibility which is ABI's application binary interface. It means that when we look at those bottom layers and change them out, we make sure that the packages that change just have security patches and that they continue to be dynamically linked to the build application and the build application doesn't have to be rebuilt. So in Cloud Foundry, our operating system package layer currently is called CF Linux FS3 and it's based off of Ubuntu Bionic. So in that Node.js app, when you have that vulnerability in OpenSSL, it's that bottom layer. And so in Cloud Foundry, we have a special way of dealing with this that makes Cloud Foundry kind of unique among different platforms. So to patch those vulnerabilities, we spin up new VMs that have new operating system package layers and these are the VMs that run your applications called Diego cells. And we take the old ones down and we do this one by one until your whole platform is patched. And so this patches high CVs and OpenSSL sort of live in production a few hours. A few hours on a big foundation could be much less than that on a small foundation. So for cloud native build packs, for build packs IO, we have a, we sort of take this Cloud Foundry model. It's also a model that's very similar to what Heroku does too. They call it a slug instead of a drop, but it's basically the same concept. We take this model and we play it. So we sort of replay it on top of a Docker registry where we keep your operating system packages in one repository and we keep your sort of app bits in another repository and maybe use other repositories to store where dependencies live to deduplicate them. And so we can upload, when there's a vulnerability in those operating system packages, we can upload a new copy of them and then just update the image manifest. Just a couple kilobytes per image to point at that new operating system packages layer. So it's one upload to the whole registry of the updated operating system packages. And we can just sort of flip a little bit of metadata for 500 images, which just takes a few minutes usually for lots and lots of images to update all of your images so they can be redeployed afterwards. This doesn't cover the deployment part, you still have to figure out how to orchestrate that, but it updates all of the images that quickly. And it sort of decouples patching the images from the infrastructure. So if you're on Kubernetes or whatever platform, as long as you can handle deploying all those applications and redeploying them in a safe way, you get the same benefits. So the features we use to do this, and why we haven't done this in the past on Cloud Foundry, there's sort of new features in the OCI image specification, which has, unlike the original Docker image specification, OCI has content addressable layers. So you can reorder the layers kind of as you want. You can sort of do rebasing without changing the IDs of each of the layers kind of arbitrarily. You can do this directly on the registry without downloading into the old layers. It's just that the Docker file model wasn't built against this idea, and so you can't sort of take advantage of it with Docker files. And there's also a feature in new Docker registries called cross repo blob mounting, where you can sort of have one repository refer to layers in another repository without having to copy the layers between them. So now we'll talk a little bit about the new build pack API that we're providing and sort of compare it to the Cloud Foundry one right now. So in Cloud Foundry we have all these scripts. We have detect, supply, finalize, and release. In this new API things get a little bit simpler. So we combine the supply step that supplies dependencies, the finalize step that sort of does last preparation on the app, and the release step that generates the start command into one bin build script. Things get a little simpler. But then to support really modular multi build pack mode where you can have lots of small build packs that are transparent working together, the detection phase across all the build packs has some metadata that gets fed into the build phase. And so if you look at the detect and build sort of scripts, we had two phases that are handled by the platform to that. So during this detection phase, an optimal group of build packs is selected. Then they all have to agree that this app is a good candidate for those build packs. During the analysis phase we look at the previously generated image and pull metadata down about it so that during the build phase we only have to regenerate layers that need to change. We actually leave layers that don't need to change on the registry, we don't build them, we don't re-upload them at all. So it's very efficient. Some Java build pack builds go down from 30 seconds to milliseconds because of this sort of advantage. And during the export phase we upload the newly generated layers and then sort of reconstruct the image on the registry to point to those new layers. So I'm gonna take you through what this looks like sort of step by step. And so for an application that's using, in the Cloud Foundry case for Ruby we'd probably have three build packs and node we'd probably have four build packs. But in the simple example I'm gonna say this is one application that has Ruby and node dependencies and pretend that we just have one monolithic Ruby build pack and one monolithic node build pack just to kind of make it easier to understand. So during the detection phase the first the Ruby build pack runs and this can actually happen in parallel to some extent but I'm not gonna go into that exactly. When the Ruby build pack runs it looks at the app and says yes I'm a good candidate for this and it moves on to the NodeJS build, or sorry, it generates some metadata about what it saw in the application then the NodeJS build pack runs looks at the app and says yep this is a good node app also generates some metadata about the NodeJS version for instance. This metadata is fed into the build phase so that when the Ruby build pack does the build phase it generates some new layers, the Ruby layer and the Gems layer it might also modify the app directory here too. Then the NodeJS build pack runs and this happens, always happens serially. It gets metadata that gets that plan metadata but without the information that the Ruby build pack used and then it gets to write new layers. So to kind of take you through this going through one build and then a build afterwards to sort of better demonstrate the efficiency if you have that Ruby node app analysis runs and doesn't do anything because it's the first build. Build runs, installs Ruby, does bundle install, installs node, does NPM install, new layers are exported to the image and then in the second build when the app's changed and maybe in this case just the node and node modules and Gems changed but the rest didn't then we read metadata about those layers. We build just the layers that need to change so NPM install and bundle install but no recopying of Ruby or node and then we export just those changed layers back up to the registry with the different app layers too. And now I'm gonna turn it over to Ben for a demo. Okay, before we head into the demos does anybody have any questions about what we just saw there? No, okay, so I'm going to do demos of three new things that you can do with cloud native build packs that you weren't previously able to do or were not able to do particularly easily and all of these are going to be centered around that pack CLI that I did in the demo yesterday and since I have a bit more time today I can explain to you a little bit more about what exactly is going on. So the cloud native build pack specification differs significantly from the previous build pack spec because in addition to defining sort of what a build pack expects to be called, right? So like whether you're going to call been detect been build things like that and what your obligations are as a build pack implementer it also defines the other side of that interaction as well in a way that we never did before defines how the platform interacts with build packs themselves and sort of the implementation of that specification, right? Of the two sides coming together is something that we call the life cycle and the life cycle is really interesting we have a reference implementation that we know is going to be used inside of cloud foundry and Pivotal products but we expect it to also be used inside of Heroku as well. So since we have this thing, this life cycle thing it turns out it's just a go binary we can wrap that on a bunch of different platforms we can wrap it for cloud foundry we can wrap it for Heroku and we can wrap it in this pack CLI that I'm going to use today. So that pack CLI then is just a very thin wrapper that exercises exactly the same code we'd expect to everywhere else. So the first demo that we're going to we're going to jump off with is basically an extension a demonstration of what we just saw Steven talk about. So we're going to do a pack build here make sure I build the correct one, okay? And so we're just going to build the same thing we did yesterday in the demo we're going to build a pre-compiled Java application I really wanted to come in here and show you all that for the first time on the Java build pack side you can actually build from source now but the conference internet is slow enough that that's just not going to happen. It begins with downloading all of Maven so I'm pretty sure you all wanted to get out of here before that was done. So we see as we saw yesterday there's a build pack group up here that we're going to skip because it happens to be the one that has RIF our function as a service in it but in this particular case we had our second one that has an open JDK build pack and the JVM application build pack in here. And so we see that we get this contribution of open JDK 11.02 to a launch layer basically we have the it says it's reusing the cache downloads so offline build pack style we have the tar file there and we have to untar it onto the file system and then we have to upload that data to a registry somewhere. And so you sort of think to yourself well that can't possibly take that long in reality it's about a second and a half, two seconds to do something like that but we're working on eking out every single second in our build packs right we want to make sure that you can get into running situations as quickly as possible and this is really really critical in something like a function as a service kind of environment and as I said project riff earlier on right we are actually already have a customer for these build packs if you have ever used anything from project riff or if you've used the PFS beta product and things like that they already use cloud native build packs for this just another place that we've another platform that has opted to use that lifecycle reference implementation. So what we're gonna do now is we're going to run this and I'm not gonna hold it because I want you to see again as we go through the build pack the build bit of it see how quickly it can actually cycle through and it's done right because this time we reuse that cache layer the spec gives you a way to without actually downloading the layer back to you whoever you are in this case to pack CLI to take a look at some metadata that's attached to the image but not the data attached to the image and do a comparison and if you are satisfied that what that layer was created from or that metadata matches what you think it is previously then you can just go ahead and say oh I'm gonna reuse this layer I don't need it and I don't need to rebuild it I don't need to spend that second to two seconds building it and another bit of time uploading it question over there. Is that meant? Right so the question is what exactly is the contract of the metadata that you put you attach to each layer? The contract is actually only between you and yourself in the future. Right? The question of whether or not I am going to rebuild this JRE is only for me to look at a piece of what everybody else should consider to be completely opaque metadata so you can imagine for example version 11.0.3 of the JDK comes out I'm going to compare that and say hey I was at 11.02 previously this time I'm going to do 11.03 instead so it only matters to me nobody else gets to view that particular metadata. Can it break your downstream layers? Sure I mean if I put something that wasn't the JRE back in a layer called the JRE yeah absolutely that can and so we generally only use layers and update layers that are equivalent things right? No matter what only a JRE is going to go here and only a particular version of a JRE is going to go here and if you ask for a different one say you ask to go back to Java 8 but your application code is compiled for Java 11 or something like that sure you can absolutely cause problems that way but it's sort of a loose coupling and you have to kind of keep your eye out if you know your source code or your compiled code is Java 8 then you need to make sure that you request a Java 8 JVM and you expect that the open JDK build pack will provide that for you maybe you know 8.101 and then 8.202 and then 8.303 but you've asked for Java 8 and you're going to get Java 8. So just to give you a comparison of what this looks like when you run it with pack or you run it in the life cycle we only download the metadata about the image and not actually the image I can't reproduce that particularly easily I mean I can type about 11 different commands and get that but we're gonna I'm just gonna pull the whole thing so you can see what it looks like so on that image there's some metadata that looks like this for each individual layer and so these are layer names right here tells you sort of what the Shah is and the data that we've opted to put into it and so this bit here is opaque but what we've chosen to do is we represent build pack dependencies I think I can do a cat build packs open JDK build pack help pack.toml like this we represent a dependency sort of like this inside of a toml representation and we use this to say if any of this has changed I want to rebuild a layer if this hasn't changed then I just want to use it as is the key thing obviously is this Shah right here as long as I planned on untarring the exact same binary I had before I don't need to do it again so let's go back here again represent it all in here but it doesn't have to be this dependency you could write any sort of data that you wanted here to help you compare later on when you take a look at it so you could imagine like the npm lock file or something like that might be a good candidate if there's any change I want to regenerate all of my modules okay so that's avoidance of layer creation next thing we're going to talk about is the way we get to use multi build pack and that modularity that I showed off yesterday to say okay I want to do some sort of replacement some sort of modification but I want to do it without forking my entire build pack so a really common thing inside of the Java build pack was I don't want to use open JDK I have Oracle or I run on top of AWS or I run on top of Azure or I'm an SAP customer and I want to use SAP machine again instead so I'm going to show you what we call a builder.toml file the plan is unless you go out and use pack.cl or the pack CLI today you shouldn't really ever have to see this we expect this to be sort of managed by the platforms but it does give you sort of a really low level view of us acting as a platform what exactly we're doing so the first thing is there's a definition of all of the build packs basically just a bag of all of the build packs that you might have access to and so we have things like SAP machine we have all the stuff for Project RIF a bunch of stuff around Node.js and NPM and of course all of the Java build packs as well then once you have those you organize them into groups and groups are both ordered and describe optionality and so right now the second group the one that detected that we saw before is one that starts with open JDK then contributes a build system optionally JVM application is required then New Relic and then so on and so on from there so what we want to do though is as I said before we don't want to use the Cloud Foundry open JDK we want to use SAP machine instead which is an open JDK compatible variant available from SAP as you might expect so I'm going to go back I need to do a create builder open source is going to take a second to actually do it probably should have kicked that off what we're doing is we're creating a builder image in order to guarantee that there is sort of a controlled environment where your application is built the same way every single time we actually create an image with all of the build packs and all this metadata inside of it and when your application is handed to us we start that up inside of a container and then we actually build your application in that controlled environment that's where you can guarantee you get reproducible builds not just on a given platform we're not saying that the environment is consistent between Cloud Foundry and another bit of Cloud Foundry we're actually saying it's consistent potentially across all of these different platforms the same on Pac-CLI as it is on Cloud Foundry as it is on Heroku so this time if I go ahead and do that same pack build command that I did before same command we did before without having modified any of the existing build packs the JVM application build pack but it would have worked for any of the rest of these as well things like the Azure application insights build pack or the Google Stackdriver one now all of a sudden the SAP machine one starts kicking in contributing the SAP machine doing some configuration of it around Bosch DNS and a bunch of other tools that go along with that and again we haven't actually forked a build pack there's sort of a conceptual idea that an operator somewhere has made some modification but not to a build pack in the way that you've had to before where you have to constantly keep up with version updates and things like that every time I make a commit to the Java build pack if you have a fork you need to make sure that you consume that and get all of the security fixes and improvements and things like that this takes that out of your hand as long as you're always staying up to date with SAP machine the fact that you've swapped it out instead of using the OpenJDK means you're gonna stay up to date without a huge amount of effort and then the final thing that we're going to demonstrate here is what I call a cross cutting build pack one of the things especially APM vendors really like about this new thing is previously if you wanted to integrate so your new relic and you wanted to integrate into the build packs you'd run into situations where sorry totally lost my train of thought here you'd run into situations where integrating into the Java build pack happened one way integrating into the node build pack happened another way and there was very good reasons we'd want that integration into the Java build pack to be consistent with all of the other integrations into the Java build pack if you're a Java developer you should expect to configure these integrations in the same way to find which versions of a dependency you're supposed to use in a good way and the same thing true on the node side you expect all the integrations into node to look very similar to one another but now if we've got this modularity we have this possibility instead of being consistent with each language that we can have a vertical slice effectively we can say new relic needs to be sort of consistent with itself across multiple languages so we'll go ahead and do a let's see pack build we'll do the Java one and I'll add some new relic configuration so effectively while this kicks off adding in that end file at the end effectively what this does is that end file exposes what you guys think of as VCAP services service payload for actually binding to new relic it's likely going to change name to CNB services to be a cloud native build pack and portable across multiple platforms again because one of the key things is we want this image to be able to go to a bunch of different places but fundamentally the payload is exactly the same it's again we see some very nice avoidance here and this time new relic fired off and contributed the version 4.1.1 of its agent and configured some properties that are going to run so now if I pull that same thing yeah that's the one I want and nope knocker run a different one that one here if we go ahead and run that same thing but again putting the end file with the new relic configuration in it we go ahead and fire that thing off we'll see that it starts connecting to new relic I'll let it go the whole way out and demonstrate that it is actually has made a connection but one of the key things you'll see in these new build packs we've taken the opportunity this isn't something you technically couldn't do but it was something we didn't do very often we actually now only resolve service credentials at runtime they never get baked into the image and in the old days there were many places all over the Java build pack I know very explicitly where we baked in service credentials into the droplet itself the service was bound during staging time which is still a requirement but then we would like read the license key write it onto the file system in the configuration file for new relic now we've been really really explicit because we want this portability between all these different platforms and all these different environments even though it is much harder to do we will only resolve those things at runtime so when I say much harder to do literally every single integration we're doing at this point has some little helper application yet another go application that has to be contributed so I've got go apps and go apps and go apps at this point that knows how to read the JSON payload out of an environment variable write whatever needs to be done for new relic or app dynamics or elastic APM or something like that into the environment only at runtime and off it goes so we've got a Java application and somewhere in here it says it's reporting to new relic right here but without making any changes I'm telling you that we are using the exact same new relic build pack we did before I want to do a pack build again but this time we're going to build a node NPM application this application is basically hello world as I remember it we're making sure that end new relic is on the environment again because we still need a signal right we still need to know that it is appropriate for me to add the node JS agent even though we're not going to look at the values in it we do need to know that the services there to make sure that all the binaries that are necessary are available the alternative is for me to pack in all of the 40 integrations into every single one of your applications and trust me you really don't want me to do that so off we go we'll go ahead and do another Docker pull of this particular application this node application and not the Java one the node one this time around it is somewhere in here starting new relic for node JS and tells us that this is where we are streaming all of our data to and so one of the last things I want to do here in our last couple of minutes is I do want to show you what sort of a modern build pack actually looks like I'm gonna use this new relic build pack as an example of it so what you're going to find out if you are familiar with the build packs as they exist today specifically the Java build pack we're switching over to go from Ruby as it was before there are reasons historical reasons why it was Ruby originally mostly because it sort of predates the existence of Go and then so many people had forked it that migrating was going to be kind of a problem and be horribly breaking but to be sort of good citizens inside the cloud native computing foundation Go is a good choice we're gonna do it you're not required to as part of the spec the spec basically says that there must be a slash bin build and a slash bin detect whatever language those executables are in doesn't matter you could write it in bash as some of our samples are but in our particular case we've chosen Go and I wanted to sort of show you the kind of things that we're doing to get this cross-platform behavior so we have sort of a language binding that we call lib build pack that sits on top of the specification if you're a Go developer I expect there to be more lib build pack equivalents for other programming languages people can write build packs in anything that they really choose to but what we're doing effectively here is say hey is there a new relic service if there is a new relic service and it has a license key credential I might be able to participate and if that's the case someone else has told me that this is a JVM application right this is one of those other key things about the modularity is we saw a lot of people when they were in situations where they were trying to do a multi build pack kind of deployment that they were having to detect in multiple places whether or not is this a Java application is this a node application is this a Ruby application and all of them use different sort of heuristics to determine whether or not those things were true now what we want to do is there is some authority the JVM application build pack in this case that defines this is what it means to be a JVM application it knows about Java, it knows about Kotlin it knows about Scala it knows about all of these different languages that make up the JVM ecosystem so now the note the sorry the new relic build pack never needs to detect it it just needs to say hey am I a JVM application if I am I want you to participate basically contribute the Java agent dependency in the future on the node side if this is a node application right if someone else has detected this for me then I want to participate in it by adding that agent dependency there we also sort of blindly ask for Python if Python is available since one of the and new relic agent dependencies can be natively compiled if Python's around and falls back if it's not so we sort of make a shot in the dark and say hey if there's Python about if I could have Python please give me some Python so that I can build this thing okay so end of the demo as we announced yesterday our first public beta came out earlier this week the pack CLI that you saw me playing around with I highly encourage you to use it all the build packs that you saw me demoing here are available inside of that and they're all based on your favorite Ubuntu based operating system image finally feel free to join us on our Slack to discuss this it's pretty vibrant judging by the number of you know sort of notifications I've gotten in my pocket while I've been standing up here seems to be going quite nicely and thank you and any questions tell us what's inside what we know at least was that you know we can make process elimination determinations seems like now we're going to know less about what's inside because it could have been a lot of different things depending on how we go so how would the address in fact it's so the question is previously we had very little bit little visibility about what was contributed into a build or into an application when a build pack was used it seems like there's going to be less the answer is actually there's going to be significantly more one of the contracts is that we build a bill of materials every time a build pack contributes something to a layer it has the opportunity and certainly we take it very heavily to describe exactly what was put onto that particular layer so that whole dependency thing where it said it's open JDK 11 version blah blah blah it has this shot and in more most importantly to a lot of people it has this license attached to it that will all be in metadata attached to the image itself so you can literally query against a registry and find out exactly what was put in there our build packs will also go through on the Java side something that people have really really wanted for a long time it will enumerate like all the jar file dependencies in your spring boot application or your web application and do a best effort like we're not going to resolve them back to names what we'll say is you know spring-core version 5.2.1.release and those will be exposed through the same bill of materials as well so I think that's probably way more than you were expecting that's great yeah it was one of those things like we knew that this was an issue but every other time we tried to solve this problem we tried to solve it by kicking data back to the cloud controller and we basically said we're not going to wait for the cloud controller to sort of deem this to be a priority it will be part of the specification because it's that important it will attach to each individual image yeah yes sir so we haven't fleshed out the integration with cloud controller yet but we're sorry oh yeah repeat the question sorry what will this integration look like with cloud controller in how will cloud controller expose you know what's in these images and what build packs they're built with and all those things so we haven't figured out the integration with cloud controller yet but because we're exposing all this metadata on the images we can expose it through cloud controller I think when we're looking at this we're looking at it like can we have a more declarative model for images where you're saying this is what I wanted my image and you're having the platform continue to build images to meet that instead of having developers go I want to restage this now because I think there may be CVs in it we want to move to things automatically get rebuilt and then your pipelines can pick up on them automatically and get promoted through environments and kind of yeah kind of be more proactive about that as a platform because one of the sort of insidious bugs that people don't realize is it's one thing for the for operators to make sure build packs are staying up to date so that every newly staged application is up to date but then you also have to keep track of all of the applications that aren't regularly restaged right and I know a lot of you know companies that do the former and haven't even thought that the second one is a vector for vulnerability yeah so we want that's why the declarative model of basically taking it out of your hands and you don't actually ever cause a build you say I want an image to look like this and builds may happen to get you there think Bosch like or something like that you can do that right now so we the PAC CLI that Ben just showed you it works in a local workstation you can build your images they build the exact same way they would in a platform we actually did the local integration before the platform integration this time so you can yeah all works it was one of the key requirements that allowed or that that sort of caused us to build this generic life cycle that everybody uses whether it's us Heroku your laptop where what have you to make sure we could get consistent stuff out of it the back when the re layering as in the rebase right so the only thing that we are committed to rebasing at this time is the operating system the operating system has a a guaranteed ABI compatibility and while I'm not saying canonical has never broken ABI compatibility we sort of depend on it for you know tens hundreds of millions of rebases across cloud foundry over the last five years I don't think I mean I I don't think so I don't think we're going to add any flags in there simply because it goes it's almost impossible for us to tell after the fact or sorry before an application starts that it might break so then you basically have the health checks as applications are started up right I think there there's sort of two levels to this right there's the talking about the operating system package level and the application layers and being able to rebase that out from under the application layers right and that that contract is very tight we do this on cloud foundry already in production for lots and lots of apps it's never broken we never had a report of it being broken so we're not very worried about that because it's just security patches from operating system vendors that do lots of testing right but then at the application level the contract between those layers is is very strict in a different way it's sort of enforced to be everything's installed in these different directories and there's a you know very particular contract between there so that there isn't a possibility of layers the order of layers mattering there or anything like that so we think the whole thing is very safe and the way we've implemented it even though Docker files might not let you do that usually okay let's call it thank you all for coming here we'll answer any questions down the floor if you need us