 Hello. Good afternoon. Thank you all for coming. My name is James Rawlings, my software engineer working at Chain Guard. We've come here today, we're going to have a little talk around Wolfie. Just before that, I like to ponder, ponder quite a bit. Ponder about many different things. Recently I was pondering about LEGO. LEGO is kind of cool. I'm always amazed by it. I was trying to understand why I'm so impressed by it and how it kind of became so successful. In my head, whilst I was pondering, I was breaking it down. I love LEGO because it's all these small composable blocks that you can build together. There's all many different shapes and sizes, but there's always a contract for how they all fit together. So you can take instructions from the object that you're trying to build. You can use those instructions to build it. Or you can get creative and take in all those building blocks and be creative and build your own designs yourselves. There's tools. I don't know if anybody's seen the... I mean, this wasn't in my age, but there's now this orange tool, the thingy magic. I don't know what it's the name of it's called. But it's kind of awesome because it helps you be more productive and work better with LEGO. And also this is the one thing that I really do love about LEGO. And when I'm thinking about it or doing my ponderings, that whenever you buy a package from LEGO, you always have just the bits needed to build the actual object. You don't have hundreds of extra pieces lying around adding that complexity, making it more complicated. It's just the bits needed to do the job. Okay, that was my ponderings on to the share. It may or may not be appropriate as we go through the rest of the presentation. So, Wulfi. What is Wulfi? Well, Wulfi is used to build cloud native images, cloud container images. It's designed specifically for the cloud native era. It's a Linux undistro built using GLibs-C. The reason I say the undistro is significant because as these are cloud containers that are running on the cloud, that's using the kernel from the host. These images don't include the kernel. Glad you could all turn up. Thank you very much, these chain guardians. It's also a collection of software packages. So, there are 1700 current software packages in Wulfi, and there's continually growing more and more and more. They are built with currently built to run on both ARM and X86 as well. And we use Alpine's APK packaging format. You can't mix and match Alpine's APKs with Wulfi's APKs. You just want to clarify that it's using a standard format for packaging. And this is one of the really interesting things as well. Wulfi is built from the ground up using Wulfi. So, I'm going back to our LEGO analogy. Imagine the factory that's producing all of those LEGO bricks. Imagine that was actually built with LEGO itself. This is how we've built out Wulfi. So, there are some goals and some efforts we have around Wulfi. This, I saw a quote from Vile, who's here, one of the co-founders at ChainGuard, and I just loved this quote from last week. Just because milk was good two weeks ago, it may still not be. It totally applies to software. We have all this awesome software innovation around open source, continually creating, adding more software, which is incredible. But things can go stale. Vulnerabilities can be detected, exploited. So, we need to be able to keep on top of this. So, with that in mind, Wulfi is taking the approach of security first, with that first and foremost in front of our minds. We want to be proactive in the package updates as well. So, for example, there's many maintainers that may be out here today. Lots of CVs that we see get when they come up. We get notified of these CVs. They've actually already been fixed or mitigated or patched within upstream projects by those maintainers, and they've been released, and they've actually been those updates being pulled into Wulfi and being made available to users. Minimalism as well. We want to reduce the attack surface so that we have these small light weights containers running in the cloud with only just those packages needed to do the job. Going back to our LEGO examples, we don't want to add a load of complexity for no reason. That's more increased to the attack surface for when vulnerabilities are detected. We can just avoid having those packages in the first place. So, this is the orange thingamajig that I was talking about earlier. Transform working with LEGO at least recently. I don't know if anyone knows the name for this thing, but maybe you can find that out afterwards. We have tools as well for working with Wulfi. We've got melange here. That's used to actually build these APKs, and then APK itself. We have a tool called Wulfi CTR. That's really helping us maintain Wulfi. There's linters in there. There's an update bot code. There's ways to create advisories, which we'll talk about in a bit as well. But this is really helping us automate and build better and improve around Wulfi itself. When we want to build these images, sorry, when we want to take those APKs and build images from them, we use a tool called APKO. There's also a Terraform provider as well for that. So, it wouldn't be a Q-con talk without some yaml. I'm not totally expecting you to be able to read this, but we're going to dive into it a little bit in a second. But at a high level, the way that we use, that we describe what a package is for melange to be able to build it is using yaml. We have metadata to describe the package, its version, its descriptions, licenses, of course, being very important. We also have a section for the build environment. We can define what that build environment, and again, taking the thinking around Wulfi, only include the tools that are required to actually compile the source code. Afterwards, after that build container has run, that container has disappeared. We actually only want to keep the resulting built artifacts from that build. We have a pipeline. A lot of people will be familiar with this to be able to fetch or clone source code, be able to build this, compile the artifacts, and then we can actually have sub-packages as well. Going back to having that minimalism, we only want packages to be very small and discreet. If you have, say, extra configuration on multiple binaries or docs or development headers, maybe you don't want those to all be in the same APK because you don't necessarily need them all when you're running at runtime. There's also update configuration as well. This tells us how to go and find out when there are new releases available at upstream. With that, we're going to take a little, get out of slides for a little bit, see how this goes. This is the exact yaml that we were seeing. We can see this in Wolfie. There are the 1,700 melange yamls here that you can go and have a look at, loads of libraries, glibc, kill, different types of libraries, there's different languages, Node.js, Java, Python. There might be databases, Postgres, Kafka. You name it, there's lots in here, and it's growing as well. If you have or maintain a project, you might be interested in adding melange config to this as well. Let's switch gears slightly. What we're going to try and do, I'm not sure how we're going to do a demo with this, but I thought it would be good to try and get out of slides. We'll give this a go. This is our melange yaml for Cjason. I picked one at random. You can see here Cjason is at the top. You've got a version, it's a little bit smaller. There's a version here. This is the upstream project version. However, the maintainer of this project is using, that's the version that we have there. We also have this epoch revision. That's for Wolfie. So if we make any changes, say we add in extra dependencies in the build environment, maybe one to add in an extra sub package, or we make a change to the pipeline, then we'd increment this epoch here. Next, we've got our build environment, we've got the pipeline, which is doing simple git checkout with a particular tag version that was referenced at the top. We've also got an expect to commit to make sure that we actually check in out the correct source code and the right version. Next, we've got some of these reusable pipelines. I'm not going to dive into those now because it gets a bit more complex, but to make this more dry, these pipelines, we can have these reusable steps for buildings with CMake in this example. What we're going to do here, this would actually create two packages. It would create a top level package of CJSON and it would create a second one, a sub package of CJSON dev. We're going to see how my live thing is if I can remember how to type. Let's add an extra sub package. So this is going to help me slightly. Let's go see JSON demo. It's not a user pipeline, not one of our reusable ones. Let's run some script and that's helped me a little bit. Let's go. We're going to use a special environment variable here. Targets, targets.context.dir. Yes, I remember how to type. We're going to put this into a different folder called etc demo. Let's remove that. Echo. Hello. Cube gone. We're going to put that into the same directory we just created under etc demo message. So when we build this, let's switch over. I've got an SDK. This is one of the tools that we said that we're using. We've got an SDK which contains Malange, EPKO, Wolfy CTL. So we're actually going to run a build. I was going to do this just with Malange, but because we're specific, we want to show you Wolfy. Let's actually run this with the Wolfy make file. Sub package, make package, see JSON. Let's see if that works. Okay. I don't know if you can see this. This is installing all the dependencies in that build environment. It's declared nothing else. Everything that we've defined in there and any of those transitive dependencies themselves. Next, it's going to run the CMake build, and it's going to build our APKs. Now, if we have a look over here, open packages, because I was running in the SDK, we've mounted a local file system, so we can actually go and have a look at what that was built there. Here we can see we've got those three packages now. We've also got our CJs and demo. Let's have a quick look at what that looks like. Open this up. Use it. Now, what we should have is we've got the folder structure that we created, and we've got our message file that we added in. It's really just a simple example to show you what's in that Malange pipeline. Of course, you'd be compiling binaries and putting those into your separate sub packages. Interestingly enough as well, all of these APKs come with S-bonds as well out of the box. When we're installing these APKs, you also get the S-bonds along with them as well. Okay, let's shift back. We've also got a container here. This is just again using the Makefile. This is using a Wolfie-based image. This has just got, I think, busybox and bash. Maybe that's it. What we can now do is APK and see Jason demo. It worked. Okay, now we can cat. Can anyone remember the folder? ETC demo was a message. So you can see you're starting to get a relationship of being able to build these APKs through a pipeline. They get created separately, separate out small discrete APK packages, and then you can start to install and use these APKs as well. Okay, that's one part of it. Let's skip back to slides. So we talked about there was an update config block at the bottom there. There was that one there was using GitHub, so we have a bot that will run every hour. It will go and query GitHub's GraphQL API and check to see if there's any new releases. We've also got, you can have another configuration of release monitoring, which is a Fedora project, which again is great. We want to be able to, this is how we are pulling in understanding when there's new releases upstream. This is a graph which I thought was interesting. Of GitHub releases, so projects that use GitHub releases, over 80% of the projects are actually they're released upstream and they're available in Wolfie for Wolfie users. So we create pull requests, automated pull requests, and have checks run on those. They're approved, merged, and available in Wolfie. This is just an example of one of the automated pull requests. You can see here there was this particular one runs every hour and it was the pull request was created, the checks validated over nine minutes, and we see Carlos then had a little check and made sure of everything and approved it and that was merged. A few minutes later it will be available in Wolfie. So thankfully, most software uses versions. So that makes it easy for us to understand when there are new versions available. That's also a version. There's been an interesting journey because there's some weird and wonderful ways that people can actually create versions. This is an exhaustive list. I'm sure there's some more interesting ones out there. But if you look down in the bottom left, there's an A character that's using, that's identifying that there's an alpha release. Open JDK have a plus symbol in there. You've got Gradle that use release candidates and also milestones. MariaDB have a prefix on MariaDB. This is interesting. Postgres, they have releases on the same Git repository of multiple major versions. That's because they provide maintenance and support on multiple major versions at one time. So we refer to these as version streams. So diving into version streams a little bit more, this is of Go. So this is from endoflife.date where we look at Go and we can see that the current in support versions of Go are versions 120 and 121. On the right-hand side, we've got how that is represented in Wolfie. So we can see we've got a separate Milan Jamal for each of those major versions, or sorry, each of those versions, those version streams. When a new version stream is created, so when Go 122 is released, we'll create a brand new Milan Jamal and that will go into Wolfie. But it doesn't stop there. We will then also epoch bump every single downstream dependency that uses Go as well. So we recompile those with the latest Go version, keeping everything up to date. There are occasions where sometimes that doesn't work. Those get caught by CI. We do have to pin them and we'll wait until the upstream projects are able to actually work with that latest version and then we'll unpin them again. So talk a bit about the Wolfie CTR. There's a bit of a Swiss Army knife. So this is an example of how we want to keep our CVEs low. So we want to be aware. Wolfie CTR scan is actually just a wrapper around gripe. It's just slightly tweaked to be able to work with APKs. You can see that we continually want to be aware of how, of any vulnerabilities in APKs. So how do we keep CVEs low? Well, going back to the LEGO example, we want to have these small composable APKs. We want to reduce that attack surface so that when people build images from Wolfie, then there's only the packages needed to do the job. We also have talked about the fast updates of releases. Largely, when CVE we see gets announced, an upstream project has actually managed to mitigate that and created a release. So we already have that patch, that new release available in Wolfie by the time we actually it's announced we find the CVEs. On other occasions sometimes that doesn't always happen. We do have the ability to patch as well, which we'll dive into, have a couple of examples. We don't always want to do this. The occasions why we may want to do this is if an upstream project has a CVE, they've mitigated that, they've created a patch that's been merged, but it hasn't actually been released yet. And so what we're able to do in that example is take that patch, have a reference to it, and because we're compiling from source, all of our packages were able to apply that patch in the pipeline and mitigate that CVE. That's not all, the only way that we actually can address some of these CVEs. Lots of times they're around dependencies. There's lots of dependencies in the world. Here's an example of Go where this is actually in the Go pipeline when for Pulumi Kubernetes operator it's doing a Go get of GRPC to a specific version to remediate this GitHub advisory here, security advisory. Again, the epoch is incremented so that this is the Wolfie version of the same upstream version software. So we run a Go mod tidy and then compile that binary. That's not the only way as well. So we also want to, we do a huge amount, a tremendous amount of triaging around CVEs. So we want to document that. It's not just a case of saying, okay, we've done all this analysis to prove something as a false positive for example. We want to actually describe the steps we've taken and add in any, the reasons, any context of why that is a false positive. That may change in the future so we want to keep that. Additionally, we might, we'll create advisories for when we actually fix these versions as well. So this is important because we'll actually generate a security feed from this, which scanners can then go and consume so that when they scan and find vulnerabilities or they do package matching, they see our packages, they'll see our versions and they'll know that we've addressed and mitigated those CVEs or define, declare them as false positives. We are rocketing through. This is great. This is the fastest I've done this. Brilliant. We also have talked about quality as well. This is going to, a continual stream of efforts. This isn't just going to get fixed overnight or addressed overnight. We want to continually improve quality of all software, but also catching, shifting left understanding of when the new CVEs are appearing. So as early as possible and continually as possible, ensuring that a package updates when they're coming through, they're not going to break anything, that we can continually validate and ensure that users of Wolfi are having, using the latest and greatest software. So we have currently a number of checks here that run on pre-submit for a lot of new packages or package updates. We've got a linter. She'll just double check some of the best guidelines that we're saying around Wolfi. Some of the Wolfi maintainers have had years and years and years of experience in many different distros, Alpine being a significant one. So there's lots of knowledge and experience that's been learned from it there. So we want to automate those checks or those recommendations and apply those into automated linters. We also have a Wolfi check for the updates so that if somebody adds in a new package, we want to make sure that where they're fetching, where we're saying go and look for that new release of software, that it really is, it does work. As we scale this out, we said 1700 packages currently, that's going to scale. We want to automate all of this and help us grow. We also build on pre-submit, compile the codes, the packages on x86 and ARM. Actually, this was, I guess, probably came in about three months ago, I'd say, on doing both on ARM as well, because there was a lot of inconsistencies. So we want to understand what those inconsistencies are and check and validate softwares running as it should do on both architectures. Interestingly as well, when we were talking about those, the LEGO blocks before about all the different shapes and sizes, but they have a contract of how they fit together, there's also a sonane check here as well to verify ABI compatibility. So when we update these libraries, if there's a new shared object, major version, we know about it and we can verify that will work with any of the other libraries as well. And you can see here we're also scanning those APKs. Two of these checks aren't actually required. This is an effort over time. We want to shift more of this effort left and ultimately get to a place where we're not adding in packages that have any CVEs that we're continually scanning and that's the quality gate going in. But of course that's not everything because CVEs, as Vlea's lovely quote to say, next week there may be a new CVE. So we are continually scanning for CVEs for all these APKs to understand if there's any new vulnerabilities, creating advisories or mitigating these CVEs, these vulnerabilities. So as far as, so we've talked a lot about creating these APKs, these packages. You don't run these APKs. We are building these to run on the cloud images. So for this we use the tool APKL. A little bit more yaml, but there's a special surprise, there's not just yaml as well. Here we can see a very similar structure to Milan's, where we are defining the repository and the key ring to be able to look for these packages. And this is where we're defining what packages we want to install for this image. So here, Wolfie-based layout is just something that's standard. It tells scanners about what the Linux distribution is being used, so it contains Wolfie. This is the package that we want to build, co-probably a bad example here, but we'll also include in Busybox and Bash. We may not want to include Busybox and Bash because it's extra packages. If there's a vulnerability detected in Busybox, then you then have to update this image. If Busybox and Bash isn't even in the image in the first place, then you don't need to remediate any CVEs. It's not going to appear on a spreadsheet somewhere to say that you need to go and investigate it. So you've also got other configuration here around accounts or volumes or environment variables as well. So familiar things for building container images. This was the surprise as well. It's not just YAML. It's also HCL as well. There is a Terraform provider. Chainguard, when we build our images, we actually make use heavily of this provider. It helps make things a little bit more modular, reusable, dry, and there's lots of examples to have a look at if you're interested in that. With that, I'm going to pop back and go back to our demo and have a look at... We have this demo APK. Originally, I've taken co-outs. That probably wasn't in there. This was me messing around earlier. So, C.Jason demo. I could say C.Jason dev, and that would just install the dev libraries. Let's go for the demo purposes. Demo. We're saying we want to install that package that we've just created. We want to run as non-route security first, and we have these uids and goods. Now, let's go ahead and use APK-O to build this. Going back up to our SDK. Apologise, this is a bit of a long command, and it may not appear very nicely here. This is just doing an APK-O build. It's got a couple of extra flags to signify the local directory of where we built our APK when we ran the melange build. So, we're also going to run build for the ARM architecture. So, that is going to take... I mean, that was quick. We're going to... That has built us a table of demo.tar containing the packages we've defined and set up the... set up this local table. What we can do now is we can load that table into Docker where we have this image, and now we can go and have a look at... Oh, exactly. It's just... No. Let's do docker... Docker run. I have to do this for my Mac for some reason. Platform... Linux... Arch... Arch 64. Okay. Minus ti image named bash because we included bash. Now, when we have this container image, we should be able to go atetc demo message. This is just demonstrating that we've built an APK using melange. Yeah, it was a bit... There's a rubbish message in the middle there. I don't know if you can see that. But the concept is there that we're using melange to build these APKs containing compiling software, and then we package them up with APKs, and then we're able to build these container images with just the bits that we want to actually run them. And let's just, as well, let's just run that. This is our image. And again, let's just do gripe. Again, as many scanners as we want, or gripe have just released a new version, looks like, but you can see there's no vulnerabilities. I wouldn't expect my message to have a vulnerability, but I did have Busy Boss and Box and Bash in there as well. So we can see that the scanners work well with Wolfie because they actually finding there's 24 packages there. There's no hidden packages that have been secretly installed into this container. So the scanners found those packages. It's reporting there's no vulnerabilities as well. And with that, just some thinkings, maybe some again, of what Wolfie could mean to you. We want, you could build, use the existing packages. You could build your own creations using these packages. Wolfie is used to build secure by default cloud images. Low CVEs, fast package updates, and a small attack surface. If anybody's maintaining a project, add your project to Wolfie so more users can benefit from it. And with that, nice little end to it all. Everything's awesome. With six minutes to go, is there any questions? Nice questions. Oh, thank you. There's a question there. I think there's a microphone. How would you compare, you know, the approach that Wolfie takes versus an approach of building with Debian and then, you know, slimming it with, like, Docker slim? I'm quite here, I'm sorry. Could you say that again? I couldn't quite hear. Yeah. How would you compare the approach that Wolfie takes versus an approach, like building with Debian or Ubuntu, and then using Docker slim to, like, tree shake out the dependencies? Yeah, no, I think that's a good question. I mean, to be honest, I didn't put anything in this talk specifically, too, because I think sometimes a lot of this comes with preferences. I think there's, I think we're, what we're doing with the Wolfie project is thinking about the way that we, how to approach this problem, probably slightly differently from maybe Docker slim. What we want to start right from the core, where we said about building bootstrapping Wolfie with Wolfie, we want to understand the entire stack right from how that was first created. If there's ever a CVE, we want to be able to find out where that source code is, and we want to be able to mitigate and patch that CVE so that everything downstream of that can actually benefit from it. I think there's value in all, in a lot of these different approaches, quite honestly. I'm sure lots of people have opinions on these things. I certainly don't understand on stage and kind of open up a framework of anything, but I think there's merit in all different options, to be honest. The idea around Wolfie having that minimalism and, you know, if you're working with Debian, sometimes there are other packages involved there that may not have been used required at that time for your purpose, and that's probably one of the significant reasons where we really want to focus on just removing the unnecessary packages for that runtime. Right from the start. Does that answer, does that help? Yeah, yeah, okay. Good afternoon. Thanks for the talk. First of all, a great project. I like the idea of it. I'm working in an air-gapped environment, and I'm trying to figure out if this would work for us. We're experimenting with Alpine and APKs, so I'm all new to that, but I'm in old hand with all the distros and things, so getting the hang of it. But we would need a local drive or... Yeah, I mean, just thinking off the top of my head right now. The example that we were showing there was building the APKs. It was actually creating a local APK index. That's what I saw. That's what I was like. Yeah, exactly. So how that could be built and distributed and shared around exactly? Yeah. And just one thing, like CVEs, we're working with the government so they're stigs, but could you solve the same problems using stigs? With scanning as well? Yeah. Is that the gripe tool? I missed the first two minutes. I was in the wrong room. Oh, right, sorry. So what was the question? So do we... Like scanning for both of our vulnerabilities? You're saying because there was a Wolfie scan APK. Is that what you're saying? Yes, I see what you're saying. So that's actually just a wrapper around gripe. One of the maintainers of Wolfie used to work with gripe, and so he's taken some of that knowledge and made sure that we can shift that letter and understand before APKs go into the images if we can understand if there's vulnerabilities at the APK package level. I can be happy to talk to you and show you where that is afterwards, and that'd be really interesting. I do agree. I think you should be able to use that. I also wanted to say excellent project looks really awesome. Do you envision when building an end-user application containerized using applications, building their dependencies out into individual packages, and then combining them together or generating their own unified package or just using it for base images? Which approach do you recommend? Ultimately, I think there's a timing thing here as well. I think naturally people are using say base images and then layering on top of that. That's quite common. I think over time as we grow to the future, I'd love to see more and more software being packaged in this way so we can manage in a consistent approach. Again, that we can patch and compile by source. It depends on different language ecosystems. We go, it's quite nice. You can statically compile that binary. It's nice and simple. Java or Python, then you're downloading lots and lots of dependencies from all over the planet, but it's still a challenge. It's still a problem that needs to be needed to have a look at because those jars can contain vulnerabilities, and how do we address that at the application level? I think it's something over time. That number of 1700 packages, that is going to grow. That is going to scale. As more and more we can actually understand, we can build these packages in a consistent way, then we can benefit in the same approach. Thank you. Cheers. I'm not sure if we have time for one more. I was curious to know how might the open source community contribute packages? I mean, we're a consumer of some of the free chain guard container images, but similar to how Alpine works with their A ports, it's very easy for anybody to submit an MR and add their tool to the package repository. How might that work for Melange and I guess Wolfie and your container, your public container images? It's a great question. I probably should have pointed this out. There is a repo Wolfie OS where you can, all of those YAML examples that I was showing, we can, let's have a look here, you can contribute to that. You can create an issue if you want some help along with that as well, but there's lots of maintainers. There's lots of people contributing to this as well. There's the best examples of the Melange YAMLs themselves. There's some documentation. There's an improving set of documentation, but hopefully when you identify something similar to what the package you're looking at, find a typical pipeline and you'll be able to contribute that as well. Open an issue if you like any support with that. Cheers. Go on, let's go for it. Do you do anything to ensure that the open source repositories you are building from are secure? Say there's like a malicious commit in Caddy? So if there was a malicious commit in Caddy, for example, one of the maintainers maliciously pushed the commit? Yeah, I mean certainly from Wolfie we are relying on a lot of the ecosystem around that is focused on that specific problem, whether that's through scanners, any way to identify vulnerabilities on malicious code. We don't do anything like that for Wolfie itself because we want other scanners to be verifying our work that we're doing here. That's not it to get out of the problem. It's just making sure that we are being fair and many scanners that are out there that they are doing the best job that we can actually use and leverage to understand. We're certainly focused on identifying if there is malicious code and understanding how fast we can actually remediate that and understand and ensure that users of Wolfie and anything downstream can actually receive those remediations and updates. Do you fork the repositories? No, we don't fork the repositories. We will check them out directly from GitHub, or if there's not all softwares on GitHub, some produce tarbles on distribution sites, so we'll fetch that, but we'll always double check the expected 2.5 share as well to taking measures to ensure there's no man in the middle attacks on there as well. Thanks. That was awesome. Thank you so much. I really enjoyed that. Thank you.