 Good afternoon. My name is Aaron Creel. I'm the director of government security at SpaceX. So kind of in that role, I own all of SpaceX's like secure facilities, clearance process, and classified systems or sensitive systems. We're going to talk specifically about that last one here. Yep, and I'm Wayne Starr. I'm a DevSecOps engineer at Defense Unicorns. I spent five years in the Air Force kind of in the software factory world working with secure software and how do you implement CI CD and some of the continuous authority to operate stuff that you may have heard about. So a lot of that stuff dealing with how you implement NIST and all that stuff within the Air Force. But now I'm in the Kubernetes world working with SpaceX. So a couple disclaimers right up front. Both Wayne and I talk very, very fast. We're going to take a real make a real conscious effort to slow it down. If you see us like start spinning off the rails, let us know and we'll try to slow down. Also, I am not very technical. Wayne is extremely technical. So if he starts going down a rabbit hole and be like, Wayne, what does that mean? So please like, yeah, I'm trying to make a conscious effort for that. So kind of the problem. About a year ago, SpaceX, well, I'll back up a little bit. So SpaceX, you know, commercial company, a number of government things. And as the years are progressing, the more government programs and the more sensitive government programs is kind of growing. We have for the longest time kind of a very manual process and how we do those things. I knew of Defense Unicorns through my previous life as a presidential innovation fellow and kind of asked them like what they would do to help make this process a little better. So, yeah, so we looked at an angle process. We looked at, you know, how things were being done efficiently and and try to kind of bridge that gap between how we were doing it kind of the very tip of the spear how like, you know, these innovation centers are, you know, doing this really quickly and try to bridge that a little bit for industry. So this is one of the, and that's the way to help me out on this. This is one of the more, this is the project kind of like that we took on in order to do this effort. It's a lot. So, yeah. And specifically, this is one binary that basically builds from a couple different projects internally. And then it gets built itself into a container. And then that gets ran in amongst an ecosystem of containers currently in Docker but moving to Kubernetes. But so this is just that one project and all of its dependencies, not concluding its web components, infrastructure components or container infrastructure, just because of how complicated some of the math and different things that they're doing with a lot of these telemetry systems are. So why solve this? And that is kind of the big question. You know, there are kind of two points of view. So the SpaceX point of view, and I think it makes a lot of sense is we're launching a rocket a week and we're landing it. We're putting 4,000, 5,000 satellites into space. Like we're doing this right. The government process is very much, you know, we're going to assess these controls. We're going to look at these things very just, you know, one by one. And we're trying to bridge that gap here. Trying to make it where, you know, SpaceX understands kind of where the concerns to the government side are. And then the government side takes a kind of realistic understanding of how SpaceX is doing it. And knowing that just because they're doing it a certain way doesn't make it less secure. But trying to kind of translate that from SpaceX to authorizing official was kind of our goal and make it more automated. So kind of two ways we could do it. One is through policy saying like, you know, this is our step-by-step process. The concern there is, well, a big concern is communicating that policy. You know, we really have one software process. We'll use it for everything, right? So whether it's Starship, whether it's Starlink or Falcon, all of our code really starts at the same place. And then it kind of changes a little bit based on what we're deploying into. But the concern is as it gets further along in the process, that process changes. The better solution is to automate some of this. You know, for SpaceX, one of the big concerns is we use a lot of custom software. And, you know, that is, a lot of times takes a manual process for the government to look at it and say like, cool, this meets my threshold from a risk perspective. So we kind of looked at three ways to do this. So on import, I guess kind of a process to do this. So on import, like initial reviews, once it's integrated into like that new stack or that new air gap region, or once it's fully up and running, when it was deployed. And, yep, go for it. The other thing we really try to avoid is once it's deployed into that new environment, how do you make sure that the version controls the software dependencies stay fresh and are continuously updated? So we didn't have an answer as we came to these guys. And really it has to be easy enough for me to understand, you know, the non-technical person, the person who gets it a little bit, but, you know, you ask me two or three questions deep and kind of just glaze over. Yep. And so getting the automation pieces, there are a lot of different tools in this ecosystem, but I'm going to kind of bring it back to basics at first. Many of you are probably familiar with CPEs, common platform iterations. They've been around for a very, very long time and are largely how you would identify a given package or product within your ecosystem. They're how CPEs are looked up and very similar information would be used for GitHub security advisories or other things. And so it is something that's important to keep in the back of your mind is to how you surface this information within your build process, within your deployment process, and how you understand that within your environment. Generally, when you're looking at a CPE, the big things you're kind of asking yourself are what's the vendor? What's the product? What's the version and what's the language? And all parts of those are very important. And there's obviously other things that are important for, if a particular vulnerability impacts a piece of target hardware you have or various other things. But those four things are really the big ones to hit. And you don't always get this information from every packaging ecosystem. Within the container world, within like say an Ubuntu system, a Debian package system, you will get a lot of this stuff. You may even get even more like the maintainer of a given project. But within say a C language thing, which we'll show later, you might be missing, say, the vendor entirely. And so you have to kind of guess at those things. So the product may not match what the CPE database has. And so you have to find ways to translate that or find ways around that within the automation. The other thing you have to think about is how is this analysis actually happening and when and where within the build process is it happening? Are these dependencies that we're sbombing, we're trying to find, we're trying to attest, are they actually there on the system? Or are they potential theoretical ones that could be downloaded from say a mirror? So a package JSON file for say a node project, it describes a whole bunch of dependencies. However, none of those unless they're in your node modules folder are actually on your system until you run npm install. So if you say had some optional dependencies in there or dev dependencies, those you may not necessarily care about in the security context of your production system because they won't be there when you actually go to production. So you have to think about the context as to where dependencies are used and how you're pulling them in. The other thing, especially for SpaceX, given how much custom code they have, is this package something publicly available or is it private to the enterprise? So can I go out onto npm or the pipi or whatever and figure out what this package is and maybe look up vulnerabilities in a public way? Or do I have to dig deeper and potentially say, okay, so this is a private package. What are its dependencies so that I can continue to keep going upstream through that dependency tree and potentially find more information about it? This was a big concern for the like the air gap deployments that we were talking about. So those dependencies may not be available in the region you're trying to deploy to. So if you deploy it and don't check that, it's not really good. You're not going to action anything. So having that checked before you actually deploy is really important. Yep. And then the other thing is, is this package actually used by the application, as I kind of mentioned, there could be optional dependencies or dev dependencies that you need to consider, you know, do I actually need to bring this into production or could I actually cut it out? Or you may just have, for example, like a fatter container image and you may just have things in there that you don't care about and shouldn't necessarily be there, but you need to know that and be able to get that information back. So in general, kind of going through these things, I'll show a little bit of this through a couple different tools. There are kind of three main issues that I saw across the ecosystem. It's relatively new. There have been tools that have been around for a while, OAS dependency check being one of them, but then there's newer tools like sift and gripe, which I'll also show. But one of the things that kind of comes up is consistency within package ecosystems. So if you look at a node package, it may identify exactly what that package is. It's load ash, whatever version. Cool. I know what that is. But if you look at a go package, it's going to show you all of, or a go binary. It's going to show you all of the dependencies of that go binary. And so you'll get this mix of information that you have to handle as a security professional as you kind of look across different packaging ecosystems. The other thing is configurability. Some of these systems are quite large. They have lots of files and you need to be able to say, I don't care about this directory because I know that's a data directory. I know that's another kind of thing within this project. And you need to be able to trim out a lot of different things because if you scanned every single file and had sift, for example, interrogate every single file, it could take a really long time to do that. So you have to be very careful about how you actually implement these tools. And then on the backside is management. When you actually get this information back, when sift outputs an spdx, cyclone, dx or json file, what do you do with it? You need to be able to know when an update happens, be able to compare them, be able to understand what's inside of them, and be able to pass them to SCAS or other folks who might be not technical. And so I'm going to show a little bit of a demo as to kind of how some of this stuff works. And so to start, I have this, basically this project. It has some node dependencies inside of it. It's simple package.json, package.lock.json, and index.js file. It has some Python dependencies in it, just an index.py and a requirements.txt file, and then it has a little shell script here. And so if I go into this folder and I kind of just want to see how that works, I can run sift against it. So sift package is done. And then you can see that it indexes them and it finds a whole bunch of things. And this is great. Like, it's a great start. And if I was just implementing this for myself, I might think that this is kind of all that's there. But this doesn't actually tell me a whole lot because what it's doing is it's looking at that requirements.txt file. Numpy, pandas, and PyTZ in this case aren't actually necessarily on disk, and neither are any of the npm packages because I haven't run npm install. The other thing is I haven't looked at that actual shell script to see what it does and what it downloads. And so one of the things that we do at Defense Unicorns is we have a tool called ZARF that integrates with sift and wraps it. And because it's an air gap and because ZARF is an air gap delivery tool, it forces you to be declarative about exactly what you bring into that environment. And if I actually were to look at this shell script, you can see that it is actually going out and reaching out and downloading kubectl and putting it on the system to run. So if I had something like this that a developer was trying to introduce to a system, I might not necessarily know that because unless I was watching network traffic or had some other policy down the line. So knowing that this is a dependency or being able to pull this out is pretty important. And so what ZARF does, if I go back, I can run a ZARF package create on this folder. It allows us to have a declarative package that will actually pull all of these dependencies down. It'll pull all the images down that are needed for this particular package because this not only pulls down this project but it also will upload it to a registry in the air gap. And then it'll basically pull everything onto disk and put it into a tarball that I can use to bring up to the air gap. And so this is a very useful thing because it basically acts as a forcing function for me to say exactly what I wanted to bring over. So it's gotten all of those dependencies down and now it's pulling image metadata for all of the different things that are inside of there and then it'll pull all of the images and then start writing it. And you can see here it's creating all of the image S-bombs so it's interrogating all those images with sift. My computer's getting a little warm. And then it does the components and now it's done. So if I want to simply look at this thing I can inspect it and then I can get the S-bombs outside out of it by doing this. So if I wanted to do that I can see that that's a package and it attracts it and now if I go into my folder here I can see that there's this S-bomb test folder. And so now I have all of the S-bombs from that package they're inside that tarball but so I can extract them anywhere I'm at wherever that tarball goes the S-bombs go with it. So on the air gap side I could see all this information or here obviously I can see that information. But I can see all of the sift JSON files which are useful because sift also supports conversion of these to whatever format you want so if you had another tool you could export these to Cycline DX or SPDX. And then I also have these HTML files that I can kind of look at that ZARF builds on on top so I could say okay I want to look at Alpine let me look at this I can kind of flip through them and see all of the different things going on. And in this case you can see them for my little software project that I have now not only do we have these npm packages and all this stuff we also have all the go modules of kubectl itself and so we now have all of that information here and kind of in one simple interface that I can refer to. The other thing I can do with this tool that we've built into it is do comparisons which was something that is really important when you're trying to update software especially update it quickly because if you wanted to get and kind of approval for any one of these modules to go onto your system it's going to take quite a while and if you do that for everything all of the time it's just going to bog you down and so being able to know exactly what changed is a very very useful thing. So if I go here and I go to one of the previous as bombs that I had ran for Alpine 316 I can basically pull that into the old file and then I can go into the one that I just generated and then I can run the comparison and I can see exactly what changed what was added and what was removed from that one container image to the next container image and so now this reduces the cognitive load on the security professionals as you manage that data throughout your system. And what makes this a lot you know really helpful is you take this delta to the authorizing official who owns that network and say like this is what we want to change versus like this is the old one and this is the new one having to do that process and have that kind of relationship that says like cool I'm looking at these you know 10 or 12 different things go for it. Right exactly. So the next thing I'm going to kind of show is a little bit of when this starts to break down because sift and gripe work very very well for the container image world, the node world, the python world, the go world. But a lot of this stuff is C, C++ code and that starts to get a little bit more hairy and so I'm going to show a project called rigs of rods which is the soft body physics simulator. It's not a satellite telemetry thing but you certainly can do rocket boosters and attach them to things and see what happens. But basically the important part about that particular project is that it uses both Conan and CMake to build. So those are two different ways to pull dependencies for C projects and Conan run by Jfrog basically allows you to pull dependencies down sort of like npm whereas in this particular case for rigs of rods they use a CMake meta project that will pull certain things in. So if I go into here and then I'm actually sorry to go this way. I'm guessing what he just said means something to the more technical people because he explains me five times I just shake my head yes because he just it doesn't work. So if I catalog it directly with sift you can see that there's all this Conan information that it pulls back and this is all of the data that it is found within a Conan lock file on the system. Even this though isn't entirely perfect because if I actually go look at that Conan lock file you'll see that it has information like it is okay there's elib okay there's libpng there's bzip2 but it is missing in this case vendor information so if I were to create a cpe I would have to go to the nvd database and I would have to look that up and so in this particular case if I go back to search here we'll see if this loads if this takes too long I'll just walk through this but if I search through this for broadly for example whoops that's not how you spell broadly but you can see it's pretty brittle on on exact precise naming of things and that's something that I'm gonna show as well because over here you can see that it found lib alsa which is a sound basically you know system for Linux but in this case in this case it found Google broadly but in this other case over here it's actually not called lib alsa it's called alsa and it's under the alsa project and so if you would try to search the nvd for lib alsa you would have found nothing and you'd think okay I'm secure but there's actually no cpe under that so there's still a little bit of stuff you have to work around with Conan but if you were to point this at the meta project for for this you basically would find nothing because it's a cmake project in this case in this part of the of the project and you except basically just gives up and can't find it which is fine because that's not something that it can do but if you were to use a different tool in this case an older tool that has been around for quite a bit longer dependency check which is made by the alsa foundation which another talk is actually talking about alsa and all the things that they do definitely recommend checking them out but if you enable their experimental flag you can actually get more information because it'll actually go through and look at the cmake lists and the dot cmake files and it'll try to find a whole bunch of stuff and so if i go back to back to that you can see this is the report that it generates and it's a little bit different obviously this isn't json it's hard to like deal with this later on but you can see how much it's kind of guessing simply from the fact that some of the names are really weird like this is called cmake import file it found a an actual thing fmt is an actual cpe within there but you can see the data is very experimental that's because i turned on that flag and it only supports this experimentally but it kind of shows that you you kind of get whatever the packaging ecosystem or build ecosystem that you run a system within gives you so in this case i would have to sort of figure this out as a as a security professional and say okay i see this vulnerability how is that happening and if you actually looked at this you'd see that the cmake import file is coming from this fmt target cmake over here and it's kind of not obvious why that pulled up the real reason is that there's a regex and it's kind of matching on this and then it takes the file name and then it searches the nvd and then it finds fmt and then it returns the cpe but what's that mean all of all of that stuff though is just kind of implementation details where it's kind of seancing its way through these things and so it's very important as you especially when you're building custom software or lots of software that may be interconnected within a larger enterprise that you think about your build system and how it all kind of comes together in the end um so yeah that was pretty much it um are there any questions it's nice rocket landing uh any questions or nobody ready for happy hour cool all right thanks