 Hey y'all, hope you had a good first day here in Detroit. We're having a lot of fun here. We're going to pry, I'm probably not going to be too long. But like you said, my name is Cole Kennedy. I'm the founder, CEO of TestifySec. My co-founder over there, Mikhail, he gave a talk earlier on Archivist. If you didn't see that, that'll be showing up a little bit in this talk. So I encourage you to go back to the feed and go ahead and take a look at that. It might add some context. So at TestifySec, we're building a platform that enables enterprise to verify the integrity, compliance, and trustworthiness of their software. And we're doing all this with open source software, with a thin layer proprietary software on top. And really what the problem we see is that ever since we started creating software, when we hand that software off to production, right, there's really no way to tell what was in that software, where it came from, what was in it, what tools are used to build it, right? And so it's just a lot of magic. And security and compliance engineers and administrators are really struggling with this, especially with a lot of the new regulations are coming out, right? There's a new NDAA that's about to get signed in the law that says no software can have vulnerabilities, right? And there's SSDF that's going to require you to certify all the steps of your supply chain and make attestations about that. If you're a small company that tries to sell software to the federal government, this is going to be really, really tough for you, right? It takes a lot of manual labor to produce these reports. And we're really trying to help these companies with this type of a burden, through automation and attestation. So what I'm presenting on today is verifiable attestations of EBPF traces, right? With witness, we can do a lot of different types of attestations, and this tester is experimental. It's not in the main branch yet. But why do we want to do it? One, we want to find hidden CVEs. If we look at the log for shell or the heart bleed attack, those artifacts that were vulnerable to those attacks, there was no way to tell that they were vulnerable just by looking at the artifact itself. Later on, some tools did come out that kind of helped with this, let you kind of sift through the things. But when a CVEs just announced like these, what tools do you have as a security administrator to find out where you're affected in your inventory? There was nothing. Now we have a little bit more. We got S-bombs and we have some other tools that can help out with this, but really it's very difficult. Number two, we want to thwart malicious actors. SolarWinds had a really bad time. I think we all remember what happened with them, right? Their build system was completely compromised. So they were shipping off signed artifacts that their customers trusted that had malicious code inserted into it. What happened was their build system was compromised to the point where there was an agent running on their build system that every time a compiler action kicked off, it looked for a specific file and replaced it with its own version, therefore injecting that Trojan. And there's really no way, by looking at that artifact, that you can tell that that happened, right? There is a paper Reflections on Trusting Trust written a long time ago. So we've known about this for a long time. We just haven't done anything until recently. And then finally, we want to automate pipeline compliance, right? One of the biggest burdens we have is not necessarily that, making sure everything's secure, but making sure the people that care that it's secure have the information that they need to report up to executives that say, hey, yeah, this is good, or your customers, right? When a salesperson is engaging with their customers, the first thing that a high compliance, high risk organization is gonna do is ask you to make attestations about your software. And this is really tough to do if you're a small company just trying to ship features. So at Testify Sack, what we wanna do is three things. We wanna account for all the build materials, everything that goes into a build. We want to account for that. We wanna account for all the build processes. If you do a static analysis test on your pipeline, we wanna have that information let you deliver that to your customers, right? If you do a go build, right? We wanna know which compiler you use to do that go build with. That way, if that compiler has a CVE that's gonna compromise your entire organization, we can revoke privileges for that workload. And finally, right? We wanna make sure that nothing was tampered with, like what happened in SolarWinds. We wanna ensure that the build materials that we expect to go in the build actually do go into that build. So I'm gonna step back for a second. I'm gonna talk about witness. Witness is our open source project. We started about a year ago. The Genesis was actually at KubeCon NA last year. Mikhail and I have been working really, really hard on this and some software around it as well. What witness does is it implements the in total specifications or parts of the in total specification and allows software producers to make and verify attestations about the software that they produce. It has integrations to projects, open source projects such as Sigstore, Inspire, to allow you to sign these attestations without having to worry about keys. In addition, we do have a platform that we're developing that will also provide this capability. We have integrations into GitHub and GitLab currently. We have a GitHub action that's experimental that allows you to do this seamlessly. And we make it easy for, it makes it easy to produce verifiable evidence for software build. If you wanna verify that that SARF came from your static analysis tool, that's what witness does. If you wanna verify that the SBOM was produced at build time from the code in your repository, that's what witness does. And finally, we support both containerized and non-containerized workloads. We understand that not every software artifact is gonna be delivered in a container. There's the vast majority of software, whether you're embedded or virtual machines does not exist in a container. In the future, maybe it will, but today it doesn't. So what does it support? Well, what witness does is allow you to make these attestors to pretty much do whatever you want. There's just little pieces of code that describe information about a system. Currently, we have attestors for GitHub. There's currently a PR for that. GitLab, AWS, GCP, open SSF scorecard. SBOM through the open SPDX generator. SARF, OCI, so you can bind it to actual container images and then materials, what went into that on build, product, what came out of that build, and then command run, how that build happened. And that command run is what we're gonna be focusing on today. So the other part of this equation is archivist, right? You have these attestations, you need somewhere to store them. Mikhail spoke about archivists earlier today at length, so I'm not gonna go in detail, but in general, it's an untrusted store. What do I mean by untrusted? Well, I mean that all the data in archivist is individually signed. So even if the database is completely 100% compromised, if you don't trust the certificate authority on that piece of data, you as a client are not required to trust it, right? What it does, that indexes attestation into a graph database, that allows you to follow the nodes and edges of your supply chain to find the evidence that you need. And then finally, it provides a GraphQL API into all this attestation data to make querying very efficient and flexible. So right now in our release branch, we do include support for tracing within witness. We currently use Ptrace to collect traces. So if you enable your trace functionality while you're doing a witness run, what happens is we attach a Ptrace to that process and collect all the syscalls from that. So that way if when that process goes out, the compiler process goes out to go touch a file or read a file, we know about that. So we stop that process, we hash that file and record it. So we have, and then we allow that process to open it and read it and the compiler can do what it wants with it. This is great and it works really well. It's synchronous. That means we stop the process, we stop the world and we record what's happening before we let it go on. But this also does have some downsides, right? Hashing takes time. When you add a Ptrace to a process that takes time. So it really does slow down builds quite significantly. And there's also some other issues with threading as well. The actual impact in real CI systems, usually they're smaller machines in many cases, but sometimes it may double or even triple build times. So currently, experimentally, we have implemented EBPF support in witness using Tetragram. So what witness does is because it's actually executing your build process, we have that full context of what that build process is. So we know what the PID is, right? We know what we also know what materials it should be touching, right? We add that context to it. So then we send that information to the Tetragram agent with the fine traces. So we say, okay, let's trace everything that touches our working directory where our source code is. If anything changes in there, right, we'll know about it. Let's, if anything changes in there, we'll know about it, right? And so we're able to record a very accurate bill of materials for everything that happened in addition to any network calls. So we're able to prove things like, hey, does this CI process make a network call? Nope, it's a hermetic build, we can say that. Did any of the files change while the build process were happened that we didn't expect it to? Nope, so it wasn't tampered with, right? But there are definitely some trade-offs between using BPF and P-Trace. Well, BPF, that's asynchronous. So what we're having to do with BPF is we get that DRPC call that says, okay, this file was touched, then we go hash it. Well, our compilation process is already read that in by the time we've hashed it, right? With P-Trace, right, that synchronous, we get to stop the world and actually see what happens. So with BPF, we do miss some things. We do miss some file hash changes. It is, in our testing, it's pretty accurate and we do get enough to determine if it was tampered or not, but again, being asynchronous, it just has some trade-offs. BPF, it can monitor your entire system, right? With P-Trace, we're limited to only monitoring that build process, but also we don't require a root with P-Trace, right? We can run P-Trace traces in GitHub actions on a shared runner. We can't do that with BPF, right? Because we require system-level access to install BPF. With P-Trace, the coding part of it is a little bit tougher and more prone to error. With BPF, the abstractions that TetraGon has made have made it very, very easy to implement traces very, very quickly, depending on what your needs are. Our current P-Trace implementation, we're not looking at any network calls. That's a future addition that we're gonna make. Our current implementation with BPF, we get all those network calls when you know exactly where your build process called out. So if your build process called out to some malicious server, we can make some correlations with that and then we're saying, okay, we shouldn't trust it because it contacted HackChinaServer.com, whatever, and so uploading information or downloading information. Like I said before, the BPF agent, you need root on this, right? So it makes deployment much more difficult. If you're a small, medium enterprise and have a smaller security staff, you may not have the ability to install a bunch of software all over your systems because you have to maintain that. With P-Trace, right, we just dash, dash, trace and it just works because we control that process that we're calling. BPF, that requires a modern kernel. And then P-Trace, right, we have compatibility. I looked it up, right? Unix V6 in 1976 is where that was introduced. I don't think we have compatibility back that far, but P-Trace does. So I'm gonna go and I'm gonna do a demo. I got some code up on GitHub. You can go follow along if you want. Hit that QR code. All right, so this demo is non-deterministic, so you may not go that well. We'll see. So the first thing we're gonna do, yeah, I'll show you what this file looks like. Okay, so the first demo we're gonna do, we're actually gonna attack solar burst type tampering, right? So I recreated a, I built a red team tool. It's on our GitHub repository that emulates a solar winds attack, but in a Linux environment and using Go. What we do is we're looking for any invocation of the Go binary. And when we see that Go binary, we attach to it and we look for it, trying to open a file called main not go. If we see it open a file that main not go, we inject some malicious code into it. We let the compiler start back up again. And then once the compiler is done, we replace the original code back with it so nobody knows the difference. This is very similar to how the solar winds attacked. So go to the demo. All right, so the first thing we're gonna do here is we're gonna actually start Tetragon. Then we're gonna start our solar sploit tool. And then we're gonna run witness. So we're gonna name a step. We're gonna output an attestation, watch prefix right here. This is for the BPF, right? This is, we're gonna watch all the files in our build folder. Tetragon address, that's a GRPC address for the Tetragon agent. Right here, we are providing a key. Witness does support keyless. So if you're running this in a build system, you do not need to provide a key. You can configure it to use OIDC credentials. And then we're going to just run our command. And we're gonna output a file called hacked. And then on step four, we're actually gonna show that attestation. And if everything goes right, you should see the hash is changing. All right, so it needs root. So now that exploit is running. It's looking for any indication of go here. I'm gonna send this to a different screen. All right, following the target. Now we're sending the stuff to Tetragon. We're setting the trace. You can see there. You can see a bunch of stuff happened when I switched over the screen. So we have it in debug mode. So we're getting all the different events that happen. And you can see here that this exploit looks like it worked. So let's see. All right, so what we're looking for is file hashes, right? All right, so you can see the file hash we started out with is, starts with a zero A, zero alpha. Go down, go down, go down. Oh, look at that, it changed. Process two, three, eight, nine, one, zero, change that file hash from zero alpha to a six, nine, right? That's probably not good. That's our main.go. Nothing should change that file. So we know from this trace that this build has been tampered with. So we should probably do some alerting on that and do some mitigation. Because witness includes a policy engine, you can actually do that with a witness. You can write a rego policy that evaluates this to make sure this type of stuff doesn't happen and make sure this policy is applied to your build pipeline or integrate witness into something like an admission controller to make sure this stuff never gets in your Kubernetes cluster. So let's go look at the policy. All right, time for demo two. All right, so here's another use case of this technology. Let's say your compiler was compromised, right? You had a malicious actor that changed your compiler or it could just be a CVE that's really bad that makes anything compiled by that compiler vulnerable to a remote RCE, right? So what we wanna do is find everywhere, every single artifact in our system, every single workload that uses compiler. It's really difficult to do this right now. You're most likely just gonna have to rebuild all your software and redeploy it again. And if you're working anywhere that has legacy software, we know that this isn't always possible. So you see, we still have Tetra gone running there. We're gonna stop this though. We don't need our exploit running anymore. Oh, I wanna show you something too, just to show you it was actually hacked. Yeah, your code is hacked. So that's what it injected in there. All right, so let's look at this demo next. All right, so the first thing we're gonna do is start Tetra gone. We've already started Tetra gone. So then we're going to build with witness and then with the experimental BPF tracing. Then we're going to create a policy. I'll show you that policy. We're gonna assign that policy. One important thing about witness is it's different than some of the other policy engines you may know about is that we require policies to actually be signed that way. Not anyone can just put a policy in there, like a break-class policy, right? You would need to sign that break-class policy in order for that to be accepted. And number four, we're gonna verify. And we're gonna make sure that that malicious compiler was not used in this build. In this case, yes, it was used in this build. So we should get a deny on this. We're setting those traces based upon the PID of this process. We get those debug messages. All right, so we're actually using our online public version of Archivist here. So we stored it there. So hopefully the Wi-Fi works when we do the verify. So we're gonna assign that policy. Actually, let's go take a look at that policy real quickly. I just wanna show you what it looks like. So what we're doing is we have this attestation right here, this witness command run attestation. And then right here, we have actually no rego policy in it. Uh-oh, I may have booked this demo, we'll see. And we're gonna verify it. Right, so this succeeded because we actually didn't have a policy on it. Let me pull that code out of GitHub. I have an old command. I'm sorry about this. All right, so we'll take this rego policy right here. That JSON looks right, doesn't it? I think so. All right, so now we created a new unsigned policy. So let's go ahead and sign that. All right, cross your fingers. And now that failed because we had that hash in there of that compiler we didn't wanna use anymore. So now we've taken something that takes a lot of time. If you talk to anybody about how long it took them to mitigate against something like Heartbleed or Log4j, if they have the policy engine like this and they have these attestations about their software, we can really lower that time. And that's what we're trying to do here. So let me go back to my slide deck. That's it. And do you wanna have any questions? What are your questions? Yeah, so witness is just a binary that requires no root privileges at all. So what we did is we actually have a GitLab runner that we integrated with. So we pulled down the GitLab runner code and I found the part where it creates a shell command that GitLab actually injects into that container and we just instrumented that with a witness. It works great. So now if you're using that GitLab runner, every single time you run a command, developers don't even need to worry about it, right? Just if they're using that GitLab runner, you're creating some attestations. For GitHub, we did something a little bit different. It's open source right now, but still experimental. So we have a witness run action that just takes some commands and then it'll run whatever's in there, right? And wrap it with witness. Jenkins, something similar. I think we're looking at maybe making a Jenkins share library or integrating with the agent for Jenkins, but that's probably gonna happen within the next couple of months or so. Yeah, so I am not the best at Rigo. But what you're doing is you'd loop through all those shashlums, right? And anything that's in your working directory is your code that shouldn't be changing, right? Shouldn't change. And so I'd create a loop that went over and verify that all those shashlums didn't change. Any more questions? If you do have any more questions, you think about anything else. I'm gonna be around all week until Friday. I'll be at a bunch of events. You see me in the hallways. Otherwise, hit me up on Twitter. DMs are open. LinkedIn, hit me up on messages there. Or you can hit a contact us on our website, test-by-sec.com. But we love open source, so if anyone wants to collaborate, got some use cases for us, we'd love to work with you. Hit us up on GitHub and let us know how we can improve this, make it work for your organization. Thank you.