 Awesome. Good afternoon, everyone. So today, we're going to talk about runtime attestations. So that's how we bring observability into otherwise opaque container builds. So a quick introduction. I'm Parth, I'm a co-founder at Kusari. And Kusari is a supply chain security startup. And our focus is building tools that help developers create secure artifacts, but at the same time answer those hard questions of exactly what are they ingesting. Previously to that, I was a solutions architect that worked in regulated industries, as well as commercial. And I'm also a maintainer on various open source projects, such as in-toto attestations, in-toto Golang, Fresca, and Guac. I'm going to hand it up to my colleague, Shripad, to continue the presentation. Thanks. Thanks, Parth. Hi, everyone. Good afternoon. My name is Shripad Nargada. I'm a cloud software architect at Intel. I'm currently leading a few core services at Intel Developer Cloud that we announced last month. And security is one of them. And I'm also motivated to address and go on the journey of addressing security challenges in the open source communities. So talking about security, almost every discussion we have leads to software supply chain security, and for good reasons. And yes, we will be talking about that in this particular talk. So let's start in some familiar place. This is the attack factor that's come from Salsa. I think most of you have seen it many times. The work peculiarity with it, it shows the vulnerability all along the way, starting from developer, when they are building their code and committing to the source repository to the way, how we are building it, how we are ingesting the dependencies, and how we are packaging it. All the tools, all the processes, they are vulnerable for compromises, which is a very risky thing. And then how do we start addressing these challenges? So there's another good characterization that I see, again, coming from Salsa, is we need to think of our different shield into three different dimensions. It consists of how do we secure our code, how do we secure our dependencies, and how do we secure our build? And there are different maturing levels in which how we can address these challenges. In this particular talk, we are going to focus on the build aspect of this particular defense shield. And it's not like we are starting from scratch. So if you see in last few years, last couple of years, we have come a long way. Again, going back to the same threat model that we just saw, we have added a lot of security controls along various dimensions. Like when the developer is committing their changes, they can sign their commits at code repository. We have multi-factor authentication, secure configurations enabled for build. We have the desired state of hermetic and reproducible build. When we ingest dependencies, we have vulnerability scanner and the some maintainability checks through scorecard that we are doing. We have pipeline signing security. So we are basically marching on this journey of improving our security. And now let's talk about the hermetic build and the reproducible build. So build aspect of this. So this is, again, desired properties and we are not there yet. So we need, this is a journey that we need to basically go together. Now zooming into the why security of build is important. So we'll basically zoom into the what build process essentially comprise of. So if you think of a typical build process, it starts with some source code, right? Source code is input to it and the source code might be what source code we write like Python or Java code. We have some dependency manifest like requirement.txt or package.log.json and then we have some build recipe which could be a Docker file. And then this is input to our build process where we have multiple phases where this build happens. So first thing is we set up the build environment. We set up our base images. We install our build tools. Then for any unresolved dependencies that the other dependencies which are not being in the manifest, those gets resolved to their correct versions. The dependencies, their registries get resolved. In a lot of cases, a lot of organizations have their private package registries like Artifactory. So this gets resolved at runtime during the build process. And then finally these dependencies are downloaded. They compile this application and we package them. So when we talk about build, right? It comprises of whole spectrum and things can go bad in any of these different phases of this build. So what things we are trying to address here, right? So when this build is happening, we want to know that what build tools are being used, right? We don't have visibility into it because most of these phases that I mentioned, they are dynamic and they are non-deterministic. We cannot determine what the package will get resolved to at build time where the package registry or my package registry mirror will get resolved to. So these are all dynamic and they might change based on build to build, right? Based on the input configuration that we provide. So we need to know, we need to have this visibility. Like when the build is happening, all these stages are happening, what build tools are being used, whether my build environment has been compromised. Again, we don't want to go into details like, but the solar wind, right? We all heard it many times last couple of years. So we don't want to pick on them, but yeah, that is essentially what we are trying to address here. Like if your build environment are compromised, then we need some visibility. We can, something that can signal us, right? Where my solar dependencies are getting resolved from. Are there any intermediary artifact getting produced and what are those, right? And again, we talk about hermetic build, which is a holy grail that we want to assure. But if you're building it and it's one of the condition or the one of the requirement is it doesn't, it should not pull any dependency from internet, right? So how do we ensure that when the build is happening? How do we ensure or who can attest that this particular build is hermetic? So those are the challenges that we are trying to address in this particular work. So what we try to do, right? So what was our approach? So we said, okay, we have some build servers. We have this build process which is running on some build server which can be Kubernetes worker nodes, which can be Jenkins agents, GitHub runner. What about we put this monitor in between our build actions, build process and our build server? And what this monitor essentially needs to do? It will collect the trace. It will collect the trace of all the network connections that are happening. It will collect the trace of all the file operations, any crowd operations that are happening and the process executions. What process are getting executed? What inputs were there and how this process is executed? And we don't want to collect this information just for the sake of it, right? As if we don't have enough data already in our supply chain. I essentially or our objective essentially is to generate some semantic understanding of this data. When we collect the trace, then make this trace, understand this trace, pass this trace and create some attestation that we can make some statement about this build process. That this build process is hermetic or this build process made this connections or it modified this set of files. It executed this list of processes and then we can build some policies and we can make some decisions out of it, right? So our idea behind this monitor is to create that attestation which can be actionable. We can take some actions on top of it. So how do we started? When we started the design, we had different choices. And as I think yesterday, Cole was also mentioning about this. So we started with the common, the S trace, P trace, where we actually started tracing this, wrap our build process, build actions into these tools and we started tracing them. We call it in-man because now we are essentially, we are the parent process that are invoking all these build actions. But then we certainly find, right? With this approach, we need to change all our existing build tools, build actions to wrap them under this protocol. It was hard to organize. And then these tools, if you run the P trace, it will collect the information about the process, right? And we need to add a lot of decorations around it. Like this process belong to this pod, this pod belong to this tech, this tech-down task or these actions and it belong to this pipeline. That all external metadata collection and mapping, that was very hard. Then we say, okay, what about we do something out of band? So we said, okay, we start with ABPF which will run as the agent. It will collect this trace, all the trace that we need and give us this information, this information transparently. Again, the challenge was the same. The contextual information gathering was a challenge. We couldn't get that pretty easily. There was a lack of policy support, right? We couldn't customize what information need to be collected, what we need to start on and off. So we couldn't do that. Then last KubeCon, this project was an austere talk on. It was like a new kid on the blog that everyone was talking about. And we said, okay, why not? Okay, let's explore this, what this Tretragon is. And then we suddenly find, okay, this is actually the right tool for the, that we can leverage and we can build on this. But again, this whole solution is not based on Tretragon. It can be applied with any other tracing tool. And as the path will go in detail, we are basically talking about one particular implementation here. And the good thing about Tretragon is it is Kubernetes aware. When it collect the trace, it already has the additional metadata where it can, we can associate this context. And then it has automated policy control. So we said, okay, this is something that we can go with. And then once we monitor the data, once we collect the trace, our next objective was how do we format this data? What do we do with it, right? So we said, okay, we need some registration format. And for this, our options was very limited. It was actually a single option. We said, okay, we know the intro is the right approach. We will go and create an intro to our station. And then we also propose a new predicate type for runtime attestation. This is some work that we have been doing. So this is the approach that we, the design choices and the approach that we are taking. And if you think of, as I mentioned, right, so this runtime monitor is happening when your pipeline is running. So if you want to see where this particular runtime attestation is happening during the scope of your pipeline. So this particular picture, it shows the lifecycle of a pipeline, right? You start with the composition. We compose our pipeline from a lot of open source component. We install or set up our pipeline. We, the pipeline gets triggered by some events, get up events or manual trigger. And then it is executing. And this is where the runtime attestation, monitor and attestation is happening, right? And the question that we are trying to answer is, how the artifact is built, right? What is happening when the artifact is getting built? Now there are other solutions that are creating the sales attestations for build, like take-out chains or some GitHub actions that you have probably heard of. Those happens after the pipeline completions, right? So once the pipeline finishes, those attestations get created, which tells you what task actually ran for that particular, your artifact or your image, for instance, that get built. The runtime attestation is happening one step, one stage before this sales attestation that are getting generated. And again, as you will see the, how these two technologies merge together. And to talk more, I will, in what parts to talk more about it. People I was saying is, we wanted to use Tecton pipelines. So for a general audience, if you're not aware of what Tecton pipelines is, is that it's a CI CD, a cloud-native solution. It builds everything in terms of containers. And it's very easy because it builds everything in an ephemeral container that can be destroyed and isolated. It also has the ability with Tecton chains to create attestations, right? It creates sales attestations, signs sales attestations that you can push into OCI and in other directions. So we started off with this tool in mind because we're gonna be working with TetraGon. TetraGon is Kubernetes-aware. Tecton is based on Kubernetes, is built on Kubernetes. It does everything within pods and containers. So we can utilize those two technologies together to create our runtime attestation. So like Shri Pao was saying, TetraGon is a tool that we chose in this case. It's an EBPF solution, right? Because it observes using the EBPF technology. We don't wanna fix to TetraGon, right? So we wanna make the runtime attestation very generic. But in our proof of concept, we chose TetraGon because of the benefits that it provides out of the box. At the same time, right? Because it's Kubernetes-aware, it can see the pods, you can see the processes running within the pods. At the same time, right? It can see file access, network connections, right? It can see a lot of those things that we do wanna collect. So we don't wanna introduce another controller, right? The Kubernetes space is already full of controllers. We don't wanna introduce another one. So what we decided is that let's just have the TectonChainsController be the one that creates that attestation for us, right? TetraGon is gonna collect all the logs for us, communicate that over to TectonChains, and it's gonna create that sign attestation for the runtime attestation and then push that into any kind of direction, right? TectonChains already has the ability to create attestations. It already can do sign, right? It can already sign attestations, push to OCI or push to a document database. It has that ability, so why not utilize all those capabilities instead of starting from scratch? So that's what we decided to do. So let's get into a real-world example of this. So real quick, starting off, I wanna show off that this is a simple Kubernetes cluster. You can see all that's running is the TectonChains, like I mentioned before, to TectonPipelineController, which has two components, the controller as well as a webhook, and then TetraGon itself, right? Those are the only pieces that are running. We're not introducing anything extra besides what's already coming out of the box with TetraGon as well as Tecton. So what I'm gonna be showing off here in this pipeline, basically, is that it's a multi-step pipeline and it's going to go through and it's gonna clone down the repo, it's gonna build the artifact, it's going to scan it using Grape, it's gonna create an Svom, and so forth. It's gonna do all these different steps, and at the same time, what's happening is that TetraGon in the background is actually capturing all the processes that are running for each one of these tasks. If you remember before I said that TetraGon is Kubernetes-aware, right? It can see specific processes within a task. Each one of these tasks are running individually in each one of the individual pods, so they can collect all the processes that are running, and we can go and actually analyze them in the future kind of thing. So it helps us, oops, sorry, got that. So in the future, it helps us go see exactly what's happening, what happened in each one of these individual tasks and make policies based off of it. So you can see it just finished. All the different steps in this pipeline completed. So now let's take a look at the actual result. So this is the build step, right? The build task that actually ran for this specific pipeline, and you can see the main point is what I want the audience to focus on is these four things here. The pipeline, or I'm sorry, the payload is the salsa attestation that TectonChains already generates, right? So that's something that we're generating, that's TectonChains doing that, as well as the signature down here. The new thing that we are adding is these two pieces right here, which is the runtime payload, as well as the runtime signature. Like I said, TectonChains has the ability to, right, Tetragon is collecting this information, TectonChains will take that information and create the attestation for us, and it's appending that into the annotation here in this task run as a label. We can pull this down and actually view it, but we can, in this case, right, it's actually getting pushed into the actual Kubernetes object. We can push this into OCI or some other object in the future if you wanted to. And because it's signed, right, we can go back and use cosign to go verify, yes, this is signed by my private key and that is valid. So taking a look at the attestation itself that I generated, so what I did is that I cut this down for size, right, because it generates a lot of data and I'll talk about that, you know, why, what we want to do about that, but for now, kind of talking through what this, what the actual attestation actually looks like, right? So like Shupa said, we started with the intoto attestation format, so you can see that the subject is up here, what artifact that it built, right, the pipeline built. And then down here is a specific, right, starting from down here, the predicate type is a specific, the runtime attestation itself. So the first thing we see up here is the monitor ID. And like I was saying before, is that we wanted to make sure that we made this attestation very generic so we can choose different monitors, right? It does not have to be touch or gone. It can be, you know, P trace, S trace, some other tool in the future kind of thing. And then down here is this specific thing that's actually building this artifact, right, building this attestation. So in this case, right, it was tecton chains, but again, it could be something else. It could be GitHub actions or GitLab or something else coming down the line. And then this is the actual task run object that we are actually looking at that I actually collected these logs for. Down here, you see tracing policies. So what this kind of does is out of the box, tetragon has some capabilities, right? But if you wanted to, if you want to include more, like for example, if you wanted to include network connections, right? If you wanted to trace network connections or if you wanted to check file access and such, you have to give it a tracing policy that says, yes, enable this and look at this information and capture this information. So what this is doing is, okay, we also want to know exactly how the tool itself, right? Tetragon or some other tools configured and how is it capturing that information or what was it supposed to capture? So for example, right, if I don't see this tracing policy, if I don't see this TCP connect policy here, right? If I don't see this one, then I know that, okay, it didn't collect any network connection, right? It didn't collect any kind of network process logs for me. So I can't say for certain that is hermetic because now it's not collecting anyways. But if I see that this tracing policy existed, there's a TCP connect policy, right? And if it did not collect any kind of network logs, then I can say for certain that it was a hermetic build. So that's why we want to capture this tracing policies. So down here, if I scroll down, is the actual processes themselves, right? So we kind of broke them out into event types. If there were file access types, exit entering, you know, what kind of binary was being used at the time and then what kind of arguments that we're taking and if there were privileged calls themselves. So if I keep scrolling down, you can see an actual example of, because I had that, the network policy active, right? You can see there's a TCP send message here and it did actually send message to some kind of public IP address, right? So right away, I know that this process was not hermetic and I can't make that claim anymore. So what do we do with all this information? Like I was saying, like it's information overload. I had to cut down the size of that attestation because it was pretty massive, right? So it seems like, okay, it's gonna give us a lot of information back, but it's collecting information for information's sake, right? We need to do something with this information, right? So because there are one time attestations and logs themselves are pretty massive, right? They can end up being 500 megabytes. They can be, you know, they might contain confidential information that maybe companies don't want to distribute or they're even hard to distribute because they're so massive, right? They're 500 megabytes or something. So what do you do instead? So we're actually in the talk in the intuitive attestation group. We're actually in the talks of maybe using, you know, the software supply chain attribute integrity, otherwise known as Sky and kind of calving and, you know, for example, in here you can see that there is an attribute that says it's hermetic, right? So saying like, okay, this artifact was built, this subject right here. Oops, sorry about that. This artifact here was built and is built in a hermetic process. And the evidence that says so is this, the runtime attestation that we generated that stored somewhere else, right? That could be stored in some kind of other public database. And then you could go access if you want to manually check. But what we do instead, let's say for example, in the future is that Tecton chains, right? It generates, right? It goes through the runtime logs to check to see if there are any kind of TCP connect calls or anything being made. It checks to see that the tracing policy was actually active. Yes, it's supposed to collect network logs. I didn't see any kind of network logs. Then I can be for certain that it is a hermetic build so I can create this smaller attestation and use this instead and distribute this out with my artifact and push it into OCI. And then in the future, we're thinking about incorporating this into Guac, right? Guac is graph for understanding artifact composition. So you can see from this diagram, it's, you can see the artifact in the middle, which is that orange piece. You can see there are two attestations. You can see there's two lines pointing towards where attestations versus solstice attestation as well as runtime attestation. And you can see it was built by and monitored by. So what Guac we're trying to do with is basically having all this kind of information, right? Like I said, we have this information now. What do we do with this information? So now we wanna check, you know, for example, I wanna have a policy that says I only want to make sure that I run any kind of artifact that was built hermetically in my production environment that I can go query Guac for this kind of information. And then it gives me back with confidence saying like, yes, I saw there's an attestation associated with it. There's a runtime attestation associated with it. And I can now run this in my production environment because it meets all those policies. And I'm not gonna go too much into detail, but you can take a look. My colleagues are gonna give a talk about this on Thursday. There's a QR code you can scan, but this talk is gonna be on Thursday if you wanna learn more about this. So the road forward for this is that we definitely want to make that. So we have a predicate proposal in place right now in Toto, in Toto Attestations repo. It's currently in draft, but we wanna make it more generic. So at this point, I think we're looking for community feedback, involvement from others to see exactly, okay, what other information do we wanna capture? And does it encompass everything that we need? So we're working on that piece for this specific proof of concept that we were working on. There's one of the things I was working on is getting the tetragon policies. So that tracing policy that I showed earlier, that was a quick code change that I did, but basically when we want a formal PR that's gonna be put in, that kinda gives you more information in a structured format of exactly what configuration tetragon is set up with. So that's that proposal. As well as formalizing the change, right? So I showed you the change interrogation with tetragon, push that informally into tecton chains so that people can start utilizing it. And then lastly, the guac interrogation, so we can use this information to create policies and then use that for governance. So a quick recap, right? So we wanted to have visibility into the build. So we wanted to use, we wanted to integrate with existing tools. We don't wanna introduce anything new. So we can use tetragon, tecton chains to create the signed runtime attestation that we can use and then take that information and so that we can generate the attribute integrity, that sky attestation that we can use to check to see yes, if this is hermetic because the runtime attestation sets so and here's the evidence and here's the evidence for the runtime and you can go query it manually if you want to. So definitely get involved. We have our own Slack channel in the open SSF, Runtime Attestations and here's our Twitter handles. So definitely contact us if you have any questions or if you have any comments or if you wanna collaborate more. Thank you. Any questions? Thank you very much. Anyone have any questions? One over there. I'm running. And real quick before I forget, we do have t-shirts so if you guys wanna pick up a cool Kusari t-shirt, see my colleague in the back and get you that. Great. Yeah, great talk, very interesting stuff. I like the hermetic build part of it and there's also a lot of other ways that people have talked about doing things with build systems to try to protect them better like for instance reproducible builds or having some kind of hardware root of trust that you use on your build system and stuff like this. And I'm just wondering if you've thought about would that use the same mechanism that you're planning that you're proposing here or would you have separate mechanisms for them for some reason? Yeah, so I think for reproducible builds like Aship was saying, we definitely use this mechanism because we can again go through and validate that yes all the same processes is ran for two separate builds so they were running to validate that they were reproducible. So there are definitely more use cases for this, right? In this case, we chose hermetic because I think it makes a lot. It's easy to connect with and easy to understand but definitely there are definitely more use cases that we wanna explore in the future with this. Any other questions? Right over here. So there was a presentation yesterday on testify sec archivist. Yes. So how does this compare and contrast with what? So Cole and Mikhail are actually working with us so they were just on a call with us last week so they are also going to take this. So we are gonna like we're working together on this in terms of we wanna formalize this attestation format, right? The predicate type so that they can also produce the same type. So we're gonna work in collaboration so that's what I was saying. So we wanna make this a standard runtime attestation that people can use in other contexts like he's actually using tetragone in this case, right? But you can and he's using another his own witness tool, right? So it doesn't have to be chains and tetragone. It could be something else and everyone kind of creates a runtime attestation that everybody can benefit from. Okay, anyone else? Going once, going twice, great. Well, thank you very much. Thank you. It was a great talk. Thank you.