 Okay, I am Santiago Torres, I am a professor at Purdue University and my college, I'm here. Oh, the Thierserys, yeah. We're going to be talking about Intodo. Intodo is a, it's been in the CNCF for a couple of years now, but it's always been a pre-scient project. It's been trying to solve software supply chain security even before solar winds, even before software supply chain security was such a large problem. So, let's talk a little bit about what we set off to solve. Intodo is pretty much focused in trying to solve software supply chain attacks. We talk a lot about software supply chains nowadays. We understand that software is interconnected and that we depend on other people's work in order to make good software, right? Now, a software supply chain attack is something that, even though there's not a very good definition out there, it's something that we're constantly redefining and understanding. It's basically the way in which hackers try to attack the left-hand side of an equation, say, your build system, your source code repository, anything really that produces software in order to attack the consumers of the software, essentially the result of your build, anything that's actually consequential to your software delivery pipeline. Now, this is a slide that we update every year because they're constantly increasing. I remember when, I think, two years ago, we were in a cumulative 400-something percent increase in software supply chain attacks. And even though we have a lot of technologies trying to solve this problem, they keep on increasing. We still keep on seeing software supply chain attacks happen. This is also a slide that we update every year, unfortunately. But this is also something that, as the entire project we've been working on for a while, which is trying to catalog and understand how software supply chain attacks take place and what shape do they take. This is a project that we donated to the CNCF Tax Security Group, I think, in 2019 or so. Even though it's not an exhaustive list, it's a very, very good resource that talks about how software supply chain attacks take place. How are they actually carried out? And while we use this ourselves to inform the design of Intodo and in order to essentially understand the nature of a beast to try to tackle the problem, I really, really recommend looking at it. Now, from this information, well, we know that there's many good solutions that came out there. You may be familiar with some of the logos here. There's things such as Salsa and Tecdon Chains and SBDX, software building materials in general. A lot of people are excited about projects like Wac or Skit or Sixdoor or Gitbomb, all of these things have a place in trying to tackle the software supply chain problem. On the right-hand side, we also have policy engines. Now, in Intodo, in and by itself, it's a metaframework. It's something that allows you to connect all of these pieces together to verify software supply chain security properties. Go back. For example, here, this is probably a very cartoonified version of the software supply chain. We have a version control system. We have a build system living somewhere. We have probably a CI checking and running the tests all the time. We have some packaging infrastructure. We're putting a package out there. This is something you're familiar with. Now, being able to connect all of these things together to write policy to say, well, I need a version control system with the security properties in place. I need a build system that needs to behave in this way. I need a packaging infrastructure that needs to be run by this individual and it needs to be behaving in this particular way. Well, that's something that you can easily express using Intodo and the semantics of Intodo to essentially tell you what's going on. On the other side, we also have libraries that allow you to capture and express the information that you collect during your regular software supply chain operations. For example, you saw the policy earlier. It described what needed to happen on this particular piece of data that you can collect using Intodo libraries or integrations that we're going to talk about. In a minute, you can start collecting these links of the supply chain, these pieces of evidence that allow you to verify the policy. For example, if you're familiar with Salsa provenance at stations, that's an Intodo predicate. That's something that you can use to describe the build system in a very expressive way and write policy around it using Intodo policies. Using these things collected together, then you can essentially compare, what's on the left-hand side, what's supposed to happen with what actually happened. You take the original layout of the supply chain that you wrote using Intodo and the collected evidence and then you contrast the two things together to check whether those things match letter to letter. In order to show the power of this, we actually prepared the demo. Again, this is another very cartoonified version of the software supply chain, but I assume this is something that you're doing probably every day, right? You have a couple of colleagues. In this case, Bob, Alice and Carol, they're working together to make, I don't know, a new release, package it into a Docker container and release it somewhere to eventually make it part of your cloud infrastructure. In this case, Bob will clone the repository, will do a couple of changes to the code, and he will submit a pull request. It's going to be reviewed by Carol and once Carol finds it appropriate, she's going to merge it, and then Alice is going to tag a release and trigger a build on a server to eventually make it into a Docker container that will eventually verify to make sure that before we put in the cloud, it's correct. I wanted to give a shout out to our two students, Alan and Rui. They actually did this demo themselves. I am super impressed of all the work they did this couple of months. Should we just kick it off? So you're going to see a lot of text. I'm going to try to explain what's going on. Part of what I like about Intodo is that if everything's working, you're not supposed to know as an end user, but we're going to make it very, very both. The diagram that you saw just a second ago, cloning, changing some code, making a PR, and so on and so forth, is going to be verified. Every single step is going to create a link. The little files that we showed a second ago. So we can actually build a very granular provenance graph. Let's kick it up. So first we define a demo layout, which is, again, this policy of what needs to happen. We just created it. It's on root.layout.file It's going to be living somewhere. You can think of this existing inside of, say, an admission controller or something of that nature. And now we're going to actually kick start the process of running over our supply chain. We're going to get cloned now. We are going to make the changes. If you see somewhere in the middle of the screen right where the pointer is, we use Intodo to trace the process and capture a piece of evidence about cloning and changing the code. We will continue doing the same. For example, in this case we'll track to get commit. We change the code. We track what commit that it baked it into. So now we know these files are connected to this commit. Then we track which PR that it didn't make it to. We'll say we'll commit it into this PR, which is going to be reviewed by somebody. And then we track as well. The next person in line, Carol, she looked at the PR. She tracks the PR she looked at. And the eventual merge commit that ended up being a result of her review and her merge action. And now we are now we're tagging a release. We are going to build a container and we do the exact same thing. We track the commit that eventually became a tag and then we track the tag that eventually became a container. And we built this paper trail of exactly what's going on and where the things come from. The idea is that you can create these policies and these layouts to describe at a very granular level how is your software supply chain supposed to work. And then you can produce evidence and ask your developers or your contributors to trace themselves what they're doing so that nowhere in between anybody's changing anything. Now we run the verification, which is the last step. We take all of the evidence that we collected and we compare it against the layout that we described in the beginning. We said there needs to be a commit there needs to be a PR that needs to be reviewed. There needs to be a tag that needs to be the result of that PR and there needs to be a container that's built using that tag. Oh, this is a part of the demo where we're going to actually show how essentially we're going to show how when Toto is able to detect if someone comes in and actually tempers with that workflow we just displayed and maybe puts in a commit that wasn't actually supposed to be there. And we're going to continue with all of the previous steps again build the container again from that previous tag and everything, but when it's time to actually verify everything like verify all the metadata that was just collected, we see that well, in total verification didn't pass for this one because there wasn't a commit that wasn't supposed to be there. Right, so basically the goal of Intotis to detect if there's any breakage in this paper trail if something happened in between that shouldn't have happened Intotis able to detect it it actually prints a very user friendly trace that says like here's where we started, we started with this code changes and eventually I got into this commit and I couldn't make it into the tag because there's no link that connects one and the other. Let's fast forward. Now this is the result of again, a lot of work from Jean-Druy and Alan this was also something that we're very happy to do if you're a student or somebody interested in participating in Google Summer of Code we started this particular demo and this particular extension to trace up operations more granularly as a Google Summer of Code and I think it went very well. This is one of the demos that we'll be showing today. Now this is the state of Intotis, this is what you can do right now. I'll let my colleague Aditya talk about it. Do you want the feature? Hi, so Intotis is primarily a specification that there are a number of implementations that we manage. We maintain four implementations and well four different languages and the specification itself can be extended using what we call ITES or Intota enhancements and a good chunk of these end up being implemented as well in one or more of the aforementioned implementations. So we've got implementations in Python, Go, Java and Rust. The Python implementation is a reference implementation. We hit the 1.0 milestone back in I think late 2020 and that's the one where we make stability guarantees and things like that and the Go implementation are slightly more experimental implementation and that's where a lot of the newer features and newer ITES get implemented before making their way to the Python implementation. The Java implementation and Rust implementations are a little less feature-complete but they're important for integrations with things like Jenkins and the Rust implementation in particular is used with the reprisable Bills project for integrations with things like RebuilderD. So we've had a number of ITES that have been accepted. We've got several more that are currently in draft status. I'm going to talk about a few of the key ITES here. ITE 2 is one that describes how you can use another CNCF project or Intodo sister project, Tuft as a root of trust for Intodo and what this allows us to do is to securely distribute not just the artifact but the metadata, Intodo metadata captured for that artifact using Tuft properties. It also helps us distribute things like the keys used to verify Intodo's supply chain securely and helps identify which batch of Intodo metadata must be trusted for a particular artifact and things like that. And this isn't theoretical. This has been implemented and it's been used in production for years over at Datadog. ITE 3 is like Sisterite which is informational and contains all the details of Datadog specific implementation of Tuft and Intodo in their pipelines. We have ITE 4 which enables all the different artifacts in Intodo. We just saw that in action actually in the demo that Henry and Alan put together to capture things like GitHub specific objects and not just Git specific objects and recording containers and things like that. It's also been implemented we've also used ITE 4 to do things like look deeper within files rather than just at the file level situations where there was a config file that needed to be modified and the modification needed to affect just one specific part of it and so we could use Intodo to validate that some fields that weren't supposed to be modified weren't modified. We didn't want the base image for example to be changed and that particular instance to something that's more vulnerable and things like that. 6. This is I think the one that's really like open Intodo up and enabled it to become the common link with a bunch of other software supply chain security projects. It allows for defining more generalized attestations. We've got a number of like we're it's a little in flux but we've got a number that are already being used like the salsa ones and we're animating Cyclone DX as an Intodo attestation as well. That's the salsa provenance spec but we've had a number of them come in like at the original Intodo links SBDX, Cyclone DX Sky and runtime trace predicates which was talked about yesterday if someone was here at SecCon but yeah and we've defined a new process that folks can contribute to these predicate types. It's a standard line 889 and we've got a number of people in the broader Intodo community helping provide feedback to new attestation types and help curate what type of attestations we need to for a particular use case and things like that. 877 is the one that added support for X519 certificates to Intodo. This enables integrations with things like full CO, six rows full CO or spiffy spire and any other standalone PKI setups that folks have and want to use Intodo with. As a matter of fact, the spire integration is already live in our Go implementation as I noted that was the experimental one so that's ready to use there but yeah. So I'm gonna like wrap up sort of with saying that a lot of this wouldn't be possible without the broader Intodo community with like you know the multiple lights they've authored and the contributions to the various implementations and new attestation types are coming up and I wanna talk we wanna show another pre-accorded demo that displays Intodo's integration with KeyLine another CNCF project so let me just pull that video up. Well he's pulled it up. KeyLine is a system that essentially does hardware based tracking of whatever's running on your cloud so you can easily connect this to projects together and say well I don't want to run anything that doesn't have any total policy attached to it and that I don't know where it came from. I want to know that every single line of code running on my infrastructure was reviewed by say two people or think of that nature. I'll let Mark describe the rest. Hello Intodo community, my name is Mark Bistofros I'm a developer engineer with Red Hat and he offers a CTO as the emerging technology security team and I'm here to talk about the work I've been doing right and supply chains with Intodo and KeyLine. I'd like to start by talking about the supply chain security ecosystem as it stands. For the past year and a half there's been a lot of great work on tools like Intodo, Topin 6 store around the beginning of supply chains. These tools do a great job of recording, testing and propagating information about code origin and this makes information about source code easily accessible and available, which means that PSOM ran a binary that used these tools and supply chain. A consumer of that binary can have very high confidence about how that code originated and where it came from which is a really cool story. On the other side we have a great story for the end of supply chains as well which is KeyLine. If you're unfamiliar with KeyLine it's a CNCF hosted project that provides a highly scalable remote boot attestation and runtime integrity measurement solution. KeyLine enables users to monitor remote nodes using a hardware-based English well. KeyLine is a toolset that allows end users to monitor what's happening on some client system and to attest that what's happening on that system, the things that are running, machine state matches what is expected. And that guarantee is backed using a hardware root of trust, which is pretty great. That's a very strong guarantee that what's happening on that client system is exactly as expected. The issues these two stories are not that well connected. With internal sticks are tough, their chain is mostly kind of end after distribution. There's really no runtime guarantee that running code matches what was compiled. That's kind of missing from that chain. On the other side KeyLine does provide that, but it still relies on user specified policy to enforce its hardware guarantees. And these policies there's not really a clear story for where they come from. They could come from the operating system render for example, which might cover the baseline or OS image, but that may go on top of your base OS. And that's what I've been working on, kind of connecting these two sides of the story with the tool that I'm calling the supply chain bridge. This is the missing middle. This connects the two parts of the supply chain together. You can see that happening in the sidegram on the left that's visualized with let's say you have actions pipeline that takes your source code, uses the wonderful tools in total and stick store to sign and upload signing evidence to a transparency log produces a bunch of artifacts. So in addition to compiling a binary, it also gets signatures as on the lake files from in Toto. Now let's be attached to a GitHub release. The supply chain bridge will take the signing materials that are now publicly available, verify them and kind of test that the providence of a specific binary makes sense and is correct. And once those checks pass, then it will forward the hash of that binary to KeyLine which can then monitor for it on end user hardware. Which then closes the loop and creates a continuous chain of trust that's end to end, which is really the goal from source code all the way to hardware. And that's what we're going to what I'm going to shoot today. We still have two repositories to show you. The first is supply chain pipeline demo. This is just a fairly basic repository that is sort of a prototypical example of what good supply chain security practices look like. It's got a very basic hello go binary as part of it, but more interestingly it also has some intono keys and most crucially a workflow that makes use of the wonderful supply chain tool demo to us. So it compiles, this workflow will compile the go binary and a test or record that that complication happening with the tonal, producing a link file and it will also sign and upload that binary to six store using the wonderful GH action six store Python action by trailer bits. This produces a lot of sign materials and these sign materials are attached to make it hub release. You can actually see if we go back to the releases page. That is in fact the case. The blaze released has not just the binary but also certificates, signatures and link files. Which is great. This is the information now publicly available on the internet and people can see it. The second repository is the key line supply chamber itself and this tool consumes that GitHub release, consumes the sign materials from it, verifies them and takes the binary that is part of that release and takes the hash and forwards it on to key line which connects the two, connects key line to the rest of the supply chain ecosystem. And we're going to show it in action right now. What we're seeing here on the left side of your screen you have, we have two virtual machines on the left side is one of them called key line Fedora one and on the right side is a second called key line Fedora two. Key line Fedora one is running two separate things, running the key line verifier in the middle here and the key line register on the bottom and the key line Fedora two on the right is running the key line agent. The verifier and register on the left running on the one of the two command and control aspects which monitor and and orchestrate the agent. So this is an agent that's working on a separate machine and the agent is what does the job of hardware attestation and making sure that everything just looks good on a client machine. So key line Fedora two is the VM that we're going to be monitoring today. To facilitate this demo I have a few files on disk to show. This is a key line policy that corresponds to the base OS image of key line Fedora two. So this has all the system files and whatever else that is part of that base OS image. Additionally I also have the hello go binary that we saw present on both machines, both hello go binaries are there. And also we finally have the key line supply chain bridge which is the repository of the tool that I wrote. And we're going to see that in action right now. I'm going to run it. You can try to bridge slash main dot p y and provide a number of inputs. We're going to give it an owner and repository on github. So this corresponds to MSRO slash the pipeline demo. So I'm going to get towards this repository and what's going to do is going to look at this repository, find the latest github release and then grab some materials from that release and then verify them. In addition I specify a local path to a binary. In this case I'm going to specify hello go. This is not required. You can leave this field out and it will just grab the binary from the release. In this case they do have it on disk so I'm going to have that entry there. I also specify a destination path. This is where the binary will end up on the machine that will be monitored, which in this case is this iron right here. So that's just at group hello go as well. And finally I'm going to give it the alleles which is the base layer OS policy. I'm also going to assign S and dash i to indicate that I'd like to do six more and in total checks. When I run this we're going to get a bunch of outputs and we're going to see what that looks like. Once we run it you're going to see a number of things happen. We are verifying the local binary at root hello go right here. I got sign materials from MSR slash supply chain pipeline demo. We are going and we're going to do a bunch of checks against that. We're going to do basic signature validation which did pass. We're going to verify against six doors. So presence of out signature and inclusion proof and it looks like that pass as well. And we're going to run a guest in total and it looks like that passed as well. At the end of this, since we've run all these checks, we can reasonably confident that this hash of the binary is correct and has province that we trust. So we're going to then forward this hash over to a key line. We're going to take the existing alleles that's present at real also create a new key line policy that has this hash. And in fact, if we look we do have this policy and if we print this out we're going to see a bunch of hashes in addition to addition to at the very end root hello go which is the which is the hash of the binary that we just compiled which is great. So that means that this key line policy now has information about our verified binary. The last step is to actually run key line and we are going to do that now with an invocation of the key line 10. We are going to provide it with our new key line policy that Jason and we are going to not delete you're going to add this policy. And this is going to invoke key line to do a bunch of things is going to set up the agents and the agents is now set up with this policy as now checking constantly checking the integrity of key line for door to door. And it's saying that everything is good the checking on a measurement list on agent text loops around and if we keep seeing that that means that everything is in good shape. And to demonstrate that things are working as intended you're going to run hello go by doing so we ran our code and everything is fine everything continues to work as expected which is great because that means that our province information that started the code made it all the way down to hardware and is being reflected when we're monitoring it. But we can actually go further than this we can we can show this working incorrectly so let's say we modify hello go let's say we do let's say we add this text the end of hello go once we do and we run hello go we're going to see that key line actually recognized it and it tripped it and it tells us that root hello go doesn't match the expected hash it in fact is different which is great that's exactly what we want it means that key line work does intended and that's it that's the demo showing that product information from source mail all the way down to to the end hardware so this is a pretty cool demo in my opinion but there's definitely some work to be done in the future there's definitely some work to be done around making this more seamless and more usable and there's definitely conversations happening in the key line in the Toto communities to upstream parts of this tool into the main code basis to be pretty cool and I'd like to continue doing that work in the future thank you very much so this is all very exciting again part of what I like about in Toto is that it really does play very well with all of the software supply chain technologies pretty much security products out there we actually have a lot of integrations with other parts of the CNCF landscape you just saw something with key line tough is the sister project we have pretty much being the home of salsa since its inception if you look at the salsa provenance at the station at the very top it says in Toto statement version 0.1 that's that's how we started to make things happen cycle in the X, you saw the extensions beyond on the open source ecosystem we are also a technique to say verify reproducible builds you can collect provenance at stations from different builders and agree and check if they agree on what the reproducibility status of a project is and well you will find us in many many places we are a very welcoming community for you to try to explore and experiment with different software supply chain solutions please, well I invite everybody to join us there's many ways to reach us and even though there's only a minute left you can also hack with us tomorrow in the Toto tough and six-door contripest I think that's a great way to get started in space we are three sister projects that are trying to set the tone for software supply chain security in the coming years and thank you I think we have very little time for questions if not at all, five minutes okay, great it's actually more of a clarifying question in the demo when he actually made the change to the hello world to like hello he will do something apparently Keylime policy engine caught it and it says the hash was mismatched did it stop execution I did not see hello, you will come in under right, so my understanding and that's a little bit more on the Keylime side the default is to just let it go and warn you so the kernel actually lets the execution go and then it tells the other part of Keylime to tell you that that happened it should be possible as far as I'm aware to change that behavior instead of warning you just killing the process I was just wondering if the guy was talking really fast I was hoping I could get the GitHub or his GitHub account that he was showing that demo I was looking online at the event schedule I think the PDFs or the presentation sorry, I think the speaker is facing that way I couldn't hear you does the individual that did the demo do you happen to know his GitHub URL Mark Bestabros so I think his GitHub handle is M-Bestabros M-Best M-B-E-S-T-R-V sorry, my accent V-R-O-S thank you great I have a question about how precise the layouts for Intodo need to be for verifying what comes, or verifying what you produced so in particular I'm thinking of like two different scenarios if you're using some sort of ephemeral key generation do you need to know what the keys are in advance in those layouts or if you have a the other scenario, if you have a multi person team and you don't necessarily know who on that team is going to perform a particular action do you need to know that in advance in those layouts as well so let me see if I can rephrase the question basically how granular do you need to be in your layouts and also in particular what's up with the identities of the actors probably you'll know beforehand who's going to merge but you know that somebody needs to merge from that company or something like that yes, there's three semantics that help you in Intodo the first one it's a little bit more about the granularity you can essentially say I don't know how this part of the supply chain works but I know this people will give me a layout to describe it so you can make like recursive calls into Intodo and that allows you to like abstract out parts of the supply chain that you don't know maybe you're like a PM you don't you really don't know what's going under the hood but you know the team that's going to be working on that the other one is you can actually give it a set of keys and a threshold so you say I don't know ten people are going to be working on this and I need at least two of those people to give me evidence of a merge I don't know which of them but I want two of them and the last one is the i7 which is the one that allows you to work with identity providers so say if you're using Keyless and Fusio you say well I don't know what key is going to be there but I want it to be from this email account and it also works very well with Spiffy Inspire when you have a familiar workers so you say well I just want a cert this chain rooted in this trust it's going to be some worker on some machine in the cloud I don't know exactly which but just verify this chain of trust all the way to this root and with this particular constraints on the cert chain was that kind of awesome anybody else wonderful session thank you