 we're here to talk about getting started on supply chain security oh it's really not that hard to get going um so i'm tim miller senior supply chain engineer at the city bank um i'm michael Lieberman i'm also a senior supply chain engineer here at city as well as one of the architects leading up the cncf security tags supply chain working groups reference architecture mike and i were not next to each other in those photos that's just like really weird all right so uh what we're going to talk about is why we care about supply chain security we're not selling anything we're a bank we're an end user so why are we here what you need to do to get started and most of our talk is going to be about a demo uh walking you through this um we'll wrap up take questions and um all right so what so why do we care supply chain attacks affect everyone this really is a chain every time you all are compromised it hurts us vice versa we recognize that problem and it's really it's a real thing um the from a regulatory perspective as well banks have to deal with something called um a lot of a lot of banks call this controls basically there's just a bunch of rules on not doing an unacceptable thing like having developers just run stuff right into production it happens it just so happens that a lot of the things you do in supply chain security are very correlated to the sort of thing following those regulatory requirements basically making sure that the segregation of duties and in addition being able to prove that you know who did what when it happened and oftentimes people result or resort to paperwork to to accomplish that goal today so we kind of get two birds one stone with being more secure and having to do less paperwork which i'm a very big fan of and then um and lastly we're not special we don't have different concerns than anybody else developing software this is a generic generic problem for everybody we just have some of our concerns dialed up to 11 all right so getting started um the first thing you do is don't try to solve it yourself uh solve one of the great things about something like salsa and there are others but it helps you think about how to start like you don't have to sit there and flounder around trying to figure out if you should be shooting right for reproducible uh that's probably not step one um so really leaning into that and using it to apply that to your environment is a really practically helpful thing a lot of just starting generating that data not even doing it just having it is a very good first step um separately deciding who it is that you trust like a lot of this stuff really ends up just being not which libraries you trust but the organizations and people that are producing it you trust them or do you not trust them um starting to put that thought exercise down first is really practically necessary these signatures don't help you if you don't know where they're coming if you don't know what what what it's telling you it's all used so really put that thought in um and then just start going well so we'll we'll we'll we'll show you what that looks like you um but the one pitfall that i think we'll talk about as well is you have to be really careful uh otherwise these these attestations can effectively lie to you um if you don't know what it is that you're asking validation for you can really mislead yourself far more secure um all right so now we'll get into the fun stuff okay um actually tim can we just swap the mics there no i apologize for your ears we'll still hear can you all still hear me all right um so uh as i sort of shared in the um the slack channel uh if you want to follow along with the code um the code is in the slack channel but uh just to kind of give a very very high level overview and a lot of this is stuff that's based on a lot of the work in reference architecture from the cncf if you want to learn more about that there'll be some discussion about it tomorrow at cloud native security con but in and um yeah and so just to give you a high level overview though using a lot of the tools that were kind of shown off earlier today so that stuff this is stuff like tecton tecton chains um you know using uh s bombs and then all that sort of stuff using cosine design images so um let me just sort of walk you through it so if i just sort of run this right now uh it's just going to be running a a build and just to kind of run you through what this build is supposed to be is it's building a docker file and uh can everybody see do i need to make it any larger okay so it's just building a basic rust application and that rust application is this hello world demo and so um what's happening okay so it's building it it's going through a bunch of steps here it's generating a cyclone dx s bomb it's uh using canico to um build the image hey sorry can you wait one second we're having some virtual things i don't know that knows how the virtual things work they can't oh can they not hear us or something okay um so uh it built the canico image it's doing all this sort of stuff it's um signing the s bomb and attaching that s bomb to the actual image repository so it's going to live alongside so the s bomb is actually living alongside the actual image and it's doing some uh you know verifying the digest and so um if i go in here and uh if i do an ls like a crane ls um of my image right and just for for shorthand i just took my image here um you can see this is the image it's the tagged with latest it has an s bomb it has a signature for that s bomb it has a chains at a station that's uh you know attached to it and it has also a signature so if i wanted to go and verify any of these things right i can go and let's say um if i want to get the s bomb i can use the s bomb signing key um to validate this and you can see um this is just for the sake of the demo uh an input s bomb of all the source files okay cool this this is this is valuable information it lives alongside the artifact great now if i go back here again and i do uh now let's say a cosine verify of some of the other attestations and some of the other signatures you can see that so i can validate the signature of the image yep it's valid um and i can also verify the attestation and you can see there's some stuff in there as well and if i want to just sort of unpack that attestation let me just do this and as you can see it's pretty much what you know Priya and others had shown with with chains this is all that that metadata in a predicate format and so on great and some other things that we were doing in here just to kind of show you is using admission controller to validate that the only things running in the build environment are signed by the tecton key so tecton images are signed by the tecton key um the cosine images we were using were signed by the sig store key and then other things that we were using were signed by our key okay so great now let's see what happens when i run it so let me pull it down that's weird it's supposed to say hello world what happened looks like my supply chain got attacked so i was validating all these different signatures but something still went wrong well let me show you and just um you know this sort of thing uh for the sake of time not going to show you how you to sort of debug this sort of thing um you'd probably use something like falco or tracy to use ebpf and sort of validate some of these things but if i whoops if i go right now and change the canico task and uncomment this and rerun it now what i have is i have a task or a step in the canico task that is actually validating that the parent image that canico is running a sign so for those of you who don't know canico is sort of an agentless runner right is not running um in docker itself and so on and so forth so it's not actually seeing the emission controller from the perspective of uh you know kubernetes so what's happening right oh and here here we see like well okay so it was using an image that's a builder image that wasn't signed and so now we validated that it's signed and because it wasn't signed we didn't verify it it somehow came in hijacked our image repository now we're catching it right before we couldn't catch it so now um oh yeah and so now what i can do is i can now sort of show what it should look like so now just for for the sake of simplicity here i also have a i'm just going to build sorry rebuild the good image and sign it and obviously you know obviously in in real life here this would be a in real life this would obviously you would go through you would validate the the parent images you would validate the builder images you would do all that sort of stuff okay so that's built now let's go through rerun it one last time take a couple seconds so while that's doing that oh that's because of a typo sure thanks thanks okay um so uh let me just last time here now it should so now it should go through it should see the um that the parent image is is signed yep so it validated that and so while this is continuing to build let me talk about the last step here which is going to be um so now that we have all this what do we do about it how do we protect production right so the way you do this right is so um you can validate uh you know using stuff like caverno and whatnot you can definitely validate signatures but we need something a little bit um more powerful at least right now that supports the ability to kind of validate also the attestations themselves and you know validates some of the metadata that is lives alongside the image so using opa gatekeeper um plus a an api wrapper around cosine we can do stuff like validate both the signature of the image as well as the attestation so as priya had sort of mentioned chains attestation format is already you know salsa compliant um it can prove some of these things so so then how can we say okay well production now requires at least salsa one or requires salsa two how can we now validate this well before doing that let me just do one last uh docker pull here just to kind of show you that now it's been fixed right and now it says hello world right and so now okay cool i want to run this in kubernetes but i also want to make sure that things that aren't signed don't have valid attestations aren't getting run and so how can i do that well that's actually pretty simple right using um you know opa gatekeeper or similar sorts of tools and emission controllers you can sort of validate that um the attestations that are signed by cosine are not um that they're valid that uh they have the right attestations and if they don't you don't allow it into production so here's an image that wasn't signed by us right just a random curl image i want to run it in production i can't why well it uses an image without valid attestations and it uses an image without a valid signature cool so it doesn't run in production but if i go and i do um so if i now run um the actual cube ctl uh sorry if i run sorry my my canico chains demo um image it does allow it why does it allow it well it has valid salsa attestations right through chains and it also has a valid signature and those two things combined we're like cool this is we think that you know this is good enough for authorization for us and so all of these things combined right are are just one of many steps that we need to take to improve uh supply chain security switch back to so i mean we're mostly wrapped up here but i mean just uh just to wrap up we have the same concerns everybody else this is a general a generic problem um aligning to these frameworks can be a helpful way to link into it basically the demo that mike just showed as a result of going through and trying to hit salsa one and then leaning up and leaning up and and just using that to prioritize what to go get done um and then lastly just every time anybody makes this easier this gets better for everybody so this is really a community effort this demo is very similar to a lot of the other demos today so i think this is all red question mark any questions um yeah so it's just a visual studio oh sorry uh oh sorry sorry sorry so the question was what ide and integrations do i did i use um so we're using a visual studio code um and that's really about it it's mostly just visual studio obviously some plugins for syntax highlight raw state that sort of thing it's it's nothing to come oh and so the so also using the tecton dashboard which is is a ui plugin for tecton so um we can't get into too many details about uh production but what i can say is for example with that demo i just showed some of those features did not exist two weeks ago so these are things that are very much moving very and i think it's something that that um need to highlight which is just the the old model hey this may be a few months or whatever these things quick and the attackers are more sophisticated just as fast so we gotta we gotta we gotta learn to real fast i heard fridays are good for shipping all right thank you