 Hello, everybody. Welcome to our final security hub and conference session about the tag security spy chain security working group. It looks like we just have a few folks at the end of the conference here, so folks want to kind of gather in the middle to help with discussion. If you're interested in getting in, yeah, it looks like you're all mostly there, just to find a spot in the room. So yeah, welcome. So we're first going to talk a little bit about what we do as the supply chain working group of the tag and we've been having some conversations recently about what's next for for this group. And so we figure this is a great chance to have a conversation about that. Does anyone want to start with what we do? Sure, so the mission of the supply chain working group is just sort of a subset of tag security, right? If tag security is focused on cloud native security, then hey, the supply chain working group is focused on the subset of cloud native security that's focused on the supply chain. So that's both securing the cloud native supply chain as well as looking at securing just supply chains in general in cloud native, using cloud native approaches. Yeah, so we've been working on a couple things recently. I'll let John talk about an effort he's been running in the working group, and then I'll talk about mine, and we'll talk about what's next. All right, so one of the most recent things we did was a paper that we, you know, a lot of the work we'd done before was very technical, and it was in some senses aspirational of what the ideal solutions to a lot of supply chain security concerns are. And we found talking, you know, out and about at different open-source groups or with customers that we work with who are concerned about supply chain security, that at the highest levels of those organizations, they don't really understand the problem in a way that allows them to help their company solve that issue. And you know, one of the kind of typical things we'd heard are things like, well, we have S-bombs, doesn't that fix everything? Or you know, if I have all these vulnerabilities in a vulnerability scan report, what do you mean there's false positives? Like people don't understand the complexity of this and the amount of time and effort it's going to take to really fix things. So we wrote, we tried to write as short as possible executive-level summary of supply chain security, kind of laying out some of the concerns and the problems and an approach to solve them. And one of the things that I think is more useful in conversations in that realm is how we tie this back to business-level concerns and how solving supply chain security can actually help improve the business, how you can deliver software faster by following best practices, having things more automated. If you have less CVEs, your engineers spend less time triaging CVEs. Like a lot of this stuff is very straightforward to us and may not be the things we talk about, but as someone higher in an organization, those are the kinds of things you may be more concerned about. So as a bunch of engineers, we tried our best to try and up-level our thought process in that way, and we're going to publish that soon. We wanted to wait a little bit after KubeCon so that doesn't get quite swept up in that whole thing. But yeah, so that was the last thing and then we worked on something else with Marina. Yeah, so the thing we've been working on recently is a supply chain security tools mapping. So there's a lot of open-source tools and closed-source tools in the space of supply chain security that solve something in the supply chain. But really, you know, the supply chain is a lot of different steps. And so tools solve a lot of different pieces of the software supply chain. And so what we did was we took the supply chain best practices white paper that this group had put together and we kind of split out the sections and looked at which tools actually address which sections of the white paper and kind of a matrix to see what solves what. And we're currently in the process of working with the CNCF to turn this into something more visual, more useful for people. So if they're trying to say, like, what tools solve my software supply chain, the answer isn't one tool. Let's look at all of these different ones. And we focus a lot on open-source tools just to give you a nearer somewhere. And we're looking to make this like a first draft of this. And then of course, as new tools come out, as we, you know, I'm sure we missed some, we can add that all to some kind of a visual representation of that. So that's another ongoing piece of work. And then I think we've been talking recently about what's next. And so I think one piece that is definitely next is a rework of that supply chain security best practices white paper. It was written, I think, two or three years ago originally. And of course, the space has changed a lot in that time. So it's time to just take another look at that, do a V2 with thinking in the new year once folks are not taking as many holidays. Take another look at that. And also looking at collaborations with the open SSF, working more on blog posts, sharing knowledge about software supply chain security. And that's kind of what I want to ask the room today is if anyone else has things that we could do to help you in software supply chain or that you can do to help us get involved and solve this big problem. So I think Aditya is somewhere with a mic kindly offered to walk around if anyone has any questions for us, comments, ideas. Yeah. Hello. Okay. So it seems to me that is becoming permanent to me that a lot of the times companies and even engineers are not willing to implement some of the technology work on until it's too late or when regulation actually like when the government comes after them. So I think it might be helpful to create something that not only because like I think a lot of the efforts we're having right now is to educate the engineers, educate CSOs, educate members of the community we talk about like we're in. However, is there any efforts in trying to educate those making the regulations like someone at the NSA or like any other agencies that maybe relevant like that we could like hope about. Yeah, so there's a bunch of RFIs coming from both NIST and CISA regarding a lot of what you just sort of said there of like, you know, what are the big sort of open challenges in adopting these standards and working for feedback from like enterprises from individuals from various projects and and so both the CNCF and open SSF do respond to some of those. Sometimes it's on behalf of a working group on sometimes it's on behalf of just the broader community and so there's definitely a lot there that that we're looking at. And I think to kind of go back to the one when you talked about, hey, you know, what about, you know, CSOs, how can we inform them? Actually, that's kind of the purpose of that sort of executive summary. And I think one of the key things there a little bit is, is, you know, in that document, you know, and we've seen this actually just pop up is like CSOs are being charged with crimes. If they are attesting to things that are potentially, you know, materially inaccurate or potentially even lies as well. And we are seeing, you know, various insurance companies no longer, you know, actually protecting against these things. So now we're seeing CSOs taking it and just, you know, generally companies taking that problem a lot more seriously. And I think for us, one of the big things, you know, that this group is looking to do is try and make that easier for then engineers to develop like the engineers are now getting that signal like, oh, you know, I'm supposed to secure my supply chain. Okay, how do I do that and make this feature go out in time. So that's that's definitely something that that I think we're looking for also a lot more end user feedback on, you know, what we can be doing in the group to to make that easier. You know, I think one of the other things is you mentioned government agencies and I think we say they're watching. They're always watching. But but from a more serious like we've actually seen the cloud of the supply chain best practices and the reference architecture architecture cited in things like SSDF. And I don't know if it was in the the NSA supply chain security guide. I think it was. But there's we've seen multiple different government documents citing the work that we've done, which is a really cool thing. So a lot of the work that we've done in the past has been has been around sort of like the integrity of the supply chain, right. And one of the things that's been occurring to me lately is that a massive proportion of the actual supply chain incidents that we're seeing take the form of like for example typo squatting and things that we're not able to solve with like signature verification for example. And I'm wondering if there's any work or things that we could work on that might address that particular problem set. Yeah, I think that's a great question. Yeah, because if someone asked to download a piece of malware, it's hard to you can securely have a supply chain that gets them that piece of malware that they asked for. I think this is it's only one of the trickier problems in in the field this offers a pleasure. I think it kind of can go back to how people are actually choosing to download packages. I think that's something I think about a lot is like having sources of truth either it's just having typed it in a file once instead of typing, you know, like, I don't know, pip install package name and having to like type that every time. So you'd have processes to verify that. Yeah, I think that's definitely the kind of thing that we can include in our guidance. And so, yeah, that white paper v2, I think is a great place for that kind of discussion. Yeah, and I think one other thing to add on there. And I think that we're seeing with provenance is being the first step is provenance helps you identify where it came from in the first place. And so, I mean, one thing that's at least lucky for right now is we're not seeing a lot of typo squatters and similar signing their packages, which is it was potentially a good thing. But I think even if they were to sign it, you know, as we begin to sort of really look at the provenance problem a little bit more and really try to tie those things back to known good maintainers, known good organizations. Then you know, you can kind of go back and say, hey, I downloaded a package and you know I checked a ton of the the attestations about that package and I can't seem to verify where it came from. So maybe I should take a closer look before actually installing it. That's kind of I think some of the things that we're starting to see in that space as well. Hey, yeah, so I've got a few things. One of the things. So I think I've talked to a few people today about the fact that I teach classes for sands and one of the things that we're adding is some supply chain security information and I get all kinds of different systems, some of which are newer to pipelines in general and others that I would consider kind of almost like peers of myself, right, like people that are pretty expert and something that would be really useful to me is to have explainer materials that kind of start at a very high level and then continually get deeper. Something like I really enjoy like the five levels of complexity or something like explain it to a child explain it to a teenager to a, you know, graduate student expert, whatever. But like some sort of bits of information like that that's both that that's communicated in different ways as well. So like potentially video explainers, those are those are great but also like text based explainers and things that would take this ecosystem from a very high level and then kind of dig into different areas of it and that people kind of choose their own journey. I think that'd be extremely useful. Yeah, I think that's a great idea. That's a great direction for this work. Yeah, if you're interested in getting involved in that, especially if you have some experience creating them, that definitely be the kind of thing we could collaborate on. Yeah, yeah. Could you let us know what's the process to get involved? I mean, I've been running circles around this ecosystem for a couple of years now and it's the first time I've come across this group and that's a fantastic question. So, so we're the software supply chain working group of tag security. And so the easy way to get involved with us is we have a meeting every Thursday, except not today but every Thursday at one o'clock Oh, sorry, 11 o'clock, 11am Eastern time and I don't know what the time that is for you, but hopefully you can make it to that time. And we have a Slack channel, tag-security-supply chain. Michael will pull it up and make sure I got that right. And so yeah, you can find us on Slack as well asynchronously if that's easier. And I think a lot of the work we do goes through the tag-security GitHub repo. We have a specific project board for the working group, which we have been working on keeping more up-to-date. So hopefully that will reflect some of the work as well. Do you have that? Yeah, so it is tag-security supply chain WG, all dashes between the words. Could you explain like the evolution of Salsa and where you might see that going in the future? Sure, yeah, I'll take that one since I'm on the Salsa Stereo committee. So Salsa for folks who aren't aware, it's a supply chain levels for software artifacts. I always get mixed up. So it is a framework, a set of essentially requirements right now focused on the build, but it's focused on supply chain security. And so right now it is focused on the build and around build provenance so that you know that a particular piece of software was built a particular way against a particular source and then had this output. And it's essentially just a signed attestation making a claim about that. And then it's also some requirements around the build system to essentially enforce some of that. So for folks who aren't super familiar, level one is essentially just recording the data about the build in a format that can then be consumed. The second one is sort of signing that data so that you're associating that data with an identity. And then the third piece is some additional properties that are trying to prevent stuff like builds from mucking with other builds' provenance data, as well as some additional, you know, things in there to prevent stuff like a build that goes malicious, signing its own provenance and potentially also exposing signing secrets and those sorts of things. In the future, you know, so that's kind of where it is today. And we're actually, we have a ton of work in progress on a couple of different threads. One is on some additional levels of the build. So some of it's like reproducible builds, potentially there's some folks from Intel and other places that are working on using stuff like trusted execution environments, secure enclaves and those sorts of things to further attest the build where you can actually tie the build not just back to a piece of hardware, but also to a particular, you know, a secure element of the processor. And then the other big thread that's happening right now is around source attestations. So these are things and actually we have a teaches here, you know, there's actually a lot of interest in looking at get tough for some of those things as well, where the idea now is like if you are testing to the build and you are claiming that I pulled the code from this sort of place, can you actually now go back and verify, you know, who are the people who wrote that code? What was that code committed by approved parties is somebody trying to impersonate somebody was the code somehow manipulated without, you know, somebody else knowing. So that's kind of where things are kind of going and then in the future we're expecting a dependencies track, most likely as well as like a package track as well. Is there any work being done in the community around a provenance around AI workflows. I'll also talk about that one a little bit as well. So one of the ones is so so far at least in most of the salsa community, we view that the AI models are by and large the same as software. So it's, you know, if software is a little bit of data and a lot of you know a lot of code, you know, machine learning and other models are a lot of data and a little bit of code. So, yeah, we view it kind of the same though there is some effort to kind of pull out a couple of key characteristics that might be a bit different like, you know, elements of the training data that are hard, you know, that are a little bit different than you would consider just like source code or something like that. Yeah, I think there's also effort more generally in the CNCF around AI and security I think there's a. I don't know exactly where it's sitting right now because you know it's a new new field but some work happening around the intersection of those in some of the tags so I don't remember if it has a home yet in one of the one of the groups. Awesome. So one of the other things that you mentioned that I wasn't aware of was the supply chain tool mapping. I was wondering kind of what state that currently is in and without knowing anything about it. One thing that I would like from it is so I've been able to use like the SSDF a lot lately when I work with developers and DevOps teams because it has the examples column. Like it's great to have these Swiss Army knives that are extremely powerful and broad. But it's also really powerful to like make it at home for somebody and have examples and so that's something I would love to see but I'd also love to know like what kind of what's going on there. Yeah, so the current status is we have a really big spreadsheet that has all the different elements of the white paper and a bunch of different open source projects and kind of. Then that then a somewhat sparse matrix of places where these things intersect with like some kind of explanation in some cases or just like indication of which pieces each tool is addressing. And I think the next step with this project is really to turn that into something a little bit more usable. I think that yeah we're getting some resources from the CNCF for some some actual graphic designers in front and people who can help us make this better. And I think that's another great idea as part of that effort is as people like click through whatever whatever design that looks like actually seeing you know how these tools can be used for these particular problems that that are that are coming up. Yeah, it makes sense. Just like a follow up. Are you thinking like something more static like a PDF or something interactive like a website or like a website. Yeah, and so if anyone has experience with front end development or things like that we would love to have you contribute service test tickets with CNCF can take a while to get responded to. But I but I do think you know one of the ideas we had talked about with this is also just getting different perspectives right so for us as a bunch of people thinking about supply chain security all of the time. We may be fall a little bit further to the the offside of you know thinking about those concerns but also really get the developers perspective trying to understand how different people are thinking about this. And also the tools all meet different requirements and different number of requirements. So we mapped all the tools to the recommendations from the supply chain best practices paper, and we would like the interactive guide in theory to be able to kind of choose your own adventure. As you go through if you pick one tool that offers more security properties for say your source code, then you don't need to choose another tool that does a smaller set of those types of things and as you go through each step of your supply chain picking. So being able to kind of create that holistic view with a set of tools that aren't overlapping. There's, there's the open SSF, we're doing security there's the confidential computing consortium, then there's the whole set of folks like catac containers and confidential containers and all those and then there's you and so is there an overlap between some of the work that's happening or is it all. Yeah, I think yeah there is a lot of overlap and there's a lot of collaboration and there could be better collaboration so as an example I'm both a tag security lead as well as a member of the open SSF tack. So yeah there is a lot of overlap and you know I see john and Marina add a lot of also the the open SSF meetings as well. And so there is a lot of collaboration there there's still a lot more to be done. The confidential computing consortium is really kind of starting to move and there's a lot of I've been talking to a lot of the folks or we've been talking to a lot of the folks over there as well about like hey where can we kind of collaborate on some of this to make sure that we're not all sort of reinventing the wheel like I inevitably like hey you focus on the confidential compute piece and we'll kind of focus more on like the cloud native piece and then the open SSF will think about it more broadly. So there's a lot of stuff there I think that we were very interested for example if if anybody in the community notices like hey why are you folks working on that when this other team is oh we probably just weren't aware as I'm sure here. Everybody knows right like the LF has how many projects underneath and how many foundations. So so inevitably there there's there's going to be some you know conflict and confusion a little bit there but I think there's a lot there and then as far as on the Cata containers and confidential containers space. You know we've had a couple of conversations with them here and there you know there's a lot of stuff also in like there's a thing called key lime for example which is around you know TPMs and so there's definitely still a lot of those conversations happening as well. Sorry I'm trying not to be a hog. So I guess kind of like a follow on to that what what does the interaction look like between the attacks your working group and like the fresco project in open SSF. Yes, so well a couple things. One just so folks know fresco is being archived just because it was more of a representative example and and that sort of thing. Yes, so for folks who are not familiar and everybody should know about it now fresco is it was essentially a an example implementation of the secure software factory which is a white paper that came out of and I helped lead up that came out of this group the chain working group and it was briefly you know mentioned and what it was was it was taking the supply chain best practices work a white paper and looking at you know one specific thing that a lot of folks were struggling with which was the build and trying to think about like what could the future of a very very secure build look like. And so we were looking at all sorts of cloud native tools and technologies that can really really help secure the build so those were things like, at least at the time you know like tecton and tecton chains, as well as a spiffy spire, like DBPF tools like Falco and Tetra gone and those sorts of things and seeing how you could potentially like line up a, you know, all that together and so fresco was actually implementing all of those things. But as I think everybody, you know, or a lot of folks probably are keenly aware it's not the easiest thing to get folks to move off of let's say their Jenkins onto a new system. So largely it was used as an example and a lot of different folks at different companies actually cite it as like hey I was looking at what was going on in there and then I figured out how to fit it into our, you know, our team city into our blah blah blah and then and yeah. Yeah, I don't think I answered that the interaction piece right and basically basically it's that it's that there was like this idea here and then it turned into a project there and we. Yeah, I think most folks here were aware about the stuff going on in the open SSF around it and try to just like support each other I guess. Yeah, I guess like part of it was I was wondering if there's gonna be revisions to the way papers on the one side would it get ported over there that give the projects pretty much destined for the attic then is there not going to be like a tangible example of the way paper updates and things like that. Yeah, so I think on that front. So there's two things. One is, I think, and one of the biggest pieces of sort of advice I've been given about, you know, just sort of open source projects in general is that it's very difficult to start a new code project within just let's say a working project is much easier for folks within the community, whether it's other organizations, other companies to come together, build a thing because obviously, you know, you know, folks are trying to sort of make money on it or whatever and then saying hey let me do that back because, you know, we see it as a good fit to this the CNCF. So I think on that end like if there are folks who are interested in developing those tools I think we are still, you know, definitely down to sort of build some interesting examples but we found at least for us a lot more success in building the white papers and a couple of like, you know, short examples, that sort of stuff, but seeing sort of other groups whether they are coming from the academic space, other sort of larger nonprofits that are sort of dedicated purely to writing code or to sort of the, you know, you know, private companies coming in looking at the white papers implementing those things and then, you know, contributing that code back to, you know, the CNCF or another open source foundation. One more question in this sort of vein here which is just like is there a, is there any sort of formal distinction in terms of the scope of OpenSSF's works in this realm and CNCF's, the tag security or the supply chain working group and like, not necessarily how they interact but like is there is there sort of different emphases or different distinctions in terms of what their scopes and purviews are? Yeah, I think that it's mostly defined I think in the individual charters of both groups. I think the biggest thing I think that we try to do is think about the cloud native angle on software supply chain security or more broadly in tag security on security to try and like reduce those kind of, not conflicts but we don't want to duplicate work in this limited resources in open source, right, there's no need to do the same thing twice. That's the way I think about it at least. Yeah, I think our group is relatively small. I think we have five or six people maybe show up on a regular week and it's a good week. The OpenSSF is a large organization and they have many different working groups focused on things from best practices for security tooling, how to generate s-balms everywhere. There's an entire group focused on supply chain integrity which has different subgroups underneath that. They're starting to take in different projects. Sandbox projects like Get Tough is actually a project in OpenSSF. S-Balmint is another one and that's like maybe in the broader ecosystem. An interesting thing that we're seeing as well of the projects like that that are based off of CNCF projects like Intodo and the update framework crossing these boundaries back and forth. And I think that as individuals the group is, the group of people working on this are doing a really good job of understanding both spaces. But I definitely think at the foundation level there could be a little more willingness to collaborate across those groups and be supportive of each other. Yeah and I think one thing to quickly add there, I think a little bit more specifically, the sorts of things that I think you'll see within our group is stuff focused on as Marina mentioned around the cloud. So containers around Kubernetes and container orchestration, service meshes, workload identities and all that sort of serverless wasm and those sorts of things whereas I think in OpenSSF you will see some of that as well. You'll see more from a breadth perspective than a depth perspective, right? You'll see just sort of lots of stuff but you'll also see folks talking about just sort of general memory safety and as sort of, as John had mentioned there, like just S-Balms generally, right? Whereas we might be focused a little bit more on how to generate S-Balms for containers.