 Hey, good morning. This is Fessel. I'll see if we were getting John Meadows today. How's everyone doing? Hey, everyone. How are we doing? Thanks. Thank you. Good morning, everyone. Morning, everyone. So those massive frenzied activity over the week on the document. It's transformed into lots of additional commentary. Just cool. Is there anyone on the line that is new to the group? Once introduced themselves. Hiding in the, in the wings. Nope. Is, does anyone know if Emily's joining us today? She can't make it today. I'll make it. So I think Emily and as you, you guys were driving us pretty nicely. The, the last. Meeting we actually got a lot of stuff done and went nicely to that document. So we're just going to reach out and that's good to do the same again. But should we go and take a look at that document, see how we're getting on and share it together. So John, just. Can you guys hear me? Yes. Yeah. Yeah, sure. So just a request. I actually have to pick up my son in 20 minutes. So I just wanted to go through my section quickly, but I just wanted to make sure that everybody highlighted and then later I can, I mean, then later you guys can discuss and I'll kind of review, but I just wanted to put my point across because I'll, I'll have to leave in 20 minutes. So I mean, if possible. You want to, you want to do that now for sale, because I know you've done a whole chunk of stuff and we got Richard online as well that was doing the, the source code piece, securing source. So let me share the screen. Yeah. Okay. So a couple of points. I do want to emphasize that this week, my focus was more around adding content. Okay. And right now things might seem here or there, right? I mean, every paragraph is different from the previous one. So eventually the focus this week was that I had content more today. Sorry. If this week and in next week, we can kind of try to smooth out things or try to remove or add things, right? But I think having content was the, I mean, first we need to have content, right? Before we can go through all the phases, right? So, so in securing source code section, what I have done is that I have started to add content to it. Okay. And you'll see a lot of content in under this section, but there are three main points that I'm trying to emphasize, right? Within this section, I'm only focusing on how to secure the source code. That's it, right? I mean, we are not addressing any operational issues. We are not addressing what are the different things around supply chain attacks or we can, how we can make things better, right? But this section is purely focused on one agenda and that is how to secure a source code, right? That is the question that we are trying to answer here, right? What are the recommendations around it, right? There are three categories to it, okay? That I have identified that we should focus on. There might be more. So please, everybody should take this opportunity to add more ideas around it that if I'm missing some category, but the first category that I'm trying to identify my recommendations are around identity and access manager management for the source code repository, right? Source code needs to be somewhere, okay? So what are the, how we can, I mean, what are our recommendations around securing the source code repository itself in terms of recommendations, right? Again, not operational issues. Operational issues can be different things, right? The second thing that I want to focus in this section, in a subsection is what kind of contribution policies we can establish at the source code repository level that will help us secure the source code, right? So the first thing is identity and access management. And the second thing is what are the different contribution policies that can be set up by any administrator or any enterprise to kind of secure the repository, right? And again, this is to help the secure the source code on the, okay? And the third thing is how to establish non-repediation and integrity at the source code level, okay? So these are the three sections that I'm trying to categorize information in. If you find later on that there is any other section, please add to it and just add few comments around it. Then we can take it beyond, right? So then each section starts identity and access management. Richard has added a lot of information here, okay? So thanks, Richard, right? Again, in recommendation, then we are going for recommendations right now, right? For each of these sections that we have described, okay? What are our recommendations, okay? So in the justification section, we are trying to add some information background to it why MFA is needed. This is just an example, right? We can shorten it. I mean, it depends eventually when we will refine or sweep the whole code, what we do. But with each section, what we'll try to do is that we'll add recommendations around it, okay? That what are our recommendations for each of these subsections, right? And they can be different, right? MFA SSH keys, okay? How to rotate SSH keys should be rotated, okay? User access tokens, okay? And there are different recommendations. I'm still writing something around rules, okay? So that we can enable four eyes principle as recommended by Emily. So, hey, Fazal, the only thing, and so I know we kind of talked about this on our call the other day, these are general IEM recommendations. They could apply to every single part of the system, every system that's going to be part of these steps. So my question for the overall purpose of the paper, you know, when I wrote it, I was really focused at contributing source code to a source code hosting platform. And like I know that all the like enabling MFA, you know, we know that that's something you need to have MFA as well on any of these systems on your, on, you know, logging into Jenkins, you know, whatever, whatever's going to be doing your build, any of your security software, like testing pools, all of those are going to probably also need that same sentence. So the question I had was, are we going to copy all of these recommendations across the paper, or are we going to have, you know, that's the part where I was like, why are we, why are we justifying it here versus anywhere else? Yes. So that's a great point. Okay. But this is only related to MFA and I will emphasize, right? SSH keys, you may not need it for Jenkins. Okay. These recommendations are specific to how to access and how to do IM around source code repositories. For other systems, it might be different. MFA is a thing that you can say is general in nature, but underneath other recommendations, if you see around, these are specific to source code repositories. I mean, there are, I mean, for example, you say how to access Amazon, right? Or AWS services, right? They will have their own recommendations, right? How to store your IM credentials and things around it, right? But the recommendation that I am giving here, they are more specific to how to manage IAM around source code repositories. Or you can call it a hosting platform or a DevOps platform, anything. We'll come back to the terminology, but these recommendations are specific to source code repositories. Personal access tokens and these things, I mean, these things are different, okay? For other parts of the system, okay? So I do not think so that you will share this information with other sections. The only thing that can be shared is MFA, right? Because it's generally, I mean, in general, right now there is a trend, right? You need MFA for everything, right? You are accessing website as well, you need MFA, right? But again, here I'm trying to emphasize this solar wind attack that happened, okay? And where this thing happened, right? Because there was a credential issue there. I mean, just read through this section, okay? With MFA, we may want to, we may want to place this section somewhere else. But if you look at all other recommendations, they have to be here because they are more directly related to source code repositories. They are not, they are not a general in nature. I mean, I wonder if to Richard's point, there are two layers, one that is overarching for all the paper. And at that level, it might be least privileged. And onto a specific sections, it becomes more, more targeted and more implementation-specific to how it pertains to that particular area. And at that point, it would be easier of like, hey, here is how we're applying this overarching principle in this particular section. Richard, is that what you're thinking? Or you'd like your structure? Yeah. I just, the thing I worry about is us writing the same sections over and over again. That's kind of my key thing here. When I think about this section, I think about it. I'm starting an open source project. I want to know what CNCF thinks is the most secure way of communicating with GitHub. There should be a very focused list of like, okay, make sure your users have MFA enabled. Make sure you're using SSH keys and not using HGP, HGPS. Make sure if you're, yeah. Do you want to say that or not? Actually, I want to ask that before we finish this. You're recommending both HGP and SSH keys and personal access tokens. But then you, I think you go on to say you're recommending HGPS over SSH. I mean, do you want to completely drop SSH? I mean, we're providing one option, right? Isn't, we're a best practice generally? A categorical? I think there's confusion and I put a comment in here. The use of access tokens is not for developers. That's not. You can. You, you can, you could, I mean, you could do a lot of things, but I don't think I've ever been anywhere where you. I mean, I've worked, I've, that's my current job. We also, from an operational perspective, it's fewer protocols and there's no SSH. I only have HGPS ingress into my environment. Right. So there actually are a number of reasons to pick one over the other. Those aren't these answers. That's not for this. This is an operations answer, but yeah, I didn't. Yes. This. Yeah. This has written. I know everything's a progress document. But yeah. Yeah. So, so personally, again, this is what I added this morning as well. Right. So, so in my view as well, both are the option. And this is how I'm trying to put it here in the document as well. SSH is an option. Okay. And there are issues around it, right? I mean, it is secure because it also creates an encrypted channel. I mean, you can see in this section what I have written, right? But if you can, you should go for personal access token, right? But personal access token, again, they are more oriented toward build agent, CI CD pipeline agents and things like that. So that's why I have put this comment and read that you need to more focus on when we describe this thing that, okay, for build agents right now, let's recommend this thing. Okay. But for you developers, let's try to focus on SSH keys somewhere, right? So the description is there. But in general, I agree overall that if you have personal access token, they should be preferred. And I miss, I mean, I, I cannot read. I have to rephrase this sentence. Okay. Yeah. That's exactly what I was saying. Yeah. Because from operational point of view, right? I know what you're trying to get at, right? And this is what, what is my experience too. That, that HTTPS is from network security teams will say, okay, HTTPS. Good. SSH. Okay. Let me review the system. Okay. So again, add comments. I'll add more things to it. But I will add. Okay. Yeah. Again, add comments. I'll add more things to it, but I will add, I will recommend both the things. We shouldn't remove any of them. Okay. Because, because there are scenarios where SSH keys will be preferred by open source community. It is there, there are many reasons for it, right? Yeah, it's not going to go away, but I'm not sure if I might strengthen a statement of one's preferred over the other. I mean, there are other weaknesses of SSH. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. There are RNG problems protocol. Like crypto gets old over time. Like those concerns are not present with a token because typically a token can be manually expired in compromise. Or can. Typically as a one year expiration or you pick a timeframe when you issue it. Yeah. So like there are. There are a lot of nice reasons that. Yeah. Yeah. I mean, I'm not sure about using personal developers. For a developer for literally as, as my, I'm not going to have an SSH. Like I will not configure. Right. Hosting platform. I, if you were starting a, an open source project today, and you wanted to be as secure as possible, you would require your developers to use personal access tokens rather than the SSH. I'm not sure. This may happen less than open source, I think, and I might actually agree with you again. I'm not sure. I'm not sure. I'm not sure. I'm not sure. I'm not sure. It's your own environment. Cause I've never encountered this period. I get that it could be incredibly. It's significantly more secure. I'm curious about the fact that you then have to do a whole separate credentials management system for those personal access tokens. I mean, you can't. You have to do a management system for your private keys of SSH. Absolutely. I mean, it's trading one for the other. It's one. And that, it's a pretty big advantage. There are always more fine grand. Yeah. The upside of keys is you have public key. And the public key is sometimes an advantage, which is actually a major advantage of SSH. Yeah, Blake. So yeah, there's, we need to consider. Let's just keep going. Yeah. Get in the comments, Blake. Yeah. Yeah. I think in order to keep going, should one of you put in there that this is for developers and make the distinction between. Yeah. The developer and machine. Yeah. Richard. Sounds like there's already enough for perhaps you to start putting. Principles. In the executive summer, maybe the place is the executive summary of here at the high level things you should be doing. And, and this things may, maybe realize differently depending on the place where you, where you do it. I did change the face. I changed the path is should be preferred to is preferred. Now that we have the content. And more assertive voice, but if you guys feel that, Hey, Yeah. To your point, Blake, single recommendation, we should agree on a single recommendation. And it's getting to where they should be versus what is viable today. Yep. Yeah. So yeah, again, please go through this document. Okay. I have a couple of minutes. I just want to go through the whole section. I know that there can be a heated debate, right? On these things, right? But overall, I have added all the content. Okay. And I, because I also use personal access tokens to manage my infrastructure. Okay. So I do know about them. And that's why I've added all these comments here. If you have anything additional, go for it, add content. Let me know what you think about it. Okay. But again, I just want to go through this whole section quickly so that then I can just kind of drop and then I give it to you guys to kind of discuss. Again, I am, then I am, we need to articulate this topic heading well, but we need to define roles, right? Because roles can help us to define permissions, enforce four eye principles and different things. So all these recommendations come around IAM, right? This first section that I identified, then in the next section as well in terms of contribution policies, again, I have given all the recommendations that I think should be there, like branch protections, require signed commits. We should prevent committing secrets to the source code, how to do it. And then there is this option as well, right? And this is the only section I'm confused about. Should it be here, configure security and vulnerability analysis features? Should it be somewhere else or in this section of configuring contribution policies at the source code repository level? So again, guys, just go through this section. I have added a lot of content here. I mean, if you have more recommendations, add it. Okay. But again, please keep in mind I am trying to focus on the security aspect, not the operational aspect. Okay. So that has to be understood. And then in the end, I have this section establishing non-repudiation and integrity at the source code level. Right. And there are two ways to do it, GBG and SMIME. Okay. In the end, we will recommend GBG keys because they are more prevalent. Okay. And they should be used for establishing non-repudiation and integrity. But generally there are two options. Okay. So just go through all these sections. Okay. I have added my comments on the right as well. There were a few questions. What is the thought process and what I'm thinking about this whole section? Okay. How to present it. But, but this is the high level three categories that I am thinking. If you find more categories, right? Please let me know. But in my view, if we present these three categories and recommendations under these three categories, I think we are good at least on securing source code. Okay. So what's the number list? Should these come like you're thinking they have a sense of priority or order of operations? One takes precedent should it be a bulleted list over a number list? Do you feel strongly about that? I don't have any, I mean, I haven't thought about this thing or to be honest, right? Now my, my point was that, okay, these three things need to be established. And then I'm generally giving recommendations. I'm not, I'm not saying this one takes more preference, right? They are general recommendations. And we can take a different approach. It's interesting. It's interesting. Kind of, kind of giving more priority. But right now I haven't thought my agenda for this week was have content. And now we can all dig in and refine it. And I think, I think that's great. There's a lot of really good content there. A lot of good thoughts, right? I think it's something that we can now take a look at, right? At least we can go through now as a group later on and sort of edit it. And perhaps, as we're saying earlier on, pull some of it back up to maybe it's more generic thoughts or even some of it in some form of an appendix and just try to sort of refine it so that we get nice clear recommendations out to people. And one of the other things I was thinking about during the sort of discussion we had there was, you know, do some of this break into different separations of security sensitivity, right? Is that, you know, some of it, okay, for the moderate and basic security. And then if you jump up to, you know, an air gap network, maybe it's easier to use one issue or one approach over the other, right? Yes. This will be considered eventually. Once we have finalized the content, we will assign this moderate or high. I mean, we will itemize this thing. I have to drop guys. I have to pick up myself. I really apologize, right? But overall, yeah, please go through the document. The content is there. I will add more content to it, but I didn't know about to refine it as well, right? And hopefully I've conveyed my point, what I was thinking when writing this section. Okay. Yep. No, it's great. Thanks for the detailed content on that one. Thank you. Awesome. So just, just trying to get back to the top of that document, I guess, or look at what additional people have added in. Do we want to go through the people on the call and, and in the similar way that Faisal has done, sort of highlight the area or the section that you've taken on and give a quick update. But you've got a lot of people in the call, right? So maybe like five minutes or less on anything that you want to bring to people's attentions for a bit of, bit of feedback or a bit of area that you need help on, a bit of area that you're not quite sure about. Should we go through that? Maybe if you want to go? Yeah, definitely. Thanks, John. Let me see if I can share. Can you share? Sorry. I'm not sure if I can share. Can you share the screen, John? The document. Yeah. Let me give it a go. So I have added the generation of a small. So I just want to clarify what I mentioned there. And I just want to know how, how far we should go in the generation of S-bombs. Where are you going there? Find out where, where you're at generation of S-bom. Yeah. Used to be close to that. I used to be at the top now. I've got a lot of additional content. There we are examining S-bombs. It's down. It's called generation matter that let me. Whereabouts? Let me find this. Also, I feel like the document keep going back to older version. I'm not sure it's just for me. Like last time I made some changes and it's going back again to an older version or something. I don't know if others notice the same issue. I have the same problem. I'm part of me. I was like, man, we should do this and get. And then I realized, no, this is, this is how things die is when you make people do word documents and get. But I think, I think I'm going to suggest in that the last time, well, maybe I'm just, you can give us a bit of insight and when you guys were doing the, the spiffy inspired document, you, didn't you like everyone did it in, in Google docs and then start to version it out or something. We had a dog per chapter. And two, three people per chapter at a time. Right. Okay. Okay. So yeah, John, I put it in the chat. So yeah, I found it. I hope you can also find it. Otherwise I will see if I can share my screen. I cannot share. Okay. Can you find it production or software artifact and associated metadata. So if you put as bombing bracket, you can find it here. There we go. Got it. So yeah, so in this area, I mainly focus on a, I think we need to rephrase that title also maybe. I think the software artifact will be produced as a part of the build process. Right. So I was just focusing on how the as both itself being produced. So, you know, so we can have multiple ways to generate as one way to generate is during the build process using the build information from build or package management tool. So how the software itself is produced. So we can generate as bone from the data. The other data is a software composition analysis. Like a, once you have a software, you can generate also, you know, reverse engineer and going back to find the components inside the software. So there are different kind of tools currently available in the market. Like it's some of them support both the SPDS and Cyclone DS, some of them just for Cyclone or just for SPDS. Just wondering what your recommendation is here, right? I mean, it's, we should definitely, we should definitely generate them. But is there any recommendation of how or, you know, I mean, I put the one, so my, I mean, with my testing early testing and the trials, I find that, you know, the build process tools has more accuracy because, you know, they can find actually what exactly the software is built of like a, like Maven tools or a node or something, whatever the build tool, if it is a Gradle or Maven, if we can use in that build process, we can have more accuracy in the S-Bomb compared to the reverse engineering or a manual process. So manually say, you know, if we have to use it, we will use manual, right? But yeah. Do you need any help on that section? So I want to ask, do we need to give some example or do we need to give some more detail? How far we should go with the generation of S-Bomb, right? Like, or is it just high level information? And these are the tools available. So it just put open source tool at the moment like how to generate best forms. Yeah. Well, my gut feel right was if there's anything open source already out there, use that. If there's anything CNCF prioritized that and at least provide some actual guidance on, you know, which of those tools to execute. And if there's a gap and one doesn't exist and there's not a decent tool, then we identify it as a gap and we take it as something that we might want to come back to and actually implement the solution. Maybe that's an option to take on. That'd be my gut feel for that. But at the same time, the landscape is, you know, we need to consider the technology landscape. Not all technology, like I'm saying, go lang. There are so many other programming languages, right? So not all programming languages may have own plug-in or big tools to generate S-Bomb, right? So there are other challenges in general. So, yeah. Yeah, I think it's really important that all software, not all cloud native software is ready to go and run some Kubernetes. Yeah. Yeah. I think it's really important just to talk about the point that, you know, whatever. Programming tool that we're using, whether it be go or some esoteric language. You know, however that, that tool records the packages that it needs during the build process. That needs to be recorded in the SPDX format. All right. So there's tools that don't exist and are going to be built to do some of these, like, let's say add up, right? We need to build an SPDX for an add up. I don't have any idea how to do that, nor does many people, right? But I'm sure it can be done if we talk about that in a high level. Sorry, Blake. I was going to ask, are we recommending one or the other over SPDX versus Cyclone DX? I think that's where we're getting to. So at the moment, in that list, I haven't. So I put both, right? Like both SPDX and Cyclone. In fact, Cyclone has more support for existing build tools and plugins. But, you know, I've also put SPDX and there are common one, like Anger, Swift, sorry, the second link that support both generation, both SPDX and Cyclone. That's exactly what it's probably spelled out here. It's probably recommended to use a standardized tool and comment their integrations with the ecosystem vary. I don't know. Is there some function that will take SPDX and convert the Cyclone DX and back and forth? No, I mean, is that possible? I don't know if that's a loss. I mean, I think you probably could do it, but I don't know. I don't know these formats enough. So that's a loss. Yeah. There is a minimum, you know, support for all these SPDX, Cyclone and S-Feed or something, the three main one. But, you know, they have minimum mapping, right? But it's not a full mapping for all these standards. But there are tools like, as an example, OAS dependency track, which consume both SPDX and Cyclone and it will convert. So you can, in theory, you know, you can import one format and generate other format from that tool also. So there are different options. All right. And the other thing I kind of want to know is, hey, what should be in the SBOM, right? Are we talking about what kind of metadata? Is it just a package metadata or is there some other things that we want to put in there? I think the guidance around that would be really helpful. I think that that's mostly dictated by the standard itself, right? So that's how the SPDS and Cyclone varies. They have their own minimum required data in the spec itself, right? So, you know, I think in our case, we need to, at least in my view, like we need to at least identify what package, what version is that, where is that originator, where is the source code. And these are the like minimum attributes out there, like in both the SPDS and Cyclone DX, right? But if you want to have more data, we need to think what we need to have. Yeah. Okay. Cool. All right. Do you need any assistance with that, Vinod? You're deep in the weeds there. I mean, I was told that someone else is, I was supposed to ask it someone else there, like I don't remember that guy's name, but I think it's your ion channel guy. So, but I didn't saw any update from him. So that's why I jumped in today. Okay. I appreciate it. Maybe reach out to him on the Slack and CEVs. He's still interested. Yeah, definitely. I will reach out to him. All right. Thanks, Vinod. Who's up next? I know Cole. And Michael, you, your own stuff. Cole, did you. Do you want to give an update on this stuff? Yes. Yes. Yes. Well, one second. Let me pull up my area here. And then I'll share my screen. Nice link. Oops. All right. Yeah, you can share. Okay. Software factory. All right. So are you going to have to zoom? I think that's quite some screen you're there. We got that should be better. So I added some additional sections here. I need to do a little more on testing, connect and wiring provision deploy. So that's going to be knocking that out. If someone wants to take some of that or help with some of that would really appreciate it. You can just assign yourself to maybe one of those subsections. Otherwise I will be continuing on. You know, one thing I did do that we hadn't talked about before really probably deserves some discussion is defining a different user roles, right? Have this idea of segregating responsibilities of the software factory. And you different different roles with checks and balances between those roles, right? So you have like a policy maintainers, right? The idea that the policies kind of be coupled from the software factory that gives you the constraints on what users can and can't do, right? That should be your encoded in organizational policy should be in cordon that with the constraints, right? Who defines the pipeline, right? Or the, you know, the specific pipeline is going to use. Cool. Yeah, you know, they'll define either the pipeline templates like in the idea of platform one and some RBAC rules for developers, automation around, you know, when you instantiate a new project, what permissions does that project have, where does that project get deployed to, right? So that's all policy that should be encoded Then you have developers, you know, those are the end users, right? So you should have depending on, you know, how large your organization is, right? You may have a federated model for this, for your developers. In order, you know, so you may have many, many, many groups of developers that don't have access to each other's projects. And then you have factory administrators, right? They're responsible. They're like the SRE group for the software factory responsible for innovation and the general maintenance and uptime, making sure that they hit their SLAs. Nice. Did you get into any of the things I thought about over the weekend and last week was the generation of the hardened containers in the first place and making sure that we create a pipeline to generate those containers or where you'd get those containers from or the API or requirements for those containers specifically. So as far as just like what the image should look like for hardening containers, no, I'd not get into that. I did. Go ahead. No, it was just that that's kind of my thought is that and you know, to go back to the DoD, right? It's all the cool work that's heading into the Iron Bank and leveraging those containers in the middle of the software factory. It's identifying. We've got really good software factory implementation we're talking about, but it's really the real heart of the matter is what does and doesn't go into those containers that are actually created and how they're hardened in the first place. Maybe that's not, in terms of best practice, it's like you're going to be a user of that platform and, you know, if you go and use it, but it's kind of advice on how to create that container or maybe even it's advice just go and use the Iron Bank. That's one way to do it. So yeah, I think that goes into, you know, where's your repository at? And start talking about that. Are you talking more of like a bottom turtle problem, Jonathan? No, I'm sorry. I'm talking about the actual hardened containers that you're going to string together within that software factory itself, right? So that each of the individual tasks you're going to put together within that pipeline, right? You know, one's going to have building it. One's going to have sastin it. One's going to have some sort of scan tool. One's going to have a test tool within it. You're going to chain them all together and how do you actually create those containers? Okay, right, right. So we talk about the steps here of what a pipeline is and all the steps that a pipeline needs to have in order to produce a hardened container. Right. So we have, you know, functional testing and testing, right? So we know that it works and it matches the specification. I think, you know, verification. I think that's a good idea, right? That's an aspect of hardening validation of the build process, right? So these are the steps that go into a process. I think there needs to be some sort of a diagram here to make that more apparent and say, hey, these are the steps that go into it. And this is what your end product looks like. Or maybe it needs to have like a different introduction or a title or something. What are your thoughts? Can I put some diagram in there? Maybe I'm looking at it from a completely different angle, Cole. And then you're out, but definitely the entire section is how to create a single container within that software factory or whether it's. John, we're in 40 minutes. Do you want to take this one with Cole? Yeah. And a breakout. Okay, cool. By the way. Yeah, good, good work. So yeah, Cole, let's see when I get together and we'll go through that one. Yeah, I got a little bit of time right after this. If you do. Yep, me too. Michael, do you want to. Sure. Step in. How are you getting on? Yep. Share my screen here. So a lot of stuff in here, mostly about the build worker and the, the securing the build infrastructure. I know that there is probably some overlap between some of the other pieces and, and as we kind of evolve, I'm sure some of this might go in other areas or, or whatever. But I think the big, the biggest piece, I think as far as the securing is like what, what to do about the bill worker. Bunch of stuff in here, but the highlights and I have this in my notes somewhere down here. So the big things I think that I've done in the past, that I've done in the past is, you know, the build worker should be single use. Why? Well, if the build gets compromised, then your build worker is fundamentally compromised at that point. So if you're not constantly sort of refreshing your build worker, whether it's like a container or VM or something else, then then you're potentially building on a compromised worker. You know, the, the main sort of principles around it are sort of have a golden image that should ostensibly only keep track. Sorry, it should only include the, the tooling that is needed for the build. So like keep that image as minimal as possible. So if you're compiling GCC or whatever, it should really only have GCC and some other few things to just support that it shouldn't, you know, consist of like it shouldn't, you shouldn't have a build worker that is for Java plus GCC plus all those other things. It's, it's way too much attack surface. The other thing that I think is big is, you know, the build environment. So that sort of consists of the dependencies and the sources that you're going to be then using to build your artifact. And so the big thing there that, you know, suggesting is that build environment should be created for the build worker. So as much as possible, that build worker should be more or less, you know, not completely necessarily air-gapped, but it should be air-gapped in the sense that it should not have network access. It should be using stuff like shared drives as our shared, shared storage and those sorts of things such that, you know, the pipeline will be creating the, or sorry, creating the build environment, which would include its sources and dependencies for the build worker. And then the build worker just knows to pick up its build environment from storage. And then the build command itself should also be passed into the build worker, you know, if it's a container, you imagine it's just when I run the container, I'd sell it. Hey, here's the command. And the command should, for the most part, just consist of something like a, you know, compile this thing or whatever, plus usually something like here is the signature of the build environment that you should be looking at and, you know, validating. And then the output should be an artifact, which is get, which gets written to another piece of shared storage, which then, you know, the pipeline orchestrator would then pick up and then on the build worker's behalf, upload that to an artifact repository, because, you know, there's just too much going on in the, you know, given that the build worker, right, is running potentially arbitrary code in building the artifact. There's way too much stuff going on to give it sort of network access, because then, you know, if you say, hey, I'm going to allow this to push artifacts, you can't really be sure whether or not it pushed the artifact that said it built and so on and so forth. So that's kind of a lot of it behind the build worker, also sort of limiting the build worker to as much as possible, just sort of a single sort of process. So if it's, if you're building, you know, multiple artifacts and then linking those artifacts together and some sort of step, it's much better to have all of those be individual builds, have the pipeline orchestrator sort of orchestrate all of that over time because then it becomes much easier to then audit after the fact that if one of those builds was compromised, you have a much better chance of catching which of the builds was compromised rather than, hey, if you have a sort of a monolithic build, then you're kind of, you know, it becomes significantly harder because you have, you know, it could be building multiple artifacts and you don't know which step is what failed or what got compromised. One though, really quick. Yeah. I think I agree with most of you said, one thing I might note, one of the major advantages also to blocking the internet egress is you get a more accurate idea of what your dependencies are. You can't pull dependencies in from the internet. So it forces you to have captured all your dependencies prior to build, which is actually a big deal. Yeah, that's great. Yep. So the main reason. Yeah, yeah, exactly. And actually one of the things along those lines, which is not necessarily captured in here because I don't think it necessarily is part of this, but it's something that I do think needs to be either discussed or captured somewhere is. So one of the things that made a lot of this sort of work for me in the past is by building sort of a Merkel tree of the dependencies and building a Merkel tree of also the signatures of in, you know, the things that you're building, which has allowed us to sort of say, Hey, if your artifact essentially self describes its, its tree of dependencies, it makes it much easier to kind of say, well, this is not a valid signature because that signature is not based on the hash of the other stuff that creates that artifact and becomes much easier to know like, Oh, well, I know that this, you know, if I'm if I built all of its dependencies, there's some amount of signatures for each of those dependencies that are essentially based on the hashes of those dependencies. And then when I'm building the final artifact, well, the final artifact is essentially it's a hash based on or, you know, it's a hash, which consists of the signature of the final artifact, which itself is also based on the, you know, validating the hashes of all the previous dependencies. Now I haven't seen too many folks sort of take that approach outside of OS tree. OS tree seems to really be pushing that sort of model. Nick's is another package manager that I see sort of dip its toe in there. But it's something that at a previous place, we sort of more or less developed internally wasn't all that difficult to do because I know that to some extent basil and pants and some of those other build systems sort of do the same sort of thing. And so by doing that, that kind of allowed us to sort of say, Hey, when, when securing the build workers, it made it much easier to sort of say, Hey, the build worker exported something that doesn't match up with what we have upstream because it has to be based on the, the hashes of the dependencies and it's giving me something that appears to not match up. And that made it easier to sort of catch those issues. Exactly what you're saying. I think in total actually does some of that. I believe it signs as inputs and signs as outputs. It signs a hash of all of its input files individually and it signs a hash of all of its output files individually, which is not quite a miracle chain in the same way. But if you post build validate all the, those chain together, you've got a very similar validation. Yeah, cool. Yeah. Yeah, just one more item in X space. Yeah, yeah. Yeah. So I've been a little, I've been a few years out of that whole space. So I'm a little unfamiliar with, with some of the newer tools, but yeah, that, that makes sense. And then so I mostly sort of focused on to go back to sort of the build worker real quick, mostly focused on the build worker stuff, mostly trying to provide guidance around, you know, for example, try and remove anything that is really not related to the build from the build worker. So, you know, if you don't need VIM on the build worker, don't include VIM in the golden image, you know, the, as much as possible is the build worker's dependencies should be given to it as opposed to having the build worker pull in its dependencies because it's much easier to kind of have a, something like a pipeline orchestrator that it's only responsibility is just to sort of move the pieces around to have that push, you know, the build environment to the build worker. And then the same way that the output of the build worker can then be pulled by the pipeline orchestrator pushed to whatever and then, you know, the pipeline orchestrator will take all that data and do whatever it needs to do with it. There's a bunch of other pieces. Oh, sorry. Sorry to Michael. I just want to clarify there. When you mentioned build worker, right? So, I think most of build tools in the market, like, you know, even, you know, maybe in Great Ale or, you know, what are the tools that we have, right? Most of them works on the pull model, right? But when they need to build a software, if they can't find in cash, they pull it from the remote repo or whatever they have. Can I just, can I just jump in a sec? We've only really got 10 minutes left, unfortunately. I just wonder if we want to take this one as a bit of a sidebar. Because again, it's huge amount of good content, right? Maybe pick this up on Slack or a separate conversation. Just keen on making sure we get to get a few minutes with everyone else on the call. There are others that are, others that have been able to contribute that haven't talked so far. Marina and Mikael. I did a couple of comments in the signing section, which I think is a couple below this. It's not quite text yet, but it's a couple of thoughts, really. Yeah. And I can, I can later turn this into something more like text. I didn't have a chance this week. Okay. No worries. No worries. Anyone else? Mikael, Aditya? I was able to add a bit to the distribution mechanism. So I basically approached it in the sense of what properties we want any, you know, any distribution mechanism must have and I borrowed pretty heavily from the properties that Duff offers. And I think it does make a lot of sense to, especially for the CNCF to recommend using Duff or distribution mechanisms. Yeah, I mean, it's perfect sense, right? Do you need any help with, I mean, you are, you are in, you are from exactly the right place to a point upon that, right? I doubt you need much help. I think I could use some help or at least someone to discuss a couple of things with, with respect to handling metadata specifically, especially, I think there was some discussion about handling metadata internally. That's a scenario I'm not very familiar with. So I'd like to go over that. I can give you a hand with that. So it's been tough, but I can give you a hand with that one. Let's catch up on that one. Anyone else that's added? Yeah, I didn't add much there. Joe was just reading and understanding what was there. But I'm wondering if there was any discussions around code obfuscation on the protecting the source code there. And if that's relevant for this document. Drop it in. Sorry. Yeah. Drop it in. Put it in there. Yeah. Yeah. I'll add that. Yeah. Today or tomorrow. Yeah. I'll do that. Awesome. Yeah. And yeah. Nice job on the article. Your team got published. Okay. Yeah. I'm really, really keen to see you and, and Andy, Andy Martin, the perspective you bring us is red teamers on to this as we're thinking very offensively about it. Sure. I'm sure. Yeah. Yeah. I'll focus more. My focus would be more on the source code stuff. That's my prior background experience on application security. So, so yeah, but I'll try to add some stuff from this article as well there. No problem. Thank you. Yeah. And feel free to reference it to be sure rather than yeah. Okay. Thank you. Thank you. Yeah. I'm on a separate note. I think I, over the next few days, I'll probably be kind of, you know, jumping in on a couple of places and securing the pipeline, which I think all was working on. And I haven't taken a look at securing the bill yet, but I am curious to see where it all makes sense. And a few of those different places. So probably also be either leaving comments or making suggestions. Andy, you were trying to chip in there, but I think we'll mute. Yeah. I constantly forget this microphone is flying above my head when I'm drinking my alcohol-free beer. That's not alcohol-free. It's not even, you're not getting away with it. Genuinely, I protest too much. The, yeah, my contributions are yet to land. I have everything printed out, but it's been so much activity that I'm actually only halfway through and going to start again. So I will have something more useful for next week. But it looks awesome. I'm so impressed with the bit that I've read through so far. Excellent. Is anyone contributed or added that hasn't discussed it today? Nope. Anyone? Nope. Sorry. So I've added a few things around signing artifacts and thanks to Michael and Maria for that commentary. I think it'd be worth having some help around validations and shares at deployment and runtime. I think there's a bunch of different components working amongst there, such as in Toto, S-bomb, artifact distribution or signatures, GBG. So I think there's some affinities with some of the previous headings. So I wonder if there's someone who could help in that section. Any takers, anyone around for that? Do you want to drop a text of that effect into the Slack channel? Gary, and if I get time after helping out the detail, I'll try and drop into that too. Yep. I think they'll be useful. Anyone else? I'll drop in there as well. I'll be good to add other people in there too. Excellent. Any other areas that people think are either a bit light or they're on and they need a bit of help? Nope. I think we're good. By the way, I think there's a huge amount of content in there. Looking pretty awesome there. Yeah. Just one comment there. Yeah. On the 2FA part, like I didn't see any, I'm not sure, I didn't see any specifications on the type of 2FA to use, like either as a mass or like an app on your cell phone or physical token, right? Like a UV key. Do you guys think that's relevant or not? I specify. In my section, I did specify tokenized 2FA. I don't know if that's a good characterization of the type of 2FA we want to use or if there's another term that, that, that works better. I mean, don't rule out smart cards either. I don't think we should recommend them for everybody, but. Yeah, maybe we can have some degrees of like how, how secure you want to, to protect your code, right? So like what level of security you want. And then you can have like, okay, any 2FA is fine. And then maybe the app 2FA is a second level. And then the physical 2FA is a third level of security. Yeah, I think that's one of the things we're trying to, trying to do, trying to start, you know, separate out what this is moderate. This is high and certainly add to that. Yeah, I like, I like, I like the approach of, of the levels there. Like what they have on the OASP open SAM document that they have some levels of like best practices for, for application security and, and secure coding. So I like that approach as well. Again, it sounds like often he's moved probably like early in the document to a section of its own because this is again, a common item. Okay. Yeah, I think we're sort of against that point on me where it was the common item section or, you know, I am in such great. Alex, anything that you want to chip in? No, I'm, I'm, I'm sorry, I'm new to the group. I'm mostly just observing and listening and I'll, I'll be reading through the paper this week and see what contributions I might jump in with. Totally fine. Just join in on the Slack channel or just, what we're doing is just go in there and assign it a section to yourself and jump in any commentary very much welcome point. Okay, cool. So huge amount of progress last week. You know, I think more refinement and more contributions coming in. I think Emily was suggesting that by the end of next week, you know, aiming to get at least a decent run through the document ready for editing. I believe that was a sort of timeframe she was looking at. So if we can try and refresh each of our sections to make sure that they look in decent shape. And if we need sort of initial editorial review, maybe we can flag that as a, as a comment as well and a couple of people can go back through it. Makes sense. And does any, any other thoughts? Yeah. The week feels a little bit tight, but. Yeah. You know, I'm happy to offer. Sudo office hours, same time slot Monday, Wednesday, Friday, people want to jump in and work through things. That same if you rather break out. I feel we might need yet another week. I think we can, we can push, but let's see. Well, should start taking. Yep. Sounds good. All right. And with that, I'll let Andy get back to his beer and wish everyone a really good, it's definitely beer. There's no way, man. I try to get the hour timing. Okay. Right. There we go. Yeah. Yeah. Yeah. Yeah. Have a great weekend, everyone. And catch up during the week. All right. Thanks everyone. Hey, Jonathan, I'm just jealous, man. I'm just jealous. I'm sure he's good stuff. Sorry. What kind of beer is that? It is Lucky Saint. Saint. Well, it's 1130 right now. So I got 30 minutes of parking. It's the only positive side to this relationship between the two countries because half the time it's 6pm and everyone's like, this is a button and jumped up on caffeine. I'm just, oh, what's happened to the day? Oh, I have a customer right now that's over in Ireland. So same, same situation. Yeah, exactly. Hey, Jonathan, are we good on that? That like base image that we kind of sidebarred on. Yeah, what I'm, I think so. What I'm trying to sort of suggest is, is as we building out the software factory, right? If you want to define what the opinionated pipelines are going to be, maybe you have one for a Java image, you're going to build a Java project, you're going to build or go project or whatever. Within that pipeline, I'd imagine that you select a number of different steps that you're going to put within each one of each one of those pipelines. So you would assemble those different steps or components. You'd have maybe a couple of linting tools. We'd have the build tool, a couple of testing tools, perhaps. Some signing thing at the end of it. And you'd have your factory effectively chain all these things together, sign each one of them using in Tato and such and distribute it. What I'm referring to is the way of building those individual images behind each one of those steps and ensuring that we highlight that, you know, we need a pipeline that actually builds those steps and you need to ensure that you have the same level of security around those builds, so that you don't just build your pipeline and then fill it full of stuff that you built on your laptop, right? Yeah, right. So, you know, we need, I think we, I can go into the process, kind of using those identities of management and how we do that. How do we bring in new images and how do we validate those images in order to be used into the fact, in order so they can be used in the factory, right? Yeah, exactly. Yeah, because, because I think that's really, really important because there's quite a lot of detail that you should put in there to make sure that people aren't just pulling the stuff straight off the web or just, you know, again, building something off the laptop and then using that as the sort of heart of the really super secure software. Yeah, yeah, I can talk about, hey, you know, this is where we need to really understand the ideas of reproducibility and, you know, distributed trust along these networks and kind of evaluate our risk of our build materials that we're bringing in that we're saying, hey, we trust these. Well, why do we trust them? What's our risk profile and how do we evaluate that? And then ultimately get into the point where the software factory itself could be used as the mechanism to build those images and secure those images, right? So, you kind of sort of get itself to eat itself, but, you know, you do need a decent level of control when you're building that sort of stuff. Yeah, who was it? Oh, Mikhail actually gave me this quote. I don't know, is he still on the line? No, he's off, I think. No, he's not. But it was, it was talking about how if you would have put like a vulnerability like the C compiler 20 years ago, you could still be exploiting that today, right? So it's really important that we figure out that we had that trust as far as the images, you know, verified as much as possible. Absolutely. And there was, I've been thinking also about the, you know, monitoring those individual components and steps, right? And the way I look at it, when you get to, you know, not all of those steps are equally important or maybe they are, but suddenly the build step is pretty damn important to make sure that no one's interfering with that build step and injecting code in that you weren't expecting and such. Or I guess what it really is is anything that actually affects the end result other than sort of a test. So if you look at that particular component that's looking at that build, is there a mechanism we can put in place to monitor the behavior of that build step and ensure within certain parameters it does roughly the same thing. We have the mechanisms for that, right? So you know, capabilities, we have, you know, we have a lot of monitoring that we can hook into. So the way I was going was there was a really good presentation about Tracy, the EBPF monitoring tool from Aqua and the open SSF team did a presentation on it. I was thinking, you know, there's some interesting sort of tooling you could maybe bring to bear on that build component and see if you can learn from what it usually looks like. Yeah, yeah. I want to, you know, I've been spending a lot of time on the code this week. And one thing that I want to do is actually, you know, put traces from build steps and in total and just make that automatic and make that one of the products, right? The trace for that build. Right. Yeah. And we can, when we can build policy around that, right? Because we have it, we can create, you know, the in total layout say, hey, these are the only, you know, CIS calls that are allowed. Yeah, exactly. Right. Exactly. And it got me thinking as well, because I read this paper from the ACM. I'll send it to you. I don't recall the name of it, but it was basically looking at, I think there's thoughts on the forensic output from a malware attack of something. It's not quite that, but basically they were saying that they'd analyzed open source libraries and then analyze the components after it been infested with malware. And, you know, there was like a thousand percent difference in the materials post exploitation. If you actually did a deep dive into the actual file structure, which, which makes complete sense, right? But, but that sort of situation is a monumental signal. So if we have that sort of diagnostics on that build step, that'd be pretty handy. And I think what was the example that they were using the other night? I think it was basically looking at BPF and anti-debug or something like that. If someone switches anti-debug in the middle of your, your open source, you know, in your build step or some of the other libraries that are getting used. It's a pretty good indication. Something weird is going on, right? Yeah. So I actually have an issue for this already, Jonathan. I think we're tracking. Oh, yeah. Yeah. Cool. For the, for the instrument tracing and the intodo. So I think we can talk about that at a high level, but definitely understand what you're talking about there. Yeah. Nice. Yeah. Cool. Good. Yeah. Just, just some thoughts about it. Some of the stuff we've been digging into around how to create those containers. Yeah. I saw some good code to do this too. I can probably just deal and get something, get something working for the next couple of weeks. It'd be fun to do. Yeah. One of the things I've done in the past is you want to limit the capabilities of the container to just like literally the bare minimum, which is, I think goes back to some of the stuff I was talking about before, like at previous spots, we just literally didn't give containers the network capability whatsoever. And so it just, you know, the thing that was doing the actual build would not have the ability to sort of go out to the network for anything, even if there was something else, you know, the saying, Hey, yeah, go out or whatever. And I think those sorts of things like, you know, you know, you know, the, I can't remember what the, some of the it's called in there, but like the anonymized users inside of the containers and, and ensuring that, you know, Yeah. The capability is just completely separate from all the other namespaces. Yeah. And that, that's why it's important to how we're building those containers, right? So, you know, put together that process of building it, constrain it to the absolute minimum you can get away with. And then use that. Yeah. And then what these build process are missing is a verification, right? How's the end user actually verify that you did what you say you're going to do. There's. Yeah. Yeah. Well, so that's why at least in the past it's been, you know, you want to limit that blast radius where you say, okay, I told it to just compile this one time. That's all I told it to compile. And so at least there I can maybe start to, you know, that's where I'm going to focus my scan. If you tell it, oh, I want it to compile these 10 things, then it's 10 things you need to worry about that maybe, you know, on the third thing it compiled, it did something weird. And so, and then in addition to that, I think the thing that we've done is sort of like you have the bill worker, which you kind of limit to a very, very small sliver of thing. And then you have a bunch of these other like helpers that do stuff like it will download all the. The dependencies and the sources, and then just hand it a network storage, it just hand it shared storage and say, hey, build worker, this is all you're doing. Like here's all the only thing you have access to, and it'll do all that. And then ostensibly, it should just write to a separate output storage. And then from there, you can have additional processes to go and say, okay, well, the only thing it wrote was it wrote this one file. I'm going to scan that file and, you know, look at all that sort of stuff. Whereas if you kind of give the bill worker, you know, in the past at least our concern was if you give the bill worker too much permission to like, let's say push to the artifact repository directly, then, you know, and then say, okay, well, I'm going to download it and do another scan. Well, what happens if, you know, whatever that artifact did, like it's hard to, I don't know. There's a lot of things there. Yeah. And we need to set up that bill. But let's say you're, you're maintaining a CNCF project, right? And I'm consuming that project as a, you know, as an end user. I need to be able to verify that you did what you said you're going to do and not in a way that I'm trusting the human being, right? Trust some sort of cryptographic process that you limited the C groups or the Linux capabilities during your build process. You isolated your build environment and didn't have any network into that build environment, right? So I think what we're missing right now in the ecosystem is that ability to verify, right? That is not mature and is out there. I think we have the idea of reproducibility. That's getting some maturity, but the idea of actually verification is not. Yeah. And it's not just the actual artifact you're building. What you're saying is the, the ecosystem that is in place as you're building it. So it comes along with like the artifact and the, the metadata along with that artifact plus the metadata from the pipeline that was used to create the thing. That's quite powerful, right? Yeah. So I was reading some work. There's some folks in Japan at some university that I think we're doing trying to do this sort of thing through zero knowledge proofs of, Hey, could you prove that a particular artifact was built in a particular way? It seemed like it was very, really on with like, you know, a very narrow set of things. But yeah, I agree that that's kind of the, the biggest challenge, right? Cause even if you sort of, even if you cryptographically sign, Hey, here is my container manifest, right? Who's to say you actually use that container manifest in the output. And in the past, the thing that we've done is what you do is you take snapshots of all the build workers and whatever, and you kind of go back and you know, if somebody says, Hey, we want to validate this thing, you go back and you scan the container, you do some other things. It's still not ideal, but it's to some extent better than nothing. Can I just confirm what, what I think I'm understand for what I think I understand, which is when we're talking about injecting all the dependencies into a thing, into a build worker, in order to essentially have a hermetic build. In order to do that with a container based system, you'd still presumably need to construct container beforehand. And it's just that one stage of artifact compilation, but actually source code transformation, I guess into an artifact that we're talking about. Correct. And so the way that I've operated that in the past, was once again, this is at a place that didn't trust Kubernetes, but the thing was you would have a, you would have a handful of containers, right? One container would just essentially set up the environment for the actual source code transformation step, whether it was compilation or packaging or whatever. And so it would set all of that up into sort of a, into one shared storage piece of storage. And then there was actually sort of three pieces of storage that the container would have access to, that the actual sort of build would have access to. One is it would have read-only storage of the inputs. It would have sort of a working storage for it to do all of its work. And then the last step would be to sort of take that and copy it to an output storage. And so, and then it would hand that off, right? It would just say, okay, now I'm done. And then once that container sort of was, you know, died or whatever, you know, it completed its action. Another container would come back on. It would go and do some basic sort of thing where we go in, validate some signatures on that artifact and then, you know, push it to the repository and write any additional metadata to an attestation database or something like that. Okay. That's, it's really interesting pattern. I can see how I'm trying to model onto the stuff that I'm sort of kicking around with tecton at the moment and just kind of, I guess, just breaking stages up for that bit more to give more transparency to composition. Okay. Thanks for clarifying. We're going to have to drop unfortunately, but thanks very much. I'll leave you going. And I'm really impressed. Actually, there's a lot of good content coming into this document. I'm actually got some time of the weekend to go back through. More to do. But cool. I just, I just hope we can get it done in time for a cube comprehension. That's, that's my worry. I mean, I, I know we're going to get it done just that timeline scares me a little bit. Yeah, it does. Cause it's hard to wrangle free work. Right. I mean, what I'm going to do is, is like, just take a day and dedicate it to it. Cause I think it's really important. Yeah. Okay. You can't, you can't really, you can't really ask more of people really graciously contributing a lot of energy into this room. Cool. All right. Thanks guys. I got to run too. Thank you. Just stop the evening. Michael, sorry.