 Good morning or afternoon or early evening for you all even on how are you doing? Oh, do you congrats, I guess I have another I'm not even able to get it It's still gated here Thank you, thank you. This is mine. This is my it's Friday hat One blue and share my screen probably here in a second take off my And Marina all right. We'll probably be pretty quick today since it's Likely Smaller crowd do the holiday Yeah, let's go ahead and dive into this So at this point with the paper at It was chatting with John a little bit about this yesterday. We are ultimately ready to just Start tackling and and getting through the media elements. There are far fewer comments than there were even a week ago I mean, we're we're down to the point where you know, there's not even simple recommendations now. It's kind of bigger question items so I If it's good with you all let's try to shorten this I would say to 30 minutes Today, let's address some of the the more meteor Comments that are on here and then I I think the the like leaving thing would be You know recommending you all go through and either pick one section just to see if there's anything else you can find but we've We're really ready at this point for external review Once we tackle I think these last kind of These around Not sure if it is my condition but It seems like you know, I'm getting some Problems slow with your broadband. Is it same for others or it's just for me like sometimes your voice is kind of Breaking for me at least for me Yeah, I mean I was hearing it a little bit but I don't know if it wasn't nearly Like I could still understand you but I don't know if it's worse for some people I've been having issues recently with with zoom where it was like I I Don't see anything on my end, but apparently it is slowing down. So I But not if it if it keeps happening Just let me know. Okay. I'll go and jiggle the plug in the route or the modem or something Figure something out the router All right, does that sound like a decent plan for today going comment by comment And actually just like trying to get through some of those issues Anybody that's a good awesome so Hands down we're actually in terms of the The introduction aspect Things are pretty solid Until we get to this securing build pipelines Section this is this has been the kind of bane of our existence and this was the most controversial aspect of when we were reviewing it as the four Last week and so one of the things we're waiting on and I'll have to reach out to Emily separately and see if she has any For that graphic Alex glad to see you joined I did Go ahead and mention Before I delete this let's wait until there's a graphic there Just so we remember what we're trying to convey Because I do think this is valuable, but I don't think it's presented in a way that's good So that that's at least my sentiment there Does anybody if you if you're familiar with this section of build infrastructure feel that Bulldozing all this text away and replacing them with the graphic that clearly shows Pretty much these elements. Does anybody feel particularly hurt by that decision? No, I actually thought there was a lot a bit too much text here. So that's that'll be good This whole section is a bit too much text which gets to this little piece by you Alex adding in another sentence I I Didn't want to say no to this there's a lot going on in this in this paragraph period a lot of like meaty this Issue and I I'm gonna be honest. I did not actually look to see how the chat went with Around the build worker versus build steps. I don't see and Blake isn't on the call So we can actually just we can make this decisions for him Always at this time What is does anybody feel strongly about this Alex there's you wrote a lot about this You basically wrote an entire paper on build worker versus build step What do you have to say? I mean, so I was attempting to So we had an extended terminology debate In the meeting where we were trying to go through this section about what they were and and I was trying to figure out a way to sort of Make sense of it all and so that's where this suggestion is coming from is and and I can rehash What else I've said about it, but but basically it seems to me that We've sort of collapsed the distinction a little bit and the suggestions we're making Where we are we are assuming That build steps are being run on containers and we're also recommending that all of those be Isolated and have a separation of concern so that each one is running on its own container And so to me it seems like we've collapsed that distinction so that the worker and the step are There there is a distinction between them There's a way to distinguish them, but they are sort of existing in a one-to-one relationship And so they are at least nominally interchangeable to a certain extent I don't know if that's helpful, but that's sort of Build worker ever Can you repeat that at least zoom is telling you are frozen again with that You should stop Give me one second, maybe I'm returning to you all hey you guys there. Yep. Yes awesome. Okay So build worker does anybody else use that term to mean what we're talking about build step here That was my main beef. I don't think any of us Blake also. I think agreed that nobody uses that word Which word sorry build worker You got things in your life that you you refer to as build workers and people would know what you're talking about Um, I mean at at my last place we we definitely did use them as build workers and we made them distinct from So I guess the thing is like what I think about like a build node or whatever the the reason why I kind of think about The infrastructure very specifically there is when securing a build step. The idea should be that you are also the infrastructure and I'm going to use that in quotes. It could be a container. The infrastructure that you are using to run the build should ostensibly be single use Right, you know, you don't pretty much you want to ensure that Especially if if the build is going to be doing something like arbitrary code as in like a compilation step Right a compilation step. It's very hard to actually know what's actually happening there. And so you want to say, Hey, assume that the compilation step could have Compromise that build worker. And so if you assume that that build worker is potentially compromise you throw that out. And that allows you to then say, you know, you can always go back to your scan do all the additional work to say, Yes, we're we're certain that That build worker did all the right things. That's the way I've always used it is in that sort of context. Yeah. Yeah. Which is what I mean that's what we're getting. That's what Alex was saying with the distinction right is that it's the Yeah, the step is the abstract task right it's it's do the compilation it's run these tests. It's whatever the task is that we're trying to accomplish and the worker is the Infrastructure that's performing the task. But if we're if we're treating of those workers as disposable and each one of them is only completing one step, then they sort of have a have a we've sort of collapsed the distinction a little bit. Yeah, so my only suggestion would just be to make sure that that's clear is just to say, Hey, look, we are making some assumptions like any sort of Step that is doing something sensitive or doing something that could be is hard to predict right like, you know, it's one thing to say, Hey, we're running a linting step, a linting step you probably don't need to worry about throwing out the infrastructure afterwards but if you do a compilation or if you're So particularly packaging something the question I have is it confusing to bring up both because to me I'm worried it is I'm worried we're going to confuse the reader where they're going to not know what a build worker is or not know what a build step is Well, I guess the thing that I would ask is like, so if I'm using, you know, Jenkins right and I'm running my CI CD There's a bunch of Jenkins workers that do different things right you know some of the Jenkins workers. Oh, here's a Windows Jenkins worker that will compile my stuff on Windows here's but you know And so I think in those cases like I I do view those as different from the pipeline steps right like the pipeline steps are saying what should happen and then the Jenkins workers are where those things are actually happening Right right right right right and There could be a way to make it easier and more you know Straightforward about what that means because I also think that the there's another reason why they're distinct is I should be able to run A non secure build locally on my machine right those build steps will run on my machine it'll run linting it'll run a compilation step and I can you know get my art you know I can get a artifact that I can test locally But that's gonna be very different than the secure build worker right which is where the build steps would run in a secure you know way Yeah and That throws a wrench into everything Alex that's like saying we need to have both terminologies and be specific in which ones we use But we're not throughout the paper Yeah so and I tried to when as I was going through it and sort of revising things I was trying to sort of make a distinction between like You know so for example in the in the sample pipeline section where we have the orchestrator says alright here's the next step And then that step is executed on x worker right and you know that that there's a worker assigned to that step and then it gets executed so I'm trying to sort of nuance that distinction a little bit in the way that I did some of the reviewing but The revisions but yes I think there probably needs to be some more More work done on that would be my guess Just going back to that definition real quick Michael build steps and workers take as inputs a golden image to your box So I would say the build worker does not necessarily that will Okay so this is once again yeah there's a lot of ways to to to so I would take okay let's take a step back for just a second Um when I think about The build I think about it in two separate ways one is like hey the build is a set of steps that can run anywhere Then there is a canonical build or whatever you want to call it the this is the official build that we feel is secure that we feel is Is is we something that would somebody would sign off on and the reason why I make that as a big distinction is there's like There's literal like money dollar you know there's money associated with it there's times associated with it and whatnot where if I have a secure build often I'm doing additional scans and running it on secure infrastructure that has a cost associated with it if I'm compiling something locally just to test out the functionality Like it doesn't you know it doesn't matter right like it doesn't matter if it's a little you know not exactly following what we would in in a secure build Um but we can't we can't detail those edge cases because you know that's not it's not definable really Yeah well so yes yes I agree with that but I think that there still is um I think what you could say is if you were defining the build here the build steps and the workers purely in terms of a secure build And just say we are not like specifically call out like we are not um putting anything in here about running this build you know on anything other than like a secure build I'm trying to think of a word for it like a developer build or you know a non secure build you know anything that would be um you know the idea here just being that The build steps and workers here as defined should only be considered in terms of you know this is an artifact that should be signed off on you know and so on Would it be helpful to explicitly write out build steps and workers are distinct concepts workers generally refer to the compute resources but steps refer to the the I don't want to say philosophical the concept as a whole And within this paper we won't and and based upon your implementation, they could be one of the same is that worth just spelling out Or is that that going to cause even more confusion and then and then what's great is that we can if we if we do that and we mentioned specifically that within the paper, you know, we expect that they might be one of the same, we don't then have to say for every single Build steps and workers. I'm worried about having always specified the breaking it down a little bit. Alex is that Yeah, so I think so so this maybe touches a little bit on one of the questions other questions that I had as I was reviewing this, which is like In my head so this is this is coming from CNCF as a document I assume that we are we are leaning into this being recommendations for a cloud native supply chain. And in my head, like one of the things that that make that is a distinguishing factor of that is that we're principally talking about processes that are running as containers or microservices or something in that sort of an architecture right. So that so that's part of where what got me to where I was on this, you know, on this connection between steps and workers is is that sort of containerized architecture is I'm assuming you know containers are a lot more disposable than if you've got a server running in your basement right. And, and so, so that's that's part of where I got to where I was. And I'm trying to tie this back, sorry, I lost my train of thought a little bit trying to tie this back to your original question. I think that So, so I'm wondering, you know, if we're if we're laying out here's the distinction between those two things. I think that this is a cloud native set of recommendations. Like, what does that mean for the for how we want to phrase that, if that makes sense. You know it's funny. I think you and Michael are actually agreeing on the exact same thing. I think you guys are saying exactly the same. I realize here that you you just spelled out what you you mean. And it covers us to have both have this sentence and then also refer to both of them here. Because just in case somebody isn't coming to this with that whole worker and containerized build environment in mind. This covers them to I don't personally, I don't think this one point of contention is going to ruin the paper. I don't think we're at an existential worry I think we're probably if anything, bike shedding a little bit on this I think we covered it by your sentence that you just put in there. Alex I know I would didn't like the addition of another sentence but I think it does actually give that preface that you need to then read this okay. I'm worried based upon how much Blake wrote here and what his response to you was in that thread that this might not be the end of this conversation. I'm going to leave this comment in here for now. If somebody feels incredibly strong about it. Let's come back to the conversation. I did like your comment here hard no CI container let's let me just read this out so that everybody in the call can hear a hard no CI container made to send from scratch, such as empty root file system suitable for statically that's not a great sentence for statically linked binaries. Distro lists such as scratch with local and public certificates or an organization's manage minimal base image, such as red hat TBI with additional public keys or internal configuration. What is that saying. Yeah, so I think there is one expert is missing there like a, there is also a pattern where scratch based images are using like a side car approach like a, you know, expertly handling TLS and certificates or floated to side car. So I think what I understood what I understood is if you if the scratch option is available then a UBI minimal base image is another option. But I think scratch plus side car is also a good alternative to UBI base image right so. Yeah, it doesn't. I mean, I actually think you could take out this entire middle section here. I mean, I because the distro list such as scratch with locale locale and public certificates doesn't actually provide any. I mean that's just saying scratch with your own stuff put on it you're still saying scratch. The whole goal of this paragraph seems to be use a scratch image as your base. Right. That's the that's all it saying. Like six options about or expanding on that by saying that you can put your put public certificates and locale information on it doesn't help. Can we just call it a minimal image and be done with it. Yeah, yeah, I mean I'm good with minimal image. So hard. I'm not even going to turn on suggesting. By the way, I don't know who owns this document. But anonymous people can still come in and edit. I know that because I stupidly didn't sign in because Gmail doesn't have or Google doesn't handle. Can we turn off anonymous edits. This is the same picture I was asking last time like to disable the editing right like I think I have the ability to. I'll bother him. Yeah, I guess I can't share it. Can I, that's not going to actually do anything. Yep. Okay. I'll bother John. I'll see if he can, if he can edit it. Yeah, that'd be it. I'll look it in the information but okay so for rewriting this particular quite long sentence. May is also the wrong word. I mean, should let's let's start with we're back to recommending we're not. It doesn't just happen by chance that it descends a hardened OCI container should descend from. I think the double quote scratch that's like from images. Yeah. Right, right. Otherwise we got to do you want to use Google's minimal image. I don't want to talk about that. Nope. Nope. And then do we want to keep in this or an organization's managed minimal image minimal base image such as red hats you bi. Do we need to include this. You probably say organizational minimal image, and then pick up the red hat stuff. I don't know if that's probably duplicate language. Yeah, because it's your minimal image. Someone's got to own it right. I think we cover, you know, a testing to what is in the image other places in the paper. Yes. You put it in brackets some examples of people who read at least some actionable things right like they know when we don't want to confuse people so at the same time right like a what is a minimal image everything is classified as a minimal image like you bi. Google container minimal image that you also they will get it later on the paper. Yeah. Okay. Do we do we say, such as scratch images. Yeah, we could probably define what a minimal images in the glossary and they say minimal images defined as scratch, you bi, whatever Google the one that Google has or similar artifact. Yeah. Minimal image in glossary. Bingo. All right, cool. Additionally tools such as open scap gas what is Blake and Blake like wrote another paragraph in the comment about the paragraph. Actually be shared. It's like so many. Testing to application testing and image scanning. All right. I don't actually to be honest this this this first sentence to me, used to validate the removal of potentially sensitive files or configurations in the container at build time what does that mean. Validate the removal of potentially sensitive files is very I mean that that that. What does that mean does anybody have a sense secrets, secret sanitation, maybe that they're removed. My guess is that this is referring to something like a Docker slim sort of situation. Well, I mean the the tools themselves look at the right exactly. Alex. I guess it verifies that you are using a minimal image, or you're using an image with. Yeah, I think by potentially sensitive files and configurations it means, you know, it has something running on it that you didn't want running in your environment. That's the that's the question right, or that's what this is this is saying, you can use tools to validate the actual container images you're running to make sure that you are you're you're not running. I don't know. I don't know if that's a good idea, or something that's going to. Yeah, I think it's exposed. But I don't think that sentence says that eloquently, or even concisely. Can we, can we collapse this whole paragraph into one sentence added to the paragraph above it that just says, you know, your bit you should then validate your base image with tools that ensure that there's nothing else running at that you didn't that's not very well said but you know something to that effect. Yeah, I think we should use a term verify. Sorry. Yeah, sorry. Yeah, sorry. Yeah, I think we shouldn't use base image because it can be the final image also right so people can run open as cap against the final limit so we should run as cap our premium tool that we recommend here, which if I only have the list one, which one would it be that probably as cap it's the MITRE tool, additionally tools such as open as cap should be used to validate that an image is truly minimal that image contents meets organizational policy. Yeah. Yeah. That images contents let's let's be meet organizational policy and security best practices minimalism. I don't know that does that does that Alex back to your point does that say what you what was supposed to be said here. I think there is a value adding and security best practice because many organization may not have a image policies like you know it is an interesting thing right like I can type I swear folks. That is called you offended by or security best practice. Oh, no, I'm sorry I was. Oh, it's alright second. I know it looks good to me. Alex, rip this out. Yep, sounds good to me. Also, I love deleting languages almost as fun as deleting code. I feel like I'm controlling a wrecking ball. All right. Let's look at this little sentence. runtime hardening should be provided by the underlying orchestration platform protections may include policies like set com app armor. Wait, what. Okay, there's something with this. Something with this sentence is off. The conditions may include policies like set com app armor and se Linux network isolation and build artifact validation he is listing within a list. And that makes it a list inception, which violates one of the Geneva conventions I always forget which one. What is the saying should be provided by the underlying orchestration. In general, like what that first sentence is like the idea there being that you know the orchestration platform I assume being something like sure netties. Right. But I agree that the second sentence. I'm not exactly sure like protections may include policies like, and then it's certain that our policies certain things that aren't policies. Right, right, right, right. This is this is. I don't need to say this just doesn't is it valuable to have this in there. I mean, back to you is actionable to say that that runtime hardening should be done by the orchestration platform. Would that be news to anybody. No, I think there is a value there but I think to get a proper value. I think that that need to be another big paragraph like it's so things out there and it's not giving a clear guidance like it's just putting some names or policies, but it is a bit complicated like it in my view so. We talk about I know we talk about network, we definitely have an entire section of these two so this is covered. Yeah, I don't know about sec comp app armor I see Linux I don't recall that being in another part of the paper, anybody. No, but I think just I think you're right but not it feels wrong to just dangle it there. Yeah, like, nobody wants to work with se Linux. Where do we talk about, where do we talk about locking down the, the build worker, right, maybe that should be long there. That's where we should be doing se Linux policies when we're talking about locking it down. environment. Let's see if if I can get it from one of our. It's not securing artifacts. It's not securing. Like we're missing the the actual title here somehow this isn't in securing we were missing this is at the wrong level, somebody changed this. This should be heading one. There you go. Okay, sorry. Let's see cryptographically guaranteed policy adherence, validate build artifacts validate environment dependencies before usage would that be what we're where you would talk about runtime. Yeah, I think this this goes right into mitigating the solar winds right this is the only thing that would mitigate is locking down those those calls. That does not seem to be the case there. Yeah. This is this is this is a bit this is quite a big paragraph. Validate environments. This has got to be it. But it does. This would usually include things like validating get commit hashes and checkings. Do we need to create a new recommendation around run time security. I mean, I think as well that that seems to be something completely missing. All right. Throw the heading in there I'll start working on that. Yeah, so how about this. Just basically be a copy of this. Validate runtime security of build workers. Not just build workers as it's a. What was that. I was saying that the runtime is also for the final application container right like you don't want the application content. But that would be handled down here and securing deployments. Yeah, I mean like this is we're still talking about the build worker at this point. And I think that's what that original sentence that we were just angry at was saying as well. Yeah. Yeah. Like we want to, we want to enforce the policy on the build workers, right? So the machine will have that policy and then we want to the containers should. You know, meet that policy, but we needed that other auto band verification that, hey, we're not, we're not executing stuff. We shouldn't be. I think it's all I'm going to try to go after that. Yeah, I think you're right. Can I just delete the other sentence. Or the other, or should I copy that paragraph down. You can copy it in there. I'll probably read it, but I'll use that as a reference and grab some of that language in there. Previous paragraph. But anonymous neon cat. Let's go back here. I'm just going to paste it under here. Just so you have the reference including the comments. Okay. Thank you call for Do we reference immutable and potentially ephemeral pipelines. I see the word immutable bolded. I would have bolded that by the way, nothing else in the papers bolded and I would have bolded everything if I knew that we could bold maybe happen when tried to come an accident. Okay. Oh, I will. The orchestrator stands up an immutable pipeline, leveraging infrastructure as code. Cool. We referenced immutability, ephemerality. I'm just going to go back here when the components of the supply chain over the orchestrator. Anybody feel strongly about the fact that it doesn't specify. Anything anything about ephemeral pipelines. Yeah, I don't know what it means by ephemeral. What, what. What we mean by ephemeral pipelines. What it means is that the build workers are ephemeral that they die after they do their run that's got to be what it means. Oh, yeah. But that's I think you're I think you're bringing up a good point Michael that why would we specify that here. Oh, we do. We do talk about that I believe just above. So actually one of the things I was. Yeah, and one of the things I was actually concerned by I was actually actually confused by what was meant there because ephemeral like if I was actually curious if the pipeline itself was ephemeral as it based on some sort of logic, it would figure out, oh, I should be running this pipeline and actually we want to sort of recommend against necessarily that sort of thing because we want to make sure that the pipelines are very clearly defined. And we don't want to actually have a I would at least think we don't want to actually have a lot of logic in how we do something there is like you know the pipeline when I think about it should not have like the logic should be very straightforward and simple to say oh the pipeline says you should be building this thing or be doing this thing the pipeline should not be saying well based on a bunch of logic and heuristics and whatever it should be running this random thing. I want to make sure that in that was where I was just a little confused by but I think yeah so I was just a little confused by what is meant by ephemeral in that case. Well, it's funny because as you just saw, he does no ephemeral credentials is fine creation of multiple ephemeral and ephemeral immutable pipeline so yet again he is using this term of ephemeral pipelines. But I still I still wonder if you can. It's normally a defined set of codes and step right like it's not really a firm like the orchestration part maybe that the containers that are built as part of the pipeline that should be a firm like. Definitely. But when you say that the pipeline is ephemeral if all of its containers that are part of it but not are ephemeral. Well, well hold on like I think the thing there that the way I would argue is right you know the pipeline if I'm if I have a pipeline right and I'm thinking of the so I guess the way I'm thinking about it is is the pipeline is essentially. You know a set of code that tells the ephemeral workers what they should be doing. Right. And if you were to say the pipeline itself is ephemeral I'm thinking that code is ephemeral as in that code is short lived or perhaps generated by some other process. And maybe it's a it's a pedantic sort of argument but but that that's the way I sort of read that. Okay, so going back to that Michael. Do you feel strongly about that like that's to me I'm like no that's not important the ephemeral. The actual pipeline know screw that that's not worth. Yeah, yeah, no organization awesome but otherwise yeah no it's it's not a huge. Okay, so are we are we offended by it being left here or no. Do you think that's confusing. We could we could do, we could do multiple comma immutable pipelines, or I guess no comma that's bad English. Let's do that, because otherwise it's confusing. Or we can say pipeline workers instead of pipeline right so that is another option. Right. Now let's not bring up the word workers any more than we have to. I think we were done here. I think I think it's still makes sense. Yes, we bought in front of John thanks John. Thanks Cole that's looking good already. Yeah I think interest I like it right there. Let me know if that meets the intent there. I just grabbed me nuts. Oh, I know I'm horrible at spelling. That's all right. All right, out of band verification of execution policy with tools such as out of band verification of like execution policy. What's execution policy. I'm, I'm, if you had to break that down to a five year old, what would we, what do we call that for three years. Single syllables please. See if we got any prior work here possible rules that should be created and applied to build infrastructure. Yeah, I'm trying to, because I see a lot of like SE Linux policy or app armor policy so I'm trying to abstract that out. How about this out of band verification of runtime environment security as defined by execution policy with tools such as does that clear it up a little bit. Yeah, yeah that adds the clarity I think it needs. So that's the environment security defined by execution policies with tools such as that that I can read and I can be like okay, there are execution policies that come from set comp app armor SE Linux that nobody ever wants to deal with because it's boring that help verify the runtime environment security that is that that makes. I can see what the importance is there. The real sets should be created and applied to build infrastructure cool. I've provoked kernel capabilities such as debugger attachment should be restricted and monitor. Yeah, finding should be forward to organizational security and systems for remediation. I thought you were going to say teams like I don't want to send it to them they're not going to do anything. I think this is I anybody. Yeah, Alex you're going to go go through it like an hour from now and be like not destroyed this paragraph. All right, I'm going to click check thanks a bunch cool that was a quick turnaround there. Next one. We should also suggest pinning the. Yeah dependency pinning who wants to take that. I got the last one. I mean, so I read that comment. Also, I might have been something in here about dependency pinning a little bit later on but maybe I just made that up in my head. Let's let's use our friend control. Nope. At least they didn't use the word pin. Okay. Of course lost it. I did have a question about this, mainly on. No, what did I just completely lose it. First off, this recommendation to remove or verify external requirements from the build process. Doesn't that seem like conflicting advice. Do you either want to remove it or verify it. So verify. Cool. That makes sense remove that's really actionable and I better see concrete examples about what I removed. Anybody have a good example, or a good good like, I mean, external sources resources can change or disappear unexpectedly third party packages that are a part of your build process can be It is not feasible or possible to do so recording hashes of any remote data. I think we should do it. I mean this has to be verified you can't say remove. We're not telling people how to remove external sources if you have external requirements, they're there already just verify that they exist. I think basically like for each requirement you should either verify that exists or remove it if it doesn't rain. Can we just say marine on that note if we if we change it to just verify. Can we have like one quick little zinger at the end. Like if you are unable to verify the external and external requirement. You should remove that reply. I'll go tell my CTO that or CEO that just remove the requirement. Good. Don't worry you didn't need the requests library anyway. All right. Yeah, but not did I just I just trigger you. It's like finance job to verify. What's up. No, I think reading that it is kind of not same across all the line programming languages. So, I don't know what is external sources if artifact itself is an external source build server. It is an external source of rendering. I don't know what is a practicality of rendering in all programming languages right so in my view. It should be like the dependency should be free and the frozen before before the build time like not put like in, you know, like NPM or Python greater than or less than or something like that and then getting a version during the build time. So instead of that explicitly mentioned the version number so that there is a consistency during the build on each time rather than is built. So it may have different behavior, because the artifact may have a new version for that package and then they will get a new version next time right so when doing I don't know how practical it is even go lang mode from vendor to go module right so I don't know it is a bit complicated area. I know what you're saying and I believe in the paper we do recommend storing all the artifacts yourself right so not pulling from, is it an external source, or is it an external requirement you're pulling it from your own artifact itself is mirroring from external like Maven Central or PyPy Central right so artifact is an internal clone for external artifacts right so it is, but even the artifact itself is outside a build system it's not a part of a Jenkins pipeline of the build process but I mean in my opinion we are over complicating this and answer or you know it is more like what John the pinning dependencies is the main thing here right we need to freeze the dependencies we shouldn't, we should be able to reproduce it every time we build it right so we know one should mention greater than less than this version or something like they should be a specific explicit version number, which should be same across all the bills right so that's my view instead of highlighting when during when during may not be practical. Is this too generic then this is this is all just too much generic advice you're basically saying the recommendations should be here in your version your external requirements and verify those external requirements are what you get during the build. Yes, I mean that that's what I instead of rendering that's what I've been looking. Yeah. Can we get a second to that motion. I'm okay with that. I don't know if that is moderate to high risk in categories. I mean that might be high. Yeah, it bit confusing different lines are the first one is a reaching out to external sources at build time. That's fine but he's a internal artifact is an external source, maybe may not be but even artifact tree. You know, sometimes if it is not in the artifact it will try to pull it from the external like memento and things like that but there can be always that challenge external resources can change and disappear. But, you know, if it is an already fetched artifact it should be in the local artifact right so sure. There will be cash. The question is what is the actual work recommendation to somebody about reproducible builds and pulling down external requirements is it that you pin the version and get the same exact external source every single time, verifying it via the hash of it or whatnot. Is that the recommendation. I don't think we can recommend that we're not there yet. So I think we can say this is what we should do, but because we can't do that the community isn't there yet, you know, we, you should incorporate that into your risk assessment of the software. Yeah, okay. I mean I'm kind of with Cole in the sense that that's great advice but it's not feasible probably for most organizations. So we just say hey, assess the risk of this if there's a bunch of stuff you don't know about it don't give it level access in your system right. Okay. Yeah, I like that to Vinod's point. Ideally, pinning specific versions of external requirements increases. Yeah, I don't I mean we can't start with, you know, let's take out ideally the entire page or paper is one big ideally sense. So we're going to be using specific versions of external requirements increases the consistency and ability to verify build assets through a software throughout soft. Look at that, I just gave the project a website. Do if we do include, I mean since we're including this do we also want to make a recommendation about specifically here about keeping, you know, the pen versions up to date, as things change in the external resources. You know, it's funny. I mean, at that shouldn't you wouldn't you get like a presumably your scanning tools would tell you right, they'd be like, well, Yeah, it might contradict with what we are pinning right so if you're saying up to date most people will use latest tag or without explicit version, which will again the control will be Yeah, so you don't know what version you're exactly getting during the bill because there will be a new version in the factory so the checksums and everything is different. So, ideally, you know, the version should be pinned and the tools additional tool should be used to get get update info, like depend about or similar tools should be there to help you with the regular base. Yeah, I was thinking along the lines of dependent bought or renovate bought and so on, rather than pinning to a latest tag. Yeah, but it's still it still means that even if you are using dependent bought or renovate but you still should pin to that one version. I don't see why you never. Sorry, can you repeat the rich actually. I mean, if you're if you're using even if you're using dependent bought or renovate by the recommendation to pin. It still exists, you should still should pin to that existing version. There is a challenge with the pinning right so what mostly developers went though once they explicitly mentioned a version. They don't change is they don't update it that frequently right like a. So that's one of the reason like some people use latest or don't mention the actual version so you know using pinning can introduce this problem of not not frequently get updated the software right so I think it's a good idea to recommend them. Dependable or renovate but along the same that to avoid any, you know legacy versions use tools like dependent bought to get a regular update versions right so they can people can, even though they explicitly mentioned the version in their form file or they will get alerted about new version they can update it to you know updating is also security piece right like if people don't update it. I just think that that should be a different, a different recommendation right. It is but this is a side effect of pinning right like if you pin explicit version, you may kind of stick with that version forever I mean. Yeah, but you should also have a system like dependent bought or renovate bought that you know even if I pin, if I create an open source project today, and I pin a specific version of a Python package dependent bought is still going to yell at me. Yeah. Yeah, that's a good thing but if dependent was bought was not there like like many other companies like Bitpocket or GitLab type of repository, they just keep that old version forever right so they can use renovate bought in that case to avoid the problem. Yeah. Who would use Bitpocket. Yeah, I agree, I agree. Does that need to be. Does that need to be one extra sentence. Keep in mind keep in mind with when you pin specific versions. You run the risk of not getting the updated version of not using the updated versions of a package later though I'm yet again I'm feel like that's rare. I'm sorry guys I know we reached the top of the hour. So it's good to add that line in my opinion but yeah but. Throw it in there can I can I put that on you either Vinod or Aditya. Yeah, yeah, sure. Just just like a quick one. Nothing, nothing too long. So, I know we're over. If anybody can continue going through, look at the comments. If it's not something that you feel confident over writing, add a comment in and feel the fire. I mean, I think our goal right now should be to clear out all these points of contention. And then, I mean that that should be don't even go and read it first let's let's actually knock out all these pieces because we can always add more comments in the next round of edits, but this is getting significantly more coherent. Getting closer to the point of being able to just read it top to bottom. And yeah, seriously anything over the weekend. I wouldn't mind next week, putting in a little bit of time to do this again I think this is this is useful to just have very straightforward pushing through. So, thank you all for coming. Have a great rest of your weekend. Look forward to finishing this out. Adios.