 Good morning folks. Morning Steve. Some new faces. Wow. Everybody get a new style for 2021 or we just got the word out. Well, there was this hack that happened. Turns out people are a little interested in that. We'll give folks a few minutes to get on. Give me a second. Right. I posted, I'll post the real link of it. When I tried to post. The agenda from my phone yesterday. I wasn't able to get that posted. So. This morning. So here's the hack doc. Folks want to sign in. Welcome to us to 2021. Hey, Steve. As arena. All right. So in. Oh, actually I didn't post it there. Let me grab the. So is anybody else having trouble with slack? On the channel. Back is down. Yeah, there's a big slack outage. Okay. If anybody's got. Well, how can I get that in there? If anybody's got the link that we had, I put it in the notary chat, but it's from Twitter. That we can. We can just post that for others if they wanted to read, or I can get them to the notes afterwards. I think I. The, the, the. The one. Yeah, I have actually got. I was just. Yeah, I think it's this one. Basically, so yeah, there's lots of news. And of course the press kind of picked up on the whole Russian angle of it. And for those of us here, we probably don't care who did it. It's just the, how did it get done? And what can we do to prevent it to. One of the things I thought was interesting was that how quickly they figured out that it was actually in the build system. So sometimes it's not just preventing it. It's being having some traceability about it also. So that I thought was pretty interesting. Okay, so we'll get that link posted. And if anybody else has any more links on more deeper. Info around what was actually done. That's always helpful. The. The summary of from what I read from this article and a couple other articles that came out in December when it first occurred. Was the build system was hacked. It was later reported that there was actually no source code. The source code that was being built per se was not hacked. And the distribution system was not. Exploited. In fact, it was the distribution system. Well, I guess the distributions was exploited. They shipped updates to the existing package that came out of the, the build system. So it was a valid build. And we're shipped as updates. So we often get concerned around making sure updates are current. In fact, we have this conversation a lot around containers of you always want to grab the latest container. Latest or newest, whatever the tag might be. And every time you do a build, you're always pulling public packages. But in this case, the, the fact that they were keeping current turned out to be the problem. So I thought that was kind of interesting. And I think the, what I also read early on, I need to go back and find the articles and they're actually in Twitter, because that's where I first got a note of it. That it was. The packages were signed by solar winds. And they were, because of that, they were actually very quickly traced it back to. That it was the build system that produced valid builds. And I think that's something that I have to double check this. That it was dot net code. Which the only thing there has kind of helped me raised on a little bit in my own and I have to reach out to a couple more folks. They're just coming back to vacation of the various ways that. If it wasn't source code. And now I'm in total hyperbole here totally hypothesizing. If it was packages that I pulled in. That how could a package. I could either get pulled in that was, had the exploit or the package list. That is part of the build system been changed. I don't know what that calls it, but there's basically a package jam or some sort of chase on file. Dot net has this injection model that decorative injection. Just by having something declared, it could, it will automatically start running code. So those are, those are totally my guesses. So just trying to brainstorm on some ideas and the ideas. I think that's one of the, one, how could we mitigate from an exploit of this type as much as we know? And is there anything we need, anything we need to adapt on the plans that we've had thus far? So that was my little two cents. I'll let others. I think that link might have been the one that was the one that analyzed the dot net carried. I just posted it. Yeah. So I put together a quick presentation about kind of tough side. I think as we learn more about the attack, I think it's definitely a little bit more on the end Toto side than the tough side because of where in the process it fell. But I can go ahead and present that and talk about that. If people are interested. Yeah. Before we get into a presentation, just because we have a bunch of new folks here. Does anybody else got any thoughts or comments? I do have a question. Yeah. Yeah. I think the code repository itself wasn't compromised. I think based on the articles of red so far, it wasn't clear whether this was an issue with the code repository itself being compromised or whether the build system also was compromised. Just the build system. Yeah. I mean, it seems the SEC filing from Zola wins says. The source code was says that the malicious code was not in the source code repository. Okay. And so. At least isn't, I mean, isn't now. I don't know on what, I mean, I think that it's still somewhat conjectural because like it could have been. In that. Potentially in the source code repository for a while and then deleted, but I think maybe that's unlikely. I don't know. I mean, it's the, some of the analysis suggested that the code was written so that if people looked at it, it looked reasonable, which made me think that either they're expecting it to be the code to be kind of reverse engineered or and maybe in testing or they were, were expecting people to see the source code for it. So I think it's a little bit hung glass at this point. Yeah. Yeah. That's fine. The stuff that I read was to this one, this article that the SEC find the first one that Justin posted specifically talked about the source code wasn't exploited. But what's interesting is what we all know with build systems is there's lots of things that are pulled into run the build system. So I don't know if a script was hacked or what. So the source code that compiles the code was declared as not being violated. The other things that I heard was what the news was reporting as part and all the various references was the early article I read and I'll double check the one that Justin put there. Talked about it would stay dormant for like two weeks so that you could not monitor malicious activity. Like if you write a scan on the code right away, it didn't show anything for normal patterns. It was only after a period of two weeks that it would start looking at that. It's on the machine is what the way it was described. So it wasn't like social security numbers. It was like, what other information could have access to that? And I have to find the, the articles that went back and reference that specifically. So it's interesting is that the source code wasn't hacked. What in the build system was hacked? Was there a bits that were laid on there? Was it the scripts that run the build were hacked? Those are the things I haven't seen more detail yet. But those are, that's a great question. Cause that was the thing that I've been working with. We've talked about ephemeral clients for build systems as well. Anybody else. Okay. Brandon, you were kind of active on these threads while I was a bunch of us were trying to take vacation. Yeah. That's why I jumped in and joined in on this call, but nothing for me to add. I was just more to be here in case any comments came up. I have to nobody else. Marina, it's all yours. Okay. Let me see if I can get screen sharing working. Can people see that? So you have these first couple of slides kind of go through what we've already talked about, like, you know, what was the solar winds attack. I think I'll go to like this. I think when I present it messes up the screen sharing. But, um, so, um, yes, there's a compromise release of solar, of the solar winds software. And, um, but they're currently saying it was somewhere in the build process and there's something between when the source code is written and when they actually made the release. And because it was the build process, there was a valid signature on the software, but that it was. Because like the build system was compromised, whatever came out of the build system was, you know, it was not signed even though it was malicious. And so, um, it's definitely one thing to look at and maybe how we can prevent malicious things from being signed by the system. Um, and once installed, the compromise allowed the attackers to generate, um, sign on tokens, um, gain access to systems and generally cause a lot of trouble once it got on the systems. But for now, it's focused on kind of that first part of how they got on to the systems. Um, and that's what we're talking about. This was a supply chain vulnerability, um, in the build system. And it was not detected either by the signing entity or the customers. So nothing after the build system had any way to detect that something funny had happened during the build system process. And so that's kind of what the, what we need to do to kind of prevent a check like this would be to give somebody visibility into what's happening in this process. Um, I think that's a good point. Um, I don't think that anything we talk about here getting exploited as, as this was definitive. We should be careful. Like we, there's, I haven't seen any information that it was Jenkins or Docker. That was part of this. Oh, yes. This isn't. Um, yes. You're definitely correct. The images. Um, illustrative. Not like. Yeah. So, um, yeah, don't take that. Recording of this. I just want to, if you share this slide, I think we should be careful. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. For example, or whatever. Yeah. This is good. This is what we were doing is focus here in the container stuff. Yeah. But we don't know what, I don't think anybody knows any more facts on this. Yeah. And this is how we can, how we can protect these systems, not necessarily. Um, their systems. So that's kind of what was here. I mean, I think we know it was mostly when it was almost all windows software. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. These are the ones we'll probably talk about, you know, how we can make them better. Um, so then I'll talk a little bit about in total and about tough. So in total, um, protects the build process from code to packaging. So it kind of the whole process between. When the freaking, when the code is written by developers to when it's actually sent out and then it kind of moves on to tough, which I'll go over. In a second. Um, and this is another example again, not necessarily what they did in this particular case is that, you know, you have the right step packaging and then. Inspection. And then, um, so in each of these stages, you can add in total metadata that says, okay, this is what came in at my stage. And I'm going to sign that that's what I got. And then the next person says, yes, I got, you know, what you signed. This is what I produced. I'll sign that step as well. And kind of a test cryptographically that each step was actually followed and the correct person or entity followed each correct step. In this process. Um, and then I think some of you might be more familiar with, with tough, but I'll, I'll go over this as well. So tough protects the release process from after software is packaged to when it is installed. And in some ways this is a little bit different because the, um, the actual bits don't change between packaging installation, which makes this a little bit more. I want to say easier, but a little more straightforward because you want to make sure that the user downloads the exact same bits that were packaged. And that's kind of the whole goal of tough. So, um, yeah. So once this package, the developer or whatever packaging entity can attach, um, sign metadata to this image. Um, there's a lot of other steps to tough, but essentially the sign metadata ends up on the client side. And then it's verified and the client can make sure that the metadata matches, um, the package that was installed. So combining tough and in total, this is kind of the meat of, of what I want to propose is that, um, you can use tough to also contain this in total chain, which then connects it kind of end to end from the first time someone's writing code to when the package is installed on the client system. And you can verify that entire chain of the supply chain using the combination of these two technologies. So, um, in the sense kind of what's said here, the tough is the transport protocol for the in total, in total attestations. And there's a way in tough targets metadata to include the in total metadata in that location. So you can kind of a test all the way down. And then both the client can, can verify that entire end to end process and anyone, um, who's uploading this to a repository can also check that ahead of time for the clients. So, um, so for example, in the, the SolarWinds case, which again, we don't know too much about, but the person who is in charge of attaching that SolarWinds signature could have verified the process ahead of time before actually attaching the signature that then clients used to verify the software. So, um, it could kind of be stopped at two different points, both by not attaching the signature and by the clients being able to check. Can you explain a little more because I, if we assume that based on the knowledge we have, that the build was valid and the, from the time that the components were built and shipped to the various clients, they were also validated. What is it that tougher in Toto would have stopped it dead in the tracks? How would that have done? Yeah. So that's actually, that's the, we got question here. So, um, again, yeah, it was definitely something in the build system that was compromised. That's, I think as much detail as we know right now. Um, so in this, this layout, the attackers would have to either compromise the signing keys for the build process, which I think for now we'll assume they wouldn't do. And then, um, which of course requires some, some work, but, um, and the other thing, or the other thing they could do is, um, they would have to make sure that there's no mismatch between the metadata that is signed. Um, I think in practice, something like reproducible builds and independent rebuilders, um, really guarantee that, um, this mismatch would be detected because, um, the artifacts would be different. So, um, the way, the most built systems, one of the reasons they're hard to secure and guarantee is that, um, the artifact is different every time it's built as a very easy example, like the, the time that it's built is often included in it. And so if you compare something that was built twice right next to each other, even on the same system, the, the binaries often don't match. And reproducible builds works to create a version that doesn't have differences between these binaries so that you can detect, um, a malicious build. Um, and I think that's the strongest protection that you can, you can have in this system is you can have, you know, a sign from, okay, this is what it was before it was built. This is what it was after it's built. And it's, it should be that every single time. I think short of that, if you are not able to do fully reproducible builds, what the in total metadata is able to do is say that, you know, this computer or this person built the software, um, which depending on what kind of build system compromise might have been able to detect the system anyway. It's like, you know, some third party computer had done the malicious build and then just giving it back to the rest of the supply chain, that would be detected, for example. But, um, if they actually were compromised, the server that contained the keys used for the build, I think that you would need something stronger, like reproducible builds. And I think someone, a little bit about like, so the way I think about every pre-producible built, and I'm not, I'm asking the, I'm stating it with a question. Yeah. Reproducible builds are awesome for open source projects that, you know, it's built by some, the open source project initiative or whatever, that I'm not trying to pick on any cloud. We pull Kubernetes images from GCR, let's just say. Um, but we've also been concerned, for instance, in Azure and other places in this well. Um, so we actually don't pull the images from GCR. We actually rebuild them ourselves and gives us a chance to patch. So in that case, the reproducible builds would catch it because if, uh, we could also meet in this case, we could have introduced a vulnerability in theory, right? Uh, because some build system validly got something in, and then the, the gazintas and the gazaltas are different. So the build system from GCR would be different from an Azure build, speak one way or another, and that way you could compare it. Is the idea of reproducible builds is also done in this case, which is solo ends as a specific ISV. There's no open source effort around the thing they distribute. I, are we suggesting that reproducible builds also help that even in, uh, a private environment that multiple machines are being validated against each other? Is that what you're suggesting? Yeah. So I can let all that other, um, I don't know if Santiago made it to weigh in as well, but I think that what I'm thinking of is that even within a private company, you could create, create, say for example, two different servers that built the software. And if they could verify that those two servers built the same software, you know, that would be better than just one server, right? If one of the build servers was, uh, hacked and they were able to inject something into that built server that in theory, hopefully other build servers weren't, and then you would see a delta between the two and you can compare. Exactly. And, you know, you could do something like, you know, different systems, different passwords, all that kind of stuff to try and ensure that they're both not compromised. Um, and again, this just depends on your, um, your threat model and stuff and how much, how many layers of protection. But I think this is definitely, if the build system is the concern, we've reduced for builds is a really key way to, um, protect that stuff with the supply chain. And then that's, Sorry. Go ahead. Go ahead. Yeah. That's perfect description actually. Um, I don't know how many of you are hobbling Bitcoin right now. Here it's a skyrocketing. Uh, but the way they, the way they build Bitcoin binaries, it's a deterministic virtual machine that removes all the sources of non-determinism. Uh, Marina was talking about three different developers will independently pull the same source code for assuming that they got the same source code, build it on, on their own virtual machines and they will all check whether they got the same hatches. If they don't, they don't release it. You can actually formalize this using tough and endoto and have a cryptographically verifiable record. It's a shame that they don't do that. You have to take the developers word for it now, but if they just added, you know, this, uh, tough and endoto, the public can actually verify what they did independently. I think there's a, uh, a bit of a jump here in an assertion. So we can, through tough and endoto verify the steps that were taken. Uh, we can't verify if any of the steps in itself was compromised, right? Like even if you have redundant, like if you have reproducible builds, you have three build machines, uh, depending on how those build systems end up getting managed, um, a compromise of one could lead to a compromise of others, depending on how, uh, the company is securing those build systems. Right. I think this is one where we've kind of talked about, you know, single root versus multiple roots, uh, in cases of like a large enterprise, they may have the resources to correctly isolate them. Uh, cause it does take a lot of effort to isolate different build systems, isolate different key systems. So I am, I still don't quite see, um, where tough and endoto kind of like bridge that gap. Um, it's kind of giving us the same assurances as any other S-bomb integration would and saying these were the steps that were followed. Um, but it doesn't really tell us, you know, if a step itself was compromised, um, unless you can go down and verify from the source codes itself, which is certainly possible for interpretive languages like Python and YAML, but I, I still don't follow us to like, you know, how, um, for something like an ISV, like it would have helped mitigate this, this threat model. Well, even, so that's a valid point, but even in the case, so compared that to what we have today, which is nothing, but Solovins, for example, you have no, no cryptographically, you know, there's, there's no record where someone can repudiate their signature right now. It's like, I don't know, I don't know who did this. I don't know who the machine was compromised. That was one of the things that was interesting about this is, and I think it was, it was the first link with the second link rather the first thing I read that first came out is that because the content was signed, that's how they knew it went back to the build system. It wasn't hacked in the middle, a man in the middle of whatever you want to call it, a distribution attack. So the content was signed. So they did know where it went back, how far it went back and where in the build system that hasn't been determined yet. And that's the part that I'm trying to think, you know, understand here is, you know, there's multiple lines, right? There's the whole supply chain. And to Niaz's point, there's interpretive languages where it carries all the way through and you kind of see it right to the end. There's other languages as again, what we know about this was .NET. So it was compiled code and whether the source was in the compiled code or not, we don't know yet. But the point is that for compiled code and binaries, which is what we think about with most container images, there's a transposition that happens. So what I can see, because there's things we know, there's things we don't know, but we know how build systems run so we can make some hypotheses around what could have happened. So I mentioned, right, the, a package that they referenced could have been hacked. And if those packages were signed with Tough and Toto or anything in that matter and they were doing checking on that, they could have caught that. If the package JSON file that is normally produced in the source code was hacked and said, hey, in addition to all the things that were in the source code, also bringing this other package that through injection can actually load itself. So that's just a pattern that multiple languages support and .NET supports it from what I know, I don't know what version of .NET that we're using. I'm considering a modern version of .NET. So that's another possibility. So in that case, if a comparison is done from what was going in and coming out of the build. So if two machines, one was hacked and the other one wasn't, that's certainly a comparison that can be done on the reproducible builds. Like, hey, these two things produced different things. What's up? Let's go check. If all machines were compromised, then it's possible that all machines were producing the same thing, but the thing they produced is different than what they expected. So the more I thought about this conversation, the tougher in total and more so, I guess, the tough stuff is, or the signings that Santiago had been doing, what's been working on with the software, bill of materials work, everywhere in the supply chain up until the point it's got built is the part that seems to be the vulnerable place here. And that would be a great place to be able to implement this. The best that we seem to have been able to do in this case is because it did officially come out of the build system and got distributed. The best you can do is identify that, hey, it did come from the build system. And we don't need to look elsewhere. We know that these were valid signatures, valid keys. And now we've got to figure out how the build system got hacked. And hopefully they catch it earlier than from March to December. And it's caught through another multi-line defensive attack where something's monitoring the code and it sees the network traffic has changed. Alarms go off, somebody investigates and then they caught it. Does anybody know how they caught this by chance? I don't actually know. I think that part of the idea of something like in Toto or another S-bomb mechanism is that if you have people looking at these supply chain, you know, attestations or whatever you want to call them, they, like the more eyes you have, the faster you'll catch it. At the very least, you'll catch it faster. And ideally you would have people checking the process before it goes to customers and the customer is able to check the process. But at the very least, this allows internally to the company for them to do auditing, you know, hopefully pretty often and detect these kinds of things very, very quickly before the attackers have six months to get into the system and get the kind of access that they did. I think that's a very difficult thing to do at scale, right? Like if you're going to go in and check every single release in terms of what steps were followed and verify whether the steps weren't done correctly, it's hard to define a standard layout, right? We need to give developers flexibility in how they build code and at the same time ensure that there is auditability in place. And so auditability can be done through a variety of different ways. It doesn't necessarily need to be an attestation on the code itself. We have like a lot of different compliance programs, for example, that different companies are part of that go and address similar kind of attestations and saying, you know, what was the process for generating code? Were there any sort of like, what's the process and monitoring in place for detecting sort of unauthorized actions? What are sort of like the escalation controls in place? How are those monitored? So I think a lot of those controls are in place. And I think this goes down into like different companies and different developers in terms of what that company or enterprise is willing to take on in attestation and saying that when I'm publishing software, this is the standard I'm willing to meet. And I think through the community, we're going to see that different companies are going to have different attestations they're willing to make because there is a cost associated with going and making these as well. So I think the challenge I see in sort of like with Tough and in Toto is that we're making a series of assumptions that if things were set up this way and if all these steps could be verified, then yes, we'd have a great working solution. But I still don't think that that's a general solution that everyone can adopt because of some of the inflexibilities that I think are going to come up as challenges. It's one where my take on it right now is in Toto should be an option for anyone that wants to use it. But I still see challenges in making it sort of like a default solution that everyone has to use and follow a certain layout. Hello. Can you hear me now? Yes, I think we can hear you. Okay. Hello. Sorry, I've been trying to talk for like eight minutes now. I think I wanted to make a couple of positions before we continue talking. One of the things that I want to talk about right now is a reproducible build right now like the project Rebuilder D is using in Toto. So you can verify that you have multiple different instances and all of them agree on the same result. The other problem here is that as you said, the build system is compromised, but the build system had a signature over the resulting build and it should have reported what it used to build. This would have given us enough information to know if it was introduced into the build system or if it was a vulnerable dependency that was introduced into the build system or was it like they actually had SSH access or whatever type of access into the build system. And the reason we don't know that is because we don't have something like in Toto in place. That's one thing. The other thing is all of what you just said in the last two minutes is exactly the problem with S-bombs as well. And the reason why in Toto is made in the way that S-bombs is made is because we don't make any assumptions as to why people or what type of information people are going to produce. But rather, we're expecting a minimal layer of information to be produced to provide meaningful supply chain verification. And S-bombs right now is a very, very elaborate piece of information that as far as I'm aware will only be useful as things made it through trust boundaries through a vendor from one vendor to another vendor. So in this case, the vendor was importing their own code, importing tools, setting up their own infrastructure. So I actually think that making this with S-bombs and S-bombs only will not fly. It's actually less scalable than in Toto. The other thing is there is a distinction between policy and producers of evidence. And I think that will also answer a lot of questions regarding the feasibility of the solution. I think that as a consumer, as a government contractor, I can define a policy and I can verify it with the information that these people are producing. If they're not producing that information, I can just deny this product. And that's exactly the type of semantic that we're trying to enable users to have. And that's why in Toto is so minimal and in reality it's just very little at stations with five fields and a way to write policies to them. So I think you can make the same security reasoning using something like an S-bomb because it's similar to X519. It's very, very elaborate. And that has its own space. But I don't know if we can claim that an S-bomb that has like its own topology and it has the last time I checked it was shy of 20-something fields with different meetings is more complicated than an S-bomb. So I think that's the definition of S-bomb. The fact that it was more of, so again, going with what we know and we don't know, but we know about build systems. So we know that the build system, we know that the build system shipped it. We don't know if different, well, we also know that this thing had been shipping for several months. So if it had been shipping for several months, let's just assume that it was either one build server or multiple build servers that were shipped for several months. So we know that in fact, a particular build server, there's lots of things that could happen, but let's go with another angle that I think could maybe shed some light on where this could help. If all build systems were built in the same thing and it was a valid build and the source code wasn't compromised, then something had to be injected into the build system. And being the user packages, we know all languages use some sort of packages. In this case, it's just yet another one. It's just another hack to particular package that was a valid package that will, a package that they were specifically referencing, and they were able to inject code into that package. So now that has taken over, or they were able to declare a new package was being added to the build that wasn't declared by the developer, but they were able to hack it to the package, package JSON file in those two cases. And then if there's more, let's talk about those also, but in the other cases, the thought was the S bomb, if there was an S bomb for the package that the, the, the package that he actually put into the build, if they did some kind of comparison, that would not match the S bomb that was certified. Now, of course, if that package itself was validly built upstream, maybe there's a question there. Or if an extra package got added to the build system, that was not part of the declaration, that there's a comparison there that could be made. Are those, that those feel like things that I could see one of these, you know, tough in total kind of scenarios or an S bomb that has enough information to have a delta comparison on it, be a way to catch that. That valid. Well, I think the concern I have is, is a little bit higher. It's a little bit different than that. Whether you use an S bomb, whether you use tougher in total, whenever any system is making an assertion, you are trusting that the system is doing the right set of steps, unless you can go back and validate from before that step and reproduce that result. You really are trusting that system that it did the right thing. Right. And which is where I think like the way we look at it, like an, if a build system does an injection of a commit, but it gives the signatures of all the other commits that it does the build on, but doesn't add in whatever in code injection it does. Unless you're able to go back to that code repository and do the build yourself, I don't see how you detect that a code injection happened through any kind of S bomb or tougher in total. So that's the gap that I think that like, you know, the way we kind of addressed that, and I think this goes outside of the signing world is that for large enterprises, like we are distributing software that's being trusted. We're not necessarily going out and saying trusted because this developer signed off on it. We have audit and compliance reports that go out and say, here's how we build software. Here's every time someone pushed a code out with our code review. Here's the approval step that went for that. And we have this attestations in place and we have this audit logs in place, which are reviewed by third party independent services. So signing is not the only way of guaranteeing that, which is what I want to get away from saying that in total is the only way to address this. There's several different ways to do it, which is why I feel this should be optional for companies to decide how they want to make this assertions, not necessarily saying here's a way to do it. And I don't agree that in total is easy to set up. It is easier than other as bomb solutions, but in terms of like, you know, when you're defining a standard layout saying here's how software needs to be generated, you are removing the flexibility if your developers to go outside of that model for valid reasons, which may arise. Right. So there is a lot of work that needs to go into producing software at scale that says, hey, like, you know, here's a standard model. Here's how we're doing sort of like, if you're doing a rollback, maybe it doesn't have code reviews, but, you know, a rollback is to a trusted code. There may be different audit, audit things that you need to kind of push through. And I think like having a standard model through in total kind of like does slow that down a little bit. So that's why I think it should be an optional thing saying, hey, here's a mechanism to do it. If you want to use it and this fits your need. But I still don't see that this is a solution that addresses the compromise, at least from what we know so far one. And two, I don't think this is something that provides the kind of software, SDLC security that we need to kind of prevent similar attacks in the future. I am honest. So a little lost here. I haven't seen any single security mechanism that doesn't get in the way of breaking the process. And if what you're saying is, well, we sometimes need to break the process, then I would really like to know how we'll differentiate between somebody breaking the process because they're good or somebody breaking the process because they're bad. Well, so that's kind of where sort of how companies constructed goes in, like how they construct their compliance reports and things. For example, FRISO and SOC go into place, right? You can have human access monitoring and saying, like, you know, if there's a change in code, like, you know, you need this F2 ticket or some other thing to track that action against, you need certain approvals and those approvals need to be noted. But there are sort of like, I think a variety of ways that different companies address that. I don't think there's a one solution that fits all. Right. So it's exactly like, there is a policy in which somebody can say, if I need to short circuit this element of the DLC, I need to have somebody to actually sign up on it. And if something goes wrong, I want to be able to walk back and see the sign off. And that's, that's exactly an attestation that you can also encode in each other. That's why it's so minimal. That's why the idea is that you can essentially keep a record of this operation in place. I don't know if I have a software version 1.0 producing, producing like the blessed way, but then I have 1.1, which is a hotfakes that needed to have some sort of bypass on security audit. I want to still keep a record of this bypass. I totally agree with that. If I may. Sorry. Could I, could I make a point? Go ahead. Thanks. I see both points of you here, which is that, I mean, yes, Nias is right in the sense that, so I think this, there might be a misunderstanding here, which is that in total doesn't tell you what to do. Right. It's actually like a plastic mold that clay thing that you play with. You have to tell it what your, what your layout looks like. Right. And so I think it's a good thing for everybody. And I agree that finalizing that layout in the real world can be very complicated and messy, right? Because you could have many, many build steps with the rules that change all the time. And I've run into practice with this myself at data though. Right. And I had to update the rules for my pipeline without breaking backwards compatibility for end users. And I don't want to get into the boring technical details here, but there's a way to do it. It can be done. So I agree with you that this are valid real world in Doto to start instrumenting every single little step in your pipeline right away. That's not how it's going to work. You can slowly iterate it and make it more complicated as you formalize all the rules. So I think there's definitely ways to go about the problem. And no one's forcing you to do anything to follow, you know, any particular rules, but it should definitely be an option for, for people who want the kind of security to, to prevent the next elements because I honestly don't know of any of this technology that would do it. I think that that's kind of like the last part is where I'd push back is there are ways of doing it today. In total is not necessarily the only way of doing it. I think in total is, is, is one of the ways of doing it. And it definitely has a lot of pros and cons. So I'm not saying that I object to in total being an optional use case. The point here is that I know that at least for like a lot of the, for the large cloud enterprises, right? As an example for AWS or Microsoft for Google, we have ISO and SOC compliances where we're providing this exact same kind of data that says like, you know, our code was pushed through following these regulations and we have audit logs that we maintain for that. I do recognize that that's not something that every company can do. There is an associated cost with generating this reports with having the monitoring in place in total can definitely provide similar kind of attestations, which is why we've been interested in the total project as well. The thing that I want to call out here is like when you talk about like a layout file going out, like this layout file then becomes another code artifact, which then becomes sort of like, you know, susceptible to sort of like a similar model of compromise because at this point you're saying that, well, we're trusting whoever's generating the layout file has generated a valid file. And at this point, like, you know, if an attacker were able to kind of go out and generate a layout file that's compromised and signed, they can then introduce additional things through there. So it's a constant kind of like things of like defense steps that we're trying to provide here, right? Yeah, good question. I mean, so actually that's the reason for using tough as the compromise resilient. It's a way to make sure that no one can pull a switcheroo on your layout once you finalize it. You can make it effectively immutable, but we can talk about it later. It's a value concern. You're definitely right. You need to set things up properly, but there's a way to, that's the reason for using tough as the secure transfer protocol here. Yeah. I'll add that that's a very, that's a very interesting point. With in total, in fact, when you verify a layout, you create an in total attestation of that verification step, which allows you to do a lot of these things. Say, you want to have a step within the chain in which you verify a subset of the supply chain and produce an attestation that you can transitively trust to know that somebody else was actually doing a pre verification of say importing artifacts or checking elements within a sub team of the of your organization and situations like this. And I think that's that goes right toward what you were saying that like this needs to scale and you need to be able to provide a cryptographic paper trail of all of the operations in such a way that you know with some degree of assurance that the overarching element is being protected, but also you don't want to worry about the little details about who wrote the policy for the version control system or who wrote the policy for the security audit and so on so forth. And something I'm trying to understand though is if we see this is like it's an iterative process going to take a while to get all those steps done because what I'm looking at is like the Docker model when you build this base images are going to update. You're going to have external stuff you're pulling in, you know, first step most Docker files are saying apt get install Maven or something like that. As you go through all these different things you're pulling all these externalities and so it's a lot of effort to go through and get all those pulled in and hardened and controlled. And so I'm trying to understand how you get some of the value because you need those reproducible builds to be able to verify this stuff and I know nothing's changed. You know, until you make it all the way through all those steps and get every last piece hardened down, do we have the value of coming out of this with tough. That's just as we're coming to the last quarter of the hour and I see Marina stop presenting so much that there's more there or more relevant to where we're at the conversation where I see a sad is there's from what we know the hack was in the build system and somewhere upstream and there's a cutoff that it didn't apparently come from the source but again it could have come from somewhere else in the build system. Once it was built and shipped they were able to validate that this was valid and it came from the build system and that is the scope of what we've been doing in the NV2 work to certify that the thing or a test make the exact words to sign the artifacts in a registry that says this is the was came from the same source or multiple sources attested to it because we support multiple signatures and that part we seem to have a consistent model for being able to validate that the what I see really highlighted here is two is things that Brandon was kind of touching is in the build system this part of the build system that is just completely inherent to whatever the language that's being used included including part of the language is the build system itself right whether it's be an ACR task YAML or an AWS I forget what this code the script that's used there to define things in their build system or GCR build and so forth there is various build systems or get get get have actions but there's also a part of the the Docker files that also say from another image so the the other place that I could see in addition to the fact that the registry is shipping something that was put into it and the signatures were associated with it is the from statements could also implement a validation that the signatures are also valid so that if I'm expecting the Debian that it didn't wasn't minute you know hacked and I'm now getting something it's not signed by Debian now if Debian was also hacked then we have the same problem we have the SolarWinds but at least you have trace ability to know where that came from then of course we're back to the conversation of how fast can you figure out once you were hacked so you start tracing back right and I think sorry I just wanted to add that to run this question is a securities and ever moving target right and the more we can make informed decisions about what happened and we can better plug the holes the point that you were bringing Steve that I think that's a that's a great one and I think for example with in total you can actually use the total transport to fetch the reproducible build at the stations from the Rebuilders for Debian and you can say different people have built this Debian packages that I'm about to install and I can take all of these at stations and move them forward for auditing or save them somewhere so I can say this individual Docker image that was using Debian packages to build this particular element was built using these packages that have been independently built by all of these different people so I know that maybe the compromise didn't come from the packages it came from somewhere else I don't know the Docker file or like some coral coral pipe SH situation or something like this I think the more information we get the more we're able to understand the supply chain in a trustworthy fashion the better we are able to say this is what happened and I think that's the more like elucidating part of this solar winds attack that we we're we're all saying this we're able to walk back all the way into the into the build system and there were kind of like we have all of this like meta information about what happened we have the decisions we can do a lot of forensics about this and we can come up with a couple of working theories but we don't have the the trustworthy information to actually make a definitive decision about what exactly happened exactly just to just to shortly add to Santiago's point which is that what's the value of iterating over all the complicated rules even if you don't have reproducible bills you can at least pinpoint exactly where in the build process this this most likely failed instead of doing a long investigation where you don't really have all the clues so we said when we we want to understand what we would do differently in the ND2 work and what I think is what I'm hearing is part of it is for the area that we do control or do have influence to have a better statement is that in addition to the the poll which in the deployment like Kubernetes and OPA and other stuff that we've been looking at for when an image is deployed that we can validate a signature is good is that we should probably explore the various build container build tools that also can validate things that are referenced in a from statement or whatever the equivalent in other build container build tooling that we should incorporate that into it as well so it's all right at least we know that this thing was signed and there wasn't mitigation there but it does sound like there is definitely more work and I think this is part of what Trishank's original comment was when we posted on this back in December that this is part of the scenario that Tuffin and Toto teams have been focused on is the updates because it was an update that broke the system it just so happens the update was considered a valid update from the build system to the distribution but at least there is work in that supply chain in those multiple lines of defense that there should be some more focus on how this is done and I think you know we continue to have this conversation and it's not meant to be personal it's meant to like an analysis of how can we make it easier and I think it's important to understand that it is hard to implement I know it might only be five properties and I might be true but we still have not seen this taken up as much as we would like I think it's still I acknowledge it might be my stupidity but I have not been able to fully digest how I could implement it in an easy way whether I understand it or all of the developers I would like to see more of this effort happening that's why I have been excited about the SBOM work is that there's definitely holes in that this could have been a compiler this could very easily there was nothing changed in the source code was the statement there was nothing changed in packages references somehow maybe the compiler could inject a bunch of extra code it's certainly possible to see the various SBOM efforts kind of focusing on is where did the packages come from did the packages come from a place that it also has an SBOM and a TUF and a TOTO and all the stuff that Santiago has been working on and ultimately does the build when it's shipped represent what was intended and then from that point forward is it considered validly distributed because what would have been interesting is that the SolarWinds attack was out of the build system fine but between where it was built and it was distributed on individual machines you know was it compromised and if that was the case then you know absolutely that would fall right into the area that we're trying to secure from and we could have learned from that what I'm hearing here is this is really part of the shift to the left of the supply chain where an exploit was created something I'd like to see TUF might have a good role to play in a lot of this stuff is I'd like to see how we can implement that and so it gets into an implementation question of with a lot of the sign work we've been doing a lot of the stuff you've been doing Steve of let's get the signature pushed up to the registry server how to integrate some of that logic with what TUF is doing into that model I don't see that yet and so I think it'll be helpful to me to understand how some of that can be integrated in with what TUF is doing on their side yeah I think we can definitely do some more tangible stuff in that space I think a lot of the work that we've done so far is figuring out where different types of metadata fit into the the registry model and I think that TUF metadata should pretty easily fit into that so I think that it's definitely if we can figure out maybe a nice demo for how you can put TUF metadata onto a registry download that, verify that and as far as this what we can do for something like SolarWinds I think that the most important thing is just to support whatever earlier supply chain solutions people want to to use and I think that's like the main point about Nintodo here is that I think that this is a simple open source solution that we should at least support so that people who want to use it can verify down to the source code and through the deployment and even if we focus mostly on that last step, the deployment, whatever it'd be nice to include options to include other previous applications or whatever Yeah, that's fair and thanks Steve for raising that point which is that you're absolutely right we should definitely, and that's partly why we're here which is that we're trying to also improve the usability for Tuffin and Nintodo right especially when developers point of view is like I have no idea what's going on just tell me what to do at least nudge them to what's doing the right thing so I definitely agree so if they're like standard layouts we can provide developers here don't think about it, if your build process looks like this just use this file and fill in the blanks with the keys and that's it definitely agree and the second thing is that I also agree with Brandon that we I think Marina's very close she actually might have a working demo too, the very least we could do is show how Tuffin and Nintodo could have you know prevented SolarWinds attack but for containers but we need your help in figuring out where to stick all of this metadata on the OCI registry level that's something that we're not here yet so maybe if we show you the demo you can tell us oh okay I see how this works now and here's where we can stick in all this metadata yeah I think that obviously the registry has lots of place to put stuff that's the point around the artifacts approaches that it is so flexible the question that I think we're struggling with is how would that have helped because we want to see where is the gap and what do we need to fill that gap and identify something that actually does well yeah I'm not sure if that's well I mean I think that's one of the things we're talking about today and I mean but there will be potentially I mean I think we all agree there'll be potentially some attacks we can't do anything about but I think the aim is to at least raise the bar given that the default right now is you can't prevent anything at all but I think you know there's definitely issues I think the reproducibility versus compromise-build-host issue is just like there isn't a good solution because the compilation process on a totally compromise machine or I mean it's a lot of the cases of a compromise-build machine to be equivalent to key compromise for that stage anyway so I mean the things that are important about key compromise are about how you recover and detection and those kinds of you know what you do about that because you know I think we're going to have to understand that there will be cases where things won't be prevented but there'll be there's still you know basic a huge number of cases where they will be prevented so one of the things that I kind of rifted a little bit ahead of the reproducible builds is we're pulling this other piece we're seeing registries you know multiply there are going to be multiple public registries we started with Docker Hub and that just brought awesomeness to the community because we knew where to get content and not have to build VMs from scratch we forgot where to get the base part of the VM so that's awesome and you know just the long list of you know between github and now aws and so forth the same image could actually it's going to be able to get from multiple locations it'll be able to get Debbie in from six different locations the thing that was kind of interesting in the reproducible builds so we start the most obvious if the same image is copied to multiple registries then that's fairly easy to validate it'll be the same digest it'll have the same signature all of that will still be valid so that that part's good the interesting one to me about this one was the reproducible builds and like I said in Microsoft I'm sure other places we rebuild the Kubernetes images all the open source code that we use as images we are moving to rebuilding all if not most of it for make sure we can service our customers but we've also seen the downside of that is we see the security vendors come back to us and say hey we have a different image than what we're getting from GCR and it actually has the same vulnerabilities in it but it's a different digest so that itself is throwing the flag what would be interesting is if we were able to compare two builds of Debian from two different environments like the one that came from Microsoft and the one that comes from GCR I'm just going to use for example I said Debian versus Kubernetes but it doesn't matter even though they're built separately is there something about the conversations we're having here whether it's S-Bahn whether it's in Toto or Tuff or something is there some introspection that can be looked at it says yes these actually were built separately but they actually do represent the same content so we're good the security scanners will give a thumbs up to both they have the same set of vulnerabilities that is known and improved versus one of the two and I won't pick which one but it's a fact and its build system got hacked like SolarWinds and that company's distribution of that same content is considered vulnerable and it's identified it's not as identified as vulnerable but identified as different and the fact that it's different throws the flag and now somebody could a human might start looking into it there's a bunch of issues a lot of that in the OCI of E2 docs the formats were not designed for better reproducibility and so there's a bunch of problems there I think those things are kind of understood at this point what the problems are but there's a lot of work to do to actually make actual reproducibility of container images easy but that's stuff that's important to do as well so I think just really quickly one thing I want to throw in here these are all great problems to go solve are those relevant problems for this working group to look at right now because the way I look at it validating the steps in the supply chain they're important but they can be done on top of the initial distribution signature validation that we're looking at I don't see that the work we're doing here prevents any of this additional verification for anyone that wants to implement it and without an agreed upon signature format we have a higher issue in software distribution which has much larger threat models that we're already aware of so I think is that something that we should have a separate working group look at as a separate project or is that something that needs to be part of the code signing initial working group we have because I think they're separate distinct problems one with the longer term trajectory where we have a lot of other things to understand and how information is stored within the registries themselves I definitely agree to a certain extent they're different problems I think that the most important thing we can do is make sure that what we come up with here has a space it can just be extra space in the metadata or whatever that can allow whatever solutions for the downstream and yeah maybe it's a good idea to also have a working group talk more about the specific build system steps that we can include in that space but I think as far as what we're doing here I think the most important thing is whatever metadata formats we come up with as part of the OCI SPAC or whatever that they can they have the space for this kind of information I think that's the in my opinion the biggest thing and Steve I think you're muted yeah I'm trying to use the space wherever not the right focus anyway I think we have something we wanted to do to Mia's point we wanted to have something moving forward because with all of these you're always looking for something to learn like I was hoping there'd be something about this that says hey what we are doing we should do this one step better because then we could have you know demonstrate how we could mitigate this to me I'm always looking to learn what we can do better fortunately or unfortunately it sounds like the path we're having at least gives us the same traceability that solar wind test they knew it went back to the build system because it was signed today images are not signed across registries and even within registries they're not really signed that well so we'll at least be able to meet that traceability which is actually huge and we've also said that we wanted that that was a phase one and phase two like once we can understand how better we can use a tough and in total kind of implementation that spans across multiple registries supports moving across registered sports private registries all the things that I think were a shift on the original scoped work that was done with no review one no review one was done there was one when the assumption was one I don't want to get into much into everybody's participation at the time but now we've had this explosion this content movement it's changed and we haven't figured out how to make it work in that private and public multiple registry scope if there is a way that tough and in total can be used in that great and we could come back and revisit that in the phase two portion of it what they were doing in phase one also is flexible enough that it doesn't prohibit an additional signature model being put in it's something I think we've taken extra time if anything it's been a little slower because we're making sure that it'll work across all artifacts and has a lot more flexibility I just want to interject really quickly that I think we've demonstrated how tough can work in the multiple repository multiple registry situation a couple of times and so I get that you know this is a process and they're different stages to that and I just think that that's if there are concerns with the way that we've implemented that I think that's a good thing to discuss but I just know that we've gone over that I think we've discussed that I mean the concerns I have with tough which are unrelated to in total one is that you know we're making an assumption that the latest release is the only valid release right I think in practice we've seen that that's problematic and that's something where there's a difference between what is stale code and insecure code that's one of the things that we haven't seen an answer to and also the other part of that like you know as we want to move from registry to repository how can we do that without having to re-sign the registry specific metadata and have that burden so if those are two questions that we can get answers to I think that makes for what we've seen container users ask for but without those addressing those two I think it's very difficult to kind of push the way tough is currently architected as a standard solution and throw a third one on that we can talk sorry I just want to throw a quick third one on top of that which is that the disconnected environments are looking for a solution as well and so we have a lot of requirements and tough for regular time stamps to be re-signed on this stuff that kind of doesn't perform well when you have disconnected environment that can't refresh those time stamps yeah I think Marina there was a number of problems we've been identifying and you've been making great progress and we appreciate all the work you've been doing the one of the things was the hashing of data that from across all repos in a registry that's not something we can do as registries we can't give any kind of access to anything across multiple customers even multiple teams in the same company so that was one of the scoping things I know there was a performance question but it was really also around a security access issue so I think there's a there's been a number of challenges that we haven't figured out how to solve but I also we also seem to keep on starting with this is the solution how do we apply to the problem as opposed to we've identified the problems that we have as we're asking should we specifically address look at trying to address these and discuss these issues next week as we're out of time I think that certainly Marina would you like would you perhaps discuss NERS's issues in particular yeah I think I'd be happy to I don't want to hold people over too much over time but I'd be happy to address those either in a separate meeting we put together for that purpose or the next week's meeting here yeah it's probably good for the next week's amazing I would have thought that we could start it on Slack oh yeah I can do that too yeah could you maybe put now Slack's back up and maybe a quick summary of of your issues and maybe if a couple of the ones that other people had to maybe we can kind of start to concretely make sure everyone understands what the issues are and why people think there's a problem and what kind of solutions we're looking for for these yeah I think that that's a great call out I'll either throw it in Slack or start a doc in LinkedIn Slack cool so next week thanks everyone let's start 2021 in a new way so we'll see everybody next week and yeah by all means Slack intermittently comes up let's keep the conversation going thanks folks happy New Year's everyone alright