 Welcome to this CUBE conversation. I'm Dave Nicholson and we're having this conversation in advance of KubeCon, CloudNativeCon North America 2021. We're going to be talking specifically about a subject near and dear to my heart and that is security. We have a very special guest from Red Hat, the security lead from the office of the CTO, new kinds, welcome, welcome to the CUBE, Luke. Oh, it's great to be here. Thank you, David. Really looking forward to this conversation. So you have a session at kubecon slash cloudnativecon this year and frankly, I look at the title and based on everything that's going on in the world today, I'm going to accuse you of clickbait because the title of your session is a secure supply chain vision. Sure. Well, what other than supply chain is in the news today, all of these things going on. So you're talking about the software supply chain. All right, you tell us about this vision, where it came from, fill us in. Yes, very much. So I do agree, it is a bit of a buzzword at the moment and there is a lot of attention. It is the hot topic, secure supply chains, thanks to things such as the executive order and we're starting to see an increase in attacks as well. So there's a recent statistic came out that was 620%, I believe, increase since last year of supply chain attacks involving the open source ecosystem. So things are certainly ramping up and so there is a bit of clickbait, you've got me there and so supply chains. So predominantly, let's consider what is a supply chain? Okay, and we'll do this within the context of cloudnative technology, okay? Because there's many supply chains, many different software supply chains, but if we look at a cloudnative one predominantly, it's a mix of people and machines. Okay, so you'll have your developers, they will then write code, they will change code and they'll typically use a code revision control system like Git, okay? So they'll make their changes, they'll then push those changes up to some sort of repository, typically a GitHub or a Git lab or something like that. Then another human will then engage and they will review the code. So somebody that's perhaps a maintainer will look at the code and they'll improve that a code. And then at the same time, the machines start to get involved. So you have your build servers that run tests and integration tests and they check the code is linted correctly, okay? And then you have this sort of chain of events that start to happen, these machines, these various actors that start to play their part in the chain, okay? So your build system might generate a container image. There's a very common thing within a cloudnative supply chain, okay? And then that container image is typically deployed to production or it's hosted on a registry, a container registry. And then somebody else might utilize that container image because it has software that you've packaged within that container, okay? And then this sort of prolific expansion of use occurs where people start to rely on other software projects for their own dependencies within their code, okay? And you've got this kind of a big spaghetti of actors that are dependent on each other and feeding from each other, okay? And then eventually that is deployed into production, okay? So these machines are a lot of them run open source code, okay? Even if there is a commercial vendor that manages that as a service, it's all based on predominantly open source code, okay? And the security aspect with the supply chain is there's many junctures where you can exploit that supply chain. So you can exploit the human or you could be a netfarious human in the first place. You could steal somebody's identity, okay? And then there's the build systems themselves where they generate these artifacts and they run jobs, okay? And then there are the production system which pulls these down, okay? And then there's the element of which we touched upon around libraries and dependencies. So if you look at a lot of projects, they will have approximately around a hundred, perhaps 500 dependencies that they all pull in from, okay? So then you have the supply chains within each one of those. They've got their own set of humans and machines. And so it's a very large spaghetti beast of sort of dependence and actors and various themes and identities that make up the supply chain. Yeah, you're describing a nightmarish scenario here. So I definitely appreciate the setup there. It's a chain of custody nightmare. Very much, yes, yes, yeah. But it's also a wonderful thing because it's allowed us to develop in the paradigms that we have now very fast. You can prototype and design and build and ship very fast thanks to these tools. So they're wonderful. It's not to say that there is a gift there, but security has arguably been left as a bit of an afterthought essentially, okay? So security is always trying to, it's at the back of the race, it's always trying to catch up if you see what I mean, so. Well, so is there a specific reason why this is particularly timely in when we talk about deployment of cloud native applications, something like 75% of what we think of as IT is still on-premises, but definitely moving in the direction of what we loosely call cloud is, why is this particularly timely? I think really because of the rampant adoption that we're seeing. So I mean, as you rightly say, a lot of IT companies are still running on a sort of more of a legacy model, okay? Where deployments are more monolithic and static. So I mean, we've both been around for a while when we started, somebody would rack a server, they plug a network cable in, you'd spend a week deploying the app, getting it to run, and then you walk away and leave it to a degree, whereas now obviously that's really been turned on its head, so there is an element of not everybody has adopted this new paradigm that we have a development, but it is increasing, there is rapid adoption here and many that aren't, many that rather haven't made that change yet to migrate to a sort of a cloud type infrastructure, they certainly intend to or they certainly wish to. I mean, there's challenges there in itself, but I would say it's a safe bet to say that the prolific use of cloud technologies is certainly increasing as we're seeing all the time. So that also means the attack vectors are increasing as we're starting to see different verticals come into this landscape that we have. So it's not just your kind of sort of web developer that are running some sort of web 2.0 site, we have telcos that are starting to utilize cloud technology with virtual network functions, we have health banking, fintech, all of these sort of large verticals are starting to come into cloud and to utilize the cloud infrastructure model that can save them money and it can make them, can make their development more agile and there's many benefits. So I guess that's the main thing is really there's a convergence of industries coming into this space which is starting to increase the security risks as well because I mean, the security risks to a telco are a very different group to somebody that's developing a web platform for example. Yeah, now you mentioned the sort of obvious perspective from the open source perspective which is that a lot of this code is open source code. And then I assume that it makes a lot of sense for the open source community to attack this problem because you're talking about so many things in that chain of custody that you described where one individual private enterprise is not likely to be able to come up with something that handles all of it. So what's your vision for how we address this issue? I know I've seen in some of the content that you've produced an illusion to this idea that it's very similar to the concept of secure HTTP and so imagine a world where HTTP is not secure at any time. It's something we can't imagine, yet we're living in this parallel world where code which is one of the four C's in cloud security isn't secure, so what do we do about that? And as you share that with us, I wanna dive in as much as we can on SigStore, explain exactly what that is and how you came up with this. Yes, so the HTTP story is incredibly apt for where we are. So around the open source ecosystem, we are at the HTTP stage. So a majority of code is pulled in untrusted. I'm not talking about so much here, somebody like Red Hat or a large sort of distributor that has their own sign-in infrastructure but more sort of in the wider open source ecosystem. Amount of code that's pulled in untested is the majority, so it is like going to a website which is HTTP and we sort of use this as a vision related to SigStore and other projects that are operating in this space where what happened effectively was it was very common for sites to run on HTTP. So even the likes of Amazon and some of the e-commerce giants, they used to run on HTTP, okay? And obviously they were some of the first to deploy TLS and to utilize TLS, but many sites got left behind, okay? Because it was cumbersome to get a TLS certificate. I remember doing this myself, you would have to sort of, you'd have to generate some keys, a certificate sign-in request, you'd have to work out how to run open SSL, okay? You would then go to a commercial entity, you probably have to scan your passport and send it to them and there would be this kind of back and forth and you have to learn how to configure it on your machine and it was cumbersome, okay? So a majority just didn't bother, they just, you know, they continued to run their websites unprotected. What effectively happened was let's encrypt came along, okay? And they disrupted that whole paradigm, okay? Where they made it free and easy to generate, procure and set up TLS certificates. So what happened then was there was a very large change, the kind of the zeitgeist changed around TLS and the expectations of TLS. So it became common that most sites would run HTTPS. So that allowed the browsers to sort of ring fence effectively and start to have controls where if you're not running HTTPS as it stands to, as it is today, it's kind of socially unacceptable to run a site on HTTPS. It's a bit kind of, if you go to a HTTPS site, it feels a bit, yeah, you know, it's kind of, am I gonna catch a virus here? It's kind of, it's not accepted anymore, you know? And it needed that disruptor to make that happen. So we want to kind of replicate that sort of change in movement and perception around software signing where a lot of software and code is not signed. And the reason it's not signed is because the tools, it's the same story again, they're incredibly cumbersome to use and the adoption is very poor as well. So SIGStore specifically, where did this come from and what's your vision for the future with SIGStore? Sure, so SIGStore, SIGStore's a lockdown project, okay? It started last year, July 2020 approximately. And a few people had been looking at secure supply chain, okay, around that time we really started to look at it. So there's various people looking at this. So I'd speak to people, various people at Purdue University and Google and other sort of people trying to address this space. And I'd had this idea kicking around for quite a while about a transparency log, okay? Now transparency logs are actually, we're going back to HTTPS again, they're heavily utilized there, okay? So when somebody signs a HTTPS certificate as a root CA, that's captured in this thing called a transparency log, okay? And a transparency log is effectively what we call an immutable tamper proof ledger, okay? So it's kind of like a blockchain, but it's different, okay? And I had this idea of what if we could leverage this technology, okay? For secure supply chain so that we could capture the provenance of code and artifacts and containers, all of these actions, these actors that I described at the beginning in the supply chain, could we utilize that to provide a tamper resistant publicly audible record of the supply chain, okay? So I worked on a prototype over a week or two and got something basic happening and it was a kind of a typical open source story there. So I wouldn't feel right to take all of the glory here. It was a bit like kind of you look at Linux when he created Linux itself, Linux Torvils. He had an idea and he shared it out and then others started to jump in and collaborate. So it's a similar thing. I shared it with an engineer from Google's open source security team called Dan Lawrence, somebody that I know have been prolific in this space as well. And he said, I'd love to contribute to this. So can I work on this? And I was like, yeah, sure, the more the better. And then there was also Santiago, a professor from Purdue University took an interest. So a small group of people started to work on this technology. So we built this project that's called RECOR and that was effectively the transparency log. So we started to approach projects to see if they would like to utilize this technology, okay? And then we realized there was another problem, okay? Which was we now have a storage for signed artifacts, okay? Like a signed record, a provenance record, but nobody's signing anything. So how are we gonna get people to sign things so that we can then leverage this transparency log to fulfill its purpose of providing a public record? So then we had to look at the signing tools, okay? So that's where we came up with this really sort of clever technology where we managed to create something called ephemeral keys, okay? So we're talking about a cryptographic key pair here, okay? And what we could do, we found was that we could utilize other technologies so that somebody wouldn't have to manage the private key and they could generate keys almost point and click. So it's an incredibly simple user experience. So then we realized, okay, now we've got an approach for getting people to sign things and we've also got this immutable publicly auditable record of people signing code and containers and artifacts. And that was the birth of SIGSTOR then. So SIGSTOR was created as this umbrella project of all of these different tools that were catering towards adoption of signing and then being able to provide guarantees and protections by having this transparency log, this sort of blockchain type technology. So that was where we really sort of hit the killer application there and things started to really lift off and the adoption started to really gather steam then. So where are we now and where does this go into the future? One of the wonderful things about the open source community is there's a sense of freedom in the creativity of coming up with a vision and then collaborating with others. Eventually you run headlong into expectations. So Luke, is this going to be available for purchase in Q1? What's the story? Yeah, I will fill you in there. Okay, so with SIGSTOR, there's several different models that are at play. Okay, I'll give you the two predominant ones. So one we plan to run a public service. Okay, so this will be under the Linux Foundation and it'll be very similar to Let's Encrypt. So you as a developer, if you want to sign your container and you want to use SIGSTOR tooling, that will be available to you. It'll be nonprofit, free to use. There's no special tiers for anybody. It's there for everybody to use. And that's to get everybody doing the right thing and signing things. The other model for SIGSTOR is this can be run behind a firewall as well. So an enterprise can stand up their own SIGSTOR infrastructure. So the transparency log, our code signing certificate system, client tools and then they can sign their own artifacts and secure better materials and all of these sort of things and have their own tamper proof record of everything that's happened. So they have anything untoward happens such as a key compromise or somebody's identity stolen, then you've got a credible source of truth because you've got that immutable record then. So we're seeing adoption around both models. We've seen a lot of open source projects starting to utilize SIGSTOR. So predominantly, Kubernetes is a key one to mention here. They are now using SIGSTOR to sign and verify their release images, okay? And there's many other open source projects that are looking to leverage this as well, okay? And then at the same time, various people are starting to consider SIGSTOR as being sort of an enterprise signing solution. So within Red Hat, our expectations are that we're gonna leverage this in OpenShift. So OpenShift customers who wish to sign their images, okay, and they want to sign their configs that they're using to deploy within Kubernetes and OpenShift rather, they can start to leverage this technology as OpenShift customers. So we're looking to help the open source ecosystem here and also dog food this and make it available and useful to our own customers at Red Hat. Fantastic. I notice the Red Hat in the background. Just a little historical note, Red Hat has been there from the beginning of cloud. Before cloud was cloud, before there was anything credible from an enterprise perspective in cloud. I remember in the early 2000s doing work with pre-AWS and there was a team of Red Hat folks who would work through the night to do kernel level changes for the Linux that was being used at the time. And so a lot of what you and your collaborators do often falls into the category of toiling and obscurity to a certain degree. We hope to shine light on the amazing work that you're doing and I for one, appreciate it. I've suffered things like identity theft and we've all had brushes with experiences where compromise and security is not a good thing. So this has been a very interesting conversation. And again, thanks for the work that you do. Do you have any other final thoughts or points that we didn't cover on this subject that come to mind? Well, there is something that you touched upon that I'd like to illustrate. You mentioned that identity theft and these things. Well, the supply chain, this is critical infrastructure. So I like to think of this as they're serving, they're solving technical challenges and the kind of that aspect of software development. But with the supply chain, we rely on these systems when we wake up each morning. We rely on them to stay in touch with our loved ones. Our emergency services, our military, our police force, they rely on these supply chains. So I sort of see this as there's a bigger vision here really in protecting the supply chain. It's for the good of our society because a supply chain attack can go very much to the heart of our society. It can be an attack against our democracies. So I see this as being something that there's a humanistic aspect to this as well. So that really gets me fired up to work on this technology. Absolutely, it's really important that we always keep that perspective. This isn't just about folks who will be attending KubeCon and CloudCon. This is really something that's relevant to all of us. So with that fantastic conversation, Luke, it's been a pleasure to meet you, pleasure to talk to you. You too, David. I look forward to hanging out in person at some point. I love that, yeah. Whenever that gets up. So with that, we will sign off from this Kube conversation in anticipation of CloudCon, KubeCon 2021 North America. I'm Dave Nicholson. Thanks for joining us.