 Hello, hello, and welcome to the first of many secure Cloudcasts, thanks to the Cloud Multiplier Group for the handoff. Seemed like a really cool demo I was catching, I'm sorry, to cut you all off so early. But yes, the secure Cloudcast, the monthly show for all things cloud, community security, open source, maybe a little Stack Rocks ACS mixed in there. So we're looking just to give you updates on all the security news in the world. I'm one of your hosts, Mike Foster, and joining me today is? Hi, I'm Kim Carlos, product and marketer extraordinaire here at Red Hat. We're really happy to have you here on our first show, and today we're gonna be discussing the hottest topic. Well, maybe not hottest, but pretty hot. Today in security, which is actually the secure supply chain. So we're gonna talk more about that and the risk that it's posing to businesses and kind of what they're doing about it or best practices. For sure, yeah, awesome. And just for the heads up for everybody who's just joining us, I'm Mike Foster, I've been in Kubernetes for around five years, Kubernetes security for a little over two now, I came over to Red Hat during the Stack Rocks acquisition, and I'm helping out with the ACS product in various amounts of ways, including the open source project. So if you are new, definitely check out Stack Rocks. Kim, you wanna tell us a little bit about yourself as well? Sure, I apologize for the thunder that you will likely be hearing because it is very loud, but my name is Kim Carlos. I just recently came to Red Hat this year to focus on OpenShift security use cases and capabilities as well as ACS. So we'll be working a lot with that. My background is actually in various areas of security. So everything from log monitoring to vulnerability and risk management or cyber risk management, I've done even threat intelligence. So a lot of different areas I like to diversify. So this is my newest endeavor and I'm very excited to be here and talk with you guys today. Sweet. Before we bring in Kirsten Newcomer, who is gonna talk everything, supply chain security as our, once we can say formal expert, I think, at Red Hat, we're just gonna give you a couple of general news updates. Kim, you wanna kick it off with some ACS news? Yeah, sure. So we just recently launched ACS 3.71, and so we'll drop the release notes in the chat so you can look at it. But there are some key takeaways that I would say there is we have a really new, really slick dashboard that'll help you increase efficiency. And then we also have two default policies to help you improve your security posture as a whole. So be sure to look into that a little bit more. We will have more information coming out on that as we have our next release, and there will be a blog post about that we'll share on our next show. Yeah, and for those who are new to the ACS acronym, that's Red Hat Advanced Cluster Security, that is Stack Rocks. So Stack Rocks was acquired in early 2021 at the, I think it was like January 7th or something like that, but it was bright and early in the new year. And so that turned into Red Hat Advanced Cluster Security and luckily we were able to keep the Stack Rocks name alive and that's open source Stack Rocks. So if you want to test that out for free, definitely come and join us. You can go to stackrocks.io for more information. And again, we have that new brand new dashboard. I think it's awesome and super sleek. I'd love to hear from you and your thoughts if you get to deploying it. From the open source Kubernetes world, KubeCon is fast approaching. That's KubeCon North America in Detroit, I believe it is in the last couple of weeks. I forget the exact dates, 20th, I believe to the 25th or 21st to the 26th, six store con. So one of the new events that's popped up, six store con is looking for CFPs. So if you are in the supply chain security ecosystem and you will look to submit a talk, I will drop the CFP in the chat for you and you can check it out. Always nice to see some of these day zero events as well and see some familiar faces. So that'd be worth checking out. And any last updates, Kim, before we bring Kirsten on for our discussion? Nope, that's all for now. The only thing I wanted to mention actually is that there is a blog posted for 3.69 and 3.70 if you want to read more about the different areas that we've touched on in the release and how it helps you align with your business goals that is available on our blog. Thank you, and I'm getting corrected actually. Kirsten, the date expert and supply chain security expert, KubeCon October 24th to 28th in Detroit. So I look forward to seeing some familiar faces there. Without further ado, let's bring on the Red Hat Supply Chain Security Expert. Kirsten. Hey everybody. One of Red Hat Supply Chain Security experts. Very true. Yes. Kirsten, for those watching, do you want to issues yourself? Tell us what you've been working on at Red Hat for? Sure. So Red Hat for about six years now, six and a half. So my role at the moment is I lead the product security product management team for all things security across OpenShift and OpenStack. That includes Red Hat Advanced Cluster Security. So and ACS like ACM supports not just OpenShift, but EKS, GKE and AKS as well. So multi-cluster security, pane of glass across all those solutions. So I've been working with Kubernetes and security for a while now. Prior to coming to Red Hat, I was at Black Duck Software before they were acquired by Synopsys. And I've been doing product management for more years than I usually care to admit at this point. Awesome. I'm just gonna kick off the first question because I joined Stack Rocks in 2019, 2020. And I remember getting on these CUBE security calls for the CNCF and seeing Luke Hines drop in and start talking about Six Store. And I remember he gave his first demo, it was 45 minutes and people were kind of scratching their heads like, what's the vision here? What's going on? And the last two years that the supply chain conversation has really just blown up. Partially maybe government, partially environment related. But with all those ripples, we were just hoping you could give us your thoughts on what does supply chain security encompass? Are there any frameworks? I know it's a big open-ended question, but just wanted to get your thoughts. Definitely work, Justin. But you're absolutely right. There is a ton going on around supply chain security these days, both from frameworks, from the frameworks angle, from the tooling angle. As you say, Luke and Red Hat have been thinking about how can we make it easier for our customers to do certain elements of supply chain security for a while? That's why we invested in Six Store. We'll maybe talk as we go a little bit more about Six Store's role in that. But really things really got hot after the SolarWinds attack, which would have been, gosh, in 2019, I think. And so it used to be that when we talk about supply chain security, people were thinking about, how do I analyze the security of the artifacts that I'm producing through my supply chain, right? Do I do, when do I do vulnerability analysis? How early? What kind of static security code analysis do I wanna be doing and how frequently? And now that, once the SolarWinds attack happened, that really shifted. It's not that we stopped caring about those things, but the security of the supply chain itself has become a key focus, right? The SolarWinds hack involved somebody actually being able to hack delivery of a patch from SolarWinds. And one of the things that I saw in maybe in 2021 was a really great talk by the folks from SolarWinds at what they have done to improve the security of their supply chain. And they are leveraging tools like Six Store, like Tecton Chains, which we'll maybe talk about a little bit more. In terms of frameworks, lots of things happening in that space as well. So there's been an executive order in the States, that's executive order 14028. That follows a previous executive order on supply chains. There's been a new Salsa.dev project that's driving standardization of how do you, what are the key things you need to do to secure that supply chain itself? There are things coming from NIST, NIST 8218, NIST also 800-161. And the CNCF released a white paper on software supply chain best practices. So again, lots and lots happening and we can dive into the details a little bit more as we go. Or I know you're also planning to have Emmy ID join in a future session. Emmy is one of Red Hat's representatives to the OpenSSF, which is helping to drive the Salsa. A Salsa stands for, I'm sorry, I'm gonna have to go look. I'm just so used to calling it Salsa, right? So it stands for, man, they don't even spell it out on their supply chain levels for software artifacts. There we go. It's making me hungry for Mexican. It smells like salsa. It is lunchtime. Yeah, in terms of frameworks, because we kind of got there, Salsa was originally brought up by Google, am I correct in that? I honestly don't know who started that, yeah. But it seems to be adapted because supply chain just encompasses so much that Salsa really broke it down into, doesn't really matter what level you're on, you're gonna have these specific components at every stage, right? You're gonna have your dependencies, your code that you're adding, you're gonna have to build it at a specific time. Salsa really adds that framework, correct? Yeah, I think what's really nice about what Salsa has done is that they've really, kind of the elements of software supply chains, chains are multiple and they're very depending on the kind of code that you're building. So, but there is some commonality across those, right? You generally need a source code control system. You need a build tool. You need something that is going to store your build artifacts. That might be a binary repository. That might be a container registry. You need, these days more and more organizations are using automation to do their deployment as well. So, you need your security testing. You need tools that do your integration testing. It's just like this whole universe of things to think about. And what's nice about Salsa is that they really have broken down all of, they kind of listed all those elements and added categories of security for each of them. And they've gotten pretty granular, right? So, there are four categories for source code. There's gosh, one, two, three, four, five, six, seven, eight categories for builds. Providence has become a big thing. Providence is something that, honestly when I worked at Black Duck, Providence was a big deal for open source license compliance, but it's a subset of our industry that focuses on open source license compliance. And so, kind of, when you look at the Salsa framework, you have this great set of vertical checks to think about and then horizontally maturity levels so that they get kind of a roadmap for how to get started in addressing, securing your supply chain itself. Do you think that's probably a good starting point for large organizations that are trying to fortify, if you will, their supply chains? It's, I think it gives you a good view of where you need to get to. I imagine when folks look at Salsa level four, they're gonna be a little concerned. There's a lot more going on. But the nice thing is you can start with Salsa level four, although it's not the only thing to think about. There are security principles that I know are applied in Salsa that also we kind of take for granted in our production environments that you can apply to your supply chain security. So one of the first things you might wanna do is start auditing the activity in your supply chain. Most of the tools that we're using, maybe not all of them, that many of them produce log files and collecting those log files for forensics is a place to start. So auditing that activity. One of the things that Salsa does emphasize in the build space is ensuring that you have a scripted build. And that's kind of the next place I was gonna go. You really want automation as much as possible. Do as much automation as you can in your supply chain. Automate is much easier also to do automated auditing of automation. If you're gonna probably still have some manual steps when you're just getting started, you wanna think about how you're gonna collect audit data on those manual steps as well. So those are kind of some of the places to start. I think provenance is another interesting area to talk about, but let me see where else you two wanted to go before we do that. I think what I was curious about is we talk a lot about implementing these best practices and they're a lot of times evolving with our industry because it moves so quickly. But when you have a large organization, what do you think are some, I guess, ways to have the least amount of disruption to the business when you're trying to implement a good supply chain security policy? Yeah, so I mean, I think that's why when you look at Salsa level one where some of the things I just talked about, auditing, collecting log files, that minimizes the impact. However, getting to something like Salsa level four, and by the way, provenance is in Salsa level one and that's something that not everyone is used to doing. Think of provenance as being tied to the concept of software bill of materials. Where did my code come from? What's in my code? A lot of folks haven't been collecting that data. The good news about provenance, there are a lot of tools out there that can help you build software bill of materials. Thinking about S-Falms has been going on for many years and standards in that space such as SPDX started being developed as early as 2010 if not earlier. So there's a lot available. But the reality is that kind of as the world, as this evolves and as you start to increase your maturity level for supply chain security, you probably need to also start looking at your tools, right? Do you want to evaluate are those tools that you've been using Lord knows sometimes for 10, 20 years? Are those the right tools to meet these new sets of requirements? And most of us in this industry are going to have to address the requirements that are in the executive order and things that are feeding some of these new standards. If you sell to the government, work with the U.S. government, you have to address these. A software bill of materials is part of that. So you may need to add some new tools. You may need to upgrade your tool set. And this is one of the reasons if we circle back to Red Hat's investment in Sigstore, the concept of provenance is key, but as you move up that maturity level, another key concept is attestation. So I think organizations are used to expecting from companies like Red Hat that the code we produce and ship for consumption by our end users is signed. So I can attest to the validity of what I pulled down from Red Hat by verifying the signature that Red Hat provides for the content we ship. But adding signatures for custom code that organizations are producing through their own supply chain is not so simple, right? It's been easy to kind of add that signature at the end when you've got that final set of code that you're shipping out the door. But the goal of Sigstore has been make it easier to add signatures and more than one signature to content as it moves through the supply chain. And traditional signing tools just weren't that well developed for or designed for working and being integrated into a CI CD pipeline. So Sigstore really has some great benefits in that space, but it's gonna take time for people to integrate these new things into their existing supply chains or CI CD pipelines. And I think the integration and the automation is the key point there, right? Because you don't want the developer, you don't want the operator going and signing all these things you want it so that whenever something gets accepted into the main branch, it goes through that self, that attestation process and that if we're ever expanding or making any changes, we can verify that the original build is correct, right? Absolutely, and one of the additional things with Sigstore, right? And they're elements of Sigstore that you can adopt independently of each other, but RECORE, the signature transparency log is key for what you just described, Mike, right? So that I can have a series of signatures and I can trace those signatures, even if it's for the same artifact. Say somebody downloads something from Red Hat, they verify the Red Hat signature, but now they wanna add their signature to say, this is approved for production use in my organization. And that something like RECORE gives you a transparency log where you can track that chain of signatures. This is one of the reasons we saw the Kubernetes community adopt Sigstore for signing the artifacts coming out of the different CNCF Kubernetes projects. So really great to see that kind of adoption. And I think that will help other organizations look at how could we do that, right? If there's a model in the open source community that they can follow. I've adopted things like that, like with Sigstore. I'm curious as sort of that being that best practice. I think we talk a lot about DevOps and sort of this movement to DevSecOps, right? And involving that idea of signing things and verifying things and zero trust networking and all of these different things you wanna be able to apply. And I'm curious, how do you see supply chain security as it moves towards that maturity model you were discussing of DevSecOps? Well, I think one of the challenges with the DevSecOps phrase, honestly, is that we don't have a common understanding in our industry of what is meant by DevSecOps, right? If you talk to the folks who really initiated the DevOps movement in particular, they'll say security checks were always intended to be part of the DevOps process. But I also think that at that point in time and even right now, I think people think of DevSecOps as looking at the security of the code that is moving through the pipeline. That's the DevSec piece, right? Shifting security left. And the SecOps pieces, what are my security solutions and operations? And ideally, if you're really thinking this through how do you feed security information from ops back to the Dev team so they can improve their security? How do you feed security information discovered in the Dev process into ops to inform your security policies at deploy time and at runtime? And that closed loop is a big challenge. But that's only one piece of supply chain security and that's the piece that has been less, that's kind of, well, the closed loop piece, I would say that we're still as an industry working on closing that loop. Solutions like Stackrocks help with that. But again, kind of the newer emphasis is on securing the tool chain itself. And so in that case, actually, let me separate those two things. So six door helps with that first DevSecOps flavor, right? I can sign content as it moves through the pipeline. I can check those signatures upon admission to a cluster. Those are key ways that six door plays a role there. When it comes to my tool chain, now I wanna look at attestation of the steps in my tool chain. And this is where a combination, if you're doing Kubernetes native tool chains or pipelines using something like Tekton with Tekton chains, which has an integration with Sigstore, for signing my pipeline tasks, that's kind of the next step up, right? So talking about attestation, not just in my content, but of the tasks themselves. And in fact, this is something that the SolarWinds organization has done. Awesome, yeah. Sorry, a lot. Speaking of tech. Yeah, we got a lot. I was actually hoping that you could break down sort of the Sigstore project. It's different components, hopefully, and because I know there's a couple of moving parts, I sometimes get confused with all of the different projects. So you mind sort of just outlining the use case and how each tool is used? Yeah, sure. So again, this is all about making it easy to sign artifacts, different types of artifacts in a CI CD pipeline in particular, focused on cloud-native containerized solutions. So some of the elements, again, signing and verification and provenance checks are important. So there is a kind of the three main parts are cosine, which the co stands for container signing, right? Making it easier to do signing of containers, OCI compliant containers, right? And making those signatures sort of invisible infrastructure, a key part, right? Those signatures need to travel with that image itself. Recor, which I already mentioned, right? A transparency and time stamping service. It allows you to kind of, it stores signed metadata in a ledger that can be searched but can't be tampered with. And then Fulcio, which is one of my favorite parts of it, but I think it's gonna take longer for this to be adopted, right? It's a free route certification authority. So cosine can be backed by a corporate CA, it can be backed by Fulcio if you want to use this free cert authority sort of like, you know, Luxe, if you're familiar with Luxe. And then also, you know, the cosine can, you can do an open IDC-based keyless signing, which is another cool thing. And I guess that's a part that I think is gonna take probably more time as well. But imagine you're working in a pipeline, you've got a login to GitHub to authenticate yourself to work with your source code. Maybe that's an OIDC-based login, right? We've got that on top of OAuth. So now I can use that data to do my auth, to do, to sign the content that I'm generating while I'm working in that environment. Yeah, that really simplifies the process, right? I don't have to reach out to an external corporate CA. My correct is Fulcio, it can be public as well, right? So if you are using something like open source, you could push the transparency, the sign, log up. So like if I download the container, I can go and check to verify that it's the correct... Yeah, I think you meant Recor. But yep, so Recor, in fact, like the Kubernetes community is, you know, we'll have, and Linux Foundation is sponsoring an open, a public Recor signature transparency log. Awesome. We do have one question trickling in. I was wondering about this one. What are the supply chain capabilities to find in the CD stages of the CI CD pipeline? So in terms of the enforcement of some of this, what do you sort of see happening? Yep, so I mentioned one of them already, which is, right, you ought to be verifying, especially, you know, so, cube and containers, right? This is kind of a big part of my world. As you're deploying images into a Kubernetes environment, you should be verifying the signature on admission. Similarly, one of the other things that it's really important to think about in this world is all of the YAML files that go along with your pod and your images, right? There's a bunch of content, a deployment YAML, maybe a Helm chart that you're working with, maybe an operator, those should be signed as well, right? And you wanna verify the signature of those on admission, right, so we may be, you know, kind of used to thinking about signing images. It's been something that's been talked about for years in our industry. Not, you know, big companies like Red Hat, they're signing what they ship, but a lot of organizations aren't using internal SIGs or for, and you know, when they're deploying their custom content into their environment, some are, but a lot they want to, but they haven't gotten there yet. And again, I think solutions like SIG Store are gonna help with that, but then the next step is making sure that that Helm chart hasn't been tampered with, making sure that your deployment YAML hasn't been tampered with, right? And so adding signatures to those and being able to verify those signatures on admission. Also managing everything as code is a key principle in this world, right? So a solution like Tecton helps me now manage my pipeline as code. I can sign my pipeline tasks, but then also I can do configuration as code for my cluster. I wanna sign those YAML config files too, right? That are gonna deploy my cluster. And this is something actually that Red Hat Advanced Cluster Management already has the ability to do is to verify, to provide signatures for your config files and to verify those on admission. Do you see a future where let's say a developer can write and package the configuration of the application, the container itself, and it can be signed all as one artifact and hand it off to the operations team? Or do you tend to see them as still sort of separate entities? Just there's things like Acorn that have popped up where they're trying to simplify the developer experience and I'm wondering how that would flow into a workflow with SIG Store. I can certainly imagine that folks would like it, would like to get to a place where it's all one and it's simpler to manage. I don't know, I'm not sure what my prediction is. Yeah, I think that one's probably a little two, three years out I think, right? Before we missed anything there. I think as you say, right, one of the things that I do think we're hearing in the Kubernetes world, is that Kubernetes, it's a wonderful application orchestration platform and it's challenging, right? What are the tools that can help developers focus on writing the code that drives business value and not have to spend quite so much time on all the piece parts that enable the Kubernetes orchestration, right? So can we provide more automation as a community in that space to simplify that? I think we've been seeing emphasis on the build environment as well. So it's gonna be interesting to sort of see how that all evolves, right? We've already gone through the helm versus operators thing and landed on it's not versus, there are cases where helm charts are your best thing and other scenarios where operators are the best thing for the situation, for the application. So maybe we'll see something similar evolve with Acorn. It sort of bleeds in the next question. I think I might be stealing this one from Kim. But with there sort of being some variability in how this is gonna be applied, things like Salsa, which are frameworks, then you have technologies like Sigstore. You're probably gonna see other technologies that pull from that open source and create something that's maybe like a startup that they're gonna sell to larger customers. So with all of those moving parts, how do you see compliance changing in the next two to three years around supply chain security? Yeah, so today security and regulatory compliance is very focused on platforms and then the applications themselves. And so we're seeing more automation for compliance happening. And kind of talk about that at Red Hat as compliance is code. We actually have a GitHub repo that is not used just by Red Hat, but by other large organizations as well where you can write code that helps to audit for compliance of a platform. You can use something like the OpenScap scanner on a container image itself to see if it is hardened appropriately for PCI DSS or other things. And so I think as organizations kind of move ahead in this supply chain maturity model they're probably gonna be looking for proof points, auditability, not just internal auditability, but how do I show my auditors what the level of compliance is that I have for these different frameworks. I think that's gonna take a little bit longer, but I think a lot of the tools that we, is it OpenScap, is it something else? But again, I think this is why if you're taking your first step and you're focusing on collecting log files and auditability of what you do in your supply chain and your software supply chain that's gonna be one of the key things. S-bombs probably also a major thing. The cool thing about S-bombs though is software build materials, right? They've been around. Tools to do that have been around for a while. It just, they just were used by a subset of the industry and now I think the use cases for those have broadened. But yeah, and the more you're automated the easier it is to audit. Mm-hmm. Well, for Kim and I, I think we're pretty much almost out of questions, but I did have sort of a too long, didn't watch question I wanna throw at you. For let's say security 101 people, you're looking to get started in supply chain security. Do you have any specific resources or documentation or call-outs where, hey, I wanna get started, where should I go? Anything that comes to your mind? So I think that, again, if you were in, I would say the Salsa.dev webpage has a great overview and plus it breaks it into levels, which makes it easier for you to kind of move up in terms of increasing the level of security or maturity that you're adding. You know, the CNCF white paper on supply chain best practices is also a great source, especially for those of us who are working in the cloud native world. You know, lots of additional information and content and that might be a place where kind of maybe you start by reading the white paper and then you go check out Salsa for a more concrete set of things that you might apply, right, that general guidance, that context provided by the white paper and then, man, how do I go about doing those things? Check out Salsa for kind of the step-by-step approach. Awesome. Well, it looks like the thunderstorm might have gotten Kim, but before we go, before I let you go, is there any place that any of your followers can contact you, any talks that you're giving? Because I know you've been talking about supply chain for a while. Yeah, certainly there are videos on YouTube. I am not gonna be at KubeCon North America this year, but I was at KubeCon EU, talking about cloud native security, not supply chain security specifically, but there are a range of recordings of me on YouTube. Y'all are welcome to do there. Being in the security space, sorry, I don't do Twitter. Yeah, so actually that's really more a personal preference. But yeah, so great to talk with you all today. Yeah, awesome. Thanks for coming on for the first episode. Unfortunately, I gotta do the sign-off without Kim. I'm kind of sad now. Something like thunderstorms, huh? Yeah, I was joking that the, oh, she's back. We gotta bring her back for the sign-off. Oh, that was awesome, okay. Awesome. Yeah, so the next show, it's going to be September 20th. So the third Tuesday of every month at this time, two p.m. Eastern, that's when we're gonna be doing the show. If you're looking for a security topic, you can find me in the CNCF Slack, email me at mfosterradhat. But we're always looking for security topics. Other than that, monthly show, we look to see you back here next month. Kim, anything that you want to say before we go? No, just thank Kirsten so much for her time and I'm looking forward to having this every month and for sure, open to other topics, but we'll go further into supply chain security next time. Cool, thanks. And I forgot to mention, I'm in Coob's Kubernetes Slack, so that is another place to reach. Awesome, thank you so much, Kirsten. Thanks for watching and hope everybody has a great rest of their day. Take care. Take care. Bye.