 everybody and welcome again to another OpenShift Commons briefing today as we like to do on Fridays. We're going to talk about cultural, organizational, transformational topics and securities high on everybody's list. And we have with us today the Director of Cloud and DevSecOps strategy for the Cloud Platforms Group, Kirsten Newcomer, who's going to walk us through a conversation about DevSecOps and what comes first, the tools or the culture. And you can ask questions if this is meant to be an interactive thing. So she's going to talk for a little bit. But if you have questions, please ask them in the chat. And if you want to join us in the video, I will happily turn on your faces and let you talk as well. So with that, Kirsten, take it away, introduce yourself and what you do here at Red Hat first. That's really interesting. All right, Kirsten Newcomer. And I do focus on security, in particular cloud native security, security for containers and Kubernetes. And really, from my perspective, that means security throughout the application platform stack, throughout the lifecycle of the platform and the applications themselves. And DevSecOps happens to be one of my favorite topics. And it's also interesting because it generates a lot of buzz. And yet when it comes down to figuring out who is actually doing DevSecOps, what does it mean to them? There are a lot of different opinions out there. And so that's one of the reasons I'd really appreciate it if this would love to have this be interactive, learn what any of you are doing in your businesses around DevOps and or DevSecOps. So just as an introduction, right, you know, this is, I think, relatively well understood at this point that businesses need the ability to innovate faster in order to differentiate themselves. IT departments need the ability to deliver those differentiated applications faster. And in the cloud native world, right, that's containers, Kubernetes, microservices, but also DevOps is a key enabler for delivering and innovating more quickly. And that's across a wide variety of markets and a wide variety of focus application focus, right, not just, you know, cloud native microservice based apps, but AI and machine learning, right, and Internet of Things. How do you make good use of the data, data lakes, all of these kinds of things really matter. And I'd say they matter even more with the pandemic, the changes the pandemic have kind of imposed on the economy, right. Everybody is doing more business in a digital fashion. And so why is DevOps a key part of delivering more quickly? Agility and innovation, by the way, agility really can't succeed in silos, right. You have infrastructure teams, you have the app platform team, you have management and automation. And of course, you have the app development team who are key to all of this. And if you really want to move more quickly, you have to do that in a collaborative way, right, in a way that allows you to work together and meet everybody's different goals. And you'll see that there's a security admin here on this slide, right, as well as the sysadmin. And at the same time, the slide just says DevOps. And so when we think about DevOps, a lot of people forget that security is actually was always intended to be part of a DevOps process. So if you haven't run across the term before, which I imagine is not the case with this group, right, these are three key elements that make DevOps work. I'm not going to read this slide to you. Culture is key, automation and technology. And I've talked with a lot of different folks about the which comes first thing, right. Do you, you know, what's most important? Do you need the tools that enable DevOps? Do you need the culture that enables DevOps? Can one come before the other? Do you need them both together? And I'd be interested to hear whether there are any experiences of any of you who are with us today, kind of around, you know, is DevOps working for you? And if so, you know, how did it, how did it start in your organization? And if the silence rolls on for too long, I'll just keep talking. So I think the way that we could do this is if you, if you're listening in here, all the folks who hear, if you have an opinion, just raise your hand in the chat and I can turn on your video or your voice and let you speak if you'd like to there and add that in. Or we can do the talking at the end, which might be there. Well, let's, yeah, we'll go for that. So the short version of DevOps, right, it's all about getting things out the door quickly to your end users and reliably. But we can't forget the security piece. And the reason I like to use the phrase DevSecOps is because it just calls it out directly. And it adds explicitly to that TLDR that getting the things out the door, the solutions out the door securely is key. So when we think about tools and cultures, you know, one of the things we do see at times is people start adopting DevOps without necessarily making significant changes to those silos, right? A DevOps team often becomes a part of the AppDev team, an extension of the AppDev team, rather than a real combination of AppDev, Ops, and the security teams, apologies for the background noise. And so we see at times, right, that that's still that kind of wall between AppDev and Ops still can kind of sometimes be there when we're talking about the DevOps team. And so unfortunately, that can lead to a behavior where there's an example of a single team of engineers thinking that they can solve the full organization's technology problems just by using orchestration tools, right, just by using the CI CD tooling. When we think about the elements that you really need to add into your CI CD pipeline to ensure that you have got true DevSecOps, we need to start with challenges, right? We have an ever-changing threat landscape. It just continues to emerge. The supply chain is clearly a new element, you know, it's always been an element of the threat landscape, but it's got new prominence these days. When you're working in the public cloud, you have a shared responsibility model. But honestly, even when you're working on premises, you still have a shared responsibility model, your AppDev team, your Ops team, your security, your network team. As people move to containers and Kubernetes, we see that while security principles still apply, oftentimes the existing tools that they're used to using and the existing processes are not really effective in a cloud native environment. And then, frankly, it can be challenging to find folks with security skills, and it's very challenging to break down the silos, which is why the culture question is such an important one. So we're seeing more and more of a recognition that to secure cloud native workloads, which are frequently ephemeral, they may only run for five minutes, they may run for longer, they may run for less. If one of those instances of a cloud native workload goes down in an open shift or a cube environment, Kubernetes is going to spin up a new instance from the image. So that means that you really can't rely on some of the traditional approaches to going in and having a human patch in environment when a new vulnerability is discovered, right? You really have to design for DevOps for your applications, but also I would say for your platform itself, right? We should be treating everything as code and we should be building security gates into our CI CD process, both for our apps and for our platforms. One of my favorite examples of a company who, an organization who really took a great journey with containers to DevSecOps is this public sector company, their federal customer. I'm not allowed to share their name, but when you get the slides, if you click on the link, they talk about themselves and outed themselves so you can certainly learn more. They decided as they were moving to containers and cube and moving to public cloud, that they were going to take the opportunity to see how they could improve security through the adoption of containers and take on a genuine DevSecOps model. And so they started with executive sponsorship, their chief of cyber defense, their VP of operations, their VP of application development sat down together, identified the key use cases that they wanted to address as a team and talked about how things would need to change to make that possible. So in this case, they started with the culture and with the people. They knew they were adopting new tooling. The adoption of new tooling was an opportunity to change some old patterns. One of the things that the cyber security team did was that he insisted that all of his security team members needed to understand how developers built and delivered code. And this really gave them a better understanding of each other. They had to talk to each other. That also gave the dev team an opportunity to understand what are the kinds of risks that the security team is really looking at and why. Many times, especially in larger organizations, the security team has passed on a set of principles or policies, but without necessarily information about why those policies are there. And the tighter collaboration makes it much easier to have a back and forth and a genuine risk assessment rather than just a thou shalt conversation where somebody says thou shalt and somebody else says we can't and here's why. And so having the what's the use case we're trying to address together help the business be successful, minimize risk, and deliver solutions faster. So the characteristics that help make this kind of change successful, open communication, respect, developing trust across those silos, that's key. Instead of matters too, people do respond to how they're measured. And so if you're primarily measured, if a developer is primarily measured on the number of lines of code they deliver, that may not be the best measurement. You might also want to be looking at how quickly do they respond to newly found vulnerabilities. Not so much is it vulnerability free code, because that can be challenging. New vulnerabilities are discovered in existing code all the time. But maybe the better measurement is how quickly can the team as a whole get an update to that newly discovered vulnerability and get that code rebuilt and redeployed. Give team members responsibility, push responsibility down the chain so that individuals feel that they can make a difference. And then finally really have to be comfortable with experimenting and knowing that some things are going to fail, and you want to iterate to improve those failures. And patience, patience of course is key. And then the teams who are really successful in taking on these kinds of things, this kind of this degree of change, they do that with executive sponsorship. They have good technical leadership, people who can dig into the technical challenges. And they often start with a pilot project, where they are genuinely they're given the opportunity to try it out, to experiment, to figure out what works, to change processes when their initial thoughts don't work, identify those key tools, and iterate until they feel really good about what they're doing. And then once they've gotten to a place, for example, one of our customers was able to shift from delivering every three months to every three weeks with a DevSecOps process. Once you've got that success, give the team a chance to brag about it throughout the company, so that their peers can learn from what they did too, right? You know, publicize what's happened. So when you think about, you know, some of the elements of containers and Kubernetes, right? There are some key areas to think about regarding security. How do I detect problems early in the life cycle? What does it mean to shift security left? What do I need to do in the build environment? Some of the best practices here, start with trusted content, make sure you use it a container registry to store any content you pull down from externally, put security gates into your build management process and into your CI CD pipeline. Protect your application platform by managing deployment in a GitOps-like fashion, right? Make sure that you are treating all the configuration information about your platform as code, you're managing it as source control, you can use GitOps, you can use Argo CD, there are a number of different ways to do this. And then of course, take advantage of the typical, you know, make sure you focus on the typical security things that matter, identity and access management, protecting data at rest and data in motion, hardening the platform out of the box and managing deployments to the platform in such a way that you can be sure that you're not unintentionally letting in malicious code. And then there's no such thing as 100% security, right? So iteration for your applications and for your platform, you know, a DevSecOps model or key, you want to protect the runtime as much as possible, you want to collect a lot of data, you want good observability about your running environment so that you can use that to notice anomalies and to respond to those. Of course, you need to protect your application access and data as well. And in a Kubernetes environment, you want to use network micro-segmentation to isolate running workloads as well as take advantage of things built into Linux such as SE Linux to mitigate or prevent runtime escapes. So I actually have kind of additional content that drills into these points. But I think, you know, and we could walk through those given that we seems like we have a lot of time. So maybe I'll go ahead and do that. Thoughts? Let me just ask a quick question too because you've hit on a number of key things from my point of view. And one of them, you talked about doing a POC first and getting that. How do you get from, how do you coach people to get from POC to a repeatable pattern? So like they do it once, you know, is there something in the coaching that you give when you're talking with, you know, our customers and other folks about this? I think that really comes back to iterating. So, and I would use the term pilot rather than a POC. So I think it's important to pick something that is an actual project that your team needs to deliver. And so you're going to pilot this new process on something that you actually need to make available to your end users. And therefore, and maybe you build a little extra time into that pilot to acknowledge that you're going to have to do some iteration. You're going to be making some changes as you go. And you also need to be sure that there's executive sponsorship. But the goal is that you are going to define processes and refine those processes using an actual project so that, you know, normally it's very unusual that you would deliver once, right? You're going to have a new version of that same app another time down the road. So if you can do that with something that is going to be long lived and learn from it, there's there's for one thing, there's much more motivation, right? Because it's again, think of it as a pilot rather than a POC. So the other thing that I was thinking, and then we'll go into your other slides and if people have questions, please pop them into the chat or I'll turn you on your microphones on. One of you talked about the technical leadership and one of the groups that you didn't mention but I think it was unintentional is like the compliance and risk officers in companies that we talk about the developers and the operations and that but getting buy in and educating that that layer of the organization has always been like the for me one of the hardest but best conversations to have because they're the ones that expect the audit logs and all the paperwork that they've had before and the sign up and if you don't bring them on board, you kind of pooped because then they're still asking you for you know, I don't know, whatever the huge JSON file, you know, I need my JSON file and go back through it line by line or something like that. That's a yeah, that's a really good point, Diane. And I think it is challenging because in a lot of organizations, the compliance teams are responsible for understanding the standards that have to be met and maybe they're the ones who talk with the auditors or those who certify a deployment to to be compliant with a specific specific set of regulations. But they generally aren't the compliance teams generally aren't deeply technical. They usually rely on the technical expertise of the security team or sometimes the active teams themselves. And I think that's a lot of what leads to kind of the the frustration and the communication is that these groups aren't necessarily talking the same language. Yeah, I'd prioritize connecting in the security team first, because they often do represent compliance. And they're usually more, you know, they're usually a little deeper in the technology and they're therefore better positioned to have that back and forth conversation. And then but you are right, ideally, you want a compliance you want you want to have that conversation with compliance. I guess I'd say that I think one of the best things to do and again, I mentioned very early on that automation is a key part of DevOps. So as you're thinking about your processes and your tools, keep in mind that you do have a compliance team that needs output in a form that they can take to an auditor, whether that's R for JSON or whatever the format is that they're expecting to get that data in and think about and look for automated tools that can help you demonstrate that compliance in a way that reduces the effort for everybody. Yeah, I think that's it. And Chris has Chris mentioned in the chat is that he could not stress how bad a pattern it is to measure folks by lines of code commit. Yes. Finding, finding metrics or DevSecOps pilots and projects is an interesting thing in this like images scan without, you know, things are because you also mentioned like in the past, you know, patches would be applied by humans, you know, and, and there will be a log file that the patch had been done. So this, this is whole thing when you're automating security and doing security is code, which is really where we all should be migrating to if you're not there yet. But the other piece of the puzzle is like how do you tell people that you've done it, you know, like you measure that it's done in a way that's tangible or, and I know I'm hooked on the audit side of things. So an auditable and that's always been the interesting conversation that I've ended up having with people is like, yeah, developers get it, systems get it. And usually we can talk the sea level people into it. And then everybody forgets to talk to the compliance and risk officers. And then then there comes this grinding halt like, well, you can deploy that until I have this, you know, and that's, that's what I'm getting at is that that's the, and that's a cultural shift. That's like including, you know, the entire and collaborating across the silos that you mentioned, and making that something that's just part and parcel of the conversations because, you know, plopping containers and images and, and pods and, you know, OCI stuff on to a compliance and risk officer is like not a good thing to do. Right. They're not going to understand it. Yeah. Yeah. And, and, you know, there are ways again to show them that you care about what they need to deal with, right? I mean, they they and when we come back to measurements, I mean, there's, there's two elements of measurements. There's, there's like, how do you measure what's happening in the, in the pipeline, which is almost the dev sec piece of things. And then how do you measure what's in the sec ops area. And, and so, you know, a measurement can be yes. And, and this actually applies to a process piece for compliance officers. Right. Yes, we run vulnerability scanners on our private registry at this interval. And I can show you dated reports demonstrating, you know, that the images that were scanned. Yes, we run vulnerability scanners as part of every CI build. And I can show you the output of that, right, that that's a documented because, and now I tend to focus on technical controls on the sec ops side of things a little bit. But it's absolutely true that compliance requires it involves process as well. Right. So, so as you kind of define your processes, you want to document them. And as they change, you want to update that documentation. And that can be a hard thing to do. But back to your point about auditing, because we need to audit the platform for technical controls, we need to audit our processes to ensure their compliant. One of the really nice things about doing everything as code is that you get audit trails of who did what and when something changed. And so maybe part of what, you know, would really help the compliance team and the auditors is helping them understand how to view that audit data, both through the pipeline and the audit data that you might produce for technical controls on the platform itself and giving them a way to kind of check it for themselves. I know because relying on humans to do patches isn't really an audible auditable trail in the real in the reality of it. I mean, we may culturally trust it more maybe in old school organizations, but it's really it's you're relying on humans. Yeah. And how good we all are. Well, and more often the way audit and audit trail is created, and that's an area right for a traditional architecture is I've got a change management system, something like ServiceNow, a request gets logged. So, you know, alert, you know, somebody says new vulnerability alert gets sent a change request gets logged saying, please patch this vulnerability. And then somebody has to, you know, that gets handed off to the appropriate team and they have to schedule all of that. And, and then so your audit ability relies on people kind of passing along this change request in an automated CI CD pipeline for an application or for the platform itself, right, you have processes in place, you can have automated triggers that say, Hey, there's this new vulnerability discovered in one of the images you're using, I'm going to automatically pull down the latest version that has that fix. And I can automatically kick off a rebuild of that application with that fixed face image. And then you might have gates around deployment, right, you need to be sure you've tested it, you want it, you don't you don't want to just rebuild and redeploy, but you can automate a lot of those gates. And then if it's for the platform, you can do something similar, because, you know, a platform like OpenShift allows you to apply the, you know, patches in a rolling fashion across the cluster with zero app downtime for well behaving apps. Now, some orgs are going to want to schedule those patches anyway, right, but you can still use automation to do that scheduling. All right. Well, then maybe we should deep dive more a little bit more into some of the other aspects of this. And we'll see if we can coach some other questions out of the folks that are not listening. And this will go for it. All right. So when we think about detecting problems early in the life cycle during the build process and the opportunity to shift left, this is I think what many people think of when they think of DevOps and even DevSecOps, right, this is the pipeline that we're going to move things through in order to get them into production. So trusted sources in for containerized apps, trusted sources is critical, right, you're pulling down all your system dependencies for your custom apps are going to be integrated with that container image. So whether that's an alpine base image, a rel UBI base image, you want to be sure that you are using content that you have a trusted source for, you can verify where it came from. You can and ideally also that it's a source that regularly updates their content so that if a new vulnerability is discovered, you have easy access to an up to a fix to that vulnerability and updated base image. A private registry is key because again, even though you're pulling from external sources, those external, even if it's a trusted one, that external source might go down, you know, there was been a couple of years now, but there was a point in time when, you know, their AWS had an S3 outage that really impacted people's ability to pull from some of the external registries that store container images. So it's always good to have a local copy and it makes it easier to ensure that you do trust but verify. So you start with a trusted source, but you still run vulnerability scans in your private, in your private repo to ensure that things are okay. Security gates in your pipeline can and should include vulnerability scanning, but an emerging area that's really become more and more important is application config analysis. Any solution, any pod app, you know, containerized app that you're deploying to a CUBE environment has a lot of config data that goes with it. And so you want to be looking for things like are there embedded secrets in the image, but you also want to be looking at what security requests are being made. What's the security context for the pod? Is the pod asking to run with extra privileges that you don't want to allow on your cluster? All those kinds of things are things that can be done really early on so that, yeah, you've got built-in suspenders, right? You're going to prevent privileged pods from being deployed, but you don't want to find that out when you're ready to go to production. You want to find that out as early as possible so the developer can fix it. And then, of course, every application should be designed in a way that it can support logging and monitoring so that we have continuous observability once it's deployed. When we think about the platform itself, again, configuration and lifecycle management of the platform matters a lot. You know, we have a number of, even though OpenShift 4 itself is highly automated, supports automated operations with Kubernetes operators. We also have customers who ensure that every part of a platform deployment is managed in an automated fashion through a platform pipeline so that, again, it's auditable, and they can verify that everything is deployed in the way they intended it to be deployed. Furthermore, this really sets you up for, just like containerized image applications, you should think of them as ephemeral and you can always be redeploying from an image. You should be able to think about your cluster that way, right? You should be able to tear down and replace your cluster at any point in time, and automation makes that possible. So, GitOps, Argo CD, many types of ways to manage that, but definitely your platform should be treated as code as well. Host and runtime security, we're going to talk a little bit more about that. Of course, identity and access management for the platform, ensuring that data at rest and data in transit is protected. In every Kube cluster, you've got an SDN, so you want to be sure that you are leveraging things like network policies for network isolation, network micro segmentation, that you're managing ingress to the cluster, and effectively managing egress from any pods running on the cluster to off-cluster services, right? And in general, you want your ingress to be encrypted, you want your egress to be encrypted, and you may have your off-cluster service, may have a firewall in front of it in order to further ensure that only authorized applications are able to access that off-cluster service. Platform logging, monitoring metrics, and then audit and compliance, Diane, to your point. And ideally, look for tools that will help you automate, audit for compliance with technical controls from regulatory frameworks, and there are many available to you, including the OpenShift Compliance Operator, but many, many folks in this space because of the automated nature of Kubernetes, because the platform is continuously changing and nodes or servers are coming and going, automated compliance and automated audit are really key things, and there's, there are plenty of solutions out there available to you. Again, automate the life cycle. We kind of hit on this. So let's talk a little bit about managing the deployment. Once you've got your cluster up and running, you've got it configured the way you expect. You're, you're all set up for managing updates to that cluster. You're monitoring for configuration change or drift on the cluster and managing that. Next, we need to be sure that as workloads are deployed to the cluster that we're managing those appropriately. So ideally, you've got your own private repo where you're storing code, storing your images. You want to set up allow lists and block lists to ensure that especially in your production environment, only, only images that have been approved for production can be deployed to your cluster. Again, we want to use a get ups and or and or Argo CD approach to that deployment continuous deployment piece. You want to take advantage of Kubernetes admission controllers to ensure ideally you found any, you know, requests for excess privileges early in the life cycle through doing app config analysis. But again, so that's your belt, you need your suspenders. So something slips through something didn't get vetted, use pod security policies or open shift security context constraints to prevent admission of pods that have privileges that you don't want to allow on your cluster. Validate image signatures, leverage something for your apps like service mesh to add an additional layer of protection for application traffic. And of course, continuously monitor for new vulnerabilities and be prepared always to rebuild and redeploy not just your apps, but your platform as well. And again, when it comes to getting started, you know, start with a pilot. You don't have to automate everything at once either, right? You can do the goal is to get everything as automated as possible, but you can start with with simple steps, right? Start where it's easy to automate. But be sure that all your team, you know, that you collaborate across those silos, and that you get use cases across those silos and find a way to facilitate those conversations, whether it's a Slack channel or, you know, an email alias or these days, you kind of don't have a what we used to call a war room, you know, but maybe it's a zoom session. And then find measurements that measure the positive behaviors that you're trying to reinforce. So, and that's that's it for slides. So, well, awesome. And, and, and that's really I think culturally and technology, like I tend to focus on, you know, Argo CD or cube linter or, you know, that goes straight to the technology. So it's really nice to have a conversation about the cultural pieces of this and culturally for security teams, this concept of experimentation and allowing for failure is probably one of the highest bars to, you know, mentally get through. Like, that's, have you found that? Is that like the thing? Yeah, talk about iterating, but allowing someone to fail in a security scenario is sounds like, you know, an oxymoron. Yeah, no, that's definitely that's that's a really valid point. I think security teams are, you know, used to trying to again, really their job is to minimize risk. There is no such thing as zero risk. But sometimes they get this mindset that there has to be zero risk. And so kind of oftentimes you need executive support to shift that mindset. But another way that the technology enables a change in the mindset, because containers are the same from dev to test to production. If you set up your environment, your, you know, if you're, you're deploying to a cube cluster, it's the same container image deployed to a cube cluster, whether it's dev test or production, if you set it up so that, you know, you're really doing that testing in exactly the same kind of environment as your production environment, then you can maybe get your security team a little bit more comfortable with the idea of iterating. And ideally, you know, if there's going to be a failure, it's going to show up in the test environment as long as you've got the right tools in place. And because clusters can be spun up and shut down so easily, I think it makes it simpler than a traditional architecture where, you know, VMs might be scarce, you'd have to, it would take weeks to get a VM, you know, you'd have to request a VM, it would take weeks to get it provisioned. The VM environment is different from test and production. And so the closer you can get these environments to be exactly the same use the same tooling, the same security tooling, the same network policies, the same configs, the better your chances of finding something early and giving some comfort to those security folks who are so worried, right, that, you know, their job is to prevent the company from showing up in the newspaper. Yeah, well, I think that to me, that is like, and to the promise of Paz or Paz or however that was always the thing that I loved about it, you know, seven or eight years ago, when the whole concepts of, you know, Heroku showed up on is the repeatability of the environments, you know, is that, you know, previously you would do development on your local machine, throw it over to tests and QA and all of this. And, you know, we've now had like seven or eight years of this containerization stuff. But I also think that you and I, we're at Red Hat, we're at the bleeding edge of the knife, right, you know, so we drink this Kool-Aid all the time, and we kind of have to realize that not everybody has followed this path as quickly as other, so I think that's really culturally still coaxing people along is a key piece of this. Yeah, absolutely. And again, you know, and it's hard for big organizations to change, right. And so I think that's one of the reasons why when we've seen it be successful, executive sponsorship matters. If you, and it's not that you can't be successful without executive sponsorship, but exec sponsorship makes a big difference. And again, starting with a small group and then trying to take that success, learning from that and expanding it across the larger organization is a win too. And it's funny what you were talking about with the Paz. I mean, I remember, gosh, oh, well, over maybe 15 years ago now, you know, this, this goal at a company I was working at to be able to more easily, right, you find a problem in production. How do I trace back from that production code to the root cause, the source code problem that in a development environment, so that I can more easily reproduce and fix and, and get that fixed back into production. And there were all sorts of different kind of ideas about how to do this and how data sharing and how do you get traceability from the source code to the binary. And we're in a world where less of that matters because it's the same image from dev to test production. It's kind of cool. So we do have a question from Dave in the chat here. And I think it's, you'd rather stay off camera. So that's okay. My hair is doing okay today. Do you see security teams being spread thin and DevOps teams? And if so, does training developers on basic security coding practices become a priority? Like, how do you spread the load kind of? It's a great question. I mean, the truth is that security folks are security skills are thin on the ground period. And, and they're expensive. And so the more an organization can do to help all members of the team understand what the security goals are. And again, the why behind them, because the technology does change. And so, you know, there are, there are scenarios where something like saying, you know, every certificate has to be signed by the corporate CA. There may be cases where that's appropriate. And there may be cases where that's less appropriate in a, in a cube cluster. You know, maybe the platform certs don't need to be signed by the corporate CA because that cluster is a closed route of trust. But you want your application certs signed. But you need to have conversation about that. So, so one thing we, I haven't heard as much about this recently. But a few years back, I started hearing, you know, about roles called like the BISO role, the business information security officer who was embedded with app dev teams. And because they were embedded, they could have conversations about risk. And the developers could have a conversation about, well, we don't, you know, doing it this way is a problem. But what if we did it this other way? Can that still meet your goal? And, and so anything that an organization can do to facilitate that, to facilitate that learning. And then again, to help developers feel like it's worth their time to better understand the security requirements. I mean, they don't want bugs in their, in their code either. But what hasn't happened, right, that the thou shalt messaging doesn't work. Developers like to understand the why. And then they can get really creative about the how. So the more we can, more communication we can foster the better. I think that's a great answer to that. And Dave, if you have a follow on, just type in the chat and I'll read it out for you. The other thing you mentioned was the rise in the importance of app config analysis. That's, you know, that, that, I think, and I, that's, I'm just coming off talk with coob linter folks from, so it's like, been top of my, can you talk a little bit more about, you know, how that is evolving? Sure. Well, I think again, some of this is, is due to change in technology, right? It used to be that, you know, so the developers responsible for doing, you know, kind of building their code and, and, and doing, doing testing in a test environment. And there's a certain set of configs that they know they need. But in production, right, it was always the ops team, the developers would say, well, here's how I need my, my VM configured. It needs to have these system, these system libraries, and it needs to have these ports available. And, you know, it kind of boom, boom, boom, boom, boom. Everything would be listed out and handed off to ops. And then, you know, the ops team would have an opportunity to kind of determine whether it met their concerns and the security team to networking team. And now all that configuration in a, in a containers and coop world, all that config data is, is in my deployment. It's in my coop deployment. It's in my pod, in my, you know, what user I want to run at, what Linux capabilities I might need, you know, what, what are my network connections that's all going to be in the SDN. It's all less visible to the ops and network and security teams, because it's all configuration is code for the application. And so one of the ways you win trust is by using that automation, those, those newer automation tools to analyze for challenges, whether it's a Helm chart, a deployment config, something in YAML image, run those analysis tools looking, it's also a way to educate your developers about what's okay to do and what causes problems. So run those analysis tools early, share the output of them, provides, you know, visibility across the team, and it reduces problems early in the life cycle. I didn't think ever at this point in my career, I would be editing so much YAML. I just like YAML this and YAML that, like everything, everything, and the access to the config files and the YAML and the Helm charts and stuff, you know, we have access to tweak that. Yeah, yeah. Chris is joking about calendar driven YAML engineers creating these things. Yes, it's great. And it is, it's like, for me, that, that, that is the, you know, the part that was when I was back in the day when I was writing applications and deploying them, that was the mystery part, like I would have it configured perfectly for my thing. And now I go into IDEs and things like that, and I have access to tweak it again. And so what this automation layer to get to the deployment, you know, you can tweak it up, having that automation in the CI CD, check it before it goes into production is really kind of key. And it has really made me a better developer, shall we say. There's one question that's coming in here. Alec is asking, what is the best step-by-step prescriptive guide for DevSecOps with OpenShift? Did you write? Excuse me. Is that a leading question? I don't know. Maybe, maybe we should write one. We do have, we do have some folks in Red Hat who have put together what they might call would be the, you know, a container factory that outlines. We can, we can dig up a link to something like that and share it. I'm not sure whether it's in a blog or whether it might instead, you know, be somewhere else. But I think Red Hat believes strongly in prescription but with choice. So. Yeah, I think that we, I think writing the book would be good. And then you just brought, if it's in a blog, I'll be really upset because one of my, my least favorite things is when we document by blogging. And then they never get updated or maintained. That's all they can monitor I have is in, you know, in all the community, you know, open source projects is, please, if you're gonna do something. But I actually think that is what Alec is asking is a title for a great book to work on. It is. It is. I like it. It works with OpenShift and some sort of a guide there. If it isn't in the container factory, hopefully not blog, but it probably is because I know our friends and family here at Red Hat. So we'll find that link and share it with everybody. An e-book's a great idea though. Yep. Let's do that. I think we could be good. But that's, you know, that's sort of taking dev step ops one step at a time. I think that's a great slide to kind of end on for everybody to think about where we're at. And if you go to your next slide, so people know how to find you. I don't think it's there. All right. Well, I'm going to have to add that into the slide deck afterwards and do that. So it's hopefully it's in some repo that that container factory. We'll look for it. And up to it recently. And I will add this. Caroline, I will add the slides. I'll take this whole talk, which thank you so much for spending the time with us today to talk about this.