 All right, everybody, and welcome to another OpenShift Commons Hour and Briefing, streaming live on Twitch and in BlueJeans as well. And today we have a two-parter. We're going to hear from Kirsten Newcomer, who is part of the OpenShift product management team, and one of my favorite security people on why DevSecOps is critical for containers in Kubernetes. Coming off a great blog post, which you should all read on dark reading. I think that's what it's called, a dark reading blog. We'll post the link to that as well. And then we have Keith Basil, who is going to talk about some of the things you can do to automate some of the compliance in OpenShift afterwards. So it's two-parters, and we'll have some live Q&A at the end. And so hang out with us for an hour or so, or see how long this goes. And first off, I'm going to let Kirsten kick it off with a short talk on why DevSecOps, did I just say that? DevSecOps is critical for containers in Kubernetes. It definitely is. So anyway, let's go for the day. There we go. Take it away from me, Kirsten. All right. All right. Happy to talk to everybody today. So, and let's see. There we go. So, as folks know, right, I'm just going to do a little bit of lead up here, but right today's world, everybody needs to work to differentiate from everyone else. And a key element of differentiation is being able to deliver whatever innovations you have been working on as fast as possible to ensure that you are addressing your customer's goals and kind of getting ahead of the competition. And so really with changes in technologies that have been emerging for, you know, probably over 5 years now and have really started to become widely adopted containers, Kubernetes, microservices and DevOps all really help to ensure that you can deliver that innovation more quickly. So, hopefully by now folks are pretty familiar with containers and how they make it easier to deliver applications faster, right? You get to package all of the system dependencies that go with your application in that container image. That means that as you move from dev to test to production, there are fewer configuration changes if any to be made that config is those system dependencies are all in the container image. Anytime you deploy, you're deploying from that immutable container image. So, you have also a simpler lighter way to deploy your application and portability across what environment. So, you can if you choose to deploy on-prem or in public cloud, you can still use that container image wherever you're deploying. And Kubernetes makes it easier to manage those containerized applications, right? You need a platform that helps you scale, that provides resiliency and built-in high availability for your applications. And container orchestration platforms really require some additional things, networking and routing, persistent data storage. All of these are capabilities that Kubernetes provides. And really, while we saw, you know, three or four years ago, we saw a wide variety of container orchestrations out there, orchestration tools out there in the market. Today, really, we see primarily Kubernetes. We know it's one. So DevOps, so we've talked about containers for applications, making it easier and faster to deliver Kubernetes, making it easier to manage your containerized applications. The declarative nature of Kubernetes ensures that if an instance of your containerized app goes down, another one is automatically spun up. But in addition, DevOps is really a necessary part because agility can't succeed in silos, right? So we're talking about an environment here that involves all of these people down at the bottom of this slide, right? You've got your network admin, your security team, your developer who's building the containerized application, the line of business who's driving the innovation that your customers or your partners need to take advantage of. Because you may be deploying in the cloud, you've got your cloud admin as well as the sys admin to think about. And of course your ops team has to think about all of these elements all the way through. So DevOps is about getting things out the door to end users quickly and reliably. And DevOps is about doing that quickly and reliably and securely. And the reason that DevOps matters so much for containers and Kubernetes is because containers and Kubernetes are both an opportunity to improve security and a challenge to the traditional approaches to application and platform security. So this content is actually based on a YouTube presentation that was done by OpenShift users, as you can see back in 2017. But this group was particularly committed to, as their chief of cyber defense put it, to improving security with containers. He really felt that this was a huge opportunity for them to change how they addressed the security of both the applications that they're deploying and the platforms they're deploying to. And they did that by genuinely adopting this DevOps model. So their chief of cyber defense, the head of the app dev team and the head of the operations team got together. They sat down in a room. They determined what were the key use cases that they needed to meet as they were embarking on this container journey. And they came to agreement not just on those use cases, but also on how they were going to meet those use cases. One of the things that was key here because DevOps is not just about technology. It's also about people. So actually two things were key. One is that this group at the upper level of the organization decided that this was an important opportunity and decided to work together to figure out how they were going to make this happen and empower their teams to make it happen. And one of the other big different big, big and important changes here was that all three teams decided they needed to learn from each other. They needed to understand more about what mattered to each of the different groups, each of the different silos, what used to be silos. And in particular, the security team determined that they needed to understand more about development tools and how developers did their work. And this wound up in a situation where now security has shifted left in their organization. More of the security tools are deployed in the CICD pipeline. The security team is alerted early on if there is an issue and can, as is stated here, can break builds if they wish. And really all that means is they can say, hey, this build can't go any further until you fix that particular security issue. And that's a benefit both to Dev and Ops, not just the security team. I mean, as a developer, who wants to hear right at the end as you're ready to deploy your application that there's a late breaking vulnerability that you didn't know about that your security team found. And now you have to rebuild and redeploy and that delays your release. And Ops doesn't want to deal with that either. They need to know right now we have OS libraries embedded in those containerized applications in that container image. And so sometimes the updates that need to be made are being made in that OS layer, which is the traditional responsibility of the Ops teams. So again, all of these teams need to collaborate together to have an effective agile and quick release process that also provides the security enterprises require. So let's kind of take a bit of the deeper dive. So containers do change the application security model, right? And you know, this is the workloads are becoming more short lived, more ephemeral, right? So when you were deploying monolithic apps to a physical machine, kind of everybody knew, you know, what was the IP address of the host that the app was deployed to. Then when we moved to virtualization, you still had a lot of that same information. Now you have VMs to configure, not just the apps. Those VMs might live for months or years, but you move to containers and containers are intentionally ephemeral. They are intended to be spun up and run for as long as necessary. And then with the addition of Kubernetes to the containers environment, right? If a container instance, even without Kubernetes, if a container instance dies, you would just deploy a new instance from the container image. Kubernetes just makes that easier and more automated. And so those changes mean that we need to think differently about our pipeline. We need to add even more automation into our application CI CD pipeline. Because containerized images again require those those OS dependencies to be built into the container image, you're always going to be using some external content in your containerized application. You want to assess that content as early as possible as soon as you bring it down into your environment. That's one of the key reasons for using a private registry to manage images. You want to think about what security gates you need to put into your CI CD process. Are you willing to have certain, you know, critical vulnerabilities break the build as our as our friends did mentioned earlier. How are you going to manage application secrets in a container world? What do you what's the runtime security posture that you need to think about. And one of the key differences that really drives the move to a DevSecOps model for building your container images is that you should never patch a running container in place. And one of the reasons again is so let's say you're running your containers and Kubernetes. You've said I want three instances of those of that web front end part of my application running one of the instances dies and Kubernetes is going to deploy a new instance of that from a container image. If you patched that running instance that patches lost. So you always need to rebuild and redeploy to address any security issues or any other type of bug of course. So that just means you need to think again about more automation and there's some opportunities here. So you want to look at what security tooling can you build into can you integrate into your IDE environment. So for example, you might want to use code ready workspaces which is a hosted IDE that allows you to kind of gives visibility to everyone across the team. About the codes you're working on as you're working on it and it means that you don't need to download any code to a local laptop right. You can ensure that the code is secured in one place because it is an IDE as a SAS offering. You can use solutions like sneak which do early on in the IDE provide dependency analysis. And so if there is a known issue in one of the pieces of the code that you're using it'll let you know what some of those other dependencies are say you need to do an update. You absolutely need to be investing in automated unit testing. Oftentimes for organizations who are, you know, moving existing applications to containerized apps, automated testing is one of the biggest stumbling blocks. So, so sometimes when you're starting out you want to start with a green field app, but unit testing is usually easier to add than some of the more complex testing that you might do down the road. But you also want to add in things like sonar cube or covariate for code quality some of those code quality issues can actually be security concerns. And then you want to think about security scanners like Aqua security clear which is available to you with red hat quay twist lock now known as Prisma cloud. And you want to take advantage of things that may be available to you in a platform like OpenShift forced to image which makes it easy for your developers to work with the latest image from red hat and image streams and combination always making sure that you're pulling down the latest base image with those OS dependencies and built in pipelines that help you automate rebuild and redeploy. But you absolutely want to be sure that you invest in the appropriate level of automation here in your pipeline and also in your production environment in your runtime environment. And so in addition to the built in security capabilities that come with a platform like OpenShift, there are partners out there who provide additional security capabilities as need that can enhance what you've got available. You can work with companies like Aqua for Aqua and twist lock again Prisma cloud for additional runtime security cyber arc for secrets management cystig for deeper data collection and analysis. And that's really kind of on the application side of things. So the dev side. As we step into production that is again more the upside. But as we continue to think about ops. Let's take a look more about Kubernetes at Kubernetes itself right so back in May of 2019 the CNC and CNCF released open source security audit that they had commissioned of Kubernetes itself and there's a link here to one of the four pieces of collateral. That was released and the overall conclusion of the audit. Is that Kubernetes is a large system with significant complexity. And so again, some of that complexity comes from the declarative nature of this the system that it has many different parts and each of those parts has declarative config data. So it's not the fact that it's declarative. It's more that there are many different parts with many different configurations. It's a dynamic system. Things are changing regularly as new applications are deployed new solutions added to the cluster. It's automated and it is an API managed platform and there are organizations who are not used to the level of automation that. Kubernetes leverages and the benefits of it and and to get the real benefits of that automation you need security solutions that know how to work with this platform in a much more automated way. Kubernetes requires an SDN. Some of the traditional network security tools don't work well with SDNs right application configs and deployments are managed in new ways. Whereas an application deployed to a VM again, you know kind of that might be in place for for months or even longer applications deployed on Kubernetes are going to change as needed as something happens. You need to scale up. You tell it you need to go from three instances to five. I've now got five instances and I don't know where on my cluster those are deployed locations change as needed. And containers themselves are opaque to many existing security tools as as well as in general the way. Kubernetes works for example there are a lot of. You know certificate management solutions that aren't used to working that that aren't used to being managed in an automated fashion teams might be used to asking for a certificate for their application that they're planning to deploy from the team that owns the CA. And it may take a couple of weeks to get that set of certs for their app where they need a new VM and it may take, you know, a week to a month to get a new VM for their application. Kubernetes once that cluster is deployed and up and running. You're in a position to just go. So again collaboration understanding what all the goals are of the different teams and how to ensure that those teams are met. Really that those goals are met really requires a highly automated set of solutions to complement your Kubernetes and container deployment to ensure. So you need security automation, not just application and application platform automation. Kind of already been saying this right so doing Kubernetes right for the enterprise involves a lot of different things. You have to have your hosts set up you need to validate your environment. You need to manage identity and access. You need to think about adding monitoring for the platform as well as for applications. There are going to be cases where you need persisted storage. You need to think about managing north south traffic for the cluster ingress and egress. You need to think about effective management of the SDN and the east west traffic. You need to think about potentially how are you going to manage charge back if you're running in the public cloud. How are you going to harden the platform. What certifications do you need disaster recovery and how are you going to manage patching and upgrades to all of the different components that make up. Not just Kubernetes, not just your custom applications, but all of the additional things you add to Kubernetes to make it enterprise ready your logging stack your monitoring stack. Your security tooling that you add. So, in addition to kind of all of that we so kind of as a high level way of summing that up right we need to think about host security runtime security identity and access management are back. Project namespaces so Kubernetes has the concept of namespaces if you choose to take advantage of those to have a multi tenant cluster. You want to be thinking about how you're going to work with that network isolation. All of these things that we've touched on with the addition of secrets management for the cluster itself. And so one of the ways we've tackled that with open shift for is we've taken advantage of a new object type available to the Kubernetes ecosystem which is Kubernetes containers Kubernetes operators. And that has allowed us an open shift for to use Kubernetes to manage and secure Kubernetes right so we are and we have delivered open shift for on rel core West, which is a containerized container optimized operating system, which is also managed with an operator the machine config operator. So this allows us to deploy an opinionated Kubernetes cluster full stack deployment from the OS, all the way up through the Kubernetes layer, including audit and logging, including monitoring and metrics, and including the CI CD tooling that developers might want to take advantage of. Of course, you know with Kubernetes you can use your own CI CD tooling outside of a cube cluster, or you can run that inside of a cube cluster. This also has changed how we manage patching the full stack that we deliver with open shift. So we are now able to provide patches to the host OS layer the Kubernetes layer, all of the other components of open shift, including the SDN the monitoring the logging in a rolling fashion through one interface. We test everything together we deliver everything together we patch things through the same interface. And the operators that are used to manage each of these components in open shift. Apply those patches in a way that enable a rolling update across the cluster in particular for the host OS the machine config operator ensures that again we're taking the approach of managing the host as an element of Kubernetes. So if there's an update at the host OS layer open shift will notice it will mark a node as being ready for upgrade it will move all of the workloads off that node and on to a node that's content going to stay in service. The OS update will be applied this host will be rebooted put back in service and open shift will move on to the next real core OS node. So we're doing zero application downtime upgrades for well bathing apps. On this on this Kubernetes environment open shift managed with Kubernetes operators. So DevSecOps at work. So when you think about DevSecOps for your teams. How can you get started if this is not something you're already doing. So just like the group that we mentioned early that I mentioned earlier in this deck. You really kind of first of all you need to identify your key stakeholders and what your business objectives are. And ideally you do this with executive sponsorship that's one of the most effective thing effective ways to move forward. And of course you also need technical leadership and you need those technical leaders to be willing kind of to step outside of their silos their boxes and learn from each other. It's great to start with a pilot project we find that that is the most effective way. And kind of define initial processes tools and then you know kind of identify your initial timeline measure progress but just like with any agile process you need to be prepared to iterate. And learn from what from your mistakes. And as part of this again because it's not just about tools it's also about people. Think about the cultural elements right so here are some of the key things that that you really your team really needs to embrace as part of DevOps or DevSecOps culture. You want to push responsibility down into the team as far as you can. You want to incent people to take responsibility and incent them to experiment and not be afraid of failure to work with each other in a respectful fashion to find ways to trust each other and learn from each other and keep the communication open. And one of our customers Shipple Amsterdam Amsterdam airport achieved a 50% reduction in application development time through adopting dev dev ops approach. This you know kind of the bullets here are a little bit not just about their dev ops elements right they use the development pipeline and option open shift they use source to image, but also some of the application tooling that they used as well. We've run thousands of containers and mission critical workloads on open shift today and we have other customer numbers that are even even higher than this. So, just kind of to sum it up dev sec op models model is really an extension of the dev ops model dev ops was always intended to include security, but in many places that sort of got lost along the way. As you look at any existing dev ops teams you're working with today. Say, look, take a look. Have they added enough security capabilities into their CIA into the CI CD pipeline. How are you automating security out the operations and how are you iterating. How are you adapting as you learn from the changing landscape and as you take on new types of application technologies. So, I'm sure and if folks would like we have teams out there that we would you know customers or or if you have an example that you would like to share with us we would love to hear your experience in moving into dev ops and the dev sec ops journey. That's what I had for you today. Diane back to you. Thanks. And that was that was great. It's a wonderful introduction to all the concepts behind dev sec ops, as well as the importance of it for the Kubernetes and the whole cloud landscape folks so really thrilled to do that. I put in the chat a link and I'll post it again at in on the Twitch stream as well to the dev sec ops dig. I said it right this time, but the other one would probably get more traction. And so, if you're interested in this topic, please join us. We're going to start kicking off some regularly occurring conversations around dev sec ops with Kirsten newcomer and our co chair as well. John Willis from the office of global transformation. I love that title. And so please do join us there if you can and when we'll send you an invite to the kickoff meetings for that. That's a wonderful way to be part of the conversation as well. If you just simply join OpenShift Commons, we will give you put you on the newsletter and get you announcements of all the other sigs and other briefings and other twitchy things that are happening to. So.