 Good morning. Good afternoon. Good evening and welcome to another edition of dev sec ops is the way today. We'll be talking about compliance. I'm Chris short executive producer host and I'm going to kick it over to Dave mirror to kind of tell us a little bit more about compliance bits we're talking about today or let him introduce himself to but Yeah, yeah. I'll introduce myself and I'll hand it over to our esteemed guests as well. Who's one of our experts in compliance but yet. Hey everyone Dave mirror. I am on red hats strategic global alliances teams basically I'm a solution architect that focuses on our great security partners around things like open shift and Ansible. And we've created this monthly dev sec ops is the way series which I'll talk about a little bit here but before I do I wanted to pass it over to john Osborne to introduce himself. So hi everyone. My name is John Osborne chief architect for that public sector. My house is has some construction going on spear anything just just ignore that. But yes so games. Yes screams, you know, yeah. But yeah so I mainly focus on bringing a lot of commercial best practices to public sector so some of things around people and processes and technology as well so right now that kind of the big industry trend is kind of this push for platforms. I've never seen that a lot in public sector and you know there's a lot of ramifications, not only just technology wise but organizationally. And there's a lot of pieces around that and there's a lot going on usually these things get kind of lump together with things like agile and cloud adoption and modernization and all these other things that really what I do is try to help customers make sense of all that and kind of point them to a well known happy paths or at least as close to it as as there is so well known happy past. Yeah, I like that term better than the best practices. Yeah. Yeah. Yeah, I'm saying that there's a lot going on I think is an understatement john just trying to get a minute of your time is pretty rare so I think it's going to be a great discussion. Let me just throw up here. Something quick before you need to know how to share stuff. All right, here we go. So move our faces over. Yeah, so as I mentioned, I don't want to share I want to present it. As I mentioned, this is a show that's part of a monthly security series. This show specifically called DevSecOps is the way it's typically somewhere around the third week of every month. But we do two shows actually within open with OpenShift TV. And one for this month is actually tomorrow. We're talking with Cystig, one of our partners around compliance and that's how we're kind of breaking it up we've got two shows one more on thought leadership, which is what this show is about, and then another one featuring one of our partners. Alongside of that you can see we have three podcasts that typically drop every month. So what we've done is structured it in a way where there's a security category every month so we started this in March. It was all about vulnerability scanning as we launched a new vulnerability certification within Red Hat. Then we have compliance this month April's compliance month. And you can see the rest of the months there next month is around really identity access and management. We've always learned more we've got a site sort of around this and DevSecOps in our ecosystem. You can see it there red.ht DevSecOps, the D, the S and the O do need to be capitalized, or feel free to feel free to email us. If you've seen me talk before these categories may seem familiar because within our ecosystem within Red Hat we've created a framework of what we call DevSecOps methodologies and technologies. This framework has actually allowed us to understand and consume all the security features and functions and methods within DevOps and really promote that DevSecOps sort of mantra. You can see over there on the left all the categories there's nine different categories which correspond to our months, but then there's about 34 different methods I'm sure it's hard to see. But what that allows us to do is say, oh, this certain method, this certain technology should possibly be integrated at this step in the DevOps life cycle. And we could take what Red Hat does, we could take what our ecosystem does, our partners and bring together a comprehensive joint solution for our customers. So, you know, at the very least it lets our customers say, oh, I should probably think about static analysis or secrets management. What we're talking about today, compliance audit and compliance technical controls at certain stages in DevOps so that I can ensure to have security embedded, automated and not slow down, you know, your DevOps process. Right, so I think, oh, the other thing I just wanted to share real quick is, you know, here's some of our partners that we work with, we have a huge landscape of security partners, but you can see kind of where some of them fit. For example, compliance, which is this month's security topic. You'll notice there's folks like Aqua and Cystex, Synopsys, Tigera, Palo Alto, New Vector. You'll see blogs and the podcast, tomorrow's with Cystex around some of their capabilities as well. But this, you know, being Red Hat and being the platform provider on the bottom, which is secure by design, but that allows us to see the entire landscape of a DevOps life cycle with all these different partners. And then we can work with them, work with the industry to understand where they all fit. So that is just my initial spiel to start off the show. In case you haven't heard of what we're doing on a monthly basis. So I wanted to then transition down and have a conversation with John about, you know, his role a little bit more about what he does. Very interesting to understand. You're taking, you know, commercial practices, applying it to the public sector. But if you could kind of paint us a picture, John, of what you do overall like in a given day, I know you do a ton of stuff, but what are some of your main major activities for our customers? Yeah, really starting off with the hard questions. What do you do? You know, it's, it really varies by the day and I have a lot of different job functions, but, you know, my bias has always been to be really customer facing. So, you know, I cover a lot of ground because the public sector is very large right we've got state and local governments we have some are small some are really large and, you know, we have federal agencies who've got DoD and Intel and all these things and it kind of spans a lot of different types of things from hey I need to do this at the edge to hey I'm just getting pretty early. Started pretty early on at my cloud adoption journey. One of the biggest things we've seen is that as customers start to move into kind of newer ways of working and, you know, move into things like cloud. There's, there's a lot of confusion. You know, I think one of the biggest things we see is customers try to kind of lift and shift their on premise model into public cloud. They're not only just the technology but they're different ways of working. So which is a really big challenge right because cloud is just so much different so the, the way you do identity management is different the way you do not work means different the store just different. And then the way you kind of work around it right this whole idea of kind of the, the CapEx the OpEx piece is really interesting. And what I do is help customers with that make incremental progress. Try to change the mindset a little bit, because if you look at public sector a lot of the compliance and things that have been mandated by law are very static. In a lot of ways, they've been the same documents or risk management frameworks have been around for 1020 years right now. So, you know, try to get them to move into the DevSecOps is more of a mind change, set in changing the mindset a little bit because you're starting to look at, you know, even making incremental progress, you know, one of the things that we've seen kind of the failure modes. I think a lot of people, they've heard about different successes in public sector. You know there's a lot of ones that are very public that you've seen on, you know, Kube cons or, you know, other big conference events. And they think, hey, we can just kind of like take that blueprint and copy it and move it over to us and it really doesn't work that way right like DevOps in general is a lot about experimenting and trying to do things and making a small incremental progress and kind of iterating on that. So, moving that into the kind of big bang, you know, everything risk management is all done up front to kind of move into, let's get something out the door and focus on improving it over time is kind of one of the challenges. So where do you see the public sector's adoption of DevOps at the moment, I guess and compare that to commercial industry as well. Are they at the top of the curve, are they just starting? I'd say there is pockets where it's pretty far advanced but in general I'd say it's where a little bit behind the curve. And there's a lot of that because of the things I was just talking about where there is a, I would say a pretty big groundswell right now to get DevOps out the door. In a lot of cases a lot faster and kind of, you know, look at newer ways of working in newer technologies and things like that but in a lot of ways we're still kind of set up the old way where contracts and what's expected and project management is probably the biggest one where it's, it's very like if you go into public sector and you look at, you know, the way people do project management, a lot of it's, you know, faced around burn charts and Gantt charts and things like that and very kind of older ways of working and not as much focused on outcomes and just kind of building things small and incrementally and kind of, you know, pivoting, you know, things aren't working in that type of thing so it's a lot more through the groundswell is there but we're still making progress I think as we go. But customers are starting to make, you know, a lot of headway especially with platforms where they can, you know, bake in a lot of their policies and security pieces and let development teams have that kind of API driven handoff as opposed to more of the ticket driven handoff that we'd seen in the past where you know you wait three months to get a virtual machine and all that stuff so yeah so we're making headway but you know there's your mileage will vary depending on where you are. I think if you look at the traditional like Gene Kim's three ways, a lot of a lot of us are still in the first way, which is, you know, focusing on just getting things through a CISD pipeline. And in general, you know, if you say dead sec ops, I think people people initially you dev ops has been around for a while it's not a novel but the second you say dead sec ops people immediately focus on the sec part of that. They get they think you're initially talking about you that by default they think you're talking about, you know, baking security tools into sec pipeline and things like that so. But of course there's other ramifications in terms of engaging people earlier in the, you know, earlier in the life cycle and you know there's because of way contracts and everything else for setup, you know, I have had customers that have development teams that have never met anyone in ops, and they've never really met anyone in security So, you know, we're working on, we, and we've had a lot of good success just connecting people together, you know, some customers are so big they're asking me like what's the other teams doing inside, you know, their own inside the room organization so. But people have made a lot of success you know I think one of the sorry from this kind of being one window here but one of the things I think that's not that dev ops really needs to pivot to upstream is that. You know, we focus a lot on the Etsy's of the world and the Netflix and the people that are, you know, the 10s right they're doing this and it's so cool and they're experimenting all the time but there's so many people that have gone from like a three to a seven. Because they've engaged security people earlier in the life cycle and they've engaged ops people earlier in the life cycle. And no they're not Etsy, but they made a lot of progress and that's something that really should be probably celebrated and talked about more because most of us have these massive, you know, mainframes or absolute production for 30 years and just, you know, moving to a lot of what you read about with, you know, dev ops best practices or use cases or whatever you want to talk about. Isn't always isn't always that clear on how to get there and kind of incremental progress probably needs to be celebrated more. You just can't throw technology at at your pipeline and call it dev ops right I mean culture is such a huge part of it sounds exactly what you're saying about bringing people together. And that's one of the things we've been trying to say as well with dev sec ops is is getting those security people to actually just meet developers. Yeah, instead of instead of through email like hey here's an email you've got 15 vulnerabilities on this application go fix it right. Trying to get them involved. Although have you seen, I guess in the public sector, our security teams large is it tough for, you know, tough for those teams to integrate with dev ops teams is so is the, is the thinking trying to get more training with the developers around practices or what are you saying there. I would say that piece. It pretty much is in line with commercial in terms of it being, you know, the famous ratios it's like, you know, 100 or I'd say in practice maybe 31 or something like that in terms of developers and to security folks but engaging them earlier at least in terms of the planning and processing they're probably not going to come to all your meetings. But you know that that will go pretty far especially if you look at all the back and forth that happens afterwards where it's like, you know, I, you know the traditional way of doing dev ops, or even the general was like very much ad hoc in public sector after the fact like a built an app. I did all these things and oh wait now I need to make like a, you know, a spreadsheet with thousands of rows in it to kind of meet this come compliance framework. And, you know, public sector we've done I think a pretty good job of starting to automate those things. And there's been a lot of movement there and then just engaging the security people to make sure that you know whatever you're building is actually what they want to sign off on because, you know, a lot of, you know, at the end they're going they're going to sign off on the risk aspect of that. You know one of the other big things I think kind of the evolution of dev ops that we've seen in commercial to but definitely in public sector is that you know there was this kind of mindset maybe five years ago or so that, you know, securities is your job. And, you know, yes, it is but a lot of people are just really bad at it right so you know we also and that's really where platform comes in as we can. Yes, securities your job but you know that's not your day to day and nor are you an expertise in that so we can't. We have some guardrails for you and that's where kind of platform can can help do that and it gives you know the security and operations teams the, the peace of mind to know that developers can use what they want but they're doing it inside the confines of, you know, approved policies and security things that we, you know, know and love and accept and all that stuff so yeah. I mean, you can only train a developer insecurity principles so much. Yeah, I mean it's just not their expertise they're good at building stuff and being create, you know, creative, but that's why you have like guardrails and controls and things like that to help me out. There's, there's so many pieces that to I mean if you think about even like the shared security model in a public cloud provider right you know I think a lot of people are still trying to wrap their, their head around that a little bit in terms of you know what what they're responsible for versus what the clouds responsible for. There's a lot of knobs you end up having to turn those knobs look different on different cloud providers. So there's there's some confusion there, but having a platform is a really good kind of way to abstract some of that as well so if you're kind of speaking the same language whether it be Kubernetes or Ansible or you know whatever the platform whatever the language is of your platform. And you're doing that on prem well when you move to public cloud just having that kind of similar language can can help I think reduce some of the complexity and cognitive overhead of kind of moving and using different environments and things like that. Yeah. Yeah, so let's let's chat about regulatory compliance and Yeah, here come the acronyms. Real sexy. Well, tell us, I guess, and I know you know regulatory compliance, some people cringe at it but it was, it was a way for companies to be able to, you know, meet guidelines easier, instead of sort of guessing and having to do hundreds of tests over and over and over and then they can just say hey I'm PCI compliant or whatever and then that would be a common standard for everybody to talk to what are what are some of the more popular compliance frameworks that that you see in the public sector. Yeah, it's interesting. I think. So right now, there are a few and I'll talk a little in a second but we've we've seen actually a lot of commercial companies using public sector frameworks as well so I don't think public sector frameworks are just for public sector. You know, even I've had, you know, I've talked to car companies and in Germany and other Sweden other in other places where you know they want to sell to the US so they have to say that they're going to use some of these certain standards and so forth. You know, the biggest one that's been around for about 15 years now is there was a law passed called FISMA FISMA Act that standardized this kind of risk management framework. And it was largely based on on prem because it was 15 years ago. Right. And so, you know, there was, it goes very much up and down entire stack from, you know, your application and encryption to, you know, how am I physically protecting, you know, my data center for smashing graph types and areas. And so about nine years ago or 10 years ago there was they passed a Fed ramp. So if you're if you're familiar with Fed ramp. That's basically risk management framework for for public cloud providers. And so that is awesome because, you know, with with on prem, you go through this whole or public cloud you go through this whole ato process if you've heard of the term ato it's authority to operate it basically says that your system can have production data in it. And so with Fed ramp that basically says, Hey, if you build your app in the public cloud and it's fed ramped. Well then, you know, you don't have to worry about the smashing grab you don't have with the with the disk or any other service that you're running on that current, you only have to go through the ato process for what you're building on top of it. So that reduced the, the, the lift for, for a lot of, for a lot of people in public sector so that's that's been helpful. A lot of other things that you've heard about like you probably heard of stigs like the operating system has a stick or your middleware platform has a state. There's all the dis stuff and, and flips, and there's, they've released, I got to work on the Kubernetes SRG. That's kind of like the upstream for the stick and there's all sorts of stuff that's happened. A lot of those just kind of roll into these were kind of to risk management frameworks and processes. So we've seen things like CS, you know, just like commercial companies are using. In a lot of cases, what comes out of public sector we've seen vice versa so we've seen kind of the CS Kubernetes benchmarks and other things come down and now into public sector people are asking about these other other standards and things too so there's a lot that's out there. But, you know, the biggest one that's mandated is, is, is FISMA and then, and then, which is that kind of the missed RMF. And then, after that, you know, Fed ramp for, for people using public cloud. So, how do those audits usually work. Is there an external party who comes and collects, you know, your documentation things like that or how to, I guess, how does that process work. Yeah, so that the normal ATO process, you have a security officer who's going to basically at the end they're going to accept risk and sign an ATO and the whole process is a lot of paperwork generation, you know, it's how and for what vulnerabilities you have in, you know, thousands of lines in a spreadsheet over, you know, all these different controls and knobs you have to turn and all that so traditionally, that process has taken anywhere from 2000 to 10,000 people hours to get that out the door. Yeah, so it's been massive, you know, sometimes you build something it's taken, you know, maybe up to another year to get it through the ATO process. So we've been working on for a while on ways to automate a lot of those checks. We've used some of security, there's been all sorts of kind of fancy terms that people thrown around like ATO in a day and other stuff but really these things have been just a way to automate a lot of the paperwork process and a lot of the controls that have been turned with open shift right now. One of the cool things that we shipped was this compliance operator, which will make sure that you're already, you know, basically your OS is stigged or gone through the CIS Kubernetes benchmark and other things by default so you don't have to worry about necessarily turning all the knobs anymore. It's a lot of engaging people and then maybe some documentation around it so there's still a lot of work to be done but I'd say every year we make pretty good incremental progress to making that lift a lot lighter for people. There's a people process that's definitely one of the things as well so I mean I've seen ATO is going through in a day, but I've also seen them take two years at rate because it's people so it's a, you know, it's a process that you're supposed to kind of accept the risk but you know some people risk is not can be subjective Yeah so where do you see the most automation is I can imagine there's obviously some automation you can do in the build process but that's just not the only point right. Yeah, so you know I think once we actually saw even going back to before Kubernetes had fully went out you know it's going back really into the Docker days I think one of the reasons why Docker became so popular was this whole idea of building apis on top of Linux containers and and you know you ship out your environment and it was your your container image had you know your application and all the configuration that needs and the environment variables it needs to run and so that was pretty powerful in itself and then you know so I think once customers started looking at just containers they thought hey this is actually a good framework for me to improve on my existing CICD process which is like there's a bunch of plugins and you know a bunch of bash and to be honest right maybe a couple Jenkins plugins and so there's been this kind of push for a while to automate some of that. So usually the first step we see is, you know, if they're going down the container path, it's no container image scanning. There's also good, good tools out there, stack rocks twist lock and core sys tag, and so forth. You know, that whole space has been very noisy and also, you know, if we're also being honest, a lot of those scanning tools produce different results. And so I think kind of the next iteration of that is, I want to redundant scanning tools so in a lot of cases customers have to scanning tools. One of those might be something from red hat so I mean we scan our own base images but not the apps on top and that's really where all of the, like you came from black duck rate which is now synopsis right and they, they do a great job scanning open source components that you would pile on on top of, you know our base images and so forth. But then there's a lot of customers will have to do static analysis tools. So this is developments done by contractors in the public sector, but some of the operations teams might be guffees of the security people are always guffees so, you know, they want more checks in terms of hey make sure the developers are don't have open ports aren't doing those types of things so there is quite a bit of static analysis that's done. So I'm just kind of iterating on that right, you know then customers might look towards the runtime. Once they have all the, they have the CI CD part down, then they look towards the runtime. I think that the challenge for a while and still kind of exists is that a lot of the tools that we always use to do things in virtual machines isn't really set up in a lot of cases to use those same tools and public cloud so you know a lot of scanning tools and companies have pivoted. So it's going to be different for every every solution but you know in the early days it was like we have to run this tool and that tool didn't do anything. So like you scan it and it's like hey it's got zero findings and it's like well actually couldn't even the container images and even mountain to the root file system so it didn't even see anything but okay. Well we had to do it so it looks good. And so, you know you you have kind of that evolution but you know now we see a lot of, you know you put up the side of the beginning a lot of the partners now have operators where you can install it and it's going to run as a demon set and those types of things. And that's been great and there's there's been a push and we did a lot of edge stuff in public sector as well so in some cases customers are, you know, not even looking to run just Kubernetes they're running to it they're going to run Ansible they're going to run real for edge or pod man. Just want to run a few containers in an edge scenario. And of course edges different for everybody to rate some edges, you know, can run a full blown cluster with some of these pizza boxes they make now are crazy you can have like a, you probably put 20 no cluster around like two pizza boxes because some of them just come so. So, they always say more is a lot is going to run out but it never never seems to do so but. Yeah, that's kind of the iteration of is the scanning redundant scanning static analysis, then looking to do some runtime stuff on top of it. The networking component as well as this is pretty important you know people want to make sure that limiting the blast radius. Because we're using a platform. A lot of times it's multi tenant. Usually not a ton of single tenant use cases in public sector, but you know at scale to we probably want. I've been pushing for multi multi tenants. So we're not getting too big with, you know, having 20 different projects on a platform. We're kind of breaking things off into kind of a middle ground where it's, it's multi tenant but you know we're not going to all move to the slowest pace of the slowest tenant on the on the platform so. That's, that's mostly the evolution of it. There's obviously organizational things that need to change around that so contracting the boring stuff is really kind of, or we're at still with some of these things. Yeah. Yeah, you're right by the way, around scanning and the vendors coming up with different results. It's more of an art. It is I mean, you know, I don't blame them if you look at where they're getting their data sources from they're like hey we got this one came off the Python notes page from like 2014 and you know this one came straight from the vendor is like a vulnerability and you know they just get their data sources from all over the internet right and so obviously, you know when you do that it's not like everyone's looking at the same authoritative authority. In other cases, you know what vulnerabilities and then Apache Commons library, you know, who's the authoritative source for that. You know, I mean, sir. Even red hat has probably five different sources where you can pull security information from that. And that's one of the reasons or that's the reason why we created that vulnerability certification program for those partners. Last month's topic, we talked about this where, you know, there's a set of requirements for these scanning partners to to consume the right security feed, and also display red hat data alongside their data on on red hat packages but it's sort of first in industry we're hoping the other distros follow suit, which would make results a lot more. A lot similar between scanners. Yeah, as you know customers will look at a scanning report and then they'll look at what red hat says about it and and it's like wait a minute they're totally two different things. And that's actually, I'm glad you mentioned that because that's actually I had, you know, maybe I just put that in the back of my mind because I think a lot of scars issue from that issue but for those that aren't familiar, you know, I think what Dave's touching on is the way a lot of scanning tools work is they just they look upstream to where there's, you know what versions have a have this have a problem, and then they do an RPM check or something inside of a container and realize how well you're using, you know, Python, you know, or, you know, Python seven and you have this vulnerability, and that works for every vendor except the red hat because red hats like the only vendor that backports in a lot of cases fixes into older versions. And so that causes a lot of problems where scan tools will say okay well, you know you have all these, you have all these findings and it's like well now you actually have to ask red hat for what's a finding because we're back porting stuff and, you know, you're not going to, it's not the same as the upstream So that's been that's been a challenge so there's definitely kind of an educational aspect to that as well so we've seen, we've seen that in the oval B2 piece for people who aren't paying attention so you know, we've had this old feed for a long time which basically tells you just what you need to fix. And if there's a patch it tells you that you know you need to go patch your system and you know we put out this be to feed to help clarify some of that those those pieces as well as more like an enhanced feed to kind of solve some of these components upstream but Yeah, in that version I guess the version two came out with a couple years ago two three years ago, they're continuously updating it. In fact, they, I think just today they released an enhancement that will help our scanning vendors consume what what they're calling the unpatched vulnerabilities so not only does oval put out fixes well here's the fix but they also say here's what's currently affecting and that's not patched either you know it's affected or we decided not to fix it because it you know it's there's no impact or another reason that it's still a vulnerability So yeah the teams are continuously updating that based on a lot of the feedback we got from the certification program which is nice to see and and so I see I see a light at the end of the tunnel, you know, for these for these scam results but it's been good so far we've got three partners certified and about five more working towards certification is not easy. Yeah, these scanning vendors it hasn't been easy for red hat to to make it work but it but it's worth it. Right. Yeah, it's been a challenge for customers because there's there's a lot when I started off at these things so there's a lot going on and there are because, you know, a lot of times people look to adopt technologies like Kubernetes and like, you know, Ansible or whatever is in conjunction with this kind of public cloud piece which I talked about is, you know, a lot of things are different there. And then they're like hey it's also a good time to let's fix our CICD problem and these other things and a lot of these components have noise and they're all learning and getting incrementally started on it and and getting something out the door is is you know what do you focus on first is kind of the is the challenge. Yeah, so yeah I mean, and there's questions in chat that are coming through right like there's just one tool that Netflix Netflix released called risk want and like you know where do you get started right like how do you even start assessing risk. Yeah, one of the questions in chat like how do you even open the door to beginning this process. Yeah, it's hard I think and that's actually in general I think regardless which risk management framework or compliance in general is, you know, compliance is is almost like the minimum table stakes to me. It's great, but that the failure mode is when you stop looking at it as a way to kind of think independently about what you're doing. You know DevSecOps is a tool, I think it's a strong tool, but you know it's we're trying to solve larger business outcomes with it. And so you need to work work backwards from that in a lot of cases. And, you know, if you if you have a bunch of vulnerabilities it's, you know, on a system that's on an embedded system and it's not even connected to anything. You know it's on you probably heard of like Windows 95 is on a fighter jet or something like that right and it's like okay well I'm sure there's all sorts of patches and vulnerabilities that are out there but you can't access it unless you physically touch the board right so you know it's not like you have a Windows 95 machine exposed to the internet so you know compliance framework we'll see that's the biggest travesty of all time, maybe it is but you know you need to kind of think that you know it's on an embedded system. There's a lot of independent thought and risk management piece around that that I think you know you have to kind of wrap your arms around and so compliance is such a heavy lift that I think by the time people are done with it they think, okay well this took such a long time and it's such effort this we're done. And really it should be automated constant. Yeah, right. Absolutely. And you know especially with a lot of risk management frameworks that's why we don't upgrade pieces either. Right we get a risk management framework out the door. And so I've seen a lot a lot and some very scary scenarios like very serious trials and things like that right like absolutely we can't upgrade windows and T for because if we do we got to go through the entire like certification process. Yeah, the whole certification process. Like, wait you're running an NT for VM. Yeah. I mean what we're looking at right now and I think what missed and some of these other bodies are looking at right now is, you know, past, you know the first way rate, moving into the second way, and the third way, and using kind of GeneKIMS three ways are getting more feedback and moving into more new learning and so forth is you talk about the automations can be now more automation of the compliance, just not just the scanning but all the documentation pieces around that like what is your architecture look like. Well, that's great. You had the architectural diagrams from the first day but we know how production systems go over time right there's a lot of toothpicks and inductive that ends up pulling the systems together over time right and so are your architecture diagrams up to date can you do those, then kind of moving into the monitoring and observability aspect to it so I think a lot of the kind of forward leaning, you know, security people in public sector are saying, Well, if you can automate all your documentation and automate these other pieces, and, and you get to the security with your monitoring and metrics and, you know, you have some of the third party tools like maybe a twisted or anchor or black duck doing scanning and maybe like a cystic doing all the runtime stuff whatever it may be. You know, then you can go ahead and move towards you know what you hear like a continuous ATO or something like that where you can update all these components nonstop and you're not necessarily worried about a point in time. ATO with documentation from last year with certain versions that are static. Now you kind of moved into this right now I can iterate really fast once you get into kind of the second way that's really more optimized for developers where they can start pushing their code. You know, a platform has all the policies and scanning and security pieces around it that you can move at the speed of or the tempo that you need for your organization that's we are seeing some of that now which is pretty exciting but let's take on way to get there and of course it's still a lot of ways to go to so. Yeah, I want to sort of go back to mentioned open shift and some of the features like the compliance operator and fits enabled right so Are you seeing a lot of time saved in terms of ensuring like a Kubernetes distribution is is compliant just because open shift out of the box by default has some of these things already enabled. I'd say the biggest thing is it gives people a warm and fuzzy feeling right so you know they, you know they they know Stiggs they know the OS. And if you tell them, you know that red hat core west is it's immutable it's, you know, treated like an appliance to same bits as as relate the latest version is the same bits as relate three, and you know we can give you your stick in an automated way. Well, that's a huge lift for the customer. If you talk about, you know, undifferentiated heavy lifting you know it's in toil and those things that get thrown around a lot. You know for for compliance a lot of that's just incredibly tedious. You know permit root login equals no and you know I got to change this file and I got to do this and yeah all that I remember those days. Yeah, by hand. Yeah. Yeah, I'll do all those days by hand. And so, you know, we've, we felt that and so that removes a lot of a toil for customers. Yeah, you know things like, you know, you got SC Linux and other things that are built into the platform by default but now you're also going to automate a lot of the stuff that they're going to have to do no matter what you know technology that they chose because they have to do this compliance framework so that's a pretty big, a pretty big thing for customers. And it's been, it's been one of the more exciting things about we still, as far as globally geos go public sectors, it's, you know, I think percentage wise we still have quite a bit of customers on on three, but for the customers that have moved to for, you know, the, the feedback we've gotten has been pretty amazing around, you know that not having to manage the OS and have it be more of an implementation detail. And having some of the compliance pieces is a little newer, but you know just in general having, you know, a lot more of the undifferentiated heavy lifting and we just talked to a customer that, you know is running one of the largest apps that you know, I think it's actually the largest app the organization and it's a financial app and it's very public a lot of people that are watching and probably heard this, this app especially if you've ever, you know, looked at, you know, public filings and things like that for for public companies but that's all running on a cluster and they've got like two, they've got like two admins running this like really large cluster because that's a lot of it's just kind of built a lot of the undifferentiated heavy lifting is built into the platform and now they're, you know, going to be using things like the compliance operator to automate a lot of the rest of it so it's definitely exciting. We still a lot of customers on three and we're trying to, you know, do a lot of the, there's quite a bit of, you know, social engineering and architectural stuff and education that required. So we're still doing a lot of that but the customers that have moved to four or new or new customers, the feedback's been pretty incredible so far. It's good stuff. And you, you mentioned other red ad products right how are, how are some other red ad products involved in compliance auditing compliance controls outside of OpenShift. Historically, you know, in terms of a lot of the stakes and so forth. It's been, you know, Red Hat Enterprise Linux and then and then Jbos CAP in the middle or side. But of course, you know, we've iterated over time for that and now we've got states for Ansible, which has become just incredibly popular because, you know, it's, it's a lot of it's like the glue for enterprise right so you've got it can talk to almost any system and in a lot of cases it's become kind of a de facto standard to talk to things like storage and networking devices and things like that. And then, you know, for a lot of customers that are looking to do more things on it on, you know, at the edge, where, you know, they're not necessarily going to have telemetry and enabled and all those other things and that's kind of a smaller form footprint. And they're looking at just using rel and they might have OpenShift at the at the home base rate and they might have a CSD factory there but the container image it's going to get spun out is actually just going to run in a plain old rel instance, you know, and we're looking at rel for Edge, which is, which is pretty cool. And it might just get started up by plain old system D and Podman. And, and, you know, we've seen that kind of Ansible automating some of that as well. So that's been kind of a cool use case as well. But we're also using, you know, the D is working on some smaller footprint OpenShift. So I think, you know, pretty, pretty soon we'll be able to give people kind of both options which be be pretty cool as well. Awesome. Very nice. Yeah, no, the compliance operator is really awesome. And folks are asking, oh my gosh, can I get this working at, you know, in, you know, my bespoke Kubernetes cluster over here and it's kind of like, well, the docs say, sorry, this only works with, you know, our costs but in theory, if there was rules for it and we built them out, you know, we could have it for rel, we could have it for, you know, insert Linux distro here kind of deal. And I feel like a lot of the compliance bits around Kubernetes are like so new and are going to have such a hard time keeping up. How do you think that's going to go in the future? Yeah, I think that's a challenge because you absolutely have to kind of have somebody that's evolving with, you know, with the technology. And so if you look at, you know, Kubernetes is on a time box release cycle. So every three months it ships out a new version. And yeah, there's been changes in Kubernetes. If you look at, you know, ephemeral containers and yeah, and these other things and there's a security implication with running new software. Right. And so, and a lot of times like a Kubernetes benchmark might, and you might be looking at the older version or, you know, when a new version of Kubernetes comes out, you know, the Kubernetes benchmark might not come out right with that. So I think having a, having a vendor, and this is probably a shameless plug, but I think having a vendor like Red Hat that's got a kind of strong security DNA to vet a lot of those things and, you know, we're going to probably shield you from a lot of that noise. And then also, you know, we have multiple layers of protection turned onto the platform of things like SE Linux and security context, which is, which has been really powerful and what is SE Linux has put all those big, bad, nasty, gnarly Kubernetes bugs and SE Linux just stopped them in their tracks for the most part, right? Yes, I've got this, I think I got it from Dan Walsh, but there's been, you know, essentially five CDEs for container breakouts and SE Linux has stopped all five of them. So, you know, it's all about multiple layers of protection to us, but you know, SE Linux is a really strong layer of protection. And, you know, even just having that managed for you by the platform is really strong because, you know, historically people have turned off SE Linux, even in environments where they really shouldn't be because, you know, we're being honest, it's hard to use. But, you know, the good thing is, you don't have to know how to use it with, you know, with a platform, it's just kind of shipping it for you and tuning it for you and things like that. You know, that's, there's a lot of value add from there just in terms of giving people more secure out of the box profiles, but then, you know, just avoiding some of the undifferentiated heavy lifting, not just technology wise, but security tuning and compliance framework wise. So, I mean, it's, it's a bear to manage on just an OS right like when there's other stuff running on top of it. Yeah, yeah, it's, it's interesting, but we all have to remember that like kernel components make up containers. SE Linux protects kernel components. So it's, it's one of those things where it's like there's a happy relationship there. Yeah, I think people always got confused because the original, you know, the original Docker container architectural diagrams I went around forever had, you know, the box was like here's Linux and then here's my container on top of it. And, you know, I think people will, it made sense for architecture diagram but you know I think people got confused with thinking that it was something just completely different than, than Linux at first but yeah it's these things are all just kind of enabled by the platform, you know, that you're running on in a lot of cases. And I was going to say as a, as a Christmas gift for all your viewers, Chris December is platform month for security. So we're going to be hopefully diving into things like SE Linux and open shift platform security, which I think will be very cool. Yeah, as we go along. What is red hat offer kind of out of the box as far as help securing clusters right like, I know we try to be Switzerland I know we try to do a lot of, you know, neutral things but do can we actually help people secure things other than open shift. I mean I know we can, but do we is that a standard practice of ours I guess. Are you asking from a compliance perspective. Yeah, from just just from a compliance perspective right like if someone has, let's say you know, four different clouds are running on chances are they're going to have some bespoke thing, or some cloud service that they're trying to work with. And there's a compliance layer there as well right. How do we manage that. Like if I'm using like RDS for example and Amazon with my open shift cluster. How do I, you know, say okay yeah everything's good with AWS and open shift. Does that compliance operator know AWS to an extent to where it's like yeah this is misconfigured try this instead kind of deal or I'm not sure john you might. Yeah, so I mean, I think you know we will first you know we do have a consulting team. That's very strong. And in a lot of cases, you know their main their main focus is not necessarily end to end security around everything you're doing in your enterprise, but it's their main focus is how can I build this in the most secure way in the context of red hat products, because anything outside the scope of that becomes just too, yeah too unwieldy and probably unrealistic right and so they know how to make our products secure and they know how to make our products secure in the environment that they're in. But once you start going out with every you know there's what like 300 Amazon services to Google cloud services so now that they're not an expert in everything but you know they know how to put things in context there and then, you know, our consulting you know a little bit more set up to be you know an accelerator for you or for other groups you're working with as well so a lot of times you know for working with customers. There's other traditional size like Raytheon or Deloitte or or whomever right and so you know I say I see an extension and Lockheed and all these all these people so and a lot of cases were enhancing other people that are doing those things as well with kind of hey this is what we've seen this is a best practice, but you know for the customers, yeah having our consulting teams awesome because not only have they done this a million other places, what they have reached back to all the engineers at red hat and other things that you wouldn't have access to otherwise so that becomes pretty valuable and then of course like internally we have some lists and all sorts of internal shared documentation and other things like that so that's been really helpful but you know they build a lot of their source solutions for things right out in the open so red hat consulting has their own GitHub repo and stuff that's actually got some pretty cool stuff on it. That's worth plugging around and one of the other things that they've been building is this. I can't believe now I'm just now talking about is this kind of software factory piece called Redsword. So it's in the Department of Defense they had built out this kind of reference designed for DevSecOps a while back and they're working on a version two of it right now. But red hat built out this open source and one of the things that we're seeing was customers you know in addition to all the compliance components was they're spending a lot of times not focused on their app. And then back to they moved to Kubernetes initially because they'd be writing all this glue code right and so you think about the DevOps engineer a lot of times writing like blue code or doing all these other things especially initially and you know that's a lot of what what it was was like how do I take all these scanning tools and components and kind of plug them into Jenkins and or Tecton or wherever they're going to be using as their backbone and so one of the things that we did was well you know red hat actually hired people out of pocket. And it helped build this kind of software factory component by default where you know either you're going to use it or you can use pieces of it, you know it's all public something to put that there's a show notes or something. You can put that in but it's one of the things that we've been working on and we're engaged with I think for customers right now with with having them leverage that so you know yes you can build in a lot of the scanning pieces and all the static security tool and all the glue code and stuff that you would need around it. And I think it's a great approach to kind of building up all that yourself because you know we want customers to fill up the platform with apps and we want them to, you know, finish their, you know fulfill their mission capabilities and not just worried about all this kind of oil and glue code and other stuff that documentation that people have done historically in the past and non value at stuff. Well, so I had a question for Janet thinking about compliance in the pipeline we've talked about different integration points the ICD we've talked about scanning a running container but if we think about compliance technical controls during that pipeline where have you seen those implemented have you seen admission controllers stop pods from starting because of compliance issues or, or maybe even as something's running maybe an automatic remediation type of control. We've seen those. I think a lot of it. A lot of it's like basic hygiene in the in the beginning so, and even in the middle stages so you know that's a lot of let's get network policy setup. I think of the traditional things you would do on premise with firewall rules and things like that where now you want to make sure that you're at least doing those in some sort of cloud native way so in having good role based access control identity management. You know people ask about things like certificate rotation. Those types of things network policies. It's not crazy stuff it's just like good hygiene things that you should be doing. And that's usually where, you know we spend the most, the most time is kind of setting those things up, you know some customers are just starting to look at newer paradigms and ways of get ops and, and things like that where, you know your workflows are actually starting to change built around those things but you know it's a, you know a lot of it is just kind of the good policy management and some of the more mature customers that have been around been doing this for a little bit or style to like policy around multi cluster provisioning and things like that where, you know they might be looking at, you know there's a, we have another tool called advanced cluster manager that does cluster provisioning and so forth so they're starting to look at those types of things but I would say it's an iteration, you know for, for at least like customers that are halfway through the maturity curve for now. Speaking of get ops, get ops guide to the galaxy starts top the next hour so please stick around for that if you're interested. Yeah, we've got what five, five more minutes left. Yeah, I'm actually all out of questions for john it's been a good discussion I know john if you had more you wanted to talk. I hadn't even looked at our notes actually. No, I think we covered everything. Yeah, nice. Yeah, let me throw this up again real quick. Those who didn't see it at the beginning of the show. So again this was one of our two open shift TV shows for the month we do this every month. This month is on compliance. The next show is actually tomorrow with cystic. Yeah, that'll be a good one. Well, we're also going to be producing three podcasts around compliance as well because the summit next week they might actually bleed into being dropped in May so just stay tuned but we've got a blog also being reviewed right now with some of the topics we talked about. And some other content as well that we're publishing but this is the monthly security series so for me that's a really appreciate john you joining us today for this thought leadership dead sec offices the way show. I appreciate everyone having me. Sorry, some of my responses tend to be long winded but you know we have a lot of fun here so I like to like anything I've been learning I was like to kind of throw it out. You know so it's not. One thing I don't do is hoard knowledge so. Yeah, we appreciate that. But if there's any questions in the chat or something. You know, not I'm not a lot of questions other than you know how to have what I do X with Y and I've answered most of them. Okay, so yeah, yeah. Yeah, I mean the biggest thing is reach out if you need us, you know we're here to help you if you're having compliance issues within your organization. What is now we can help absolutely Jay Osborne at redhat.com or open shift fed on Twitter so yeah, reach me anywhere. Yeah, thank you very much everybody appreciate it Dave I'll see you tomorrow. Yeah, thanks Chris. Thanks Dave great chat with you. Take care everyone.