 Hello, everybody, and welcome to another OpenShift Commons Briefing today. As we like to do on Fridays, we are going to have someone from the Global Transformation Office talking to us about things organizationally transformative or systems thinking. We have all kinds of topics that we do on Fridays. But today, I'm really thrilled to have with me again John Willis, who's going to talk about some DevSecOps topics, and his title today is The Blurred and Broken Lines of Defense. And I'm going to let John introduce himself, and we will have a live Q&A at the end of this. So please type your questions in the chat, and we will have a conversation afterwards. So take it away, John. Hey, thanks Diane. It's good to be back. It's been a little bit of a while before I've been on. I love the Friday OpenShift Commons Briefing. It's fun. Anyway, thank you all for being here. One of the things, for those of you who know me, for probably like five or six years, I do these sort of, I pick a theme near the end of one year, and I start thinking about like that's going to be, it's not the only presentation I do all year, but it's sort of the main, like any of the keynotes and things that I might get invited to. And so this is the first sort of attempt at this presentation. I'm calling The Blurred and Broken Lines of Defense. So and it sort of, as it turned out, it's turning out to be a little bit of a meta piece, too. So you'll see what I mean by that. But one of the things I would love is, you know, I'm a botchka loop, there's Twitter handle, or Jay Willis at Red Hat. I would love your feedback, right? Because this is my first version. You know, I think, you know, I've submitted this to SwampUp. I've looked at it, I've been accepted at SwampUp for Jay Frog, and probably DevOps Enterprise Summit. So I'd love to get any good, you know, really sort of any type of feedback, negative, positive, you could have done this, should have done that. So that would be very helpful. All right, let's get started. And so for those of you, you know, so quickly I waste too much time. There's a lot of information about me out there. You know, I'm old as dirt. So I've been doing a lot of stuff. Like 10 startups or 11 startups and 10 or 11 books, like I've lost a chance. But probably the most biggest claim to fame in sort of this space is I co-authored DevOps Handbook. I was co-author of a project called, it's an audio book, but it's called Beyond the Phoenix Pack with Gene Kim and then some, I think, really important working groups that I've been on. And I'll talk about these two, the green one, which is DevOps Automated Governance. We're about to start the second version of that. And something called the Automated Cloud Governance. And I'll talk about that. Anyway, I've done a whole bunch of startups, so I do play guitar too. So not really well, but I do play guitar. So here's the question. You know, when I said that, like, I went down, I had a very distinct sort of motivation for this presentation and I started building the presentation. I got sort of very sort of meta in the way I was thinking about where are we at? Like, so DevOps is, you know, give or take 10 years. So I think it's a little more in there. But, you know, 2019 was the 10-year anniversary officially from the first DevOps days event. You could argue that DevSecOps is about five years old. We've done some really good stuff, right? Like, I mean, like big time, like, in both of those spaces. And, you know, again, DevSecOps is lagging a little bit behind our sort of all the things we've done well in DevOps. But you know what? Let's face it, it's still a mess out there, right? I mean, that's the bottom line, right? Like, we're still struggling. We're now calling it digital transformation. And that's okay. But like, just these are sort of, like, we're having a hard time. That the complexity is growing faster than our ability reasonably. And that's a whole nother presentation. So then the question I was thinking about when I started writing this presentation, which is, and stay with me here, folks. What would DevSecOps, what would the conversation of DevSecOps as a phrase or a term or a meta movement be if DevOps never existed? And the reason I'm saying that is, have we made a whole lot of choices about security with the bias of what we accomplish with DevOps? And then, where are those the right choices? Right? You know, and, you know, we can talk all day long, even today. Most sort of hardened security people will say we're still running on, you know, 20, 30 year old security models, right? And, you know, in DevSecOps, we've done some very cool stuff, right? Like, we've sort of added automation. But in a lot of ways, I guess, you know, and I'm going to pick one particular area to sort of explore, which is the, we'll talk about the lines of defense. But I guess the question I'm adding is, like, yes, it's additive what we've done. You know, DevSecOps reference architecture is, you know, putting automation of things that were done manually outside of pipeline. Yes, yes, yes, yes, all great stuff. But are we significantly making things better from a security perspective? You know, I mean, I was going to post, like, all the sort of the security vulnerability and breaches that just happened within the last three days. And it was like it was going to fill up the screen. Anyway, so, you know, and then, you know, we put into sort of agile security and we have those conversations. So hold that thought for a second. And then I'll say, okay, let's go back and say, if I say that DevSecOps is five years old, Shannon Leitz overed into it. If you haven't followed her work, you should. She's brilliant. You know, she coined it, you know, I think before 2015, but this was a sort of seminal posting that she did, you know, and she's, you know, she owns this sort of coining of the DevSecOps term because she, you know, I was actually just, you know, honestly, I was talking the last night. I wanted to say, like, I showed it so a little bit of my presentation, like, if I could be, it's just nonsense. It's just good Shannon, you know, make sure I don't make an idiot of myself, right? And, you know, and I think one of the things she's great at is, you know, she sees the world, you know, way different than everybody else. And so if you go back in 2013, 14, she's in into it. She's trying to solve these really hard problems against adversaries and she sort of has this, you know, it sounds simple now, this epiphany of this idea of DevSecOps. At that time, it wasn't, you know, because nobody else thought of it. And the thing that's sort of interesting about that is, you know, that first sentence where she said, everybody, everyone is responsible for security, and that's 2015, right? And, you know, like, 2021, like, we really haven't solved this one. I mean, we have, we're doing a lot of things, but like, I, you know, I do a lot of interviews and I do this qualitative data analysis. Some of you might have seen my presentation, you know, the last one I did on this forum, where I interview hundreds of people in organizations and certainly talk to risk and I don't get the sense that we're, you know, we're like, you know, we're not even in the Bs, we're like C plus maybe, to be honest in this. And I don't say there aren't pockets of, like, excellence all over the place, but globally I think we're not. And then, you know, and I think we're still trying to figure out who should do security, right? Like, you know, one of the areas I spend a lot of time is something, you know, sort of modern governance or sort of a new branding of, or a new way to think about automated governance. And a lot of that stems for sort of a very siloed approach between risk, IT sec, sometimes the, you know, internal audit, which may be the same, not the same, and in dev and ops and infrastructure and who really owns this and how, you know, and so a little bit of more memory lane, you know, I've been fortunate enough to work in Gene Kim's organization. We write forum papers, you know, it's about 40, 50 us to get together every year. We normally go to the Portland. Unfortunately, last year and this year it has to be virtual. But we write these papers and there's been a number of sort of collaborative efforts over the years. The original one was 2015, Unlikely Union, Dev Ops and Audit. In 2018 it was a sort of tongue-in-cheeked ear auditors, which was an apology letter to auditors. But it's actually more than that, right? Because if you actually like, so maybe the first three pages is this apology letter, like how we screwed up. And then, you know, the next sort of 30 pages is a lot of control regs and trolls that we promised to be better at this one. Look at this. And then, you know, and then a project that is sort of near and dear to my heart, which is in 2019, I got a group of people together as part of that forum, Nike, Capital One, PNC, John Resotasi PNC, Top of All, Capital One, Courtney Kisser and Nike at that time. Saber, Mariah Carey, Dwayne Holmes and Mariah Carey. Like, there's some other people I'm forgetting, but and we tried to sort of proof out like how could we do audit efficacy, reduce audit efficacy in toil. And these are all creative commons. And I'll talk more about that in a little bit about what we did there. And then one other, I think, historical marker for me, thinking about this whole DevSecOps, for those of you who didn't know me, I was the only American at the first DevOps Days in Ghent back in 2019. Andrew, myself and Damon Edwards ran the first, and actually Mark Hinkel, ran the first DevOps Days in the U.S. together. We collaborated on that. So I've been in the DevOps game for quite a while, but you know, the DevSecOps thing sort of started sort of ignoring at me about 2014, 15, you know, about like, oh my God, we forgot security. Like how could we make that mistake? Like, and Topo Powell had written this article in 2017, Capital One, about how they did pipeline sign, right, creating better pipelines. And they called it, it's kind of funny for you, sort of geeks out there. They said the 10 gates that they had set up, which is hex, right? So it was 16 gates, right? But the, you know, and they said that, so the thing that you had to read between the lines was, they would say, they would tell the organization, hey, you know, you have to, they weren't telling the organization, you have to do this, you have to do this, you have to do this, and then sort of fight that battle. They were basically saying, if you could show up with your service or application or code or whatever, with these 16 sort of, the ability to show evidence of these 16 things, then you will get certain privileges in how you deliver your software. Like one, for example, you might not have to go to a Wednesday cab meeting, right? And at that time, you know, I was hanging out with Topo and we were having a lot of discussions about, like, what could we apply a blockchain model to this? And I think we, you know, the industry found out really quick, like blockchain is probably not a great model. And like, what was really lacking here is, what the industry was doing great is gating a lot of things. Like, you know, everybody was sort of maturing very well on how do we stop the build if you don't like, you know, if it doesn't come from source control or if there's not a pairing on a pull request or whatever. And then also if it doesn't have, you know, a clean build or even furthermore in the DevSecOps discussion, does it, you know, does it pass a vulnerability scan? Does it DAST and, you know, SAS DAST and in whatever, you know, software composition analysis, all those things, right? And that's a reasonably advanced model. And I would still argue it was still not changing anything fundamentally there, but we were doing good there, but like the real, another question that was creeping up is back to the audit question. Were we reducing any toil for internal auditors? And were we reducing it all when it came to, you know, control regulations or, you know, GRC, if you will. And were we improving the efficacy of what we were actually capturing? And so the discussion came is like, if we're going to capture this stuff for Gates anyway, why couldn't we just be building some sort of attestation, digitally signed attestation stuff? I'll come back to that a little bit. And so the two projects that I've worked pretty heavily on over the last few years is, you know, one called the DevOps Automated Governance that I talked about or about the second version of that. And there's some really fantastic stuff that has been designed and developed since, you know, the first paper in 2019. A couple of sort of organizations have really taken this pretty far. We're doing some really cool stuff in Red Hat with it. So I'll show you that. And then in another group, I got invited to the beginning of last year, which was a bunch of large sort of healthcare, health care, health care and financial organizations that wanted to do something around cloud loosely related to the work that we did in this paper. So I chaired something called the Automated Cloud Governance. And I'll talk a little bit about that too. So that actually, we're about midway through the second working group on that. So really interesting stuff. And I'll put it in context as we move. And, you know, one of the things that, as we think about the sort of the problem statement of like in today's world with all of the complexity that just grows and grows and grows, this is more cloud focused, but this is in general, this is a slide from Onug, the group that the second working group. It's the largest networking user group. These are the guys that were heavily involved in SDN definitions, SD-WAN, and now doing a lot of stuff in sort of DevOps and in this case, cloud governance. But the idea that like the complexity of what, like what we get hit with every day from sort of, you know, just in a cloud world, right, is overwhelming and at any point, it's sort of a moving target of what your minimal viral security posture is. And so, you know, again, in this reflective mode that I was in when I was trying to put this thing together is, you know, traditional security models are in any pattern, right, that's why sort of even the, sort of the hardened sort of security people laugh at themselves. They're like, yeah, you know, I was just having a conversation with a friend of mine this morning who was like a security guru, you know, like 15 years ago, right? Like insane, like, you know, all kernel stuff, right? And then really just gone into a lot of other stuff, like, you know, sort of development and was pulled back into a deep security conversation and he started off with apologizing that, hey, you know, I haven't really done anything in security for 15 years and they're like, you hadn't missed anything, buddy. I had a little tongue-in-cheek, but, but again, I think the sort of question then is, yeah, I mean, DevSecOps reference architecture is cool, right, like, I mean, like, you know, automating a lot of stuff we did decoupled and manually are now sort of in pipelines, which is good, you know, sort of our SaaS, that's automated pen testing. But then, you know, the real question is, is all this, you know, how much, how, Cheneleast uses a term called secure ability, like that this should be an ability. And can we ask the question, are we actually more secure? And I think, you know, I mean, like, some presentations, I try to scare everybody to death and show you all the stuff, right? I'm not gonna do that this time, but like, take it as default, it's pretty freaking scary, you know, you know, the complexity of the problem we're trying to solve and the intelligence of the adversaries, and I'll talk about that a little bit. All right, so that's a given, and, you know, are we, you know, did DevSecOps like really fix this? Are we, I guess the sort of the main point I'm trying to make here, which is have we force fitted, remember I said earlier, what would DevSecOps look like if there was no such thing as DevOps? Would we have made all the right, the same decisions about how we are trying to deconstruct or maybe just inherent as is security models, as opposed to like what Shannon tried to do originally in 2015, which is break it, blow everything up. In fact, you know, some of the stuff that she's gonna be publishing shortly, which is adversary analysis and adversary intelligence, she's gonna blow it up again, right? That's for another presentation. So back to the sort of what is the title of this thing, the blurred and broken lines of defense. So one of the things I hear a lot from clients when I come in, least clients that are sort of, at least clients that are sort of in this DevOps, DevSecOps, IT, ITSec, and are savvy about the sort of constraints of their control regulations and audit and internal audit. And they'll talk a lot about sort of the three lines of defense model, right? And so this idea, if you've never seen this before, right, like there's like the first line are sort of the control owners. And again, like don't scream at me. It's like, you know, like the IIA or the ISAC, you know, you wanna repeat devours and go argue with somebody else. The second line is really sort of control owners. And so the original three lines of defense, right, which was, you know, they're sort of based on, like there was sort of, it's the financial control, the security control, the quality control, right? Like you can already see problems here if you're sort of thinking DevOps. And then you have this third line was internal audit. So like, what's wrong with this picture? And for those of you who were screaming out and saying it's been updated, you know, the IA is updated, I'll get to that slide in a second. But it doesn't look very DevOps. In fact, it looks very similar to that walls of confusion. You know, it's, you know, it's, you know, everybody sort of has their piece in this almost throw it over the wall, right? Which this is sort of the dev, you know, space, space, space ops, pre-DevOps discussion, right? You know, and so, you know, and so I was talking to somebody else about this, you know, it's great to have smart friends, right? You know, about this whole thing about three lines of defense and they, you know, sort of brought something that should have been obvious to me that wasn't, which is the whole Conway's law. And I like, I started thinking like maybe there's sort of a new, it's called the called the Taylor Sloan Conway law, right? Like, you know, and I won't go in front of Taylor and Sloan and, you know, this command and control structure. But, you know, that if you're familiar with the sort of obviously overused in presentations, but sorry, because I think it, the reason probably is overused in presentations because it actually, it helps us define some of the constraints we have in modernization opportunities, right? So the Conway's sort of definitions, any organization, the design system, the cruise design structure is a copy of the organization communication structure, right? You know, there was the sort of original sort of added instead. If you have three groups working on a compiler, you're going to get a three-quest compiler. Well, you know, what do you get here? You know, three lines of defense structure, right? And then, and like I said earlier, like, you know, in all honesty, like, like this isn't like cave men mentality, you know, they've like in 2020, the IAA has updated this to be sort of more modern, but it's still a solid approach, right? So okay, so there's a governing body, which is awesome. It actually got rid of the defense to sort of promote a more proactive posture as opposed to everything sort of wait to it, like it happened, and then we got to deal with it. But again, like I will argue that, you know, they sort of collapse, you know, the first line and second line and sort of this sort of quasi, you know, you can say, okay, give them the benefit of doubt that it isn't siloed there. I would argue it still is, but you are definitely siloed from internal audit. And by the way, that's one of the biggest problems we have is the miscommunication, non-dialogative opportunities that we don't have with internal audit or risk when it comes to IT, right? That's why we have thousands of control regs that nobody understands. And I was, you know, I wanted to test check myself. I won't name the organization, but it's like a well-known big four, you know, consulting company who was heavily involved in sort of external, internal audits and all that. And in their sort of late 2020, you know, sort of the banking regulatory observations, you know, part of the response to the financial sector, ISAC, basically, you know, I'll try to read this quick, but, you know, another common issue is having, so your capability is missing or encountered, basically what you're saying, I read it, you know, the four example, if the business and a first line of defense does not sort of still using defense, right? Even though the IA has changed that sort of nomenclature, is not testing capabilities, risk and the client's functions to the second line might perform testing. However, if the second line, does that anybody remind you of like where sort of early DevOps or pre-DevOps was about QA and development, right? Like this idea like the next one will catch it, right? Like it's just not like, you know, and like this is the sort of common mindset that like we're trying to sort of force fit into the, oh yeah, three lines of defense, yeah, it fits perfectly with, and I'm not saying this is terrible stuff. I'm just saying I'm not sure that, you know, we're actually sort of thinking about the sort of the security problem the right way. We're having an honest conversation, and so I've been, you know, I've been doing this qualitative data analysis stuff for quite a few years now where I basically interview hundreds of people in our organization and I have this sort of presentations I've done in days gone by called the Seven Deadly Sins of DevOps. And so I'm thinking a lot about like, you know, and you know, to me like it all boils down to Seven Deadly Sins is that, you know, your security, you have security compliance data, like your audits basically, you know, sort of create incredible toil and have like terrible efficacy, right? And that's even before you get into your sort of containers and platforms and clusters and even worse, like, you know, event-driven architectures and serverless and stuff like that, right? Like I'm saying, like even in your sort of virtual environments that haven't been containerized or Kubernetes-ized or OpenShift-ized, you're terrible at this. But in the gap gets worse and worse the more you modernize, right? You know, the ability sort of to tie some of the ephemeral activity that happens in pipelines to change records of somebody describing that complexity in human-to-human discussions, right? And that's what auditors are sort of going by. Notice like, you know, what's the sort of common theme of one in order who truly doesn't understand they ask for screen prints? So like, that's the sort of, you guarantee no, you've not done it correctly. And so the other thing I've been thinking a lot about, so I've been in these multiple working groups, I've been exposed to sort of just a lot of brilliant people or like literally people who sort of run all cloud security for like five billion IT a year budgets, right? Like, I mean, that's like a mammoth job. And I've been fortunate enough to these working groups to have conversations with a lot of people. And I keep thinking about how everybody's got this sort of own agenda of what DevSecOps is. And so I don't even want to call it DevSecOps. I'm just saying it's modern governance. There's something here. And so I was thinking about, okay, what are the things that like I'm passionate about and what are the things I think that they cover a lot of ground? Like it's not everything. One of the hard things I find that if you sit down with a bunch of people and say, okay, we're gonna say, we wanna have a sort of a holistic presentation of you about DevSecOps. And I would tell you in every conversation that's even tried to do that, it falls apart really fast because there are so many security related things that are hard to categorize at a primitive level. And I'm not trying to do that here. I'm just saying like, I'm really only gonna cover probably primarily depending on time, the first one. And maybe for another day, I'll talk about sort of defense and trust, but I've been very focused in risk. That's that automated governance stuff I was talking about, which is, what are we doing to reduce toil and increase efficacy when it comes to sort of our, our risk controls, you know, usually around our governance and compliance and usually comes down to some type of automated form of attestation, sort of evidence in some sort of digitally signed structure and not going to that or a gating like the sort of Capital One model of enforcement. A second area that I like through the cloud governance project realized that all these people are highly involved in the sort of like, again, overloaded terms, but they're all building data lakes, cyber data lakes, right? That's the thing, all these data lakes are really just elk stacks, right? But, and I'm not, that's not saying that's a trivial thing, like to build those things is very complicated to be able to pull and sort of normalize and enrich all the data and get the right meta constructs into a data that actually is searchable and intelligent and can give you sort of predictive and I would categorize that of all the work that's happening under what I would call the Toilet and Apricot of Defense. In fact, that's what most of that project is all about is all the stuff that we get from say, Amazon, Google, IBM, all the major clouds are all have different type of contacts, even though they might actually have the same meaning but the poor cloud administrators like have to figure out, interpret like, what was a permissive grant and how is the wording of that from Amazon versus Google versus Azure versus, right? It, there's a lot of toil in that and then how do you get that into a data lake? You know, and if you've seen some of the presentations, so Onar will have a spring conference and you'll get to see if you sort of pay attention just Google own NUG, you'll get to see some of these companies and the complexity that they have, they're creating on their own, great data. And then the last area just quickly, just because I think a lot about this and I do less work in the third category which is about trust, right? And that's all related to identity, you know? And I had a CISO tell me that at least I was like, getting one of the largest defense contractors in the world said that John, it's all about identity, right? Like we think about all the breaches like it's some form of sort of, you know, an account that sort of is shared or it's a service side request forgery, like all the sort of big ones we've seen is all about identity. So then we get into like, you know, and it's not just getting in, like it's like once they're in, how did they find secrets? How did they find, you know, passwords? How do they get into systems? How did they get to the Amazon metadata server and do a fake identity, right? So there's a lot of stuff there, right? So again, I just wanted to get that. So let's talk about the automated governance, like, which is sort of that first category of risk, right? And so one of the things that when you think about like the whole idea of risk, you know, or this way to create digitally signed evidence as I described earlier, right? You take what Capital One was trying to do with the enforcement, what if we could turn that into digitally signed evidence? So I like to say think blockchain but like don't use blockchain for this problem. But the idea like is that instead of an auditor going ahead and sort of having to go back to a bunch of change records and emails and give me this log, give me that log, how come these two logs don't match? Well, they never will, they're not the same, right? And give me screen prints that what if we could just say there's basically a digitally signed event that's a list of a digitally signed evidence that's all immutable, can't be changed. In fact, the store is immutable and like you're done. Like, okay, I mean, there's like no discussion. Like we know that this is truth. And so the then question becomes this verifiable data audit model which is how do I prove that I'm safe? Okay, and like we do that all day long, right? I have arguments with CIOs like not since I've been in Red Hat, but when I was independent I would go in and I'd do this analysis company and I'd go to, and I'd start pointing out that Seventh Deadly Sin and like they would say I would get really upset when I would tell them that their audits are basically theater. I mean, really, like they defend themselves and then I give them proof about what I learned and their proof would be, well, no, no, no, John, we get awards and look at our sort of our, and they're all these subjective like things that they say that they're safe. So the proof is this subjective manifest of change records and archer databases and all this stuff, right? And but then the real question is, can you like when they ask me, I'll say, well, okay, here's a story I want to tell you that I learned. Tell me how you would demonstrate that that's secure. Even though in your subjective model you said it was from an audit perspective and the answer was you can't. Sorry about that. And so the real question is how do you do both, right? Like that, how do you get to sort of both and so then what we fall upon and in fact this started in the first working group of the devil's automated governance is like, how do we move from implicit trust-based model for controls to an explicit based proof base? Sort of a lot. And then the how is how do we change the subjective nature or subjective evidence or we just call them attestations into objective attestations, right? Like how do we instead of like having sort of a telephone game of, you know, Sue says, it's going to be like this. Bob said, did you do this, this and this? You know, Jane says, well, it's got to have this if it's going to go in production. And, you know, and then we spend 30 days to 40 days a year sort of reconciling these discussions as opposed to we already have the automation, couldn't we just create digitally signed events of the things that happen with no human intervention and have that be the evidence. And so this idea of like sort of an objective evidence in closed feedback loop when it comes to security, right? So let's reduce audit times from like 30 to 40 days a year. So I've got, I know banks that basically just have groups that just reconcile auditors who's all year long, right? To like zero time, right? It's continuous, it's hit and error. Like anything that gets deployed, the evidence is there in a mutable format. And then also because it's actually coming from the automation, it's not tampered by humans, the efficacy starts getting increased. The scenarios where, you know, the CS is, you know, John, I'm not going to accept that. And I said, well, let me tell you about this, this sort of microservice that uses DynamoDB that has this and like that and a container and it branched up to that and use service mesh, the Istio Envoy stuff, the route. Tell me where the evidence is for that. Like, and I'm not saying automated and ground-mount cells, all that, but at least you can have webhooks. You know, the opportunity to create is much evidence as you have. And in fact, what actually gets more interesting is you move from a reactive mode where most sort of service people, developers, are in a sort of reactive or even conflicting mode with internal order of risk. And then, oh, geez, they always want us to do this. And the thing I always hear is like, you know, we don't tell auditors things they don't already know. Like it's a common thing I hear. And you move out of that to a proactive mode where you wanted to tell signs that things are working way better is you see an increase in self-identified risk controls. That means not only are these sort of service delivery people on board with this new way of doing things, they're actually starting to think about how to protect the brand. And is that a latency issue? That might be a brand-tarnished opportunity, right? So all those things, shorting feedback loops, moving away from cabs, enabling just, you know, and I do these loose surveys, but 90% of most controls and most banks are still manual today, right? You know, and so what we did is we structured, oh, there's one more thing I want to say too, I think this is interesting. I heard a charity major say this and I thought it was sort of a brilliant observation and I'm using it to extend the opposition. So she was saying, one of our sort of interviews, she owns Honey Column, she found a brilliant one. She said, do you like this thing about what's, if there's a company A, this is sort of a DevOps conversation. I'll sort of give you my DevOps spin in a second. If company A, what's the difference between company A where company A has a couple of thousand developers and they're deploying, you know, on demand sort of all the time, hundreds a day, you know, gazillions a day, I don't care. And company B has a couple of thousand developers but they deploy quarterly. You know, what is the real answer of the difference between those companies, the real, real, the real important answer is that company A learns, capital L learns, maybe two orders of magnitude faster in company B, right? We understand that. We like, that's everything we learned from Lean, it's shift left, it's why do we do that? Like when Amplify feedback loops, we wanna catch it early so we can sort of fix it early. All right, and it's the whole sort of moving out of waterfall processing, right? Small batch, iterate. But think about where we are with security. We're in a waterfall, like the fact that like, and because of one more point, the important point is like, when we write code, but in the DevOps scenario between those two companies, they're all hypothesis, right? We write code, so it's a little bit art. I mean, some would say it's a lot of art and they're hypothesis and how do we actually prove our experiment, right? It's when it's in production. And that's why sort of that shift left and Amplify feedback loops and the sort of fast iterations is very important because we learn quick, we get to make our decisions. Well, you know, control regulations and our hypothesis as well, you know, despite maybe what sort of some, you know, Sistine Chapel, like, you know, building in your sort of financial institution things, you know, with the vaults of the papers and the regulations stacks of books. I mean, they're hypothesis, right? And so the question is if you're doing audits once a year, you have all the ills of what we had pre DevOps, which is by the time you actually try to reconcile that everybody's forgotten about it. They've been on like four more projects since then, right? So then the question is, are we getting immediate feedback loops on our control? And I would argue that this type of automated governance structure is an ability to increase efficacy through that process. So sort of quote unquote, terribly phased DevOps seeing security. And so this is a structure. It's in that Creative Commons books that you're sort of welcome to get on IT Revolution. You can download it. As the first reference group that I talked about, we tried to figure out what are the stages and what were sort of the common attestations. It's a really good book for just understanding, you know, control actors and common gating or attestation manifestations. And again, I won't go through all these. And actually this has been updated to sort of, this is not, this is, there's some of this in the original reference paper, but you know, it's sort of in my travels and work that I've done internally and externally. Like you can see in the development stage, you might have code quality from SonaCube or customs or change size or psychometric complexity. Was there a pairing on a pull request, branching, an optimum branching strategy, clean dependencies, build performance, build version, linting, SAST in the build stage. You can look at, and these are just examples of certain places that we've done a couple of internal hackathons. So we use, you know, some of the easy killed products like SonaCube and, you know, and then artifact versioning. So it would say maybe come from Nexus, meta package, code signing. You know, I'll tell you kind of funny story a little bit about code signing. Container scan and then pre-pod. And then one of the things that we did after that original paper is a couple of the organizations, we're going to really address this heavily in a second. You know, originally we were calling policy code, but that's such an overloaded term. Like, you know, for our purpose it's really more like a risk as control. And I don't know that we're going to stick with that as the name of the DSL, but this is very specific to the automated governance model I just described. In other words, there would be a collaboration between risk or internal audit and developers. It's like sort of cut out the infrastructure people. Like cut out them. It's like office space. Like what do you do here? In other words, let's just have like a design requirements. Let's have sort of, you know, risk or internal audit sit in and collaboratively create an artifact that is an agreed communication structure. Because the other problem you have is, you know, unless you're sort of constructing something that the automation is going to use, you're going to have like mismatches in terminology and thinking and context, whereas this forces you to sort of spell out the same enforcement in attestations that need to have and you, you know, just the YAML itself creates sort of a meta agreement. Internally in Red Hat, we've been, when I got to Red Hat, you know, I've been working externally and then I got to find out there was like some really awesome projects of you imagine as a large open source company that we have and something that is called the trusted software supply chain that was originally done as some of the government software factory models that they were trying to sort of drive or a red sword they call. And this is really cool. And you know, I'm getting a little tight on time. So like there are lots of really cool presentations here. But this is just sort of foundational. This is an automated governance by itself, but it is, you know, it's just my whole thinking. I really like this one because it's an opinionated reference but not an opinionated implementation. In other words, it's a composable sort of YAML based structure to define whatever with sort of opinionated structure in terms of stages, but completely unopinionated on the choices. Like you want Jenkins, do you want bamboo, do whatever it did like it doesn't care. And it does a lot of cool things. Like not only will it sort of, you know, sort of inject like Sonacube to run a scan, it will actually take the artifact and scan and store it. It was all necessary things you need for automated governance. Because what you'd like to do in the attestation is not only say this happened, but possibly ta and then Shah that lock, right? As a mutable event in the chain. So it's, this is brilliant. And so the model then like takes that work with some of the open source work that I've been on in this sort of model that does collecting and testing and enforcing uses a DSL, something like a YAML based structure. Originally we're using Graphius. I'll talk about that. It doesn't necessarily have to be with Kubernetes or OpenShift, but like in our examples, we're using OPA and Rego. And then this is the TSSE project. It's actually being renamed to this TSSE Python project. And the other thing, again, this is like what's really cool about we're gonna run at is another team that was building basically, what it's called a verifiable audit data store, which basically based on some of Google's work with something called Trillion, which again, I'm going really fast, but you can look it up, is basically a true immutable storage structure. In other words, it just can't be changed, right? So now instead of putting our attestations in this open source Graphius tool, which has worked very well for the last couple years for us, but now we're trying to look, because this is based on a Merkle tree and it's just like the perfect solution for some of this stuff comes out of what Google's doing. Actually, a lot of it came from certificate transparency and key transparency, but like you can also use it for this idea of verifiable audit data structure. It's just really, you put all this together and it's really cool. So here's the sort of obligatory, like we're gonna talk about security, you gotta mention SolarWinds, whoo, whoo, but I have a real sort of point here. So I had the opportunity to do some research about the SolarWinds thing and I won't sort of talk about why I had to do that, but you let your imagination be your guide. And I went out a lot of places and like KPMG had some good stuff and Krebs on software is always good. In fact, he's the one that pointed me to this, the crowd strike, and they, to me, they did a fantastic job of how the original kill chain happened inside of SolarWinds. This is like before it gets out to the wildness, everybody in it. It's an ingenious, I mean it like yet to Shannon's point, man, the adversary analysis is overdue because these people are ruthlessly intuitive and creative. And so I won't go through the whole thing, but like one of the things in the blog article, which is brilliant is they used, if you're familiar with the MITRE attack framework, like we're using heavily in our cloud automated governance project. So I will say this is my sort of matrix, but I took directly some of their matrix and sort of merged it with, so here's the point. What I wanted to show is how automated governance might have been very helpful and instrumental for SolarWinds based on the attack surface here. So we'll go through a couple of these, but like the two meta points and these slides will be variable. So you can have fun sort of finding original article and see how I said, you know, this attestation would have been like incredibly helpful for them. The one of the things that the crowdstrike started to point out, which was there were all sorts of hash mismatches. So it just, in short, they hijacked MSBuild. They basically went ahead and sort of inserted their own code into the build. They did some log masquerading. I mean, they did just crazy intelligent stuff, right? And they were doing it for many years too. So they had a lot of time to perfect your system. And all I was saying is that they had a really sort of mature pipeline structure, which we don't know, but have to assume they didn't have proper hygiene. But even furthermore, if they had automated governance, they could have been like signing mismatch, code signing mismatches. If they were to use something like Trillion as the attestation sort of like you couldn't, it's a miracle. You cannot mutate it. Like so some of the mutations that they did for log masquerading. So, you know, so I tried to sort of point in like their sort of observation and, you know, and their tactic and ID from a wider perspective. So I thought this was fun. And, you know, and I think it, to me, you know, again, I'm taking advantage of the situation, you know, hello and say, oh, pay attention everybody. But I really wanted to make the point that, that, you know, had they had automated governance, not automated governance and saw everything. Like, I'm not, we can go in closer to that. But it's a level of hygiene that can produce sort of a bill of material or something that, you know, like this idea that, that like when I go in and, you know, Shannon was saying this to me too, like when you go in and you talk to people and say, what's your evidence that you're actually doing this correctly? And like, and if you can't even show me some form of a bill of material for everything that you create from a software factory perspective, then you're not, you're wrong. Yeah, that's, I think that's it. So I got covered everything I wanted to cover and so. You covered more than everything you wanted to cover. There's enough tips and tricks here and leads to other topics that we could probably go on for a month of Fridays. So thank you very much for this. I think this is, it's what I'm gonna have fun with is getting all of the additional resource links for this cobbled together, I think. So thank you very much for coming, John. You know, there were a couple of questions I think I handled mostly about, you know, how do I get a certification and DevSecOps from out of Red Hat? And, you know, that's an interesting question in that, you know, how would you certify someone in this topic? There's so much to it. You know, I have dear friends, dear friends that I really liked that actually do this, right? And so I try to be sort of like split the middle, but you can't, you can't certify it. I mean, again, if the goal is to sort of, you've got an uninformed organization that requires it and you're gonna play their silly game because it helps you achieve the things that you don't know why this keeps happening, but I'll turn this off, but sorry about that. You know, then yeah, then like, God bless you, just do it, right? But the truth of the matter is, like I said earlier, when I try to sit down with really intelligent people and say, like, what is the holistic landscape of DevSecOps? Is it identity? Is it authorization? Is it, you know, and even in the identity space, is it secrets management? Is it, is it certificates? Is it, you know, is it encryption? Is it encryption at rest? Is it, you know, does it include discussions about vaults? Does it do discussions about SPIFI and the service match? And like, like, right? And then, okay, so let's take another topic, audit, okay? You know, like, forget it, we can spend months on audit. Like, do you gotta know Archer? I would say no, you know, like, that's sort of an old way of thinking. You have no three lines of defense. If we talk about, if we talk about cloud governance, you know, then oh, God, like, you know, then like, you know, like, what are all the sort of things in Sims or Sores or, you know, how do you create cyber data lakes? And then, you know, what role does sort of the traditional Sims and Sores give you versus now all the stuff you get from sort of Amazon's web services that are all very immature from that perspective. And then even if you got that, right? I mean, like, again, I know I'm over-rotating on this, but I want to just make the point why certification is in general nonsense. Because even if I got all that right about Amazon, what if you're going to use Google and Microsoft? Because that's all another, so I mean, when we just use the banner, I mean, in DevOps, I would give you a little bit of wiggle room on a certification. Because I would say, okay, if you understood the meta, if a certification test, I still sort of fundamentally disagree with these certifications, but if you understood the sort of meta around certain principles that I think were very important to have in DevOps, sort of the culture, the automation, those things, I mean, Damon coined something called CAMS years ago, course, all the measurements. Like if you understand those things as primitives of behavior and how the technology and the social technical aspects of it, like I think you can answer those questions to do a pass-fail. Maybe. But I think when you get into security, you're like evidence that even the security people make fun of their, I just don't see how you... I mean, even in the training, you have to just go for Sam's Institute, just to learn traditional security. Yeah. All right. I gotta say, don't get me started, but I've already started. I know, I love getting you started. That's like, you know, working you up on this. I think the short answer is like, Red Hat certification is specific to products for the most part. There's some, you know, and so there is a certification as a specialist in security for containers and OpenShift container on the OpenShift and the OpenShift container platform. Where you learn all the security bits of that platform, of that technology. But it's not the cultural thing. It's not the, you know, automating all of, you know, the security and governance and the audit and the risk. Yeah. And that's a great point, Diane, right? Because I mean, as much as I rant, but you're absolutely right. Not because I wore a red hat, but you're right. I mean, if the question is like, you know, we acquire stack rocks, right? And then like, you know, will we or, you know, we will have a certification about the sort of how do you use that tool effectively and even some sort of meta aspects of using it. That's a valid certification, right? Or any other sort of specific tool that we have, like container, you know, our version of sort of container adoption and so like that, I agree. But I think I sort of drives me nuts on Facebook when somebody posts there like, I am DevSecOps certified, you know what I'm like. Like, come on. Like, it's crazy to say, like, tell me what you know. And how you can get all this out because I've been struggling for at least on the security question for like five or six years to try to figure out what to, and it was this presentation that finally got me to at least create those three boiler plates that I talked about. Like, and that's taken like five years for me to get to even build that slide to say, in my mind it's risk, defense, and trust. Yeah. So like, if you go back to your, you know, you don't have to do this, like, if you go back to your, like your software factory slide, you know, there are pieces of technology in that, that people should get certified on, you know, and should know in depth. But it's, and this is kind of why I have you on, on Fridays when we're talking about organizational and transformational and systems thinking and higher meta things, that these things are a level above the traditional sort of things that you can get certified on. And- But that's GTO too, right? We didn't spend a whole lot of time. I know our team is, you know, on this, this forum quite often, right? But like, we're sort of about, like we didn't get hired because we were like red hat experts or we didn't send those. So we, you know, like, we got hired because we, you know, all four of us, you know, Andrew Clay Safer, if you don't know, Kevin Bear was co-author of Phoenix Project and Jay Bloom is probably one of the smartest guys, one of the top three smartest people I've ever met. The, you know, like we're all, you know, we try to bridge that technical and social, right? We try to, you know, make sure that we're not over rotating on meta, but we're also like, you know, so we're trying to hit the middle ground and that's what we do here at Red Hat, you know, and that's our sort of bullhorn too, so. And that's why it's wonderful having you here and having you back this week. I think that's a great way to wrap this up too. I think that's what you bring and that's what, you know, organizations have to also wrestle with that this is both a technology issue and problems and things that we can solve with some training and some certifications, but it's also a meta problem and a cultural problem or initiative within companies that we're thrilled to have you guys here helping us work with all of our customers and ourselves internally too to wrestle with these things and help drive them forward. So thanks again for coming here today and we'll make this session available up on YouTube and tell us again, John, what was the conference that you're gonna be giving this presentation again? Oh, so I'm confirmed for Swamp Up. I think that's in May, that's Jay Frogs. They're great people. I've been speaking at their conferences for quite a few years now. They've got this great community and just their CEO is just a lovable guy and I like that environment and so I just have a blast. So I speak there almost every year. So I'll be giving this presentation at their Swamp Up and I don't know if I'm confirmed yet for the DevOps Enterprise Summit, but it'll be the first time since its existence that I wouldn't be speaking if it isn't accepted, but that'll be basically what we call the virtual London and then I think it'll be April, May and then they'll probably repeat that at the end of the year for the virtual, hopefully maybe it's not virtual, by the time I get to Vegas. I don't know, probably maybe. So I'll work with you, John, to get some of the, make sure I get the right references in for this and if anyone out there is watching this after the fact and has feedback for John, reach out to him, here's his contact information and join him for further conversations on this topic. And obviously- Yeah, I'd love any, like I said, the first time I've given it, like did I over rotate on something that you, I would love any sort of really hard constructive feedback would be very helpful for me. All right. Thank you, Diane. Thank you for you doing what you do. Like it's great to have this forum to come and experiment and just be part of the community, so. I love giving away the podium. So if any of you are out there and listening to this and have another take on it or a deeper dive that you want to do, just reach out and I'll definitely give away the podium. So thanks again.