 Well hello everybody and welcome to another OpenShift Commons briefing today as we like to do on Fridays. We are going to have someone from the Global Transformation Office talking to us about things organizationally transformative or systems thinking. We have all kinds of topics that we do on Fridays. But today I'm really thrilled to have with me again John Willis who's going to talk about some DevSec Ops topics and his title today is The Blurred and Broken Lines of Defense. And I'm going to let John introduce himself and we will have a live Q&A at the end of this. So please type your questions in the chat and we will have a conversation afterwards. So take it away John. Hey thanks Diane. It's good to be back. It's been a little while before I've been on. I love the Friday OpenShift Commons Briefings fun. Anyway thank you all for being here. One of the things for those of you who know me for probably less than five or six years I do these sort of I pick a theme near the end of one year and I start thinking about like that's going to be it's not the only presentation I do all year but it's sort of the main like in any of the keynotes and things that I might get invited to. And so this is the first sort of attempt at this presentation I'm calling The Blurred and Broken Lines of Defense so and it's sort of as it turned out is it's turning out to be a little bit of a meta piece too. So you'll see what I mean by that. But one of the things I would love is you know I'm about to go loop there's Twitter handle Jay Willis at Red Hat. I would love your feedback right because this is my first version you know I think you know I've submitted this to SwampUp I've looked I've been accepted at SwampUp for JFrog and probably DevOps Enterprise Summit so I'd love to get any good you know really sort of any type of feedback in negative positive you could have done this should have done that so that would be very helpful. All right let's get started and so for those you know so quickly I waste too much time there's a lot of information about me out there you know I'm old as dirt so I've been doing a lot of stuff I like 10 startups or 11 books like I lost chat probably the most so biggest claim to fame and sort of this space is I co-authored DevOps Handbook I was co-author of a project called it's an audio book but it's called Beyond the Phoenix Pack with Gene Kim and then some I think really important working groups that I've been on and I'll talk about these two the the green one which is DevOps Automated Governance we're about to start the second version of that and something called the Automated Cloud Governance and I'll talk about that anyway and done a whole bunch of startups you know so and I do play guitar too so not really well but I do play guitar so here's the question you know and I said that like I went down I had a very distinct sort of motivation for this presentation I started building the presentation I I got sort of various sort of you know meta in the way I was thinking about where are we at like so DevOps is you know give or take ten years all right it's a little more into but you know 2019 was the 10 year anniversary officially from the first DevOps days event you could argue that DevSecOps is about five years old we've done some really good stuff right like I mean like big time like in both of those spaces and you know again DevSecOps is lagging a little bit behind our sort of all the things we've done well in DevOps but you know what let's face it it's still a mess out there right I mean that's the bottom line right like we're still struggling we're now calling it digital transformation and that's okay but like just these are sort of like we're having a hard time that the complexity is growing faster than our ability to reason with that's a whole not the presentation so then the question I was thinking about when I started writing this presentation which is and stay with me here folks what would DevSecOps what would the conversation of DevSecOps as a phrase or a term or a meta movement be if DevOps never existed and and the reason I I'm saying that is have we made a whole lot of choices about security with the bias of what we accomplished with DevOps and then where those are the right choices right you know and you know we can talk all day long even today most sort of hard and security people say we still are running on you know 20 30 year old security models right and you know in DevSecOps we've done some very cool stuff right like we we've sort of added the automation but in a lot of ways I guess you know I'm gonna and I'm gonna pick one particular area to sort of explore which is the we'll talk about the lines of defense but but just a question I'm adding is like yes it's additive what we've done you know DevSecOps reference architecture is you know putting automation of things that were done manually outside of pipeline yes yes yes yes all great stuff but are we significantly making things better from a security perspective you know I mean I was gonna post like all this sort of the security vulnerability and breaches that just happened within the last three days and it was like it was gonna fill up the screen anyway so you know and then you know we put into sort of agile security and we we have those conversations so hold that thought for a second and then I'll say okay let's go back and say if I say that DevSecOps is five years old Shannon Leitz overed into it if you haven't followed her work you should she's brilliant you know she coined it you know I think before 2015 but this was a sort of seminal posting that she did you know and she's you know she owns the sort of coining of the DevSecOps turn because he you know it was actually just you know honestly I was talking the last night I wanted to say like I showed her so a little bit of my presentation like if I gotta be it's just nonsense it's just good Shannon you know make sure I don't make an idiot of myself right and you know and I think one of the things she's great at is you know she sees the world you know way different than everybody else and so if you go back in 2013 14 season into it she's trying to solve these really hard problems against adversaries and she sort of has this you know it sounds simple now this epiphany of this idea of DevSecOps at that time it wasn't you know because nobody else thought of it and and the thing that's sort of interesting about that is you know that first sentence where she said everybody everyone is responsible for security that's 2015 right and and you know like 2021 like we really haven't solved this one I mean we have we're doing a lot of things but like I you know I I do a lot of interviews and I do this qualitative data analysis some of you might see my presentation you know the last one I did on this this forum where I interview hundreds of people in organizations certainly talk the risk and and I don't get the sense that we're you know we're like you know we're not even in the bees or like C plus maybe you'll be honest in this and I would say there aren't pockets of like excellence all over the place but globally I think we're not and then you know I think we're still trying to figure out who should do security right like you know I'm you know one of the areas I spent a lot of time is something you know sort of modern governance or sort of a new branding of our new way to think about automated governance and a lot of that stems for sort of a very siloed approach between risk IT sec sometimes the you know internal audit which may be the same not the same and in dev and ops and infrastructure and who really owns this and how you know and so a little bit of more memory lane you know I've been fortunate enough to work in Gene Kim's organization we write forum papers you know it's about 40 50s to get together every year you normally go to the Portland unfortunately last year and this year it has to be virtual but we write these papers and there's been a number of sort of collaborative efforts over the years the original one was 2015 unlikely union dev ops and audit in 2018 it was a sort of tongue cheek dear auditors which was an apology letter to auditors but it's actually more than that right because if you actually like so maybe the first three pages is this apology letter like how we screwed up and and then you know the next sort of 30 pages is a lot of control regs and and and trolls that were so we promised to be better at this one look at this and and then you know and then a project that is sort of near and dear to my heart which is in 2019 I got a group of people together as part of that forum Nike capital one PNC John Resotasi PNC top of our camera one Courtney kisser and Nike at that time Saber Maric between homes of Maric group like it's just some other people I'm forgetting but and we tried to sort of proof out like how could we do audit efficacy reduce audit efficacy and toil and and that these are all creative comments so and I'll talk more about that in a little bit about what we did there and then one other I think historical marker for me thinking about this whole dev sec ops for you know me I was the only American at the first dev ops days in Ghent you know back in 2019 you know Andrew myself and Damon Edwards ran the first and actually Mark Ingle ran the first dev ops days in the US together we collaborated on that so I've been in the dev ops game for quite a while but um but you know the dev sec ops things were sort of started sort of gnawing at me about 2014-15 you know about like oh my god we forgot security like how could we make that mistake like and and a total pal had written this article in 2017 capital one about how they did pipeline design right creating better pipelines and and they called it just kind of funny for you sort of geeks out there they said the 10 gates that they they had set up which is hex right so it was 16 gates right but the you know and they said that so the thing you had to read between the lines was they would say they were telling the organization hey you know you have to they weren't telling the orders and you have to do this you have to do this and then sort of fight that battle they were basically saying if you could show up with your service or application your code or whatever with these 16 sort of the ability to show evidence of these 16 things then you will get certain privileges in how you deliver your software like one for example you might not have to go to a Wednesday cab meeting right and and at that time I you know I was I was hanging out with it was topo and we were having a lot of discussions about like what could we apply a blockchain model to this and I think we you know the industry found out really quick like block chain is probably not a great model and and like what was really lacking here is what the industry was doing great is is gating a lot of things like you know everybody was sort of maturing very well on how do we stop the build if you don't like you know if it doesn't come from source control or if there's not a pairing on a pull request or there whatever and then also if it doesn't have you know a clean build or even furthermore in the DevSecOps discussion does it you know does it pass a vulnerability scan does it DAST and you know SAS DAST and in whatever you know software composition analysis all those things right and it's a reasonably advanced model and I would still argue it was still not changing anything fundamentally there but we were doing good there but like the real another question that was creeping up is back to the audit question where we were reducing any toil for internal auditors and where we're using it all when it came to you know control regulations or you know GRC if you will and and and what we improving the efficacy of what we were actually capturing and and so the discussion came is like if we're going to capture this stuff for Gates anyway why couldn't we just be building some sort of attestation digitally signed at the station store I'll come back to that a little bit and so the two projects that I've worked pretty heavily on over the last few years is you know one called the DevOps Automated Governance I talked about we're about to start the second version of that and there's some really fantastic stuff that that has been designed and developed since you know the first paper in 2019 a couple of sort of organizations have really taken this pretty far we're doing some really cool stuff in red hat with it so I'll show you that and in another group I got invited to the beginning of next year last year which was a bunch of large sort of healthcare health care health care's and and find financial organizations that wanted to do something around clouds loosely related to the work that we did in this paper so I chaired something called the Automated Cloud Governance and I'll talk a little bit about that too so and that that actually we're about midway through the second working group on that so so really interesting stuff and I'll put it in context as we move and you know one of the things that as we think about the sort of the problem statement of like in today's world with all of the complexity that just grows and grows and grows this is more cloud focused but this is in general this is a slide from ONUG the group that the second working group it's the largest networking user group these are guys of people that were heavily involved in SDN definitions SD WAN and now doing a lot of stuff in in sort of DevOps and in this case cloud governance but the idea that like the the complexity of what like what we get hit with every day from sort of you know just in a cloud world right is overwhelming and at any point it's a sort of a moving target of what your minimal vial security posture is and so you know again in in this reflective mode that I was in when I was trying to put this thing together is you know traditional security models are in any pattern right that's why I sort of even the sort of the hardened sort of security people laugh at themselves like yeah you know I was just having a conversation with a friend of mine this morning who was like a security guru you're like 15 years ago right like insane like you know a kernel stuff right and and then really just gone into a lot of other stuff like you know sort of development and and was pulled back into a deep security conversation and he started off with apologizing that hey you know I haven't really done anything in security for 15 years and I'm like you hadn't missed anything buddy I had a little tongue-in-cheek but but again I think the sort of question then is yeah I mean DevSecOps reference text was a cool right like I mean like you know automating a lot of stuff we did decoupled and manually are now sort of in pipelines which is good you know sort of our SaaS that's automated pen testing but then you know the the real question is is all this you know how much how channel east says uses a term called security like that this should be an ability and our can we ask the question are we actually more secure and and I think you know I mean like some presentations I try to scare everybody to death and show you all the stuff right I'm not gonna do that this time but like take it as default it's pretty freaking scary you know you know the the complexity of the problem we're trying to solve and the intelligence of the adversaries and I'll talk about that a little bit alright so so that's a given and you know are we you know did DevSecOps like really fix this are we I guess the sort of the main point I'm trying to make here which is have we force-fitted remember I said earlier what would DevSecOps look like if there was no such thing as DevOps would we have made all the right the same decisions about how we are trying to deconstruct or maybe just inherent as is security models as opposed to like what Shannon tried to do originally in 2015 which is break it blow everything up in fact you know some of the stuff that she's gonna be publishing shortly which is adversary analysis and adversary intelligence she's gonna blow it up again right that's for another presentation so so here's so back to the sort of what is the title of this thing the broken the blurred and broken lines of defense so one of the things I hear a lot from clients when I come in least clients that are sort of at least clients that are sort of in this DevOps DevSecOps IT ITSec and are savvy about the sort of constraints of their control regulations and audit and internal audit and they'll talk a lot about so the three lines of defense model right and and you know so this idea if you've never seen this before right like there's like the first line or sort of the control owners and again like don't scream at me if like you know like the IIA or the ISAC and you know you want to repeat the ever isn't go argue with somebody else the second line is is really sort of sort of control owners and so the original three lines of defense right which was you know this sort of based on like there was sort of it's the financial control the security control the quality control right like you can already see problems here if you're if you're sort of thinking DevOps and then you have this third line was internal audit so so like what's wrong with this picture and I and for those of you who are screaming out and saying it's been updated you know the the IA is updated I'll get to that slide in a second but it doesn't look very DevOps in fact it looks very similar to that wall walls of confusion you know it's you know it's you know you know it everybody sort of has their piece in this almost throw it over the wall right which this is sort of the dev you know space space space ops pre DevOps discussion right you know and and so you know and so I was talking to somebody else about this you know it's great to have smart friends right you know about this whole thing about three lines defense and they you know sort of brought something that should have been obvious to me that wasn't which is the whole Conway's lore and I like I started thinking like maybe there's sort of a new it's called it called the tale Sloan Taylor Sloan Conway law right like you know and I won't go in front of Taylor and and Sloan and you know this command and control structure but you know that if you're familiar with the sort of obviously overused in presentations but sorry because I think it the reason probably is over used in presentations because it actually it helps us define some of the constraints we have in modernization opportunities right so the Conway's sort of definitions any organization that design system to cruise design structure is a copy of the organization communication structure right you know there was the sort of original so added instead if you if you have three groups working on a compiler you're going to get a three-quest compiler well you know what do you get here you know three lines of defense structure right and then you like I said earlier like you know in all honesty like like this isn't like cave men mentality you know they've like in 2020 the IAA has updated this to be sort of more modern but it's still a solid approach right so okay so there's a governing body which is awesome it actually got rid of the defense to sort of promote a more proactive posture as opposed to everything sort of wait put like it happens and then we got to deal with it but again like I will argue that you know they sort of collapse you know the first line and second line and sort of this sort of quasi you know you can say okay give them the benefit it out that it isn't siloed there I would argue it still is but you are definitely siloed from internal audit and by the way that's one of the biggest problems we have is the miscommunication non-dialogic opportunities that we don't have with internal audit or risk when it comes to IT right that's that's why we have thousands of control regs that nobody understands and I was you know I wanted to test check myself I won't name the organization but it's like a well-known big four you know consulting company who is the heavily involved in sort of external internal audits and all that and in there sort of late 2020 bank you know sort of the bank regulatory observations you know part of the response to the final suspect ISAC basically you know I'll try to read this quick but you know another common issue is having so you capability is missing or in character basically what they're saying I read it you know the for example if the business is the first line of defense does not sort of still using defense right even though the the IA has changed that sort of nomenclature is not testing capabilities risk and the clients functions to the second line might perform testing however if the second line does that anybody remind you of like we're sort of early DevOps or pre DevOps always about QA and and development right like this idea like the next one will catch it right like it's just not like you know and like this is the sort of common mindset that like that like we're trying to sort of force fit into the oh yeah three lines of defense yes fits perfectly with and I'm not saying this is terrible stuff I'm just saying I'm not sure that you know we're actually sort of thinking about the sort of the the security problem the right way we're having an honest conversation and so I've been you know I've been doing this qualitative data analysis stuff for quite a few years now where I basically interview hundreds of people in our organization and I have this sort of presentations I've done in days gone by called the Seven Deadly Sins of DevOps and so I'm thinking a lot about like you know and you know and you know to me like it all boils down the seven deadly sin is that you know your security you have securing clients data like your audits basically you know sort of create incredible toil and have like terrible efficacy right let's and that's that's even before you get into your sort of containers and platforms and clusters and even worse like you know event driven architectures and serverless and stuff like that right like I'm saying you like even in your sort of virtual environments that I haven't been containerized the Kubernetes eyes are open-shift eyes you're terrible at this but you and the gap gets worse and worse the more you modernize right you know it's you know the the ability sort of to tie some of the ephemeral activity it happens in pipelines to change records of somebody describing that complexity in human to human discussions right and that's what orders are sort of going by notice like you know what's the sort of common theme of one in order really doesn't understand they ask for screen prints like that's the sort of you guarantee no you've not done it correctly and so the other thing I've been thinking a lot about so I've been in these multiple working groups I've been exposed to sort of just a lot of brilliant people who or you know like literally people who sort of run all cloud security for like five billion IT a year budgets right like I mean that's a like a mammoth job and I've been fortunate enough to these working groups to have conversations large people and I keep thinking about how everybody's got this sort of own agenda of what DevSecOps is and so I don't even want to call it DevSecOps I'm just saying it's modern governance but there's something here and and so and so I was thinking about okay what are the things that like I'm passionate about and what are the things I think that they cover a lot of ground like it's not everything one of the hard things I find that if you sit down a bunch of people and say okay we're gonna say we want to have a sort of a holistic presentation of you about DevSecOps and I will tell you in every conversation that's even tried to do that it falls apart really fast because there are so many security related things that are hard to categorize at a primitive level and I'm not trying to do that here I'm just saying like I'm really only gonna cover probably primarily depending on time the first one and maybe for another day I'll talk about sort of defense and trust but I'm very been very focused in risk that's that automated governance stuff I was talking about which is you know what are we doing to reduce you know to to reduce toil and increase efficacy when it comes to sort of our our risk controls you know usually around our governance and compliance and usually comes down to some type of automated form of attestation sort of evidence in in some sort of digitally signed structure not going to that or a gating like the sort of capital one model of enforcement a second area that is like through the cloud governance project realized that all these people are highly involved in the sort of like you know again overloaded terms but they're all building data lakes cyber data lakes right that's the thing all the other these data lakes are really just elk stacks right but um yeah and I'm not that's not saying that's a trivial thing like to build those things is very complicated to be able to pull and sort of normalize and enrich all the data and get the right meta constructs into a data that actually is searchable and intelligent and can give you sort of predictive and and we I would categorize that of all the work that's happening under what I would call the toil and efficacy of defense in fact that's what most of that project is all about is all the stuff that we get from say Amazon Google IBM all the major clouds are all have different type of context even though they might actually have the same meaning but the poor cloud administrators like had like have to figure out interpret like you know what was a permissive grant and how is the wording of that from Amazon versus Google versus Azure versus right it it there's a lot of toil in that and then how do you get that into a data lake you know and if you've seen some of the presentation so on our will have a spring conference and you'll get to see if you sort of pay attention just Google own and you see they'll get to see some of these companies and the complexity that they have they're creating on their own great data and then the last area just quickly just because I you know I think a lot about this and I do less work in in the third category which is about trust right and that's that's all related to identity you know and I had a CISO tell me other day at least I was like getting one of the largest defense contractors in the world said that John it's all about identity right like we think about all the breaches like it's some form of sort of you know an account that sort of shared or it's a service I request forgery like you know all the sort of big ones we've seen is all about identity so then we get into like you know and it's not just getting in like it's like once they're in how did they find secrets how did they find you know passwords how do I get into systems how did they get to the Amazon metadata server and and and do a fake identity right so there's a lot of stuff there right so again I just wanted to get that so let's talk about the automated governance like which is sort of that first category of risk right and and so one of the things that when you think about like the whole idea of risk you know or this this way to create digitally signed evidence as I described earlier right you take what what Capital One was trying to do with the enforcement what if we could turn that into digitally signed evidence so I like to say think blockchain but like don't use blockchain for this problem but the idea like is that instead of an auditor going ahead and sort of having to go back to a bunch of change records and emails and give me this log give me that log how come these two logs don't match well they never will they they they they're not the same right and give me screen prints that what if we could just say there's basically a digitally signed event that's a list of a digitally signed evidence that's all immutable can't be changed in fact the store is immutable and and like you're done like okay I mean there's like no discussion like we know that this is truth and then so then the question becomes this verifiable data audit model which is how do I prove that I'm safe okay and like we do that all day long right I have arguments and CIOs like not since I've been a red hat but when I was independent I would go in I do this analysis company and I go to and I start pointing out that seventh at least in and like they would see how we get really upset when I would tell them that their audits are you know are basically theater I mean it really managed like they defend themselves and then I give them proof about what I learned and but and their proof would be well no no John no we get awards and you look at our sort of our you know and they're all these subjective like things that they say that they're safe so the proof is this subjective manifest of change records and and you know an archer databases and all this stuff right and but then the real question is can you like when they ask me I'll say well okay here's a story I'm gonna tell you that learn tell me how you would demonstrate that that's secure even though in your subjective model you said it was from an audit perspective and the answer was you can't start at that and so the question is how do you do both right like that how do you get to sort of both and so so then this what we we fall upon and in fact just started in the first working group of the devil's automated governance is like how we move from implicit trust based model for controls to an explicit based proof base sort of a lot and then the how is how do we change the subjective nature subjective evidence or we just call them attestations into objective attestations right like how do we instead of like having sort of a telephone game of you know Sue says it's gonna be like this Bob said did you do this this and this you know Jane says well it's gotta have this if it's gonna go in production and you know and then we spent 30 days 40 days a year sort of reconciling these these discussions as opposed to we already have the automation couldn't we just you know create digitally signed events of the things that happen with no human intervention and have that be the evidence and so this idea of like sort of an objective evidence in closed feedback loop when it comes to security right so let's reduce audit times from like 30 to 40 days a year I got I know banks that basically just have groups that just reconcile auditors who's all year long right to like zero time right it's continuous it's hit enter like anything that gets deployed the evidence is there in a mutable format and or and then also because it's actually coming from the automation it's not tampered by humans it the you know the efficacy starts getting increased so those the scenarios where you know the you know the CSS you know John I'm not gonna accept that and I said well let me tell you about this this sort of microservice that uses DynamoDB that has this and like that and a container and it branches up to that and use service mesh the Istio Envoy stuff the route tell me where the evidence is for that like yeah and I'm not saying what I mean is go amount solves all that but least you can have web hooks have the opportunity to create as much evidence as you have and in fact what actually gets more interesting is you move from a reactive mode where most sort of service people developers are in a sort of reactive or even conflicting mode with internal order of risk and then oh geez they always want us to do this and or I you know the thing I always hear is like you know we don't tell orders things they don't already know like it's a common thing I hear and you move out of that to a proactive mode where you know wanted to tell tell signs that things are working way better is you see an increase in self-identified risk controls that means the not only are these sort of service delivery people on board with this new way of doing they're actually starting to think about how to protect the brand and is that a latency issue that might be a that might be a you know a brand opera you know brand tarnished opera too right it's all those things you know shorting feedback loops you know moving away from cabs enabling just you know and and I you know I you know I do these loose surveys but 90% of most controls and most banks are still manual today right you know and so what we did is we structure oh there's one more thing I want to say that too I think is interesting I heard um a charity majors say this and I thought it was sort of a brilliant observation and and I'm using it to extend the opposition so she was saying one one of her sort of interviews that she owns honey column she found a brilliant woman she said um do you hate this thing about what's if there's a company a this sort of a dev ops conversation I'll sort of give you my dev spec out spin in a second if company a what's the difference between company a where company a has a couple thousand developers and they're deploying you know on demand sort of all the time hundreds a day was you know gazillions a day I don't care um and company B has a couple thousand developers but they deploy quarterly you know what's their what is the real answer of the difference between those kinds of real real the real important answer is that company a order learns capital L learns maybe two orders of magnitude faster and company B right we understand that we like that's everything we learn from lean it's shift left it's why do we do that like when amplify feedback loops we want to catch it early so we can sort of fix it early all right and it's the whole sort of moving out of waterfall processing right small batch iterate but think about where we are with security we're in a waterfall like the fact that like and because of one more point this is an important point is like when we write code sort of in the dev ops scenario between those two companies they're all hypothesis right we write code which is a little bit art I mean some would say it's a lot of art and the hypothesis and how do we actually prove our experiment right it's when it's in production but that's why sort of that shift left and amplify feedback groups and the sort of fast iterations is very important because we learn quick we get to make our decisions um well you know control regulations and our hypothesis as well is you know despite maybe what sort of some you know um Sistine Chapel like you know building in your sort of financial institution things you know with the vaults of the papers and the regulation stacks of books I mean they're hypothesis right and and so the question is if you're doing uh audits once a year you have all the ills of what you what we had pre dev ops which is by the time you actually try to reconcile that everybody's forgotten about it they've been on like four more projects since then right so so then the question is are we getting immediate feedback loops on our control and I would argue that this type of automated governance structure is an ability to increase efficacy through that process so sort of quote-unquote terribly phased dev ops seeing security and so this is a structure it's in that creative commons books that you're sort of welcome to get an IT revolution you can download it as the first reference group that I talked about we tried to figure out what what are the stages and what were sort of the common attestations it's a really good book for just understanding you know you know control actors and common gating or attestation manifestations and and again I won't go through all these and actually this has been updated to sort of this is not this is there's some of this in the in the original reference paper but you know it's it's sort of in my travels and work that I've done internally and externally like you can see in the development stage you might have code quality from sonar cube or customs or or change size or cyclomatic complexity was there a pairing on a pull request branching an optimum branching strategy clean dependencies build performance build version linting sas in the build stage you can look at and these are just examples of certain places that we've done a couple of internal hackathons so we use you know some of the easy-killed products like sonar cube and you know and then artifact versioning so it would say maybe come from nexus meta package code signing you know I gotta I'll tell you kind of funny story a little bit about code signing um container scan and then prepod and and then one of the things that we did after that original paper is a couple of the organizations we're going to really address this heavily in the second um you know originally we were calling policy code but that's such an overloaded term like you know for our purpose it's really more like a risk as control and I don't know that we're going to stick with that as the name of the dsl but this is very specific to the automated governance model I describe in other words there would be a collaboration between risk or internal audit and developers it's like sort of cut out the infrastructure people like cut out them it's like office space like what do you do here in other words let's just have like a design requirements let's have sort of you know risk or internal audit sit in and collaboratively create an artifact that is an agreed communication structure because the other problem you have is you know unless you're sort of constructing something that the automation is going to use you're going to have like mismatches in terminology and thinking and context whereas this forces you to sort of spell out to see enforcement in attestations that need to have and you you know just the YAML itself creates sort of a meta agreement internally in red hat we've been when I got to red hat you know I've been working externally and then I got to find out there was like some really awesome projects of you imagine as a large open social company that we have and something that um is called the trusted software supply chain that was originally done as some of the government software factory models that they were trying to sort of drive or red sword they call and this is really cool and you know I I'm getting a little tight on time so like there are lots of really cool presentations here but this is just sort of foundational this is an automated governance by itself but it is you know it's just my whole thinking I really like this one because it's um it's an opinionated reference but not an opinionated implementation in other words it's a it's a composable sort of YAML based structure to define whatever with with sort of opinionated structure in terms of stages but completely unopinionated on the choices like you want Jenkins do you want bamboo do whatever I think like it doesn't care and and it does a lot of cool things like like not only will it sort of you know sort of inject um like sonic cube to run a scan it will actually take the artifact and scan and store it those all necessary things you need for your automated governance because what you'd like to do in the attestation is not only say this happened but possibly ta and then sha that that lock right as a mutable event in the chain though it's this is brilliant um you know and so the the model then like takes that work with some of the open source work that I've been on in this sort of model that that does collecting and testing and enforcing uses um a dsl something like a YAML based structure originally we're using grapheus I'll talk about that um it doesn't necessarily have to be with kubernetes or open shift but like in our examples we're using opa and rago um and then this is the tscc project it's actually being renamed um to um to this um you know tsc python project and uh and the other thing again this is like what's really cool about we're gonna rent out there's another team that was building basically um you know what it's called a verifiable audit data store which basically based on some of google's work with something called trillian which again i'm going really fast but you can look it up is um basically a true immutable storage structure in other words like um like that it just can't be changed right like so now like instead of putting our attestations in in this open source grapheus tool which you know has worked very well for the last couple years for us but now we're trying to look because this is based on a merkle tree and it's just it's just like the perfect solution for um you know and some of this stuff comes out of what what uh google's doing it actually a lot of it came from a certificate transparency and and key transparency but like you can also use it for this idea of um verifiable audit data structure it's just really you put all this together it's really cool so here's the sort of obligatory that you're going to talk about security got to mention solar winds but i have a real sort of point here so um you know i i um had the opportunity to do some research about the solar pinsting and i won't sort of talk about why i had to do that but you let your imagination be your guide um and i you know i went out a lot of places and like kpmg had some good stuff and you know crev's on software is always good in fact he's the one that pointed me to this the crowd strike and they to me they did a fantastic job of how the original kill chain happened inside of solar winds this is like before it gets out to the wildness everybody in it it's an ingenious i mean it like yet to shannon's channel is his point man the adversary analysis is overdue because these people are ruthlessly intuitive and and creative um and so i won't go through the whole thing but like one of the things in the blog article which is brilliant is they use if you're familiar with the mitre um you know the mitre attack framework um like which is we're using heavily in our cloud automated governance project um so i i you know so i will say you know this is my sort of matrix but i took directly some of their matrix and sort of merged it with with so here's the point what i wanted to show is how automated governance might have been very helpful and instrumental for solar winds based on the attack surface here so um we'll go through a couple of these but like the two meta points and these slides will be variable so you can have fun sort of finding original article and see how i said you know this attestation would have been like incredibly helpful to them the one of the things that um the crowd source crowdstrike started to point out which was there were all sorts of hash mismatches so it's just in short they hijacked ms build they basically uh went ahead and you know sort of inserted their own code into the build they did some log masquerading i mean they did just crazy intelligent stuff right and they were doing it for many years too so they had a lot of time to perfect their system um and and all i was saying is that it had they had a really sort of mature pipeline structure which you know we don't know but have to assume they didn't have proper hygiene and um but even furthermore if they had automated governance they could have had you know there could have been like signing mismatch code signing mismatches um if they were to use something like trillion as the attestation so like you couldn't it's a miracle you cannot mutate it like so some of the mutations that they did for log masquerading um so um you know so i i tried to sort of put in like their sort of observation and you know in their tactic and id from a minor perspective so i thought this was fun and you know and i think it to me you know again i'm taking advantage of the situation you know hello and say oh pay attention everybody but but i really wanted to make the point that that you know had they had automated governance not automated governance and saw everything like like i'm not going closer to that but it's a level of hygiene that can produce sort of a bill of material or something that that you don't like this idea that that like when i go in and you know shannon was saying this to me too like when you go in you talk to people and say what's your evidence that you're actually doing this correctly and and like and if you can't even show me some form of a bill of material for everything that you create from a software factory perspective then you're not you're wrong um yeah that's i think that's it so um i covered everything i wanted to cover and uh so you covered more than everything you wanted to cover there there's there's enough tips and tricks here and leads to other topics that we could probably go on for uh a month of fridays um so uh thank you very much for this i think this is it's what what i'm gonna have fun with is getting all of the um additional resource links for this yeah cobbled together i think um so thank you very much for coming john um you know there were a couple of questions i think i handled mostly about you know how do i get a certification and dev sec ops um uh from out of red hat and you know that's an interesting question in that you know how would you certify someone in this topic there's there's so much to it you know i have dear friends dear friends that i really like that actually do this right and so i i try to be sort of like split the middle but i you can't you can't certify that i mean again um if the if the goal is to sort of you've got an uninformed organization that requires it and and you're gonna play this silly game because it helps you achieve the things that you don't know why this keeps happening but i better turn this off but um sorry about that um you know then then yeah then like god bless you just do it right but but but the truth of the matter is like i said earlier when i i try to sit down with really intelligent people and say like what is the holistic landscape of dev sec ops is it identity is it authorization is it you know and even in the identity space is it secrets management is it um is it certificates is it um you know is it encryption is it encryption at rest is it um you know does it include discussions about vaults does it do discussions about um spiffy and the service mesh i'm like like right and and okay so let's take another topic audit okay you know like forget it we can spend you know months on audit like do you gotta know aren't you i would say no you know like that's a sort of old way of thinking you have no three lines of defense if we talk about if we talk about cloud governance you know then oh god like you know then like um you know like what are all the sort of things in in in in sims or or sores or you know how do you create cyber data lakes and then you know what role does sort of the traditional sims and sores give you versus now all the stuff you get from sort of amazon web services that are all very immature from that perspective and then even if you got that right i mean like again i'm i know i'm over rotating on this but i want to just make a point why certification is in general nonsense because even if i got all that right about amazon what if you're going to use google and microsoft yeah because that's all another so i mean when we just use the banner i mean in dev ops i would give you a little bit of wiggle room on a certification because i would say okay if you understood the meta if a certification test i still sort of fundamentally disagree with these certifications but if you understood the sort of meta around certain principles that i think were very important to have in dev ops you know sort of the culture the automation those things you know i mean daemon coin something called cams years ago course all the measurements like if you understand those things as as primitives of behavior and how you how the technology and the social social social technical aspects of it like i think you can answer those questions to do a pass fail maybe but but i think when you get in the security you're like evidence that even the security people make fun of their their i just don't see how you um i mean even the training you have to just go for sands institute just to learn traditional security yeah all right yeah i gotta say don't get me started but i've already started i know i i love getting you started that's like i love you know working you up on this but i think the the short answer is like red hat certification is specific to products for the most part there's some you know and so like there is a you know certification as a specialist in security for containers and open shift container on the open and the open shift container platform where you learn all the security bits of that platform of that technology but it's not the cultural thing it's not the you know automating all of um you know the the security and governance and the audit and the risk and you know and that's a great point diane right because i mean as much as i rant but you're absolutely right not because i were read up but you're right i mean if if the question is like we you know we require stack rocks right and then like like you know will we or you know we will have a certification about the sort of how to use that tool effectively and even some sort of meta aspects of using it that's a valid certification right or or any other sort of specific tool that we have like container you know our version of sort of container adoption and so like that i agree but i think i sort of drives me nuts on facebook when somebody posts there like i am dev sec ops certified you know like yeah like these graduates say like tell me what you know and how you get all this out because i've been struggling for at least on the security question for like five or six years to try to figure out what to and and it was this presentation that finally got me to at least create those three boiler plates that i talked about like and that's taken like five years for me to get to you and build that slide to say in my mind it's it's risk defense and trust yeah so it like if you go back to your you know you don't you don't have to do this but if you go back to your like your software factory uh slide you know there are pieces of technology in that that people should get certified on you know and should know in depth um but it's it and and this is kind of why i have you on on fridays when we're talking about organizational and transformational and systems thinking and and higher meta things um that these things are um a level above the traditional sort of things that you can get certified on um and but that's gto too right we didn't spend a whole lot of time i know our team is you know on this um this forum quite often right but like we're sort of about like we didn't get hired because we were like red hat experts or we knew sento so you know you know like we got hired because we you know our all four of us you know and to play safer if you don't know kevin bearer's co-op at fiends project and j bluins probably one of the smartest guys you know one of the top three smartest people i've ever met um the um you know like we're all you know we try to bridge that technical and social right we try to you know make sure that we're not over rotating on meta but we're also like you know so we're trying to hit the middle ground and that's that's what we do here at red hat you know and that's our sort of bullhorn too so and that's why it's wonderful having you here and having you back this week i i think that's a great way to wrap this up too i think um that's what you bring and that's what you know organizations have to also wrestle with that um this is both a techno technology issue and problems and things that we can solve with some training and some certifications but it's also a meta problem and a cultural problem um or initiative within companies that um we're thrilled to have you guys here um helping us um work with all of our customers and ourselves um internally too to wrestle with these things and help um drive them forward so thanks again for coming here today and um we'll make this this this uh session available up on youtube and um tell us again john what was the the um conference that you're going to be giving this presentation again oh so i am confirmed for swamp up i think that's in may that's j frog so they're great people i just i've been speaking at their conferences for quite a few years now they just they've got this great community and just their ceo is just a lovable guy um and i just i like that environment and uh so i just have a blast and so i i speak there almost every year so i'll be giving this presentation at their swamp up and i i don't know if i'm confirmed yet for the dev ops enterprise summit but it'll be the first time in in since its existence that i wouldn't be speaking if it isn't accepted but that that'll be uh basically what we call the virtual London and i think will be april may and then they'll be i'll probably repeat that at the end of the year for the virtual vague hopefully it's maybe it's not virtual vague by the time i get to vegas so i don't know probably but so i'll work with you john to get some of the you know make sure i get the right um references in for this and if anyone out there is watching this after the fact and has feedback um for john reach out to him here's his contact information and um join him for further conversations on this topic and yeah i'd love any like i said the first time i've given it like so that did i over rotate on something that you i would love any sort of really hard constructive feedback would be very helpful for me all right thank you diane thank you for you doing what you do like it's great to have this forum to to to come and experiment and you know just be part of the community so love giving away the podium so if any of you are out there and listening to this and have another take on it or a deeper dive that you want to do just reach out and i'll definitely give away the podium so thanks again recording has stopped recording has stopped yeah so um john i'm going to edit a little bit off at the beginning where i was talking too much at the beginning i felt um and if there's anything in here that you you felt like you over rotated on um once i do you know let me know and i can uh i think i'm