 Hey, good morning all Yeah, so this is a little bit of a departure, but I'll explain that I called the 7th deadly disease and something I've been doing in the last couple years called the 7 deadly diseases of DevOps and My name is John Willis. I'm new to red hat. I'm part of what we call a global transition office with Andrew Schaefer one of the founders of puppet labs Andrew and I with another person created the first DevOps day in the US. I was the only American at the first DevOps day Again, he was puppet labs. I'll give my background a minute and then Kevin bear who is the co-author of the Phoenix project and J. Bloom has been working with Kevin for years at PhD and a gizillion things anyway So just very quickly who I am just so they sort of you know, why am I up here? We won't over rotate it 40 years. I'm pretty old. I've done start. I've done 11 startups I was actually first private cloud. This is pre Open a stack. I was getting confused open stack My brain says open stack when I say open shift and I'm just just terrible. No, I don't work at red hat The but worked on the first. It was a eucalyptus. You're bound to enterprise cloud. It was terrible I was just seven person in chef helped build that sort of technical system I sold a company to Dell that you've never heard of I sold a company to Docker that you never heard of I'm really good at building companies of that people never heard of but it was a So for the fine networking tool, so I spent like two and a half year two years And he what Solomon called the eye of the hurricane Docker and then for three years. I was doing this seven deadly diseases stuff October I came on board of red hat then 11 startups Again, I'm old for decades of technology actually started my careers and I'd be a mainframe developer I went to 12 books and I am considered one of sort of progenitors of the DevOps movement. Anyway, they're good Oh, yeah, a couple shameless pugs. I did a out of all the books This is an audio only if you're into like lean and and resilience and those things and how I affect ups Me and Jean did an audio only book one audible credit y'all just one audible credit I got two kids in college so The I'm gonna come back to this So Jean I came if you don't know the author the Phoenix project I've been fortunate to know him as a good friend and a collaborator now almost 10 years and He wrote the Phoenix project the every year he invites about 40 of us up to Portland to work on this sort of study group things And we we produce usually ebooks You know 50 70 pages. There's about 30 of them out there for the last seven years It's a great IT revolution forum papers like brilliant papers from lots of really big companies So this year I decided to sort of do one on something I've been thinking about what's automatic governance And like I said, I'll come back to it, but there's a creative comments book in the end I'll talk about like what this is. So that's if you IT revolution Again, you find me Jay Willis at red hat like I'll give you any resources that you need But so one of the things that I talked about the definitely diseases too. So what happened was I left Docker I'd been sort of I spent a good part of my career operations transitions Back at age where the technology was so terrible that you could never really transition anybody And then for about getting you know little short of 15 years. I went into vendor land you know the the sort of chef and canonical and and and then I was leaving Docker I Noticed that ship was sort of sinking a little earlier than most people anyway, I Trust me like I've lost a lot of money on that deal. But anyway So like if you're like, hey, why are you making fun of Docker? Like talk to me on the break The so I thought okay like I've been hanging out with all these people who have been doing sort of dev ops I'm part of Gene Kim's dev ops enterprise summit, which is only enterprises, you know In working with them and I took all the toolkits and I wrote the dev ops handbook And I like okay I'm gonna go out out of end the world be completely independent and really sort of change the world And I had all these notions of these tools that we use for dev ops lean value stream mapping all these things, right? I've got lots of presentations on this stuff. I found the first bank. I was actually was a capital market company I went to and I kept finding that all these sort of prescriptive notions of how to get all I really wanted was truth Like I just wanted to find out exactly how you were operating In the minute I did a value stream mapping thing. I lost a whole layer of truth, right? So I I literally I didn't have a name for it and it was a terrible business model Because I couldn't sell it to anybody only about three banks bought into it But they were the biggest banks in the world, right? So, you know, I didn't have you 400 500 people over a month period just finding out these incredible sort of cracks in the armor of like, how do you really operate and I came up with this and I have full presentations I only have probably 20 minutes total, which is really impossible for me to do but these sort of seven models and I call it seven deadly these because it's clever it gets you selected for keynotes because It's a cool title sevens a cool number But they were they were patterns that just absolutely repeated over and over and over now that for the purposes in this presentation I really wanted to focus in on sort of the last deadly disease, which really so it's a compliant certain compliance dinner in other words your Hey, CIO your audits are Fill in the blank, right? They're terrible. They don't match reality. They don't like I haven't been to a bank capital market or insurance company where when I talk to people on how they're doing things and And how their audits are it's like clearly disconnected, right? It was interesting because all this funneled into it again There are longer versions of this, you know, what I call in fact I'm probably gonna do an audio book on the seven deadly disease with gene, but here's the thing, right? So for those of you who follow the Phoenix project it was based on a book called the goal by Ellie Gorat Who who's focused on sort of theory constraints and I'll throw a lot of stuff at you right now, but The Ellie Gorat had this great audio series about 20 years after you wrote the goal Which again is sort of if you've done industrial engineering degree You probably were forced to read this book and if you read the Phoenix project like oh my god It's the same story, but only a Java stack and a system programmer, but he had this thing that um He's about complexity and how certain scientists like particularly physics think about complexity and in during the thing He asked this question like here and he has a slide deck that it goes with he says which system is more complex a or b and Sort of a social science or not a physicist I'm not a physicist not even gonna even to try to behave like I act like one on TV But the most people would probably fall into its system B and And he would say that a physicist or somebody who looks at complex systems would say its system a Big it has more degrees of freedom Right like this one is already starting to try to tell you a story Right that one. There's no story You know and this is why I never run it. I always run out of time My dad used to my dad was like this like you know like he didn't really do anything But he'd say you know what son veterinarians are the smartest people the stars can't tell him what's wrong with him, you know you know I had to get that one in that was ad hoc too. I'm here every Tuesday night tippy waiters But here's the thing right so when I go into a company and I'm trying to look for these patterns, right? Most people want to tell me the positive somebody wanted a deposit because oh my goodness this guy's gonna get me fired But but also it's like, you know, it's like it's fine John Yes, you know, they're trying to describe the already abstraction layer that's taking me to a place where it's not fine Sir, you know that like it really is on fire. How do I get there? Alright, so let's go into security Like so, you know, I wouldn't be I wouldn't torture say how many you can tell me what this is It's hard to read even if I said that It's the vulnerability. It took Equifax down Right, it's the struts to Jakarta that you could literally send in depending on where the the Tomcat server was hosted Literally that command was running and you could do you know, that's just an example echo of in but I've been echo But it could have been like whatever Right and and so it was funny though is As this came out, you know after they realized, you know, eight months later that they had the vulnerability and it lost tons of money And and again, I'm not sure Equifax like most people know the Equifax was was the most costly breach In the history of all systems play if it was they lost like five billion in market cap Right and the the recovery is another story But then it's sort of the answer was you know still investigating, but I think we got it There was a person who failed to patch the system That's ridiculous Yeah, yeah, but given this is what we do right and we we tend to sort of abstract of everything and and you know And the thing was that um You know, and you hear this a lot right like I'll give you another example of one more recently But so there's this brilliant paper and normally this group doesn't do a lot of brilliant work But but this is a really good the end of 2018 right? Yes 2018 yeah, they Congress US Congress did Sort of a retrospective of that breach and it's brilliant. I mean it really is brilliant I mean they they walk through the kill chain. They do everything's really cool because the things are in it This is what I love like the IDS systems the systems are supposed to catch man out anomalies and protect the perimeter Had 18 month expired certs right the CISO Worked for the chief operating office or cheat not worse than that chief legal officer So in the testimony when she was asked Why didn't you notify the CIO of the breach by the way the the CIO had left for like a two-week You know sail around the Mediterranean She said why didn't think of it And like again like you say well, oh my god fire But the the real answer was they sort of created an organizational structure They gave that answer before they even started in her head was and it could have been anybody It was I need to think about what my boss the legal and the legal implications it is She wasn't thinking about it So there was just a number of people who do forensic on airplane crashes like they don't look for pilot air They look for all the things that possibly went wrong, right? So again threading on how do you get to this truth? But sort of just digging and digging and digging and don't accept first-order second-order or third-order answers from people Again, most people are the capital breach right it was it was You know, I think summer last year and again here the answer was it was basically an inside attack from an ex-Amazon Person who like this, you know, we got it. It's good. We'll just make sure this doesn't happen again But the truth of the matter is Anybody in the room could have done this with two commands And I'm oversimplifying but basically what happened was Somebody who's in like hurry-up mode had to get something in had to basically break process They didn't have XPCs. They put up our own WAF an open-source WAF And they took the defaults and the default had bypass on and so if you know what that is It's basically a bypass to guess what the Amazon metadata server And they happen to pick a VPC group that was all we'll use the one we used last time it happened to be authorized and For a couple of weeks anybody in this room could have basically dumped all the VPC and WAF credentials Which that's one command from outside in Right, so like there's a contrast between the difference between and then there's a whole lot of things that led up to why that happened, right? And so really what this was what they call server side request forgery The point being you have to sort of dig a little deeper In sort of the frenzied or whatever to sort of get to the bottom I talked about that beyond the Fiat's project one of these we cover pretty heavily in that audio book Which we look at some of the people who who actually go out and do forensics on airplane crashes or Catastrophes in hospitals where they use systems thinking to figure out how to really how do you really get to the bottom of? How things really work or don't work and and so I've got a longer version it is because I do want to get to sort of the end Part when we get back to this automated governance But what I find when I do these interviews with all these companies and just get to talk to everybody, right? I was there like let me talk to the edge the people who put their fingers on the keyboard And really sort of listen to the executives, but really find out what people like they're out of their mind They have no idea that we have to do this and this and this right because of their incredibly terrible decisions But um, you know so toil especially in enterprise right like this is a thing I don't like like God bless anybody who's doing a Greenfield startup and like I've done plenty of those I've lost kind of money. I made some money like I'm really interested in the ugliest of the ugliest of the ugliest enterprises The ones that basically have Spreadsheets of policy that are basically hold on to your seat 400 columns long Right that somehow have to and by the way 90% of them were written in the 90s Right like, you know like I love this one. I'm gonna have to steal a little time down the the one I my favorite all-time one right is Having to write a business continuity description for storing data on S3 All right, so if you know sort of data at rest in S3 is like 30 something nines That means that every molecule in the universe except one would be destroyed and then the data is lost But every time because there's some NFR to sit in stone It's in a policy that nobody can have a medium of discussion about like why can't we get rid of this? How many people copy and paste like monster? Clips into forms to get application Raise your hand if you've ever done that come on everybody else is lying lying You know I told right like it's a thing right like how do we sort of look at cloud native and idle right there like They're two like ships going this Pulling apart. I mean they are I Mean, you know, I mean again. I'm not an anti idle thing right like it's a brick-and-mortar But like like it's not even talking about service mesh and you know node discovery and you know I mean, you know API extensibility Kubernetes operators like like let's talk about serverless I mean like there's like no glue between what most institutions have in their sort of 400 column spreadsheet and what actually is I say this there's like the the idle people on the sixth floor the continuity people are on the seventh floor Right, and you know so and you dev ops are on the eighth floor and they all use different elevators And they don't talk to each other and they all use overloaded terms That's the reality of most large banks You know the risk right we still in myopic about perimeter based Even if our searcher expired for 18 months the I'll talk about subjective governance models You know low attestation You know, I think like I always I dig like sort of twist lock a queer. I know queers open source They those are like will wake you up go look at their reference Description and you realize oh my god, I got a lot of new problems because configuration in this world is a new attack vector There are so many ways you can get yourself in trouble by missing a comma or taking a default Configuring and we go on and on about sort of Docker Kubernetes And only that is another whole thing running at you which is like your your sassas service now configuration those things Not all the source, but they're attack vectors that are new misconfigurations and things like service now or social calm a new Scary attack vectors configuration blind spots And inconsistency like it is unfathomable you to be walking a larger operation and not have a consistent definition For some form of infrastructure as code chef puppet answer or whatever Like that not be sorted DNA based in today's world. It's just you know where organizations in some cases still waiting Like six weeks to get a server And that's a VM Like I thought we solved that problem You know, I swear I will retire when somebody says I need like a I need sort of a computing since in a cluster For a Kubernetes and it takes four weeks. I'm done. I'm out of here like I have to find a new line of work So the opportunities again, I'm gonna sort of zero trust for risk beyond I mean shift left security Data ops. I'm gonna be talking a lot about data ops late this year Like how do you do attestation models for data? Those data ops is sort of adding supply chain to move into data very interesting subject Like let's not just move data anyway, but by the way all the major breaches that I've studied it wasn't like You know struts to or some vulnerability in node I mean it was but what was it was the data and it didn't have any problems in half the time It's developers who are copying data at will to different places and forgetting to clean them up or half Anonymizing it or no no attestation for how the data got there You know consistency I talked about this Again, I go on on about sort of date driven versus velocity and the hijinks that come out of people's mounts When they try to tell me they're velocity based and I'm like, no, no you're date based No, no, no John. We do story points. I'm like give it to me. It's like don't like please don't go there So the deadliest disease we finish I talked about security appliance theater There's a boiled plate. There's um, you know, one reason I'm involved with this with Diane, which she does wonderful work Is I've got a sequel dev sec ops. I'll put this on the last slide. Let's So join the party if you will You know, you'll review boards. I say like you're a suck factor of your company It's a multiple of how many review boards you have to do go through to get your sort of project Appointed the arm the XR the prom like I want coming in XML board review They they had literally created application was so ingrained with their job schedule or like they literally for major changes Had to get not only be all the other review boards and if it's only use it on me They got said like all review boards will be on Wednesday, right? So A vulnerability theater policy theater right again back to this so one of the things that I started thinking about two years ago I was in his dev sec ops. I'm like, yeah, there's a lot of pieces here. What about identity? I just like I don't know somebody else's job like I have to I say like what like could I get my teeth around in sort of dev sec ops I'm gonna solve all the problems was one that would really sort of I think have some teeth and I I started I actually started out with a large bank that was sort of doing this as A gating system they were defining sort of gates if you will or we call them control points that said that like in order to get Auto-proof you had to evidence these 12 or 15 or 20 control points And the ones that you would obviously think about in a dev ops discussion like it had to come a source control There was a pairing on a pull request. It had to have a green build It had to have a green vulnerability scare right on and on and on right like and so I would thought about like what Orders do right? Orders sort of come in and look at a change record And they follow this subjective discussion of Like people sign off Bob says yeah when I make this change on this like Two and a half billion dollar budget IT system Let's think about that Bob is gonna make a change on a two and a half billion dollar year But IT infrastructure, and he's gonna describe it in a change record And then the Sue is gonna look it on the board and go hey Bob Can you give me like two more sentences on that s3 thing? And then Joe is gonna go not on my system Tell Bob he needs to put another paragraph in about this Then the order comes in it says Subam okay. Yeah, give me screen prints. I trust any yeah, right and Like 2020 screen prints for audits, right? Well the people laughing are the ones like no, this is true, right? And so then so I said okay like if these banks are building these sort of models for sort of gating Shouldn't those be attestations so change the model from a subjective discussion to an objective model could we actually build evidence in the pipeline That meets our sort of what are you know sort of GRC governance risk compliance a risk policy and so I Back to this book. I invited to last year's Gene Kim sort of pajama party. We call it the I invited Nike Capital One Marriott Sam Guggenho at At Microsoft PNC and Mike Nygaard over at Saber and for two and a half days Which we said three things we try to look at one was Could we increase the efficacy of an audit actually the first thing first which was could we turn a 30-day audit into a half-day audit? Like could we do that like I mean not like solve the world But who we write a reference architecture where we could actually create a reasonable discussion to like everybody in this room to say Okay, I think that makes sense second egg who we increase the efficacy of an audit from nonsense You know 20% to maybe 90% efficacy a third which is the sort of dying Sort of you know kitten if you will of DevOps, which is Going to Seattle and say get rid of the change of eyes report and like how I Gotta know what it eat dumb idiot, right? Like so could we actually between those companies? Like sit down and write and we did and we sort of created You know again, it's it's you know if you if you go to it revolution.com. There'll be a second called forum papers I'll have this up a reference to it on the sig Shortly and it's a it's a creative commons. He shows you what we did 75 attestations, you know, and nobody would run all those 75 but the It was sort of the aggregate of like at Meret. We would do it this way You know and Microsoft for like how we run is your we do it this way and so it's a kitchen sink And we did is we had we had to come up with like our we weren't trying to do another PowerPoint slide to show you how the pipeline looks The like the 480 with version we were sitting back and thinking like we don't care what it looks like everybody else What are the sort of boundaries if you were thinking about attestations and as long as discussion here We described why we did this it matches most of what you said the biggest difference probably most is the difference We build in package because in today's world the packages, you know Either sort of a jar file or your file or a container image, right? Like, you know, so there's this sort of new world that that deserves its own sort of boundary marker for attestations And then we had so control points was the terminology very well described. We did our disclaimers all that stuff It really um My only credit was given to really really smart people for big companies in a room One of these one of the companies who's followed on left and is actually implemented this Using a consultant firm and so we found that like once you get this thing kind of running You start seeing like is that idea like you look you finally can see over one fence And you see the next fence down the road you didn't see so he's really cool things that are popping up one is Now they're able to create these sort of Yammer files for policy people So the policy so what you have now is you know the in most organizations you have this so again another Subjective discussion about policy somebody understands polity so somebody understands it to somebody who can implement it to this So the idea is what if we could create sort of a manual right the ammo files for policy people to inject that Automatically in the pipeline none of this is seamless. It's all embryonic But like that that idea that we could cut out that sort of that multi-step And then the other thing is you know talking sort of big five auditors like like could there be a template that a big five Order just says a PSI DSS template. Yep use this one again all these are like open discussions and I think this is the really coolest thing and Which is one of the things we found with implementing this at a large bank was when you go through this You know a lot of what I talk about and we talk about lean as it maps to sort of dev ops to sort of turn We talked about commit to deploy And the truth is the whole dev ops discussion and all these things sort of fall apart at ideation And I know there's a longer discussion there There's a lot of things like if I'll have an argument with like an old lean IT or you know David J. Anderson who event in Kanban like you like like lean manufacturing does not map IT And I'm like baloney like from commit to deploy I can map it brilliant But what he means from all the ways from ideation and they're right to the point What we have found is you can build attestation models in ideation Now this gets really brilliant and if you want to talk to me on a break I can tell you how if this is a subject you're interested It sounds that 1990s 400 column spreadsheet problem, and I'm writing dynamo DB apps on Amazon today How do I make that happen without having committees aboard? I'll leave that as a cliffhanger if you want to talk me on a break. I can give you more Because I got like one more slide. So they had this thing about the octopus We don't even know what's out there. So the consulting company they've been working on the implementation of that reference architecture at PAZ Bank Or literally last week One of these we tried to use is graphius, and it really didn't work right I talked to the Google people. I'm gonna see if they can like change the model. It didn't work for what we needed You know, I think it's great. It's the only attestation data store out there. It's open source But it's just modeled incorrectly for how we need like for example one of the attestations might be a successful build Which might be a shot of The bill lock a tarred the bill lock shot that now is in a chain of events And by the way never used to wear blockchain just make it look like blockchain. Just don't call it blockchain So they actually wrote a sort of a nice abstraction, which is sort of a fork of graphius, but not quite But it's like it last week Right so I mean they had the advantages like I fed I was an advisor for them I fed on this they were working on the implementation now They have actually a malleable model for attestations open source I Also mentioned my good friend Nick lipis who runs on argue haven't heard it It's one of the largest regional network user groups. It's mostly large banks out in New York They've been doing this for years. If you look at the board advisors It's just insane seesaws and and network architecture designers from like every big bank in New York I haven't mentioned to him this thing and immediately like four of his biggest sort of Advisors who like let's do a cloud version of this and the thing is my automated governance doesn't care whether it's cloud Whether it's data whether I think my the one I've been sort of driving like it's agnostic people like will it work for past? Yes, well, no, it's just a model of attestations And so I don't care what whether it's cloud or not cloud or pass or not pass But what I found is we started this working group Hopefully in May will this will all be open and Creative Commons as well is we're gonna sort of drive a whole cloud Specific discussion and even though I was sort of saying I don't want this to be cloud What I found was so this is not open yet But like once I found like these, you know the cloud security VP cloud security from a really large bank was like yeah, this is great. Love your model John But like what about trust models? What about um, yeah serverless, let's you know secrets manager as a part of this or regional standards, you know some of the language of multi-cloud providence and localization anyway So it's been fun for me just sort of sitting in that and listening to sort of CISOs Talk about like I like this model, but can we add that these kind of things? So this will all be public And anyway, so you know follow me at the sort of the DevSecOpsIG The Google group and I'm Jay Willis at Red Hat. Thank you so much for