 All right. Good morning. Good afternoon. Good evening everyone wherever you are hailing from Welcome to the Red Hat OpenShift which I am very very very excited to be joined by not just John Willis the person but John Willis the friend John Willis the mentor and John Willis The guy used to work with and now we're coworkers again So, you know, I'm a technical marketing manager here at Red Hat and I will let John introduce himself Hey Chris, it's great. You know, I think you're talking about my you know My kids I was talking about twitch and I really didn't fully understand what it was But now I could just go back and brag to him and say hey, I've been on twitch guy. You've been on twitch. That's right You've got your own thing. Yeah, my twitch badges with my youngins Yeah, no, you know, my name is John Willis. I go by the handle of botch a loop on Twitter I've been doing this mucky stuff that we call IT specifically around operations for 40 years I'm pretty old But I would say the last 10 years. I've been very focused on maybe 15 years on on really the what we've been calling so Precursor to DevOps DevOps cloud Distributed computing if you will and that's you know, I won't give you my boring history prior to that But mainframes and all sorts of crazy stuff, but you know about somewhere less than 15 years ago I got involved in open source projects that related to Systems management, you know, you know some of the original open source tooling. I got into infrastructures code stuff CF engine puppet chef Ran that mile cloud and then Yeah, just been deeply involved. I built I mean I've been done startups my whole life Mostly failed miserable failures. I mean like most startups If they talk about the whatever doesn't kill you make you stronger like that's me for startups But the last five years or six years have been really good to me I sold a company of Dell I sold the company of Docker and and so anyway It's just long story short last October a good friend of mine and other DevOps luminary if you will Andrew Clay Schaefer Said hey, what would you think about coming with me over the red hat and I can be perfectly honest I was like Andrew. Did you but I'm the wrong person? Yeah small company kind of guy You know, okay Andrew like if it was anybody other than you like we wouldn't have this conversation But and then he said he was forming a group with Kevin Barry and I've known Kevin ever and he's He's you know, he's amazing person. He's co-author of the Phoenix project. Yeah and and we you know we You know, I got that me Jim, you know, and you know, like Kim my buddy You know, I met him three times, but but you know, it was Andrew's like, you know, let me show you how serious I am and red hat is about this. Would you want to get on the call with Jim wire? Okay, that's pretty serious. Yeah. Yeah. And this was for frame of reference because Jim is president at IBM now This was back what October? Yeah, it was October last year the conversation. We started in October And and and I fell in love and I had read open org and you know, I knew I knew his backstory because I know I'm also like Like, you know, like, you know, I have this on my day job and then there's my family and then there's like three jobs I do so about two in the morning And and a geek about aviation and Delta and all that stuff has always been yeah Yeah, so same same kind of thing. Yeah, his background was just fascinating to me and then they get to meet him in person Was just so uh, you know, I was telling somebody just the other day that he's the real deal You know, like there's nothing if you see him on a sort of promo where he's taking a call an incident call Right. He's taking an incident call. I mean, I I fell in love with him and you know, so the stars aligned and we We were brought in as this global transformation office And uh, you know, we're we're trying to find our way and see if we can't help You know, one of the things that we talked about is You know, we've done a great job in DevOps, right? Like 10 years 2020 all pat ourselves on the back It's you know, it's a solar giants. It's a collective You know, people say, you know, john's one of the founders of doubts. Yeah. Well, I was there. Yeah Very early. I was the only American in in the original conversation, but Like don't ever let anybody's imply that it wasn't thousands and thousands of people exactly who helped build this But but here's the thing in 2020 Okay, we've done good, right? Like like everybody take a moment Pat yourself. Yeah. Yeah. Yeah, but but you know, like and and again, no disrespect to sort I use this as an example because I do think get ops and Are great things but if five years from now if we're still just ci cd and get ops Then we fail Yes time in 2020 and this is what gto What is all about is in 2020? Let's think about the story for the next 10 years And and that's what we've been trying to do and we're trying to meld the ideas of the existing red hat You know, you know me and I you know getting to work for you is always a blast We've got to do some really interesting projects together and yeah, your ace is in my book, bud. So Well, thank you, sir I appreciate that now just to give everybody context john and I worked for a startup together What two or three years ago now? It was a while ago. I was down on years, man. Yeah But yeah, we we worked together on some projects john was actually my boss for a little while and It was it was a lot of fun. Let me give you a heads up, right? At least one thing I'll tell you know, I'll go in and like, you know, again I I don't know how like I guess we don't have to be politically correct per se, but like I it kills me Be nice to our competitors and everything else But yeah, yeah But like one of the things is that like I go in environments and I see what they pay for spunk, right? And you know me and you but you did the heavy lifting You replace spunk in a lot and like a top 10 bank with uh without I mean that's That's a pretty impressive resume Thing, you know, like the idea was the the guy who championed it You know, I was able to get the deal in but You know, we put you in there and like you not you banged it out and that bank today is basically taking spunk off the balance sheet Yeah, no that they're they're saving a ton of money now and it it was just you know Having grown up in, you know, north Carolina and spent so much time working with linux and being in raleigh at the time You know when when we were working together. Well, actually, no, I was up here But you know having spent so much time around red hat and it being just kind of a red hat company Walking in the door. It was kind of like all right. This is what you want to do Well here, you know, move the objects out of my way first and we can get it done So, yeah, uh, it was funny one of the first projects that I worked on there to give me an idea of how like Far behind they were in their dev ops journey like they They had a it was a remote position and you know, I'd come on site every you know a few weeks or so But they literally had a problem Confirming whether or not the VMs they got from ops Were of the size they requested right like keep in mind This is put in via a ticket and the first piece of code They wanted me to write was an actual answerable playbook that they could run To verify that the system meets the requirements of the environment. It's about to go in I mean that tells you how like Back off the dev ops scale. They were right like they didn't not only did it take weeks to order VMs But they would come in and be the wrong size or the wrong specs or whatever And yeah, like having to go through that I think was very eye-opening for me from a dev ops perspective Having gone through so many companies where like that was just table stakes, right? Like if if you wanted a VM, you just got it Well, you know, there's not that we do on talk about automatic governance But not to take too much of spin, you know as I go in and I do these sort of organizational forensics or what I did before I'm in red hat and you just interview thousands and thousands of people Not thousands hundreds and hundreds of people 400 500 people But um, it's that trust thing, right? Like I could see why somebody'd want that. It's not just that they made a mistake It's you know, it's the You know infrastructure doesn't trust the developers, right? So I go to the wall. There's a legit wall Yeah, the developers would say, you know, yeah, we always have a problem And you know, like I don't know why like if we ask for a You know 16 gig or we ask for this amount of this space like They just trust that we're running a billion dollar revenue service And and then you go to the infrastructure and people like we can't trust them We have to cut all their requests in half. Like yeah, they support a service that brings in a billion dollars worth of revenue a year Like and yeah, yeah now that's that's a very much a common like dev ops world is There's an instance like an edict on one side against The other kind of deal and it's it's sad but true. So So cool, man, you want to talk about I don't made a governance. I hear Yeah, you know, uh, it's been uh, you know, you know that um, you know So I my career has been sort of finding new areas that you know, I'm not the first in You know, um, but you know, it's like, you know, I think somebody way back had this analogy about when they go they only go to countries where This this may not make sense now where where at least they have one mcdonalds in the country, right? You know, like they're not fully developed But they but yeah, but they have some infrastructure So I'm really good at sort of finding the people who are are working on something They very very few people are sort of aware of And um, and then it just really, you know, if it hits sort of it Sort of mind melds with me So, you know, I I did that with with with chef, you know I came in very early at chef and like I fell in love with chef You know dev ops, of course and you know cloud, of course and then um But about three or four years ago, uh shannon leitz over it into it She defined this thing called dev sec ops Really didn't want crazy about the sort of morphing of the doubt But I got to know shannon and she's incredible And and you know like you start forgetting like who cares about the name. So I got I I dove all in on on dev you know dev sec ops, right in And and I you know, I've been sort of working through a lot of these like reference architecture discussions and What does it mean to bolt down security and the pipeline, you know, all that stuff What does shift left actually look like? Yeah less security and all this stuff But it was again like anything else. I guess it's my biggest problem is I get bored easy, right? And I was Explained the the whole like last 30 years of security In context of dev ops. I'm like, yeah, like I don't want to do that like identity Yes, it's a thing, but like it's not my thing, right? And um when I found is I was starting to get a little bit Jaded on dev sec ops Not because of the name but more of like it was so such a big discussion to have and And I didn't want to be one of those people that had to go research and lane. Anyway, I want to start short I ran into a couple of banks that were like discussing this new problem, which is you know, how do you start moving? You're you're if you think about what what why do we do a lot of the risk? It's for audit, right? Like we we want to sort of prove we have this governance risk compliance We have this you know this risk posture that's bounded with our legal It's our business like is protection of the brand all these things, right? And then how do we how do we evidence that? Well, the ultimate evidence is an audit Right, but but then um, there's this like Incredible gap between what like is in some like 500 column spreadsheet Bank and I'm not joking. I've seen one. No, I've seen it too. I've worked for those companies. Yeah And and what the auditors do right now? I started calling this uh security and compliance theater, right? Oh, it's total theater. Absolutely. It can see even worse I'm talking pre cloud data It was terrible. Yeah And and we'll talk about like how cloud native just like it just leaves the universe in terms of separation But but these a couple of like really really interesting people that I've gotten know through sort of gene kim's tribe Um, you know, you know people that we we work with devox enterprise Probably at the forum And we'd have these conversations at these events about like how does this have to change about and it all comes down to this, right? When you do an audit, you're basically showing evidence and those evidence can be just tightly called attestations and so the the most modern organizations the way they They produce evidence is to what I call subjective attestation models Which means that You know, basically, you know, this everybody knows this you just haven't heard it in these like fancy Terms which is you create a change record The change record describes some human description of how things are going to happen Which right off the bat Like if you have any understanding of complex systems and all that right? Yeah bike I just landed from mars and like really I don't know if that's going to work And then that human telephone game continues somebody on our advisory board says, you know what can you give me two more Sentences about that subjective Novel of what the change is going to do They go back they do that and then somebody in operations who has to approve getting into production Says, you know what? I want a really good backup plan and I need to know You know when does it look and all this discussion is A thousand miles away decoupled from any cognitive activity deployment strategy. I mean like all right, so And and like there are people that you know So both of us know that our industry leaders in the dev outs that work for financial organizations like this is terrible We spend 30 days a year doing audits Yeah, and when sometimes worse, but you have no continuing It's always this sort of event based. Oh, you know the next 30 days everybody be prepared Um, and it actually creates anti-patterns too because people start hiding stuff on purpose Exactly right like you stick things in the in the drawer under the corner Yeah Fully a new technology because don't tell them about that piece right because right that's too Reminds me of my story at duke that I'll have to tell later, but yeah, go ahead. You keep going and so So yeah, and so that you know The question then became like oh, I get this subjective evidence And then the auditors come in you have 30 days and you know, they watch screen prints to prove things and like Really that's like they actually want like screenshots on paper kind of thing. It's yeah Backwards to anything in tech And then and so and then the efficacy of these are just laughable right like I've had zero zero point zero Yeah, zero point zero no kidding You know, I could tell you this great start out this large bank of these people are building What amazon dynamo db on an application that was going on, you know, iPhones that was Crushing in revenue and they were scared to death to bring the auditors in Because they were just like they knew there was no connection To there are so that so when we can get back to that But the point is there were really three things that were sort of hovering one was audits are just a waste of time Every it's a vicious cycle of like I'm not going to tell you stuff because if I do it creates more work for me The second efficacy is like really in we'll talk about cloud native stuff zero point zero And then and then like how do we get rid of the cab? You know, I mean or how do we start having a better conversation? About what is the point of a cab and what is the evidence and if the evidence is Sue tells jane and jane tells bob and bob tells mike Of this like abstract Discussion of what really happens in a complex system. And so we we started mulling over this idea of what what I've sort of coined as objective attestation So what if we could like what if it was very specific about the policy now we make the policy people like, okay Now can you really tell me what you want? And um, oh, oh, you want like I can't just hand you this 400 column spreadsheet Oh darn no no now I want to know exactly what you want And I will automate that so that no human touches it that all the attestations happened during the pipeline And and I you know the first rule of the blockchain fight club is never call it blockchain Right, but it's basically a form of a blockchain of immutable attestations that can be linked So now you have that And now the auditor walks in and you say well guess what this is, you know, this is basically, you know This is a technology that can't be broke. It's immutable Um next no screen prints and so the idea and so we worked on a project last november It was the the participants were no, I'm sorry not in what last april 2019 it was capital one pnc bank marriott Nike sangu kohaira from uh from microsoft and mike nighard from saver Saver group and we put together this it's a creative commons. It's out there. I'll point to it We put a first crack at what a reference architecture could be It looked like where it was sort of a clean room No humans are creating the attestations. They're all digitally signed attestations And I I mean I I love what we did there and that paper's been out for about a year And then so that's sort of one thing and then one of the banks have taken it to a whole new level They they create policies code and I we can show you So one while we work on a paper We we were having these dinner conversations and we were like wouldn't be cool if we got this to work Where we could give policy people human readable constructs To drive this automated attestation engine And so one of the banks that I work with actually went out and built this over the last year Where now the policy people write yaml And that yaml becomes the driver for the attestations and they built an enforcement engine. So that's one thing That's nice. Isn't it awesome? And then the second is Because of that I got pulled into this large open network. He used a group. They had seen the paper And now they went a little different route, which was what we call automated cloud governance similar But in this case we were asking the cloud providers to create Their evidence so that we could use in this You know, um object attestation and I can talk a little more about like we finished the paper It's going to be another creative commons and should be out in a couple of weeks about like and ask for um for the cloud providers To for evidence for certain things they do that can secure our trust or a tenants trust All the things so the in in general I can go in more detail, but in general what we're saying is We need to be able to know if they're making changes that they don't think are of consequence to the consumers But in some ways they might change network routing. They make change a posture That gives adversaries ability to attack. So it's it's right all the most exciting stuff. I've worked on in my whole career Wow, okay So, yeah, let's get off this intro slide. Let's get into it This is the the red hat team I talked about the gto guys You know, I you know the battlefield is interesting. I mean, uh, you know, I think um, you know, if you look at You know, I've taken a lot of stuff to some from derrick weeks and other people. He's over at sonatite He does great presentations on the other. I mean, this actually one is the struts too That happened a couple years ago and just yeah, I remember seeing this when it first came out and I was just like Wow, like the the amount of time we had to potentially Go right was very very minimal, but it was there Yeah, no, the data is like some of the reports that that you know, sonatite continues the software supply chain stuff is You know, like only about three percent basically deal with a zero day Within the first day Three percent or post three months 60 percent of our industry Three months three months for a zero day like not just a zero day But I mean the struts too was like a like a like a heart bleed, right? Like that was if you were using it you were done Like I mean, I mean the the struts too I think some of their data is like 40 000 organizations typically download that that vulnerable library The year Today, right? Still. Yeah, they're still downloading that, you know, there's the node stuff. I mean if you heard like the Yeah Year early become a committer With the sole goal of knowing that there was a bitcoin operator used this library Yep, you know, just you know waited waited out the time got the permission embedded some malicious code into that library, you know, I mean Yeah, like, you know being in the kubernetes community. I worry about stuff like this, right? Like I'm welcoming people in and you never know who you're bringing in. Well, and that's that's another dangerous fact and the whole configuration, you know configuration in the cloud error or cloud native error is the new scary attack vector, right? Yeah, yeah, yeah This commas or an extra stanza that somebody to copy and paste from stack overflow and like it works. Hey, you know I don't know but it's why you know, and like oh that other stuff just created incredible permissive opportunities for adversaries. I mean that in fact, you know the echo fax thing has been told so many times But I love the you know, again, I've got a lot of presentations on this I won't overwrite to but the the capital one is like incredibly interesting, right? They're all sort of the same. They're at least sort of they're they're almost like if you look at any we talked about like the sort of aviation if you look at The forensic of aviation disasters like it's they're they're very systems thinking complex adaptive Um, you know forensics and and so if you look at capital one like capital one to be like, oh, yeah It was somebody who worked for amazon they had inside. It's baloney Like you know what it was it was based on what they call a server side request forgery And it was a team that basically Had to work outside of the process for whatever reason went out and grabbed their own open source WAF Didn't configure it properly like the defaults. Look get this um, they um, so they left bypass on in the WAF And this is doing no good Well worse than this and then took sort of their last known sort of Amazon credential stuff. So they actually were working on a sort of authorized vpc An instance and yeah, like this is the perfect storm of air. It's like the airframe before seven the um everything had to go wrong so basically What ultimately happened was? um, this adversary just probably pokes every I know pokes every bank continuously Hit a time for a very short period Where they were able to do, you know, you know example.com or capital one dot com your bypass url equals the um amazon metadata server Wow, so are they just services server side request forgery and in the metadata service basically, oh, okay Well, you say you're an instance. It's authorized here. Let me tell you everything you miss to know about the security credentials I mean that's how and by the way, this is a common attack that happens because the metadata server as has some Has a lot of information that could be very useful for anybody trying to exploit things And you know one of the problems about sort of one of the you know, sort of the um You know the uh that the glass half full about amazon is They did a lot of things to make automation the prime Motivated right using it for all these days Which meant like metadata server was critical to be able to find out your ip address Right, you had this disconnected call now. I need to know stuff about you've given me a notification that the Instance is ready. I mean, but the glass is half empty is that These automation things, you know, if you if you get this sort of you know, um It's imitate somebody who's authorized That's almost every one of the major breaches that happen where you know people get into like the nsa But they got into top secret data on s3. They're always service side request forgers um Anyway, um So this is the DevOps automated governance, you know, I I tried to slap some slides together We're still, you know, working mostly off the paper is less presentation, but um, you know the idea of putting um digitally signed attestations no human kind of clean room with the objectives a short note of time increase audit efficacy and reduce cab activity And this is the guide it's out there actually under it revolution.com Foreign papers It's creative commons. Um, it really I I was literally getting, you know, sort of goose bumps When we were working on this we'd have sam guggenheim who's over at, you know, Microsoft He's like one of the the guys who's been there forever understands the all the backbone of everything they do from the security um, and then you got, you know Uh joined homes from maria You have the the the first fellow at capital one And you have pnc and they're all sitting at the whiteboard. Well, you know the way we show evidence, right? Like i'm like, oh my god Like I get to live this life to sit here and and and and so at the end it was 75 attestations No one company would do all 75. It was really an affirmation of what mariat did what pnc did what and uh, you know and Really proud of the book, um, you know, and you know, this we built a model What we did is we had to figure out like, um You know, it's been I always tell me well, this isn't like the the 101th way to show you what the pipeline looks like Have a thoughtful goal here the goal is we need to contextualize a discussion of how would you describe attestations in Breaking into stages, right like we need to be able to say like this is the checkboxes, right? Like this is the guide rails that you live in and and how do you contextualize? Uh, you know a discussion about attestations that have to be At least granular enough to be in defined in different stages Interestingly enough, you know all these experts, you know the hundreds of ways to describe pipelines in the dev ops, you know slide You know arena, you know side deck arena It took us a half a day out of a three-day discussion to really get this right Because it had to be work From like how could we? Bound attestations and you know you look at it. Oh john that seems pretty simple But like if you think about the people that are in that room, but we we we got it to Basically seven models and you notice dependency and artifact have their own cycle Yep, but you know you have your you know, you know development you build and the other thing we realized is I and we don't see a lot of sort of dev ops supply chains where they separate package And if you think about it from an attestation model, that's really important Yeah, whether it's our file or a container image, right? There is this uh mechanism of packaging and And we built the um, you know the the common actors, but really important what we call controls And uh, you know, I think this whole thing here. I thought it's subjective But I do want to get this one thing. So this is the guide again Um, did I yeah, here's this is something that's important So capital one in 2017 wrote this interesting article about called, you know Focusing on the dev ops pipeline and there was a section in there where they called creating better pipelines Okay, and and this is um, you know because capital one is a big part of gene's forum and all that and so You know and so I got to meet some of the I've been sort of asked not to mention names of capital one Sure. No, I totally understand very sensitive about you know, sort of You know the things that happened to him last year. So I'm being very, you know respectful there But these are public things here. So, um, and and this was an interesting article because it said, you know, hey You know, everybody's sort of new capital one is a poster on fact It's sort of really sad that that breach happened to wanted to probably the the poster trials of fin Yeah It's just that it is those complex Like all the things have to align in the worst way to happen and it could happen to anybody, right? But you know that breach but anyway, you know, they They wrote this um thing about how they put gates in the pipeline, right? And it really was um designed around if you wanted to have auto deploy, right? You know, sort of sort of dev ops authority, if you will You had to show evidence As the service owners the service developers That you could create evidence of the originally I think was 16 things Right that it came from source control. It was um, you had branching strategies or like, um, at least Optimum in terms of the banks, you know, architectural design You did static analysis 80 percent you get you can go through vulnerability all these things, right? And if you could show evidence of these things Then you were giving, you know more privileges about auto deploying and stuff like that But this started me thinking about well, if you're doing this anyway, why couldn't these be the actual evidence? As opposed to the change record And that's actually why I sort of created this working group Which was you know, let's you know, let's take this sort of a gating or control point model or gates and Let's build um, and the other thing that happened is Around the time I started thinking about a google open source to project called grapheus I vaguely remember that yeah Project that originally got a lot of hype and then just disappeared because nobody really understood How to use it. Yeah, and I sort as This is what they do automated governors Right, right like they're giving us this beautiful secret of and actually I found out the back history was I'm not sure This is perfect correct is um, the reason it was created was when they were creating the registry The container registry. Okay, that is a place. They had to have a lot of audit ref evidence, right? You've got you've got tags. You've got all your code right there. Everything's packaged up ready to roll So you better have that stuff So they open sourced that thing and I was like, whoa, wait a minute I think we have a great opportunity and really that was the sort of the genesis of that original project Which was you know start thinking about his grapheus and and we found there were a lot of deficiencies in grapheus I've been working with a couple of companies that just really take in this a lot further But you know as you can see like this is an example of this is right out of the creative commons book that I just talked about but at the source code repository stage the controls would be peer review unit test coverage clean dependency Right and we built everything in this sort of input output model Right because that's how it has to be right like Yeah, it's a change now build stage you'd have you know build source control immutable build You know these unit test and linting You have input output output becomes the artifacts and then you know again I won't like unless you want to stop me but the also where we found a lot of sort of really cool discussions around How do you actually manage dependency management artifact? Repositories right like there's a there's this like especially when you're talking about high consequence You know services, you know, you know customer facing top 10 bank Code right, you know the the the things that you have to start thinking about is You're not not just sort of you know Are you is it a trusted? So is it a sign trusted source? Right, is it what is their license checking built in is is the what is the age of the artifact right like having Approved version the same thing with package stage right all that stuff again I'll you know, I'll just tell people you read the book but And you know artifact stage Is it immutable right we get into at what point should everything at this at what stage should everything be mandatory immutable Right, and you know you get your prod and and and again here's sort of and the other thing we did which is really cool is We we tried to figure out like if there was going to be attestations, where would they come from? So we weren't right like one vendor over another, but we we literally walked through A matrix that sort of develop a level you might get You know at your clean dependencies from like, uh, you know a check mark for black dark or nexus, right? Or sonic cube or and you know say Anyway, um, I'll I'll sort of run through these real quick, but um, because I do and then and then um And then there was like after that one of the banks went back and said hey by the way, you know that whole thing We didn't reference guide Like we need we need a Kafka around it. Oh, of course we do. Of course you do. Of course you need How could we not um, how could we not have a production operation without Kafka? That like access station for auditors, right? I need secure. I need to secure that right And uh, and then you know, and then there's like um, one of the companies has gone a lot further And so what one of the things they're doing now not only have they gotten into policies code YAML files and all this stuff Um, they've actually now, you know, one of the things you realize we never talked about enforcement in the paper Right if you want to do this you got to do both right you got to have the Evidence and you're gonna like why not like have the and so it realizes you had to create sort of an enforcer model And and then what's really cool. Why not use opa and rego, right? So they're Using opa and right so all the stuff. This is what's so cool the stuff that I had in my head I got sort of you know, gene gives us this opportunity to go to portland every year and work on these papers I'm like, I got a good idea. You know, I quote like the we actually Like in three days built this really cool guide and then one of the banks was so energized by this They extended it to YAML files that policy people and today they have, you know, that built in as the final gate is Driven by opa You know, and he had great I mean that's opa open policy agent for everybody to edit into like it's a cncf project it is in Some degree of heading towards graduation, right? Like it is a very trusted project. Yeah, no I mean my only point is I I'm not positive. You know, I am I was scarred very early my career with prologue, you know But you know, and I know rego is not exactly prologue, but you know, I just like, you know, maybe king for day I would have started with a more Malleable language, but but other than that, right? It like you said, it's there. It's everybody's got it everybody's using Wi-Fi it and and what they've done is basically created a nice sort of um, you know translation and Relation coupled relationship between the the yaml pack files the policy is code files It's really beautiful. Um And you know, you see how all that and you know, sort of decoupling policy Uh, it was um, this is sort of an example. It's a policy is code All right, like but that's just yaml, right? Like yeah, imagine that just you know, and I Project that we've used a gherkin model this okay Very simple, but actually so you've kind of guessed the bank because they they've actually built They've got some of their stuff like graphius. There was a bunch of things about graphius That really at first we like we were gonna punt on like, okay. It was good idea, but like nobody's really Thought about how could you use like the the notion imagine that one of the attestations is taking the build log tarring it And then turning that into a you know a shot and putting that wow. Yeah Now you have real real immutable Yeah, like meta immutable Like yeah, the graphius like didn't give us that and then um And then graphius doesn't really have uh, you know a relational database or some really Permanent store, but you know what they stuck with graphius and now they've they've got like a Graphius my sequel which is awesome They've uh, they've modified their own version of graphius to sort of make work in this model And they're basically trying to contribute. I'd love to get red hat Excited about graphius. So you know again, maybe maybe maybe this thing I mean this entirely possible somebody at red hats already looking at it I don't know could be because what happened was it came out There was a whole everybody got all excited about it And then they couldn't really figure out how to fit it into their work because that was a model graphius It would like look great. You used it. It had all these sort of really honestly architectural floors for Like if you weren't google it didn't work And p&c stuck with it and now they've got it where it actually works for a bank and they have this go badge thing This is really cool too. So one of the things that um that I love about automated governance is that it's just once you get to for the first step You get to see all the things you couldn't see until you got to that step That's right Like we had to have the attestation model to be able to say. Oh, well, why can't we create re human readable? Why do we have to have policy people go to it people have it people then create? You're like, hey, I see all the office space, right? Like yeah, have you read the What exactly do you do here? That's exactly And then the other thing was that um as they started thinking about um How do you put all this together? You know one of the things that like all the stuff is there, but like the They started thinking about like it's sort of you know, everybody's cmdb out in Everywhere in the enterprise. It's just publicly, right? It's just not But yeah, you have this notion of like the service name or this mnemonic that describes This thing, but it's just if you go back to cmdb and you look at the configuration item It's like like there's no possibility in the universe to reconcile that with cloud made No, like yeah, no like the fact that you can maintain an active configuration management Database with pods just spinning up and down left and right Thousand changes, you know, I mean, I don't think it's like amazon but thousand changes, you know You know the police in an hour right or whatever, right? Yeah, like just keeping keeping that under wraps and like having that as a valid database would require another like whole System just to keep that up and running almost the equivalent size of the existing system Some of the some of the things about legacy like sort of risk is like you like even if you wanted to solve it It'd take a three-year You know all hands on deck You know vomit to solve but but here's the thing though This is why I like that what we're finding is people who are implementing it We're actually solving some of these complex problems where You know the alternative would have been a three-year summit with everybody in the company Versus emergent properties. So like in this example, they they have this go badge thing Which is it's an open source project. So in bit bucket it sort of shows but the The componentry behind it is interesting in that Basically, they've actually as part of the or DevOps automated governance So every change has to be sort of a pneumatic of the service It's got to be the component definition and the version And so now that when when you actually want to sort of create pack files or start this policy as code initiative You have to define Your component at that level. So they're actually reverse engineering a configuration management database Because the pack file can't exist without that three-level identification Right that three-level indication now is the evidence In the attestation store And now they've been able to do a simple thing like developers can see At the you know, this is sort of in bit bucket, but could have been any other tool continuous evidence Of their risk and compliance posture And then then you get into if you have that This notion of audit and replay. So now Every change that happened imagine order not only order now can just look at digital evidence Right with the back start from, you know, say grapheus and my sequel, right? That's there. In fact at the end of day They could just be hitting enter every Day or every hour and day But they can actually say I want to know a little more about this one And we know that because now we can tie that to sort of the the service mnemonic the the component id and the The the the the version right to change itself, right? Like you you can put the shine that commit shot Yeah, like you could have the the git commit show as Now I've actually solved one of the hardest problems between Sort of service management and idle and cloud native, which is how the heck do I get those to the work? Well, okay Here we go like basically if an order to walks in and says I want to see this one Okay, well, I can just go right back to the attestation store that stuff will take me right back to the history and get um, and uh, you know and so um You know, that's all there and and so now I have a mutable evidence both not only on the attestation But I have a mutable evidence back to The deployment process that we're just commonly using this, you know, get up get whatever right It's all there right so in a sense that the policy that was the actual change that you made the policy is code that was like, um the um The abstraction definition for that delivery are now completely cobbled and evident So the order can come in look at the mysql back You know sort of uh definition this fine-grained detail definition And then if they need to ask any real detail questions beyond hey, do you not trust the blockchain? Um, we could go right to the evidence history of that change the commit right and and it's it's right there You just click the link off you go. There's the change. It was approved by two reviewers that kind of thing And those were all part of the the pack file that reinforced like maybe one of the attestations is appearing on a pull request Yeah, and in fact There's even they don't all have to be automated like the you know one of the problems That comes up a lot is you know pen testing like pen testers like oh, yeah You need to pen test this like okay great. Well, uh, give me a time window. What do you mean time window? We're right like what do you you want to prepare your systems to do what exactly? We don't have the sense of a window anymore, right? Like and that's another problem Like if you look at the change records, right from the audit perspective, it's like, you know Who you know who authorized the change? Well, it's whoever's in the service owner to see him to be. Oh, yeah Right. Yeah, exactly. I had one client where the ci for all activity on amazon cloud was the account So that means from everything that they did from an audit perspective by the book from service management, which was This change happened. I had dynamo db. I did this I did what I you know, I credit s3 But whatever yeah, if the auditor came back they went to the ci and there was one ci Which was the account it was actually for they had four accounts This is like a top five bank, right? The it wasn't pnc by the way Top five bank and so the auditors would literally and I confirmed this they would literally look at any of the activity on amazon They would basically okay. Where's the who's who's the change owner the service? Wow And the service owner will be somebody who actually set up The credit card or the the the when that wouldn't be credit card at that top side bank, but the right right You know supply change structure for working with amazon who has nothing to do with anything Right, and then the second criteria for the first order is the service owner Which is like somebody who's either dead or has nothing to do with it and then the second is the change window There's no change window. It's like it's now it's yesterday. It's like in three minutes from now you know, but one of the things about pentesting then is You say, okay, well, this is real easy You do you pentesting whenever you want and here's your attestation model And if your goal is to make sure that every once a month there's a pen test against this We just throw that in as an attestation. It shows up in the continuous compliance review again, there's all these like really You know, you find all these like Opportunities, you know, once you drive down this path. Anyway, that's um We might have used up the whole hour on that one and have to come back for cloud automated governance So we got a little bit of time I guess. Yeah, we got us measured enough time. Yeah, go on All right, so that was and you can always come back John. You're always welcome You know me. I love to I'm a I'm a big old blowhard. I I love to talk, right? So The second project which is really interesting. I told you that like we you know, we we created that That's what I made a governance guide. That's you know, if you need to ping me j. Willis that read out I'll get you pointed to it's free creative commons. Um, you know, we're actually starting another working group We're going to just really focus on policy The first one was really about our reference arbitration. Now we're going to like what have we learned. So imagine this So you give all those things I just talked about Could you then do error budgeting for policy? Oh, why not? You actually have all the mechanics to actually write like you can say like this, you know everything Yeah, I mean it's there now. You actually have evidence to do something. So like s l o s l i sort of activity All right. Yeah, all that happened and then um, I've been loosely working with the guy who runs onug Which is the open network user group? It's these guys are really big and sort of defining the white papers for sd And uh, you know sdn and all that a really really powerful group It's it's a user group But it's like this monster monster group with like all the major banks of wall street and you you know They're all on the board. Um, the guy runs it. Nick. Let's just gave me a call and said, hey Like we've been looking at this paper. Would you like to sort of work with us? I'm like, yeah So the next thing you know in december of last year we start this working group with fedex Signa Kaiser Permanente. I live a trauma life my friend. Yeah, you do It's a lot of luck with a lot of hard work, right but um the um signa Kaiser Permanente and um fedex, uh utc, which is Raytheon and um b of a and Um, basically loosely golden sacks, but the guy who ran all engineering And so we we sat down and we said like let's use this and the first thing everybody wanted to do is focus right on cloud I'm like, hey, I'm not That's fine. You know we could build on this or and so, um, I sort of sat back I was the chair but you know the chair meant that I just make sure that we focus on stuff, right? And not get a pause but uh immediately they came up with like three Really interesting industry problems with the cloud providers And so we wrote um and and this will be available in a couple of weeks. Actually the wall street journal is going to do an armor on it But here's the three that we in this paper and again, I can get people sort of early looks at some of this stuff, but We were we're asking and and one of the things was we weren't asking the cloud providers because that like, you know Like yes, like naive us are going to tell amazon and google and and microsoft what to do, right? So I was very clear about we're not asking the cloud providers We're asking the industry To see if they accept these questions And then maybe the industry will go back to the and by the way that is happening now So and the three questions were this quite simply first was Could we get all the providers to give us a normalized way an attestation maybe some event or a signature or a checksum That tells us the current status of their boot sequence Don't want to know any of the ip behind the boot sequence, right? Don't care about that. No that when I'm looking at my sort of security posture My trusted posture might turn it from your provider Your provider that I have a known state and nobody's you know electricity to a hypervisor I just want to I just want to know that you've given me some identification Some checksum that I know what that state looks like. So if somebody tomorrow decides Oh, nobody would even care. I need to change some code in the boot sequence Like why should I even document it to the customers? Um, it's scale Let me say it again. Yeah That may change the security and trusted posture Well, very simple and and all these requests are and again, this is a big ass But that's why I was very clear about we're not asking them But like let's get a hundred, you know fortune or global Them with trillions of buying power, right? And so one person's not gonna move Yeah, even one FedEx signal, you know, I mean there's actually a trillion dollar work of mine about right just in the working paper, but right the point is like big You know Little b big ask is could you give us these three things? I'll describe the other two but one is sort of an attestation that we that we understand what What your boot sequence looks like now? We'll know if it changes The capital big ask is we want that in a normalized format from all the vendors Right, so we don't yeah, yeah Yeah, like give me a standard here. That's right. And again, we're being clear not to say standards, too the second Every cloud provider sees all our ingress They're the ingress requests to them, right? Right from us, right like the api everything we do like so from a tenant perspective We know that amazon has an entry point to see everything that we're requesting right What if they could just expose that? Back to us through an event gateway And again here big ask capital B all three of them in a normalized format That isn't packets, but it isn't very specific to the the actual We create sort of a and there's a lot of work to be done here, right? Like yeah, absolutely I mean, it's not this isn't trivial stuff, right? Like you're talking about feeding all the events back to the customer Yeah Now but but in a way that you can listen on it because now think about what you can do So what do people do today, right? Most people don't do anything but the advanced people have to scrape hundreds of logs Just on then they have to have a whole not a set of code and when the log format changes Imagine we just get rid of all that and we just have an egress event gateway or event process We can sit on with something out lambda. What can we do now? We can remember those server-side request forgeries or If somebody look at this attack that's coming at us right somebody is Somebody is is become, you know doing server-side request forgery and they're trying to add something And then, um, you know, and you know, so that there's a lot of sort of To be there's yeah, there's a lot of potential there Well, but imagine all your infrastructure. So I've actually let me go sort of deeper here and show you this You know, so I I we worked on this using gherkin as pseudocode So imagine that all the infrastructure So in the second ask which is give us this sort of gateway model The third ask is we didn't really get to finish as much Which is the have all the cloud providers give a common Security framework they all have their own security frameworks exactly But the second one to me is the most interesting which is so imagine now You've got this advanced stage of excuse me for saying this way dev ops But where you know, you're using ansible or terraform and all these things everything built Is actually built by some automated Um Tool so now you can put meta labels and tags and all that stuff Yes, and now in the in this ingress egress model from the cloud provider They expose back the labels and tags Right, so now a simple so like again, I won't go into this gory detail, but it'll be in the paper So a simple sort of gherkin can say I got you there adversary. You didn't get the memo There's a blank meta field here Right and going back to policies code. Maybe the meta file is the is the mnemonic the component and the version Yeah, right and and oh, it is no meta field here. I wonder what this is. Let me look at a white list for crypto miners Oh Or if we start seeing like, uh, you know the average statistics of our requests I'm just going to make up numbers for is like You know a hundred requests an hour for this service and now all of a sudden we saw a thousand requests Right like hello something broke Logs but good luck pal and you know make sure you have a full-time person Maintaining logs just for amazon right, you know not and then another for google and so the and actually the second paper That we got so much good press out of the first one um the um four companies have now sort of gone back to amazon google and uh And uh in microsoft and I've actually said, you know, I'm trying to get somebody read. I told you about this, right? Like um like passes should be certainly part of this electricity to hypervisor, right? Like yeah Yeah, yeah Anyway, so that's where we're at and uh, you know It looks like there's a shot that the actual cloud providers might be involved in the next working paper this I mean so Well, that would be that would be an amazing feat and it would also be something very good for the industry overall to have A standard coming out of every cloud provider, right? Like that right there would be Big right like if everyone agreed that yes, this is a standard for Anything I think that would be a win But if it's a standard for something like this that is so crucial and so powerful and so needed Uh, but people don't realize it's needed because it is so big and nebulous Like that is a huge win You know christa tied all the way back to the beginning like I was telling about sort of dev ops and I was getting a little jaded You know, I one of the things I came to realization is like how do how can I be? Effective here and it like me Becoming an expert on identity and all those things would be just a waste of everybody's time including myself But when I found this I'm like, you know what this isn't the answer to everything security, right? No, well, but Man when we talk about cloud native and all the problems that we're having with cloud and all the stuff and all the sort of You know theater of how we're doing audits and the exposures that we have You know, I talked to dj sling who used to be a top security guy at etna now. I think he works at sonite now Yeah, he does Listen to him. He's like this is this thing like once you see it you get all these Like like secondary ideas. He said yeah, you know what I would do if I had that if everything was built from an attestation model When I got pwned I would just dump the attestation data store And then I would look for You know set a needle in the haystack. I'd be right I inverted the forensic because yeah, so it's just a red flag sitting there in front of you don't have Somebody's talking my system. It's like the label thing. Yeah, let me find all the things that are here that have happened Like let's look at this particular event. Let's backtrace it And oh my god, there are no attestations for this. Oh, how did that happen? Oh We got an old Jenkins running over here, you know, and yeah, so yeah, you'd find things in your infrastructure very quickly Fun stuff my friend. Yes, it is sir And again, I appreciate you joining. Uh, we're running up on the top of the hour I will be back later today with you know our good friend Diane Muller from uh OpenShift Commons and she will be Having a presentation That's a good from um, Alvin Kirkway, I can't say his name right but for he's from ken folk He's going to be talking about ebpf superpowers and that's here in an hour So join us in an hour for talking about ebpf and unleashing that on your grouvernet's cluster for maximum power So thank you again, john willis for uh touching base with us Your your twitter handle your email. I've dropped all that into twitch chat Uh, anything you want to say before we sign off here, buddy No, it's cool. I mean if this stuff gets you excited, we're starting and I'm starting two more working groups These are all creative commons open people that want to come in and share ideas I mean, it's just beautiful to see multiple bank centers This stuff never happened 10 years ago No banks sitting in the room and sharing because this is not your competitive advantage This is table stakes. Right like this is how you keep your operation running So it's pink. They will set uh, you know at redhat.com, you know, yeah Have a shout out. Thank you again, john. Appreciate it and uh, see y'all soon. All right