 It almost sounds everybody gets up and says hey Vancouver. It's great to be here, but I have such great memories and pivotal Events that happen in my life in Vancouver. So I mean you can trust me when I say Hey Vancouver, it's great to be here Yeah, I'm right Oliver as I mentioned It's good to be here as well. I know quite a few of you from past get-ups events like at the weave company It's good to see old friends. So Really excited to be here All right, so Today we have a little story to tell you in the beginning and then we'll get into what we're here to talk about So in 1979 Two brothers Hubert and Stuart dry fists conducted a study in This study they were trying to understand one simple thing, which is how individuals acquire mastery or skills You know understanding the levels of mastery They particularly took a look at airline pilots and how Airline pilots respond to emergency situations or an emergency response procedure In this study they had novice pilots and they had expert pilots and They had the expert pilots write a script or a playbook if you will That the novices could follow in order to do emergency response more rapidly the experts in this study were mostly instructors So they were pretty familiar with giving them some, you know, maybe series of steps In that study they found that when the novices use the playbook They obviously were much faster. They were more performant. They made better decisions in any given emergency response situation But then they turned around and had the experts follow their own playbook And as many of you are probably familiar with in tech, this doesn't always go so well The experts when they use their own playbook their performance dramatically reduced Compared to when they weren't using it at all So why don't we think this is well when they took a closer look at those experts in particular and when they removed the playbook from Their hands they noticed that in every single emergency response situation Those experts weren't using any scanning pattern, you know, there was no like in the novice case There was usually some sort of series of like check, you know this dial this dial this dial flip this thing whatever in the experts They almost in every situation responded differently And this is because at the highest level of expertise in mastery they concluded that Intuition is the primary form of decision-making This study actually ended up creating the five stages of expertise in mastery which are used today In the field of nursing and several other fields and the deciding, you know the pivotal point of level five or expert or master level is You are no longer using scripts or documentation. You are pretty much using intuition for every single decision you make To draw a parallel to technology If any of you have read the Phoenix project, which I assume most of you have If you've ever worked with that character Brent in real life When you pull Brent into an emergency situation or an outage If you told Brent to use a script it would slow him down and In most cases if you've pulled that person in It's because the script no longer applies or it's not going to work for that given scenario and in many cases By the time you you're not even finished describing the problem to this person They already kind of know what to do next or where to look It would kind of be like in the SRE world where you have maybe a level one level two level three By the time you've called the level three you've already gone through the scripts and the playbooks and you just need somebody that knows What to do next they need that they need to use that intuition So if any of you have read the book the software craftsman by Sandra Mancusco He talks about how agile without technical excellence is just process He also talks about the reverse of this which is that when you have technical excellence when you have expert Programmers when you have seniors in their field or domain experts on your teams, but you have an overbearing poorly defined process The same exact value reducing thing occurs in that you're not utilizing those experts as you you're not getting value up So today we're going to talk to you about an enabling pattern that massively reduces friction It gives you a process that will work well with those experts and those people that use their intuition What sort of compliance are we talking about? One answer could be what we'll do in the next few slides It's the compliance that most of us are used to seeing in a CI CD pipeline But the takeaway we want to do early on is that really as you engage in this exercise within your organization It's all the compliance that might be in there Contractual obligations PCI things like that could be governmental regulation In the form of you know a lot of different things it could be socks compliance Internal processes the organization just practices a way of Meeting you know investor and shareholder requirements all those things that translate into work that might appear in a developer's life Really can be subject to this and if you think about it more and more You know the VIN diagram of the overlap of those things is starting to look like a circle But we're gonna focus in on one particular set of areas that that's really central to CI CD So how many here have heard of continuous delivery? You know, it's like who is this guy? But the second question is the more ironic one and how many here have heard of ThoughtWorks? You know, it's like okay, it's not too bad You know, I don't know if you realize when jazz and Dave Farley wrote that book they were ThoughtWorkers and they've been amazing contributors to this space but the domain of thinking about Using automation and using CI and CD to for a developer to do his job You know, we introduced that with cruise control and it's something that's central to what we were doing It's what we do today what we will be doing in the future and we're always thinking about What's the impact of that? Is it benefiting it? It really has become in essence and it was for a long time central tool of the trade surgeon scalpel for developers and It's morphed from there. It looks somewhat different Probably around 2009 or so. It was like Jenkins made it on the IT ops radar Not surprisingly they're the sort of the first line for a lot of stakeholders within an organization that have Compliance needs financial needs you name it and they will kind of look to IT ops to get those things done So they saw an opportunity from a sort of traditional human division of responsibility way to deal with that and by seizing control of the pipeline and saying We'll address these things here and we'll set out our goal and then we'll make it your responsibility And that's what we'll tell the auditors as we move forward But if you think about what Brian was talking about What effect does that have on your developers as they increase in their knowledge of the domain which they're shipping and as their general skills grow It begins to have a very high friction opposite and outcome effect Maybe not in terms of client strictly but certainly in terms of the productivity of what is a very expensive and human labor intensive activity So we're gonna just focus in on sort of the everyday piece of that and what goes on in these pipelines Some of these have always been a part of developers concerns those engineering standards that is a team they put together and want to hold themselves to They continuously building their code How they maintain it inside of git they've always been there But there's a whole lot of other things that we do right now I've been growing and doing that go much deeper than that security issues cost controls data protection and we can think about all the ones that are there It's kind of another lens of looking at it a lot of the Static and dynamic testing that goes on and we found a way to do that there But how does this end up looking how does this you know? How did it begin when when the pipeline, you know was a visible to everyone and they wanted to get control of it? and use it That is a content-dense slide Which I made so it's my fault, but It really captures it, you know, here's real life informing art How many of us really are familiar with this experience when you're out there working? By natural design, I guess from the way we traditionally work generally except that accounting principles you name it We created all these separate teams Took away the pipeline to start with Then persistent data sources. That's a central part of what developers do especially and even more now that we have Many more of those databases and our distributed compute architectures But we set up another team to build that and think about how that should work both technologically and also from a sort of security And controls and data protection perspective We set up cabs to do things If any of you have been in some of the senior roles executive roles and companies You know that you know oftentimes we set up cabs just to make things go slower Because we were breaking stuff as people wanted to do things more and more quickly But that's what developers are facing and so now now take this and project it over time not just one sprint But year over year as your organization is growing and shipping more and more of their customers experiences in the form of software And yet we're also growing and evolving this Approach to dealing with all the security concerns that we have Begins to really drag things down so How do we solve for some of these pain points, you know, taking a look at those different stops that we saw along the way and The the thing that we sort of are putting out there and we've coined a term for it as compliance of the point of change and What we're talking about here is the separation of duties between the verification or the doing of compliance work Versus the verification of compliance work Meaning we're splitting the responsibility of doing the assessment and the responsibility of verifying that that assessment happened as An example separate the verification of infrastructure configuration from the making and the doing of the infrastructure configuration itself In other words, we want to shift the doing of compliant development left as far as possible as close to the developer But the enforcement of compliance at the point of change where it actually occurs When we do this we meet the needs of security compliance audit We're also giving developers the freedom to optimize their own pipelines We do this with the concept that I mentioned earlier compliance point of change and this term I bring it up again is because it's Been coined by Nick Carl Nygaard Martin Fowler several others So to give you a more concrete example of compliance to the point of change something that many of you are probably familiar with is the Kubernetes admission controller and This is a API that can be used to meet the pattern of compliance at the point of change Now what the Kubernetes admission controller does for those of you that aren't familiar is when you send a deployment to your cluster To the boundary of your environment as we talked about earlier There's an object there that is going to be parsed by that admission controller And there's different technologies to do that you can do it with pure go or OPA or something else We'll talk about those But the point is it's going to take a look at the different artifacts and things that represent that deployment Now the reason that this is important is because when you think about the traditional audit or compliance strategies of past When you do an audit, it's like once or twice a year Performed by some random company by some random set of auditors and the way that they go about Doing verification of compliance or about it is completely depends on the person in this case You're doing it with every single deployment in a consistent and repeatable pattern So to give you a more concrete example, let's look at maybe sneak if the developer owns their own pipeline and they own Sneak in terms of that pipeline that might have like a circle orb or something that they've written they're going to create a report artifact or a CVE artifact that's attached to that container hash and When that deployment hits our boundary our cluster environment or admission controller our admission controller is going to call back up to sneak and say Does this deployment have any high or critical vulnerabilities or any of them accepted by the business, etc, etc But the point is the verification of compliance is done at that point of change. You're at the deployment the actual doing of compliance work was already done by the developer and They were able to do that. However, they wanted to it might be the first step in their pipeline It might be the last they might have another automated process that generates those reports daily hourly minute Lee Some of you have probably seen or tell us here at the top that one does it dynamically, but the point is Those CVE's and those reports are generated however the developer wants to They're just meeting the needs of the boundary of the environment when they actually get there So how might this how might this look we've got two or three particular examples that we want to put in here And starting with this one is continuing on with CVE scanning and I've put up here the security domain team So if you think back to that dense slide I had with all the different teams on it traditionally those are like functional silos within an organization and So if there's optimizations and things going on they tend to be happening inside of those functional silos And so if any of you have worked with and thought about lean engineering and what that means and and instead of thinking about things in Their silos, but end-to-end optimization and how to improve the actual outcome of that The question you have to ask is to be successful with that end-to-end approach is can any one of these individual teams? Have we have we drawn the boundary correctly? Is it the right domain and in this context? It's a way of saying well, okay Can they can they arrive at what they want to do in such a way that they Self-serve enable and completely don't block what the other teams around them are doing So it's very much a domain-driven design strategy like we think about when we're architecting our application that we're shipping to our customers The same thing applies here and it can work if that team is approaching it this way meaning One of there's many but one of the things that they're responsible for is hey We don't want to ship code In into our customers user space that has known vulnerabilities that we haven't figured out or accepted or mediated And how can they do that? Well, you know the today we pick tooling to help us automate that process But the idea is yeah get that in place provide keys when teams come on board and they're using these things give them keys that they can Interact with and get that feedback early and do the work of remediating those issues or patching them whatever it takes to fix it and in this case we use kind of a circle CI example the idea is yeah That security team could own Building these versioned reusable bits of pipeline code that then the developers are on their pipeline can just drop in and get that Feedback what are the vulnerabilities if any what our mediations are available and they're responsible to make them Likewise that security team could build and ship the admission controller the validation that goes in and say hey Do I see in our sneak logs? For instance, do I see the artifact that says yeah this particular get commit this Shaw was scanned It had the right results at the right time And so I'll go ahead and this deployment can go through or alternatively if it's not there We'll block it at that point and so this compliance now we're talking about at what point do we enforce that compliance? We'll do it at the point where the change occurs and what we're checking is did that work take place? Did it happen did developers on it and whatever you do if you're gonna hold that domain for instance in this case the security team Provide these three these tools that they would need to give that feedback and be compliant and completely independently consumable self-service sorts of ways We have kind of expression we bring to this former thought worker. He's now head of engineering at Richie Brothers a great Vancouver company But put it this way if you have an opinion then you better have developers And the idea is that either if you're the security team for instance you provide the outcome and developers can self-solution to meeting it or You have developers that create service interfaces API endpoints Reusable bits of pipeline code and automated gatekeepers at a control plane level that finally do the enforcement so that people aren't Opening up tickets to get that worked on they're not engaging in that necessary human interaction for it They have a self-serve way of achieving it Put another example in Using the same team in this case, but bill of materials is something that's becoming more and more important And in here often what we're doing is saying hey, you know Here's an orb for instance and go ahead and generate a bill of materials for the information inside your image And this kind of distributed compute architecture Here's another tool where you have the means to be able to sign these things so that we can trace Providence all the way back to that sign commit when it went in to get to this image That's now going to be shipping and running the same exact image from dev all the way through prod if it's a you know If it's released here's the tools to generate those things and get them into the OCI registry so that we can put a Admission control on the other side that says okay. There's a deployment happening. I'm going to go do a quick check Is this sign is it signed by the appropriate team? Is there a bill of materials with it? They don't generate those things but they do do a check on it to make sure that it's valid And so the entire effort this is one where in fact Maybe none of those those automations in the orbit there But you could define those outcomes very clearly and teams could sell solution in a way that would easily meet it And yet still write for instance a security team could still own and write the admission controller in that case So for our last example We're looking at maybe a slightly different case from the others where we're seeing scanning tools we can talk about product ownership or development or the creation of user stories and While at ThoughtWorks, we don't recommend doing You know manual gating your deployments like having production owner sign off and then it's deployed We understand that some enterprises still need to do this and so in that example most of those tools JIRA rally etc have an API and so what you can do is call out to that API at the point of change ie the admission controller still and See if the product owner has signed off on that deployment or that user story To do this in a better way what we would recommend is maybe we define product Approval or product owner approval as a series of criteria or a composition of criteria that have been met of all at the same time IE story has been admitted into the backlog We've got a bunch of acceptance criteria that are be automated and tested and they're all passing We have smoke tests We have code quality all those things when they're met at the same time our product owner has already said Yep, if all those things are met my approval is met and we can additionally still in that case call up to our JIRA or other api based product ownership tool and See oh is this deployment allowed to go through or not? And maybe there's certain acceptance criteria that haven't been met or haven't been tested against well Then we can immediately deny that deployment into our environment until that is actually met Well if my part just to summarize We consistent we see this consistently and it's not just a sprint over sprint sort of thing or a month over month But it's a year over year return on investment to say everything that that makes up what a CI CD pipeline is That that the actual ownership of that and the the day-to-day code interactions with it to maintain that in the development teams Hands whatever other requirements there may be give them self-service ways of meeting that of doing the work to make it Compliant, but also when you have to have and we do want enforcement gates That's a that happens early. It's all the way left in the sense that that first data center deployment these gates are in place, but You don't take away control from the teams to be able to do that And it's one of like say there's many things that create value and accelerate teams but this is one that's consistently among the most valuable and Just to go back to our sort of an earlier point And there's two things we want you to walk away from here The first one is the single most friction-reducing thing you can do for a development team is give them ownership of their DevOps processes By adopting compliance to the point of change you can do exactly that while still meeting the needs of your security security compliance and any other teams that have Checks that need to be performed and you can allow your development teams to do what they do best Using their intuition like we talked about earlier in order to drive their own processes You know as you develop domain experts on your teams You know if you have an application team that's been together for one and two and three and four and five years Eventually you're going to have experts on that team and they're going to get better with time And they're going to want to change the process because they realize that there's changes that they can make Well if that process is controlled by some arbitrary team or architecture board like Nick said or something else They're not going to be able to improve and get better The second point that we want you to walk away from is from the audit perspective by taking this approach We are instead of doing one or two off audits every year We're doing it continuously and we're doing it with a pattern that everybody understands and expects and can speak to And this is immensely powerful for both sides of the house operations and development I think that's that's the end of our talk I've got another talk later today on flux at the point of change two o'clock feel free to come to that and Feel free to reach out to us for questions Thank you