 So imagine a thing with human faces or a treat and a delight and I get to stand up not worry about being on mutes And and get to use my clicker and everything. It's quite exciting. So Elephant in the room policy is a super dull thing. It's kind of hard to make it sexy But I'm going to at least try and get your attention. So bear with me So to set the scene I'm in a lift and yes any American friends We really do call them lifts and four people walk in I think to myself Chris This is your moment now or never as the doors close I position myself in front of them a captive audience. They're mine. I've got them the doors still shut behind me and I take a breath. I Looked the first person on my left. She's in a suit. She looks really important to her. I gesture to her see She looks back at me as if to say yes. Go on. I she knows. Oh, ah Perfect the CIO the policymaker the one whose neck is on the block What are the chances of finding you in my imaginary lift today? I ask her well, what keeps you up at night? She tells me I don't know what teams are really doing what the volume of risk and what I should be showing more interest in Setting and changing policy is slow Hard to communicate and people just go off and do their own thing They think they know better and to be honest often they do but then I'm left playing catch-up with the risk that they've signed me up to Okay, I say trying not to sound like a patronizing snake say Patronizing snake or salesman. I can help. I Turn my attention to the second person in a suit slightly less important though. I make a guess. Let's face it This is my imagination. It'd be a bit weird if I was wrong Portrait manager. I say they nod the whipcracker. What's important to you? So managing risk mostly opportunity risk the fear of missing out So getting features out the door and avoiding getting bogged down with they glance to the CIO Bureaucracy that is designed almost seemingly to just to slow me down Awesome. I say this is your lucky day. I Turn to the next person dressed in overalls. I'm in a trendy part of town. They could be the CTO Before I ask they sent me staring at them Cleaner say odd. How did you get my imagination? I'll come back to you My attention goes to the last person hoodie headphones around the neck. Ah, my stereotypical developer. Yes. I know you well What code you write I ask it doesn't really matter Python cool Have you got anything to work you updated to work with I pause Python 3 they offer. Yes Python 3 that must have been super hard. They don't know it But I can almost won a bit of their trust. What's important to you? So staying on top of patching things so we can react to the next fire knowing what rules exist What was I can bend what I can break and ultimately what might cause me to lose my job? Writing consistent good quality code and avoiding technical debts Ultimately the rest of my team being able to cohesively work as one Do you use any tools to help you with that? I ask that. Yeah, Linters code quality test coverage that the kind of usual Great, I say I write code to let's be friends and I hand them a printed QR code Here's my public GPG key. So you know that you can trust what I say. I Return my focus to the cleaner. I've got it. How do you get told what to do and when it changes? Well, we get a memo or something stuck to a notice board So last week we got a memo saying that all of the meeting room whiteboards needed to be cleaned every night Interesting say well, how does that work out? Well, it's then up to us to then maintain the to-do list so that we can on board new people and such Does it go wrong at all? So yeah, sometimes if we compile that operational manual wrong when we miss a memo or don't apply them in sequence We do things do go wrong They glance apologetically to the product manager like when we hadn't updated the guide that the meeting room on the third floor Was being used as a dedicated war room and we've wiped all their boards down. I Looked at the dev so sound familiar. I asked they nod so turns out we're not all that special snowflakes. Hey Okay, all's not lost. I knew that there was a reason that I imagined you here today The lift is slowing. I feel it coming towards its destination. Great. I've got the silver bullet for you, too The CIO looks ready to buy literally whatever it is. I'm selling they ask me as the door is open Who are you and what team you in as I move out the way so to stop obstructing it? I Answer I don't actually work here I'm just here to fix the lift people have been complaining that it actually goes to the top floor and it's actually pretty slow My audience storms out furious heading towards the stairs and the door shut and I get back to my job. Okay So if any of that might sound vaguely familiar and you can relate to my imaginary friends Then I might have something resembling answers for you So what if I said you could update policy easily even releasing several version updates Not just in a year a month What about ten updates in a single day and seamlessly communicate that to the people that need to consume it without derailing them You could have visibility on compliance using tools that perhaps you already use and that policy could be readily consumable easy to pass Demonstrate compliance and make sense and not be bureaucratic to change when it needs to ultimately not get in the way That same policy could be treated as a dependency and operate like a linter So you could run compliance checks locally in CI and then guard production That multiple versions of the policy like a dependency are supported So emergencies like you must update now because there's now some no invulnerability type updates Effectively just become a business is usual activity to carry out and within your business Interesting. Okay, hang around About doing for time. I wasn't allowed to lock the doors, but this would have been the point I would have allowed them to be unlocked. So hopefully I've at least got your attention It's it's now time to kind of introduce myself and start explaining things. So my name is Chris Nesbit Smith I'm currently an instructor for learn k8 and also for control plane consultant to UK government minute the cram prosecution service and generally a tinkerer of open source stuff I've spent a fair chunk of my professional career now working in kind of UK gov and large organizations Where problems like this are rife We should have some time at the end for questions and heckles so please kind of prepare a really good one And I've got t-shirts for the best questions. I think it would really good one. If not, I've got pink trousers I you won't miss me. So find me afterwards somewhere So given that I've got the luxury of a live audience and most of you have got clothes on for a change Thank you. So by show of hands Who's with my CIO and has set written or applied some policy and any sort of guys before? Any sort of coding standards or anything like that cool. Almost all of you Okay, thanks. You can put your hands down now How many of you have sort exemption or consciously bent broken circumvented ignored bypassed whatever a policy with at least good intentions Cool, you fell for it. Well, so thanks to the organizers of this event and now the photographic evidence I've got all of your names and employees details down. So can I put your phones down? Lend me your ears. The stakes just got raised. So Where do I see policy as code going wrong? Well before we dig into that. What do I mean by policy? So it usually comes in one of two forms So security enforcing say like data at rest being encrypted for example or consistency such as code style So tabs being two or four space indented maybe Or maybe you can think of some others But in any case, it's hopefully intended to mitigate a risk of some sort However, with the best of intentions, these are often emotionally led rather than being grounded in a proportionate control Which is the open door to case-by-case exemptions being sought When you come against a situation that you weren't expecting So this is not unlike how the laws of the land were created So with case law making for a complex to navigate rule book and harder still to measure compliance It often can look like the thin end of a wedge where the precedent which may have been an uncomfortable pill to swallow the first time round Becomes dangerous with others looking to expand upon its scope Which can lead us to sometimes wonder if the cure was actually worse than the disease But that's not how we at least typically develop software. So why does this all have to be so hard? We codified everything else. So maybe this is the answer Well, yes in part but my point of this talk is that we often do it wrong So maybe some of you are kind of mentally screaming your favorite product name at me in your heads And it's not and it's not and you're not wholly wrong But the devil's in the detail Throwing some curly braces or yamal at something doesn't inherently fix things If it's a security control, it's often tempting to keep that policy a secret Exposing it could maybe used against you by an adversary However, that does not support us kind of shifting left at all It results in devs effectively reverse engineering what the policy is by finding out when we can smash our heads up against it It doesn't therefore take much imagination to see that in the scenario of an application deploy Fighting one resources non-compliant and rejected would leave us the overall deploy in an inconsistent state likely resulting in some downtime Which begs the question of was the policy better than the downtime that you incurred Especially if that leads your engineers who are all hopefully plenty smart people at finding Inventive ways should we say around the computer says no response that they've got This is then further exasperated when updates to the policy desired So maybe you get a pen test or something goes wrong So you form that case law effectively and need to apply new policy So maybe say all s3 buckets now need to be encrypted a change that could be considered breaking Sure, you might say that you provide say warnings on at least the less important issues or new emerging policy Which is great So long as someone sees them But if you've adopted githops or at least some CI CD in some fashion is anyone seeing any of those warnings I mean who studies the results of a successful build log every time Anyone every time Well, if you are I Lightly suggest you're perhaps missing the point of CI CD. You should ultimately be able to trust your job status Okay, well, I'm not here just to throw stones. So remember my implied promises to my four imaginary friends of what this utopia might look like Well, there's nothing new under the sun here funnily enough. We've already unwittingly solved these problems elsewhere We just need to be reminded and join the dots Well, the first is something that if you're doing policy as code, you're probably already doing so put it in version control The thing however, you might not be doing then is making that visible So at least in a source it by which I mean allow anyone within your walled garden of employees and suppliers and etc To see the policy I'm not saying to give away all your kind of threat intel and monitoring rules Rolls away. You can probably keep that all to yourself But I'd argue that visible policy and the gaps therein is often better than the downtime reverse engineered workarounds and opaque legacy exemption spaghetti soup If you're brave you might even open source it you'll find that it unlocks the ability to work well with prospective suppliers without NDAs and what not and Ultimately widely distributed secrets are expensive to maintain difficult to handle and often only stay secret for so long after all Okay, so we're off to a good start a policy is visible now to those that need to see it So many of you are no doubt used to Semba, but a quick recap So the first segment for it is to indicate breaking changes say perhaps conflicting ones So in the context of policy, let's say it's requiring resources to have a department label Maybe that will help with some internal cross-charging. He knows I'm re-judging An increment to that might look like requiring that to be from a predetermined list rather than free text The second segment is to indicate minor changes that shouldn't really break anyone an Increment that might look like correcting a spelling mistake on one of the department names The third is to indicate patch changes. So these should be a no-brainer for everyone to keep up to date with An increments that might look like say adding a department to the list of available options Okay, our policy is visible in a repository its version so that we can easily communicate the policy We can tack on release notes and expectations are managed by semantic versioning In software, we're used to handling dependencies. So what if your policy was just another dependency? So you might unwittingly already be doing this if for example, you have saved ESLint as a dependency in your JavaScript package, perhaps Okay, so our policy is visible in a repository its version we can easily communicate the policy We can tack on the renotes we can expectations are managed by Semba. It's beginning to look a lot more like software Okay, I know testing is a dirty word But in order to make this an asset that can everyone can depend on and also provide some really useful get known good examples Tests are essential to give everyone confidence and stability and surface potential side effects before they end up hurting everyone involved Consumers of this policy need to be able to test themselves against the policy locally and in CICD Thus shortening the feedback loop and better informing things So it was a bonus. We should be able to find our consumers able to rely on the artifact that we're sharing with them Okay, we're well and truly on the home stretch It's a dependency so updating it should be no different to any other and we can even use some magic like github's dependable or mend Who were downstairs earlier renovate to do some of that for us? So think automatic pull requests tests even auto merging if you're brave Okay, the check you're all still awake Can anyone tell me a recent event that caused everyone to want to know what version of a certain? Java logging at do Hickey. We were potentially running Literally everywhere in the estate finger So yeah, as you know all presentations this year are actually contractually to required to reference log 4j Even when it's almost entirely out of context and includes the memes in just a few short months I can remove these and hopefully just point broadly at a scary looking list of CVEs in order to command your behavior through fear What I'm getting out here though is is that situational awareness piece around software supply chain is something that your organization has already Can I hopefully thinking about? If not already addressing so if our policy is a dependency this is not a new problem So software bill of materials for the win, right? Which can allow us to also then measure the compliance across the entire estate Okay, I've just covered quite a lot of ground and hopefully sounded at least vaguely convincing and it's not just a fictional utopia painted in PowerPoint It's time to look at how you might be actually able to do this And I know you really came here wanting to see a million words on a slide and not just the odd emoji or two So we've reached the point of the show where I get to show some code hooray But to maintain scope I'm going to limit this to talking about two things to prove This is not just one tech or one tool of arbitrarily picked terraform and kubernetes But I could have probably picked anything naturally. I'll need some tools to help me go go with this I'm too lazy to invent that much here. So likewise. I'm going to pick two tools But again, I could have used some or even all perhaps So in this case check ops going to be doing my terraform whereas Coverno is going to be doing my kubernetes If you want to browse along with me The link will be at the end of the deck as well But I'm not expecting you to kind of read or grok the code on screen. So don't worry about it It's just to prove that I made something that's vaguely real So the policy is stored here So here's where my policy starts at fee 1.00 So I've got policy that requires that department label or resources so long as it's set doesn't matter what it is I've written tests for this So note how the passing test cases are a really useful great example of what good and bad can look like We pushed a tag in git we've added the release notes I can sign it to provide further assurance if my heart so desires Which of course it does but moving on version 2 looks similar only now that department field has to be from a pre-determined list So like before tests exist release notes are written tags assigned 2 1.0 is where we notice and correct a spelling mistake of one of the options in that list of departments 2 1 1 and I've now added a new department list Couple more repos in that org with app one and info one and they depend on version 1.00 of the policy It's not compliant with version 2.00 beyond but how do I know that? Well, I've configured renovate to be automatically making a pull request So when there's a new version of the policy It's super obvious if I can just update the dependency and I can see clear feedback about where and why I'm not compliant I Can also see all the pull requests over the org so I can measure the compliance of my policy Okay, moving on from that app to complete repos so app to an infra 2 well They depend on version 2.00 of the policy However, we could merge the open pull request all the way up to 2 1 1 Finally app 3 and infra 3 other two repos are dependent on 2 1 1 and they get a gold star from the CIO There is a small touch of magic and it's not pretty I've written some bash Don't judge me even though. I've probably definitely absolutely constantly written much worse But what this allows me to do is from my dev laptop or NCI to evaluate my code against the version of the policy that I've declared Ideally, this might be a bit less cumbersome, but it is what it is for now pull requests are very welcome And the last piece of the puzzle is managing the lifecycle of the policies so and allowing multiple versions of policy to be accepted And evaluated within a single runtime I've cheated a bit here. So Kubernetes gives you admission controllers It's not so easy to going to get the same policy from a cloud vendor and be able to actually Evaluate that about about stuff that you've got locally Again pull requests contributions collaboration all very welcome on that front so You may have noticed the way that the policy is declared and designed and distributed lens itself Well to coexist with itself in a Kubernetes cluster Which brings us to another repo cluster one which describes a cluster that accepts all the versions of the repos All the versions that we just what policy that we described so far So likewise another repo cluster to only accept 2.00 and greater We can automate this all using Kind which you should keep in a season Docker for CI in order to deploy the apps and Then we have it a full org all done all compliant policy or version than the CIO all aware of what's going on This is great, right? But just one more thing Wouldn't it be awesome if the policy carried a story about why it exists? After all if our agile teams are even half effective They should be rejecting anything that they perceive as friction if they don't see the value in it and It could allow our devs to know why they're compliant And if they want to do something outside of what the policy permits They don't need to have any exemption granted per se They can have a well-reasoned and informed debate with rationale behind a pull request to the policy So imagine if you will this going through a stage of versions with risks that inform the mitigations Manifested as policy all maintained kind of as one living thing So when the risk landscape changes your policies can move with it So when some new Privacy regulation comes out so your latest marketing strategy pays off and you acquire more data For example, even if your policy was perfect at one time the risk and the appetite for it will stand still for no one So we can liken this to over provisioning that we might be familiar with from elsewhere We're lead times along changes hard and there is a significant pressure in nailing at the first time Which can lead us to hedging bets Against what some future state might be rather than proportionate mitigations to the risks that are more tangibly real in the now And that's where the real culture change is needed and the execution of that is likely a long series of talks in of itself So now this is really kind of all over to you Honestly, the best thing that you could do right now is tell me it's madness Already done irrelevant or otherwise just unachievable something my esteemed echo chamber appears have yet to do So beyond making some pull requests and developing the theory some more I'd really like to start building some more case studies with willing kind of real organizations and Allow me to continue to swap out my imaginary friends for some real ones But the most important thing that I want you to remember from our talk together is and please do feel free to say this out loud with me Purposeless policy is potentially practically pointless policy Which I've been practicing far too many times. So I'm Chris nose Smith. Thank you so much for your time You're now free to leave. I'll destroy the evidence of your guilt admissions earlier Like subscribe, whatever the kids do on LinkedIn GitHub, whatever There's you'll be rest assured that there's no spam or really much content at all since I'm awful at self-promotion CNS.me just points at my LinkedIn It's a bit easy on typing my name and talks.cns.me contains this and other talks and they're all open source Questions very welcome on this or anything else. I'll hold the stage for as long as I'm allowed or finally afterwards I'll be somewhere where there's some probably some some Guinness somewhere I imagine But yeah, thank you very much I've got t-shirts for the best questions In the sense of what sort of scenario you're thinking Yes Yeah, yeah, so the point of conversing it is and having it in that way is that it should be able to Be more of a living thing like our software is right So you have a any requirement for an exception exception in the software you codify that in so that given this then that is Okay, so rather than just going fine, but that team doesn't need to apply to comply to a policy or that thing having like a Thing around that team or that product or whatever it might be It's where you codify that into the actual code So you say well on these conditions and you can have that in a number of ways Like you could codify that through like attestation through I'm don't have any PI I therefore I don't need to care about encrypting the bucket. I'm just using it as a cheap dirty CDN fine like that That's the thing and you could If you what if that would if the org have accepted that it was appropriate to And it was within the risk tolerance to just describe that through kind of metadata annotations on the bucket and apply your policy that way Right, and again like that's a version increment that you've applied applied across the org That makes sense Yeah, make a poor request I Know not in its current guys, I guess and it's hard to like know how far like from a I guess from a central point like the How far on they've got ultimately if you are You can come back to my point earlier. I think the if it's treated as a dependency then it's It's the same tooling ultimately to see like how compliant your your your org is and you shouldn't have that dependent You shouldn't be depending on that version if you're If you're not compliant with the policy so like I like fail the build ultimately as the team until I until the And you can be on an old version of the policy That's okay so long as the the organization is kind of happy with with that version Can I and it's within the life cycle of that version and The runtime that you want to run it in so if your production cluster accepts like two and greater Then you can work on one But you know that you won't be able to actually deploy it to production You might be able to deploy it to an environment that does accept a version one of the policy Yeah Yeah, that's the that's the the utopian vision reality of it We'll see but Yeah Yeah, it's hard. I gotta say like the bigger challenges that is people text easy, right? I might might not be in the right room for that I'm sure we spend plenty time looking at code and and struggling but Text really easy when you compare it to people people are really hard and cultural changes is super hard to achieve And that's what most of this is the tech is stuff We already do elsewhere when we build software and we build things that other teams depend on like it's no different Sure. Yeah, yeah I generally would focus on Both with this and any other bit of tooling around can developer experiences I would generally stay towards like make the experience good for the developers to start with Naturally like developers are all kind of smart people and if you can articulate the benefits that it provides them that they know that they're working in a safe space And they know they can it works in a way that they're familiar with Then it naturally becomes something that you can run in CI because they'll run that in the same way as they might like run a lint check or Run the unit test, right? It's no different to that fundamentally And it's also up to them to kind of run their pipelines and usually done For their for them to sort out for how their application gets built But yeah focus on the developer experience first and foremost would be generally my advice with both this and kind of other Any sort of other kind of cultural change of make that work really well get them fully on board with it And naturally they'll that will kind of filter through to the end. I think the whole point of a lot of that is has been able to run the checks all locally Rather than as I said, I just like smashing your head up against when you find that some things not applied properly And you end up with production that's broken often because the production cluster or production environment, whatever it is is Someone said well, we should have more strict policy their developer environments lower down. Well, they can run whatever We can be more relaxed on the policy and controls that on a lower environment and monitoring anything else Production has to be kind of higher Yeah, exactly. Um, yeah, I mean it's that's that is I guess the DevOps movement or whatever but or other similar kind of buzzwords, but Yeah, like focus on on that experience and if you've nailed that then you shouldn't really see Exemptions being raised further us or exceptions being raised further up and any failures because they built it right from the ground up and they baked in The understanding of why they're abiding by the policy early on right and you've actually articulated to them the reason of Why it's important Oh Nothing, I think I use cavern. I could have used keyboard and I could have used I'm thinking any of the others there's millions. I mean keeping us uses to unopinionated as all of them No particular reason for not using API and whatsoever. You could do it in any of them It was just picking something arbitrarily of the other as the thing Yeah, I had to do weird things for the terraform of Putting the version number of the policy that I wanted to use as a variable and then I have the the am I ugly bit of bash Look gets the variable number checks out the version that the policy repo at that tag and Then checks the terraform against and same thing happens for the Kievernesses and checks it against that super ugly Please pull requests, but that's kind of how I can have hacked it together to make it work Proved a point. It was all kind of proof of kind of concepts and and and value and right and to make a make a talk That wasn't just PowerPoint. There's something that's some code that runs somewhere and Is there one or that owns the risk ultimately? And then they're the ones that the owner and can articulate down. They can delegate down the online kind of thesis kind of really is is that is that risk conversation that finds its way ultimately articulated to awesome policy The implementation of the policy could vary between different ones But ultimately your risk owner has going to have some appetite that they can I think you need to help them articulate of what they're willing to accept and some principles So kind of risk of like PII kind of get a personal identifiable information as a main gov So acronyms are common So if you if your concern is like PII being leaked has that's a risk and a problem and a concern You might your principal might be where we encrypt or PII fine And then your policy might be that that that's defined as like how that is evaluated against in terraform and then another one against a Cuban exes and another one against a VM somewhere or something else Yeah, I didn't I Didn't however, I can if you if you if you'd like to them they're all open you can download the ones that are there but I paid a I Found a Ukrainian Got graphic artist to do it And they did a terrific job. So more than happy to refer on if you if you're interested and pass on her details Yeah, thanks Different right, I guess but I've only got five. I think we've run out already It's good question. Um, so Ultimately like they're knowing the success of like it being Adopted by your org then like look for failures that happen beyond the local development environment like back to the point earlier Like nail the developer experience early Shift that all left so they can have a first-class experience of it operating on their developer laptop and That being the identical result as they get when they try and do a deploy to prod, right? so you shouldn't have a scenario where you do your deploy to prod and It's And it fails but it passed in earlier environments if you've done that then then something weird has happened along that developer experience And you've failed fine. That's normal. We do that all the time. That's part of the problem The next bit I guess is like the risk owner whoever that is and where if it's delegated to just Understanding and articulating the risk that they're signed up to so if it's been communicated well to them and the But the way that the policy is shaped and the risks that that both mitigates and introduces They understand then happy days If they if they're uncomfortable then you've got a problem Ultimately incidents will always happen fine things will always go wrong fine but they the risk owner needs to be aware of The risk that was a that was presented and understand that that there is a possibility of those incidents And it's when surprises happen when they weren't aware That's where your policy ultimately has failed and your risk management as an organization has kind of failed you Grand yeah, find me if there's anything else. I'll be obvious from somewhere, but thank you so much for your time