 Thank you so much. Yeah, thank you really appreciate the intro and thank you so much for having us and thank you everyone for joining I'm Matt Johnson. I am a developer advocate with Bridge Crew part of Palo Alto and Prisma Cloud and yeah I am just super interested in kind of security and all things Kubernetes and all things Infrastructure as code so in this session We're gonna kind of look at those coming together and kind of what the security landscape looks like when we have you know code We have infrastructure configurations And you know how we need to kind of consider those as a single entity when we consider a security posture So as I said, I'm a developer advocate with Bridge Crew I Have been involved in containers since Solaris branded zones, which is probably showing my age at this point I'm a hobbyist pen tester. I'm way better at the breaking stuff than the writing up the pen test reports afterwards But if you do want to kind of carry on this conversation I've put the Palo Alto Twitter at the end, but I'm at metahertz on Twitter We also have a slack channel for kind of all things infrastructure as code security at slack dot bridge crew dot I oh So by all means kind of find me there afterwards if you want to take any of these conversations offline Also the Q&A window I can see in front of me here It's always more fun to kind of handle questions as they come in So if you're interested in a specific thing, let's be a wildly off-topic If you want to fire a Q&A over and I will try and get to them as we go through so with that said I'm gonna set some context and jump right in and For this context, we're gonna start in the persona of a developer as we're gonna see there's a few different personas for Security obviously in the software development lifecycle But let's start as the persona of a developer I as a developer have access to a Kubernetes cluster and In writing this talk based on on the research I'm gonna show I was gonna make a joke that a children's illustrated version of this is coming soon because it sounds like it Could be a children's book and then I realized that actually that already exists Yeah, it's called an illustrated children's guide to Kubernetes and the CNCF already wrote it So, you know, if you have kids and you really want them to understand the difference between a pod and a deployment that book is for you but anyway the story of a Developer that has access to a Kubernetes cluster and needs to deploy Some service that the business needs now are we going to expect that developer to go and write the deployment manifest themselves? or realistically, you know their function is a Building that code base their function is and let's say that code base is running on top of engine X For example and using whatever modules engine X has built in to allow that code base to run really easily Are they going to want to build a engine X deployment themselves? Or are they just gonna want to package the code that actually adds business value and Deploy that onto an existing engine X deployment that they can push to their cluster So Realistically, they're probably going to go to their favorite search engine and they're going to look for an engine X deployment And so that will probably bring them to he says if his mouse was working for changing slides That will probably bring them to Here we go to he says Small demo goddemons always always especially at this time of the evening This is probably if they are looking for a pre packaged Kubernetes engine X engine X deployment in kind of the Linux and kind of modern cloud native CNCF landscape. It's probably going to bring them to somewhere like artifact hub.io, which is a repository of kind of everyone saying hey, yeah, I have usable Helm charts. I have usable packages for that kind of ecosystem. And so we find the helm chart We find an engine X helm chart And we have some options here as a developer we can either download and run it We can run for the door or we can go no I'm going to spend the time and effort building one myself from scratch when one already exists And as we all know the chances are We're just going to download and run and see if in a demo environment that solves our needs and chances are it will so we have engine X And we can now proceed to deploy that on our cluster and do whatever we need to do with it to to add business value and We previously Wrote a few blog posts on actually looking at the kind of default security posture of Helm charts available publicly not necessarily just an artifact hub also from sources on github and we scanned thousands of Reel well hundreds of helm repos containing thousands of helm charts And wrote this up in a three-part blog series about kind of the default posture And what we found is you are more likely To download from a misconfigured helm repo And within that you are 46 percent likely to have a kind of misconfigured Helm chart by default if you take the defaults within that home chart What do we mean by misconfigured? Well, we mean It won't it won't by default Hold up to kind of the security posture and the kind of good security practices you would expect for a productionized Kubernetes deployment the resulting Kubernetes templates just just won't be there And it's something we expect and see in the industry because if you are trying to get a simple working example of engine X That's easy to explain and easy to understand. You're not going to want your documentation to be pages and pages long Explaining all the nuances of a hundred different Kubernetes options You know, it's that security versus usability and always in our industry usability seems to win So it's not unsurprising that most default helm charts Are insecure by default? So to do this research we took our open source scanning tool help check off which I'll Demonstrate in a mo and we went and found as many helm charts as we could pass them out into their resultant Kubernetes configuration and Process that data and as you can see We have a load of built-in Kubernetes policies and there were some quite shocking results So for example belt embraces like not not locking down the security context Not making sure that you cannot run with privilege escalation You know so many containers still running as root and then kind of some Less so critical things, but it's like, you know, 90 80 percent Have no CPU limits. No CPU requests. No memory limits. No memory requests now Apologize and by all means kind of skip this little section if you're not deep into kind of Kubernetes manifests but these are all things that Kubernetes relies on to know whether an application is Out of control whether it's about to swallow up all the resources on that node Whether it should be shut down and you should raise an error like if you don't have CPU and memory limits in your application configuration Then that application is at real risk of going wrong and kind of swallowing your clusters resources So maybe not a security issue, but definitely an availability issue, which is, you know Just as just as valid since you know dos is usually get a CVSS score and I'm not trying to bash helm or Kubernetes at all here as I said, this is a security versus usability problem and We did the same set of analysis and found roughly the same percentage Across Terraform another infrastructure as code framework that some of you may be familiar with and yeah 44% chance you are likely to get a Terraform module from a public source that doesn't meet, you know today's expected security posture and The reason is very very simple. The reason is in a single slide this And we'll post these links. You don't have to copy them from memory into the chat but we created a Both a Blog post follow-up and as I said the three-point helm research where we actually go into this This is a valid piece of Kubernetes. This will happily run a Demo application from my colleague Steve's podcast actually where he kind of goes through a load of Kubernetes security news and it will happily deploy, you know a single replica of a single application and That is valid Kubernetes and you can throw that at a Kubernetes cluster and it will run whatever we're suggesting to run as a container image however, if you actually want to pass the Kubernetes CIS 1.6 security benchmarks your Kubernetes manifest that does exactly the same thing for the end user as in the application is running on your Kubernetes cluster Actually needs to look like this. It needs to be it doesn't fit on the page levels of Kubernetes and Obviously the problem with that is most People working on Code bases most people working towards their company's missions and their features Don't have a deep passion for all the options in Kubernetes. They are not Security engineers first they do not have the luxury of kind of sitting there and considering all these options They're trying to get a job done and so, you know, if runs ish Kubernetes works That's generally what you're going to find in kind of public home charts and in hey, this works for me try it out kind of community environments and so We have a lot of Usable public charts and a lot of usable public charts are probably going to be this kind of size How do we know as an engineer? What we are missing? How do we know? Whether that's secure or not and how do we get to kind of this and preferably how do we get to this? larger more secure Kubernetes without Without writing it all ourselves Specifically and so as I said, I'm going to introduce what we use to generate this data set We have an open-source Instructors code scanner called check off check off dot IO And it comes with unlike kind of having to have policy packs It comes with over a thousand checks built in across all the cloud providers terraform cloud formation Kubernetes helm We're just adding customized support today. The PR is ready to be merged Serverless framework, you know on templates for Azure you name it check of is designed to find misconfigurations in your infrastructure as code and Help you go. Hey, that's probably not something you want from as simple as That estuary buckets got public rights on it in Amazon all the way to you need to minimize access of root containers in your in your Kubernetes pods and clusters So it's designed to be used in CI so not designed to give developers any extra headaches or any extra work And so yeah, we have a GitHub action for example that you can just drop in if GitHub actions are your things And like any other good tool. It does one thing. It does it pretty well I'd say and it will obviously return a bad exit code so you can pretty much put it in any of your CI CD environments So when you are starting your journey for more secure Deployments and you realize that infrastructure Misconfigurations still play a massive part of that as we've just seen with the Kubernetes example You're going to run into two problems Problem one is where to start as developers. We are already very familiar with CVEs and security vulnerabilities within libraries that our code depends on or within our code itself As you see on the right of this slide. So for example a quick scan of the next cloud Apache Docker container highlights seven critical 34 High four hundred and six vulnerabilities in total across a load of different packages You know developers are very familiar with thousands of CVEs that they are expected to deal with and patch and maintain and what you're going to find when it comes to Infrastructure as code which realistically if you're using Kubernetes you already have an infrastructure as code usage. You can't ignore it What you're going to find when you run check-off against your Kubernetes much like the small versus big Kubernetes example is Added to those CVEs you are also going to have a load of misconfigurations that aren't security compliant with your Instructors code like I said memory limits not being set Image pull policy should be always to stop someone poisoning a local nodes image cache Don't mount secrets as environment variables because some error pages dump environment variables, you know all very sensible CIS benchmarks for the common good of creating a more secure environment, but to a developer that just adds more Complexity that's more things they need to do that aren't necessarily kind of quote-unquote getting their job done and security is everyone's job, but if you've worked in any real-world environment, you know what I mean by that And so that's problem one and I'm gonna focus on that mainly problem two is security isn't a point of time We can't just fix all those issues Once locally and then push and we're done and we never have to look at that code base again because Docker images change new CVEs come up like entropy guarantees that something left on its own Will go bad in software engineering even if it was good at point of release And so because of that kind of the bridge crew in the Palo Alto kind of mindset on this for kind of dev sec ops and Looking at secure deployment pipelines is you need to run the same policies from kind of day zero all the way to day x You should be checking the policies At the id level at the pri level through ci cd with check of and then also checking those same policies against the items Created in runtime To make sure you don't have drift to make sure there You know Someone hasn't gone in and manually changed something to make sure that an image that was secure is now not vulnerable because of a new CVE Unless you have that kind of complete cycle Um that matches your development lifecycle. You're gonna miss something. Um, so that's problem two but problem one is the thing that most people get caught up with the first because You're introducing developers to screens like this which Basically are just lines saying more work more work lots more work and so problem one is quite hard to solve and We recently had the opportunity to work with the wonderful friendly hackers um in the unit 42 cloud threat and security um Research team and we actually shared some of our kind of helm scanning data That they kind of went and did really cool things with to build The cloud threat report that came out. Um at the end of 2021 And it was amazing to kind of work with them to see that what we thought was kind of interesting with You know all these Infrastructure misconfigurations and then you know how that can affect supply chain and how that kind of ties into CVEs and it just kind of highlighted how Wide a problem this is when you've just got all these CVEs all these issues um And what do you do to kind of get through that noise? Because realistically if every project is giving a developer that much noise um It's probably not going to get fixed or it's definitely not going to get fixed In the areas that need the most attention the the quickest because it'll just be sat on a backlog and who knows if you're actually getting the priority items and so this This got us thinking um at bridge crew about the concept of well, what does matter? You know, yes every every vulnerability is bad. Sure and every misconfiguration is bad, but There is there is definitions of badness. Um, and this got us thinking of this concept of blast radius So I will give you an example If I have a load balancer That allows anyone in from any ip on the ipv4 internet so anyone on the internet can get to this load balancer Is that a misconfiguration? Yes, probably we probably didn't mean to do that However, the load balancer has no back end service attached to it. So the traffic gets dropped on the floor It's it cannot attack a service because there is no service Yes, there is a misconfigured load balancer. We would say that's an infrastructure misconfiguration You probably don't want something allowing all ports in from the whole internet But if it's if there's no attack chain, if there's nothing going on there I wouldn't necessarily prioritize A developer fixing that issue first because the the risk of an attack chain leading to compromise there is Negligible, you know, arguably. Yes, there could be some internal Hair pinning of the load balancer account of other potentially far-fetched scenarios But in its default configuration, that's not a massive priority, but it is a misconfiguration Whereas you take something like this you take A Misconfiguration first approach where we have the same open load balancer to security group But that load balancer has ports open That are not meant to be open and there is something behind the load balancer There is this container and this container may have an application that is meant to be exposed publicly via the load balancer But it's also exposing a monitoring endpoint on a different port that isn't meant to be exposed and because that monitoring endpoint is internal It or it's meant to be internal. It might not have authentication. It might be running an old unpatched Um web browser within that code base because it's not public so quote unquote who cares and so because of that Infrastructure misconfiguration We're suddenly now in a scenario where we have access to Potentially a cve within that monitoring endpoint that gives us access to that container So that's how a misconfiguration can lead to something which otherwise might have, you know Successfully been a cve that never got attacked for years and years because there was no access to that cve from a an attacker's kind of attack chain Um, and then once we have container access again misconfigurations that could easily be ignored in the list of output for kubernetes Misconfigurations might mean that there's still a default service account in kubernetes mapped to that pod Which means we have some form of kubernetes api credentials that the attacker from Inside the cluster can then use to query the kubernetes api and potentially find lateral movement Into other parts of the infrastructure into other kubernetes workloads Um, look at what else is happening in that namespace, etc, etc You know equally the other way around let's say we have a public um Let's say we have a public website that is running log 4j for example an unpatched version So the cve is the first thing an attacker can get to the website has to be public because the website is meant to be public That's the whole point of it. Um, but because of that cve Um that yeah that cve would normally get us into just that pod and if we're very careful We can do things with that pod, but if we crash that pod, you know We get a new one and we have to start all over again But if we compare that cve with Misconfigurations that have been ignored in the infrastructure We might have again a default service account in that pod which allows us to query the kubernetes api lateral movement into the infrastructure Or alternatively, we might have misconfigured network policies. Um within our kubernetes, um Network policy objects or even within our wider cloud service provider that allow us to use that as a hopping off point to download some tools and Hairpin into other infrastructure probe other ports probe for kind of internal services That let us again lateral move into other parts of the infrastructure. So Whether it's cve first or whether it's um Infrastructure misconfiguration first do not rule out One being more important than the other like attackers will never do a single thing and go. Yep hack done Like it's a it's a chain. It's a a game of chess and What we're finding is infrastructure misconfigurations or just as common if not more so as the kind of first Stepping off point to a successful breach And so that's all well and good But how do you get diagrams like that when you are looking at Data like this was our next point like this is This is not useful This is not useful at all Especially as I said if you are developers aren't necessarily uh full-time security advocates They are they're trying to get a job done like how do we make this easier? How do we make the flow something more like this? So we took the data we'd already generated and we created like a little bit of like a Hockey beta visualization And we've put it online here At bridge cryo github Helm scanner and what this is is basically a point-in-time snapshot of every helm chart in artifact hub.io and what we're trying to highlight is The the kind of likeliness of an attack chain by mapping All of the misconfiguration items from the kubernetes the infrastructure configuration through to the containers that are used within that infrastructure configuration through to the cv's we have found within that container and powered by Palo Alto's image scanning capabilities that are now built back into check of And powered by check of infrastructure scanning capabilities We can do all of this in one tool and then we're effectively spitting out These dynamic graphs for each so if I quickly Go to that page so bridge cryo.github.io slash helm scanner You can see we have a bit of information here and then what we have is a searchable list of Every Helm chart we found and the search does work So you can go here. You can search for a single cve. You can search for a specific infrastructure check in check of Or you can search for the helm chart itself And then for each we can see the list of cv's and infrastructure issues But if we click on the blast radius graph for this We can then kind of start zooming in and really understand Whether this is somewhere we should be Um Focusing our time or not or whether, you know, maybe we have other issues. So first of all, yes, this docker container Quite a lot of cve's you can hover over them to see Uh, the cve the the name we are only showing by the way in this whole website Um cvss scores above five. So if you have something like this, it's it's probably a a good example of um a very insecure container image But we can see that this is coming from Um this helm chart It's only got one deployment called the default deployment, which is a home resource And we can see that this um has numerous Infrastructure misconfigurations provided by check off. So here, uh, it's not using a read-only file system. There's no liveness probe root containers are not intrinsically banned It's not using a char tag. So it's using uh, either latest or blank Which means it'd be very easy potentially to do upstream attacks on that container And run on people's infrastructures And the idea behind these is it helps us have conversations around prioritization. It's not perfect We did it just to kind of, you know, highlight these issues to You guys and girls the wider audience But it gives you something to play with to kind of start modeling how important it is to not just consider cvs Not just consider misconfigurations Uh in your infrastructure. So if your infrastructure team that does your kubernetes deployment is still completely separate to your teams that deal with your code and think about words like cve Um, hopefully this diagram explains why, you know, those teams need to start working together to look at prioritization of threats And so this puts us in a scenario where You know, we can start going well, okay, um I can see that in this particular home chart um I have an issue a quite a serious issue where I have roles and cluster roles defined that are using wild cards So if someone got the hands on the credentials For the roles i'm creating there who knows what they could have access to generally speaking if you find stars in anything be it iam or Security configurations. It's a bad bad thing. So, okay. Um, that's something I should probably look at But could anyone use that? Is there an attack chain going back to kind of if someone popped our container with a cve Um, is there a way of getting anywhere else into our infrastructure because you know, if they just stick in the pod That's a pretty good day for an attack if they manage to end up in our kubernetes cluster. That's that's not so great and so We we've seen that they have wild cards But we've also seen that the deployment that's, you know, within the same home chart These roles are set up for this deployment but this deployment exports service account tokens mounted into the containers so Whatever we were giving wildcard access to um will be available in that pod Um, if someone, you know gained gained kind of control over that pod So that's not necessarily a good thing and kind of, you know, you could then go well, okay Um, let's go back and look at the csvs within this content the containers within this home chart What are the chances we have any kind of remote access cve's that could get to these misconfigurations in the first place? And it just allows you to have a much more kind of combined conversation about security posture and prioritization And by the way, this isn't the end of the research and You know what we're planning to build into the product here It's just kind of a really nice way of visualizing how important it is to treat these two things as um You know the same when it comes to security posture And then for kind of people that are dealing with this every day again still very much on the developer persona but a lot of this is you just have to have the context of The people deploying to kubernetes the people writing code that is going to end up in in these environments be at kubernetes Be it terraform Other considerations specifically around hell You know A lot of images we scanned, you know for cve's don't just have cve's they also have tools that Don't really need to be there. Um, they're tools that are useful for developers, but not useful for running production code So it's amazing how many containers you have running in your production workloads will have Uh web browsers like curl installed and tools like ns enter which allows you to try and break out of You know linux c group namespaces Um, you might have sue installed that allows you to you know try bashing against Um root accounts, you know sue is a usually a chmod Root binary probably something you don't want on your image. Um, so all these tools not necessary They give a hacker extra options Um, like downloading files really easily with curl Um, if they do manage to get themselves into a pod, they do manage to get a shell on that pod Um, so with those tools that image is easily equipped to enumerate the network with curl serve malicious content You can abuse our back with ns enter Um, and potentially a life of breakout to the host You know, whereas if you are making it hard um If we're if you're making it harder without having those tools, you know It's defense in depth and it's just something else to look out for while we're talking about security considerations Um, as I said problem two and we'll come back to this now is this Security isn't a point in time. Um, we've got a question by the way on the q&a I'll answer it live. Uh, it's check off similar to what one can achieve with cubescape um Not being intimately familiar with cubescape. Um, if it is kind of a policy engine that kind of Looks through Your kubernetes configuration and goes hey, yeah That's that's a bad idea. That's a bad idea. Um, you know, you probably need to tweak this deployment to Secure this or to not ruin root containers Yes, if if it's kind of rules engine based and then check off does the same kind of thing We have a load of built-in rules. Um, but check off is also multi infrastructure as code. So it doesn't just do kubernetes as I said it does kubernetes it passes home charts automatically it does customize as of this pr things like that So the helm scanner data second question as well And by all means anonymous attendee on the cubescape thing if you want to reach out to me on twitter afterwards I'll have a look and give you a much more in-depth answer um Karen um Thanks for the question. How will the helm scanner database be updated? At the moment it isn't at the moment. It's a point of time Um, just because we were hitting the artifact hub API is pretty hard to download all of the home data It does run and generate a new set of data. So we might turn that on as a github action CICD job in future Um, if that's an interesting use case for you if you know api Access to that kind of data is interesting again Love to have that conversation building use cases on this finding other people kind of interested in kind of similar research So, uh, yeah, definitely reach out Um, so yeah problem two kind of security isn't point in time Um, this is you know more of a solved problem Which is why I wanted to kind of touch on kind of the new research around blast radius um bridge crew part of Palo Alto is Literally a one-stop-shop to solve this problem Regardless of whether you're using kubernetes Or other kinds of infrastructure as code like terraform cloud formation. You name it Um, and the way we deal with this is we basically offer a platform Which applies the same check off policies effectively, but at every single stage of this um This diagram automatically So, um within bridge crew you can integrate it with your source control be it github or any of the integrations You see there you can integrate it with cloud providers to get um real time runtime information Um, and then you can also integrate through check off or other cicd plugins like github actions Um to make sure we can see not only the code at rest We can see whether that code has gone through cicd on its way to production or been blocked or not because of violations But then we can also see the resulting Uh objects that have been created in your cloud environment And we can tie those cloud environment objects back to what has been deployed through cicd and where that lies in um your code base which allows us to do some really cool things so we can Automate fixes um to basically give you like a virtual security advocate By scanning pull requests as they come into your team um and Once they're there we can go. Hey actually We know that you need to fix This piece of terraform and this kubernetes manifest here is how you would do that here are the issues with that manifest again Just kind of providing some context and providing a developer that maybe isn't particularly security savvy or You know doesn't have particular contextual knowledge of the security configuration of that project that there is something in this pr which they should probably send for further review that could affect security um Here you can see more of that pr where we are actively saying. Yep. Here are some issues um, and in some cases adding Pull request data to actually remediate those and go. Hey, yeah Do you know what that's a boolean flag like versioning is either on or off for a s3 bucket for example So bridge crew allows you to will suggest those kind of fixes And allow you to raise prs back into your organization To kind of zero touch fix those issues And then because we are tracking runtime and the original resources in git We know of the format that you wanted them to be in so we can also and again This is why it's important to do everything across that diagram We can also do automated drift detection. So you can see here. This is a live aws environment And what we're saying is This piece of infrastructure is code defined that the acl should be private For whatever reason be it indication of compromise be it someone logging into the cloud environment manually To to fix a bug or to to do something You know we can see that that private is now missing from the runtime resource like something has changed something has drifted And we can alert on that As well, which obviously if you were only scanning your cicd pipeline and just assuming once it's in production, it's fine You'd miss that Um a couple more questions while we have time So, um the sue curl ns enter someone was asking what do those mean? so these are Common linux commands the developers will use on their kind of desktops and on their servers To do various things. So for example curl allows you to retrieve downloads from A web browser a web server. So just like chrome but on the command line effectively However having these in your Containers where you are running your production code base. There's no need for them So any docker containers that kind of have these kind of tools You're just exposing Unnecessary tools that an attacker could potentially take use of Not a security vulnerability in themselves not hacking tools by any means but again just things to watch out for minimizing that kind of Attack surface And then richard asks from a kates aspect any advantage security wise using customize Over helm or even palumi Um good question. So hang on. Let me break that down into a couple um No real security advantage because Chances are you're not going to find like if you go and find a random snippet of Kubernetes online It will from my personal experience be just as vulnerable as the Available helm and the available customize charts and that's purely because they're not trying to be vulnerable um, it's that security versus usability issue It works and it's simple to understand and therefore people will use it because it's simple to understand So it gets upvoted and it gets reused and it gets reshared um, so no effectively all customize and helm do is template out to Uh, kubernetes manifests anyway. So in terms of security posture, they're all much of a muchness I'd say customize and helm charts are more secure And hear me out here because they're in a format that is designed to be shared Um, it means that you know teams like myself Palo Alto, it means that you know security researchers are more likely to target submitting pull requests and fixing security posture issues Or reviewing the security posture of something that is we can tell from artifact hub has been downloaded millions of times Uh, because it's designed to be reused rather than just a random snippet of Ter of kubernetes or terraform in a github gist, which you know, it's just someone's Manifest that isn't really designed to be reused. So, um reuse is definitely good and hopefully it gets more people looking at those items so Yeah, uh over time with kind of these issues more and more highlighted Uh, hopefully helm and customize will end up with a much better security posture And then um, I'll get to Yuri's question in a mo. Um, I just want to get through the, um Little bits of advice we have here. So yeah, definitely check out bridgecrew.cloud And our blog at bridgecrew.io slash blog for information on kind of question two Because we do literally live and breathe dealing with end-to-end infrastructure as code scanning And and looking at how we can tie blast radius Into everything I've just showed you in those screenshots as well. It's a free trial. Go check it out. It's actually really awesome um a bit of advice, um I wanted to give now and again I'm going to split this into two personas because everything we've kind of looked at It's kind of deep dived on the coalface like people writing code and um, you know people deploying kubernetes manifests and I hope you appreciate from the nuance of what we've been talking about They are the people you need involved in this in these security conversations because That's where all the context is that's where Whether things are around accessible or need to be accessible or whether that does or doesn't pose a security risk Gone are the days where you can run a single security scanning tool against a vm and go Yep, that's fine. Uh, and that can be a separate team So from a practitioner's point of view, um, like I said, we have uh, a lot of blog posts where we are focusing on um Taking an insecure kubernetes manifests and looking at what we actually need to do step by step Um, and what each line that we're adding does and why we're adding it and context around that So, uh, my colleague steve was responsible for the large engine x Manifest that was secure and he goes through that in a blog post and step by step shows why each of those are there and equally I've done the same for a vulnerable helm chart where I go through and make modifications to that helm chart to make it secure in terms of all the kind of kubernetes checkoff policies And then um, finally If you are in a oh my god, I've got thousands of infrastructurers code issues We also have a baseline feature which allows you to run checkoff in ci and only block on new issues Rather than issues that were there when you first ran That's a really good way of kind of, you know, freeing up your developers to Make, you know, know that there's no new issues being introduced by new code and new infrastructure And kind of stops that technical debt getting bigger and allows you to then go back and prioritize and kind of look at Blast radius and and work out which of the existing issues you need to deal with and again Give bridge crew a try it is definitely your ally for Automating a lot of this if you don't fancy kind of rolling your own with checkoff and other tools And then to enable us to You know c-suite managers kind of anyone not doing the actual manifest and not doing the actual code My advice would be First of all security isn't no So, you know, there needs to be that security has to have a Security has to have a share of resource within any Within any team within any development work And it's not going to be solved by One big push one big sprint. Hey, let's just let's just spend this sprint getting all the security issues done because as I said Entropy will guarantee that that will never work Little and often, you know, allowing Through prioritization through kind of looking at the the checkoff Baselining mode Allowing a little bit of time More often to kind of, you know, chip away at high priority issues Is is definitely a more sustainable kind of way to deal with that Try and embrace passion where you can you might find that, you know, there are People that have kind of that enjoy this nuance that talk about security issues on your team more than other developers That have kind of that hacker spirit that that are almost already seen as like a Security advocate within a dev team or an ops team. Um, you know Nurture that use that give maybe eke them out a bit of time To kind of start thinking about a strategy for dealing with this Checkoff and bridge crew are going to let you know what you didn't already know and you know Having that person be able to come to you with a prioritized list of issues is Um, you know a massive part of the challenge once you once you turn your unknown Into kind of knowns then you can start prioritizing what you want to do about them Um, and there is a lot you can do Um in a small amount of time To as I said use use checkoff use tools to at least understand your posture But even just getting low hanging fruits even just making sure you've not got kind of keys or you're not exposing Kubernetes tokens where you don't need to or making sure you don't have any wide open load balances or You know all that kind of good stuff There's a lot you can do in a small amount of time It just requires a bit of work to kind of prioritize and understand your posture Um, and then yeah, finally here is my slide on takeaways. Um, the one I'm going to highlight is The more you automate bridge crew checkoff ci Um For every hour spent automating is going to free up potentially an hour once a week once a month once a day once a once You know, whatever for one of your devs and that next hour They've saved next week can be used to automate a bit more and a bit more Um, it will feel like work. The defaults are not secure You will start digging your way through thousands of issues if you already have a reasonably large code base Um, but misconfigurations are just as likely to expose you to compromise as CBEs Especially when used together. So, um, yeah, take this stuff seriously. I know everyone does Um, but it will feel like work, but um, yeah small steps and hopefully this was really insightful for some of you. Um I'm gonna quickly if we have time I think about like 30 seconds. Um Is it possible to run checkoff uh tool already running on prem bare metal? Yeah, absolutely. Um checkoff will happily run against your kubernetes, um manifest. So if you have them in a repo before you've deployed them, um, we are also in the process There's a PR for it open right now. Uh in the checkoff, um repo We're in the middle of building an admission controller Which would then run a subset of the checkoff policies when you try and deploy or update something in kubernetes Um, so you know immediately, uh, without having to have a ci pipeline or anything like that on some of the more critical issues Uh checkoff is completely open source checkoff dot i o Let me find you how to spell that. There you go c h e c k o v dot i o Um, we love suggestions pull requests contributions. Um, yeah, it is fully open source and we have a massive community at, um slack dot bridge crew dot i o because uh bridge crew Are the authors and still sponsor and and maintain its development to this day. So come and join us say hi And uh, hopefully i've got to everyone's question and yeah, thank you so much for listening. Hope uh, hope this has been useful Wonderful. Thank you so much matt and thank you to everyone who joined us today This recording will be up on our youtube by end of day today and we hope to see you back for a future webinar. Thanks everyone Thanks, everyone