 Hello, and welcome to our talk, Learn by Hacking. We're infinitely grateful that our arcs are the Cloud Native Enlightenment, and we extend a heartfelt thank you to you, the attendees, volunteers, contributors, and the Cloud Native Computing Foundation involved in the vision and delivery of Kubernetes. Documentation and bug fixes do not write themselves. And the incredible selfless contributions that drive open source communities have never been more freely given or gracefully received. Security controls are generally more difficult to get right in complex orchestrations with the functionality that Kubernetes is known for. To these security teams especially, we thank you for your hard work. The tax security CTFs are a reflection of the pioneering voyage of the good ship of Kubernetes out in the choppy and dangerous free seas of the internet. We have a free book available to download written by myself and my veritable co-author, Mr. Michael Hasenblas. There may be useful tips and tricks for individuals looking to assail the Cloud Native infrastructure if you're intrigued enough by this talk to join us on Thursday and play our new scenarios. So we're going to talk about how we run CTFs at Cloud Native SecurityCon over the past years. What we did, what we helped to achieve for you the attendees and how we maintained our fragile sanity while provisioning thousands of notes on demand per day. Over the years, we have run CTFs in person virtually. Planning for these events began in the ethereal and misty nights of 2019. As we looked to determine what it would take to put together a CTF running on Kubernetes, for Kubernetes, without having the entire infrastructure come crumbling down around us, nothing is more terrifying than building a self-hosted security game and inviting all and sundry to have a blast at it. Our goals are to communicate how best to secure Cloud Native infrastructure. They are red team scenarios with full demonstrations, teardowns, and remediations. The best form, as Sun Tzu has told us, of defense is attack. Know thyself, and we look to give something back to the community that is so generously shared, so much with us. Here stands the nefarious pirate captain, Hashjack, leading the charge with his eponymous bag of Bitcoin and the untitled goose that sparked so much security enthusiasm across the last few years. This is a learning experience for all skill levels, starting with an introductory scenario at each event, building up in difficulty, progressing as a game that gets more and more difficult throughout the journey to captivate and intrigue all skill levels and help nurture and seat the interest in and around security. With dedicated infrastructure, we look to emulate as much of the real world systems at some cost of complexity that we are going to dwell slightly further into the presentation. The goal is not to find the ultimate security ninja, but the rising tide that will lift all boats and that players have fast feedback and support to enhance their learning experience. But first, here is a digital reconstruction by the street artist Banksy of what a bunch of pirates would look like playing a cloud native CTF. With a splash of stable diffusion for effect as well. So there is no such thing as secure software. As fast as we can possibly paint over the colorful selection of vulnerabilities in our infrastructure, new features are shipped. Software moves quickly. And those new features potentially feature untested code paths. And any of those untested code paths with security side effects are by default vulnerabilities. We prioritize shipping features over fixing vulnerabilities at our peril, of course. And this is the reality of modern software that we balance in our day to day every day. Resemblance to the friction we see between red teams and blue teams, any resemblance in that last animation is not a coincidence. Organizations not only evolve for competitive advantage, they must do so in order to survive because their adversaries are also evolving. So how do we represent that complex puzzle in a safe learning environment? As SNARIO authors, we look to construct the most realistic SNARIOs that we can manage. That includes building infrastructure diagrams and threat modeling SNARIOs. It helps us develop with greatest degree of realism and is born from our experience consulting with regulated industries and startups alike. Here we can see a supply chain attack in progress. Captain Hashjack has attacked systems through various network systems in the past, including packing repositories, container registries, and the notable hypertext coffee pot control protocol. With no stone left unhindered, running the platform on real infrastructure gives us the opportunity to emulate simulations we see in real life. Coffee pots and IoT pots and IoT attacks are not the exception. And we are looking to educate as many people as possible as effectively as we can. Here we see a pod security treasure map from the book Hacking Kubernetes. These scenarios are intentionally increasingly complex and difficult. This helps to crystallize players' understanding of a wide and complex security landscape. And as with all CTFs, attendees and players must enumerate the visible horizon, escalate their privilege, reverse back from dead ends and intentional misdirections. This pod treasure map details many of the routes that an attacker might explore within a Kubernetes cluster. But with each scenario, there is only one intentional route for a player to find. With the notes and caveats, the players have found multiple bypasses, unintended shortcuts, and wily workarounds as noble routes to completion. So how's the CTF plate? With a terminal, IP address, and optional authentication credentials for an SSH session, a Kubernetes cluster, or other mystery piece of infrastructure. Here we see the first view of a cluster that a player has when placed in a root-enabled container running in cluster. They must enumerate the visible, uncover the invisible, and start to explore potential routes of compromise in the cluster. This may include hijacking sessions and credentials, attacking APIs, and data storage routable from the starting point, and escalating privilege through any and all identity mechanisms a cloud-native system may have. My personal preference is to bring a portable set of bash scripts to speed up enumeration and exploitation. However, there are no rules what tooling you can bring with you. You just need to find somewhere to execute it from. Over the years, we have looked at a whole host of vulnerabilities and misconfigurations from container image file system foobars and runaway runtimes to secure config, malicious mounts, unsurquested secrets, privileged pod security policies, sensitive service accounts, no firewall networking, awful admission control, exposed at CDs, terrible TLS implementations, federation failures, chronic control plane configurations, and unlimited user exposures. And there's more from ingress to the kernel. We've played this remotely, we're playing it in person this time around, adapting to the pandemic and running from virtual events, online gatherings, running the infrastructure over Slack. Peace was never an option. So how did we go about this high-pressure, live deployment of 2,500 nodes across each day we play the game? We start with the open source breach simulator tool built to help control planes clients secure their clouds. One five node cluster is provisioned for each scenario. And cluster reuse is not enabled. We add a double dose of supermassive provisioning engines backed by Knative scaling to zero. These are not centralized in a unified engine. This sits on the back of a PubSub queue and re-entrant state machine, which tries to ensure that even half provisioned clusters are forced into life and reinvigorated when half deployed, which just means distributed systems, as we know are hard. Then the unique credentials are distributed to each player for each scenario cluster pairing and the games begin. Our trusty taskmasters are on hand in person and on Slack. To hand out hints, offer suggestions and guide people through the path of the challenges. All this adds up. We spin up the clusters as needed and we hit the lofty heights of two and a half thousand nodes spun up per day. With 80 to 100 participants, we hope to beat this record on Thursday. What has gone wrong? Would you really believe us if we said nothing? There was one time an AWS internal Ubuntu mirror went down. As you can imagine, nothing completed provisioning. Simulator was just host, dead in the water. Health checks decidedly unhealthy to showcase that. Typical distributed system problems when EC2 wouldn't provision VMs, a very red dashboard, a rising sense of panic came from it that required us reprovisioning into a different region post-haste. That time, somebody figured out how to bypass the initial SSH tunnel sandboxing, which was also interesting. Securely provisioning insecure infrastructure is hard. We didn't want it too insecure. When the cluster admin RBACs, the RBAC credentials disappeared, gone, found their way into a bastion node when the test suite only covered the happy paths also. And through it all, we seem to have hit our goal. We're thankful to everybody who has played the game, given us feedback and give us the opportunity to iterate on this wonderful, thoroughly enjoyable platform that we love to play. CTLs can often be big, can feel daunting. So we hope that you focus on the learning experience. Walk through the yes scenarios and if you feel like smashing machines, channel that please. And our clusters and scenarios today are of course open source running on this cloud native breach simulation underneath the Kubernetes simulated project. There is a hosted version and control plane are running a private beta if you would like to be involved. As you can see, nothing is perfect. There's an out of bounds exception on the end of the slide. If you would like to pop some shells with us, please come and play tomorrow.