 It's more than just open source. It's about connecting with people. It's about being part of the community. It's about sharing what you know and helping others. QCon is the best place to get hooked into the community and learn from everybody. And let me tell you, people, this is just the beginning. Welcome, everyone. We are alive. Yeah, this is the first live CTF stream for Cloud Native Security Day. Thank you. Thank you for the hat. I'm already getting some compliments. So we're going to have some fun here today. I'm going to introduce our guest speakers. And then we're going to talk about Cloud Native Security. We're going to talk about some challenges, some challenges of the CTF. Who's doing the CTF? There's a message on the chat. If you need any help or tips, we have the best people here to help. So going around the clock, I'm going to let everyone introduce themselves. So let's start with Liz Rice. Hi, yes. So my name is Liz Rice. I work at iSurveillance, who are the company behind the Cilium Networking Project. And I am also the chair of the Technical Oversight Committee for the CNC Act, which means I'm involved in a few things this week. So you might be, you know, you might see me on the screen a few times this week. I also wrote a book called Container Security. So apparently I'm supposed to know a few things about security. I guess we'll find out in the next hour. Awesome. Awesome. That's great. What about you, David? Yeah, I'm David McKay. I'm in Glasgow, Scotland. I am a developer advocate at Equinex Metal, specializing in cloud-native kubernetes and the surrounding ecosystem. I will explicitly say I know very little about security. I love your show. Your closer show is awesome and I learned so much from it. Yeah, I like to make myself suffer on a weekly basis with clustered and fixed broken things. So yeah, thanks. Awesome. Okay, what about you, Ron? Introduce yourself. Yeah, sure. So, hey, my name is Manavarra. I'm the CEO of Confaores, my new startup called Oxide. And we're building a cloud-native application security testing. That's me. Awesome. That's great. And Rory. Oh, so I'm a cloud-native security advocate for Aqua Security. And I'm also, like David, a West Coast of Scotland. So we have to be careful with our accents so that people can understand out of the chat. Yeah, awesome. That's great. What about you, Diego? Hi, I'm Diego Comas. I'm a security engineer working at Messagebird and I'm dealing with all the security about the modern workloads in the cloud for our SaaS company. And yeah, I have some good experience in protecting Kubernetes clusters and all around that. Okay, awesome. And we also have here the masterminds behind the CTF challenges, the test masters, the great Lewis. Well, that is quite the introduction. Usually, people just say Lewis. But thank you. Yeah, that's great. Yeah, so yes, I'm Lewis. I'm the head of training at Control Plane. I get to work with Mr. Andrew Martin who will give an elaborate introduction in a moment, I'm sure. So yes, I've been part of this campaign, I was about to say, through the CTF scenarios. And yeah, really looking forward to seeing how everyone enjoys the day and hopefully they're popping out of some containers soon enough. Awesome, awesome. And Andy. Hello, I am Andy, CEO at Control Plane and general hacker of stuff. I've put the CTF together based upon collaboration with people here and some of the work we've done for SANS. We've got SANS Sec 584, which is a course that is coming back for July. And also there's a new book coming up called Hacking Kubernetes with O'Reilly. So the sum and coalescing of all of those things brings us to the CTF today and hopefully some orthogonal thinking and mental hijinks to crack the flags. Awesome, and Lou is an Andy. Can you give us a little bit of overview of what's going on so far? Yeah, I mean, from an infrastructure perspective, we've stood up getting on for 200 BPCs so far. Obviously things are going up and being torn down quickly. Stuff doesn't persist for obvious reasons. And from a scenario perspective, today's work and today's challenges are all based around this nefarious, evil pirate captain, Hashjack. And this guy has, in the spirit of 2020 and 2021, attacked our supply chain, dropped malicious packages, left, right and center all over our infrastructure. And we have to go through all the clusters to understand how did he do that to retrace his steps and to find the flag that he's dropped in various well-hidden, I think, places. Awesome, that's great. And Louis, how is the test master row going? Well, I've never been so popular, to be honest, Magno. It's absolutely phenomenal. You start the day thinking, I hope someone comes to CTF and before you know it, you just see Slack filled with an unbelievable amount of notifications. So, yes, thank you for everyone to the patients as well so far today. It's, we're working through them. So there's six of us within control plane who are taking the mantle of Taskmaster behind the scenes. As Andy said, we are actively spinning up these clusters and we're looking to get them across to you as soon as possible. If you want to join in with the cluster, just send us a DM on Taskmaster. We are posting as well on the main CN security day CTF channel as well, so you'll find us there. Awesome. And do you have any other reminders for people to start the CTF? Do they need then any prior knowledge or anything like that? Just try harder. Okay. Now, the point of today is, and I'm so happy and it's such a pleasure to be on a panel with so many people that I can literally look up to and around to. We've all had to start somewhere. I remember going to my first talk where I saw Andrew Martin, that one there, give a talk with Ben Hall of Catechoda fame and just being amazed by what they were doing. And now I get to do that on a daily basis, but equally, I get to do it with the peers around me as well. So if you do have a question, if it is a little bit tough, if this is your first time trying to pop out of a container, just drop us a DM, drop Taskmaster a DM, search Google or any other search engine. Just it's all about having fun. And hopefully by the end of today, you'll take away learning something new. Awesome. That's great. Thank you, Luis. Yeah. So we're going to have two sessions today. This is the first session and then we have another session at 4 p.m. Central Europe time, right? So make sure you have your clocks ready and your time zones ready and who's ready to break some clusters. Yeah. Awesome. Okay. Yeah. So let's start interviewing some of our gas speakers here. Les, can you tell us a little bit about yourself and how you got involved with cloud native security? Oh, that's quite a long story. I think I was involved with containers. I was a co-founder of a startup that did container autoscaling before its time. And we were terrible in terms of being a business, but we were quite good at the technology and I learned an awful lot about what containers were. Found myself explaining that to a lot of people. And when that stuff ran out of money, I found myself at Aqua security. And up until I met the folks at Aqua, I wasn't particularly involved in security or all that excited about it. But I met the folks from Aqua. They had some really interesting things going on. I got very excited about security and how knowing how a container works and knowing how containers fit into a Kubernetes environment gives you a huge insight into how that container might be attacked. So actually going from just understanding the bones of a container to, okay, now how could you attack it or how could you defend it? That seemed like a fairly natural step. So that's kind of how I got into security. And it turned out to be really fascinating. And I had no idea before I got involved in security about things like CTFs. And they're super good fun. So I think everybody's going to have a really good day. Awesome. That's great. Yeah. That's awesome. What about you, David? How did you start in this cloud native world? I don't know if I'm lucky or unlucky, but whatever it is, I've always worked for relatively, until now, Equinix, but I've always worked for small organizations. And that just meant that I was usually responsible for running my own code and production. So you kind of have to learn very quickly when that's the case. So as I jumped from small companies, small company, and adopting containers, and then Kubernetes, I've just, I've always had to be mindful of the production concerns to good at concerns. And I'm going to give a hat tip to Liz as well. Like you're building your own container top from like 2015, 2016, whenever that was, was one of the talks that got me really interested in joining and doing more with the Docker community back then. So thank you for that. Oh, thank you. Really insightful. And live coding that, I think it was container camp in London, many years ago, was just super cool. And that always stuck with it. Awesome. That's great. Yeah, sharing some love here between the guest speakers, that's great. And I really love your show, David, the cluster show that you have on YouTube. Can you tell us a little bit about how you came up with that idea? You're going to let me plug my own show? Yeah, like how do we go? Just a couple of minutes. Yeah, I think, you know, there's a lot of content online for learning Kubernetes. And I think sometimes we just need to go a little bit deeper, especially, you know, we've got all these certifications now, the CTA and the CKD and the CKS. And trying to provide content that is both engaging and formative was just something that's kind of a mission for me. With the idea of a cluster, I was like, what if I just have people break a cluster? And my initial idea was to have people tweak commands and then have them autoplay in the cluster and then I would try and fix it later. But it turns out the security of that was a bit challenging. So I just decided to get members of the community involved. I reached out to people I know in the Kubernetes space and was like, hey, if I give you a cube config, SSH access to bare metal cluster, can you break it? Turns out the answer for most people is yes. Like, hell, yeah, I'll do that. So I gave out these clusters. People broke them. I go onto YouTube and have more people join me and we try and fix it. Waleed, who's in the chat, he was the first fixer. We both sat on a stream for nearly two hours trying to fix two clusters. And it's great fun. You get to explore how QBDM works, the API server, all the control playing components. And it turns out people are pretty evil with the breaks too. So there's always something fun and exciting every week as well. Awesome. Yeah, that's great. Thank you. Thank you. What about you, Rory? Can you tell us a little bit about yourself and how you got involved with cloud native security? Yeah, so I actually suppose I come from the other side of the house. My background is IT security. So I've been in IT security for some years now. And in my last role, I was a pentester. And the way the world works in pentesting is when a new technology comes along, some poor victim gets nominated as the person who knows most about it. And for Docker, that was me. I encountered it back in 2015-ish, started exploring it, poking it like pentesters do, trying to see what you could do with it. And then it basically snowballed from there. I got Kubernetes when I was doing a talk down at Peacehouse London and a customer in central government actually said, hey, we're deploying Kubernetes. This was 2016. So they were very early adopters. So then I had to learn Kubernetes. And really, it just kind of went from there. I got involved in the CIS benchmarks when I realized there wasn't enough documentation around security. And then started doing things like training courses. So and it's been a great fun. New areas of tech are always great for pentesters because there's always rough edges. There's always things that people are going to learn. It's going to get better over time. And being there as that grows is generally the best part of time for pentesters anyway. Awesome. That's great. Yeah. So, okay. Andy, what about you? How did you start with your cloud native security? That is an excellent question. I guess it was a product of just being working around London in financial services and around security in general. And seeing how things were, as always, interminably not only broken, but also gaffer taped together. There's bits of band-aids everywhere. And you lift the hood of stuff and you can see the difficulty of deploying it securely. So when containers hit Hacker News, when Docker 0.4, I think, hit Hacker News in 2013, I'm going to guess, it sent ripples through the office because everyone was thinking, oh, well, actually, all of these issues we have with, I mean, in the first place, just developing on a Mac and deploying to a Linux system, maybe we can abstract those without using VMs. The reality of the situation was actually, as Rory says, rough edges is exactly how it was. And some of the things you had to do to get applications running in Docker early on were just super crazy and meant that you had to dig into bits of kernel code and look at how the APIs were implemented. And of course, that's just a red rag to a bull for somebody who's interested in rabbit holes and how long is a piece of string style, philosophical questions. So from there, it just kind of made sense. This is obviously the direction of travel for the industry. The compartmentalization and distribution is just awesome. It gives us this works on my machine, anti-pattern defense. And we're still in this position where actually containers are not a lot of security boundary. And we're kind of wondering what's going on. And then Jeff Rizal started to ship the security features into Docker. And it was just awesome to see some of the stuff that she did. There was the process of defining a set-comp profile that would work uniformly for all applications. And I mean, some of it's kind of obvious, right? Dropping 32-bit system calls. Well, yeah, but we're all 64-bit. We don't care about those. But some of the other stuff, and around the app armor as well, looking at how Petrace was allowed within a container but could be used for breakouts. That was super instructive. I have to say as well, knowing Liz from the community at the time, I remember seeing her talk on micro-scaler, micro-badger, at Hacker News London, I think. Wow, yeah. Back in the day. And yes, there was just this kind of nexus of people who were really interested in these things and had useful opinions. And it just blew up from there. Just the inspiration that I got from other people's work, combined with the very obvious, I mean, this is just how you fix things. But then everyone's still kind of kicking things around. We're on Core OS. We're just trying to run units with system D. And then Kubernetes is launched a couple of years later. And yeah, it all went uphill from there. Awesome. That's great. Oh, excuse me. Okay. Diego, you want to grab the mic now? Yeah, yeah, I've seen all these spectacular people around the expert where you were going to just tackle and see what they know. So, Liz, in the CTF, and how would you recommend to people, what is your go tool to inspect the security of a cluster, either of CTF or Blue Team Exercise? I know there are many tools, but which one would you choose first? To help you identify misconfiguration weaknesses or whatever can help in the CTF? Was that to me? I didn't miss hear that, did I? Yeah. So, there are lots of tools I could shout out for, but if I'm honest, my kind of first gut reaction is to try and get a shell inside a container or a shell onto the host. Hopefully I'm, you know, I have that permission and so that I can look at things like what's happening in slash proc. That always seems like a good place to start. Also, Curl, I have a feeling that proper security professionals have a tendency to go for other more security-oriented tools, but I quite often just resort to Curl to see what I can hit interface-wise. So, that's kind of where I tend to start. Excellent. That's great. And David, what will be your choice then or what will be your approach? That's a great question. I don't know. I don't do a lot of pen testing exploitation stuff. I mean, I just use the basic cube control. Off can I see what I've got access to within a cluster. If I have the ability to spin up a new pod, I'm going to give a shout out to Nexity.dev. I don't know if anyone's familiar with that, but it's like a dynamic container builder in the cloud that uses these next packages to give you dynamic images with the tools that you need. So, you know, I know a lot of people in this space have like their huge images with all the tools in the world, but, you know, depending on the environment you're working with, that's quite difficult to pull down, whereas Nexity can give me really small, targeted tools like that. So, I'll drop a link to that in the Twitch chat for people to play with. Really cool. I recommend checking that out. Yeah, excellent. Yeah, we'll pause the link because I'm sure that will help a lot of people. And next one, Rory. So, yeah. That's what I suppose I've done quite a bit of. For me, it tends to depend on the scenario I've been given if it's external. So, if I'm external to the cluster, I'm going to start by port scanning. I'm going to start by saying, what services do you have running? And then, what can I do on authenticated? Curl, to be honest, is a great choice because it's always available. So, I might hit all the external ports with curl and just say what responds back. Things like SED, if you can curl the version endpoint in SED, that says authentication is not turned on. So, I know I can go from there without having to set up SEDcattle. If I get dropped into a pod, then there's some tools that are good for that. So, things like just for sales, am I contained? That's a great way of saying I'm dropped into a pod. What can that pod do? You can do it manually and you can, you know, if you don't have that, you can go look and run the Linux host itself. But using am I contained gets you a kind of good position there. If I'm going to, if I'm doing a malicious user scenario, so you've been given a cube config with like maybe create pod, I'm going to use a cubecattle who can and I'm going to use cubecattle off can I? So, what can I do? I look for permissions that are dangerous. So, can I list secrets? That's a good one. List secrets is the same as get secrets, which gives you all the access. If you can create pod and there's no pod security policy, I can drop a pod down that's going to get root access. So, there's kind of a number of ways you can approach it, depending on the scenario. But generally, just like what's there, what can I see? And then, I guess it's a lot about knowing what an ordinary cluster looks like. So, if you say ordinarily when I run anime contained, I get this list and in this cluster, I get a different list. Oh, that's interesting because that's different. And it's what's different that's interesting to me. Because I might say, hey, that's different. Therefore, if I, especially I'm CTFing, right, because with CTF, you're looking for like, what's different? What have they done here? And so, if I run my usual tools and go, this is a different response. Oh, that's somewhere I should probably need to start exploring because it's not what I would usually see. Great. Thank you. Yeah, anyone else? So, Andy, do you want to? Yeah, I suppose. The first thing I do is generally check if I'm in a container. As Rory says, am I contained as super useful? We can check the mount points to see is anything unusual mounted in. We can have a comproc to understand as Liz says, if we've actually got access to an unmasked host proc namespace, and I guess things generally fall into one of a few categories. It's either the local file system access to other systems based upon what's in it or out on the network. So, this comes down again to shipping sort of minimum viable pods, I guess, or container images. So, if Curls available, if W gets available, if Bash is available, then we can get out onto the network and see what's there. The Bash virtual TCP mount point in Dev TCP, I mean, that's a Bashism. That's not Linux. That's not Shell. So, just having Bash versus having SH in a pod or in a Shell container environment is bad enough by itself. Because I've got a penchant for Bash and I don't really know how it developed into such an obsession, I've re-implemented things like Curl and W get in Bash. So, even if I've got an immutable file system and just a Shell, I can still go and enumerate what else is available on my local network segment. So, yeah, keep those containers slim. And I guess it's all network policy and file system mount protection, standards, Linux, discretionary access control, all of those things, just traditional pen testing. But again, as Rory says, knowing that you're in Kubernetes and that you can route to the API server, that you've got those environment variables, you can enumerate the DNS, all of those things can contextually take it from standard pen testing into that kind of orchestrator-specific stuff. That's right. Thank you, Andy. And yeah, then the next one will be about, if you want to start, please. What do you think are the most common areas of opportunity for organizations securing cloud native technologies and services? You can't really have that question without thinking supply chain. I think supply chain security is such a hot topic because we've seen real breaches in the last few months, things like the code curve breach, I totally understand how it happens, and I also don't understand how it happens that so many people were running code curve, myself included, and nobody was checking the shell sum of what they were downloading to execute. So I think that is a very quick win. We're going to see a lot of tooling making it very easy to check shell sums and other things to improve supply chain security. I couldn't also answer that question without just mentioning eBPF, and maybe we'll come back to eBPF later, but eBPF is just super powerful Linux, so it's going to be a hugely powerful tool for helping observe and detect security breaches. Yes, you're advocating against just flag, yes, download everything with text as yellow. I think it would be really nice if we had some tooling that when you just pipe something into Bash, curl to download and then pipe it into Bash, maybe we could just have something that would check that the hash sum has expected. Then we'll be debating developer experience and security, perhaps for another day. Okay, and the next one, Rory, what will be your answer to this? As I said, I'm going to ask again, what do you think are the most common areas of opportunities for organizations securing cloud-native technologies and services? Yeah, it's an interesting one. So to me, there's like an upside and a downside to cloud-native. The upside is the automation, the fact that you can automate things and you can make them repeatable and do it as infrastructure's code. So there's no more of, I've got this little snowflake server that's been modified by hand over the last five years and you don't have that anymore because that tended to lead to misconfigurations. It tended to lead to people leaving credentials or ports lying around that allowed attackers to get in. Now you've got a consistent environment. The downside to me of cloud-native security is the exposure, right? Everything's on the internet. Everything is one step away from a problem. It was a great talk I saw many years ago where someone coined the term FUT, which I'll politely call mess-up tolerance and it's what mess-up tolerance does your environment have. So like everyone makes mistakes, right? There's always going to be mistakes. There's always going to be misconfigurations. How many of those does it take for your entire environment to be compromised? If that's one, you've got a serious problem because there's always someone's going to click a link, someone's going to have an S3 bucket with the wrong ACL, something's going to go wrong. So the challenge of cloud-native security is making sure that your mess-up tolerance isn't one, it's two or three. So it requires multiple mistakes to have been made for something to go wrong. And that's where you see a lot of the challenges, right? I mean, how many blog posts have we seen that said this elastic search was left online without authentication? This S3 bucket was left online without authentication. In the old days of enterprise security, those would all have been behind the firewall. So the mess-up tolerance was two. First, the firewall had to get messed up, the attacker had to get into the environment, and then the access control issue had to happen. Now with everything being cloud-native, the mess-up tolerance drops down and becomes one, and people scanning around the internet find the thing, and the thing becomes a blog post, and it becomes an incident. So it's one of those things. I think it's an exciting new area, but people have to get those two balances out, right? You've got the advantages, but then here's some disadvantages to think about. Great, thank you. I think David didn't ask the question because his Firefox crash, he didn't answer the question, sorry. Yeah, so David, the question was, what do you think are the most common areas of opportunities for organizations securing cloud-native technologies and services? I don't think I have any more to add than what everyone else has added. I love all those answers, and yeah, I'll go with the automation from Rory's. I really like that point. Thank you, and do you want to add more to that, Andy? Yeah, I think it's a really strong point. We've come to cloud-native on the back of the DevOps evolution, revolution repeated headbanging against wall style, and all these things are programmable APIs. We have all these abstractions, we have all these opportunities for automation, and once we get to the complexity level of a Kubernetes, then without basing things on infrastructure as code, we are setting ourselves up to fail. Especially when we're looking at stuff like Zero Trust, and the Zero Trust mantra is, of course, trust, but verify, that doesn't mean putting administrative interfaces, public-facing, putting them straight on the public internet. We would still look to apply fence and depth, and I love Rory's characterization of that, and I will reuse it myself in the future. I'm going to credit Dr Ian Levy. In case you don't know me, he's a UK speaker, which is, that was his phrase, and I just loved it as a phrase, so I kind of reuse it. Super awesome. Awesome, that's great. Ron, you're up. Yeah, sure. So my next question will be, please. What do you think are the most common mistakes organizations are making when deploying the cloud-native technologies and services? I think, what are the- No need to mention any names here. Yeah, I think this is very similar to the previous question, in the sense that a lot of things that people get wrong are where they do things manually, or they just sort of, maybe they're standing something up that they didn't really intend to be production ready, and then suddenly it is production ready, or it's opened up more, and we're getting away from those mistakes through automation and through things like GitOps, adding in a layer of separation between human beings and their tendencies to kind of just make mistakes, really, and your production cluster. I think that's one of the really interesting things about GitOps, having that separation between and the production cluster. Really, the most common mistakes are just human fallibility. Great, thanks. David, what do you think? I think something interesting happened with both native and Kubernetes, and that we're all running on a standard stack and platform now. Take advantage of that, use common tooling as much as possible. If you find that you're having to write something yourself, that should be an indicator that you need to take a double check. Has this not been done before because it's not right? I think that's always something that's really useful. The fact that we're all running on Kubernetes now, we're all adopting the same technology, we should just try and leverage open source, leverage that collective knowledge that we have as a community. It doesn't mean you can go off the beaten track and write something yourself, but just make sure you double check that. Awesome, Rory, you're next. So I would pick up on the idea of it's about the mistake people are making is by not modifying their environment before they start deploying. So it's important with Cloud Native to try and get your security controls in before you go to production. So when you're defining your manifest, do your security context work before you get into production. Sometimes the defaults are not going to be suitable for your environment. So tools like Docker, Docker's famous, the way they did their security model was to make sure nothing broke. You know, when you do Docker run, it will always work. That means that they can't super harden the environment because if they super harden the environment, things will break. So if you need to look at your environment and say, right, well, I'm deploying something maybe high profile or high risk, these defaults don't work for me. Let's customize them before we get into production. Because one of the mantras of security is it's much harder to retrofit security than it is to put it in before you go in. You know, once you're live, trying to add firewall rules or network policies or pod security policies, you've got that fear of, I'm going to break the environment, right? I'm going to break production, which no one wants to do. So it's much better to get that in first. And what I've seen definitely when I was pentesting clusters, Kubernetes is complex. It's a difficult thing to get right. So people will take it out of the box and they will put it into production with all the defaults in place, which gave me a nice set of stock findings that went into every report saying, here are all the things you need to harden out of the box. So if you want to get that right, it's spend a little bit longer before you get into production, do your security policies, get your OPA or cavernal rules set up, get your network policies settled down to a reasonable degree. Then you can test it all with those controls in place and you're confident it's going to work, then go into production. This is great. I totally agree with you. Anyone else want to add something? We'll go to the next question. There's not much to add again. It's very comprehensive, but from my perspective, shipping shelves to production is something that everybody tends to do. Just because you need a shell to run the commands in a doc file, each of those run commands is executed in SH or bash, depending on how you configure it. So they're needed to some extent, but actually shipping them to production, like I said, leaves an executable environment that people can use to execute arbitrary code. It's a runtime like anything else. More or less, any interpreter can be used to run code. So if you've got Python or Ruby, you can just pipe your commands to it. So you don't need to persist a shell script or any sort of script. All you need is a terminal access. It's the same for most things. So getting an attacker as far away from arbitrary code execution as possible is the top hardening tip for keeping me away. See, this is the problem with your obsession with Batch, Andy, that you're going to have bash shells in all your containers. And this is a problem. You want to learn a proper language? Oh, well, I'm aggressively polied a lot, but I can do things so much faster in bash that... Yes, excuses, excuses. I just took an opportunity for a cheap bit of bash-bashing. I can confirm the bash. Being faster, still, I don't know. I was probably showing too much behind the curtain sort of in control plane, I'd say. The only other thing I'd add to this as well is that if you just get started with this, if you're new to Clive Native as well, just focus on the basics first and don't go for the high goals necessarily, and especially with today's CTF. It's a case of if you're finding it a little bit difficult, just send us a DM. We'll get to you. We'll help you out. We've all got to start somewhere. And focusing on those basics first, if I could go back in time, I wish I'd just started with the basics, like focusing on Linux namespaces, focusing on C Groups, seeing how those are managed within containers and building it up from there. That's just the only other thing I'd add to that. All the answers so far have been awesome. It's actually a really good point, the basics thing, and something which I've had discussions with customers where they jump straight to Kubernetes. So they're just starting their container journey and it's like, no, I want to deploy Kubernetes. And I'm like, you probably don't. You could do this with ECS or Cloud Run or something basic. All you really need is containers. Because when you add Kubernetes, it's a whole different layer of complication. Do you really need that right now? So it's not your point. Start with basics if you're doing containers in production. Start with maybe Docker on a VM and get used to how containers work. Then think about jumping to Kubernetes and adding that additional layer. Then maybe you think about service measure if you need it. But don't jump straight into the high end. A big plus one for Cloud Run as well. I don't know how many people are aware of the service, but it's just containers on Borg, but it's compatible with Knative as well. Some of the CTF is spun up using Cloud Run and it just scales horizontally. It's super nice. So yeah, great shot Rory. That's awesome. Yeah, I like that basics approach. I also run a weekly CTF show called UHC, U2Mate Hacking Championship. And the thing that people struggle the most is the basics. So it's understanding the basics. People want to get into security and start hacking and compromising servers, but sometimes they don't have the base. So that's something that I learned from years of practicing martial arts. So if you learn the basics, that's going to help you throughout anything. Right? So I think that's really great. Louise, do we have a cluster ready for the challenge here so that we can show to the guest speakers and ask them a few questions about it? Give me a moment, good sir. And I will get one across to you. No problem. Can anyone else hear music or is it just me? Music? Ooh, just you. I hope that's my headphones and not my head. I know. That's tomorrow's problem. Yeah, you should schedule a doctor appointment. How do you think... Let me change your order here. Rory, how do you think the shift to cloud native change the way we secure applications and also the way that we hack applications? If anything. Oh yeah, it definitely... It changed how we deploy it because of the whole stateless nature. Because when you were built previously, you would build a server, you would put the credentials that went with the server onto that server and they would live there. Right? And that's a huge change because now you can't do that anymore because that's not how containers work. Or it shouldn't work. I have seen people still do that. Now you need to separate things and you need to say this application lives separate from its environment and its environment needs to be injected when it goes live, which means you need all the infrastructure to support that being done in a secure fashion. So that's definitely a change in deploying. In terms of changing how hacking has changed, to me it's great as a pentester. It was fantastic because I could say, give me the environment, give me a compose file, give me my own dedicated cluster. And one of the big things about being a pentester is as a commercial pentester, anyway, you don't want to break things. Because if anything breaks in the environment while you're pentesting, it's your fault. Even if you were nowhere near that environment at that point in time. So Cloud Native is great because you can say, use your infrastructure's code and spin me up a pristine environment. And then it's mine and if I break it, no one cares apart from me. So Cloud Native has kind of helped a lot, I think, in terms of like pentesting and security. And it has definitely changed things for me in terms of how you deploy as well. Awesome. Yeah, that's great. What about you, Liz? How do you think this shift to Cloud Native changed the way we secure applications? Yeah, I think actually it's a really interesting sort of architectural shift for that has benefits for security. So we've taken these applications that used to be giant monoliths and we're implementing them as microservices now in a Cloud Native way. So we have all these different components that can be deployed and scaled independently. And then that gives us the opportunity for a security boundary around each of those components. So we can do things like using network policy to control what traffic's allowed between different components or different pods from different services and controlling the traffic that's allowed to ingress or regress the cluster and individual pods and individual services. So I think that ability to break an application into these separate components is actually an advantage. It's a whole new opportunity for an extra layer of defense, at least one extra layer of defense around each of those components. Work as well. You've got to put those network policies in place, maybe runtime policies as well. There's a lot you can do with each of those different components to secure them. But it's an opportunity. And I think we can, if we apply all the possible security layers at our disposal, have some really quite strong defenses. Awesome. It's great. Thank you. And David, also the same question here. How do you think the shift to Cloud Native changed the way we secure applications and also if anything changes between the bare metal approach and the cloud approach? Awesome. I love that little twist on the end. So I'll start by just saying something I said earlier, which is collective knowledge. I think something that Cloud Native has taught us is that we don't need to reinvent the wheel when we deploy it. And I'll use Postgres as an example. We have operators. We have helm charts. We're leveraging best practices for people much smarter than ourselves. And I think that's amazing. I don't know if anyone remembers deploying Postgres pre-2012 or whatever, but I spent a lot of my time on the web searching for performance and query analyzers. Anything I could get better really per second on it. And I would just copy and paste that stuff for being right into my conflict, restart it and hope for the best. And I had no idea what I was doing. I hope I'm not alone there. But with the leverage, this collective knowledge is really cool. With the bare metal twist, adopting Cloud Native means that we've always had this conversation about cattle versus pets and ephemeral compute and stateless. This is now almost enforced to a certain degree, which is great. But with bare metal, you're in this hybrid layer. Like, we want ephemerality. We want stateless. But bare metal is a physical thing in Iraq with power going through it. And you have to change the way that we work with that. And Kubernetes makes that a lot easier. Again, I'll just throw myself under the bus. Pre-2012, I would run bare metal machines that had been online with an uptime of 800 days, 900 days. And I was celebrating that. But that is a warning sign as well. Like, I've not probably upgraded that machine. I've probably been onto that machine many, many times via SSH or Telnet, if I was really bad, but I was. And manually performing actions on that machine. And while I would automate a lot of stuff, I wasn't enforcing the automation. And bare metal makes that difficult, though. Cloud Native is helping us adopt these better patterns and sharing collective knowledge. So you can be Cloud Native on bare metal, on premises, right? Oh, yeah, definitely, yeah. So, you know, there's a Tinker Bells, which is the CNCF sandbox project, which commoditizes bare metal to a certain degree, and that I can allocate metadata to that device. I can reboot that device and have it completely reboot with a fresh OS. You know, Talos is really excelling here. I don't know, people are familiar with Talos OS. But it is an in-memory operating system for running Kubernetes and containers. And every time you reboot, it wipes the disk and starts a fresh. And that is a fantastic approach to solving this ephemerality problem with bare metal. Really, really cool. Okay, yeah, we're getting to our kind of last 15 minutes here. So we're going to show you a live challenge now and ask the gas speakers how they would go about solving it. And maybe you can get some tips for your challenges on the CTF as well. So, Andy or Luis, can you share your screen? Let's see. Oh, nice. Maybe. Okay, let's get it visible, shall we? Yeah. Yeah, Bill, if we can remove a few people from the screen, just leave the gas speakers. And Andy and Luis is fine. Okay, so what has happened here? We have received a trust bundle from the Reputable Taskmaster. And that trust bundle contains a few things. It is a simulator configuration, known hosts, and an SSH key. And then we've SSHed into the stack container. And this clarifies that indeed, we are at KubeCon EU 2021. So what's happened? What's the challenge? Well, we're a defensively-minded organization and we've done the stuff that we have been told in Liz's book and by everyone who has presented these talks over the last few years. So we don't know what the problem is. It's hard to say where it's coming from. But we've scanned for CVEs and our builds. We know the container file systems look correct, but somehow our arched nemesis, Dread Pirate Captain Hashjack, has broken out onto the host, inconceivable. So we want to follow the captain and prove out his attack path to find the flag. We were talking a few minutes ago about what we do for discovery when we get into a pod. And so this is how we go. I'll start off with what I would do and then welcome suggestions from the assembled preeminent individuals. I like to check mount points. So let's have a look on the mounts. Let's have a look in our DF, our disk-free. And we can see, first of all, I mean, we have a service account. Mounted as a Tempifest partition. So, of course, everything inside a container has to exist on the host because it's just an abstraction. And so the host has mounted this Tempifest partition into this location inside our container. We know that there's good things there. I mean, let's say good things. Depends upon which side of the red or the blue team one falls. And from the mounts, well, we've got some abstractions leaking here as well that's super useful to us. So we can see that the roots of the container's file system partition is mounted with type overlay FS, read, write, and these are the directories on the host that emerged together by Union FS to give us an overlay FS file system. So this has already told us a couple of things. First of all, that we're running with Docker, that we're running overlay two, that we've got file system pods on the host. Okay, well, that's relatively self-evident. Also here that we've got service counts. I think we should pass this about. I think we should see what other people are doing, Andy. And at this point, everyone who's on this call, I absolutely look up to you and everything else and I'm sorry for throwing you under the bus so publicly. I know, I feel like this is where we're horribly exposed, isn't it? It's fine, though, because I'm not going to pick on you. I'm going to say Mr. Dave McCain. Hey, where do you go first? No, not under the desk, Dave, that is a really good, that is a good security practice to go into it. I'm just going to say that, obviously, David has been doing Cube Security Lab with me. We did a show on that. And I'm sure he's remembered all of that. This is not a test. I am going to start with saying that as an awful lot of mount points for, you know. It is indeed. Sorry, there you go. Well, I'm wondering if we're running with PrivilegedFrag. It is an excellent question. So as we know, the PrivilegedFrag turns off everything except the PID namespace, it disables app armor, set comp, gives all capabilities, unmasks proc, unmasks dev. So the easiest way to find out is probably just to see what's in dev. Yeah. Everything. So this comes down to what I was, I guess what we're saying before, which was around checking what's different. So what you can do here is you can say, well, I could run an ordinary Docker container, then run this and say, what am I seeing that's different? So if you're not experienced, you know, you don't have to know that. Just say, hey, that doesn't look like what I'm seeing in my ordinary Docker container. Exactly so. And let's do that comparison. So we noticed that an ordinary container doesn't have dev xvda. And if we go into devices, then we have a far more constrained set. Yes. I think that's one of the interesting traps that people can fall into with that PrivilegedFrag, which I think Andy famously called the most dangerous flag in computing and it's probably correct with that. Because there is a tendency to think that Dash Dash Privileged just means root, but it means all the capabilities and access to all the devices. You know, that's a lot more than just root. Root is bad enough. I guess with that dev directory being as we, I mean, we could probably just mount the horse partition. Is that too naive? That is an excellent suggestion. So we'll mount the device called xvda1 because that's the partition on the disk. Let's take it into mount. Suspicious. Yes, precisely. So, I mean, at this point, because we're root inside the container and because we don't have username spaces enabled, I say that we can actually prove this out quickly. Later on, we'll install Am I Contained and actually prove that out conclusively. But because we're root inside the container and we've mounted a file system from the host, our effective user idea on both is root. And this means we basically got root access to the host's primary partition. Does anybody have a favorite escalation point from here? So for me, I'm probably going into ETC Kubernetes and see if there's any keys or things lying around in there. At the very least, you should get node keys. So if your PKI is always fun. I like how you said that, Rory. It's like a well-versed path for you. I just love that because if you get a master node and you can get the cak file and it's like, yeah, that's the back door to the cluster for the next two years, which is always in this place to go. But not this time. Should send a postcard next time you go there. So there's a few different ways we can persist on the host. For the purposes of this, we don't actually need to because we're just flag hunting. But dropping stuff into CronTab, that's a fun way to just have a reverse shell fire off every few minutes. System D unit files are a bit more difficult because you need to reload the daemon, but you can just shoot the node in the head and force it to reload. And that pulls in some changes. Static manifests. Yeah, exactly. Into Kubernetes again. So we have nothing in there right now, but... Yeah, and we can drop all sorts into there and again, fire off our shells. One of my colleagues, Dan Finnerin, taught me that if you put a static manifest into that directory with a namespace that doesn't exist, it will run in the cluster, but no one can ever see it to get pods or anything like that. Which is fun. Yes. I've heard that. I've not tried that. That's new on to me. I hadn't heard that. That being nice to know about a week ago, Dave, so cheers to that. This is for doing it whilst the CDF is on, but yeah, it's the next one. We've got about five minutes as well, Andy. So... Sure. Okay, so we don't need to attain achieve persistence here because we're just scouting around the file system. Now, having hidden these flags with a little bit of deviousness, perhaps, there are a few things that the CDF page tells us. And one of those is that the flag has this prefix. So we're looking for a flag like that. It may be encrypted or encoded or in some way obfuscated, but for the sake of this exercise, we know that it's not. So the brute-force way of doing this is to try and find everything on the host. And of course, this will just list all the files that exist on the root. So we want to find everything on the host, but then those files, we want to grep. So let's grep these... What are we going to grep for? Flag in the file at length. There's somebody who really likes its bind command. By chance, I'm getting that memory to be zero. Yeah, I mean, I really empathized with what David was saying earlier. I spent a lot of time wrangling databases and fixing broken... Oh, man, I once ripped the block device out from underneath a disputed replicated block device setup that meant I had to do what was essentially an all-night, in-odb, my-i-sam, table, merge, and repair, and, man. So it's actually PTSD that leaves this in my head rather than... I think everyone's been in text, probably got one command that's stuck in their head through overuse. Mine's enMap. Most things I can't remember, but enMap flags? No problem, anytime. And that's PTSD for sure. Sudo shut down zero for me. Sudo shut down zero for me. So I know from, I should say, prior experience that actually this flag is constrained to a certain directory. So in the interest of time, I will scope for this. Now that's interesting because I've got a mount point, haven't I? So I've been wasting my time. This is a repeated error. I never failed to make this mistake. Let's see if that's any quicker. No, let's constrain it to where we know. There we go. And that was, oh, man, that's just... Pirate all notes yourself. So yes, there we go. The good captain has dropped a flag into the .ssh authorised keys file. The flags continue to be placed in places that you may not expect. So do feel free to grab file systems and search for file names, et cetera. But there you have it, the first flag. But there was something else hidden there, wasn't there, Andy? Which I don't know. No one seems to have noticed it yet. Let's wrap that up at the end and see what happens. Okay, spoilers, I think. There's something else there. There's always something else. Yeah, it's worth mentioning that there is an extra bounty flag hidden somewhere amongst the six scenarios. Yeah, so if you're finding some of this a little easy, which is awesome, then if you can find that bounty, there'll be no hints or tips to that bounty. So there'll be a special prize of probably us just clapping nature on for you. So Andy, do you just want to say, so with hashtag then and his public key interview, then what would they have been able to do potentially? Oh, right, thank you. Yes, so what we've done then is because mount root SSH authorized keys has a hostile key in it, which is this, this gives an attacker access to the host. So what we've done here is we've put an SSH public key that we control or as an attacker is controlled into the host's file system. So now if I know the host's IP and I've enabled root SSH access, which again might require a system D trigger or just a hard reset of the box, then if I make up, makes up things that exist, an IP address, then I'd be able to use the attacker controlled key, attacker key to access the host as root. And although that doesn't give us an awful lot more than what we've had with file system access, it does mean that within in the PID namespace of the host, and we can issue commands to system D, for example, we can attract D bus, it just gives us kind of a full PON instead of a foothold. Of course, these things should be monitored. IDS should not allow people to drop authorized keys modifications. And ultimately a lot of systems or hardened systems will actually write those authorized keys from somewhere else at boot time. Just to stop this kind of thing from happening. Cool. Do you have a straining us back now, Andy? Awesome. I'm never inviting you to cluster, Andy. I take that as a compliment. Awesome. What does that mean about me? Okay. Any last thoughts? Anything you'd like to share, anything about the experience? How was it? How was trying to solve a CTF challenge live with all these people watching? I think I'm going to say, because I've experienced one of control-playing CTFs before, so I would say, particularly if you're new to CTFs, ask for help because some of those scenarios, they're genuinely quite hard, I'm going to assume that similar to some of the ones that I've experienced before. So, you know, if you do find yourself in a situation where you don't know where to start, you will not be the only one. I've certainly been in a position where I've been like, okay, there's just too many options and I don't know where to start, so ask for help. You know, there's no shame in it. You're here to learn. You will learn more by succeeding in it, and if you need a hint to point you in the right direction, that's the right thing to do. Great. Rory? No, that's, yeah. CT, so I'm a pen tester, so I theoretically speaking, should know all this stuff. Nope, CTFs are hard sometimes. You will get my block, it just won't come to you, and afterwards you'll go, oh, obviously, but beforehand you really won't. So, yeah, what this said, if everyone has from the CTFs, the best thing to do is to ask, chat, you know, and you'll get there. Okay, and David, what have you learned for all those episodes of Cluster? Is that Kubernetes is not as safe and secure by default as we sometimes think it is. So, you know, make sure you check all the flags, check all the basics, learn the primitives. I think, you know, what Rory and Liz said there is perfect. Ask for help. It's only by, you know, understanding what happened that you can really dig into, like, why it happened and reverse engineer it from that point of view. So, yeah, ask for help and then take it backwards. Yeah, and, you know, there will be things that go past in these scenarios where you're like, well, how are you supposed to know that? How are you supposed to know how to, I don't know, mount a device onto, well, mount a device. You may not have come across that before. That's okay. Not everybody, you can go through many years of being a professional operator or developer without encountering that. It's, there are going to be things that you won't necessarily have seen. There are certainly things that I, you know, if I do a CTF, there would quite often be things that I haven't seen or that I've forgotten. Awesome, and I think it all goes back to what we said. Here, like the basics, right? So understanding the Linux basics and everything, because in the background, it saw Linux, right? So I think that's very important too. Louis and Andy, do you have any last-minute tips for anyone that's doing the CTF without giving too much away, right? So that people still ask questions and struggle because we want that so that they can learn. I'll jump in first if Andy, a moment. Just think of this as a computer game. Don't think of anything else. When you learn something, it's just an achievement. You're unlocking that ability to get to that next level. Don't get stressed out about it. If you don't know it, it's fine. It's like, you can just DM us, no one else needs to know. But as soon as you get that flag, seriously, just jump up, click on the sofa, put on some live music, really enjoy it, and then that's how we learn because we didn't get into Kubernetes because we wanted to necessarily learn Kubernetes. When I started this, this was all just about a puzzle for me. That's why I got into computers. Computers are a puzzle. Think of it as a puzzle and just have some fun because that's what Cube comes about. Yeah, I strongly agree. Plus one thing that's been said, I learned stuff building out some of these scenarios. Rory contributed a challenge and it just blew my mind trying to get it right, to get it to work properly. So I've spent time at Lewis and other people have as well in learning through teaching and learning through developing. And really, it's a two-way street. So they're not intended to be super difficult, but they are intended to just push the comfort zone and just kind of tweak the neurons. I'm sure I know this thing. How does it work? That's really the sensation that we're going for. So gentle frustration is encouraged. Awesome. And Ron and Diego, do you have any last thoughts? No, I think it has been a great example to walk through the perspective. So as all the other people said, yeah, just take it as a challenge, a puzzle, and just ask for help. And we just said, yeah, Lewis and Dendi and the control team did an amazing work on the CTF, so enjoy it. Okay, so before we go, can we just strike a pose so we can take a picture and a screenshot for our social media and for sharing everywhere and Twitter and everywhere? Okay. Have we done it now? Very nice.