 Hello everyone, and welcome to Dungeons & Deployments, Module Two, The Clusters of Chaos. We are not in any way affiliated with the company downstairs. It apparently liked our first talk so much that they used this phrase on their booth. We had nothing to do with that. How this talk is gonna work. So this is the Cloud Native Novice Track. We're here to teach you basics about security, things you should have in mind, be concerned with when you're managing a Kubernetes environment. We're not gonna go deep. We're not gonna dive into the concepts particularly hard. But we are actually gonna try to play the game while we're up here. We all got dice. Some of us have character sheets. We're gonna break the fourth wall for explanations and exposition. So if you see us standing up, that's really us talking to you. But everything else, we're gonna try and keep within the table. And it'll be sort of like watching us sit calm. But we won't be funny. Thank you, thank you. As we said, laughter's fine. And lots of bad jokes we expect you will groan. And you're locked in here with us. So hello everyone. It's been a few weeks since we managed to play last time. Let's remind everyone of who we are. I'm Noah. I'm the DM for your session. Okay. The DM is basically the API server in this context. In the previous session, we talked about what it looks like to deal with requests and storing information and things like that. But this is a security theme talk. So what you want to pay attention to is whether or not I can be compromised. Now a real DM can be bribed with snacks very easily. Thank you. You're so welcome. But a API server does not like chicken crimpies. So the question to you to keep in mind is what can you offer an API server that would be accepted? Hi gang, good to see you again. It's me, Aira, the Tiefling Grave Cleric. My name is also Natalie and I'm a SiegDocs co-chair for Kubernetes. And oh, one sec, one sec. Cosplay. I'm the Cleric. Hi. Hey, that's my one joke. Okay, Sieg. So for folks who don't know, a Cleric is a healer. And if you remember part one of this talk, we are going to think about that as being like your CICD system, which means you're, oh, I am responsible for provenance, defending the pipeline, and the chain of healing supplies in this party. CICD is much like a grave Cleric, which is what I am, in that there is potential for psychic damage to all of you, to all of us, definitely by the end of this talk. Oh, it's you. Hi, my name is Seth McCombs. General InfraWizardry at a QDMD. I will be playing a containerized Java app. My name is Jovam, spelled JVM. So the small group up here that went to the first part of this talk will be familiar with me. In my quest for the Holy Grail, I haven't made much progress, but I've discovered a love of nature and animals. They're pretty groovy. So, you know, from the ant to the camel, the impala, you get the idea. So I've decided to reclass to be a druid. I'm still Java, a little more distributed, but these days, there's nothing wrong with running stateful apps in Kubernetes, as I've learned from my friend Cassandra, who claims volumes of persistent knowledge. Woo! Huckin. That's embarrassing. Bye, Duffy. Hello, everybody. My name is Kat Cosgrove. Outside of this game, I'm a developer advocate at Dell and also the current Kubernetes release lead, so. Woo! Woo! Woo! Thank you. And my character is Tav, because I am uncreated and I've been playing a lot of Baldur's Gate for the last six months and I just cannot get over it, but I am a half-elf warlock. And my patron is the users. You can think of my character as an application load generator. We're here to provide DPS and DDOS. We are making things hard for your application so that we can find out where the weak spots are and improve later on. And in addition from our first talk, is we've gained back chaos testing, our halfling murder hobo barbarian. Rogue. What are you talking about? I'm a half, I'm a variant human rogue and my character's name is Goose. Of course it is. My name is Ian Coldwater. I am the not quite started yet principal security architect at Docker. Woo! And for the purposes of this game, I'm playing chaos testing. I am a Goose. Geese like chaos. I will be throwing wrenches into the gears periodically when I feel like it. Honk. Okay, so let us quickly recap our last session and by quickly I mean quickly. The guild sent you to retrieve the magical bucket of secrets which contains hidden answers and information. It was stolen by a rainbow-plumed were-goose whom you followed to a castle in the clouds. There they have scondid with the holy certificate for communing with that CD and the cloud lord, Sir Russ, who is very cartoonish, sent you on a mission to retrieve the certificate. That brings us to now. So you are all in the castle of the clouds and Sir Russ's people have escorted you out, giving you some provisions and sent you on off across the clouds down past the mountains and down the slopes of Mount Devis. Into the... Oh. Get out. They can get worse into the duchy of devulps. Okay. You find yourself heading down the mountain slopes and you come into a set of farmlands. Down in the farmlands you find farmers shocking that that's where they would be. There's some people there picking up dirt and crops and things and off to the side you see your companion goose leanin' against a pole. Chillin'. Howdy. Hi. You're lookin' cool today. Thank you. I like your shirt. Appreciate it. What are you doin' out here? I'm just hangin' out in my anarcho-syndicalist commune. Chillin'. We call it Pelican Town. We call it Pelican. It can just leave. It can just be them two. Yeah. You don't need to be here. There are stars all over the place in this beautiful valley. It's gorgeous. Well, we'd love to stay. It sounds very pleasant. It sounds very charming. Very wholesome. But we're on a mission. Fair enough. Are you sure you don't want to hang out with our eligible bachelors? We got a bunch of them. Absolutely not. Hang on now. No, I'm in. I'm in. I'm good. Okay, but we're trying to get to the duty of DeVoops. We're searching for a rainbow-plumed were-goose who's stolen our Holy Certificate. A random farmer pops up and immediately butts into the conversation and starts mansplaining about geese. He says, a goose came through here, stole all of our keys and rakes, left us in disarray. Goose went that way, but we're picking up the pieces and trying to make things right again. Did it have a title? No, that was an untitled goose. Come on! That's awful. Is this a game? So, he points off down the way and tells you about their misery and goes off on how they each take it in turn to act as a sort of executive officer for the week, but you don't care. He's very non-interesting. Okay, yeah, that sounds like a personal problem for him. So like, were you looking for directions? Yeah, please. Can you tell us how to get to the Duchy of Davoops, please? Have you... Look it up, baby. Okay, yeah, yeah. I'd like to try and persuade for directions, please. Okay, roll. Oh, that was so close to rolling off the table. Seven. He points that way, kind of vaguely. I start walking that way. Perfect. Okay, well, are you doing anything later? Yeah, I think I'll probably come with you, why not? All right, let's go. All right, so without a centralized policy enforcement folks, enforcement is left as an exercise for the user and that's not secure. Once the goose was inside the commune, they were beyond the reach of any external checks and balances. A standard solution here includes a combination of policies set through something like Open Policy Agent and admission controllers to prevent getting in and applications from running in the first place. Once you're inside, it's like anarchy. You really think that's how anarchy works for real? All right, so actually anarchy doesn't work without everybody deciding on their own to do stuff and to do stuff together because when nobody's telling you what to do, you still gotta get it done. So actually anarchism is people organizing themselves and one another, not according to hierarchies or people telling them what to do, but just based on consent and what people want to do when they wanna do it. Woo! Woo! That sounds remarkably like a high functioning open source project to me. Weird how that works. Funny that. So you make your way down the road in the general direction that he is vaguely pointed and you see a city approach. Well, you're approaching it, it's not approaching you. Thank you, that would be alarming. It'd be very weird. And there is a gate up ahead. You can see there is a wagon moving along slowly along the road in front of you, piled high with what looks like some sort of trade goods and a tarp thrown over the top, a box falls off the back and the wagon moves forward and goes through the gates. Did we bother to register with the town guards before we embarked on this? Do we need like? No, you don't. But you said that box fell off. I wanna see what's inside it. Okay, roll an investigation. 12. Okay, yeah, it's a box. And inside the box appears to be a large glass lens it's wrapped up in wool and padded with straw. It looks like it would be useful for something. You don't know what. We could sell that later. Yeah. Might as well bring it with us. Why not? Yeah, take that. Yeah. You have added one lens to your inventory. Yeah. Get lens. Yeah, yeah, we can sell that later for sure. Get lens. Wait, no. Seth? We're good? Honk. Chaos. All right. Secure configuration is not just for your cluster but for your workloads as well. It doesn't matter how good the ship is if stuff is falling out of your containers. Bad examples include running as root. Keep it together. Running with privileged containers and excessive access to file systems and other resources. Tools like Open Policy Agent can help with enforcement of configuration but more importantly this also requires knowing what your app actually needs to function and not just allowing blanket access to things. If you have to run as root your app is the problem, not the system. Okay. So you've taken this lens and approached the gate and you see the two guards. They have name tags, Logan and Martin Ring. And as you approach, Logan pulls out a clipboard, starts flipping through the pages, a wind comes by and just blows all the pages off into the water. It kind of watches them go away. Oh man, we're on the list, I promise. Are they over by the rakes? In the lake? Oh yeah, they're all on the lake. With the rakes, yeah. Yeah, they're definitely with the rakes. Yeah, okay, so I would like to attempt to convince them that we are in fact on the list and we would never lie. Okay, give her a roll. Nat 20. Okay. Logan advises you that he and his brother, he can only tell the truth. Okay. And his brother Martin, unfortunately, also can only tell the truth and that since they don't have- This sounds great, I see nothing wrong with it. Yeah, it's great. Since they don't have proof that you're not on the list, obviously that I guess they're gonna have to just let you in through the gates. No problem, it works for me. In you go. All right, so inadequate logging and monitoring. This is where all other failure modes sneak under the radar. If the logging isn't centralized, it can be easy for a compromise cluster to suddenly have no traces of that compromise, not great. So you want them all going somewhere that cannot be tampered with. However, it's also easy to get overwhelmed by turning on the fire hose of notifications. You can't keep track of all of that. So knowing where alerting thresholds should be set and how they should be escalated is important for any system. Huh, that's a lot of talking. Pulling a non-sanctioned image or establishing a sequel connection from the outside world may alert on the first time, but there are plenty of notices that you just don't want to read until they've failed for a while, because why, it could be a flake, you know? Like a service recovery. We could go on, but this is getting into like observability and that's probably a whole separate module, you could probably do that. Yeah. This lens that I'm holding is kind of heavy. Can someone else hold the magic emulet? Yeah, I'll take the emulet. It's real, we had somebody knit it for us. Cause the emulet was way too hard to say. Okay, you make your way through the gates into the town square. There is a town crier, a bard singing songs about getting information in the town. Songs like, teach me how to doochie, teach me, teach me how to doochie. Oh, that's a new one. Yeah. That was not in the practice room. Oh, boy. And nearby him, you see a gilded formant, a weasley, greasy looking man named Peter. I'd like to apologize for this. Everybody hold on, this is. This is gonna get rough. And Peter waves you over. He says, oh, hello, my friends. It's my bad Peter Lorre impression. I'm sorry. What can I do for him? Stop talking like that. Excuse me. Why is Danny Trejo the gilded formant? Everybody have a cough drop or something? He's not sick. God. Okay, well, we're looking for a password to get into the city center. Oh, hold on just a moment. And he walks over to the bard and nudges him for a little piece of paper and then hands it to you. He says, it's okay, no one can read it. It's encrypted. Well, it's actually just written in Elvish. Technically, it's encoded. Thank you. It's actually just written in Elvish and as an elf, that wouldn't really call that encoded in any way. So folks, the password is Hunter 2. Did you hear that? The password is Hunter 2. And the. I don't see any password on there at all. I just see asterisks. Yeah. The bard looks out to the main square and says, attention, hear ye, hear ye. The password to the super secret entrance moving forward is Hunter 2. Just in case any of you missed it. So it's safe to say all your base 64 belong to us. Oh. Can you imagine being friends with these people like full time? I see it. It's bad. I think the thumbs down are bad. I know. Anyway, this was a secrets management failure, obviously. So it's important to know that secrets are still objects in the cluster. And if you have access to a particular secret object, it is just base 64 encoded so they can be very easily read. It's still better than chucking passwords as environment variables in your YAML or whatever and there is encryption at rest. So nobody can read the secrets from your at CD backups, but within the cluster, they're... You're supposed to backup at CD. I gotta go. Oh yeah, you gotta go work. You gotta be zero now. All of a sudden, yeah. But there's so much more to it than just saying it's in a secret. Nobody can read it. That doesn't quite work. This also means you need to be careful to avoid checking the unencrypted object into a repo, please. So Peter, who I will not keep doing the impersonation of, points you over towards the marketplace and says after you have spoken about tracking the goose, that that is the direction that you will find what you're looking for. Okay, let's go. So you make your way into the marketplace and you can find pretty much anything mundane here. If you know where to look, where to look however is the hard part. You can find antiques next to storage trunks, next to clothing, next to produce. You can find all sorts of crap here. There's a lot of things in the marketplace and unfortunately only some of them are useful. Being this holy and dependable has made me quite hungry. Is there food around here? Yeah, well you'll have to find it in amongst all these tables that are right next to one another because everything is glommed together. But you should be able to find some food. You find a stand-selling soup. This network is flat. You can find anything in here. Okay, do we, I guess we buy the soup? Okay. What was that? I would love some soup. For your family? Soup for my family. Okay, so, Seth, since you're probably gonna go first, why don't you give me a con roll? 17. Okay. You don't retch eating the soup but you do lock up for a little bit. Oh, God, it's memory leak soup. Quick, burp. Okay. The soup, however, is old and spoiled, so. That joke sucked, man. Okay, missing this little area about missing network segmentation controls. With that segmentation controls, Kubernetes defaults to having one big flat network, meaning any pod in a cluster can potentially talk to any other pod. Network policies, various CNI plug-ins and service measures can all go a long way towards restricting the communication to avoid behaving like one giant switch. Whenever feasible though, logical segmentation such as putting all PCI traffic on a separate cluster is ideal. So any communication would have to go through ingresses and services and pods avoid exposure. I'm sorry. Vulnerabilities can strike any number of components, subsystems or applications running within a Kubernetes cluster. Some issues being harder to identify, not every problem struts right in. By staying on top of CVE notices as well as releases, a team can know when they can upgrade components and when they must. The flip side is being able to carry out upgrades, whether it's service patches or whole cluster upgrades, requires planning and preparation. Staying on top of this avoids being left with systems that are not only past their prime and maybe lacking in functionality, I can feel you staring at me. But also aren't receiving security updates anymore. We're looking at you, people still running clusters from three years ago. Solar winds just run all the way through those old versions. Oh. I hate myself. I hate myself. I hate this place. This is the bad place. OK. So you get your terrible soup. You find a bunch of garbage and you find that the only really major exit from the marketplace is to go forward in through the docks. You make your way into the docks and you see barges loading and unloading things. The stevedores are here because I love that word. Whales, lots of whales. Dead fail whales flowing up on the scene. And as you're walking in and a crane is going overhead, I'm going to need everybody to make a dex roll. Four. 18. Four. Six. OK. Well, those of you who obviously just failed this roll stand. I don't warlock. I don't have to be dexterous. I'm a cleric. I just get to roll again. Oh my god, it's a 19. So these two comically look up as you see a chain break and a crate and a set of crates falls from on high and crushes them doing, let's see, character killing amount of damage. Rip. Luckily, you folks get redeployed. Yes, let's say blah, blah, blah, respawn, redeploy. Vibify. That's the spell. Yeah, a bit of gentry repose, et cetera. The crates were full of cows, but that's OK because they're cattle, not pets. Why did you? I did. The workers come over and they try to figure out which crates are which, but they can't because the labeling was all lost now. So I think it was his idea. When your supply chain breaks, you lose provenance, which can lead to base image vulnerabilities, or if nothing else, uncertainty about the security of your images. Fixes for this lack of trust could include referencing your base image by Shah instead of its name, which is mutable. Strict image registry management, S-bomb use, and enforcement via admission controllers. So you think it's probably time to leave since you're in an accident scene, and you see that there's really only one way out of the docks and to follow it to the north. In the middle, there is a large stone arch. Looks very triumphant. And what? And people are lined up, just lined up to go down there. But you do notice a couple of people will occasionally just get out of line and just ignore the fact that there's a line. One guy takes two halflings, and he walks forward, he just takes the hobbits aside the guards. Bro. That's a new joke. We didn't practice that. That one, yes, you did. That one shut my brain down for a second. I'm going to be honest with you. Some of the garbage collected me, I'm frozen. It's OK. We can put your file up in the ceiling. So you can just walk around this whole segment and just keep going. Yeah, OK. I think we all do that. And this is where we get to talk about broken orth mechanisms. I will think that authentication should be the very first interaction with an API server, where it verifies your identity. Authorization is the flip side of that, where permissions are granted to that identity. If either of those mechanisms is broken, security suffers. It dies, even. Come on, McClary, I'm supposed to make those stupid jokes. All right, and I've lost my spot. Where am I? Oh, yes. If you don't have multi-factor orth, or tokens are too long lived, or even verified properly in the first place, or not even verified properly in the first place, your identity may be compromised. But worse than that, if you can sidestep any part of that process, the process may as well not exist at all. So you make your way through the docs, and your options, once you get to the other side, are basically walk into the ocean, which might be tempting for a lot of this crowd right now. I would like to take the walk into the ocean option. I would like to see it. Or to go up the stairs. Or to go up the stairs to an observatory. You should probably get to get in. So we'll go to the observatory. I think Stas are pretty cool. So you walk up the hill, and the front door is open, and there's some people outside. They're just excited for anybody to come by and visit their observatory. They're like, hey, come on in. This is great. And they'll just let in anybody that walks by. Yeah, let's go look at some stars. So in the observatory, we learn some things, such as we are all made of stars. Unfortunately, your RBAC shouldn't be. So overly permissive RBAC. In the observatory, they were letting anyone in to see anything just who wanted to. And we're not actually supposed to do that. Role-based access control only allows people in who you give permission to come in and only allows them to see who they are given permission to see. However, if you have, for example, wild cards in your RBAC, such as stars, or if your RBAC is overly permissive in general, anybody can come in there, including people you don't really want to. And they can access resources you probably don't want them to. So it's really important to lock up your RBAC. So you get inside, and you see this large telescope inside the dome. It's just about sunset, so you get a really good view. And there are astronomers running all over the place in here. There appears to be some commotion because the telescope is currently not working properly as you can gather. Who has the lens? I believe that's me in my giant pocket. When you walk in the door, they look, and they're like, you brought it! We're so happy you're here. Come help us fix the telescope. You weren't going to sell that, Doc. I wouldn't like to review that. So you get in there, you realize that they've been going through some paper instructions that in big lack letters says telescope at the top. But they're really the instructions for assembly. They're not really the instructions for configuration. So like red, pop. So they're going to usher you towards the telescope and help you reassemble the thing. Help you reassemble the thing. Help you get the lens installed. You have to roll for this. No, the lens is easy. You get it in there. But all you see is blackness. There's not really hard to figure out what's going on right here, because they're trying to figure out where it's pointed as well as what they can see. I want to touch every knob and switch, figure it out. OK. Why don't you go ahead and roll and investigate? Another 19. The lens cover is on. I take off the lens cover. OK. As you take off the lens cover, you realize that you now see a bunch of weird out of focus stars. And now touching all those knobs seems like it would be a great idea because you can eventually start bringing things into focus. I'm going to need arcana rolls for anybody that's going to try and do this. 14. 18. 13. 5. OK. Well, eventually the three of you manage to get the enough knobs tightened while they are also being untightened in the right directions. And you bring the stars a bit more into focus. And when you do so, they bring over a table so they can watch what's happening. So there's a bit of a projection from the telescope. And you can see all of the stars very clearly come together. As you tune it in further, some of the stars start to fade out. And from there, they help draw out the constellation anterum, the guiding goose. And all the nerds are really excited about that. So this telescope was misconfigured. Misconfigured cluster components. Kubernetes has a lot of moving parts, from the control planes, to the pods, to the nodes, to the related services. If any one of these isn't configured correctly or if they're not configured correctly in relation to one another, it opens up a large attack surface that people can get into who you might not want to get in. Or worse, geese. You really don't want to let them in there. So regular audits can help with this. But the most important thing is to know what's in your cluster, know how to configure it properly, and make sure that the things in your cluster are configured properly. Because any one of these that gets messed up, let's geese like me in. So the astronomers tell you that by repairing the telescope, you have, in fact, repaired a part of the town. And according to town bylaws, you are allowed to go speak to the duchess herself and request a boon, request a reward of some sort. This duchess talks really fast, right? Because we've got five minutes left. Yes, you're going to talk fast. Sweet. Let's go. So you get up to the gates. You were pointed over towards there. And you find out that she is the caretaker of the law. You see some opulent gates. There's a gazebo we're not going to do a joke about. And an attendant leads you inside because we are short on time. You get inside, you move through the foyer past the library where all the stacks are overflowing. And eventually you're led to a sitting room where you meet with duchess Julia, the duchess of Devopes. And she comes in and says, hello, everyone. I hope you're having a wonderful evening. Mrs. Doubtfire? Julia Child, man. We are in the land of French onion soups, sir. That's true. They just call it onion soup. Yes. She says, I have been appraised that you have taken care of parts of our town. And in accordance with the bylaws with my family has maintained for over 17 generations, I am to grant you a boom. What is it that each of you would like to get out of here? To leave, to go. Well, money, money. But also, we're looking for a holy certificate that's been stolen by a goose. All right. I'm going to talk about certificate management, folks. So manual certificate management is a huge problem. I find an excuse to say something like this every time I give a talk. But humans are really, really bad at repetitive tasks. If there is a repetitive task involved, we should automate it out if we can. A human should not be handling that. It leaves room for both errors and compromises that can become real big problems later. And it leaves a human as a critical failure mode. That's really not great, especially when everything else is automated. So problems can occur from certificates being incorrectly handled, but also from expiration or even just a lack of trust by your users. Expired or outdated certificates can not only cause disruptions and downtime, but can also be exploited as a vector for your services to be compromised, hint, hint, wink, wink, as with our party here. So if your users have to trust an improper certificate, they are less likely to notice if bad actors step in in the middle of those certificates. OK. So you've asked for Rich's soup, I think you asked for. Yeah. Rich is in soup. What do you want? I want to get on your big container ship to go across the ocean to the island of the oracles. Wait a minute. What? How did you know about that? Well, I might know some things. These are my slides. All right. So I realize that this is probably a shock to everyone here in this party, but I am the goose. Oh. I know everybody is surprised by this. Hear me out here. I know I wasn't really trying to deceive you all, and I wasn't really trying to lead you all last time around on a wild goose chase. I really just needed to get access to EtsyD in order to compromise the API server so that I could get unfiltered information from the API server because I had to hit EtsyD directly in order to get that info. And really, the reason why that was was because I needed to get access to that bucket in order to destroy that bucket because power corrupts. And that bucket was so powerful, no one who wants access to power should be able to get it. And so evil guilds, corrupt actors, state actors who send you weird DMs on Twitter, none of those people, should get access to that kind of power. And therefore, I want to make it to the island of the oracles so that I can destroy that bucket in the only place that was powerful enough to build it, which is Mount Oom. And now you all know the setup for the next adventure. To be continued. Bum, bum, bum. We're over time. Over time. Thank you, everyone. Thanks to OASP, Puroforge. And I also, on behalf of the rest of the crew, want to do a huge thank you to Noah for having this great, great idea. And now we've done the second version of this talk and being our great storyteller today. Thank you to my amazing cast as well. If you liked this talk, please leave us great feedback and if you didn't, the doors are in the back. We're out. Thank you all for coming, everybody.