 I don't hear you. I wonder if I have too many things going on or you're also maybe just muted. Actually, yes, it says you're muted. Okay, great. Was I muted the whole time? You were muted the whole time. Okay, well, I was just saying that I'm not going on YouTube live with this ugly background, but like, I guess it's kind of acceptable now, but. Oh, that's really funny. Yeah, no, no worries, no worries. I think it's just fine. Okay, because like, there's this towel on the door because my dad is like still renovating this basement. Okay, I have a notebook in case I need to take notes. I have the list on my other device so that my people start coming in. I can check their boxes off or whatever. I can check their boxes off as I let them in. Amazing. Anything else? Anything else is I'm going to go and like review the document just to make sure I don't forget anything. I will wait here until Magno joins and then I will hand over hosting controls. Okay. I will play room code of conduct. Oh, yeah, I think I read that at some point. Well, if anyone's being. If anyone is being, I feel like you've got it. If anyone is being a jerk, like, you know, warn them and otherwise, like, you'll have this, you'll have like, next to their names, the ability to kick them out. Has that's going with you? It's going all right. It's stressful. Yeah. But, and I'm tired, but it's okay. Do you like take like a week off or two weeks off to do this? Oh my God, no, I took Wednesday off and then I took basically three days off and then Monday's a holiday. So I get to recover a little bit when I, when this is in person, then it, I actually take longer off because it's just so physically exhausting, but this time, like I'll just be like tired and that's it. But like, honestly, usually after I like inverse and more sec, I get sick because it's also like you're, you're exhausted. Everything is like crazy. And then, and then also you meet a ton of people and stand in tightly packed crowds and, you know, have food and drinks that are shared for three days, actually no more like five days with people who have traveled from all over the world. So it's just like, you're going to get it. That's fine. Yeah. I'm just honestly, I haven't listened to any of the talks or anything like that. I've just been like doing my work and like, like I'm so tired of like online talks. I know everyone is, it's honestly, I'm shocked we have such good participation because we're like, okay, we know this is going to be hard because it's like, you know, like there was a lot of enthusiasm last year because it was like right at the beginning and it's like, oh yeah, like it's not entirely canceled. You're going to do something. And now it's like looking at a screen why, especially when it's nice out. Anyway, yeah, but we got to keep it alive. So yeah, we'll get there. Is there anything I need to do for the YouTube thing? No, the YouTube just handles itself. And when the Magnus starts presenting, no one else's face will show. Like it's just going to show the presenter. Oh, okay. Oh, so I didn't even have to worry about this. No. Yeah, you don't have to worry about it at all. Okay. Cause I just remember there was something about, we encourage you to have your camera on because that encourages people to participate or something. I think it's something like that. Anyway, fine. Oh, is my name, my name? No, your name is definitely except John. Okay. Okay. I need to fix that then, cause that's weird. Where is it? Damn it. Participants. It's weird that, oh, but if I'm host, I can probably rename other people to, I think except that we could do that. Maybe if they have like a rude name or something. I just think it's weird that like to change your own name, you have to go on the participants list. Kind of weird, but anyway. Yeah. It's just, I don't know, we did a, we did, I think it was actually a mantra hack at the very beginning of COVID where we were a little too liberal with the Zoom permissions and we got Zoom bombed. And so now it's like, okay, no, we've got to be careful about it. Have you? No, it's not something to do with you guys as settings. It's just like Zoom's, Zoom's UI. Anyway, I didn't know you would do your mantra because it was like, I've been participating in those for a while, well, semi-participating in those for a little while and I'd never seen you or I just had no time. I've been a few times like a while ago, like a while, while, while ago. Like when it was in person? Like, yes. Okay. Like probably period days. But yeah, I should actually do them. I should. I kind of stopped doing them because I figured that's not even worth it because I need to go and like read the stuff. 101 before I jump into the, I don't know, the CTF challenge that no one solved. Yeah, it's like, it's often extremely hard. This year in the CTF, actually, there's an easy track. Oh yeah. Which is really cool. Is there any, is it this weekend or next weekend? It starts tonight. Oh, it starts tonight. Yeah, no, I'm not going to do it. I looked at some of the other workshop schedules and all of them were like fully booked. I said, well, whatever. Yeah. Totally. Yeah, I got this like humble bundle that had like a bunch of, it was like a bunch of security humble, and it had like the pentesting book that you let me that I still have. Cause I never, I got like halfway through and then I stopped. I think though that the thing is, it's really, for this kind of stuff, it really helps with its hands on and practical. Yeah. I don't know. Like the workshop that I did last year, because I didn't end up practicing it afterwards, it's kind of like gone. Like I still have like the list. Like I probably, I have to have it somewhere, like the notes that were given out, but like, and I think, anyway. Like I think it's like, I don't know, for me anyway, I have to like go through like books or tutorials and do it like on a regular, okay. Yeah, I hear you. That's also why I think hands on makes a difference because it just sticks in your brain more. The workshop is like just, it's so intensive. Yeah. There's that too. Like it has to be the kind of thing that's sponsored by a work order. It's not going to make any sense. Yeah. Like you can't kind of do it on the side. It's just, Yeah. I tried to get like, I asked my work about like whether they could sponsor the ticket and they're like, oh, we still don't have a lot of money even though we got a lot of money. We don't have it. I'm like, okay. Cause we had to spend it on making a few other teams to build others. I don't know. But you said you're still, you're having fun cause you're learning, right? Yeah, I mean, honestly, yeah, I mean, I don't regret the decision like at all, at all. It's like, Yeah, yeah, fair enough for that week. I've been like so much, doing so much more broad stuff. Like I was just doing like Appium there and then I just started doing Selenium and I started doing, now API testing and all of that. So it's like kind of like, I don't know, full stock testing. You know, I don't know what full stock testing really means, but like more things and more like self management and like whatever. You know, I haven't really, they were like open to me contributing to like product code, but like I haven't really done that cause I just don't have time. I just realized like, well, you just need to take ownership of the role and do it. But I feel like once I'm over all this crazy personal stuff, then I should be able to find some time to just start developing skills that I want to. And I've been doing like, I did some like courses and things that I've been like participating in like online communities and people like kind of know who, I don't know. I've made a few contracts, I guess. So I think that's a good thing. Awesome. Anyway, why isn't it Magno? I can't know. Yeah, Magno. So he was just on the main stage, I feel like, and it ran super late. I feel like he will be here soon. I'm sure he had to like, I don't know, like run, get some water, et cetera. I'll book him in a second. Oh, okay. Words says he's just good. So I should just use the Kubernetes channel, but I guess if someone's like struggling to get in or whatever, I can DM them. Exactly. They're on the list. Yeah, like if someone is like saying that they're lost, I mean, they can't find their channel, you can DM them, et cetera. Try to stay in the main room. Okay. Well, it would be handy if this list actually had the total number of participants, but it doesn't. About 53, okay. Poor Linux. There's all kinds of issues. There really is. I mean, just that the Zoom version is not that well supported. Like it works, but it's got a lot of the little things don't work great. It's like there's a conspiracy. No free operating systems. If you try to use one, you're gonna be screwed over by every single app you try to use now. I don't know. Oh my God, there's only five minutes left. It says something that I have to send you to YouTube. No, it's okay. Cause I'm the one who started it. So I had it. Okay. So it's out of date. Yeah. Yeah. I didn't, like after Hugo, I think realized that the reason I wasn't able to get the role or that I wasn't able to delete was that I didn't have a 2FA enabled on Discord. Oh, but it didn't give you a proper error. That's... It didn't tell me anything. So... Hi, I know. Okay. So I said it would be a good idea to add this to the documentation somewhere. That is very true. Hello. Hey, Magno. How's it going? Good and you? Excellent. This is Mona Magno. I love your background, by the way. I was not in the Zoom earlier, so I couldn't tell you that, but good times. Thank you. All right. I'm making co-host Magno. Okay. And I am making Mona the host. And now I can't do anything anymore. Okay, so I see that there's 12 people in the waiting room. Yeah. I will drop off, but if there's anything you guys wanna just figure out before you start letting participants in, go for it. And just poke me on Discord if there's anything. I'll be there. Okay, great. All right. Is there anything? Yeah, thank you. I just wanna go grab water quickly before the workshop starts, but is there anything that you wanted to go over quickly or? No, it's fine. Yeah, once we let everyone in, I'll give them the instruction so we can start from scratch and everyone. And yeah, if you can grab the links, I'm not sure, like are we enabling chats here or just on Discord? So the moderator guide that they sent me said that don't encourage the use of chats in the Discord, in not, sorry, let them use the Discord. I think it's because Linux people cannot see the chat or I don't know, there's issues with it. No problem. I don't think we can disable the chat. That's not what I was told. I was told that, just don't encourage the use of the Discord. But was there anything else? Were you planning on using breakout rooms or? No, no, no. Polls, anything like that? No, no. Okay, that's gonna be pretty straightforward. Yeah, exactly. So yeah, it's just sharing the links. Like I'm not sure if I post some link here, if you can post there on Discord or vice versa, right? So that everybody sees. Okay. Also, I don't know if people follow me on YouTube and other streams, if they can follow as well. Like, so yeah, basically it's the slides and the documentation with some commands that I'm using and so that people can check and they don't need to type themselves. They can just copy and paste. Yeah. I think it would be best if we focus our communication through the Discord, cause that's what I was told, but if anyone is not able to find the channel, then I'll probably have to troubleshoot it. Sure, sure. No problem. Sounds good. I forgot to ask is if I think they need to send a flag to flag PopBots, I think they were, I think they are supposed to get it in their list. Let me just ask Flo quickly. Yeah. I'm just asking her quickly like, okay, there's some other people there. She, JJ. Okay. So I'm gonna quickly go grab water. I don't know if you wanna start letting people in or... No, yeah. Let's give five minutes, I think. I'm gonna grab something too. Okay, perfect. I'll see you in a minute. Okay. You from Montreal? Oh, sorry, I was on mute. Yeah, I live in Ottawa, originally from Brazil. Okay. People without last name, so... Okay. Okay. So yeah, let me know when you wanna start letting them in. Yeah, I think you can start letting everybody in right now. We can take at least five to 10 minutes until everybody is ready so that we can start. Okay. Are they in alphabetical order? Okay, I don't know who... Okay, I see. I think so. Oh my God, this isn't... Okay. Okay, I figured it out. Okay. I figured it out. That is really important. So for those of you who have just been let in, we're just letting all the participants in now. The workshop will start briefly. If you can DM people who... There's last name we don't know. I think, yeah. I think it can message them through zone here. Well, I'll DM the channel. This is someone... Hello, everyone. Hi, we're just adding the attendees, so please be patient while we let everyone in to the room. Yeah, it should be starting in around five minutes, 10 minutes at most. Yeah, I'm seeing a lot of names that I'm not finding on the list. What's up with that? Okay. People have long names. Hey, Magnus. Hey. I just like your slogan behind you. Oh, background. Thank you. You're from Brazil? Yeah, sure. I know Luan Pacote. Oh, okay. Yeah, yeah. I studied with him. Yeah, yeah, yeah, yeah. Yeah, we worked together for a couple of years. Nice. Okay, are you in the Discord channel, too? No. Okay. No, no, no. I just discovered this event for a chance. Yeah, so take a look at some instructions that they sent to the Discord. So to access the Discord channel for this workshop, you need to send a hash string to the flag bot that's on the Discord server for Narsak. So make sure that you do that so that we can start. And you can ask questions. I'm gonna post some documents there as well. And yeah. All right. Can you repeat where I can find the link of the Discord? You should receive the link when you signed up for through Eventbrite. So take a look. It might be on your spam folder. Some people said that it went to the spam folder. All right. Take a look there, please. Okay, thanks. Thank you, Jeremiah. So yeah, just a couple more minutes and we will start. Let me see my slides here. I've got everyone in from the waiting room. Okay. Sounds good. So we can start then. Yeah, so. Okay. Yeah. There's still, you know, if anyone shows up late, should I let them in? Yeah, yeah, no problem. Okay. Sounds good as long as they catch up. It's fine. Okay, let's see here. You guys see my slides, full screen or presentation mode? Is it full screen or presentation mode now? We need to switch. I think it's a presentation mode. Okay. Zoom does that all the time. Okay, is it full screen now? Yes. Awesome. Thank you. Sounds good. Okay, so yeah, let's start. Yeah, thanks everyone for coming. This is the Kubernetes Security 101 workshop edition for NorvSack. We're best practice to secure a cluster. My name is Magna Logan. I'm part of the team NABLA at Trend Micro. So first I'm gonna introduce myself and then we're gonna go into the details of setting up our cluster. So because it takes a little bit of time to create your cluster on AWS, EKS. So we're gonna do that first and then while those clusters are building and being created, being deployed, then I'm gonna go a little bit, talk about Kubernetes, the architecture and then we go back to the cluster to configure the web application and start playing with it. So we're gonna do some attacking some clusters and then defending the clusters as well, right? At any time, feel free to ask questions. You can either unmute yourself and ask questions here or try to ask the questions on Discord. I'm not looking at Discord all the time but I'll try to keep an eye on it. So yeah, if I don't see it, feel free to just unmute yourself and tell me that there's a question there. Okay, let's start and then I'll give you some details on how the infrastructure is going to work. So yeah, my name is Magna Logan. I'm an information security specialist and senior researcher at Trend Micro here in Canada. I'm a member of the cloud and container security research team. I'm also a member of the CNCF. It used to be called SIG Security Team but now it's Security Tag Team. They changed and I forgot to update the slides. Sorry about that. I'm not a Kubernetes security expert yet. I'm still learning. So yeah, feel free to, if I say something that you don't think it's right or that's totally wrong please let me know and we can chat about it or otherwise we can talk after the workshop and can discuss more about it, right? I have a personal blog here at katanasac.com. I published some blog posts at least once a month and there is also all my contact information there and all my previous talks from since 2011. All my talks and slides and videos that presentations that were recorded are there as well. So you can take a look at anything from application security, DAVSEC ops and lately Kubernetes security. Any questions so far? There's a question in the discord about cloud nine. Yes, I'm gonna show a cloud nine and how to deploy it. So we're gonna go from scratch. The only thing that you need prior to the workshop is to have an AWS account, right? Preferably a separate one that you don't use, right? I don't think there's going to be a lot of costs involved, but some of the services might cost you a little bit. So yeah, and especially it should be a separate AWS account because we're gonna do some stuff that can be considered dangerous, right? So don't use your work account or something that you're running in production, right? So don't do that. So the only thing that you need before the workshop right now is create an AWS account. Let's see here, chat. There's a possible to use free tire for this workshop. So we're going to try to use most of the free tier stuff but some of the things might not be free tier. So with the KS just to spin up the cluster, right? You pay for the control plane as well. So AWS charge you a small fee per hour for having that control plane or that master node available for you, right? So you might have some charges there but it's not gonna be a lot. And as long as after the workshop, you delete everything and I can show you how to do that easily. But yeah, I can't guarantee that there won't be any charges. Okay. All right, thanks. So yeah, this is the agenda for today, right? We're gonna go at each topic, going to the Kubernetes architecture and then we're gonna talk about the MITRE attack framework, attacking Kubernetes and then defending Kubernetes, right? And yeah, before we go into setting up the environment, I just wanna mention here that last year I created this list on GitHub as a awesome Kubernetes security list. I've been studying Kubernetes security since January last year and a kind of full time since June, July last year. And what I've been doing is, okay, there were a bunch of links and resources and books and videos that were available out there but they were always scattered around the internet, right? And it was sometimes I needed something and it was hard for me, even for me to find. And yeah, thanks Igor for posting the link there. So yeah, so I decided to create this GitHub project there where I put everything that I find related to Kubernetes and Kubernetes security that that's interesting. I put it there so that everyone can benefit from and you can fork it, you can submit new PRs and yeah, feel free to use this as you wish for your Kubernetes journey, right? So there's a lot of stuff that the basics on how to start with Kubernetes and then we go into more advanced security stuff there as well, so there's a ton of stuff there. Okay, so before we go into the presentation and some details on, I wanna go to the AWS account so that we can set up the cluster and while this is building, we can talk more about this. Let me stop sharing here for a second. There's a question whether you can post the link for the PDF, someone share your GitHub. I don't know if that's... So the GitHub, let me post this to Discord. The PDF, it's not a PDF, it's a Google doc. I posted that already on the Discord channel, it's on top here. Let me kind of... The instructions, right? Pretty much everything that we're gonna do, almost everything that we're going to do it's there on this Google doc so you can copy it, you can open, right? So yeah, spoiler alert, there's stuff there that if you don't wanna know right now, don't check it out yet, but I'm gonna do everything here with you guys and explaining everything that I'm doing, right? Okay, so let me come back here and let me share the screen again, one second. Okay, yeah, can you guys see my screen? Can everyone see my screen? Okay, so yeah, once you log into your AWS account, you can type here, Cloud9, right? And it's going to show you this option, the Cloud IDE, you click it, right? And it's gonna give you the option of creating the environment, right? So make sure that everyone is there. If you're not, please let me know, right? So that we can all be on the same page while we go. So that once we set the cluster to be created, then we can talk more about Kubernetes and how it works. So can you just redo the last part? I'm sorry, I missed that. Sure, yeah, did you log into your AWS account? I did, actually. Okay, so just type here on the search, Cloud9. It should show the option here for this service, Cloud9 service for AWS. Yeah, if you're on a different region, right? Make sure that you use that region throughout the workshop. If you wanna use just US East 1, it's fine. But if you wanna use a different one, yeah, it's okay. Just make sure that you keep the same region. Sometimes some regions might not be available for creating EKS clusters, so we're going to see. But yeah, preferably use the US East 1 or something that you know that it works for you, yeah. Okay, so here on AWS Cloud9, we're gonna create a new environment, right? So we're gonna create a new instance for Cloud9. That's basically a virtual desktop or virtual developer environment that we're gonna use to create the cluster, right? So here I'm gonna type Kates101 workshop, right? And here it's optional, any description that you want, I'll just type the same thing here. Next step, pretty much the default settings here works fine for us, right? So create a new EC2 instance using the T2 Micro, instance as well, so that's free tier, so we don't pay for that. And using the Amazon Linux platform to Amazon Linux 2 platform as recommended, right? You can also leave here for cost saving options. So after 30 minutes of idle time, right? The instance, the Cloud9 instance shuts down. So you forget to disable or shut down the instance, it's gonna shut down automatically for you. Only the Cloud9 instance, right? Not the EKS cluster or anything, right? And yeah, the service row, we're gonna create a row specifically for your Cloud9 instance. So don't worry about that right now. And yeah, next step. So it's just going to show you the options here. So to review and create the environment. Let me see some questions here. So why oh? Yeah, okay. Cloud9, yeah, okay, that's good, good. So yeah, that's gonna take a few minutes to be created. It's basically, if you played around with VS Code, right? It's basically a VS Code online on AWS that it can use to do development work, right? The virtual desktop or a VDI way of doing things, right? I think GitHub has launched their own kind of Cloud9 option as well. Azure has their own too. So yeah, it's just a way so that everyone is on the same environment with the same configurations. It's easy for us to play around and create the cluster and make sure that everything it's going to work for everyone, right? Because we have the same settings. So that's easy. Yeah, it should be created soon. Any questions so far? Let's see, Discord, right? Yeah, if you already have an account there for AWS try just try a different email. Yeah, okay. So yeah, that created here for me. So I can close this and just I'm gonna increase the font size, don't worry about it. Let me close that as well, okay. So here to increase the font size, let's see if there's something, okay. To increase the font size, I can click on this engine here on the top right corner and go to user settings and terminal and the font size here. So I'll put like 20. Yeah, I think that's, is that okay? Is that good for everyone? Can everyone see this font size? Okay, good, good. Another thing that we're gonna do on the preference here is disable the AWS manage temporary credentials. So if you go here to preferences, AWS settings, this is enabled by default. We don't want that, we want that disabled. So click here and make sure that that's disabled. Okay, sounds good. Okay. So on another window or a tab, right? I'm gonna go to the console again. Sorry, Magno, it's to disable AWS management. Yeah, and the address manage temporary credentials, right? So we don't want that, okay? Sounds good. What? What did you do? I'm sorry. Did you do that, all right? Did it work? Yeah, yeah, yeah. Okay, so yeah, just open another tab on the console so we don't leave here. We stay on this tab, it stay on the cloud nine instance. And here we're gonna go to IAM, right? The identity and access management. And we're gonna do something here that's not safe, but it's a way to make sure that everything works from our workshop, right? We're going to create a new row and we're going to attach that row to my cloud nine instance, right? So here I'm gonna go to IAM, right? Type IAM, click here and then go to rows. And I'm gonna create a new row. Is everyone here on the same page? So here on the row, I'm gonna select, this is selected by default, AWS service. So I'm gonna click on EC2 because I wanna apply that row to an EC2 instance, right? And then gonna click next here on the bottom on permissions. For permissions, I'm gonna select this administrator access. That's not a good thing to do, I remind you, but like for this workshop, for this training to make sure that we don't have any issues creating our cluster, we're gonna do that. That's why it's important to have a separate isolated AWS account, right? No, I think since at least one person was behind with creating their account, maybe you can just quickly go over everything that was done up to this step. Okay, sounds good. Yeah, let me just finish that and then I'll go back. Okay. So you can create a row name like cloud nine admin or something, it's up to you. Any name, just remember that because you need to attach that later to the instance and you create a row. Did you skip the tags? Tags, yeah, I skipped the tags, we don't need that. All right. Okay, so just recapping a little bit. Has everyone created their cloud nine instance already? Or may I ask who hasn't created their cloud nine instance yet? Is anyone like behind creating the cloud nine instance or has everyone already created it? Let's see, anything on the chat? No, you're on mute Mona. Yeah, if someone's typing, that's crashed, they're trying to create it again. Okay, sounds good. Yeah, so on the cloud nine instance, once you create that, the two things that you need to do, actually you don't need to do two things, you only need to do one. I did two things. I did one thing for the font, for the terminal, so that it's easy for everyone to see. But the main thing that you need to do here yourself is just disabling these credentials, AWS managed temporary credentials, that's it. That's what we're gonna do now and we don't do anything on the cloud nine instance yet. After we've done that, we go to another tab and we go to IAM and we're going to create a role that we're going to attach to that instance later, right? So I went to IAM, identity and access management. I went to roles and then I created a new role, right? For this role, I'm just gonna show again, I'm not going to create. I selected AWS service EC2, I went to permissions and I selected the administrator access permissions and then I basically skipped this tag options. We don't need to add any tags and then I added the name here, any name cloud nine admin or something like that. It doesn't matter as long as you remember because you're gonna need that to attach to the cloud nine instance, okay? So far so good, any questions? How we're doing there on Discord? Is everything good? Did they just, for the cloud nine instance, did they just use the default settings? Yeah, default settings, pretty much, yeah, exactly. Okay, so let me cancel that. Oh, good, okay, thank you. Yeah, it's nice that you guys give me some feedback either on Discord or Zoom so that I know that everyone's on the same page. I can go back and explain later, no problem. Just a off-top question, this will be available on YouTube, this workshop? Yeah, it's being recorded, it's being streamed to YouTube already and it's going to stay there, I think, pretty much, yeah. All right. Okay, so after, okay, create the role, let me show again. So after you go to IAM, okay, sounds good. So after you go to IAM, you go to roles or, yeah, I don't know the name in French, sorry. Roles and create role. Create role, you go to EC2, it's the first option here on top, EC2 instances, right? Next, permissions and on the permission, you're gonna select administrator access, right? So that basically gives that EC2 instance, administrator access to everything on your AWS account, right? So that's very dangerous, that's very permissive, but we're doing that just for the purpose of this workshop, right? So don't ever do that on a production environment, please. After you select the administrator access, you go to tags and you don't need to add anything here, only if you want, but it's not gonna affect anything. So you just click review and add the name to your role, right? So it could be cloud nine admin, admin, two, okay, right? And then you create role, yeah, exactly, exactly. So yeah, if that instance gets compromised, right? Anyone can get the key from that instance and those API keys are going to have permission across the board on your AWS account, right? So they can basically do anything. So that's not a good practice, okay, okay, now that we've done that, we're going to EC2. So go again to the search panel type EC2 and you should see one instance running. In my case here, I have three because I tested everything before, right? So that's working. So you should have AWS cloud nine something, the name that you gave to your instance. You should not have those two below here. You should have at least one instance, right? So I'm gonna select that cloud nine instance and I'm gonna attach the role that we've just created, right? To this instance. So here actions, I think it's security and modify IAM role. So I go, I select here this instance, I go to actions, security, modify IAM role, right? On the modify IAM role, it's gonna show me an option here, cloud nine admin or the name of the role that I gave when I created the role and I'm gonna save it, right? So that's it, let me repeat it again. So select the instance EC2, the cloud nine that you've just created, even. So this one actions, security and modify IAM role. And then you select the admin role that you just created before in IAM and you click save. Just have a question, while we didn't create the role directly from the view you were before, when you gave the role to the instance, there was a box where you can create. Yeah, it's another option. Exactly, yeah, same thing. Yeah, no problem. Just another way, because I wanna show that the steps step by step differently so it doesn't get, anything doesn't get too complicated, right? So one thing at a time. Perfect, right? Yeah, so you should see if you attach the IAM role here properly, you should see it here on the instance description, the IAM role name. Okay, so far, so good. Okay, is anyone behind? Does anyone need help? So once the role is created, yeah. Okay, yeah. Let me just move this here. Okay, now we're going back to the cloud nine instance. So once the role is created, where do they go? Oh, sorry. Once the role is created, you need to attach to the EC2, right? So after you created the role, you go to EC2 here, virtual servers, right? And you select your cloud nine instance. It should be only one for yourself if it's a brand new account and region, right? So it should be only one. It's gonna show AWS cloud nine something in the name of the cloud nine, the name of the instance that you gave to the cloud nine, right? You select that, you go to actions, then security and then modify IAM role, right? And you attach, you select the role that you just created before and you click save. That okay? Okay, can you guys see my cloud nine? My cloud nine browser here? No. No? Nope. I just see the panel of the EC2. Okay, let me go back then. Is that okay now? Yeah. Okay. Okay, now we're going to install the tools that we're going to need to deploy the cluster, right? So the tools, the main tools that we need here for deploying an EKS cluster, right? So EKS is the managed Kubernetes service for AWS, right? So we're going to need first Qubectl or Qubectl control or Qubectl and I'm going to explain with more details what it is and what it does. But it's basically the command line interface for you to interact with your cluster, right? There is some dependencies that we're going to install as well, some libraries and all those stuff but also we're going to install EKS CTL. So EKS CTL is a third party tool developed by WeaveNet, WeaveNetworks, yeah, third party company that doesn't belong to AWS but it was kind of, it belonged the, it became the official solution or the official tool to create EKS cluster. So it automates a lot of stuff for you so you don't need to worry about and you're going to see how easy it is just to create that EKS cluster. So now if you want to follow along on the Google Doc that I shared on Discord, right? We're going to go to step one, step one of the environment setup, right? So below there's some topics there or what we're going to do and step one is installing cloud, Qubectl on the cloud nine instance, right? So the basic, the first command here is just to install Qubectl, downloading that from Amazon EKS. Okay, after that, we're just going to give some execution permissions to Qubectl, oops. So this is in your, in the repo you shared in which... It's on Discord. Oh, it's on Discord. Yeah, it's the first link that I shared on Discord before the workshop at 3.04 p.m. Oh yeah, the Google channel. Which channel? Oh, the Kubernetes channel. So if you don't see the channel, it's because you need to send a flag to the, to the flag box. Exactly. I never used Discord before. I don't know. I don't do that. So there should be a user flag bot in the, in the users. User flag. All right. And there's a flag that I shared in the chat here. In the Zoom chat. Okay. And you should copy and paste that and send it in a direct message to the flag bot. Okay. Like a message. As a message. Yeah. Okay. Is that okay? Please take a look here. Looks like there is a problem with the link. Can I resolve host? Yeah, yeah. I'm checking that one second. Okay, cool. I'll try to help if I can. I've just sent to this flag, but I don't know what. You sent a message? It should open a channel, Kubernetes channel on the workshops list. Is it normal that you have two-curve new requests? Yeah, sorry. Yeah, the command is wrong here. So I'm checking my notes. Let me update there. All right. Kubernetes. Oops. Okay. Here. Let me... Oh, yeah. Cube, CTL. Okay. Yeah. Yeah, exactly. Okay, now it's right. It should work now. Let's see. Oh, no, it's silent. I forgot to. Okay. CTL. Why is it not working? Let me grab that. Is it attached or not? The credentials are here. Let's see if there's any permission problems here. We can skip the step two. This is just to check this one. Check. Sorry. We can skip the step two, right? Why skip step two? No, we need to do that. But yeah, let me... Step two is to install the dependencies. Let me do that first and see if anything is missing. I think I needed to do some... Let me update here. Make sure that everything is up to date. Let's see everybody on instance. Let's see discord here. Let's see the chat here. Yeah. Okay, dependencies. I think it was working now. Yeah, so I need to install those dependencies, right? JQ, GATX, and bash completion. Let me go... Let me try to... I cannot access. No such... It didn't move. Oh, okay. Yeah, it didn't move there. Can you show one more time how to connect the role to the instance? Oh, sure, sure. I'm not there. Here? Yeah. Okay, here... Okay, yeah, I got there. Actions. Oh, okay. Security. Modify IAM role. Thank you. Got it? Okay. So yeah, I'm missing one. So... Yeah, I'm missing... Okay. So we're still at the step one, right? There's... Okay. Yeah, we're still at step one. I'm gonna make sure... I'm gonna make sure... Yeah, no problem. I'll let you figure it out. Yeah, yeah. Yeah, we're gonna do a lot of debugging today. Don't worry. I can't... I can't... Okay, now it works. I have data... I can't secure... Can you check the latest command on the document? I just updated to make sure that works, right? So it wasn't transferring the file to the right location. So that's why it wasn't finding. But now it is there. Yeah, it's working. Okay. And, okay. Step two, install the dependencies if you haven't done it yet. And step three, I'm going to enable bash completion for kubectl. Okay? Okay, so we're at step three. We've done step one, step two, and step three. Does anyone have any problem so far? I was following the AWS. The what? The AWS... Instruction. Instead of your restriction. No, you should follow the Google document. All right. Yeah, yeah. Yeah, if someone can post the comments there on Discord, yeah, that's great. Unfortunately, I can't do everything at the time. Exactly. Does everyone see the doc? Does everyone have access to the doc? Yeah, it should be enabled to anyone with the link. Yeah, so I see a lot of people connected there. So here I'm installing the latest version of kubectl which is the 1.20.4, right? So that's the latest one available for Kubernetes. So far we've done step one, right? Downloading kubectl. I don't know if I have PIN permissions. Do I have? I can't PIN, sorry. I think I just pinned the doc. Okay, got it. Yeah, I don't have PIN permissions on the Discord. So yeah, first we've done, we downloaded kubectl. We put that on the slash usr slash local slash PIN, as kubectl, right? And we just gave it the binary execute permissions, right? Plus acts, the chchmode command there is just to give executable permissions to that binary. I can execute the binary file. You can't? Yeah, I'm just getting this error message. Can you also? I have already set seagumode to exec this binary, but... So make sure that you transfer to the right location, right? So there is on the first command, there is a location dash o, and then I tell, let me show here the command. It's here. Okay, so here, right? This is the first command that I'm doing, right? Yeah. User local. Yeah, user local being kubectl, right? Yeah, that's it. Okay, and then sudo chmod plus acts all together. I think the first time was like with a space. So I fix that and then the location of the binary. All right. Got it? Wait, I have already installed the completion. Yeah, so do step two and step three. And if anyone has already done it and want to move on with the other steps, it's fine. Just let me know if you have any issues, right? Just want to make sure that we're on the same page until we create the cluster so that we can go back to the slides and I explain a little bit of Kubernetes and how it works before we start playing around while the cluster is creating. Okay, so step two is just install dependencies, right? Step three is just setting up a batch completion. And then step four, yeah. If you have not completed until step three, yeah, let us know. Now, step four, we're going to install EKS CTL, right? So that's the other tool that we're going to use just to create the cluster, right? So basically step four, have it here, download EKS CTL from their latest release, right? You move that to the binary location as well and you enable batch completion. Step five, that's right, oops. So step four and step five now. Create, download EKS CTL, the latest version, right? Move that to the USR local bin and then enable EKS batch completion for EKS. Has everyone done that? Is there anyone behind? Does anyone need any help? I'm just getting sure that I don't miss anything here because I just follow the instruction of the AWS. Yeah, focus on the document there, the Google Docs. All right, just removing some lines of the Berserker CR. Okay, so moving on, another thing that we're going to do, right? I also posted a GitHub repo on the Discord channel. It's basically just two files, two YAML files, one for creating the cluster and the other one for creating the objects inside the cluster. So that's step six. You can just git clone that repo, right? And I'm gonna show you what's inside there. So if I go here, okay, let me see if I can increase, yeah. Okay, here, yeah, okay, that's good. So basically what we're going to do, right? We have two files here, two files here, right? This is the first file that we're going to use. It's EKScluster.yaml, right? I already have the command here to create the cluster as well if you need to remember, right? Just a YAML file, basically I'm setting up the cluster name and the region where you want to click the cluster. So if you're using another region for your Cloud9 instance, it's a good idea to change, to put the same region here for your cluster, right? So that you can access it on the same region. I don't have any delays or any issues, right? And basically what I'm doing is I'm creating one cluster, one Kubernetes cluster with one managed node group. So what does that mean? The managed node group is a way for you to scale instances on AWS. And here, the managed node group has instance type of T2 small, right? I'm telling that the minimum size of this managed node group is one. So it should have at least one instance, right? And the maximum size is three. So it should have maximum three instances, right? Three worker nodes. But I'm telling it that the desired capacity, right? It's just one. The desired capacity is going to start with just one node. And I can, later I can change this here and it's going to scale my cluster. So if I change later this to two or three, it's going to update and deploy a new C2 instance as a worker node for my cluster, right? Labels, right? Just telling this is a worker node here and it's not that much, right? There is many more configurations that it can do here. But for this workshop, we're just doing the basics, right? Oh, the volume size as well. So the volume size of the worker nodes, the default one, the eight gigabytes, that's more than enough for what we need. Yeah, and that's it, right? And the other one, the other object, right? It's some Kubernetes objects. I can explain that later what it is here. Okay, close this. There's a question or two. Okay, which questions, we're on Zoom. I think it's about the commands that are being shared with the dot. What is the meaning of the command line beginning by a point? What's the meaning? I think it's just best completion, yeah? Just to enable best completion. Da, da, da, da. See much? Yeah, exactly. Yeah, I think that's the right, Chicago is right there. Yeah, good, good. Okay, so now step seven, at step seven, we're going to create the cluster, right? So make sure that just to a note, right? So just note that on the, it's just, it was my mistake here and I'm sorry about that, but for the EKS cluster file, right? The YAML file, this one, it has a dash, right? And the cluster objects, it has an underscore, right? So make sure that you note that once you create the cluster, you're gonna have, you might have issues with the names, right? But on the document there, I put the right dash or underscore so that we don't have any issues. So now we're going to create the cluster, the EKS cluster. And that's the command that takes a while, usually 15 to 20 minutes to create. And that's why it's important that everyone is on this step so that we let everyone's cluster creating and then we can go back to the slides for me to explain a little bit more of the what Kubernetes is and all that stuff. Okay, is everyone here? Okay. Oh, yeah. So my- So everyone should be starting step seven at this point. Exactly. So yeah, since I have already another cluster with that name, if you've done it already, you need to change the name here. That's why it didn't work, my command. So now it should work. Okay, now it's deploying. Yeah, you already have a cluster with that name. If you got an error, you go to the EKS, you can go and delete the first cluster and then use the same name, but that's gonna take some time. So just change the name here on the metadata of the YAML file, save the file, right? And then run it again, right? So that's running. So if it's here, if you're here, then yeah, your cluster is being created. So I just wanna make sure, and I wanna pause here, make sure that everyone is on this step so that we can go back to the slides and explain a little bit before why this is being created. So we're stopping at step seven right now on the document. So I guess at this point, if anyone is stuck, they should really speak up, because otherwise it's gonna be harder to catch up if they haven't started this. Exactly. Yeah, change the name and try again. Yes. Yeah, if you forgot it. Actually, I ran into that problem and I didn't have to change the name, I just like delete the cluster name and rerun the work. Yeah, but then you have to go to the cloud formation, right? Did you go to the, how did you delete the cluster? The command is actually in the logs, they... Yes, it is provided when you have an error. Okay, yeah, this one. Correct. Yeah, exactly. But yeah, that works too, yeah. Yeah, it's sometimes that depending on the stage that the creation is, it takes a while to delete the cluster, right? It might take a few minutes, right? So that's why sometimes it's just better to rename and move on, but yeah, you should also delete that. Okay, let me see what's wrong here. Okay, so yeah, that's a common error, right? That happens on AWS, unfortunately, they have a limited amount of options of, let's see, IPs and everything that you can create on specific regions, right? And we're using US East One, which is one of the most used ones. Yeah, exactly. Be careful with cloud formation there. So either once we get this error, we can either try to change regions or just try creating again. Let's see. It's, I know it's a pain. Let me change the name, but yeah. Yeah, I'm gonna delete everything later, so I don't wanna bother right now. So either you change, you try again and it might work because when Ikea CTL is creating, right? It's going to create, I think one VPC and two subnets for your cluster, right? And that's what it's telling here. That's okay, cannot create cluster because the availability zone doesn't have sufficient capacity to supply the cluster, right? You can either retry and choose, right? We're not specifying specific availability zones. So that's one, maybe one improvement for the next workshop, but that's an issue that I faced before many times on AWS, right? So you can either try again because it's going to choose another availability zone or you can specify here on the AKS cluster, but you need to know which availability zone has the capacity to do that, right? So that's, I think, yeah, that's some, either try again or try changing regions. Yeah, exactly. So let's see. So I hope that everyone is here on this stage right now and trying to create their clusters. If anyone is behind step seven, before step seven, please let me know. What does this Coup Ctl apply? That's step eight. That's step eight, we're not there yet. All right, sorry. No problem. So let me go back to the slide stand and please let me know if you have any issues. Stop sharing. Let me go back. Slides, switch. Okay, is it full screen for you? Yes, it is. One is asking a question and need, okay. Yeah, it takes some time. It takes some time there. Don't worry about it. We're going to leave it creating the cluster. So just take a look in every minute or so to make sure that it's still creating. If it gave you an error and failed, right? That's because of the availability zone. Like I mentioned, try changing the region or changing the name of the cluster and yeah, trying again. I know because everybody's creating the clusters and if we're creating everything on the same region, so AWS might complain about that. Yeah, so yeah, once from time to time, we go back to the cloud nine instance, just check to see if it's still creating the cluster, right? Okay, so yeah, I'm assuming that if you don't know Kubernetes, right, I'm going to explain it from scratch. Basically, Kubernetes is an open source system to automate the deployment scaling and management of containerized applications, right? So basically, it was originally developed by Google in 2014, actually released in 2014, right? As, and based on internal projects from Google such as the projects name board and Omega, right? So it now belongs to the CNCF, which is the Cloud Native Computing Foundation, right? And you can find pretty much everything about Kubernetes on their main website. That's Kubernetes.io there on the slides. The Cloud Native Computing Foundation is a sub foundation of the Linux Foundation, right? It has many different projects. There are considered Cloud Native and I'm gonna go into details of what that means. Some other details, fun facts, actually. Kubernetes word comes from Greek and it means helmsman, right? Which is the captain or the pilot that kind of drives the ship. I don't know the right word, but the ship, right? Usually in an analogy for a nautical teams, right? We have Docker, which are the containers, right? And the ship is Kubernetes. Kubernetes is the captain of the ship, actually, that's driving those containers, right? And hopefully they're not getting stuck like that, the other ship that we saw in the news recently. Kubernetes or Kates, right? Kates, K8S, the abbreviation is used as a way, the meaning of this abbreviation is just to tell you the number eight is that there are eight letters between the K, the K and the S, right? The first K and the last S in Kubernetes, right? It means that there are eight letters between them. So that's why K8S. Okay, let me just check here my cloud nine, okay, it's good. So, okay, so why do we need Kubernetes, right? So why Kubernetes and are there any other better solutions out there, right? So that's why I bought this comic here from Dilbert, right? Kubernetes is not the solution for everything. You need to understand not just because Kubernetes is very complex, at least in my opinion, right? You need to understand what you're doing and if there is going to be a really a need for using Kubernetes for the foreseeable future for your environment, right? So if you're using microservices, if you're using deploying a lot of containers and you have applications and environments that are changing very frequently and you need to update them and deploy rollout updates or perform rollbacks in this environment, right? Then yeah, maybe Kubernetes is for you. But if you just have like a website, one blog or a simple WordPress site that you use for writing your blog posts, your articles, right? So I think Kubernetes might be too much, right? It might be too complex and too complicated for you. Well, of course you can do it, right? But it might not be necessary. There are other options besides Kubernetes, right? So Kubernetes now leads the leader in this container orchestration solution, right? But there are other options, right? You can also try Apache Mesos. You can try Docker Swarm. You can check HeadHat OpenShift, which is basically Kubernetes in an enterprise version. And also there is the HashiCorp Nomad, which is also a orchestration tool as well. So yeah, the CNCF, as I mentioned, what is the CNCF, the Cloud Native Computing Foundation, right? It's a foundation that supports and promotes all these open source projects that are being used by many organizations that they're using usually on cloud environments, right? So the CNCF creates a sustainable ecosystem and promotes the communities to support those projects, right? The health and the growth of those open source projects. As I said, the CNCF is a sub-foundation of the Linux Foundation, right? And on that link there on the slides, the l.cncf.io, you can click, you can check it out, and it's gonna show you the landscape where you have, like, this is just a sample of the projects, right? If you go to this link, the landscape of the CNCF is hundreds of projects that are under the community or the protection or the support of the CNCF and help them grow, right? And Kubernetes is just one of them, right? Also, Kubernetes relies on other open source projects that are also part of the CNCF. For example, Core DNS, right? Container D, right? So all of those help for the charts for Kubernetes, at CD as well. And if we have time, we're also going to see some security projects that are used for protecting your Kubernetes cluster, such as OPPA, Open Policy Agent here, and Falco as well. So if we have time, we can take a look at those, but yeah, we'll see. Let me take a look at my Cloud 9 instance. Yeah, it's still creating, so that's good. And what is Cloud Native, right? So what does it mean to be Cloud Native? As a definition from the CNCF itself, Cloud Native means applications or software that are considered have at least these characteristics on the last, right? They're scalable, they're dynamic, they're resilient, they're also loosely coupled, manageable, observable, and automated, right? And some examples of those applications or those software are containers, service meshes, microservices, right? Immutable infrastructure and declarative APIs, right? So usually an application is API first, they run on container environments or they are considered microservices, they have each service for their own goal, their own function, right? The domain-driven development kind of way. So that's this considered Cloud Native and there's the proper definition here on the CNCF GitHub. Okay, any questions so far? Are we good? Okay, so yeah, if nobody's saying anything, then I'm assuming that's okay. Good, good, okay. So here is the Kubernetes architecture and that's very important for us to understand before we move on to the next steps and play around on our clusters. So there's two main components, the two major components for a Kubernetes cluster. You're going to have the control plane, also known as the master node here on the left, the big one here with the dotted line. This is a control plane. And on the right side, we have free worker nodes, right? So just an example, you don't need to have free, you can have one as what we're doing in our cluster on EKS. We're only creating one worker node here, but here in this diagram, we have free, right? And inside those two major components, we have smaller components that each of them do something specific for the cluster, right? As you can see here on the diagram, one of the main components for the control plane, right? Or the master node is the Kube API server. And why is that, right? The Kube API server is basically an API server like any other that handles all the communication that goes inside the control plane, but also outside to the worker nodes, right? So you see that all of the other components from the control plane talk to the Kube API server as well as the Kube API server talks to the external components for the worker nodes, the KubeLit and the KubeProxy, okay? Sounds good so far. Let me just fix that here, okay? Yeah, and so the Kube API server is basically responsible for exposing and receiving API calls, right? It's just gonna do that. It's going to be the central location for the communication inside your cluster. Under the Kube API server, we have two other components. We have the at CD and the Kube scheduler. The at CD is very important here because at CD is the database for Kubernetes. It's where Kubernetes stores all the internal information of the cluster. And like for example, which nodes are part of the cluster, what resources exist on the cluster. So everything is there. So at CD, as I mentioned before, it's also a standalone project. It's also part of the CNCF, but here in Kubernetes is a key value store that's used for Kubernetes to store all the information from the cluster. And we're gonna see that why is that important, right? Because in Kubernetes we can do, there's two ways to tell Kubernetes to do stuff through the Kube CTL, right? There is the imperative way where I tell, okay, like we did right now, oh, create cluster, right? Or apply objects, right? So there's a way to do that. But I can also tell Kubernetes, okay, here's my ML file, right? And I just want you to apply that to your cluster. So basically what we call the desired state in Kubernetes, I don't care how Kubernetes gets things done, right? I tell it what I want and then Kubernetes is going to do its magic, right? It's gonna handle itself to create what I want, right? Whatever I define in the YAML file, I Kubernetes needs to understand that and make sure that whatever is on that CD is reflected on the cluster, right? So for example, on that CD showing that I should have a one worker node, I should have one worker node running, right? If I have three worker nodes like this diagram, Kubernetes is going to remove two of the worker nodes from the cluster because it's saying that should have only one, right? So this is kind of the desired state. It does that for everything, every object in Kubernetes, right? If I need to create replicas of my containers, my services, load balancers, all that stuff, right? So that's considered one of the main differences and main approaches of Kubernetes that's different from, let me see one question. Yes, exactly. No, when I say kubectl apply, that goes through the API server. It's saved on at CD. And then the other component that I'm gonna get into a second is checking at CD, which is the controller manager actually. It's checking at CD to see what's in at CD is actually what's running on my cluster, right? So it first gets saved on the at CD, right? Everything that I send to Kubernetes if it's in the proper format and on the correct YAML file, it's gonna get saved on at CD. And then the other component here, the kubectl controller manager, which has a lot of... Oh, somebody got there closer already, nice. So the kubectl controller manager, which has a lot of other subcomponents inside Kubernetes, inside the controller there. So we have the pod controller, the service controller, right? They're checking that at CD through the API server, right? So the kubectl controller manager checking at CD to see, okay, what's in at CD is actually what's reflected on my worker nodes, right here. Demi, Demi, I'm gonna get to that in a second. Just hold your question a bit. Once I finish explaining this architecture and we're going to understand why we don't have a control plane in our EKS cluster, okay? I'll be there, yeah, just a few minutes more. Other components here, the kubectl controller, right? Next to the at CD here, it's the one that schedule the pods, right? So we have the kubectl API server, we have at CD, we have the controller manager and the scheduler. The scheduler is going to decide where to execute the pods, right? So through the API server, it's gonna go through one of the worker nodes and it's gonna tell, okay, please start this pod, right? And pod, before we go into the definition of pod, consider pod as a container, right? It's not actually a container, but it's an encapsulation of that. But for now, consider pod as a container and on the next slide, I'm gonna explain you what a pod is. Okay. Before we go further, can I ask if the presentation slides are being shared? Yeah, I'm sharing the slides, no? Oh, I mean, is it going to be sent to the participants? Oh, yeah, yeah, later, yeah, no problem. Okay. Yeah, I can share that as a PDF. Yeah, no problem. So yeah, at CD is not a queue, right? At CD is not a queue, it's a key value store, right? It's not a queue. It's a database of key values, right? Have key values there. There's also a cloud controller manager here that manages everything that interacts with cloud providers, right? Kubernetes was created to run in the cloud, right? So things like load balancers and disk volumes that are usually not managed, not created by Kubernetes itself. And we're going to see because we're creating a load balancer here in our cluster. So that manages that stuff. So the cloud controller manager manages interactions with the cloud providers. On the other hand here, on the other part of the diagram on the worker nodes, right? So we have three main components. We have the kubelet, right? Which is responsible for managing the container runtime, executing containers when necessary and collecting execution information. We also have the kubeproxy, which basically it generates managers, the communication between the containers itself and also external communication, right? And we also have here as showed as a computer, right? But we, this is the container runtime engine, right? Or CRE. The container runtime engine is actually the one that creates the containers for Kubernetes, right? Usually by default it was Docker, but I don't know if you heard about that news that Docker, excuse me, Docker Sheem is being deprecated in Kubernetes. So because Docker doesn't follow the container, CRI container resource interface or something, I forgot the name there, but because no, the OCI, sorry, the OCI, the open container interface, since Docker doesn't follow that, Kubernetes had to do what they call a Docker Sheem, which is basically like a support implementation to help Kubernetes to support Docker. Let me see the chat, where I can see the chat here. I don't see my chat. Yeah, thank you. Thank you for, yeah, that's why we posted this news about Kubernetes and Docker, right? So you don't need to worry that too much. So that's one of the one container runtime engines, right? You don't need to use Docker necessarily. You can use other container runtime engines like CRIO or container D. And I think container D might be the one that's going to be the default from now on once Docker is completely deprecated. Right? Okay, so far, oh, okay. This error from Mona on Discord, it might be because it took too long to create your cluster. So that's why the security token is expired. I think it's half an hour. Did you just make sure that you disabled the AWS credentials from your Cloud9 instance? That might be one of the issues. Yeah, I'm not gonna forget. Okay, yeah, that might be one of the issues. That's why this is happening. Is it normal to take too long? It shouldn't, it shouldn't take too long. It should take at least 20, at the most 20 minutes. Mine is created here. So if you're having any issues, just please try again and I'm gonna wait until everyone is on the same page so that we can proceed. My cluster was created in 30 minutes. And my manager, manager NG, NG, I think. Yeah, some minutes. Okay, good, good. Yeah, so if your cluster was created, make sure that you type some command there if you know kubectl get nodes or kubectl get pods. So to make sure that you can access your cluster from the Cloud9 instance, right? That's very important. Okay, any other questions there? Yeah, 15 to 20 minutes. I know that they reduced the time to create the clusters recently, sometime before like last year was half an hour. So I know that they worked on that. But I'm not sure how long exactly it's taking for everyone. So yeah, between 15 to 20 minutes, I think. Okay, so going just one more slide here to the architecture before we move on. So here is just some objects, right? Some what we call Kubernetes API objects that are part of Kubernetes, right? So we mentioned here on the previous slide that we have the cluster, you have a control plane and you have worker nodes, right? Okay, but once that cluster is created, right? You're going to create objects inside your cluster, right? And so as a representation here, you have a node here. And so this node means, okay, this is a machine, right? A worker node, it could be a VM, it could be a bare metal, right? But it's a worker node, right? Inside this worker node, right? It could be the master node as well, but let's focus on the worker node for now. I can have namespaces, right? And so namespaces is one of the things that Kubernetes use to separate projects, right? So think of namespaces as folders or a way it's a logical separation between projects, right? So you can have a namespace for dev environment, you can have a namespace for Q&A and you can have a namespace for prod, but it can also have a namespace for team front end and you can have a namespace for the backend, right? And yeah, so you decide what you wanna do with those namespaces, but it's basically a logical separation. There is no, by default, there's no boundary, there's no restriction between namespaces, right? Deployment here is one of the objects at Kubernetes that helps you deploy and update and rollback. So basically scale your Kubernetes cluster with pods, right? So it's abstraction, right? You really, when you create a deployment, you're really not creating anything, right? It's just adding that information to at CD to know, okay, there's a deployment there, but inside the deployments, that's where you create your pods, right? So as I said, pod is the smallest unit in Kubernetes, right? You can't create containers in Kubernetes without pods, right? And so here is that where we make the difference between pods and containers, because pod is a kind of an abstraction and encapsulation of containers. I can have one pod having one container, but I can also have one pod having multiple containers as I show here on the slides, right? So I can have a pod that has my main application container and a separate pod, which usually we call sidecar, collecting the logs of that other container. And why is that, right? Because it's a best practice for containers to run only one process at a time, right? So we don't want your container to be running your application and also your login processing software or process, right? So we don't want that. That's why we're separating, we're breaking into the processes and the tasks that we want for that specific pod in different containers so that they can be isolated and work independently, right? So loosely coupled again, right? But when you create a container on Kubernetes, you need to use a pod, right? You need to use a pod. You don't need to use a deployment necessarily. It's just a good idea to be able to update and roll back easily, right? But you need to use a pod. Okay, is everybody okay on this part so far? Yeah, I have a question to the after before slide, the previous slide. No, yeah. What is the Kubelet? I just a quick answer because I missed your explanation about that. Sure, sure. So the Kubelet is responsible for managing the container runtime, right? It's executes the containers when necessary and also collects execution information, right? It's like a demon that tells the container runtime, okay? See, you see here that the Kube scheduler is the one that starts the process of creating containers, right? So the Kube scheduler is gonna send a message to the API server. The API server is gonna send a message to the Kubelet to create a container on that specific node, right? This dotted line here or dashed line, this is a VM, right? Or a C2 instance or a Bermato is a server, right? That's the worker node. So the Kubelet receives that information from the Kube scheduler that was sent via the API server and the Kubelet tells, okay, container runtime engine, which is usually Docker. Okay, it starts this container for me, right? Start this container on this node for me, right? Thank you. Okay, good. And in Kubernetes, there are many different API objects, right? So we have pod, we have replica sets, we have demon sets, stateful sets, right? I'm not gonna talk about everything here. That's not the goal for our workshop, right? So in each of these objects, they have specific goals and functions. So they're different among themselves. So it's important for you to learn about them. If you're thinking about using Kubernetes and developing for Kubernetes environments, you need to understand that, right? And they're all, they can all be created using YAML files, right? They have kind of a similar format and standard, right? For developing them. And as you can see, I'm gonna show the cluster objects that we're going to create in our cluster right now. So I'm gonna show that and explain what we're doing because we're creating some of the objects there. Okay, before we move on, just talking about Qubectl, right? Which was one of the first tools that we installed on our Cloud9. So the Qubectl is the CLI tool that allows you to control Kubernetes, right? You can, it's when you create your cluster, right? When we told EKS-CTL to create a cluster, it saved inside Cloud9, it saved a file, a config file inside your home directory is slash dot Q directory. It saved the config file which contains the credentials and the security information that you can use to access your cluster, right? So if you wanna check that out, check that file out, it's there, it has some jock tokens, it has some certificates information. So you can take a look at that for you too just to know more about it, right? So it's similar to the Docker CLI tool, right? It's for Docker containers. So if you played with Docker before, you shouldn't have a problem with using Qubectl or Qubectl or Qubectl, right? Whatever name you prefer, it's a very similar format to Docker commands, right? So that was the intention that when they created Kubernetes and Qubectl to make an easy transition between Docker and Kubernetes. Any questions so far? How we're doing on time, let me see here. Oh, 440, okay. Okay, good, good. So yeah, let's go back to the cluster and create our objects, Kubernetes objects, install our web application so that we can start attacking the web application itself. Okay, let me stop. If you don't mind, if you don't mind, Magno, about explaining just one of the objects of Kubernetes, I'm curious about the config map objects. We here use a lot in my work, but it's like a mess in my head about what is exactly. Yeah, config maps are basically to store configuration information. So they're not a container, they're just stored on HCD, right? It's usually for non-sensitive information, right? So if you have a configuration about your environment, how it's going to be set up, stuff like that, you can use a config map to store that information, right? If you wanna store sensitive information, usually you should use other stuff, right? For Kubernetes, they have the secrets, but the secrets in Kubernetes, as we're going to see here, they're not very secret, right? They don't protect your information because they are not encrypted. They're only base 64 encoded, right? So yeah, that's a problem there. So if you wanna store, oh no, that's why I'm here to hear stuff like that. Oh, sorry, what did I say wrong? Let me share my cloud nine. Okay, can you guys, no problem. Can you guys see my screen? Okay, set the right stuff, good, good, good. Okay, so yeah, if your cluster was created properly, right, you should not see any errors. It should say that your cluster is ready on the region and the name of your cluster here, right? And so let me just clear that. And yeah, so I can do kubectl get nodes and I see only one node here. I can do kubectl get nodes and I can do the output wide to have more information, right? And I can see, okay, they're internal IP, it's ready, the status ready, not rows, right? The version of Kubernetes that EKS is using, all that stuff, right, external IP, Amazon Linux, okay, good, good. So that means that my cluster is working, right? And I can use that to play around with Kubernetes now, right? So exactly, and I'm gonna answer the question from Demi, Demi, and Dapp Pereira, yeah, on a second here, right? Exactly what we told EKS CTL to do, which was to create only one worker node, right? That's what it did. There is only one worker node here. And so they asked it before, okay, where is my master node? Where is the control plane, right? So that's the big difference between what we call unmanaged cluster and unmanaged cluster, right? In unmanaged, you create the Kubernetes cluster yourself, you have to create the worker node, you have to create the master node or the control plane. And you have to manage both of them, right? On the managed services, like EKS, like AKS from Azure or like GKE from Google, you don't need to create the master node, right? The control plane is created for you. You don't need to handle that, you can't see it. You don't access the control plane, right? Actually you do access, but you don't access via SSH, right? That's what I want to, that's what I mean, right? So if you wanna see where is the control plane, you can go to your AWS console and go to EKS, can type here EKS, and you're gonna see here clusters. Yeah, I have two because one didn't work, right? So which one is mine is the insect 21, okay? So here you should have, if you're on the, make sure that you're on the right region, right? You should have at least one cluster here running and active, right, at least one, right? You can do, here is to update the version, don't do that right now. You can click on that cluster and you're gonna see some information here, right? You're gonna see the workloads, okay, here, configuration. Sure, so on configuration, you see that there is API server endpoint, right? So that's how you talk to your control plane. That's how you talk to your master node is via that API endpoint. And one thing that's very important to know is on EKS itself, right? This API endpoint by default is public, at least as far if it didn't change. So if I wanna hit that API, just copy that, right? And hit that API endpoint on another tab, I can hit that, right? I can see, okay, I got, of course, I got an error, I got a photo free, but I see from that response here, let me make it bigger, from that response here, I already know that from this format, this is a Kubernetes cluster, there is a Kubernetes cluster running here, right? Of course, the name of the URL already gave up as well, right? EKS, I already know that there's a Kubernetes cluster, but what's happening is that attackers are monitoring those managed environments and they're trying different URLs, right? They're basically brute forcing or fuzzing those URLs to look for clusters that are exposed to the internet, right? And so, okay, yeah, right now I just asked, I didn't have the permissions, right? So basically what Kubernetes is telling me here, right? It's failure, forbidden, because I didn't pass any authentication, any tokens, right? I couldn't get the permissions here on that API endpoint, right? On that CUBE API server, right? So that's why on managed services, you don't have, you don't see the control plane, right? But because the control plane, you don't have access to it, right? Only through the API endpoint, right? You don't have SSH access. You can't stall anything on the control plane server. So that's one thing to think about if you're deploying, if you want to deploy a Kubernetes cluster, right? You might have more autonomy deploying it yourself, like deploying as an unmanaged cluster, but also you're gonna need to protect that. You're gonna need to monitor, right? You're gonna need to be aware of that. Okay, okay, good, good, it's clear. Okay, so now let's install our objects on our Kubernetes cluster, right? So here I have the YAML file and every free dashed lines is a different object, right? So here, I just put them all on the same file so that we, it's all together, but we're creating separate objects, right? Here, I'm creating an object as of a namespace. Yes, yes. So pretty much for EKS, I'm sure, but I'm not sure I can't say for EKS or GKE for actually for the control plane being public, but for the control plane not being accessible for SSH, yes, they are pretty much standard. You can access that because they have actually they have a cluster of control plane to make sure that your cluster is highly available as well, right? Okay, so going back to the objects here, you have a namespace, right? I'm just telling API version, I'm creating a kind of namespace and the name of the namespace that I'm going to create is web app, right? It's just basically a folder on my cluster that I'm gonna name web app. If you don't specify a namespace on Kubernetes while creating objects, it's going to create a default, it's going to create everything on the default namespace, right? So if you go here, kubectl get ns or namespace, right? So it's going to show you four namespaces already, right? So those are created by default when we created the cluster, right? And I'm gonna tell you each namespace what they mean. But basically you have the main namespace for Kubernetes or the namespace where the master node components are located, which is the kube system, right? You have a kube public namespace, you have a kube node lease and you have the default namespace, right? As I said, the default one, if I don't specify a namespace, everything is going to be created on this default namespace. Okay, so that's why we're creating a separate, oops, separate namespace here. Here's the fun part. Here, I'm not gonna explain everything here right now, but basically this is a bad practice, right? I'm doing a misconfiguration on my cluster here on purpose, right? What I'm doing is I'm telling the cluster that my namespace web app, which I just created above, right? It's going to have a row of cluster admin, right? So I'm telling the cluster here that's gonna have the admin and I'm doing a cluster row binding and everything. And I'm gonna explain more about our back, which is row-based access control later, right? But this is a misconfiguration, right? This I'm doing just for the purpose of the workshop and you shouldn't do that on your cluster, right? Okay, we'll come back to that later. Here, the other object that I'm doing is a deployment. As I said, deployment helps you deploy multiple instance of the same application doing updates and rowbacks, right? Basically, I'm telling that I'm creating a Drupal deployment because we're deploying, we're going to deploy a Drupal web application, right? I'm telling that I want one replica for that Drupal, right? Only one container, only one pod of that web app. And down here, right? There's some other stuff here, but down here I'm telling which container I'm going to use, which image I'm going to use, right? And I'm telling that, okay, the image that I'm going to use is Drupal 8.5.0, right? Can anybody tell me where this image is going to be, get from? How does Kubernetes, where does Kubernetes is going to get that image? Oh, that's not a container from Docker Hub or something like that? Exactly, Docker Hub, yeah. By default, if you don't specify the full URL, if you're not using a private registry to store your image, right? It's going to look first for Docker Hub, right? So it's going to look for the Drupal image on Docker Hub and download the Drupal image of this version, right? And it's going to expose the port 80 of the container, right? Magno, when you specify one replica to be deployed, it's one pod, right? Yeah, one pod, yeah. All right. Who can manage the container? The number of containers inside of this pod. It's Kubernetes who will scale these all containers inside the pod. Yeah, I can manage here. I can add more containers here, right? So if I added like, okay, Drupal, I can add containers. WordPress, right? So I can add that. Yeah, I can do that as well. So yeah, it doesn't matter. But it will be deployed just one container in the pod or this is Kubernetes skills to define how many containers will be running on the pod, into the pod. So right now I only have one. So it's going to have one container inside the pod, right? If I provide more containers that I want to running, I just need to tell it here, okay, I want more containers running inside that pod and it's gonna run there. Like the difference between running one pod with multiple containers and running multiple pods, which one with one container is basically just the way that they access each other. Like for example, if the two containers are on the same pod, they share the same network namespace, the network PID, yeah, I think, yeah. So they can access each other through local hosts, right? If the container is on another pod, they have to use the IP to access that container, right? So it's basically putting them together. It's usually only if I need them to be kind of a tightly coupled, right? Because okay, I want them always to be together, right? If I have my web app and I want my other container to collect logs from that web app, right? I want them to be together so they can always communicate faster. And also if one of my container dies, they can create another one and keep collecting logs from that application, okay? Right, okay, thank you. No problem. Here I have a service and services are a way to expose my pods to the cluster or to the internet in a kind of easy way, right? What happens like, okay, containers, they are ephemeral, right? They go up and they go down every time. So imagine if I have a website, right? And I'm exposing my pod to the internet and then my pod stops working and I create a new pod for that website and then this new pod has a new IP, right? Then my website is gonna stop working, right? So because if I don't assign the new IP to the DNS and now that stuff, my website is gonna stop working, right? So as a way to a layer on top of the pods, right? I put a service on top of it and then everything that wants to talk to my pods inside my cluster go through my service, right? And my service is just a object there that knows which pod it's connected to and then it can do load balance as well. I can have like three to five pods under my service and it can do the load balancing of that load of those requests to those pods. And the IP of my website will be the IP of the service? Yeah, so if you expose here what we're gonna do here we're gonna create a service. The type of the service is load balancer, right? So we're going to access the Drupal web application through the IP of the load balancer which is the IP of the service. All right. Okay. And the last thing that we're going to create here and we're going to create a secret, right? So I use that for CTFs and we're gonna see that there is a flag here that's base 64 encoded, right? So if you wanna check what does it means it's just a flag for us to access, right? But make sure that keep in mind that I'm creating a secret and I'm creating that on a different namespace, right? I'm creating that on the kubesystem namespace. So that's not the namespace that I'm using here the web app, right? But it's a namespace that already exists on my cluster that's the kubesystem, right? So for example, kubectl got pods and I type namespace kubesystem, right? So I can see which pods are running on this kubesystem namespace. And as I said, this is the main namespace of my cluster is the most important namespace. What is the goal to creating the secret with different namespace? I'm gonna show you later. Yeah, it's just a kind of for CTFs, for challenges, right? I'm putting that secret in a different location to make it harder for people to find, right? But it can be created anywhere in any namespace actually, right? It doesn't matter, okay? Okay, so now we're going to apply all those objects to my cluster and we're gonna apply all of them together. So you have one, two, three, four different, five different objects and we're gonna apply that to my cluster. So going back to the doc file, right? Now we're going to apply that step eight. Step eight of my kubectl. So kubectl apply and the dash f, it tells a file, right? And the file that I want is the cluster objects. And you can see that's gonna be quickly created. It's gonna create the namespace. It's gonna create the row binding, right? The R back, it's going to create the deployment, the service and the secret. Everything is created here. So now I can do kubectl get pods dash n web app, right? And you can see that there is a Drupal deployment like I named here Drupal pod running already on my cluster, right? When the kubectl logs created after the name, after the name of the object, that means that was really created, actually created, or we will just be putting the attcd component of the master node. Sorry, say that again, I didn't understand. All right. When you run the kubectl apply and the logs come out, we've created, all right? It's really, it's actually created. It's mean that actually created, or it's mean that this object will be created is just put into the itcd. Exactly, yeah, yeah. So yeah, you're right. So when it says was created, it was validated through the API server, right? It went through the process of validation and authentication and that information was the format of those objects were validated as well in the YAML file and they were stored in atcd, right? But right away, as Kubernetes is very quick to figure that out, right away, you're going to see it creating the objects, right? So if I, for example, if I do kubectl delete any s, what about, okay. So I'm deleting the namespace web app and everything that was created under it, right? If I type, okay, apply again and right after I type apply, I type get pods, I'm going to see that the status of this pod here, it's not going to be running. It's going to be like container creating, right? Because it's still starting the container, right? So yeah, yeah, you're right. Just to know if it's a real time response or it's near the real time. Yeah, yeah, it's near real time, right? Because it needs to update atcd and store as everything there, make sure that's working. And then the controller is going to check, okay, something has changed on atcd and that's not matching my cluster, right? And then the controller checks that and then the scheduler asks to tell the worker node to create the pod, right? So I'm going to do that real fast here. So, no, it's running, yeah. So it's so fast that sometimes you can't notice, right? Oh yeah, you can check that out later, no problem. Okay, thank you. No problem, okay. So what we want right now is check if our web application is running, right? It might take a little bit of time after you create it, but if you check kubectl.gat.svc, which means get service, right? We create a service to expose our web application here, right? And I'm always typing dash and web app because I just want stuff from the namespace web app, right? And you get, if you do that, you're going to see that there is a service called Drupal SVC as a type load balancer, as we told in the cluster objects here, load balancer, right? And there's the cluster IP and this is external IP, right? So that's the URL of my load balancer. One thing to note here and that we should do, we're going to do right away is that that's exposed to the internet, right? When you create the load balancer, it can be accessible from anyone anywhere basically. Okay, so now if I got, if my cluster objects were created correctly, I get the installation page of Drupal, right? I'm going to install Drupal now because it wasn't installed, they just created the container, okay? So basically here for the installation, we're going to use the default settings, the easiest one, we just want to have Drupal running so that we can exploit it. So choose language, choose whatever language you prefer, the standard installation as well. And here on the database, we're going to use SQLite, ask you our light, let me change this here. Okay, yeah, so SQLite database and it's going to have the default setting is as well. So you don't need to worry about that. And yeah, it's going to install the site. So make sure that everyone is on that page. If you're behind, just let me know and feel free to ask questions on the chat or Discord. For the site name, you can choose any name you want, right? So I'm going to do dodge coin. It can be any email address, don't worry about it. Here also it can be any username. Yeah, let's see. I'm going to generate secure password, doesn't matter here, we're not going to need that username. So don't worry about it, it's just set that up. You can see that the full settings, right? So the site name, email address, it can be anything, the username and password can be anything as well, okay? Okay, so this is my Drupal site, right? I'm just going to change, this is not necessary, but I'm just going to change, no, I'll leave it here. No, don't worry, yeah. I was just going to change the image here for the dodge coin, but no problem. Okay, yeah, let's leave it here. Okay, so now my Drupal site is working, my Drupal web application is working. So one thing, now we're going to start to attack it, right? And we're going to start to attack it, and that's the fun part. I know it took a while for us to get everything set up, but I wanted to go from there and explain everything to you that was happening so that you understand what is a cluster and what everything is doing, right? So for the attack part, as we have on the attack scenario, we're going to need either, I don't think we can use the Cloud9 instance, right? But we can, I don't think we can do that. I'm not sure if it's going to work. We're going to need another instance to attack that this web application, right? That's running inside the cluster. So let me just quickly check if that's going to work on the Cloud9. I don't think it does, but we'll see. Let's see here. One second. If not, then we're going to create a separate instance on AWS that we're going to use as the attacker machine, right? So it just has very small instance for Linux, Amazon Linux, right? So that we can install this, basically this Git, right? Install Ruby to download this exploit, right? And I'm going to make you explain what that means and what the exploit does. Let's see. Yeah, it's not going to work. Load, yeah. Let me, let's go back to the EC2. Let's see if that's going to work. So basically I have here an attacker machine, but basically what we want, right? So we're going to create a new instance. It can be Amazon Linux 2 as well. It can be the T2 micro, free tier, doesn't matter. Here default as well. The storage is eight gigabytes, that's more than enough. Don't worry about it. Note tags and the security group, right? So that's the important part. I don't want to leave my instance exposed to the internet. So I can just type, leave it. I'm not actually going to, oh, I'm going to access it. Yeah, yeah, okay. I can leave it from my IP. So it's only accessible from my IP here and yeah, review on launch. Okay, I can, if I don't have a key pair, I need to create one, right? So I can create a new key pair here. True, download key pair, save and launch instance. Okay, good, good. So yeah, why are you that's loading? If you have any question? I missed the first part where you created the source part. Okay, yeah, I'll go back. Sorry, I think I went too quick because I think we're almost out of time and I want to show more stuff, but yeah, let's see. Instances, launch instance, the Amazon Linux 2 AMI, the default one, the T2 micro, micro. The instance details, we're going to leave default as well. Add storage, leave the eight gigabytes of storage as default. And tags, we don't need to add any tags right now. And the security group, you put my IP, right? It's gonna tag at your own external IP and apply here to the security group so that you can access the instance via SSH. So you need that to access the instance. So far, so good. Do you want me to go back anywhere? Getting there. Okay, so yeah. Sorry, I'm at the part of the network. You were saying that you don't need to face the internet, so which parameters will you change? Yeah. Sorry, you need the instance to access the internet, right? Here, so don't change anything here on the defaults because that's gonna allow your instance to be accessible to the internet. What the storage, right? Eight gigabytes, no tags, that's default. Here, you don't need anything. The security group, that's when you tie that to your own external IP, right? So if you put here my IP, AWS already gets your external IP, right? And you need that to access your instance, right? And then just review and launch and it's gonna ask you to create the keys so that you need that to access via SSH, okay? Sounds good. Let me see where everybody's at, yeah. I think, let me see the chat. Okay, good, good. I have my shell, great. Yeah, you can try to use, how can I see connect? I think it works here. I'm not sure if I've never tried the instance connect or the session manager. So what we're doing is I'm accessing through the SSH client, right? That's what I'm doing. I never used this one as well, but you can try, it's up to you. You just, we just need to send commands to that instance. Let's see if that SSH's instance connect is going to work. Let me see, I think that's gonna work. Okay, so what we're doing is through SSH, like, okay, after downloading the key pair, right? You need to chmod the key that you attached to you created for the instance, and then, yeah, just access that using that key. Is everyone, is everyone here? You have your Drupal shell? Oh, nice. Nice, that's cool, man. Okay, we're getting there, almost there. So yeah, okay, after, what we want to do is the attack setup, right, on the Google doc, right? So let me stop share here. Let me, let me go to, let me share my terminal so that I can show you what I'm doing. Okay. So here, accessing the, is it there? So basically what I'm doing, I'm SSHing through to my, to my attacker instance, right? And I'm telling it to use the key pair with dash I, right? And using the EC2 user IP, which is the default one, basically that, right? So that's why it's important to allow to only my IP to access that instance so that nobody else, if they get the key pair can access it. Okay. So you should see, receive that information here once you connected to the instance. Let's see if I can use terminal. Is that okay? Can you guys see my terminal, okay? Yeah. Okay, good. So basically what I'm doing, I'm gonna do here, I just create a folder in NorthSack, but it's not mandatory. So basically what I'm gonna do is, probably your instances doesn't, don't have a git or other stuff. So it's a good thing to do sudo yum update or upgrade. Okay, nothing because I've done that already. So sudo yum style git, because we're gonna use git. Mine is already installed. Yes, exactly. That's good. Yeah. Basically, yeah. Once installed git, git clone that project. That project, that GitHub project there is where our exploit is. So the Drupal get and vulnerability. So the Drupal version 8.5 is vulnerable to an RCE or remote command execution vulnerability that we're using that code from GitHub to do that, to exploit that vulnerability for us. So the only thing that you need to install before executing the exploit is a dependency that these exploit has. That's the gem, the ruby gem, right? The sudo gem style highlight. Exactly. Thank you, JJ. Okay. So sudo gem style highlight. If you don't have that yet. Okay. Is everybody on this page now? I wanna make sure that everybody is now on this, you're on your attacker instance and installed git and you downloaded the exploit, the Drupal get on to exploit from that GitHub URL that I, that's on our Google Docs. Let me share. Okay. Is everyone there or is anyone behind actually who needs to, needs help or needs, needs some more time to access their EC2 instance? I'm getting the error. There is no gem command, not command so. So yeah, sudo gem style ruby. Right. Yeah, sudo, if you don't have ruby as well, sudo gem style ruby and it's gonna have the gem inside ruby package. Okay. Books. Good, good. Okay, good. So now we're going to exploit, right? So of course, usually we're gonna, we want to do a reconnaissance of the environment, understand what the applications are running and the versions, right? But we don't have time here. So for the purpose of this workshop, we're going to grab the load balancer URL, right? From that cluster that we created, the dodge coin or Drupal website. And we're going to use that to exploit the Drupal, right? So basically here, I go inside Drupal-Gadon2. And there is this file, the Drupal-Gadon2.rb. So basically I type Drupal-Gadon2.rb and I type the host and port of my target, right? Which is basically the load balancer URL because it's running on port 80. Can you guys see my terminal here, fine? Can everyone see my terminal? Yes, we do. Okay, good. Okay, so yeah, once I run that exploit, as you can see here, you're gonna see some information. It's testing, it's finding the vulnerability. It's exploiting that. And at the end, we get a shell, right? We get a shell inside the container. So we just exploited a Drupal web application running on a Kubernetes cluster. And how do I know that that's a shell that's I'm running inside the cluster, right? So I can do some commands here to do some basic information gathering, right? I can do the ID. As you can see, I'm running as www.dashdata. You can do like your name, dash A, right? So I can see, okay, it's a Linux, a Drupal depth. Okay, what is that? Amazon Linux, right? Amazon 2, right? I can do some like, okay, ps dash af. And yeah, I see it's not a lot of process running there, right? So that's already an indication, a red flag that this is a container or a cluster, a Kubernetes cluster or a pod, right? So there's other commands that we can do there. But what we want to do just to understand, okay, what we're doing, why it's important to protect your web applications as well, right? It doesn't matter if your cluster is secure, if your cloud environment is secure, but your web application is not up to date. It's not secure properly, right? If it has a critical vulnerability such as this one, it's a remote command execution and it's from, it's not that old, it's from 2018, if I recall correctly, right? So that's a problem there. So there is a tool here that we can use on the inspect the Kubernetes environment, right? Which is called MI Container, right? It's also available on GitHub. You can easily download that and execute it, right? So it's gonna tell if you are contained it, right? So it's the name of the tool is already a question, am I contained it or not, right? Okay, so let's see if that's going to work. So I'm gonna copy that command here that's on the inspect the Kubernetes environment and I'm gonna download the tool and execute that inside my pod, right? The shell that I have here off my pod. Let's see if that's going to work. Okay, so if I run that, I can see that the output of that command tells me that I'm running the container runtime is Q, right? Whoa, okay, this is already a good indication that this is a Kubernetes cluster, right? It shows here it has namespaces, the PIDs equals true, right? It doesn't have the fbarmor profile, right? I can see the capabilities they're enabled for that container as well. Seccomp is disabled, right? And the syscalls there are blocked as well. So this is good information, right? This is already good information from the attacker. And right now I haven't done anything. I'm just getting to know my environment and understand what's inside that environment, that machine that I got a shell on, right? To understand what it is, what does it, like, okay, what can I do here, right? Another command that I can do to understand more, okay, make sure that this is a Kubernetes environment, right? It's check the environment variables, right? So when the pods are created, they have some environment variables to access the cluster, right? So these environments variables here, this command and pipegrap-icube, it's gonna show me some environment variables of my Kubernetes cluster, right? So I can see that as well. I can see the Kubernetes port, I can see the service, the IP, right? Everything here is telling me that this is a Kubernetes cluster, right? Other things that I can do inside that and we're just inside one pod of the cluster, right? We're not doing anything fancy here, but if those environment variables are set, right, and I have a pod, then maybe I can reach the Kubernetes API itself through the pod, right? So this curl command here, it's gonna check the Kubernetes version of that cluster from the pod accessing the API server. No, let's see here, okay, see? Now I have the version, right? So with curl requests, I have the version. I can see that the major version is one, the minor is 19, right? So 19.8, there's the go version, the platform, everything. So this is great information for an attacker to understand the environment, understand the version of Kubernetes to check if there is any other vulnerabilities related to that version as well, right? So this is very important. Okay, now, okay, but now we're just doing some recon, right? We're not doing anything fancy yet, right? But if I try to get, for example, those API server details, okay, yes, yes. We'll always be available for the container, yes, to the container, yes, yes. I can even, and we're gonna see later, I can even reach the instance metadata API of my node, of my worker node from the pod itself. And we're gonna see that, that's very fun. Okay, so yeah, okay, this is just some recon information that's nothing fancy here, right? But there's a problem, right? Kubernetes stores their service account that stored by default on the pod, I can access from here, right? And it's always on the same location, right? So for example, let me move that. Okay, so if I do this, I'm gonna check. So inside the pod, there is a directory inside the pod that's gonna show the service account. Oops, I deleted that. So yeah, this directory has the certificate, has the namespace and has the token, right? So if I do like slash namespace, but I do like cat, okay. So I see that this pod is on the namespace web app, right? And there is a token and a certificate file there as well. So I can do that, CA.CRT and it's gonna show me my certificate, right? And that's sensitive information from an attacker and also the token. Okay, so this is a JotToken, right? This from here to there, right? It's a JotToken that gives access from the pod to the Kubernetes API server. Sorry, what is the cert used to secure? It's, I think it's the communication between the pod and actually the pod and the Kube API server, it's the communication between them. So let me show here what we're gonna do. Okay, now let's see, yeah, let's jump this. Yeah, let's jump this. Okay, so that it doesn't become harder to type. I'm gonna get that information from the token and I'm gonna put that into an environment variable, right? I have environment variable token and I have an environment variable for namespace. That's what I want here. So let's do that first, namespace, NS and token. I'll give you guys some, I've won some time to do that. And let me know if you have any questions in the meantime. All good? Okay, so now I'm going to do another request to the API, API server without any authentication or authorization, right? So let's see what's going to happen. Oh, I got failure, right? So I got the forbidden user anonymous cannot get path API. So that's very similar to the error that we got when we try to access the endpoint, the API server endpoint API from the browser, right? That's very similar to that issue. So, okay, right, path, I also get an error. Only version is available, right? Because it's available to everyone for like health stats and all that kind of thing. Okay, so I can't access API if I don't have authorization, right? But what happens is that there is a token inside the pod and we can use that token to impersonate the pod and access that information, right? So for example, if I use, instead of just sending that call request, if I send that, so now I'm adding a header here and as the header, I'm adding the authorization bearer and the token, which is that environment variable that we saved earlier as the job token from the pod, sorry. All good here? So yeah, if I access that, oops, not working. Let's see, okay, it wasn't actually set to variables. Yeah, okay, good, good. So, yeah, okay, so instead of doing that, I'm gonna copy the token to make sure that it works. Yeah, so here, going back, if it wasn't saved, I'm gonna just paste the token here instead. Yeah, okay, now it worked, right? So now, thank you, yeah, so now it worked. As I said, the token here, I made a request to the API and now it was, my current request was accepted and it sent me information about the API, right? So there are other things that I can do here. I can just play around and keep always sending that same token to connect to the Kubernetes API server and depending on the permissions of that token, I can do pretty much a bunch of stuff here on this cluster. Let me see, other things that I can do as I mentioned, okay, I can still, I can even access the instance metadata from that instance that my pod is running. I don't know any tools like Pakoo for Kubernetes, like for the, there is one that's called CubeHunter that we can play around if there is still time, but I'm not sure if we're gonna have time today, which is the CubeHunter from Aqua Security that's used to do a recon of the cluster and find any vulnerabilities. Once you created the attack machine, you, if you download the SSH keys, the dot pen file, right, you can access through SSH. I can show you later, let me just show here the instance metadata and then I'll exit, I'll exit the pod and the instance and I'll show all the steps, one second. Okay, let me just, so the curl command here that I'm doing, right? So there's instance metadata, which is a specific IP for AWS and there is a similar thing for GCP and Azure where you get information about that instance, right? So for example here, I'm doing a call request from my pod itself. I can get the instance metadata, the latest metadata here, right? And I can get the IAM info, remember that we gave administrator access to the EC2, to the Cloud9 EC2. Imagine if this worker node had also administrator access, right? So using that, I would be able to get the API keys from AWS for the IAM role for that instance and use that to do other things, right? So that I could get the keys and put that into like something like a tool called Pacu, which does automate a bunch of exploitation inside AWS instances and I can go from there, right? So attackers could use that and they've been using that to either, either once they compromise your cluster, I'll explain the IP reason. Once they compromise your cluster, right? They can either deploy new containers to start mining cryptocurrencies or they can use your own containers that are already running to mine cryptocurrencies, usually Monero cryptocurrency because it's kind of harder to track that. But they can also use get the IAM API keys, right? The metadata API keys there from your node and use depending on the permissions, right? They can access your cloud environment and deploy new instances outside of cluster, right? Can be even in different regions. It can be even instances that are very big, right? And very expensive so that they can mine more cryptocurrencies, right? And very fast. DIP 1.169.264.169.64. That's the default IP for AWS where you can access the instance metadata. So that's the fault IP, it's always this IP for AWS. I think it's the same for Google or Azure and one of those two, they have a different one. It's either IP or a URL. So I don't remember, but yeah, right? So for example, we've deployed a similar environment like this as a honeypot for our research, right? And in last than 24 hours, last than 24 hours the attackers had compromised the cluster that we created and they broke out of the cluster and they created like large instances to mine Monero cryptocurrencies in last than 24 hours. So that's why it's really important for you to be very careful when creating your cluster and deploying that and doing any kind of permissions and misconfiguration on your cluster. So yeah, luckily we detected because it was a honeypot so we were monitoring that very close but otherwise the AWS deal would be very, very big, right? So we wouldn't like that. Let's see where we are right now. We have about 20 minutes. There is other stuff that you can do here and I left some comments for you to try, right? So you can try to break out of the pod itself and there is a command there on the docs that you can use to deploy a new pod which has some privileged pod with some permission specifically to do that and basically access the worker node as if you were breaking out of the container, right? So it's not exactly a container breakout but yeah, it is because you're deploying a privileged pod that can access basically everything on the worker node that your pod's running. So the kubectl run. So yeah, we can also install kubectl on the pod itself, right? So let's do that before we move on. Okay, let me go back. So we have here so I'm just defining here tape as a path when you get. So we can download the kubectl. This is gonna download an outdated version, but it's fine. So here I'm just downloading kubectl, the binary that we installed on the cloud nine instance, right, to control our cluster. I'm going to download that inside the pod that I compromise it, right? And I can do that. Of course, because I don't have a full shell, I can't see everything. And of course, I can update that to a better shell but we're not doing that right now. We don't have a lot of time. So now that I have kubectl, there's one command from kubectl that tells me what permissions do I have, right? So this command here, kubectl off can I and dash dash list is going to list which permissions I have from that pod itself to do with kubectl. Oh, let's see, not fun. I'm not sure if it's going to work. Let's see, might sound working. Okay, good. See if that works. Okay, good. Awesome, thank you. Now why it's so working. So here I can see that I have a lot of permissions from that pod, right? Of course I have a lot of permissions because as I mentioned in the beginning, when we created that cluster on purpose, we gave that namespace admin, cluster admin permissions, right? So cluster admin is basically the same as root on the cluster, right? So here it's going to tell me, okay, all the resources that I have on my cluster or that I have access to, and it has the verbs, right? So the verbs is what I can do, create, delete, get, list, patch, update, right? So I have a bunch of information here that I can do a lot of things from that pod itself. And I can ask this command cube CTL off, can I create pods? I can ask that as well. So, okay, can I create pods, right? Yes, right? So it's just reply, okay, can I create pods? Yes, I can create pods. I can get secrets as well, right? So if I want to grab secrets and I hope you remember that we created one secret once we created the cluster, right? Okay, get secrets from the namespace web app, right? It's showing me the service account token off web app. But if you have one he remembers, we created a secret on another namespace. Do you remember which namespace? Web app, I think? No, not web app. Cube system, yeah, not system, but cube system, right? So which is the main namespace for the Kubernetes components? Okay, a bunch of secrets here. That's great, a lot of information. Oh no, cube system is a standard namespace. It applies for all Kubernetes versions pretty much, I think. Maybe not mini cube or the other ones that are installed on your machine, local machine, but it's a standard for Kubernetes on unmanaged and managed versions. So you can see here that there is a flag or secret here named CTF, type of opaque, right? I can get, describe secrets, CTF, let's see. Okay, I have some information here. Okay, show, let me get, get secret CTF, yes. What is the other command? Flag, it shows the flag 27 bytes. Okay, get, see, secret, no. Did I get, describe, it's not describe. There's one command here where I can see, I forgot the which one now. There you can list, let me see, one second. Let me get that, one second. So yeah, basically that secret is just base64 encoded, right? Which I just wanna show that's only base64 and then we can easily figure out what that means, right? Get secret, yeah. If we can't do that through here, we can do through the API as well. Trace, let me try to do this through the API. API, give system, yeah, nothing in the token. Okay, let me grab that. I just need to update the token. Yeah, that's the one, that's what I want. Getting tired at the end, I'm forgetting things. Sorry guys, sorry everyone. No, I'm not gonna do that. Token's not there. Wait, wait, let's do this smart. Okay, I have the token here. Yes, good, good, that works too. Let me just, let's see if that works from an Ops LAN. So, okay, let's see if that works. Otherwise I'll do through the API too. You can do it the way. Um, there, yeah, there's a specific command. It didn't work for me here. See, do I need that? Not working. Yeah, maybe that one, the needcube system namespace. Okay, good, good. How the secret name to CT, yeah, sorry. Oh yeah, my secret. I was just, sorry about that, I didn't read. Okay, good, yes, exactly. That's what we want, thank you. That's what we want. Yeah, okay. So yeah, okay, good. So we got the secret and we already decoded with this command here already. And we see that the secret is a flag, right? Flag, okay, all your clusters belong to me, right? So that's the secret. And it's very important to specify the cube system namespace there. So that because if it's on a different namespace, right? You need to type, otherwise it's gonna look for the default namespace. It's not gonna find anything. It's gonna say, okay, I didn't find that, right? So you need to be aware of that as well. Okay, what else? Talked about secrets, good, good, good. Exactly, right? So that's the problem with secrets in Kubernetes, right? Because by default, it's not very secret, right? It's just basic C4 encoded and you either need to encrypt that secret with a specific object that's called encryption configuration object that you need to apply that to your LCD to tell what you want to encrypt. And yeah, you can do that. Or you can use a third-party secret solution either from your cloud provider, like a secret store manager from AWS or you can use like something like HashiCorp Vault, right? Now, nowadays, they already have a SaaS version of HashiCorp Vault where you deploy it on their own, on the HashiCorp cloud and you just need to connect that to your cluster, right? So yeah, be very careful when using secrets on Kubernetes so that if you have any, you know what's the issue, right? Okay, yeah, so with Qubectl, we could do this command here at the bottom that I break out of the container. The problem is that sometimes because we're on a EKS cluster, right? There are some protections around this, right? But it may work, but it sometimes can break your cluster, it can break your container, right? So I'm gonna leave that to the very end if that's we're just having at least 10 minutes so that we can try to break out of the cluster later. But yeah, because it's EKS, if it were an unmanaged cluster, then yeah, it shouldn't be a problem, but with managed clusters, they have some protections around, right? Because technically, you shouldn't access the control plate, right? And breaking out of that cluster might may allow you to access the worker node which can give you some permissions later to access the control plate. So be very careful with that. Yes, exactly. So one of the main issues here and thank you for noticing that is because the permissions that we gave to the namespace were admin, cluster admin permissions, right? Of course they don't need to be that permissive but if you give anything that's too much, anything that can give me my namespace to find secrets from another namespace or create pods, those are very dangerous, right? So you need to be very careful with our back, right? Our back is the row-based access control which basically you have four different objects on our back, right? You have the row, right? Which is the profile or the permissions, the set of permissions that you want to give to a specific user or service account. And you have the row binding, right? Which is the object that binds that row, that set of permissions to a user, right? Let me show the slides here real quick to talk about our back. One second. Okay, share screen. And now we're almost out of time. It should be, at least the attack part we got pretty much out of the way. Okay, let me jump that and I'm not gonna talk about it. Okay, here, our back, here. So our back, right? As I said, you create a row, you can, you have four objects, right? Row and row binding, cluster row and cluster row binding, right? The row is the set of permissions that you want to apply to a specific user or service accounts. And to connect those, the users and the row, you, can you all see my slides just to make sure? Good, okay. Yes, yes. Okay, good, good, just wanna make sure. Okay, so yeah, so to connect the row and the users, right, to use that object, the row binding object that's gonna tell, okay, this group of users, they have this group of permissions, right? On the row. The row means it's a set of permissions on a namespace level, right? So the row only applies to a specific namespace, but there's other object called the cluster row where those permissions apply on a cluster level, right? And for that you'll have, it's basically the same thing, but it's that it's more broad, right? It's more permissive because apply to all the namespaces on the cluster, right? And the cluster row binding here, it's basically the same thing as the row binding, but it's just that's used for the cluster row and applies everything, all the permissions for those users from that cluster row to all the namespaces on the cluster, right? So basically here, what we did on the pod itself, if we go back to the cloud nine instance, let's see, where is it? I guess this one. Yeah, so if we go back to the cloud nine instance, right? This is the problem here, right? This is, look, we have a cluster row binding and we have a cluster row with admin permissions, right? So we have, this is the problem of most of the problem that was caused on this attack was because of this row, right? So RBAC and RBAC is something that most web applications have already, right? So if you go to any kind of web application, they have like the regular user, they have manager, they have admin or root or something like that, right? This is RBAC, right? You have the profiles and you put the users on those profiles and they have the permissions from that profile, right? Basically that's how Kubernetes manages authorization, right? So you need to get that properly and something to, the first thing that we would do to fix this cluster is what was going to be removed this, right? So if we had time to go to the defensive part to fix the cluster itself would be, okay, remove this and apply that to the cluster again and remove that admin. So that's going to create a new default account that doesn't have for that namespace, doesn't have permissions to do everything, right? So it's just as a way for us to do the workshop and for me to show, okay, this is an RCE. It doesn't mean that every cluster that you're gonna find, it's going, every pod that you get a shell on, right? If you're authorized to do that, right? That's gonna, you're gonna be able to create pods and to do everything, right? It's not gonna be that, but usually clusters that are running for a long time that are outdated, right? Can be running for one year, two years or more. They usually don't have that because this thing of protecting, creating a separate service account for each pod and the namespace, this was done recently on like the recent updates of the recent versions of Kubernetes, right? It wasn't that a few versions ago, right? So you need to be aware of that. The new updates, the new versions, yeah, it's protected. It doesn't mean that Kubernetes is secured by default. It's not, right? There are other ways to get access to the cluster as well. This is just a way for a simple way for us to demonstrate the workshop and demonstrate the attack on the web application. I think I'm gonna leave everyone here. We still have a few minutes and I'm open to questions. We would have more time. If we would have more time, we could do some defensive stuff, right? And I'm gonna share you the slides, but yeah, I'm open to, if you wanna try more stuff on your cluster, if you wanna try to secure it, we can talk offline. I can send you some links and sites for you to do that. But basically the first thing that we were going to do was remove this and also try to protect our API endpoint. So that's not exposed to the internet, right? I hope you guys, everyone enjoyed this workshop session. I'm sorry that we couldn't cover everything. I know it's a lot of stuff and it was important to get the beginning, the setup of the cluster properly for everyone. But I hope you enjoy and yeah, thank you. Exactly, yeah. I'm gonna show, yeah, before we go, remember to delete everything from the Amazon account you created because otherwise you're gonna pay money to AWS, right? So for the cluster itself, if you wanna delete the cluster, right? It's basically here, EKS, CTL. It's the same command that we did to create the cluster. You can use that. Can you guys see my screen? Can you see that? Yeah, okay. Yes, we got it. So yeah, basically the same command is going to delete your cluster. If that doesn't work, if that doesn't delete it properly, go check on the cloud formation, right? Because all the thing that EKS, CTL does, it's using cloud formation, right? So it's all in the background, it's using cloud formation here, right? So you have some stuff here, right? Be careful if you don't wanna delete your cloud nine instance, this is the cloud formation for your cloud nine. I think someone did that before, but go here and delete one by one, right? Start with the node groups first because the node groups are inside the cluster and then after you delete the node groups then you go delete the cluster. And if you get Ehors because it didn't delete, try deleting again and it should work, right? So for example, okay, delete here. This one didn't work, so I'm deleting as well. Make sure that you delete everything here. And also on the EC2, right? So go to the EC2. Yeah, if you don't wanna use it, if you don't wanna play around with it anymore, right? Make sure that you delete and terminate all those instances because even if you just shut down, you're still going to pay some money because of the storage, right? The hard drives of those instances. So if you don't wanna pay anything, make sure anything else more, I don't know if there's already some cost there, but make sure that you terminate all those instances as well. So go one by one and go instance state and terminate instance, right? Okay, yeah, yeah, don't forget to do that and don't leave it for tomorrow or anything because then you're gonna forget and then you're gonna pay your AWS bills. And I don't think you want that. Anything else? Stop sharing. I'm gonna share the slides, the PDF. I'm gonna generate the PDF and send on the Discord channel. If you want to continue doing other stuff, there is some stuff on the docs as well that you can play around. As I said, you can try to break out of the cluster and that's one fun thing to do as well. There's other stuff, but yeah, I think that pretty much covers the basics of attacking a Kubernetes cluster by hand, right? There are some tools that you can use like KubeHunter to find vulnerabilities for you, but I don't think they're very advanced. They're basically just running some curl commands that we can do ourselves, right? They're just doing a job for you, but it's important that we understand what commands we need to run and which API endpoints we need to access to get that information. Any other questions? So for the deletion, you said that you should delete the nodes and then the... The node groups, yeah, the node groups on the cloud formation first and then you delete the cluster. So there is a cloud formation for the managed node groups for each cluster. And after that, then you delete the cluster itself. Otherwise, if you do the other way around, it's gonna complain, oh, you have a node group attached to the cluster, you can't delete the cluster yet, right? So it's just that. And yeah, if you give it some errors, just try to delete again and should delete everything but make sure that you don't leave anything, anything on your cloud formation and if this is a brand new account, right? You shouldn't have anything on your EC2 instances and anything on your cloud formation templates there as well. So that is so you don't pay any money. Yeah, I'm open for questions. If you have, I know we started a little bit late, but I can be, I'll stay on Discord if you have any other questions thereafter, we leave the Zoom chat and I'm open to explain and talk to everyone here later as well. If you need more information about attacking the cluster or defending the cluster itself. Yes, I can. The resources for defending, I didn't put on the Google doc because I was going to do that. I didn't have time, so I was going to do that with everyone here just on trying, okay, let's install OPA, let's install Falco, right? Yeah, thank you, everything. So I can share the slides on the PDF and it's gonna tell you which tools you should use to defend your cluster, like Open Policy Agent, which is a admission controller that checks whatever goes to your cluster. Okay, it checks if it's not running with privilege permissions and all that kind of stuff. It's like a policy gate or a policy as code for your cluster, right? It's Falco that does the runtime protection as well. So if something is fancy, like if I compromise the pod like that that we did with Drupal and Falco was running, Falco would detect that. Falco has, it's a rule engine for runtime protection on Kubernetes. And I'm gonna share the PDF on the Discord channel. So make sure that you join the Discord channel if you haven't already. And I'm gonna share the PDF there. So it has a bunch of more stuff and explanation. It was just a backup. If something went wrong and I need to explain on the slides, but the idea was more to do more hands-on with everyone here. So I hope you guys, everyone enjoyed. And yeah, thank you for coming. Good, yeah. If you don't have any more questions and I think we can end the workshop, right? You okay, Mona? All right, thanks. Yeah, thank you everyone. Enjoy the rest of NorthSek. Go to some other workshops and hope to see you at the CTF. Sure, awesome. That's starting tonight. Okay, yeah. Thank you very much. It's a great thank you. Thank you guys. Thank you. Thank you very nice. Thank you. Right.