 My name is Pushkar. I work as a sub project maintainer in Kubernetes security. The sub projects name is tooling. So anything related to security that we do for Kubernetes. Are we good on audio? I hear an echo. All right. So anything that we can do by writing code or helping people write code that is going to improve security for open source Kubernetes is what my sub project lead responsibilities are. And then the other things are I'll end up helping out other sub projects, et cetera. We will have a couple of folks who will help you out if you get stuck, especially Tabby, who is our chair for security who has graciously come out to help out. So bit more about me. If you saw the photo on the previous slide, I'm in the SFB area. One of my favorite places to go for a car ride is Mendocino County, which is what the sweatshirt is from. I really love the coast. Like I said, apart from security, I'm also tech lead for CNCF tax security. Great question to ask what's the difference between tax security security during the breaks or after the session. I am also recently associate member for Kubernetes security response committee. So anytime you get a CV notification, we are one of the people who send it out to the rest of the community. I'm formerly a staff security engineer at VMware Tanzu. Like I said, I love the area. Also, I love many things that start with C. So cycling chai, Indian curries, camera and cricket. If anybody follows, we have a world cup going on and you can find more about me on the link on the slide. So we will divide it into small blocks so that we don't get overwhelmed or get really tired at the end of the session. This is the intro section. We'll have 20 minutes for the first task, including the five minute break, 25 minutes for the second one with five minute break included, and then 30 minutes with five minute break included. Eventually at the end, we will wrap up and I'll share some links for further reading if you want to continue to explore more and more about this. So just by raising your hand, how many of you are on macOS right now if you're going to follow? Okay, perfect. I have tested this on macOS. So most of you should hopefully be able to follow along. How many of you are on M1 macOS? Okay. All right, and Intel. Okay, cool. So there is in the install prerequisites part, there is a small change for people will have to do for once on M1. Basically, instead of Darwin AMD 64 binaries, you will be ensuring Darwin ARM 64 binaries. So we'll talk about talk more about that. Anyone on Windows or Linux computers or virtual box? Okay, cool. So we have similar setup for Linux and Windows binaries as well. I haven't been able to test it, but even if it breaks or doesn't work for either of you, all the scripts that we will use will be available publicly after this session forever. You'll also have have access to the recording of this session. So you'll be able to follow along the demo videos even later. So don't be afraid if something doesn't work. How many of you were able to look at the session description and install the prerequisites mentioned in the link? Okay, perfect. If you haven't installed, that's also fine. We'll spend some time to install it. And only thing I would suggest is if possible use Docker desktop on Mac Windows on Linux environments really like Docker binaries are also fine. But the main important thing is please don't run any of the scripts on production servers or shared computers because this is a lab. Things could go wrong and we don't want other people to suffer because we were trying to learn something good. So as long as you're running this on your personal laptop with Docker desktop, we haven't added advantage of it. Everything running in a virtual machine that Docker desktop has included by default. So things would be much safer if you're running Docker desktop. And again, don't run this in production, even for folks watching online later on or watching live right now. So now back to the actual content. If you are, how many of you are security people and how many of you are like ops admin people? Security and ops admin. All right, some some of you are both, which is fine. I'm also both. Many times when people ask how do you want the security to be for your system, everyone says secured by design. It should be built in versus bolted on. It should be transparent and it should just work. And I think the one thing people whisper, which is, can it be cheap? So because nobody wants to spend lots and lots of money securing things. So this is exactly the part that we will do today. We are not going to ask you to install anything paid. Everything we will do is open source part of Kubernetes community. Most of the things we will do are actually part of existing Kubernetes releases. So without really installing anything apart from what's available open source, we will try to secure as much as we can for all the clusters that we work on. Anyone who is a complete new BA on Kubernetes or has just been playing around for like a couple of weeks or so. Okay, that's that's totally fine. Don't feel bad in case something doesn't work or ask questions if I am missing something in terms of giving context. And you can always continue to try this later on. And anyone who is familiar with Kubernetes, if I repeat something that you already know, just be a little bit patient and I'll quickly move on to some more advanced stuff. So we'll focus, like I said on built in Kubernetes security features. All of this is mostly work of Kubernetes community. Since at least since I've been here in the last couple of years, we have tried to do a lot of things in Kubernetes where the security can be part of the project versus being reliant on other things that we need to do. So we'll divide, like I said in three tasks. First task is how do I verify sign container images? How many of you know we have been signing release images of Kubernetes in the last couple of releases or so? Okay, that's good. So now you know for folks who don't know. And the idea behind signing is essentially you want to trust whatever is being sent by Kubernetes release. And that's the reason why we're signing and all the focus on secure supply chain doesn't hurt doing all of these things. Second one is how do we enforce baseline port security standard? So anybody using PSPs or security policies, anybody tried port security admission? Yeah, so we're actually going to try in doing using the port security admission and enforce the existing baseline port security standard. We are also going to work on second filtering. So all the container runtimes like container D we have another helper Nadir who is going to help you out in case you get stuck. All the container runtimes like container D cryo come up with their default runtime second profile. What we are going to do is let Kubernetes actually use that because by default it doesn't know that I can actually use the runtime profile. So we'll see how that works. Some prerequisite knowledge. Knowing basics of curl file redirects file editors, I'll try to go through it in a way that it is clear, but ask questions if needed during the break, or even if you have a small question, we can do it during the session with the time we have. I may not be able to answer all the questions or like unblock everybody in all the tasks. So it's okay if you are halfway there and we have to move on. We can always continue later follow along for the rest of the task. You can run the scripts later on because all of all of the things that I'll do here will be available for you later. Kubernetes basics, water names, spaces, spots, images, those kind of things would be great to know. Pre-requisite tooling. So for folks who installed it, great for folks who want to install it. I would say at least have Docker installed. There is an installed prerequisite script that I will show that will install most of the other things, but see if you can install Docker by yourself right now. Use the five minute break to relax. Also ask questions and I'll be hanging around after the session as well. We have lunch break. So it's really up to us how much we want to talk, but happy to answer questions later. So now let's get started. So this was the link in the session description. If you have sked.com access or the Kubernetes schedule access, you should in the session description of this session, you will find this link. Just run this command, get clone the URL of the repo and you'll be able to download it with all the scripts that we are going to use today. So now let's go back to doing the demos. I'm going to quickly mirror my display so I can see what you're seeing. And let's go here. Is the font small big? Is this better? All right, cool. So like I said, this is the GitHub repo. Everybody been able to find the URL so far. All right. Okay, cool. And this is where you can basically also find the URL and do a git clone. Now going to install prerequisites. So remember for M1 folks with macOS in this section, basically anywhere you see AMD 64 and AMD 64 here, just change it with arm 64. And if you run the script, it should work. So going a bit deeper into the script, basically what it's doing is setting up this so that we know when something fails in bash. Next one is setting up paths so that when we install few things, we can use it as a command line inter prompt directly. This is for different operating systems. So when you run the script dot slash install prerequisite, just add an argument for your operating system. So windows Darwin or Linux, and then it will pick up the binaries that you need. So let's pick Darwin as majority of you are on Mac. What it is essentially doing is downloading the cube CTL, which is the client side interface for interacting with Kubernetes API. It's installing kind. So for folks who are not familiar with kind kind distance for Kubernetes and Docker. And what essentially it's doing is I have a cluster. I want to play around with it, but creating cluster is hard. Maintaining it is even harder sometimes. So this creates test clusters using Docker on your laptop, and you can essentially use it as a test environment. So we will use that quite heavily, especially in the second and the third task. If you're familiar with kind, most likely the pre the whole tutorial, especially the second and third would make a lot more sense. But I am hoping that at the end of the tutorial for folks who haven't been introduced to kind, you'll actually fall in love with it and use it after the session when you're day to day jobs. Don't run kind to run production clusters. Use it for test environment and development only. And so that's pretty much it. So I've I've installed this already to make sure the Wi-Fi doesn't break. I've been informed that the Wi-Fi should be pretty good specifically in this room. So hopefully when you run this script, it should work for all of you. But start running it. The basic idea of running this would be pretty simple. So I have this installed here. Is the font okay for the people at the back? Yes. Okay, good. So I will I won't run it. But basically it will look something like this. And if you click enter, you it will install cosine. You'll see a lot of things getting installed for cosine because it's using go install. And then eventually it will start installing kind and cube sick deal, which will be much quicker. So I take take some time. Let it run while you're looking at the slides next. Let the script run. And by the time we are ready for the first task and actually implementing things, hopefully it will be installed by then. So everybody start installing this and let it run. We'll switch back to slides until it is installing any questions. Yeah, good. Okay. Yes, that is a good idea. So one thing I would also recommend in true open source fashion. The repo is public and any things that you find that could be improved in the repository, like adding Xcode as a dependency, please create PRs. I would love to hear from you. I would love to see all the work that we are doing here be useful for everyone later. So add PRs, maybe now, maybe later. And one more thing I wanted to show is for now as well as for later, if you are stuck and it's hard to like share what is wrong in your terminal, what error it is exactly face to face. One thing that I've set up here is if you click new issue, it will give you a issue template. You can still, okay, you can still see this. Open issue template. Help me debug. And if you click on this, you will be able to basically write all the things that are going wrong on your machine right now. And you can put it down with the description. How did you can you reproduce? And if you have any screenshots, you can put it, but really it's optional logs would be great and which environment you're running. So I'll try to see if I can follow along on future issues as well. If you when you try this at home, this is definitely not something I'm being sponsored for. So I will do my best in my personal capacity to answer questions later on. But even during the session, if you want to share something and probably many of you are facing same issues, creating this kind of issue would really help. So please use this as much as you want. Just to repeat, go to issues tab, click new issues and get started and then add any errors that you have. We'll take a look at all of the new issues we create during the break between the two tasks. All right. So we talked about install prerequisites. We talked about how to share how to share when something is going wrong. So now let's go back to the slides. All right. So the first task. All right. Essentially, for folks who are not familiar with cryptography and what how to use it. I'll just give a brief idea as an analogy. If you have a passport, it is basically given by an authority that everybody trusts. And because everybody trusts it authority, when somebody shows like this is me and see the name and the face matching, you trust that the person who is showing this is actually who they are. So that's the main idea of signing and verification. And the stamps for what it's worth are like attestation to your images. So signing always happens where the makers only have the private key. If the private key in a public private key pair is compromised. Essentially, there is no value in anything that's signed by that compromised private key. The verification can be done by a public key, which as the name suggests can be public. And that can be used to verify whether somebody who signed it was actually the holder of that private key. It tells you where the artifact came came from. So in case of Kubernetes release images, right? It will let you know that Kubernetes release actually signed these images. And when you verify it, you'll see an email address that belongs to them. And because it is very fine using the public key, that's the other side of the pair of the private key owned by release team. You know, it's actually signed by them. The next one is how do we actually get all the images in a release? So how many of you have seen the software bill of materials, which is called S bomb of a Kubernetes release? Okay, not a lot. Okay. So it's quite big and it's huge as you can imagine. Kubernetes pretty much depends on many, many things. So we have a way to access as bomb of any specific release on this URL as bomb dot kids.io. And what I'm doing here is we have a constant, which keeps changing based on a new release when it comes up. And we store it in this specific URL, SCTPS deal dot kids.io release latest.txt. If you want the latest stable release, you just change latest with stable.txt. So latest would have alpha releases, beta releases, stable will only have that are officially released like 125.2. And then slash release is when you will know that now I'm getting the S bomb of this specific release. So the dollar curl will actually look like, let's say 126. something. And you will get the S bomb of that specific release. The next one is package name registry dot kids.io. So kids by the way is abbreviation for Kubernetes. We will see a lot of it. The registry dot kids.io is the registry where all the Kubernetes release images are stored. And what we are looking for in the S bomb is get me all the images that are part of the release. And then essentially we do some bash magic and get a list of all the images in a file called images.txt. So after that we take it as input and essentially run a for loop. So while FS read hyphen our images going to run a for loop on every single line of the file and every single line represents one image. So it will loop through all of them. It will verify all the images. And then this cosine experimental equal to one. That's the most important part. Even if you ignore all of the other things, this is where you know that you're actually very fine things. So cosine experimental equal to one. I won't go into too many details. Essentially for now consider it as a as the standard way Kubernetes release has decided to verify images. And in background cosine experimental is essentially using something called keyless signing where I don't have to worry about maintaining my private key or public key. The six store service is going to help me and keep the private key secure as well as the public key will be available whenever we have to verify it. There are ways to do this without where you have access to private key in public right now for the releases. That's what we are doing. So then we run a command called cosine, which is one of the things that you're going to install in the prerequisites and then verifies the command name registry dot cage dot IO is the registry and then QB API server is one of the images that we are actually going to verify. So if you want try running this by yourself if you have installed cosine by now. And if not, we will actually be running this live in front of all of you in case you're facing any issues with troubleshooting. Feel free to call Tabby or Nadir. They might be able to help you out, but if you get stuck and can't move on that's totally fine as well. So let's switch back to the demo. All right. So hopefully the prerequisites are working. Now if you see we are here. And we are going to go in this. Oh, oh yeah. Good point. Thank you. Okay. How about now better? All right. People at the back. All right. Cool. So we are going to go in this directory. And if you see here, we have the images dot Txt file downloaded. So let's look at it for as an example of how it looks like. So if you see each line is an image, like I said, the tag is actually the release. So like I said, because we're using latest dot Txt, it's going to pick up any release. Instead of the officially released Kubernetes releases or versions. So this one is using an alpha version of the upcoming 126 dot zero. Every release, regardless of whether it's officially GA version or not will be signed and you can verify it. So we have about 15 plus. I want to say images in total and you can see every one of them is for different architecture. So we have AMD 64 arm, et cetera. Now let's try to run this. Okay. So the script that we are going to run is essentially the same as the ones we saw on the slide. If you want to confirm that, I'll just show it to you quickly. So same thing, except the first four lines which are standard for bash to make sure like if anything is going wrong, we know. And then the idea is after this, we will see a lot of output that shows that we are verifying. So brace yourself for a lot of text on the command line once we run this, but we'll go through some of it one by one. And hopefully all of it will make sense. Oh, okay. So cosine command not found. So why this would happen for you also is if you go here and install prerequisites, what I'm missing is I have cosine installed, but I have not set up this path. So if I go back and run this, I can try and see, okay, cosine is working. So now I will go back and run this. So like I said, lots of text is essentially trying to share all the metadata that you can get about all the images and it will run for about a minute or so depending on Wi-Fi and it will. It is basically doing the same thing for all the images verifying it one by one. And now it's pretty much done. So let's look at what it's actually doing. So if we zoom it out a bit verification for registry dot cage dot IO cube proxy MD 64. That is what it is actually verifying. The thing it's trying to do is our is the whatever I'm claiming it to be like I'm claiming that my image is signed by cosine. It's verifying whether that's true. First, second, it's trying to do is are these claims also in my transparency log. So without going again into too many details with the usage of keyless signing, you end up being able to use transparency logs where you are able to use the public key that will allow you to verify things. Folks who are maybe more familiar with six store cosine than me, feel free to correct me. Any certificates were verified against the full show roots again. It's part of the six store ecosystem. And then the main thing to look at here is what image I'm trying to verify. So this is the Shah of that image that I'm trying to verify. And if we scroll down a bit more, this is the interesting part. So the private key is mapped to this particular account that is owned by Kubernetes. The least engineering team. So if you see it's called K rail life and trust gates dot release engineering dot prod, and it is a basically Google service account. So this gets linked to the private key. And as a result of it, only somebody who has access to this account is able to sign it. And as a result of that, because we can see here that this is the account that's being used to verify it, we are able to know for sure that it's actually signed by Kubernetes release. Okay. Make sense so far. Cool. Okay. Anybody got this running like I did on here? Okay. All right. I see some hands. This is probably going to be one of the easier installs. And the other ones, if you have kind and Docker would be simpler. If you don't have kind or Docker, it might take a while, but that's fine. So the same thing happens essentially every for all the images. And that's how you get what we saw. So now how are we doing on time? One second. I'll extend again. Okay. So about half hour is almost done. So let's take a five minute break before we move. If you want to stretch, drink water, ask me questions. Just close your eyes for a bit. That's fine. Make sure to come back in five minutes. Water should be right outside the rooms. And if you have questions, I'm all ears. And I'll repeat the question so we don't have to move around taking a mic for the people in the virtual attending virtually. Any questions so far from what we did? We quiet. Can you speak a bit louder? The long running command for verifying the images or the big output that we saw or that was the install prerequisites command. So if you have installed the prerequisites before, then you don't have to do it. But otherwise it would basically take a while only because cosine takes a while to download other things should be fairly fast. Okay. I'm back here because people won't see me in the video if I'm standing there. So any other questions from anyone? Anyone stuck where we can help you? Anyone has put any issues in the GitHub link that we shared? You have. Okay. Let's look at it. Okay. All right. So. Great. I think that's always useful to see for folks watching online. So anyone with Mac OS who doesn't have a code at win 140. So. I'll just use your GitHub handle has suggested how you could install Xcode before installing all the other things and take a look at this link if anyone is struggling with the same. Any other questions anybody stuck that we can help anything that felt like can merit more explanation? Go ahead. Speak louder. Speak louder a bit more. Right. Yeah. So how many folks are familiar with public key infrastructure? Like how private key public keywords? Okay. All right. So we can go quickly like a three minute summary and see how it goes. So essentially it's like I want to let you verify that what I'm claiming is true without sharing anything secret. So how do you do that? And essentially that's where the private and public key pair comes into picture where I will not share my private key, but I'll give you an option to verify that I am the only holder of the private key. And what cosine essentially does is it creates a framework for all of us to use that. So anytime we do a release in Kubernetes, we will use the private key that release engineering team owns to sign the image. So once it's signed, the signature gets stored alongside the image. And then when you want to use the image, you can choose to verify it like we did. And when we verify it essentially again, cosine will use the existing six store keyless signing to find the appropriate public key and then use that public key to confirm that the images are actually coming from the release team or not. So that's essentially what the whole big output looks like. Most of the other things are metadata about the images. The main thing to look at was the identity of the signer which we looked at which was that email address of Google service account. And that's basically how it was in summary. Any other questions? All right, let's move on to the next task. So, all right. So the next task is about pod security standards and how to enforce it. If you have been using pod security policies for a while, they are essentially gone from version 125. We have deprecated it since version 121. Many people from six security and SIG Auth which is another SIG as well as many other SIGs in Kubernetes have worked for multiple months to make sure this deprecation and eventual removal process is as convenient and simple for anyone who is an end user. So hopefully you are aware by now that we have removed it and it's been deprecated for a while. Anybody using pod security policies? There was actually a great talk on how to migrate from an existing set of pod security policies to something like the pod security admission that we have which happened yesterday. I attended it, I really loved it. They also explained something of a policy migrator tool that they have created which will allow you to figure out what PSPs you have and what you need to do to make sure it runs with the new pod security admission. It should be on YouTube with all the other talks soon. So make sure to check it out. I think anyone using that will actually benefit from it. So this is I love analogies, if you know me and I felt like instead of going into depth of looking at the yamls of all the pod security standards this might be the best way to explain what they are. How many people either still commute or used to commute before the pandemic to their offices? Wow, there are a lot of remote workers looks like. So if you commute or went to school in the past or the main idea is you want to be there safely and as quickly as possible. Same thing goes with pods and workloads that you want to run. You want the pods and the workloads to be safe and you want them to essentially do the job that they are defined and has to do. So if you think about it there are three modes of commute I have used all of them at some point in time in my opinion and obviously everybody's experience is different being on a train and going to some place feels the safest to me because I don't have to depend on my driving skills which is very important. There is a lot of less chance of some other train coming in front of me and we getting into an accident so that feels safer and basically what this represents is the most secure, highly restricted configuration of pod security standards which is actually called restricted. The second one is baseline which is somewhat of my current commute where I go through a very complex set of streets and roads where we have bike lanes where we have trams where we have people driving in their cars through buses and that feels about right because it can cover a lot of people and the way people commute but there are some things that they are doing well like if I haven't been in a bike lane I feel safer than being in a bike lane if I am in a tram I feel safer that I have a dedicated route there so same thing for bus routes also but it's still not as safe as you would feel if you are in a train and then the last one is privileged which is basically like driving along the beautiful freeways that we have here and if you have a car of yourself you have all the things that you need well said but not everybody can afford a car not everybody has a car so sometimes if I cannot imagine myself biking on this particular highway the one in the photo because I would be very very scared so if you have the means to do it it is great but probably it's only great for you and I wouldn't share the road with any other types of workloads which is what port security standards are which is basically anything goes you're on your own at your own risk so that's what port security standards are they are opinionated only three because we can't cover all the different permutations and combination of port security fields and from lowest to highest risk they are restricted baseline and privileged so port security admission is the replacement of port security policy it is enabled by default as of version 125 but it's not enforced by default so the feature is available but you cannot use or benefit from it unless you configure which particular standard you want to use in your cluster or namespace the setup is similar you can set it up in a cluster regardless of which standard you want to follow or you can set it up per namespace level as well and let's look at how to set it up in this specific tutorial we'll focus only on how to set it up at a cluster level focus on the colored lines those are the most important ones in the slides essentially this is a configuration in kind which is defining an admission configuration so we use a lot of admission controls when we use Kubernetes and what we are doing here is setting up an object called port security admission .config.cage.io v1 is saying that this particular feature or API is GA or stable and the defaults if you notice here we have three pairs of lines so first one is baseline and if you see enforce version is latest so you can configure it for an older version even if you are running a version like let's say 125 but you can configure it for an older version also but we are going to go with latest for now and then restricted and again restricted but if you see the last two are audit and warn so enforce audit and warn are the three modes in which the port security admission is used and then we have something called exemptions where you can say I want this particular user to be exempted from all the standards that I am enforcing I want a runtime class maybe you have a windows workload or maybe you have something else that doesn't work well or you don't want that particular node to use port security standards then you can exempt that and then name spaces is basically I don't want this name space to be under the cluster level port security standard that I am enforcing or warning on or auditing on so this name space could be for example for all the privilege workloads in your entire cluster where you are you have to run some workloads in privilege and if you put it all in one name space and just exempt it and for all the other workloads which are running your business applications you can enforce it at cluster level so I talked about three modes basically the enforcing mode is I am going to fail the port creation if any of the port security standard that I have defined in enforce mode actually is not meeting the standard that I have defined so I am creating a port and it doesn't meet the standards of baseline port security standard the port creation wouldn't work same thing for restricted, same thing for privilege and one and audit are somewhat similar the main difference is the audience of both the modes one is going to warn the user who is trying to create the port so when you are running kubectl create port you will get a warning when you are running it saying well we have set this standard let's say restricted in one mode and your port is going to be created but we are going to tell you that if restricted was enforced your port would fail because you are missing these things in the port spec and then audit is similar but it will actually log that same warning or similar in spirit in the Kubernetes API audit logs so if I am an administrator of a cluster I want to see all my users of the cluster if they would be able to create pods if I convert all of my port security standards to enforce mode in restricted I can set the audit mode for restricted standard and then I will start getting API server audit logs for all the pods that get created and which ones are meeting especially which ones are not meeting because I will see that in one of the log lines in my API audit and exemptions like I said depends on these three things namespaces, runtimes, users and then we will use a namespace exemption where some of the privileged workloads that are needed to create the cluster are put in kubesystem namespace so we will be exempting it because it would need extra permissions but we want all the other pods that are running in normal applications to have different set of pod security standards and force and next one, next two slides and then we will go to the demo so how do we create a cluster with pod security so remember I mentioned we will use something called kind KIND and this is one of the configurations for it you can figure it out based on kind.x hyphen kh.io and kind also uses another tool internally called kubedium which allows us to configure all the flags and the settings that we have for any specific cluster so here what the cluster configuration is saying can you pick up the cluster level pod security standard yaml file that I saw here when you are creating a cluster so the colored lines that we saw is basically content of this file so what kind will know if we said this is when I am creating my cluster I am going to automatically enforce in this case baseline pod security standard so for any pod in that cluster that I am creating you will have that pod that pod would have to pass basically the pod security standard otherwise the pod creation fails it will warn you on restricted and it will audit on restricted as well and then essentially this minimal pod spec actually is going to work if our baseline pod security standard is enforced because we are not adding anything in security context here but it is going to work because baseline does not expect you to add anything that is minimally possible in a pod spec it will allow but because we have set it to enforce it is going to give you this kind of a warning that if you want this to work when restricted standard is enforced you would have to set allow privilege escalation to true you would have to drop all the capabilities you would have to run as normal and you would have to set a second profile you can keep it blank so that is what it is saying and then this allows you to then make those changes if you want it to accept a restricted pod security standard configuration and this is an example where it would pass a restricted pod security standard if you really want to run it which is something I would recommend as well because it is the safest way to run a pod if you are using pod security admission so lots of information we will jump to the demo and but if you have any questions I can take a minute to answer them before the demo all right cool we will go back to mirroring if we go here so pod security admission and if you see here so I am going to quickly check something if I have existing kind gate clusters so this might happen for you also what it basically is if you are using docker desktop your docker desktop is not running so I will start it and the best way to find that out is this is the socket unique socket that docker desktop uses and if it is erroring out there that means docker desktop was not running so now it is starting up it will also show up here with the veil icon of docker and once it started we will be able to use kind because kind wouldn't work if we don't have docker installed same thing for anyone using docker binaries on linux if the docker server is not running on your system you will probably get a similar error so let's see this command one more time and for folks wondering like I installed a bunch of stuff later after this session how do I clean it up so the best way to do this is kind delete cluster name all so we will create about four clusters if you finish everything with me and you can delete them one by one just like this so I had a cluster called all which we will also try to install and I deleted it now if I see get kind clusters no kind clusters found so now going to the part security admission if you see we are going to run this file and if you are curious how it looks like it's basically the same thing that we saw on the slides so it's saying can you create this cluster level psys.yml and then when you are creating the kind cluster can you use my file there are some extra things here which is extra amounts and extra volumes so essentially that is needed because kind is using docker so it would need to mount something in the right path inside a container so no need to like worry about the details if you don't understand this but the idea is I am going to ensure that when cubeadm is going to create the cluster I am going to pass it on this config file that is going to let it know which part security standard I want to enforce so let's run this now so kind with cluster level baseline part security enforced and if the demo doesn't work I have backup videos so you will be able to see how it looks like so what it is doing here is getting a node created with version 125.2 where we can install and configure all of the things that we want then it is writing the configuration that we set for cluster level pss with baseline enforced restricted warrant and restricted audit and now it is going to start the control plane this is the step that takes the most time and this is the step most times if you are using kind where it feels so we will see how it goes it worked so now it is installing CNI installing storage class which is the standard thing kind does now what it is doing saying is I have a cluster that is created if you want to use it you have to make sure that you set the config of kubectl or context of kubectl to point it to this cluster so kubectl is essentially like a client if the client doesn't know which server to talk to essentially the client is not useful so this command kubectl cluster info context will tell the client go talk to this kubapi server that I have here now the control plane of kubernetes is on this particular URL with this port we have core dns and we got the warning so I will bring it up a bit this is essentially what we wanted to see so we created that minimal port spec with nginx and then we got a warning which said oh this port violates restricted latest port security standard if you do then the reason is because you have not set all of the things that we talked about but if you notice it still ended up creating this and if you notice this was the port that we set to allow it to pass a restricted port security standard and you don't see a warning for this so the reason is because it is set in a way that the restricted port security standard would still work and if you are curious how the port spec looks like you will see this one is the minimal port spec and this one is where we set all the things that restricted port security standard was looking for so that's pretty much how you can use existing port security admission to create clusters with something as simple as baseline which would work for most workloads but gives you much better protection than not having it enabled at all or everything running in privilege which is basically you don't have any container isolation boundaries so that's the demo let's go back to the slides and we will also take a break for questions alright anybody has any questions anybody needs help we have couple of people ready to help does this give you a better idea of what port security standards are how to use port security admission yeah go ahead how do I get the if you run kubectl get parts you will see the parts I'll show you quickly so oh I see what you mean so I can do like a get pod but can I do something like get PSA so that I can see what is actually configured for in my cluster I'm not sure if anybody knows feel free to answer but essentially the idea is anything you define in the cluster level PSS YAML file that is exactly what the APS server is using so it's essentially declarative go ahead describe pod would it would show up if you're using PSA using name spaces because name with PSA with name spaces you need to add a label saying for this name space I want this particular PSS config to be enforced but because we are setting this at cluster level we will not see it in the but if you set it to like one like we saw you will see like okay restricted was in one and this one didn't work same thing will apply for API audit logs there is a good chance that if you look at the API server logs not the audit logs the logs of the binary itself in the system logs when the cluster is getting created and the API server is coming up you will see that it will try to basically hook into the config that we set for port security standard and admission and you will see that reading the config from this particular file so that will give you an idea that it is actually working or not yes I'm not aware of hand of a way to inspect like in one shot from QCTL get or inspect something right another okay so for folks virtually basically our experts who are helping us here said couple of things if you are using port security admission for namespace you will you can use QCTL describe namespace and then you will see that label where you can set the pss second is there has been an issue open where we want to get a better idea of what is my configuration of an existing cluster for a couple of years but we probably need a cap and it's probably a great thing for somebody looking to contribute and familiar with kubernetes to take it on and help the community anything I missed please correct okay okay let's actually yeah so let's take a look quickly for folks attending virtually the suggestion was maybe it is already part of the describe output as an event so if we take a look at it we might be able to see it so if you see here let's pick the nginx1 kubectl describe I think it's part nginx yeah so let's take a look okay so probably it should be here if it was implemented right so yeah so it's not here but we did get a warning so that's a pretty good idea or a pretty good indicator that the cluster config actually is working because we wouldn't have gotten a warning if it actually wasn't set correctly so that's how it would look like and now going back to the demo since we need to keep moving let's go to the next task anybody added any issue in the github repo we can take a look after this task as well so the third task is about runtime secom profile how many of you are familiar with what secom filter secom profile is okay alright so I have another analogy for you it's a system call filter system calls are essentially ways for you to ask kernel to do something if you are in user space and filtering of those system calls is going to allow you to reduce the exposure to your kernel if you are running something malicious in user space of linux so linux just very briefly divided into two roughly rough parts user space and kernel space user space is lesser privileged has to depend on things in kernel space and system calls are a way to move around between those one of the fundamental things is that the system call filter isolation has been secom filters since docker became popular there have been very great very important work done multiple years ago by somebody called jessie frazzle who created a secom profile which really worked for vast majority of workloads all around the world and she writes in her blog she literally tried it for multiple images in docker hub and made sure that whatever profile was being created was actually going to work and not break things in production for anyone so massive thanks to her most of the new secom profiles in new container run times are using that work and pretty much the profile looks similar so we will take a look later the idea is when I am going to block some system calls I want to block the ones that are going to break my container isolation boundaries so if you are running a container somebody else in your team is also running a container you end up being in the same node you don't want those two containers to be able to affect each other in any way so what the first container does should only affect this container second container shouldn't even know about it when a system call could actually change the behavior of the other container that's when we want to block it because it's essentially making the isolation boundaries and the reason you see the tea and the saucer and the filter is if you make tea with leaves or like dried leaves you need that filter in between so that when the tea goes into the cup you don't want the dried leaves in your mouth when you are drinking the tea so the filter is going to remove the stuff that you don't need so you have a very smooth very nice experience when you are having tea folks who make tea at home probably can understand what I am saying so for folks who are not aware how many system calls are there are a lot 300 and counting which makes it even more impressive the work that was done before to see which system calls would work if we removed it and whether it would break anything or not the runtime default of a Seccom profile is essentially what my container runtime is doing so in the beginning if you remember I mentioned container D, cryo and even Docker in the beginning when it was used as a runtime for Kubernetes would have a Seccom profile Kubernetes long time back made a decision for things I am not aware of to not use that container runtime profile as a default so if you are using Kubernetes and you think container D, cryo by default has a Seccom profile I am going to automatically benefit from it that's not true at least not yet unless you do what we are going to do in the rest of the tutorial so the runtime default is a new feature which tells Kubernetes can you use this container runtime's default profile for all my pods that are going to create containers under my container runtime so as a result of that it's minimal work it is not going to break most things there is no guarantee it wouldn't break anything there is a chance it would break so always anytime you want to change any configuration or cluster the highly recommended thing is to test it in non-production development environments let it soak for a while let it run for 6 months with that configuration and if things don't break then you try to see slowly maybe in one cluster then second cluster and then eventually have also a role black plan where you can quickly move back to original configuration if needed there are three other fields or the same for the seccom profile you can set it in three ways one is unconfined which was the default for a long time which is essentially don't use the seccom profile I have second one is the one that we talked about local host is another one where you can go really crazy and there are some really nice tools about it where you can define your own custom seccom profile any specific pod in your cluster so as a result of that many times the things that are blocked by default are great but you can block maybe more if you are using doing something simple as running a web server or running some gRPC peers who want to talk to each other then you need way fewer syscalls than you would actually need in a full-blown java application for example so you can do that with using something called localhost which is custom seccom profile for any pod that you want we won't go into too much depth details into it but that is a good option if you want to advance even more the security in your cluster if you haven't been using seccom runtime default is definitely a good first step towards improving the security of your cluster what does the default seccom profile look like for a container d or a cryo workload so these are the links to it we are slightly behind time so I won't go into details of looking into it but basically it will have a block which says which syscalls are blocked it will have a block which says which syscalls are allowed and that is the profile that the container and time uses anytime somebody like a container space says I want to use this syscall and the kernel can you help me and the filter will come in between and say no you are not allowed to talk to kernel with this syscall that's basically what it is and seccom filtering with customizing it looks somewhat like creating a pizza where you have a dough where you have toppings and there was a great keynote about pizzas in Detroit so you can create pizzas that are square shaped pizzas that are circular you can create pizzas with different types of cheese paneer, mozzarella you can make an indian pizza you could make a chicago style pizza but at the end of the day everything is a pizza so everything is a seccom profile but your profile might look different than somebody else's so that's basically how the customizing works what we are going to do with testing isolation boundaries is it blocks syscalls like I said it also the default runtime profile also blocks something called sys underscore time which is essentially something that allows you to change time system time of your container and the tricky part of it is if you change the system time in your container it changes the system time in the node so all your containers suddenly go back even if you don't want all the containers to go back in time and that can create some weird attacks that you could do so we won't go into what kind of attack would work but that is possible and that's why it is blocked because it breaks the isolation boundary so we look at one where we don't set the seccom profile to runtime default and see how the time changes and we look at another one where we try to change the time and it actually doesn't work it says operation denied so setting up the seccom profile this is this texture is a bit small hopefully you can still see it the bold ones are essentially the settings we need so the cubelet extra arguments this is similar to the other one where we had a kind cluster we had a cubium config and if you see here it's called seccom default set to true and then another one for worker nodes as well which is seccom default set to true so and then there is a feature gate called seccom default true so all of those will make sure that it is by default going to point to the runtime seccom profile and when you use it you get the benefit of it in kubernetes now so the demo is going to look something like this if you try to exec into a container that is running the default seccom profile and you try to change the date you will see that it will say oh I cannot set the date operation not permitted and if you check the date it is actually not changed it's still what it was at the time if you run it without seccom profile it will actually change the date and if you see later in another pod so if you see forever asleep again is the other pod it will actually change the date in that pod even though you change the date in just the forever asleep pod so that's big breaking container isolation boundaries we don't want that that's why having the seccom profile set to runtime default is useful so let's look at the demo now so if we go back to the terminal and go to seccom runtime default there are two files one with and one without so let's run the one with sorry without first and we'll do the same thing it will create a cluster it will use the configuration we set to default to the seccom profile that's used in your runtime and then we'll try to change the date and see if it works or not how many of you are liking using kind so far right yeah it's really cool I really enjoy using it as well any questions while this is running was the introduction to seccom did it make sense any questions on that okay cool so we got the cluster we are installing cni same thing cluster info context it's setting to the cluster name oh you had a question go ahead yeah so that's where the previous task actually comes into picture if we use privileged the seccom default can actually be overridden and so if you remember we had three port security standards restricted baseline privileged if we run a pod as a privileged pod it will essentially run as seccom default as unconfined so it will not be filtering anymore using the runtime default and as a result of that if there was anything like ptrace for example which is sometime useful and is blocked by default you'll be able to use that if you run it as privileged another option is custom seccom profiles where you can customize the existing seccom profile create a file that says I want this call I want to remove all the others and then use that as your default for that particular name space then you're able to do that as well in my experience personally the blocked ones I hardly had seen to be needed for any standard application like a web server or a database or anything like that anything that is available by default most of the time works out so we got this two pods now created pod forever asleep pod forever asleep let's see if this is working so ok I am already in root forever asleep now what I am going to do is try to see if I can change the date using this command so so we should get an operation not permitted here oh no we should actually not get the operation not permitted because we are running without a seccom profile so the date should change and if I go here again the date is actually October 28 let's try again ok so there you go so the date has actually changed most of the times if you're known as an NTP server it will reset the date so you're fine most of the times but this is an example of like it can actually do it and if you want to see how it actually breaks the container resolution boundaries we would do something like running it in another pod and see if the date has changed so if I go to this tab and how much time we have ok we are almost there so if I go here and go to seccom do kind without so it's the same command like before what I'm going to do is this is already done so we'll go here and instead of asleep I will set it to asleep too alright let's see if the date is so the date has reset let's try to change it here so date is changed here and date is changed here as well so it's two different pods but because you change it in one place it changes in the other pod which is not great which is why it's blocked so now if we go back here exit run the same thing with seccom front time it won't even allow you to change the date in your pod and as a result of that you can't affect the other pods that are running did that make sense so some of the things I was wondering about is like if I change the date and the license is set up for my software to expire after some x number of years or it cannot run for an older date than January 1st 1970 what would happen so that's an interesting thing to think about and you don't want it to happen to any of the pods we are running which is why having that kind of seccom default is always useful there are many other syscalls that are blocked as well not the only one but in terms of demo specifically this made more sense in terms of really understanding like oh I am actually breaking the containerization boundary here so joining the worker nodes again and now we'll have another cluster with seccom with runtime default and we will be able to try and run the change the date again and hopefully we should get operation not permitted error anyone been able to follow along on their laptops completely so far okay great all right I see a few hands okay so this pod is now created so you'll see that this prompt stays there for a while the idea is when you ask the pod to be created you get a response that the pod is created but it's not sometimes available because it has to go in the running state until you can execute it so I put a sleep 30 which is a random second number of seconds and after those 30 seconds or so most times the pod actually is running and then you can execute it so that's why it stays there for a while and then it goes into this patch and if we now try to change the date okay I got the wrong one so let's go back here and oh maybe in the slides so copy back here change the date as you can see it says operation not permitted so that's the goal and you'll see the date output still but if you see here the data is still old still the current one so we can try again if it was not visible earlier and if we see here the date hasn't changed so that's basically one of the ways we can do the second filtering now this was the demo and we'll take about let's say five minutes break for questions and then I have another demo where you basically end up setting PSA second together in one cluster so for the if you don't have questions we can try that otherwise there is a script in the repo where you can try that and one of the things I'll share that I didn't mention earlier is by default there is a capability called capsis time which also blocks this operation so if you use restricted port security standard and if you remember back to some of the slides it was saying you need to drop all capabilities if you want to use restricted port security standard so let's say for some reason you don't want to use or you can't use seccomp or you want to do it later if you switch to restricted first and drop all the capabilities and even if your seccomp is running unconfined it will still not allow you to change the date because you're dropping all the capabilities and capabilities is like a linux term what it essentially means is what am I allowed to do when I talk to kernel and there are about 15 plus capabilities if I remember right which give you a big set of permissions so one of them is this time then there are some other things like cap netraw which is another somewhat dangerous capability to use so if you drop them with restricted don't use seccomp it should still work if you use seccomp default don't use restricted it should still get blocked because even if you add the capability like I added here I added the capability but because I was blocking it in seccomp it didn't allow me to change the date so it can purely for system call filtering you could work around it you could use both for defense in depth so it's really up to you play around with it both of them can work together and that was one of the demos but yeah I would love to answer more questions instead of the demo unless people are interested to see the demo so who wants to see the demo who wants to ask questions demo okay alright let's try that so we go back here I will mirror again oh it's already mirroring okay great if we exit then remember the cluster I deleted that's the one we are going to create now so all together is basically combination of PSA plus seccomp and there should be one file only and it's going to do the same create a cluster pick two configs now one is the port security admission config second one is the seccomp config which is basically saying please use the runtime default seccomp and with those two configs it will create a single cluster and both of them will be set and enforced at the same time so what we should expect is the nginx pod for example should still get the warning that we got last time the forever sleep pod which we are using a minimal pod spec for it should also get a warning and if we go in that pod and try to change the date we shouldn't be allowed to change the date so that's what we should expect because we are setting both of them in the same cluster the good part I'll share later is most of the things that we talked about the official documentation in Kubernetes also goes in much more depth many of the things I've written myself in the docs and some of them I have reviewed so please continue to use that as the reference even for future because I may not be able to continue updating the repo as the new versions for Kubernetes come up but the official docs will always be updated because it's all community work not just me trying to maintain it so I'll share links in the slides for all the official docs for all the things that we did so if you see here we got the pod nginx created but with a warning same warning as before and what we should expect now is when we create the other pod it should also get the same warning because we are using minimal pod spec just like this and now if we go inside that particular pod and try to change the date it should fail any quick question anyone has before we move on and wait for the pod to go to sleep the forever asleep name is basically because we are trying to put it in sleep because if I don't run anything as the exec for kubectl the pod will basically die after it is created because it has nothing to do so we just put it in sleep so I can exec into it and then do whatever I want so it failed here it's now inside the pod so let's go up here again and hopefully I have the date command in my buffer and if you see here it's not it's saying operation not permitted and the date is still the current date so that's what we expected that's what we saw and both of them are working together essentially in this case now back to slides I will extend it again and so we brought it all together we could do the same verification step before these two as well because it would work for any release that you are using then you we could create a cluster like we did with both together so definitely I would say try this at home when you are done setting up all the three tasks one by one and see if it works try to run maybe a sample pod that you think could work or not and see if it gets whether you get a failure, whether you get a warning try to play around with PSS, different PSS baseline restricted, privileged try to set it up in different modes and see what happens and yeah try it around see if it works I am open for questions if you have any I would love to answer more and hope this was really useful thank you yes you had a question the official docs if I remember correctly say version 19 is stable but one of the things I should have here I will add it in the slides in the SCAD later there are links to the specific docs that cover all of those things and every Kubernetes document will have at the top from which version this document is valid or not so we will be able to take a look at it so if you have any questions or comments or want to say thanks or have follow up questions specific debugging questions definitely add issues to the repo if you have specific questions that you come up with later and thought I wish I could have asked if you are on twitter please tag me there I will try to answer I should be able to have some time until the end of the weekend and if you want to give specific feedback for the session to CNCF this is the QR code if you found it useful if you found things could be improved I would love to hear from you for any future sessions I do with that thank you very much hope to see you for the rest of the kubecon lot of good security track sessions for the rest of the day and if you are heading back home happy journey and see you next time