 Okay. All right. So, Hunter clearly didn't talk for a long time, and he had to get it all out. Okay, so he gets a talk. I want to get a talk, too. Who taught security? Yeah, I think people still awake, I'm not sure. So, who said security was going to be exciting, right? Not many people. All right. So, at SWOT, SWOT is a company I just joined like half a year ago. We also had to implement a kind of secure environment. Well, there I say that I made a very secure environment, and I wanted to share a little bit of what we did. I think this talk is going to be a bit focused on a certain use case. So, I just want to know, like, who's using Amazon Web Services? Right, that's a good amount of people. Who's using Google Cloud? Okay, that's... And Azure? Okay. All right. So, at least I'm using Amazon. So, who's using the hosted or managed Kubernetes clusters in Amazon? Okay, two people. Everybody else. Who's using K-Ops? Kubernetes operations. Okay, just me and the X on the handle on this, people. All right. So, that's maybe going to be a little bit less... Okay. I like Kubernetes. And my other image is gone. So, I also like K-Ops, which is the Kubernetes operation. It's a tool that you can use to provision Kubernetes clusters in Amazon. And I want to highlight a few features that I use a lot. So, for example, one of the features of K-Ops is the ability to create, like, an organization where you manage, like, your POPs template. And then you have your defaults. And then you have, like, different clusters. So, in this case, I have a template in K-Ops. All the clusters are defined as a manifest. Very similar to how you define anything in Kubernetes. For example, deployments use a manifest. So, in this case, I can use a... How does this thing work? All right. All right. The right one. Okay. So, the API version here is cops, not Kubernetes. And then, in this case, I'm talking about a cluster. And I have parameterized the cluster name, some buckets where I put some add-ons, and things like that. So, using this, I can specify some defaults. For example, my cluster, it has a default cluster domain. Zone IDs are not the real values. And a version that I want my default to be for my clusters. This is very long time ago, right? I'm not using 1.9 anymore. Hunter was saying how everything is changing so quickly. I don't know who's using Kubernetes 1.9 still. Running Kubernetes 1.9. No, I'm not. Who's using 1.10? 1.11. Okay. That's all right. 1.12. Okay. 1.13. Okay. Nobody. All the Google Cloud platform people should be like 1.12, right? Anyway. So, then you can provide some variables. And also, for example, you can provide your cluster-specific one. In this case, I've created a demo cluster. And I have created dedicated subnets for it. So, if anyone is familiar with K-OPS, they know that by default, K-OPS will create a bunch of subnets for you. But I will go over some reasons why you don't want to use that. Actually, I want to go really fast. And like I said, this is going to be quite K-OPS oriented. I'm just going to go really fast. If you want to learn, after this presentation, you want to learn K-OPS, go ahead. Then you can go back to the video and see, like, hey, what was he talking about? Maybe I can use some of these things. If you said I never want to use K-OPS, that's fine. Sorry. Sorry. It's mine. I'll try to go fast. So, yeah, using subnets and all those parameters that you want to pass in for cluster-specific values. Next, I love make targets. So, in this case, I create a little make target that uses the toolbox template function that uses my default template, my default values, and then my cluster-specific values and manifest. In this case, I just need to run make cluster demo, and it's going to pull everything together and generate the whole cluster definition. So why do you want to separate your subnets? By default, I don't want to use the K-OPS. After you've used K-OPS for a while, you realize that actually I think the main reason is routing tables. But there could be other reasons that you want to manage your subnets. I think in Amazon, the main reason to manage your own subnets is to set up your own tags and to set up your own routing tables. For example, if you do a lot of VPC peering, if you do a lot of connections from other data centers, and if you want to do some egress controls. For example, as Hunter mentioned, Istio allows you to do egress whitelisting for the outgoing domains, like if your containers are connecting to Google or to other domains to make sure that you allow those connections. In my project, unfortunately, I could not push Istio yet, so I had to find an alternative outside of Kubernetes solution to whitelist the outgoing connections. So for these reasons, you might want to run your own subnets. So how do you do that? I like Terraform, so Kubernetes Terraform helps. There we go. So in this case, I'm defining a bunch of availability zone subnets, and then I specify my shared subnets where I put in a couple of tags. These tags are important. They define the let Kubernetes know where it can create external load balancers. So whenever you create a Kubernetes cluster in Amazon, if you use K-Ops or anything else, they're always going to use tags to let Kubernetes identify the subnets where it will create load balancers. So that's one thing. And then after that, I actually have animations. Everybody else lost theirs. After that, I have to create private subnets for my specific cluster. In this case, I have two types. I have my edge nodes and my compute nodes. This is also something that was mentioned in HunterTalk. It's about segregating nodes using taints and affinity, I think, to push certain containers onto certain nodes. So I have a couple of edge services that I want to run on edge nodes and a couple of application services that go deeper in my network. So in this case, I create two separate subnets for this. Again, the tags on the subnets are important. I am letting Kubernetes know that this particular subnet is spawned. So it's not shared with any other cluster. So it can usually do more stuff with it. And also that this will be used for the Kubernetes API internal. This is actually not a specific Kubernetes tag. This is my personal tag because I need to do some VPC routing. So I use it to identify some subnets with this tag. And then finally, I have my edge where I basically tell here, if I need to create internal load balancers, they go into my edge subnets. So earlier, we saw this tag, which is for the external load balancers. And if I need to hit from inside my VPC, I'm going to create it in the same subnet as the edge. Is this good? Maybe not. Maybe I should create my load balancer in a separate subnet where I was lazy. Next. So this is where my subnets and I entered and spliced into my cops definition, as I mentioned earlier. And in the end, I get a nice cluster stack generated like this where I have my subnets specified and my edge subnet specified, allowing me to then create different instance groups. So why do you want to use edge nodes? Compliance requirements. If you need to have, if there is a customer requiring you to hit certain nodes before you hit other nodes, you might need to run edge nodes or different types of nodes. How? Like I mentioned, using a dedicated instance group, which you use to tag and put tints. And then also by using load balancers that can be limited to edge nodes. So let me go into those two points real quick. So the first one is an instance group definition and chaos looks like this. So we have chaos definition instance group and I am adding a label that this belongs to this cluster and I'm adding a taint. This will force explicit toleration so that means if I want to run anything onto these nodes, I have to tolerate this taint. That's a way to schedule workloads onto specific nodes and Kubernetes. Another thing that I do is I put a label on top of the nodes which is the instance group. I need this because I need to be able to select my load balancer only a hit the edge nodes. My load balancer should never touch any other nodes, not my application nodes directly. So I have to put a label on the nodes as well to find them later. Then finally, how do I set up a load balancer that only hits my edge nodes and don't hit my application nodes directly? The problem here is if you use Kubernetes service type load balancer, by default this will always hit all of your nodes. So if you use Kubernetes services, then you have three different types. A cluster IP which is only accessible within the cluster, a node port type which will open the port on every node in your cluster using IP tables. And then you can use load balancer. So if you use load balancer it's going to create a node port and then it's going to create with your cloud provider a little bit like what we talked about earlier, service catalog is going out and create a load balancer and then sets up the load balancer to hit every node on that node port that it was created. So the solution, I cannot use the default service type load balancer. There is this great project from Zalando which is called Q-Indress AWS Controller which allows you to provide custom filters for your nodes. So earlier when I set up my edge nodes with the label for edge, then I can use my Ingress Controller to hit only the instances that have been selected for edge. So this looks like this. When I deploy my demon set, basically the controller is ran to identify here the custom filters. So I give it the controller ID and then I use the Amazon region and the custom filters that I pass in through a templated function. This is basically Terraform templated manifest and I will explain more how I use that. The entry result is when I use my Terraform to bootstrap my cluster, I'm going to create a module. I have a special module and I will go a little detail on that and basically in my custom filter I specify that the Kubernetes tag has to be this particular cluster on nodes and the key for the cluster autoscaler has to be a node template taint for edge. Actually I'm not using the label. I'm using the taint. Interesting. So I don't need the label. Okay. But I do need the label for something else. I think for my morning today. Then I think earlier Hunter highlighted that, I mean Hunter highlighted a lot of things and there was a lot of things to think of but one of the things was in terms of user authentication and alteration. So alteration, this is the same thing that you mentioned I think, the R-Bag, other events and image-wide listing with admission controllers. One thing he mentioned also was the OpenID Connect which you can do with Google and Betsy which is what we're using almost every right. And then the other thing is to use the Heptio Authenticator. If you're on Amazon and you run your own Kubernetes nodes you can integrate the authentication. If you use OpenID Connect, you can integrate with anything. You can integrate with GitHub. You can integrate with anything that talks the OpenID Connect protocol. Then, and if they don't talk OpenID Connect you can use text to talk to GitHub for example. And then if you're running on Amazon and you don't want to use a third-party authentication you just want to use AWS Identity Access Management then you can use Heptio Authenticator. So this is very technical and I'm going to go really fast. So the Authenticator requires these steps. First, you need to create an Identity Access Management role inside Amazon that your users when they want to access the cluster need to be able to assume role. Then you need to set up certificates for authenticating like setting up the WebHook TLS and you need to also reconfigure your Kubernetes API server to use a WebHook. Then your Authenticator Demons needs to be scheduled across all your masters. How? I like Terraform and I like K-Ops. So I use Terraform, K-Ops, Hooks and Add-ons. So real quick, first Terraform to create a role just create a current account trust policy document that basically says the root of the account is allowed to assume this role. So we trust this, so users can assume this role through the root of the account and we then push, sorry, what's this? This is the user group. So I have a user group and then I basically give the user the ability to assume the role. Actually this one is to give the Amazon server the ability to assume the role and this one is to give the user the ability to use the Amazon service to assume the role. Next, I have to specify the TLS certificates. So I'm using a private key based on that private key to generate a self-scientificate which is signed for local hosts because the authenticator only listens on local hosts and the Kubernetes API server talks through local hosts to authenticate the user. Then I generate these certificates and use S3 to push all this information into Terraform to push it into S3. And finally, I need to set up my API server to authenticate and basically set up the base 64 encoded certificate signature and then provide the URL that this API server needs to talk to local hosts to authenticate. So using the local hosts, it will talk to the daemon set which is the authenticator service. So the authenticated service then needs to have those variables so again this is templated in. So this is how I use a Terraform template and then finally push that configuration into S3 as well. And when I bootstrap my nodes with chaos I basically can say every node whenever it gets created if the role is master, not my compute nodes. If they are master nodes, then it will quickly run a Docker container because this is CoroS I don't have any AWS, Python or anything. So I need to, whenever the CoroS node starts I need to do a Docker run to quickly copy from S3 all of the data. So the TLS certificate is the web hook configuration everything and then on the API server I have to set up the API server authenticated to use this config. So I know that at this point I've done the set of the role, set up the TLS, set up the configuration for the web hook and then bootstrap my masters to pull all the certificates from S3. Next, finally, Kubernetes chaos operations has this add-on bootstrapping thing. So whenever you create a clusters pack you can specify add-ons that you manage. So in my case, I have my own add-ons that are having their own channel. So the way a channel looks is a channel of add-ons can have multiple add-ons. So at SwapMobile we use many add-ons that we want to use. For example, our ingress, like I use a special ingress controller, right? So I have a special add-on to manage my ingress and I have a special add-on to automatically provision my authentication as well. Okay? There's a point to this presentation that I'll get to. And then I can do this. You can watch the video if you want to get the details of how this is. Basically, this is a demon set that is running on all of the masters. So this demon set run on the masters will be able to expose the authenticator for the web hook on the masters. And then this is templated as well. So the cluster ID is templated. So again, I have some templated add-ons that basically are pushed into S3. All right. So I wrote this post on Reddit this morning. Basically, what are we doing wrong? All right? Our DevOps is trying to replicate Google Cloud service locally. Only a few wizards know how to use Kubernetes. And why are we also running self-managed promises? So this was an interesting post because after everything I told you and everything that Hunter told you... It's so simple. It's so simple, right? It's so simple. So why even... He mentioned the DevOps guy has been working on this for over a year. So... And this was... I don't know why I put this side here. Because the next part is also... That's one part. But this is also about the security. Like, if you want to secure your Kubernetes components, you want to set up 2LS between your HCD and Puber bootstrapping. And you want to set up all of this, right? So, as you see, when to offer custom deployment, right? When you do your custom deployment, you need to set up your symmetric keys that need to be rotated. Kubernetes talks with TLS, so you need to rotate those keys over time. You can probably use Cube Admin now, but at the time I was setting up the cluster, it wasn't production ready. Then also we need to isolate HCD. We need to do all of this, right? So, what you get is you get a leading edge. You get a choice. I can choose my machine configuration. I can choose my operating system, my storage backend, my network against my Ingress configuration. I can choose everything, right? I have a lot of choice. But the trade-off is you might end up in Kubernetes nightmare, where you have only a few people that are able to do this. And doing this for a while, I started to realize more and more that actually going with a managed Kubernetes solution kind of starts to make sense. Am I getting... I mean, only two people were using Google Cloud hosted Kubernetes, right? And there was one person using Amazon's. I can understand why you're not using Amazon's hosted. So, who's running Kubernetes on bare metal? Okay, yeah. All right. So, after doing this for a while, I started to realize this is hard. Definitely if you come down to the security part, right? We are not all experts, and we cannot all make this work perfectly. So, and then as an add-on on to Kubernetes... Sorry, on to Hunter's talk. You are now called Kubernetes. On to Hunter's talk. Part of it was regarding application lifecycle. As Hunter mentioned, with containers, we have immutable images, and we can move the scanning or we can start verifying vulnerabilities right at the build stage. So, we need to integrate our security more towards the CI pipeline, more towards the left side, which is where the build starts. And while it's being shifted, so we need to check the registries that we're using as Hunter mentioned. And then we need to also, during the running part, we need to assure that the images that we are running are authorized. So, this is a slide actually coming from AquaSec. And then, I mean, at least this part is coming from AquaSec. And then, as a recap, that was pure on containers. But then we talked about container orchestrations such as Kubernetes. We not only need to do the scanning, but we also need to possibly set up admission hooks to make sure that the images have gone through the whole pipeline that they have been checked. We need to do process whitelisting, we need to do binary whitelisting, no protection, and set up RBAC and ensure least privilege approach. So, there is a lot of open source projects, right? Initially, when I look at this, I was like, okay, Claire, Koro is there, open source, I can deploy it. Graphia is open source, awesome, I can deploy it, right? Yeah. I decided that this time I'm not going to go there. And I actually looked at AquaSec whitelisting secure and there was another one mentioned earlier as well, no? No. Those are the only ones that I could find. And we have a lot of whitelist stickers. But basically what they do, they, and this is again an AquaSec image because their images are just nicer, not only do they provide you with this initial scanning phase, but they also give you the runtime protection. Basically, they will intercept all the calls to the Docker team and they will intercept and implement webhooks, admission webhooks to your Kubernetes platform and they will also provide container firewall and allow you to do threat mitigation. They also integrate with secrets back end. So if you want to use vault or you want to use something like that, then you can set up rules to inject secrets inside containers. The great thing about this is that when you inspect the container, if you normally provide environment variables into a container and you inspect, you can see all the secrets. If you have access to the Docker API to talk to and look at the container, you can see everything. If you use their system, they kind of obfuscate the secrets from the container level, only injected within the container. And I actually had a bunch of knowledge I wanted to share about how to inject secrets, which is probably another topic because the HALM talk also touched on it. Basic secrets are not really secure and it's important to look at that. So I didn't want to make the mistake of running everything myself. I chose the platform. They're not necessarily cheap. It depends on what is your requirements and your compliance requirements. So that's basically it. I hope that made some sense. All right, you have a question? Yeah, I'm not familiar with COPS, but does that handle your host private network investment business? With Kops, you can also provision bastions, but we took that out as well. We provision our own VPN and everything. So actually we run all of our Kubernetes clusters only privately exposed and we manage the VPN outside, so we don't want the bastion for that as well. This depends on your use case, but yes, Kops, if you want, you can specify utility subnets and then bastions will be provided there. And the bastions and utility subnets shared by the ELB by default. So where your external open access are. Any other questions? No? Yeah, which one? Okay. Do you want to bring in the cluster like a public cluster or private cluster? Private. So which means the A guys server is also like only private access. Yeah, so my client cluster that needs to be secure is private. My personal, like the you will probably still find some public clusters. I just want to guide myself before you attack me. But yeah, everything is private and then the problem is how do you use CI-CD and things like that. If you use hosted solutions for CI-CD for example, sort of CI and things like that. It's going to be hard if everything is private because you need to have some way to hit the API, right? That's why I run my own CI-CD. Yeah. Unfortunately, yeah. Sorry? Fascist. Fascist. Well, SSH tunnels maybe, VPC peering, if you have CI-CD hosted provided that allows you to do some SSH tunneling to hit the API and things like that that will help. Otherwise an interim private CI-CD. Okay. Any other questions? All right. I also didn't talk for a while and now it's out. Thank you very much.