 For the next talk, we have Thomas Fricke here, who is hacking containers and Kubernetes. So as I learned from him with a little chat up front, it's a little effort to hack it, but it's also a little effort to write a few lines of code to protect it. So applause for Thomas Fricke. Yeah. Welcome, everybody. So let's go into the details. That's me. I'm partner, and I have been CTO at Endocode, and now I have the title of the chief cloud wizard. This means absolutely nothing, but it describes mostly what I do. I've dealt a lot with system automation, DevOps, cloud databases, and software architectures. So that's what you can ask me about. Personal history is I have more or less the time from the Apple II to Atari, then digital or open digital OSF Unix, and I've been using Linux since 0.95, actually. So I'm pretty old and have a long history, and by the examples, I think you will notice that a little bit. Okay. A disclaimer. We have partnered with CoreOS, which has been bought by Rated, which has been bought by IBM, with Rated themselves, and Google. So our business is consulting, trainings, workshops, and audits on Kubernetes. And this talk is not about the Kubernetes code audits. It's just about what from a user perspective you can find in Kubernetes and where you have to be careful and what you can do to protect this entire ecosystem. So the history of Kubernetes is exciting. The project has developed really fast and was really fast. I mean, in 2013, there was nothing like this. The Borg project in Google in 2014, they published it, and immediately IBM and Rated joined and forced it. Then it was a little bit of testing how it works. We did our first Kubernetes application in 2015. This was Telephony in the Amazon Cloud with all the G10 stuff. The G10, which protects your communication. Then we went into the design pattern mode. Helm came out. And then a lot of training started. It was like introducing beginners to Kubernetes. And from the history, you see the Borg system. You might remember 7 of 9 in Starship Voyager and Star Trek. The next generation has been the Borgs. So 7 of 9 inspired the 7 spokes at the wheel of Kubernetes. So it's actually the friendly Borg system. So this internal joke. We started with security trainings in 2016. All the applications more or less have been toy applications. This has to change in the last year. People came to us from a security critical environment. This means energy control, trading, traffic control. And at this moment we recognized it's time to make Kubernetes and the entire ecosystem really secure. And the talk is how you can do it. And maybe I want to inspire you to look into the stuff on your own and get a real impression on that. So this is where Kubernetes runs at the moment. And I missed the back ends of the car industry. And I even know some projects which I don't support in the surveillance industry. Face recognition in Kubernetes for airports is actually something I don't like. But it's hosted by some of the usual suspect companies. For those of you who have not heard about Kubernetes, it's a Greek word for hamsman. Same root as governor and cybernetic. And for you, what is important is it's 100% open source. It's written in Go. You can change it on your own. You can install it. You can compile it. It's not a product as a Linux kernel is not a product. Some people make products out of it. So the main message is Kubernetes manages applications and not machines. This slides from Google. So this is a 10,000 foot view on the left side. You have the user clients which connect with API or command line interfaces or user interface. Then you have the master servers which only have data which is persistent in the ETCD part. The ETCD is a distributed database. And you always talk only to the API server and everything is stored in the ETCD. So all your secrets, configurations of the entire cluster, the state of the cluster is in the ETCD. Schedulers and other controllers are starting processes. And on the worker node side, the very right side, you have cubelets which more or less take over the role to orchestrate Docker or any other container runtime to start your processes. And this is what you should really, so this is the marketing care about. You have the user side, the API and the container cluster. Okay. This is not what we are doing on this kind of events and congresses. Let's look into it. My main message is, Kubernetes is part of the DevSecOps landscape. DevOps is more or less dead, but it is revoked as GitOps, DevSecOps, SecDevOps or configuration as code as you want, how you want to name it. It's not well defined. And together with the typical stack of security, so confidentiality, integrity, accessibility, Kubernetes adds automation and what most people don't know, the possibility to audit your cluster out of the box in the latest version. So these are the missing things. And there is no good definition of DevSecOps. So for me, it's a philosophy. You take DevOps and add security to every step. And this means it's an educational process. It's about the mindset and mindset needs to be implemented in what we are. So in your brain, it must be clear what's going on. And if you have a map, what all belongs to DevSecOps, this is a typical map. And you see Kubernetes is doing a lot of this stuff. And cloud providers, and you can write your own cloud with Kubernetes, need to do the missing part. Just to remember, this is a complex map and we have to deal with it. So this is the background. Let's go technical. What is a container? So it's often compared to a light-white virtual machine. This is true and not true at the same site. For developers, it very often looks it's a virtual machine. I can install everything in the container. From an operations point of view, it's the wrong approach. Like you see on the right side, you can do it, but it does not fit very well. So from an operations side, you should follow the 12-factor philosophy. One process per container and have a well-defined behavior that the containers can be controlled. So what are containers? Actually, for me, it's not only, oh, Docker is a container. Yes, it is, but containers are a way of isolating and restricting Linux processes. You do the isolation by namespaces, capabilities, and have restrictions like in the C groups or second part of the Linux kernel. And the layout is what you have on the left side with virtual machines. Always needs a kind of hypervisor, typically QAMO KVM. Where you have processes, you can emulate every operating system. On the right side, you see containers are running in the Linux process space. So the real Linux processes in the Linux kernel takes care about the isolation. Whereas on the left side in virtual machines, you have simply the Intel or other processes that guarantee that you isolate with ring zero and other stuff. Linux namespaces are not so well known as intended. So here are the typical Linux namespaces. And you have all the namespaces. Most important, most important, you can set the namespace per process individually. You can have processes in the same network namespace, which are completely isolated in a process namespace with a typical thing, which is something you have to have in mind, because the Kubernetes ports are designed in a way that they share common network namespace. Capabilities are also something which is not so well known as the Linux kernel. These are 38 capabilities in a 64-bit area implemented in the kernel. And you have things like cap net admin, then you can change the network layout, can run IP tables, set up private devices, things like this cap net row, which gives you raw access to the network. CapSys admin is in root here. If you have capSys admin for a process, it can do everything. CapSys time, for example, can change the time in a container. You normally have the time, same time as in the host machine. So with capSys time, a container could change the clock. One word of warning, there is a lot of wrong stuff. And unfortunately, even the Bundesamt for Sicherheit in the Information Technik does not know very well about containers and Kubernetes. So it's more or less 50-50 truth and false, what they say about it. And unfortunately, even the EX article in the Heisen magazine is not really helpful. So don't believe what's done there. Just think of your own. And this is going to explain what is a Kubernetes port. A port is where you share the network capabilities. This means you have in both containers the same local host and the same IP tables rules, the same DNS, the same whatever tunneling if you do tunnelings, and everything can be separate. So one container could have access to a different part of the file system. Definitely has different users or mount devices, process IDs. And this is an example. On the right side, you'll see the simple spot. It's just starting an NGINX server. You should simply know a single container is always a port, but containers are really only good if you glue them together in a port. This is an example where you can ensure, can secure an SQL connection with a WordPress server. So I don't like WordPress very much, but if you have to create a WordPress server, it needs access to a database. And here you see on the left side, the green part is how the WordPress server is configured to use a local host, my SQL database. But it does not have a database running. So the right side is another container which is Cloud SQL proxy, which goes directly into the Google Cloud and connects you with additional credentials, which have a different life cycle than your passwords, and encrypt it and open a TCP proxy connection. The TCP proxy connection is secured if you don't like the Google approach. The Cloud SQL proxy is open source. You can, as everything here in this talk is open source, you can use it and redirect it to your own database cluster if you like it. It's not as difficult. So this is a use case where a port makes the database connection more secure than before. The secrets are stored in Kubernetes, so here is a way of generating the username, password, secret for the MySQL connection, and below you see the certificates and private keys for the Cloud SQL connection. So this is typical stuff if you set up a database in Google Kubernetes engine, JKE. If you want to look deeper into this stuff, here are some things I can recommend. For example, if you don't like the BSI stuff, start with the NIST stuff, so the National Institute of Standards from the American government has published a very useful report on how to set up Kubernetes and other container systems secure. So the security architecture is based on a very subtile system with users, service groups, service accounts, namespaces, roles, role bindings, and when you have pod secrets, cluster roles, role bindings, all the blue stuff is just an object in Kubernetes. You can also connect the Kubernetes service accounts and all the stuff to the Cloud identity and access management IAM. This has to be implemented by your Cloud provider. If you want to be your own Cloud provider, I would recommend to implement it on your own. The next thing is pod security policy. It's more difficult because here you protect the local machine from something inside the containers, especially the data layer is important. If you configure Kubernetes completely wrong, then you get access to the host file system and then a privileged container can do more or less everything. I have an example here how you can get access to... in a few minutes... how you can get access to Docker sockets. This is a security system here. What you should consider is setting up your own admission controller that you definitely control what's going on. The pod security policy implementation isn't admission controller. This is controlling the way you start pods in Kubernetes and it even can change the configuration of a pod or a refuser. This is recommended things you should read. If you start deployments, one thing I showed you is the capabilities. Docker has a default set of capabilities which in our opinion and most of the people is completely useless. So drop all the capabilities of Docker which are mentioned there. For example, you have some things like deck override. This is a disk foundationary access control. In Linux, it allows root to bypass, file rate, write and execute permission checks if it's forbidden for root. In my opinion, Steve Grapp is absolutely right and says nothing should need this. This is complete nonsense. If your container needs this, it's probably doing something horrible. So please switch this off. The only thing you could use is a net bind service which allows your user space application to use privileged pods in the container. But if you use unprivileged pods, you don't even need this privileged. Yes, the privileged system is complicated and we have even things like the exact VE system can escalate privileges if you don't have a flag which says new prifts added to prevent systems like this. This is in the Linux kernel 3.5. It's not a good idea. It's absolutely stupid idea and it's exposed by Kubernetes. If you say, I don't want to do privileged escalation, then you don't get these privileges with the following exception. This is weird. When a container is privileged, okay, I can understand this. When CAPSIS admin is added to the container, it makes sense. But when a container is not run as root and UID 0 is added, so this privilege is added to prevent breaking said UID binaries that you can do as a Zoodo in a container which is nonsense. This is complete nonsense and you should switch it off. Unfortunately, you have to check it with this cat proc 1 status in the second line here that it's not really getting escalated. Thanks to Andreas Peters-Petty from Spiegel online pointing to this issue. I have shared the link where this is discussed in detail. So what can go wrong with some metal configured cluster? You all know the bash-bomb. Here is the cloud-bomb. If anyone has a Kubernetes cluster up to version 1.6, this probably will work. Don't try it in production. Still works in MiniCupid without protection. Here you can start from inside busybox, a Docker process, getting access to the Docker socket and running the Docker binary from inside a container which is nice if you want to build containers, in containers, a security point of view. It's a serious flaw and it should be definitely turned off. We have talked a lot about complexity and everybody agrees complexity is worth any me of security. This is a site from Bruce Schneier's interview in Hong Kong 2012. What he also says is we absolutely love complexities. This is not... I don't want this kind of thing. We have to discuss it. Effectively, my personal conclusion is a philosophical or religious discussion. What we need and what we want is different. We can talk about this, but this is not a technical issue. Here is a network view. You have been warned. More complexity. What you see as a blue part again is what happens in Kubernetes. The example here is if you set up an Ingress which controls the way Kubernetes is accessed by the outside, you can use a certificate in your cluster. In the Google Cloud, it can be connected to regional or to the global load balance which is unseen, as I think, and then you get a global rule to your cluster, which means you have exposed the TLS certificate to the Google global load balancer which is implemented as Maglev which will be in talk on its own, and then you get a certificate. It takes 8 minutes or 10 minutes to get ready, and then Google injects the traffic into your network endpoint groups which gives you 40 gigabit per second performance on the high end. So this is definitely what you want. On the other side, this complexity must be handled somehow. And if you want to write your own Kubernetes application for your own provider, you have to implement kind of custom controller which definitely takes over these roles. On the light blue level, this is on the Linux level. You see here running containers they are orchestrated through container network interface. You have traffic rules, border gateway protocol, and all this stuff. The bottom layer is the hardware which is configured through your Kubernetes stuff. So we have virtual machines, physical fabric, software defined network, virtual private cloud VPC and things like this. So it's a perfect buzzword being what I'm doing here, but this is more or less the main part of the network system. Okay. What we do is customers come to us or we need more security and then I started to offer audits which means we look into Kubernetes clusters of our customer and give them advice what could be better and what should be fixed or start with writing a security model. You don't even know what you want so how can you implement something with security? These are the top findings, the first thing is storage which is not part of Kubernetes. Then images, installations, pod security, audit logs and networks, I will go through this. Most pain point is everybody does distributed databases wrong. So here's the reason for it, if you want to have a positive talk how you can do it right, look into the Berlin buzzwords talk from Bloomberg, Houston, Putman did it quite right I think and anything else is try to understand at least the theory. Just the beauty database has a cap theory. You can only have two of these three parts and replication is hard even if you do it with MongoDB where replication is implemented the easiest way so try to understand it and databases are not well-factored don't do it in Kubernetes if you are not knowing how to do it outside. Next thing are images so you need image policies, registries, image stream things like this and what we see here very often is you are running code from an unknown source S route with full access to the cluster resources which is a terrible idea. It was always a terrible idea and this includes something like kubectl create minus F which can be hijacked, this would be a different talk but it's actually it's a 30 minutes exercise to do it on your own to implement it in any language if you want to learn go it should take 30 minutes and take I don't have time to go into the ham chart part it's the same for the ham charts. We have seen malicious code here in 2014-15 there was hardly any ghost in images and we would never forgive Microsoft for such a mess and this was the problem two thirds or three fourths of the container had high or medium priority insecurities in the docker hub in 2015 and you can protect it by adding special checks in your system in clear or every cloud provider nowadays has something like that and unfortunately it wasn't better in 2018 and I would not give a coin or give a send on that it's better now so be aware check your images you have to set it up your own the next thing is the cluster set up setting up a cluster is easy but maintaining it under production level is hard so keep the cluster up to date you need five people for this we only see two in the projects you have to manage service accounts and users please automate everything in your system add for the version and try to deploy on the latest version of kubernetes always because kubernetes is lowering the security so it might be that you have an application which is not running on a future version because the security has increased put security add put security to your system definitely limits liveness when it extracts our best practices especially don't switch off se linux or app armor use the host file isolation I show you something like this if you want to check for privileges this is a cube ctl command which is quite easy here can you check for the kubernetes security context especially in it containers and this is a go template version and here you see the red things are there is no such thing like security context the greens have good ones typical mini cube run so it's quick and easy to check it check your run time so docker is still the default but it will change this here that kriu is in ud4 rocket is completely outdated you can now at least in the google cloud have a one click solution at gvisor gvisor isolates the files on top you see what you do on a typical system without gvisor and with run time equals run as c you see gvisor is blocking most of the critical access in your system so you can add it and in the future the good news is the run time class will make it configurable gvisor here this is a restrictive port security policy please apply something like this this excludes everything allows only some important and secure volumes and blocks the host network host IPC and things like that completely so the good thing is on the other side you also can add audit logs to your system for example if you want to do step sec ops in health care environment well the game was you build it and we have all kubernetes yaml files in git secrets in a special secret branch you can add the audits to things audits means you can see every time and somebody access in a container or changes something it's noticed and it's written to a logging system in the google version it's stack driver then you can export it to big query then an auditor can only use sql queries and to inspect what's happening in big query if you don't like google you can also do it with elastic search elastic search has a raw base access model on its own you can use it and simply do it this way okay next thing is the network network policy is a must see here is you can add calico which is add network policy in the google kubernetes engine by d4 and here you have the picture every time something changes in the network configuration of a port the cni the container network interface is called and then it changes the network layout which means changing the routes border gateway protocol ip tables the entire physical fabric is actually very fast but it looks complex network policies are like ip tables rules on labels namespaces and ports so this is a typical network policy rule matching roles like the database should also only be accessible by the front end things like this so if you know ip tables this is something you'll really get into the next hot thing are service mesh so buzzword is istio again it's one click solution in google kubernetes engine it allows private public mix clouds so like gke on premises it's also necessary for k natives so the serverless application on top of kubernetes it will be part of open shift 4 it has developed mainly by google ibm and wetted so looks important big names impressive architecture should be secure let's look deeper into it this is a picture from istio shows secure defense in depth 0 trust network all the buzzwords and what they are using effectively are sidecars so here is the layout of the port where you have the istio envoy container which does a proxy work and what they don't mention very public is they running an init container changes the ip tables rules and it's in user space effectively so it has a net admin capability net admin capability is some people say it's nearly like net sysadmin and on the side they also have a ca and citadel and all this stuff so let's look into it this is a picture of a service mesh so from right here you can configure everything you are running sidecars sidecars control the network your microservice application is not is not able to escape the rules of the sidecar really here is how you can escape from the rules so sidecars are used in user space so you can revert it but simply running this ip tables rules from a different container and it takes the istio proxy container itself to remove these rules again and here you see how we classically remove ip tables rules and then all your network security is voided and this is really a problem for the cloud so you have added network security but you have lost multi tenancy because if everybody who can deploy something like this can remove the network security you have a problem so it should be added at least the pot security policies that you are not allowed to do this so I found this file the bug in January included this exploit disclosure limit of six weeks I forgot the issue then I received an answer in March so eight weeks okay seven weeks from Tau Lee Google Florida and I was not the first who invented this bug but I think this is a script curly approach it has been confirmed and they told me that in istio one dot two the container network interface will take over the thing so the same thing which is more or less not in user space is done now and the container network interface can be used and cannot be controlled from from a user port so istio one dot two is ready now but it's not rolled out out everywhere so it's still a side car so be aware that you have to fix something not many people are running istio at the moment fortunately but in the future we will see more of our customers doing this implementations with istio so the conclusion it's lengthy but actually it's a lot of fun working with Kubernetes and for me it feels like working with unix and linux in the early 90s so you see a lot of low hanging fruits and security if you know the good old system engineering and you question the marketing slides you can find a lot of flaws I think and I did not even look into the go code DevSecOps in general is a good idea we see it in our customers projects but don't forget the sec if you dive deeper in a system like this you will find flaws it's inevitable the flaws are presented here the docker flaws are parts of docker and not part of Kubernetes and part of the weird capability system of linux which simply should be cleaned up you will see a lot of buzzwords and you will see the typical thing you focus on one security aspect and then somebody has a back door and does something completely weird behind your back trust nothing for me it's a buzzword you always have to trust somebody you have to trust the code you have to trust your cluster you have to trust your users if they are doing but okay you can trust no network cluster now you can support it if you look deeper into it by writing and responsibly disclosing a plot the community is really friendly and supportive so you can get back some things and that's everything what I wanted to tell you so start using it use it responsibly and just have a way of looking into the security so if you have questions try to stay above all these clouds giving a workshop on this in the sea base area at 3pm so let's go for it okay thank you very much questions has anybody a question please raise your hand is there any questions from the internet no questions from the internet okay go right away what's the security issue with that is your sidecar in particular actually you can switch it off by deploying this net admin container additionally so it's just in user space you have trust your containers if they would have done it right they would have written an admission controller which does not allow a net admin capability on its own would refuse this kind of container and then add the sidecar container it's just an internal attack so you think you are secure but you can simply remove all the security so it's not so the proxy container is not controlling your connections but it can show you the exploit working if you come to the workshop the actual application itself is the actual application itself is still running unprivileged yeah it's unprivileged but for example it can telephone to the home base so it definitely can circumvent all the egress and ingress network restrictions if you got project level permissions me you don't need project level permissions you need simply setting up this cluster and then you do the Istio injection and then you this container is running after Istio has injected the security rules and then it's removed again thank you welcome hi there thank you much that was a great talk I'm a CNI maintainer by the way so hi have you seen cube let me in so anyway we can talk about afterwards but there are some well known vulnerabilities in most Kubernetes providers and if the project cube let me in is script kitties for Kubernetes and it can exploit like GKE OpenShift those things like that I believe that it's possible if you collect all the things together you have a nice collection of this kind of stuff any more exploits you find we can put it together yeah great thank you thank you please go on but please talk a little slower it's hard to hear on the stage thank you for the talk it was very interesting I was wondering if there is any way to make helm the package manager more secure or you don't recommend using it at all what I would recommend is to limit to a certain namespace so you have a very well known option minus minus tiller namespace and then you can inject this tiller application which is really not necessary in the future it will be removed in helm 3.0 and then you have it in the namespace only and then you can the damage is limited to this namespace so normally now it has full system access and you can remove everything and this is a problem any more questions from the internet ok thank you very much I close this talk now big applause for Thomas Fricke