 And yeah, let's talk about cloud native deployments in air gap environment, which is a contradiction in itself. So the title contains a contradiction. But we try to make it possible. Who I am, it's not really interesting. I'm doing Kubernetes for seven or eight years now. From the very beginning, founded companies consulting in the energy business and healthcare and the German government. Created some pro bono projects, but this is not interesting. So this is a slide. What are we talking about? And we are talking about critical infrastructure, which is now here the German definition in all other countries, there will be a similar definition. So we have, in German, we have this criticism regulation and it's a very rough classification just to introduce you. It needs more than 500,000 people are involved if something goes really wrong. This is a very bad measure because then this means that the insurance for your glasses are more important than the waterworks. And definitely you can live without the insurance for your glasses, but not without water. Waterworks are mostly too small. And then, yeah, we have to deal with that. I'm also doing something in transport and traffic. Traffic lights, which is also not necessary because traffic works better if the traffic lights are switched out, this has been proven. Exception is tunnels where you need ventilation. But the most important area is energy. So if you lose energy, then we have a real problem. And this means if you want to do covenators in critical energy, we need to be especially careful. Why? So the priority zero is the transmission grid, which is more or less managing the electric current on every level. And what I'm talking about is the upper part here where you have 110 kilovolt and up. This means this is the area where the power plants are, where you have all kinds of medium and large industrial plants, something like this. And this is attached this way. It is run by transmission system operators. And effectively it's a very sophisticated IoT. And IoT, you know the joke, the S in IoT is for security. And that's the same here on that level. We have to fix this by isolating the environment. And we have to install everything air-gapped. So the isolation means we have a special need to work in a completely air-gapped environment. So security requirements are complex. We need a lot of resilience against floodwater, drought, topology on a European scale. Very often it's assumed, yeah, the transmission grid is just a copper plate. No, this is not the case. We have to include every transmission line and have to do it really carefully with very old protocols, SCADA from the 90s. It has a lot of operational technologies and the future there will be smart meters and the highest what we have at the moment is 500 kilovolt. And yeah, a lot of IT is involved of course. The future will even be more complex. So if you have some devices here and then you switch to that level and in the future to something which is called vehicle to grid, you have lower voltage, but you have much more local networks, much more devices. The energy flow is B-directional. So you can load a car and you can use a car as battery and you will have a very complex security demand if you have followed the news. Some people needed two minutes to hack a Tesla and if you think you have one million of these cars producing energy or consuming energy in the transmission grid and they can be hacked, you'll see what kind of problem we will have in the future. Why Kubernetes? So yeah, of course we need more control. We have more devices, we have uncertainties in this area because we have to manage the weather, sun and wind for photovoltaic and wind energy. Locally you have consumers and in the future producers. And we also have energy trading in that business. So this means depending on the price of the energy, energy price even can negative, then you get paid if you consume energy, adds a lot of need for flexibility and the computing power can only be done in a very complex system like at the moment the natural choice is Kubernetes. So we have all these kind of devices and stakeholders. On the security side, this is one of the most security critical systems you can have. We have seen this in the last year when Russia attacked Ukraine. And what happened is the Ukrainian military and the German ENBV, which is a local energy provider, lost the control about wind turbines because they used the same satellite as the Ukrainian military. And what does it mean if you lose 5,800 wind turbines? This is the equivalent of 13 gigawatts. And it is known that the grid stability is designed for maximum hit of three gigawatts. And the modems have been bricked, so nothing happened immediately, but they lost control and they had to go to every device and reinstall software, update things and definitely get back control. Also physical attacks to operational technology in the field are possible. We are seeing drones around power plants or transmission systems and all the systems are quite old and do not scale. So the rescue is on the long run use microservices. And now we have to talk about the security model and effectively this is the security model. Here you have the internet and then you have more and more secure zones until the very restricted high security zone. But effectively if you compare it, this is a security model. It's a medieval castle. You have moats, walls, towers, bridges, canes like this. This was quite stable, so it surrendered after 18 months of siege. So actually it's not as bad as the security model unless you think, yeah, this is your battlefield. So there is no way of doing moats and walls in a sea battle. And we have this system here. So we have standards. For example in Germany, you have this BSI Grundschutz about continentalization about Kubernetes, which is a subset of CIS and NSA demands and what completely misses in the moment is some security for supply chain. So we don't have a collection of protections for supply chain and to the rescue, I always use the DevSecOps Enterprise Container Hardening Guide and documents from the CNCF and the DoD, which is the best we can have but we don't have a European or even German regulation at that moment for supply chains. And this makes it difficult. So let's look what does CNCF and the DoD say they use the DevOps process and added security to every part of the DevOps process. So this is DevSecOps. You will see it starts with a threat model, secure coding, security as a code, SAS does, so this is more or less over step. This is a SonaCube, pen testing is manual, digital signing, secure transfer, configuration scans, patches, in production, audits. This all is necessary if you want to implement this process. I don't know a single company who has implemented everything, but you can try as most as possible to get as secure as possible. So this is Department of Defense and the CNCF who are supporting this. Another thing you see in this kind of environment, I all want to implement the software factory strategy. Software factory is not completely defined, but it's all about enabling, and this is a cloud native part. You want to enable by using private or public clouds. You want to see ICD pipeline and you want to have the DevSecOps flavor of DevOps. And here in that area, this is a kind of agile part and agile is also part of the software factory strategy. So you have to implement all this and this normally introduces in an environment where this is not happening, it takes five years. Effectively, two years, if you only cause 80% of the core functions, but in such a conservative environment, it takes five years. So what can we do? Yes, to get everything into that environment, we have to check and scan everything. So the idea is to have one cluster which is at the border of your castle more or less and here you scan everything, you check everything and you deploy everything on every cluster inside. And then finally, you also report everything which is happening to Splunk or Elastic or whatever you can do to get the information out of this cluster. So let's go through all the components. First step is Harbor cloud native project and it has a capability now of definitely scanning images so you should scan every image which goes into your environment. This means we have to find a way that we use only images which are hardened and comply with our security regulations. The next thing what we want to do is kind of GitOps so we want to have a Git available and then this is the argocon, what comes into the game if we want to deploy it with argocd because normally it's easier if you have a graphical user interface. Normally the people who are running these systems are Windows administrators and then it's better to have something which is understandable for them. Finally, here you see we are logging everything so it goes into either Splunk or Elastic depends on your budget or your manpower and you can definitely get information about everything. So you know that, unfortunately I cannot show you real system from a customer but this is the idea you can deploy everything with a single mouse click and even do a rollback if you want. The setup of the management cluster is designed for easy maintenance and high security so Harvard, Trivi, this means all images are scanned. It also serves as an internal registry and we are exporting CVE lists. This is a funny thing because if you're able people to create CVE lists, the last thing I want to have in my private email is the CVE list of a critical system because CVE lists are normally highly sensitive because it contains all the information you would need to hack into the system if you're already there. Please don't send me your CVE lists. This is something I always have to introduce and into this project so yes, I don't want to be CC'd with this kind of stuff. Better you don't send them anyway. The only thing you should do is export them into some kind of monitoring solution and don't mail it. A git is necessary if you want to make a git so git is one of the first choices but we're now looking into Forgeo. ArgoCity is definitely as we have seen the most promising project for rolling out githubs with the user interface. And monitoring, with monitoring you get definitely this continuous security monitoring. Continuous means daily at this moment. And this is a central place where you can look into all your changes. For example, Harbor has an interface to curl the CVE list and then you can export the CVE list and push it directly into monitoring solution. And then the monitoring solution can be used to alert you if a new CVE is introduced last night. And then you can decide how do you want to deal with the CVE. We are in a critical system and this means if a CVE appears we cannot turn off the critical system. So we have to deal with mitigation. We have to deal with pushing a new version rolled out without all these vulnerabilities. So these are the tools we use and finally Argo is the tool to roll it out. What are the results of the things from the trenches? My personal record was an image with 735 vulnerabilities in critical infrastructure. 20% of them are critical or high. The typical Microsoft containers based on Ubuntu if they come out of the box have 140 vulnerabilities. And I'm a big fan of strict enforcement of security so we have to deal with it because if you allow this enormous number of vulnerabilities and you have not a single container, we have hundreds of them. This means security is completely blinded by false positives. So you find in some of these containers you find development so this 735 vulnerabilities means we had a development environment in that container which is unnecessary. And typically you have two versions of Python which is unnecessary. And we are pushing back and then you see okay it's getting better and better and sometimes you see regression like from one day to another we had a lock for JShell container in critical infrastructure. And then we talked to the vendor and our first statement before that was our Caesar would never allow to deliver insecure containers and then you can show him you delivered as a lock for shell. What is your statement about this? Somebody it's sometimes even embarrassing because third party vendors are lying to you and then security really becomes unpleasant and this is not a problem I have. I can become unpleasant too if I see this kind of containers in production. And we have definitely to improve our security culture this is out of the box because I cannot show you the internal containers of a critical system. This is a Tomcat 9 out of the box with a lot of critical CVEs critical and high and this is not the state of security we can tolerate anymore in the future. If the bad guys are done with Microsoft Exchange and Outlook and they will turn to Kubernetes and we will all be punished if we tolerate this kind of containers. The limitation of this concept we are not able to detect architecture flaws. So lock for shell one wasn't JNDI architecture flaw. We have to introduce and inform separations of concerns. This is an old version of Elastic before they introduced operators. They just in the very beginning the init container needed special system CTL command and then if you separate it you can have a secure version and a privileged version which is just going into Paul's statement as a demon set. The experience all to collect it the entire system works. The impact of scanning containers is dramatic for the project. So I'm running a government project at the moment and then we talked about enforcing distrelas containers because it's the only way to get rid of this crap and this needed definitely some time because then the developer said yeah we need more or less have a year to get our stuff distrelas. This is okay. It's an effort. The customer, the German government will pay for this but it has to be done. There's no excuse. Argo CD was one of the nicest projects integrating into that workflow. You can give a very fine-grained access to every cluster. I have user trites per application. Takes one day training to educate the staff but not every ham chart works out of boxes. This might change. I hope for the next versions and also central monitoring helps a lot in this systems. One major problem in all these environments is the authentication. You need to configure log ins and IAM for every application. So problem is the customers very often use active directory then you have to configure it four times. One times for your Kubernetes, one times for Harba, one times for Git, one time for Argo CD. So there is definitely a need for having a unified IAM for all applications. If you go into OpenShift, there is one of these integrations but this is not the standard. And active directory is definitely a major pain. You have to isolate it because you have Python one-liners which are able to hack into active directory and take over the total control. So you need network policies just in the case you have an active directory to isolate the active directory from the workload of your cluster as a side effect. The next steps in the future, we are not done with it. We want to sign everything. So we want to use six store and sign the artifacts. Definitely also uses the entire certificate authority. We want Rekor as transparently locks because I don't want artifacts in the systems where I don't know the history. It could be that a developer signs something and it doesn't show up in an audit or in a review and we need to know what's exactly in that in that portfolio. It's monitoring and route trust. So we should have a good chance of doing this. So what will we do? We will start with get signing and we sign the build artifacts and it protects the system itself. And then in that in the future we will promote only signed images. So as a Kubernetes container one time we use probably Creo. We can use Hama or ROCD and flux just to support signed artifacts. And this means here in that picture we can guarantee that here a third party vendor has signed something and only signed images go into the system and they will additionally scan. Then we have the signed images here. We will use ROCD for just promoting signed Git help charts. So this is the place where the help charts live. If we could have OCI images as help charts then we would use this mechanism and then it's only promoted if everything is signed. This is the future and this is part for example of the German cloud native government strategy. So every image there must be signed. What we are doing here is we are transitioning an active system to security by design. And this means security must be easy. We don't see a corporate security culture. Sometimes we have to really do a hard enforcement and we see a lot of movement in the coordinated security so mostly to the better. Basic flaws are removed. We see tools, scanners, config signatures and ROCD here as a cornerstone. And that's it. Thank you for your attention. Here are my contact addresses so you can mail me or get me on LinkedIn or on KOS social master done. And at the bottom there are the resources for the links so security trainings, how to harden containers and how to deploy this management cluster. Thank you very much.