 Hello everyone. My name is Ohtem and welcome to the Open Source Summit, Latin America. In today's talk I'll present you the building secure by design pipeline with an open source stack. I would like to walk you through the process of building an end to end security pipeline. And, you know, and want to walk you through from misconfiguration in infrastructure as code to Yama and to overlay permissive CPU limits and even securitizing between development and production. We'll take a deep dive on how secure code and configuration the pipeline itself and its integration and our Kubernetes deployment to ensure we have continuous visibility with the right monitoring controls right in place. I will demonstrate it all with the common open source tool, of course, and a bit of coding interface. So before we start, a few points about me. I'm a developer for the past 14 years or so. In the past five years, I'm in the DevOps industry, especially Kubernetes expert. And in my free time, my hobbies are yoga and watching my favorite basketball team. So that's all about me. I'm working currently in ARMO, which are the maintainers and builders of CubeScape, which is an open source end to end Kubernetes security solution. We were founded in 2019, and we based in Jerusalem in Tel Aviv, Israel. And, you know, let's just get started. So basically, a word without CITD is a chaos, as we're talking today. DevOps and modern engineering have enabled us to provide higher quality code at the greatest speed by introducing guidelines and checking to automate this continuous CITD processes. However, with the security becoming a more pressing matter and more critical zero-day traits arise and misconfiguration is basically introduced through production system and the same time application and development processes are moving fast forward right in the CITD process It's becoming critical matter, critical point for enforcing security during the CITD pipeline and not just, you know, at the end and the deploy in our clusters in our dev production environment. So I think we're all pretty familiar with this diagram, right? But as we are familiar, I'll go through in this talk about all the phases and try to connect the security guide in there. But speaking of that, Darian, that you all very much familiar, do we really know what's the CITD goal? So I checked a bit and I found that the CITD pipeline goal is to reduce the risk involved in deploying software and plan our problems ahead. What does it mean to plan our problems ahead? Okay, that means we will have problems, but we need to find that out in early stage of the pipe, in early stage of the deployment, in the planning, in the coding and not when it's getting into production and that talk will focus on this topic mainly. And of course, reducing the risk. Okay, I don't want to, that the risk will be higher as I'm going to production. Okay, we want to reduce the risk as the early steps are involved. So let's go on to our CICD pipe steps with some examples of which solution there are within those specific steps, although one by one and explain them. So source code, what do we have there? So there are basically two directions of source code. The first one is the applicative code, the application code that we are writing and the second one is the infrastructure as code. Okay, I combine them in this step together, but we'll talk about it separately. As you see here in the icons, we have Golang, Terraform, which is infrastructure as code, the YAML manifest file that we're deploying in our Kubernetes cluster, Helm. We're writing in the visual studio code and any other IDs. And then we're going through and pairing our code to some kind of a repository, GitHub, GitLab, BitPacket, you name it. And then we're stepping ahead to the CICD server. Okay, I just presented here the Jenkins and the GitHub actions. There are a lot more, I will focus on those in this talk. So I'm building my application, okay, in the CICD server, I'm building and wrapping it as an image or docker image or any kind of image that I want to wrap that. I want to put in some place that I can just pull it from there when I need and, you know, I'll save there the history if I want that or not. So I want to put it in a repository, in a registry of artifacts of all my images, there are lots of examples here. We have it for the Microsoft one, we have it for the Amazon one, Docker have Harbor, there are many, many, many registries that we can dive into them. And then we're getting to the deploy phase. The deploy phase is basically the phase that we're putting this image inside our deployment environment, right, when we want to take this image and deploy it in the cluster, in the Kubernetes cluster if we're talking Kubernetes wise. We are deploying it with several tools, of course, we have the Terraform that helps us to deploy that, also the Argo CD, Ansible, and many more tools that are helping us to do so. And essentially what you want to understand is, is my application okay, right, is it alive, is it increased RAM or CPU, or any other metrics that we want to monitor. And that is why we're using Prometheus and Grafana as its visualizer to monitor application, most of the application has their own Prometheus exporter, and that's why that's how we're monitoring it. So what we're seeing here is basically a type, right, that helps the developer to rapidly deploy, right, they're just, you know, pushing their source code and that's it, automatically, right, it's right there in the production I can trace it and see, you know, how my application is going, right. In a few minutes I can see my application in production, even not in dev environment. So what developers are doing is basically shifting right as we're going from the code to the, to the deploy part to the monitoring even part. But if I want to drill down a bit more I will present the, you know, the detailed pipeline that I created here. It's just, you know, a user flow that I created in, and we will talk about this user flow basically the entire talk we will focus on each and every one of those sections, and see how can I have the security inside this, you know, it's not a small pipe so I want to have security in most of it, or part of it, if you say. So, the developer right here is sitting and developing his code, and then he develops it in go or, or creating any manifest or deployment files to support his application. And PR in this code to his application repo, and, and PR in infrastructure is put to, to that repo. After that, the application, the PR is approved. And then it's one to the CI CD server, right, I can see here the GitHub action and Jenkins, I'm building my application, and I'm making all kind of tests if you want, and then I want to push this image this new image to the container registry. And from container registry and from the infrastructure is code is combined with going on just presenting here I go. Sorry, I'm just presenting here I go but basically it's automatically deploying taking the images taking my manifest and my terraforms and deploying it in my Kubernetes clusters. Okay, one many, it doesn't matter. And eventually, the user wants to see in the Grafana, the, the metrics about his cluster security metrics or any other metrics. So, basically, I think that nothing can go wrong, right, not a lot can go wrong. We're talking now on security matter. So what can possibly go wrong in this case. So, again, there are a lot of, there are a lot of points of security pain points that we can find in, in each and every one of the stages I've shown before, called misconfiguration zero day CVE is dependencies management And of course, when we have kind of a collision third parties that we're not familiar or just importing it and we don't know which CV is right. I'm not very familiar with that. Warnings. Does anyone pay attention to warnings, even grammar warnings, we're not paying attention right there are many of us that are not paying attention even to grammar warning so security ones. Third party object as we talked and our configuration violation, etc, etc. And of course there are many more that this paper can can show. We don't want you to panic in when showing those pain points of security and I'm basically here to calm down a bit and to show you exactly where should I put the security guidelines and security guards in this pipeline and to keep me safe, as much as we can of course. So, do we have a real solution. I think we're very close to there. Okay, but basically our solution is here to minimize the risk remember what's the CICD goal to minimize to reduce the risk. So, let's drill down and see how we're going to do that. Shifting left. Okay, again, it's like a bad word now in all the DevOps industry. So shifting left is shifting left term is the effort of a DevOps team to guarantee application security at the earliest stage in the development life cycle as part of an organizational pattern, known as DevOps. If we saw before that developers are rapidly shifting right, then security is shifting left, because we want to find our security issues in the early stage, the early stage of our coding of our planning right there. So, why shifting left. I think most of us are very familiar with this with this graph this graph represents the software development life cycle. And it says that detecting bugs in a software life cycle costs less as much as it's in early stage. Right. So, the cost is getting higher when it's in the, in the last stage of deployment. That's exactly the same pattern in, in security. Okay, when we're finding security or misconfiguration issues in the right side of the map right in the, in the cluster, it costs us a lot more. In 10 days of fixes, check all the pipe from deploy to the code and see where to fix it. Then if you would fix it in the early stage, right, early stage coding right there I want to catch as much as I can, of course, okay as much as I can I want to catch it in the early stage of development, the, the featured application, the infrastructure is code. So now I'm going back to my diagram that the user flow that we talked about before. And because we're shifting left, let's start from the right. Okay, we'll start from the right from the deployment section from my Kubernetes clusters that you're seeing right here. So the deploy phase within the cluster. How can we discover what's in it, what's the security inside my cluster. So we have two major points here. The first one is the vulnerabilities in cluster scanners. Okay, we can install and deploy the in cluster component that's recognized which vulnerabilities are standing in the, in the cluster using as bomb and combining it together for example, I just run. In a mini cluster we just read this. The vulnerability scanner of anchor, and that's what I got. There are many CV is there. Some of them can be fixed some of them, some of them won't. And that's for another talk about how to prioritize the CV, but that's like the first thing about about vulnerabilities in scanner. And the second part of the deployment phase about the cluster is the misconfiguration. Again, we need to check it inside the cluster. So I'll present here keepscape, which also checks for vulnerability and misconfiguration. I'll check, I'll just show you the, the result I just scanned my mini cube cluster. This is my simple keepscape scan. And now I see that my cluster was scanned. Those are all the controls, basically all the tests that I've seen in my classroom. If I put here a minus V, then I can see it very in very detailed view I can see exactly that my resource has failed on which misconfiguration, what is the system, what is remediation, how can I fix it, let's say I don't have CPU limits in my deployment so I'm going and fixing that, or I want to check the network mapping spaces without any service account, sort of things that are that can harm me in sort of ways in, in my cluster. So here it is, we wait in a minute, it will run. And in two minutes, we will have the data live demo. Okay, so what I'm seeing here is, for example, I see here that the resource kind service account and the name is replica set controller of the system name space has failed misconfigured of data destruction. And what I see here is exact path of its failure. Okay, that's the basically the path of the ammo they related object rules, resources, etc, etc. So I see here all of that there are many, many controls in my mini cube that failed and again in by the end I can see the, the summary of all the resources that failed and where the risk, which risks are standing so I'm like, in good shape. So that's a misconfiguration check in my class I ran it inside my class and my mini cube. So if we'll go back, we talked about the vulnerability in cluster scanner and they are also tools that are doing that such as Claire and for trivia and the misconfiguration also the tree and cake kit and of course a cube bench by Aqua. So, there are tools that are doing that. So, I recommend of using that I showed you keep keep one of course you can choose. So, from deployment we're shifting left to the container registry. And what can I do in container registry I can basically just scan this registry as I told you before, we have the docker have the harbor the CR, etc, etc. And what I want to do here is to scan this registry before I'm taking this image into the deployment into my class and see whether this image after I build it right after it's after I have this image with all these dependencies is not vulnerable. I don't have the any CV is or minority of CV is so that's in the artifact section the registry one. Again, there are some tools that are doing that. I'm trying to put here some tools that are only open source one. So we can use it. And now let's move on to the CI CD server. Here I'm mentioning the GitHub actions and the Jenkins the process is well known. Okay, we're getting the, the, the code we're fetching the code from the repo we're building it, eventually making some tests and then putting the image in the container registry. So, what we have here, we have the GitHub action and the Jenkins job. So, focus on the GitHub actions here. We made it also with CubeScape also Jenkins flagging. Okay, we made a workflow for adding workflow and to, to the GitHub actions workflows that I'm running. So I'm setting up the job. I'm pulling the GCHR checking out installing CubeScape I'm standing the file and then I'm publishing and getting the results by the end. Okay, so that's for the, for the GitHub actions of you. Also for Jenkins. Okay, it's just an example for for that flow. And now we're really in the shifting left side, right. Now we will focus in code. So as I said before we have two branches, right, we have the applicative branch, and we have the infrastructure code branch. So in the applicative one. So let's say I'm using the visual studio code, and I'm writing in goal line so it needs to find some way to scan my code and see that there are no any vulnerabilities inside. So if I'm talking about the infrastructure is code then I have my yaml's. For example on my ham charts, and I want to scan that and understand that there isn't any issues in my in my call any misconfiguration. So there are two ways of looking at it. We can look at that at the source code security plugins that I'm embedding in the ID. I'm going to get repo scanner. Okay, when I'm firing my, when I'm going to PR my code, then I want to check that I don't have any, any issues in this repo before we're going to merge it and going into the whole pack. So one step before. So in the secure source code security plugins, I want to show you Chiefscape plugin, which basically it's a YAML file of some kind of our application. And what we're seeing here is I install the plugin Chiefscape plugin and what I see is on the fly by saving this file. And you can see if I have misconfiguration in the file. I can see here which, which control is it. It's the NSA test. And the reason that it's marked it is that there are no CPU and memory limits defined the container. And that tells me exactly what should I do. Okay, what's what basically the description and what should I do with that and if I'm catching it that early stage, then it's not inside my repo. So I'm catching it in a very, very basic early stage. It takes nothing to change it. In this stage, I want to catch it. Okay, the infrastructure. When I'm talking about infrastructure is called that the stage I want to, I want to do that and, and for get reference scanners, I want to scan the code. Right. So I want to scan that my goal language. If I'm scanning the applicative code, I want to scan it. What I wrote in Go is okay, no CV is in that. And the other hand, if I want to scan the repo, okay, of my IAC, then I want to put my repo in GitHub like Chiefscape scan, Chiefscape repo or helm repo and then scan that and get the results. So those are two steps that are pretty much aligned. I'm calling that as the source code, both of them because we're at the early stage and that's the stage that is for your, for your the commit and the push and that's the stage where it's a lot healthier to have our security checks. And here it won't cost us much. So, there's just another one that we missed, which is the permit is exported. So, when I want to monitor my application. Okay. There are many, many, many exporter available for the cluster inside and basically they are, we provide here the security exporter that checks the cluster in after basically the deploy phase right we're always checking if I still have the same resources the risk is still low or it's got higher and then I can see some kind of a drift in the Grafana view and see maybe where I'm getting higher and when I'm getting lower okay according to the resources applied in my cluster. So, we're getting into the end and I want to summarize that and show you that in each and every step of the way we can have security gates and the security gate stands in all of those stages. So, really check different things. Why do I need all those security gates because I'm not alone in the world, right, like if I'm pushing a code, or I'm writing my code, I, and my friend is writing and we're emerging to the same branch and I'm not sure he, you know he plugins of the visual studio code and so and so his code can be, you know, vulnerable. So, I want to check it in that stage okay I want to check the repo after I pushed it. I want to check it in the test because they have some more third party that are combined inside after I'm building I'm building with some third parties that are not part of my code. And then I want to stand and in the container registry to see if the image as a whole is secure and it's not vulnerable. Of course as much as we can. At the end, I still want to know in each and every time what's the status of my cluster. So, is it secure isn't. I mean, is it the most secure that can be. So, the gate there is still a persist and we need to check it every time and for me to use exporter helps us with that, because we can just rerun it all the time that mitches exporter runs all the time and says okay those are the metrics this is the trend of your risk in your cluster. So, I hope I didn't scared you. I hope I gave you the tools for putting your security gates in your CI CD pipeline again we're talking here all about open source tools so use that just put it one and then you'll be safe, you know, you can go to sleep very carefully without thinking anymore security. So, thanks a lot for having me here. You can just contact me each and every one of those medias, Twitter, this board, GitHub. You can scan the QR code and just reach right out to me. Thank you very much. And I hope you enjoyed this session.