 I'm Fars Siddiggy, I'm a Specialist Solution Architect at Red Hat. And I'm Roberto Guardela, a Cloud Services Black Belt also working for Red Hat. But today, Roberto and I have got a slightly different role. We're a part of a dynamic and very busy DevOps team called the Tiger Team. At the moment we're working on a project which is developing a web application for one of our biggest open source customers. And because in our day to day activity we need to orchestrate a lot of thousands of Kubernetes jobs we need a tool that help us run and build this workflow. And this is why we're using Argo workflows as the main part of our tooling. Like many other development teams out there we are running on regular sprints. And as it seems just today earlier today we did have a release day and a latest version of our application is already deployed. Like I said we're a very busy development team. We're always working on deploying new features, going through bug fixes and all the weird and wonderful things that all developers do. So we don't really have too much time to think about security. But it's worked for us so far. We developers, we develop and we deliver and it's working, we're meeting our deadlines. So we're not bothering about that just yet. Now, like I said our release date was today. The latest version of our app is already deployed and if you'd like to see what our app looks like please take your phones out and scan that QR code on the screen. If you'd like, is it okay at the back? Can you scan it? Okay, right. So if you've already scanned the QR code you'll be able to see that's the open source client we're working with. We are developing the web application for the ArgoCon event organisers. We're going to show you how our workflows in Argo workflows look like in a minute but just on a high level so you know what steps are involved in our workflow. So we first do a git clone in our workflow. Then we build and push our container images and once they're ready we just deploy them to our Kubernetes clusters from there. And this is how we currently deploy our application. So you're very well aware that the time of the event is approached and we under so much pressure to release a lot of last minute changes. Now in fact our managers just called us and asked us to deploy some really pressured last minute changes immediately. So this is what we're going to do and Robert is going to show you what that workflow looks like. So the first thing that we need to do is to deploy the new pipeline that we are already showing and we will kick the second pipeline using the exact same workflow template. We have our workflows UI and as you can see we are running the CICP pipeline and we will show the CICP pipeline that we already deployed before to run and deploy our application in our Kubernetes clusters. So we have three different steps. We have the git clone, we have the container build and we have also the apply manifest and update manifest. So all of this is based in an open source project. You can go into the GitHub repo and you can see for example this application. This application is a small application based in Golang and it's this application that you can see in here. Also during this we clone the repo using this clone repo. We are building the container and pushing this container to the GitHub registry as well and after that we are using the version one because this is the version one deployment and also we are applying the different manifest within the repository and as you can see the final step will set the image and we will update the image deploying our final destination and our final image as well. As you can see we have the GitHub registry that is already pushed and we already have the second version so we run the exact same pipeline as well that is running at this moment so the exact same pipeline is happening. We have the git clone, we have also the container build. This container build is the exact same thing but we are storing for example and we pushed this to the registry using the version chip because we are releasing a new version and also we applied the different manifest and now it's updating the different deployments and using this to deploy our application. So our pipeline is finished. Again same as the steps we saw earlier we're doing a git clone, we're doing a build and push this time for the second version of our application and then from there on we deploy to Kubernetes. If you refresh the app that you've just scanned and the QR code you'll be able to see the latest changes that we've just deployed under so much pressure. Have a look at what those changes are. That's not what we were trying to deploy. What's going on? We've been hacked and we've been held ransom for 2 million handcoins. What's happening? So what happened? If we dig a little bit deeper we can check that our supply chain is hacked. So the first thing that we can see is our source code was compromised and somebody, a group of hackers tried to ask for money and I run somewhere of 2 million handcoins and they also compromised the Docker file as well introducing some changes and they also hacked the supply chain and we can check that we have compromised and they changed the base image and now they are using a base image that contains a CVE of Actors that compromise not only this base image also the whole cluster put this Kubernetes cluster in danger as well. Since we fell victim to this vicious attack we started caring more about security and what we learned was that what happened to us is what's described as a supply chain attack. A supply chain attack is when a third party software gets attacked or gets compromised and as a result all the end users that are dependent on that piece of software or piece of product will be affected. This is often described as a more mature type of cybersecurity attack exactly for that reason because it's a lot more impactful than just attacking one single target. You attack a third party provider and as a result you're reaching thousands of people or thousands of end users as a result of that. So it turns out that the Docker image that we were using was downloaded from Docker Hub and that was the sort of our compromise and as a result anything we built on top of that image was compromised as well. There was a report that was published in 2020 in that report they analysed the data which was related to the supply chain attacks and this is what the graph shows. This graph is copied from that report and as you can see the number of supply chain attacks per year are growing exponentially. As you can see something really bad happened in 2020 does anyone remember the famous attack that happened in 2020 and we've got red hat t-shirts for the correct answer, the first correct answer I guess a red hat t-shirt. Yes, was that was it you? Yes. So we've got an X large and a medium. I'll leave them here for you at the end of the talk. Thank you very much. Thanks. Yes, so SolarWinds was responsible for that big jump in what you saw in the graph in the year 2020. What happened in SolarWinds was that a group of Russian attackers called Cosy Bear they got hold of the update server of SolarWinds which is a very popular network monitoring tool and they just injected some malicious software into the update server and as a result for that over 18,000 organisations who use that update were affected. It somehow reported that the cost of recovery from SolarWinds could be something on a board of $100 billion. Another example of supply chain attacks is what happened to another network monitoring server which was Casaya. Again, they got hacked and then the hackers were asking for $70 million in Bitcoin in exchange for the universal decryption key. Another widespread, perhaps the most widespread example of these attacks is the Luck4j. The reason is the most widespread is because in Luck4j is a very popular logging library for Java applications as we know and a lot of applications out there are using Java so it's the most widespread, as described as the most widespread. So these are just a few examples of supply chain attacks but we see that they're growing and it's not just limited to the ones that we're just talking about here. So what is being done about this? Founded in 2020, the Open Source Security Foundation, or OpenSSF, has begun to devise, improve the defences against the software supply chain attacks. One of the projects is a six-stop project that improved the defences providing a method for guaranteeing the end-to-end integrity of the software artefacts. So the answer of securing this software supply chain lies in digital signing the various artefacts that comprises these applications. But there are several challenges facing this software supply chain because sometimes, and especially in the Open Source projects, the budget for these digital tools is very, very low or non-existent for example and the software itself is changing very frequently and the management of these private keys are challenging. So six-stop, that is a project of the Linux Foundation, it's a new approach for signing, verifying and protecting software. It offers a set of tools that the different developers can use to sign the software releases and on the other hand, also the users can use it for verifying these signatures as well. And we have three major components in the six-stop project. First is RECO, COSINE and FULSIO. RECO is a built-in transparency and some service. On the other hand, we have the FULSIO project that is a free-route certification authority, a free-route TA and we will focus in this demo on the COSINE for container signing, verification and storage that makes signature invisible infrastructure. So we're using six-stop in order to sign our images and push our signatures but we still need another product in order to be able to verify those signatures against our images and this is where Qiverno comes into play. Qiverno is a Greek word and it means to govern and it's a product which is a Kubernetes policy engine and as a Kubernetes policy engine it allows you to verify, mutate or generate Kubernetes resources. One of the most attractive features of Qiverno is the fact that it's all in YAML. So if you're already familiar and working with Kubernetes, there's no need to learn a new language. It's already there and the good thing about YAML is that as human beings we pretty much could understand what the code does just by looking at the few lines of YAML. Now, if you look at that code, we've got one more red hat t-shirt to go if anyone would like to share what they think this piece of software or this piece of YAML does. Yes, what I can repeat your answer, there's a microphone there, it's a bit far away. Correct. Yes, that's exactly what it does. So it takes an image, first of all it's a cluster white policy as you can see in there. The action is enforced so it makes sure it does it and it verifies the image against a public key that is already got and then based on that it decides whether the image passes or not. Now, let's look at what are the toolings and how are we using this? What are the toolings we're using and how are we using these toolings into securing our demo? So to secure the supply chain we are using different tooling that we have in this picture. We have our workflows, Cossine, Caverno and AgostiD. Our workflows we already discussed and showed the UI as well for running the different pipelines that we have. Also we are using Cossine for signing and pushing the different signatures to the grid health registry. We are using Caverno to verify and check the different images as for example we checked with the cluster policy and as well as Agro workflows in a separate use case we will introduce AgostiD to show how Caverno can use to protect also the Kubernetes clusters in continuous deployments or CD processes as opposite as the CACD in Agro workflows. So this is to show you that we are using AgostiD for deploying all of the components that we need to install the demo and this is all managed by AgostiD using GitHub as well. We can go directly to the Agro UI. We have three main applications, Agro apps. One is the Agro workflows. We have other deploying Caverno and finally the workflows and blades as well. We have this Agro app of apps pattern that is managing the different AgostiD applications as well. If we go to the workflow templates we will see that AgostiD is managing the different building blocks that we already have in this pipeline. We have different, for example in this case we have the sign image that is a workflow template and is fully managed and fully deployed using the Tops and AgostiD. So if we go directly to the workflow templates we can see different things and also two different CACD workflows. So we will deploy the next, first we need to deploy the policy. This policy is as you saw before a policy that it's a cluster policy for checking the image against a public image and on the other hand we will kick another pipeline that includes in this case this sign image. This sign image is a step that additionally will use cosine for signing using a private key the image and will automatically push the signature against the GitHub repo. In this case in the GitHub repository we will have in the GitHub container register as well we will have two different things. First we will have the image and also the signature because we are using this for pushing the signature as well as you can see that it's running properly and it's cloning the repo and we can see the different steps. So with the workflow that Roberto just ran we are rewinding to the first step when I started setting the story about deploying our application. But the difference is this time rather than ignoring security we are implementing those security steps which are signing our images and then verifying them against the sign signatures here. So instead of doing the steps that we were doing which was the git clone build and then push and deploy to Kubernetes. Now we are doing the git clone we are building our image and now using six store we are signing the image and also pushing the signature to our container registry as well. And using Kyveno as we just saw then we verify this images against the public key that we hold and based on that we decide whether we want to deploy into Kubernetes or not. So we will check if now it's in sign state it's signing the image so the first place we are building this container image. This container image will have a specific version that we have the version 3 tag that is v3 signage. This is pushing also this image to the GitHub registry. Then we have the sign image as we commented the sign image is signing the container image that we already deployed push to the GitHub registry. And it's pushing also the signature into the GitHub registry as well. And if we check the GitHub repository we will see first that we already have our container image with this specific tag and also alongside we have the second signature that if we checked matches with that. So in this pipeline we are first building the container image and pushing the signature to the GitHub registry. And also Kyveno it will grab this signature and will match and will verify against the public key as well. So in this case already we deployed our application and if you run again your application you can see that effectively it's deployed. So if you already had the page up on your mobile phones if you just refresh the app or just rescan this QR code you'll be able to see that now we have a healthy version of our application running again. Now let's go back to the scenario that we had to release under pressure and we were ignoring security because we didn't have time for it. But then we got hacked and then we realised now we need to care our security. And knowing what we know now about the tools we just talked about let's see how we can use those in an event that we're being hacked to prevent from those hacked images to be deployed to our Kubernetes clusters. So this is what our steps look like. We've got the git clone and build image and if you remember when we were doing the diff on git we were attacked on those two different sources. One of them was the GitHub source and one of them was the container image that we were using. But now because we're using Kyveno and we're only allowing the signed images to be deployed to our Kubernetes clusters because these images are tampered with they're not going to be allowed to our Kubernetes clusters. So we just put the stuff in there. So even if we run the workflow you're going to see in a minute the workflow is going to get to stop. So we will rerun the exact same hack and meanwhile we will present the second scenario when we are waiting till this finished and we can check also the last scenario that we have. That it's deploying the ArgoCD and securing the ArgoCD applications as well. So we will deploy the last demo and then we can get back to the previous demo in order to check if Kyveno did unsafe today as well. So we are using ArgoCD for deploying also our applications but imagine that somebody hacked our source code and also the GitHub repository that ArgoCD is linking and they pointed out to one specific tag for the image registry. And this GitHub registry shows that effectively have hacked as well and somebody pushed this image registry and hacked our image registry. But using Kyveno we'll check and verify the image. We'll check for the different signatures and then we'll deny the request because the signatures will mismatch. So as we can check directly in the ArgoCD we can check that effectively we have this application that we have a sync failed for what reason. Because Kyveno through the admission webhook this blocked the different resource and denied the request because the image verification failed because when verifies the image shows that the signature mismatch. And for this reason it's out of sync because cannot deploy the application because the signature mismatch and then Kyveno saved the day. And on the other hand the exact same scenario using our workflows as well show that Kyveno will prevent of deploying the wrong image because it's happening the exact same thing. It's preventing to deploy this application and as we can see in here if we check the logs of the container the Kyveno admission webhook denied the request because checked against the cluster policy of the check image and verifies that the image verification failed because the signature mismatch. Then also in both scenarios first the Argo workflows and also in ArgoCD Kyveno saved the day and also along with the 6th door signing the image. So we saw today how some parts of our github's workflow could be susceptible to supply chain attacks and we also saw that how using Kyveno and 6th door together we can prevent this from happening. So next time you have five minutes perhaps over a coffee break I'd like to invite you to have a think about where this solution might be applicable to the way you're running your pipelines and your github's workflows. And also how you're going to take this learning and implement it to your work and not only your work the work that you do with customers partners or colleagues of yours because after all it's a chain and we're all in it together. Thanks for today. Next slide we've got all the links we used in the demo and if there are any questions we'll be here over the break as well. Yeah.