 Hey, everyone, welcome back. Hopefully you are nice and fed, and hopefully warmer. It feels a little less cold in here now. So yeah, thanks for coming back. So next, we're going to be talking about securing CICD pipelines. We have Sharipad and Jim over here. So they're going to take it off. And welcome to this talk. And yes, we are going to talk about protect the pipe. It's a policy-based approach for securing CICD pipeline. Just to introduce ourselves, my name is Sharipad. I'm a senior technical staff member at IBM Research. I'm currently leading some innovations around supply chain security. And I'm a member of tech security and supply chain security working group. Hey, everyone. I'm Jim Baguadia, co-founder at Nirmata, also a maintainer of Kevurno, and do other things around the Kubernetes projects. Yeah, so why we are talking about security, right? So if we revisit our software supply chain, so it starts with the developer writing code, pushing it to some source-controlled systems like GitHub, GitLab, which then eventually triggers our CICD, right? And Capp pipeline specifically. We run multiple DevSecOps checks. We run, we have Git client. We run some static scan. We generate S-bomb. We build our artifact. And finally, we push them, right? So in this, we push and deploy our application on the runtime. Now, in this whole spectrum, right? So if we look into CICD, this is very critical section, right? Why it is critical is because there is a growing open source ecosystem around CICD pipeline, where multiple ready-to-use task are getting basically being available into Marketplace like Tecton catalog or GitHub Action Marketplace. And just like we are securing our open source dependencies for applications, we need to pay attention to basically securing these CICD open source element as well, right? And the other aspect is our CICD pipelines, they have access to a lot of critical user credentials, right? For instance, we have GitHub client which need access to our Git over tokens, so it can basically push some changes or create a pull request to automate some of the task. We have our build system, which essentially has access to our registry credentials, so it can push images. So they have access to this lot of critical information, credentials. So we need basically right way of securing our pipelines, so it cannot be compromised. And now, in today's talk, we are going to primarily talk about four technologies and how they come together. We are going to talk about Tecton as a CICD platform. But again, this scope of this talk is not limited to Tecton, right? It applies equally to other CI platforms like Jenkins or GitHub Actions. We are going to talk about six store, intro to Kivarno, and how these all technologies come together to our rescue. So first, talking about Tecton, right? It's a cloud native CD building blocks. It runs natively on Kubernetes. It has a bunch of CI IDs, like it has pipelines. We have tasks. Pipeline encodes the task layout, like what is the workflow that you want to execute. A task encodes the execution logic. We have task-run pipeline run, which encodes the execution trigger properties. And Tecton, again, has multiple dimensions, right? So one dimension is TectonChains, which is a new project which has about the supply chain security of pipelines themselves. It creates an automated attestations of the tasks as they execute. Then as I said, there is open source catalog for ready-to-use Tecton tasks. And then there is one element that we are going to talk about more in detail is Tecton Bundle. Like it allows basically us to package and distribute this Tecton task and share them through an OCI artifact. The other project, again, in Toto, it's an open extensible metadata standard. It is used to basically encode the software supply chain tool chain, right? What is the workflow that you expect? And create an attestation of those. And again, create a verification of those through some automations. Sixth door, again, I think all of you probably already know. So this is basically emerging as this public infrastructure for signing software artifact. And in this work, we are essentially using it to sign various Tecton resources, which includes our pipeline definitions, configurations, policies, images. We sign them using cosine. We store those artifact into artifact store. We are using OCI as the artifact store for storing all this. And the sixth door is basically it has broader scope. It has record for transparency log, full CO for key management. And then we have Qverno. I think Jim is the best person to explain that. Yeah, so Qverno is a policy engine that was designed for Kubernetes. And what it does is it uses Kubernetes's declarative configuration management system to manage policies itself. So the goal of Qverno is not just enforcement or validation or checking of resources, but also to automate security. So to be able to generate resources, mutate resources in a first class manner was very important to Qverno from the beginning. And as you'll see in the examples, this gives you a very powerful toolbox to use similar patterns that you would with any Kubernetes controller, any Kubernetes CRD or definition, and take that into policy management itself. So we are going to switch to the other. Yeah, let me pull up a bigger view of this. It'll be easier to see. Here we go. So we essentially came up with these deciduous attack threat modeling for the Tecton. And I think we are probably familiar with the legend. So the boxes in the gray, they basically represent the facts, the ping boxes. They represent the attack vector. And then the blue essentially represent how we can secure them. So for Tecton, if you look into this, as we just discussed, it starts with the definitions. We have the pipeline definitions that encodes the workflow. Like what are the tasks that you can be executed? What is the order in which these tasks can be executed? Then each task, it has a certain set of steps that encodes the execution logic. Like in this particular task, I want to execute these particular steps. Each step, it's run as the container, as the pod and on the Kubernetes cluster. And in this particular workflow, or in this particular spectrum, there are various attack that are possible that Jim is probably going to explain in detail. Yeah, so before I go into that, just if you're wondering what deciduous was, and I had to look it up a few weeks ago when I started work on this tree. So apparently, deciduous means that trees where I guess they shed their leaves every year. So something which is temporary or what changes, right? Which is a nice tie-in into the security context here because security is always changing. And every organization will probably have different needs for their security concerns itself, right? So what we did over here, like Sripad mentioned, we have three major facts, right? So you have your pipelines, you have your tasks, and then you have your steps. All of these are operating as native Kubernetes resources. Each one is gonna run several containers. Each step runs as a containerized image. So every possible attack you can imagine on Kubernetes also applies to Tecton. And of course, now because you're building your software itself, it's almost exponentially critical here, right? So some of the things we did, and this is not a complete picture, right? So don't take this as a representation you can kind of take and put into production, but the idea here is to come up with, okay, what can we enforce, what should we check? And the exercise that Sripad and I went through is, can we convert these into policies expressed by Kiverno? So the first thing we check for is to make sure that everything that executes is executing from a pipeline bundle, right? So that's one policy. The next thing we check is to make sure that you cannot just run tasks without pipelines. Now, Tecton, like Kubernetes, again, is very flexible. It lets you run just like in Kubernetes. If you wanna run a pod, you can, but it's not a good idea. You'd rather run a deployment or some other controller. So similarly in year, the idea is instead of running tasks, you run a pipeline, and better yet, run a pipeline with a signed bundle which is securely stored in your OCI registry, right? So those are some policies up there. But then every step, like I mentioned, runs as a container. So at that point, you wanna make sure that that step is secured. It has the right pod security or, well, in this case, step security context, or the same definitions that you would apply to a container, like run as non-root, don't elevate privileges, things like that, you wanna enforce for that step. Similarly, in a Kubernetes environment or your workloads, you would run each workload in a namespace. Hopefully we all know that's the best practice to use namespaces for segmentation. So similar year, what we did is we applied namespacing to say every time you execute your pipeline, it should be in its own namespace, right? So, and once you have a namespace, you can generate security defaults. So going back to what I mentioned about Kiverno and the fact that it's to automate security, so why not generate those defaults instead of having to manually configure them or come up with some other processes for them, right? And then finally, besides checking for signatures, we also came up with policies with check for attestations in total format. So this is to verify that the images actually have a scan performed. You can even check how recently the scan was performed or what type of vulnerabilities you might want to allow or prevent, right, or block from executing. And then of course, signatures for all the images was the final policy you see over here as a leaf, right? So that's the kind of attack tree that we came up with. Now, of course, again, this will vary based on your organization's needs, but hopefully it gives you an idea and I found this a pretty useful exercise of organizing this in terms of the facts, the possible attacks, the mitigations, and the end goal, right, of the attacker, which is to get access to your cluster or to your host. And by the way, there's an app, if you're not familiar with it, it seems pretty cool to go and it was fairly easy to model and kind of draft this up. So with that, let's go into actually looking at some of this in action itself, right? So I'm gonna switch, let's make sure, yeah. So I'm not gonna cover this policy list that I already spoke about them briefly, but let's go to the demo and see this in action itself, right? So the first thing we're gonna do is I'm gonna actually kick off a pipeline because it takes a while to run this and I guess with different wifi speeds, et cetera, this might take longer. So let me go and like I mentioned, on my cluster, right? So we already have tecton and other things installed, including Kiverno. So I'm just gonna show you that the policies I have installed for Kiverno, they roughly match some of what we just talked about. And what we'll do is first thing I'm gonna try and do is run this pipeline outside of a namespace, right? So instead of just creating a namespace, let me just do, I'm gonna say, and the sign pipeline, I'll show you what this is if I go into all of the declarations. And by the way, we have this available as a git repo. So you can look at this later. If I go in here and look at sign pipeline, it has a run step and the run step is very simple, right? All it's saying is in this pipeline, I want to execute it from this bundle and it doesn't have any namespace or anything defined so I can just run it in any namespace. So if I try this, what I'm expecting to happen is it gets blocked. It says, no, I need a namespace, right? So let's go ahead and create a namespace for this. And I think I already have something called run, so I'll just call it run two. Whoops, we wanna say create namespace. And now that we created that namespace, let's go and see what happened in the namespace itself. So I see that there are already some things populated for me, like secrets, there should be a role, a role binding. So this is everything this pipeline needs to execute and Kivarno has automated that. So the way that that got populated, if I go into my policy set that I have over here, for security, we looked at some of this to block those resources. But if I go and show, let me make this bigger, some of these policies which are actually generating even things like PVCs, right? So Tecton shares information through volumes, right? So you can configure this in many different ways. But here what we're doing is we're creating a volume and we're creating the PVC in there to make it easy to now execute that pipeline. So now that we have that in here, if I haven't executed my pipeline, so let's go ahead and do that. So if I do run, and if I say create sign pipeline, that should start the execution of my pipeline. And we can see, and it should spin up the pods in a few seconds. And we can look at this also from the Tecton dashboard, right? So let's go back here. If you look at it as pipeline runs, we should see there's a new pipeline run and it has started executing. So these are the different steps that are in the pipeline. So maybe Tripod, if you wanna quickly explain what this pipeline does. Yeah, so this is essentially a template pipeline that we have created. And it encodes a typical steps that we see in the most common pipeline, right? And this is again, a shorter version of it. So it starts with cloning the repo, which essentially is getting the artifact into the workspace. Then we run some static scans on this like GoSec, which if it is a Go application, it checks for it statically or check for vulnerabilities in the code. Then we have, okay, this is not string. So we have build and push. Then once we pass those, it essentially build and push the registry, the image to OCR registry. It signs them with Cosign and it also adds the attestations, right? So we see this, there is generateBomb. We generate the sBomb again, attach it to the build image. It runs a vulnerability scan and again create an attestation for it. And finally, if it, you see there are no vulnerabilities, all attestations have been created properly. Then it goes ahead and deploys it, this particular application onto the cluster, right? Again, this is not meant to be used in production, but this is just to give a context and some scope for us to basically play around with this demo, right? So yeah, it should take maybe a couple of more minutes to finish, but yeah. Yeah, so typically, I think the last time I tried it, it took about five to seven minutes to go through all the checks. And obviously as you're adding some policies, there is some time added to fetch data because some of this data comes from OCI registries. And let me quickly show you while that's going on what some of these policies look like, right? So if we go back to the policies, one of these is just to require a bundle. So that we already, you know, and we can try this by running a task directly to see if it gets blocked or not. But basically, as you can see, there's a very simple policy we're just saying, you know, everything should run into the bundle and don't allow, you know, bare tasks to be executed, right? So let's give that a trial. We don't need to watch this. So if I do, let's try. We'll actually do a create. And what I want to do is in my policies, I have some test resources. And if I look at that, what I want to do is to just say, you know, I'll just use the sample task run to try and run that, right? And again, what I'm expecting here is that this gets blocked because it's saying, first of all, well, it got that there's no signature on it. So that policy blocked it. And it would, if that passed, it would also be blocked to make sure that, you know, it has to come from a, you know, sign bundle, which is stored in our OCI registry. So going back to again, the set of policies, there's this other set which are now verifying signatures, right? And what that looks like, so it's a pretty interesting scheme because what you're trying to do is you're enforcing a policy. So by the way, this pop-up that shows is the VS Code integration because give or no everything's a CRD, you get your help right in here. So that's a nice little value add, right? So we're checking signatures here. And the one thing which is a little bit complex is you're not really running a pod, right? So it's simple enough to check a signature. Well, not exactly simple, but it's possible to set a, say, check a signature on a pod. But here what give or no is doing is it's using this new feature which we introduced in 1.7 to extract images out of the configuration. In this case, it's looking into the spec of the pipeline reference and it's gonna check for, you know, value called bundle and the name. And then it's gonna check that that bundle was signed using the key that you're declaring over here. So now this key could be in a config map or somewhere else, but ideally what you're, now what this policy is doing is it's going through all of the, you know, the actual runtime configurations. It's looking in there and into the spec checking for all of the sign bundles and then verifying the signature. So similarly here, this one's a little bit different is now we're looking at task runs. So remember there's pipelines, there's tasks and there's steps and each one needs to be verified. You wanna make sure each one can be trusted and executed. So similarly we're checking whether the task reference is coming from a signed pipeline. And finally, you know, for the steps, here there's a check to say that for each step, make sure that if once you extract the image, the image is also signed and verified, right? And in addition, we're checking for a predicate. One of these things I've commented this out because we actually found that one of the steps has a high severity vulnerability, so that was blocked. And there are ways to kind of allow that exclusion, but what we did for this demo is we just commented this out, but we're checking that every step here has an attestation which is actually a vulnerability scan created in this case by gripe, right? So you could use whatever tool you want and then you can write, if you were to kind of check for vulnerabilities in the policy, it would look something like this where you're looking inside the JSON data and checking for that payload of that attestation or in fact in the predicate to make sure that you're in this case, we wanna make sure that there's no severity higher than eight available or in that report itself, right? So those are all of the things that would be enforced. One last thing I wanna show and we'll check on our pipeline too is I wanna actually run an unsigned pipeline or a pipeline which is not from a valid repo, right? So what I did was I just copied this pipeline that Tripod created and I pushed it into my registry, my private registry or I guess my GitHub registry and I want to try and execute that just to see what happens, right? So let's go ahead and create another namespace. We'll call it run three and then what we'll do is we're gonna run now this unsigned pipeline, right? So just to show what exactly that is. So if I go back to here, this is our signed pipeline and if we go to unsigned, what we should see is it's using a different image registry to run, execute this pipeline bundle, right? So what should happen is again, because of that missing signature check, that should get blocked. So let's go ahead and try that. Yeah, so here we're seeing one of the, because we are using a public key to match it saying there's no valid signatures which internally, Kirano integrates with cosine from six store, it's making that call and validating that pipeline bundle itself, right? So that's more or less how you would go ahead and secure this entire pipeline. As you can see here, it's still continuing, it's towards the final steps of it completing. So again, depending on, because it pulls a lot of information, these, it might take a little bit longer to complete, but I think overall the idea or it should be fairly, I guess, clear through the policies, et cetera in terms of what needs to be checked. So with that, let's go back into the slides and we'll wrap up and we'll come back and check on the execution maybe after. Sripad, if you wanna summarize? Yeah, so I mean, to summarize here, we're talking about Tecton, right? But as we mentioned, right? It's applicable to other CI systems, but Tecton is a powerful CI solution, right? It's built and operated natively on Kubernetes. CI system needs to be secure, right? We need to treat them just like our production workload, not just the integrity, but also the configurations. We need to make sure they are configured properly, whatever we are executing, they are signed, they can be trusted. And of course, the Kibarno policies are the perfect fit to achieve this, right? Which allows us to perform the validation, mutate, generate and verify images. And all this demo, it's available on the GitHub, so if you want to try it out, it has instructions, all the policies are there, you can try it. Any closing notes, Jim? No, let's just check one more time to see if the pipeline completed, but otherwise, I think we're done with what we wanted to demo and happy to take any questions. I think it actually completed, yeah. All the tasks are done, the pipeline is done, so there you go. All right, any questions, thoughts, feedback? Thank you. Yes, it is actually in the Git repo itself, so yeah. But don't use it as is, create your own please, you know. Any other questions for anyone? All right, if not, thank you very much. Take care.