 Hello, everyone. So today, welcome to Cloud Native Live, where we dive into the code behind Cloud Native. So I am your host, Shahriar Aung Sankemito. So I am a science ambassador, and I will be helping the folks speak as our chair. So every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things, they will break things, and they will answer your questions. So in today's session, I'm stoked to introduce Anushka and Anusha both. So and this is an official live stream of the CNCM. And as such, it's subject to the CNCM Code of Conduct. So please do not add anything to the chat or questions that would be in violation of the Code of Conduct. So basically, please be respectful to all of your fellow participants and presenters. With that, I will hand it over to both Anushka and Anushka. So let me add them both to the stream. So, yes. So, hey, Anushka and Anusha. So I hope you're fine, right? Hey, hi, Mithil. Thanks for having us. Yeah, that's great. Yes. OK, so I guess we can start, right? So yeah, you can introduce yourself or let's start the process. OK, let's go. Yeah, I'll just go to the slides and we can continue from there. Sure. We know when the screen's visible. Yeah. Yes, seriously, it's visible. OK. Hello, everyone. Welcome to the webinar, Policies in CICD Pipeline. I'm Anusha. I'm a technical product manager at Nirmada. I've been at Nirmada for over a year now working on Kubernetes security, mainly policy and governance in Kivirno and other related areas. Prior to that, I've worked in the infrastructure side of Kubernetes. I worked with the project called Cluster API. And yeah, happy to be here. Hey, everyone. I am Anushka Mithil. I am a software developer with Nirmada. I'm also a Kivirno contributor. I've been with Nirmada for about seven to eight months now. Previously, I've interned with them and also been an LFX mentee. So yeah, today we will be, before we get into Policies in CICD Pipeline, we'll set the context with an overview of Kivirno, talking about our new cool product called Kivirno JSON, talking about shift left security and the pipeline workflow for it. We'll follow that with a demo and just wrap it up with some key points to take away and take questions here and there, wherever. Yeah. Great. So before we even start talking about Kivirno, let's talk about policies. Why policies? So just like rules and regulations are important for things to work properly, you need rules and regulations in your cluster to have everything functioning properly. You want rules and regulations that is via policy to have security in your workload, define certain guardrails to ensure consistency. You would essentially add policies to help reduce the risk of any misconfiguration, any intentional or unintentional security violation. You end up setting your security posture with policies, what to expect and what not to expect. All of these policies in your workloads help you have reliable, safer functioning with Kubernetes, have better collaboration where you don't have to worry about what the new person can do wrong and there's just a lot of more things that you can do with policies. Here enters Kivirno. Kivirno is your open source admission controller. Kivirno means govern in Greek, just FYI, of course. It's a lot more than an admission controller, really. It's native to Kubernetes. First of all, when I say that, it means that if you, you know, written a deployment or a config map in Kubernetes, you'd probably find it very easy to write a policy in Kivirno as well. It looks, it's YAML, it looks like everything you've seen before in Kubernetes. Now, Kivirno offers, of course, validation. It also offers mutation of resources to generation on certain triggers. You get sort of the option to verify images using policies as well, clean up resources using policies. Finally, beyond Kivirno, we have the Kivirno CLI as well, where you can test your policies before having them in your workload. And finally, there's policy reports to make sure, you know, what's happening in your cluster. So that was a lot of pre-Kivirno architecture. Let's talk about the Kivirno architecture. So right here, let me see if I can get the laser pointer. Awesome. So right here is what you'd call the Kubernetes Admission Chain, where there's an API request that goes through the HTTP handler, through authentication and authorization, finally through mutating admission webbooks, if any. So this control is sort of, you know, by books that you configure and register with your API. And if the server detects any, your API request will go through them. Since there's a chance that your request could be mutated, the next step is validation of the schema. And finally, you have validating the admission request, the API request. Once there's a decision made, you know, for or against the request, if it passes all the checks, it goes and gets stored in your HCD. Great. Now where does the Kivirno magic happen, right? It's right in between mutating admission controls and validating admission controls. Here is where we extend to Kivirno. So this large image in the middle of the screen is what Kivirno looks like at a very high level overview. So what happens, the first thing you need is your webbook configured, you know, with mutating admission webbooks and validating admission webbooks. So you have a webbook monitor that is constantly looking at policies that you have defined in your workloads. And according to that, it sets the webbook configuration. Once webbook configurations set, it essentially would reflect here and any API request that matches this configuration would be directed to Kivirno Engine. Within Kivirno Engine, this resource, this request will, you know, go through a lot of policy checks with whatever policies you have defined in your workloads. And Kivirno will be able to make a decision as to whether or not the request is something we want to allow or, you know, just have it recorded in your reports and allow or completely deny a request. So yeah, apart from, of course, just the validation and mutation, you have report controllers that would sort of take care of the reports generated. The admission reports, the background scan reports, as well as the policy reports. The background controller here, it's supposed to, you know, check for everything going on or that would have been going on before you had Kivirno in your system. It mainly handles update requests. All right. So that was as brief as I could be about the architecture because we can entertain some questions in a bit. Let's, you know, before getting into the meat of this webinar, let's talk about policy as code and what the policy management looks like, right? So any policy, you would first write the policy, you would develop it and test it to meet your requirements, whatever that would be. Next, you want to distribute it using the Kubernetes management tools and make them available. Next step would be to enforce these policies. That would mean, you know, have them available, have them in your workload ready to be applied or ready to be used. Next, you'll observe these policies to make sure they are doing what they're supposed to be doing. They're not really eroding out and they're not wrong in some way. You want to ensure that these policies are correct and working as you require them to. And the last step in the cycle would be to manage these policies in terms of, you know, there's some upgrade and you need to make sure your policies reflect that or just, you know, rolling out policies as and when you need them and so on. Kibirno in this management cycle, you know, I could just mention a couple of points where Kibirno helps. So Kibirno provides low code policies, easy to develop, easy to maintain, easy to understand. Next, you know, distribution. Kibirno works with all your Kubernetes management tools. With Kibirno, you can enforce in the CI CD pipeline using actions. You could have them in the admission scans or the background scans. You can observe. Kibirno provides for policy reports, like I mentioned before. So you can observe the policy reports. Kibirno even triggers events that you could observe. And finally, there are metrics that you can obtain to understand what's happening. Finally, the management policy exceptions in Kibirno where you could, you know, define that, hey, in this namespace, I don't really want my resources getting blocked for so and so reason or yeah, you just can define policy exceptions. There's, you know, you could roll policies out and there's provisions to upgrade and things like that. Yeah, that was a very good overview, Anushka. I think Kibirno is one of the most popular, you know, policy management tool out there. Over the past, I think three years, it was donated to CNCF in 2020. So ever since then, we have seen a huge uptake in terms of, you know, popularity or the number of community members that we have today. So let's talk about what is Kibirno JSON, right? Like, if many people are not aware of this, we could probably, you know, talk a little bit about it. Some background around Kibirno JSON. So one of the most common thing that we have heard from our users is that Kibirno is mostly used by DevOps engineers or by platform engineers. And as a platform engineer responsibility, it is always much more than Kubernetes. They have to identify what tooling they want to use to just provision, say, Kubernetes infrastructure or even other cloud resources like they could use Terraform, you know, Crossplane or Anushka even has an example for Pulumi later on. So platform engineers have to manage how to roll out the infrastructure, the infrastructure itself and observing the infrastructure and a lot other things other than Kubernetes. So while Kibirno worked great for Kubernetes resources and when I say Kubernetes resources, it works for other custom resources as well. The main requirement to write a Kibirno policy is that you need to have a custom resource. So therefore, now for a platform engineer to validate if the Terraform files are correct, if they are, you know, running correct images in ECS containers or ECS tasks, they need to look for other tools to write policies for this. And as in when the number of use cases increases just like Docker file, right? Now if there's so many things that can go in wrong in a Docker file, you can use base images, you know, that you don't want to which have again, you know, vulnerable dependencies and so on. So I hope you see where I'm going with this, right? As the number of use cases increases, the number of tools that you have to, you know, adopt, learn and manage also increases. And that adds to a lot of, you know, pain in the long run. So as a solution to all of that, we introduced when I say we, it's the Kibirno community, Kibirno community introduced, Kibirno JSON, last KubeCon, which happened in Chicago just a few months ago, three, four months ago. And the use cases that it solves or it addresses are tremendous. So just like in this slide, we can see Kibirno JSON, it can act as a CLI. It can act as a, you know, web app using REST APIs. You can put it in your, you know, Golang applications. You can, you know, make it part of your code itself. So you can use it in any shape or form. And the beauty is that you can write policies now for Docker file, Terraform files, ECS or any other, you know, cloud resources, of course, Kubernetes itself. And also we just said any IAC type. So now it could be Terraform, Pulumi, CDK, CF, you name it. You can write a policy for it. And not just any JSON. So you give us a JSON resource, we can write a Kibirno policy for it. All adhering to principles of low code, easy to write and understand, you know, everything that users love about Kibirno. We've made it available in Kibirno JSON as well. And again, the reporting structure of it is same as Kibirno, right? You have policy reports available, which can tell you past fail. And these policy reports are again, you know, standard APIs. This API was, you know, introduced by the policy working group and the Kubernetes landscape. And because this is a standard API, now you can, you know, confidently build abstractions on top of this. Say you want to write a dashboard that has, you know, that monitors across all these different types of resources. You can do that because your reporting API is fairly standard. So Kibirno JSON comes with a lot of benefits and, you know, it unlocks a lot of use cases as well. Yeah, moving on. Yes, this is a breather slide that, you know, Anushka asked me to add because we spoke a lot of things about this is Kibirno, this is Kibirno JSON. So yeah, again, can you go forward? One of the key takeaways we want from this webinar is not just understanding how to write policies or, you know, what are the different use cases. We want you to go and start implementing these policies in your CI CD pipelines right away if you're not doing it already. And we know, you know, doing something in production or introducing an additional step in your production pipeline requires safe business justification. And this slide gives you those numbers that you can go and pitch to your management and convince why is this so important, right? And the CNC blog that I've linked over there is actually pretty, you know, it's a very good blog, one of the blogs that I really love and I share in almost all calls. It tells you, you know, it gives you a nice derivation of how it arrived at these numbers but giving you a quick overview, let's focus on the graph that we have over there, the red curve. So what it tells us any issue, like be it, you know, a Kubernetes issue or just issue in your application code. If you find there is a bug or something wrong, say in the coding phase, it just takes one or two developers' effort to go and fix it. But as and when the stage progresses, you have a functional test stage, you have, you know, system test and so on. But in the release phase, you can see if you identify there's an issue, the amount of time and resource you're going to invest is way too high. Like you have so many developers now going and debugging what happened in production. You have SREs and it's already in production. Your customers are affected by it. So the loss that you incur is tremendous. So it's just 640X is mind-boggling, right? Like how some things trivial could have been fixed in the coding phase. You're now investing so much in a release phase. That is where it is very important you start adding these different guardrails in every phase of the pipeline, when I say you have to now do everything in coding doesn't mean you don't do anything in between. So the same amount of testing, security checks have to be implemented in each stage of this pipeline. And just adding numbers to this cost per defect, pre-production or pre-deploy phase is just $25, but in production it's like 16,000. And this is just for defect. And imagine that in a per cluster basis, it is so much. I can't even read the number to be honest. So yeah. So this is why this is a good business case that if you have to put forth to your management as to why you need to do this. Yeah, so moving on. So security was most often an afterthought in most organizations up until recently, only after we have seen a lot of security attacks in the recent past. Organizations have started to give that importance to security. So in many organizations, security is now a top three priority for them. And not just, let's deal with it when something bad happens. So that is where the industry is moving towards shift left. They want to have the security checks and balances much earlier in the deployment lifecycle. Yeah, so moving on. Yeah, so these are the different phases in any typical CI CD pipeline. You have a commit phase where you do as many number of code changes to your PRs and then you build it, you build container images, say push to a certain registry of your choice. And in the deploy phase, you pull from that registry, you actually run your application in a cluster now. And then finally, the run phase, this is everything that happens after you deploy an application. That is the lifetime of the application after it is deployed. So in each of these four stages, you need to add guardrails. And there are different ways you can add guardrails. It's not the same check you're doing in every stage. Also, which you should be doing if they're applicable, but there are different kinds of checks that you can be doing. Like in the commit phase, the most common things that people do today are things like you have unit tests for your code. You do some sort of linting, make sure you don't have, say, secrets in your application code or Docker files. So these are some of the basic checks, but you can still do some advanced level checks when it comes to infrastructure itself. So things like, you know, if you're doing port security production, you could be doing port security checks in the commit phase itself. Like how cool would that be, right? Like you're stopping a developer from, you know, making sure they're actually writing proper deployment files or proper Docker files because, you know, Kubernetes is complex and it is a niche skill and not everybody is aware of, you know, every small minute configuration detail. And then you have the build phase where you build your container images. Again, you can sign your container images here. You can verify, you can attest different artifacts and you can even scan your registries. Then in the deploy phase, you have for admission controllers, like Anushka mentioned, Kivano is one amongst them. You can have admission controllers that can actually stop your resources from entering your cluster if they don't match certain rules. So you can, you know, you can be rest assured that your bad resources are not making it to the cluster in the first place. And then you have the run phase. So, you know, things like CVEs, they come in anytime. They don't, you know, see that, oh, this was tested at build phase or deploy phase. So maybe you're safe. So you should be continuously monitoring for the misconfiguration, CVEs, everything, even after an application is deployed. So these are the different checks that you can employ at different stages. And one interesting thing probably I could share that can help the rest of this webinar is that in most organizations or customers that I talk to, and like I said, security is a talk concern right now in most organizations and people are doing a good job in adding guardrails for the deploy phase or the run phase. They are introducing new monitoring tools so that, you know, they can help with runtime checks and other things. But still the commit phase is something that is, you know, not received much love. And, you know, people have just limited this to LinkedIn and a few other basic checks. So in this webinar, we want to show you, you know, all the powerful things that you can be doing in the commit phase itself. And it's very easy using Kivarno and Kivarno JSON. Yeah, I think over to Anushka, but are there any questions at this point? So, yeah, by now we don't have any questions, I guess. We can continue, Anushka. Just to confirm, am I audible? Yes. At the same time, we can see your screen also. Okay, yeah, just opening something relevant. All right. So going back to the slideshow for now, really the image that, thanks, Anushka, for the image especially, that image is, you know, leaves me or every time I see it, it's crazy. And yeah, so in this session, like Anushka mentioned, we want to look at what you can do in the commit phase and how Kivarno can help you. So before that, you know, we prepared what the workflow looks like in this discussion. So you have, you know, two independent teams. You have a platform or a security team that's defining rules and regulations, policies. They're making sure they know and define things that they want and they don't. On the other end, we have the DevOps team, you know, owning the CI pipeline, setting it up and really just adding configuration. So what we wanted to do here is when the DevOps user or, you know, the team adds to a configuration, we want to pull the policies that are centrally stored, that can be centrally stored by the platform or the security team and scan the configuration, the new configuration against those. And you know, let the DevOps user know, hey, something you've done is wrong and you need to correct it. And before you do that, I can't let you get that in my workload. So the demo, in the demo, this is what I'll try to cover. And of course, the actions that I'll be using in the GitHub workflow, these are readily, you know, these are available on the GitHub marketplace and can be used, the ones called the Verno CNI installer and the others for Verno JSON installer. So let's go to the demo. So we have the positive here, I'm going to increase the font size a bit. Okay, so I have a team that is independently working on the controls. The controls essentially are the policies that are the best practices for each manifest and number of manifests. And on the other hand, we have something called configuration files where we are setting the configuration for different manifests. Here I just have, I have four, sorry. Doc file, Kubernetes, Pulumi and Terraform. What I'm going to do is I'm going to add GitHub workflows and these workflows are going to run on a pull request or a push to the main branch. So these are really simple workflows. All I'm doing is, you know, installing Verno JSON in this case. I am, excuse me, oops, yeah. Okay, I am checking the version and I'm doing a, I'm running a scan to see if my configuration file is violating any of the best practices said by the security of the platform team. This was, of course, a use case of Verno JSON scan. Here in the Kubernetes scan, I'm using the Verno CLI. I'm using its action, checking the version and applying my port security policies against the configuration files I have. So what I'm going to do now for this particular work demo is I'll go to one of the deployments in Kubernetes. Let's edit something. Something minor doesn't have to be huge. I just want to see the workflows in action. I will commit these changes. I'll create a new branch, propose changes. Let's add some description. Update. I'm going to do this over here. And, you know, let's go ahead and create that full request. So once the DevOps new work creates a new configuration that they want to add, they'll see the different workflows running. These workflows are, of course, just Verno and Verno JSON scans. And then a couple of just a little couple of seconds I'll see that the Pulumi scans passing, the Terraform scans passing. There's something wrong with the Dockerfile and Kubernetes scan. Let's have a look in the Dockerfile that's failing. I can see that, you know, it failed, it failed. And the reason for its failure was that your policy setting authority is not okay with using add instruction in Dockerfiles. So because that exists in the current Dockerfile you're trying to add, it will just fail the workflow for you. Exit with a code one. And it's very visible to you that something needs to be changed. Let's go to the Kubernetes scan that's failing the workflow. And you'll see again here that there's a policy failing. First of all, there was one policy applied to two resources. And you'll see the policy name, let's allow host name spaces failing, the resource that is failing, the rule that's failing and the validation error that it produced, right? You'll see what passed, what failed and it'll exit with a non-zero exit code. Going back, this makes it very evident. Like I said, the DevOps user whoever's trying to create this configuration that this is not okay. And, you know, you can't merge this unless you go back, fix your configurations to make them compliant with the security policies that your platform team has, you know, put in. And once they go back and make those changes, they'll come back, see all the scans, see all the workflows passing and they'll be able to merge, making the commit phase much more secure than it was before. So with that, that was really, you know, an easy take on what your workflow could look like using GitHub Actions, using Kibono, Kibono CLI, Kibono CLI Actions in your workflow. So, I guess before we go on to the next part, let's take any questions if there are any. Not so, I think we can move, no issues. Yeah. All right. So we are reaching the end of our session really. So for references, you know, Kibono community, the very active community, we have, we are online on the Kubernetes as well as the CNCF Slack. We're always ready to look at what users are facing and if there are any issues, feature recommendations and so on. And we have weekly meetings where we discuss these things as well. Of course, Kibono has a really large set of libraries and if I'm not wrong, it's one of the largest. Set of policies available. And of course, we also have a public Kibono policies repository. Yeah, with that, I think we can, I can say we're good with the session. Unless of course there are questions. Yeah, well, I think folks who are watching us can ask their questions in the meantime. So also, I would like to add a few questions for the folks who might have asked this questions. Okay, so let me ask this question. Let's assume that different teams are responsible for defining controls and writing policies. So how can app teams or DevOps teams who own the pipeline tie all of this together? So what's your opinion on that? Yeah, I can take that one. Yeah, thanks for opening up the slides. So in this slide, we mentioned there's a different team that's writing policies with the platform team and the DevOps team or the DevOps user. It could be app teams as well. So they own the code. So they own the CI pipeline. Now it could be across GitHub, GitLab or any source version control system that you use. So, and this is a common question that we get when we talk to users because the security team or platform team, they define controls. They define what should be the policies that the entire organization has to adhere to. And because these are different teams, it's not like the platform team now go and add the policies in every repository of every app team. Like in the demo, we had both application files and policies side by side, but that is not usually the case. That is the case only in a demo. So usually the policies are here or written somewhere else. So that's why here we have defined two different boxes. We call it central policy storage just because those are centrally defined. They are applicable to most teams and they're defined by a different team altogether. And let's consider it's a completely different Git repo and you have a different Git repo where your DevOps team is using it. And you can just pull policies from anywhere. You can pull policies from another GitHub repo in your pipeline. So instead of just referencing a local policy that's written, you can reference a policy in another Git repo and given also supports pulling policies that are bundled as OCI images. So that is another great way to distribute your policies. So you just like any image that you have to use within an org, right? You can just push a policies image to your registry and different teams can just consume that. So making that policy definition central, you can make your app teams life and your life also easier. And the beauty of this is that the same set of policies that you're applying in your commit phase are actually the same set of policies will be deployed in production as well. So you get the same production level readiness or checks in your commit phase. And this is the most common thing where application is done, everything else is done, but your code is not being rolled out to production. Like it's getting delayed. Why? Because security teams said no, you don't have certain checks passing. So to avoid such disappointments, just use whatever policies they define and define wherever it is in your coding phase itself. Yeah. Find the mutually exclusive working of these, in numbers of team, while making sure that the security standards really apply to all kind of cool. So yeah. Okay, great. So also like, I'd like to add another question. If anyone is kind of new to policy writing, so is there any resource or how can they learn more about it? Do you guys have any suggestions on that? Because more of policy is out here. Yeah. Yeah. I can take that off. So there's of course now Kibirno and there's Kibirno and Jason. So first of all, we can go back to the Kibirno policies library. And there's always looking at these policies to understand what they're doing. You're not really starting from scratch, where you don't have reference. You can always look this up. There is the Kibirno playground, which is really cool because you can put your policies, try them out, put a resource according to the policy and play around to finally get a policy that you need working for your requirements. For Kibirno and Jason on the other hand, I'll just take you to Kibirno policies as well. And yeah, you'll see that in Kibirno body go to the terraform best practices. Right, so in Kibirno policies, we've also included the Kibirno Jason policies. We're trying to build a library here. So please feel free to contribute. And yeah, so we have ECS for example, and there's some best practices for ECS and there's S3, and I think there's Docophile as well. Yeah, I think there's Docophile as well. Where is that? Oh, okay. I don't see it for some reason. Yeah, okay, I'll make sure it is there. Yeah, probably not the full request. But yeah, even for Kibirno Jason, we have a playground, let's quickly take you to that. And yeah, so the playground is pretty cool because you can have your policy input and the resource input and really see how things work out. And there are some examples here as well. So Docophiles in spirit, slightly more aware of it. Quainted with it, that's one of your Docophiles. That's a validating policy that's checking that no external calls are allowed. And you can just run the policy and see what sort of result you'd get. So even when I was writing policies, this is how I'd play around and learn how to come up with the best logic and write a policy. Yeah, sounds good. Okay, so I think after that, we will end with one question more. And also I encourage folks to ask their questions. So as I'm like, my app teams are kind of using different ways to provision resources like Terraform and CDK. So how can I have consistent policy enforcement across all of my teams? So yeah. Yeah, so again, going back to whose responsibility it is to define the policies and whose responsibility is to actually adhere to it, right? So again, it's a central security team that says things like in our org, we cannot create S3 buckets in say US West region. So that is a guideline that they provide and everybody else has to follow that. But now the problem comes when you're, when like there are different DevOps team in a large organization and there is not really a mandated way on how they deploy say S3 buckets. Now, one team may prefer using Terraform to provision the resource. Another team may prefer cross-plane, somebody else may use cloud formation. There's so many ways to do just that one task, right? But the guideline is simple. Don't create S3 buckets in US West region. Now that you have different ways to do something, should you have different ways to enforce that? And that is something we want to solve with Kivano JSON. We just want a single way, Kivano policy to ensure that your S3 buckets are not created in US West region. That's why we say here, we support different kinds of IAC types. Now it could be Pulumi, cross-plane, anything. Of course you cannot use the same Kivano policy, unfortunately, because every infrastructure provisioning tool has their own nuance, right? They have different syntax that they use for just defining this is the name of the bucket, this is the region. So those will be minor changes that you'll have to write, but you will use Kivano JSON CLI or a single tool to write policy across these different types of IACs. Yeah. Yeah. Okay, great. So yes, I guess we can end our session with that. If we have, yeah, thank you so much, I guess, Anusha and Anushka for an awesome session. I just like nice talking with you guys and learn a lot about Kivano JSON and policy also. Yeah. Thank you for having us and this was really cool. Thanks, Mr. Yeah. Yeah, this was amazing. And if there are like any questions or even after the webinar, so feel free to reach us out on Slack because the Kivano community is very active. So I'm sure somebody will answer your question if you have any. Okay. I think we have one question like, which tool can we use to keep policy centralized? Yeah. Again, interesting question. It depends on your org. So we most commonly recommend a Git, it could be GitHub or GitLab just because your policies will be versioned. You can keep track of who made what changes. And the way to distribute this policy is like an OCI image or just give your Git reference to your app teams so they can just pull from there. Yeah. Just before we were in the session, we had a question. So yeah, we just took the question. I guess that was okay. Okay, yeah. I guess now we can end the session and okay. So thank you so much, guys. And yeah, that's how we are going to end our session then. Thank you, Anusha and Anushka. Okay, so thanks everyone for joining the latest episode of Cloud Native Live. So we enjoy the interaction and questions like what we are expecting to happen for the audience. So thanks for joining us today.