 Hello everyone and welcome to cloud native live where we dive into the code behind cloud native. I'm it I Shakuri I'm director of open source at Aqua security. I'm also a CNCF ambassador and I will be hosting today's show Every week we bring a new set of presenters to showcase how to work with cloud native technologies They will build things and break things and answer your questions Join us every Wednesday here for cloud native live and this week We are lucky to have Jim and shooting from Kiverno here to talk to us about Kiverno in production So before we get going just a quick Reminder that this is an official live stream of the CNCF and as such is subject to the CNCF code of conduct So please don't add anything to the chat or questions that Could be in violation of that code of contact Basically, just be respectful of everyone Thank you. And with that, I'll hand it over to Jim. Would you like to introduce yourself first? Yeah, thank you and thanks for having us I'm Jim Beguaria co-founder and CEO at Nirmata and a maintainer on the Kiverno project It's really happy to discuss what Kiverno does in the policy management space How it compares to other projects and some of the new features and things we're bringing in Yeah, and and shootings, how would you like to introduce yourself? Yeah, thanks E. Ty. I'm Hillary one. I'm shooting down from Nirmata Currently the maintainer of the Kiverno project and thanks for having us Our pleasure All right, so Want to tell the audience who are not yet familiar? I mean the topic is Kiverno in production So probably like more advanced stuff, but you want to maybe give a quick introduction To the project itself Yeah, absolutely. And let me I'll Share my screen as we get started. Um, but just to kind of set the context and you know Talk a little bit about what Kiverno is where it fits why you might want to use it in your production Kubernetes Or should definitely consider this as one of the add-ons you add So Kiverno is a policy manager, right and we the Kubernetes of course has built-in policy objects Like we've all heard about pod security policies and You know, what's going on with their life cycle We think a year about things like network policies But Kubernetes is also of course designed to be fully extensible, right? So this allows third-party policy management external policy management tools Like Kiverno, which is a cncf sandbox project or like opa, which is also a cncf project Along with gatekeeper, which is it's you know, kubernetes policy manager These type of you know external policy managers can be plugged in into kubernetes So kiverno was created, you know, just for kubernetes and We'll talk about i'm just gonna so you're right. This was the kiverno website, by the way kiverno.io I'm just going to click on the documentation link and we'll browse through some of the In the topics here But the idea was to create a policy manager for kubernetes And really adopt and embrace some of the kubernetes concepts, right? Everything we I guess No lovin or maybe even sometimes hate about kubernetes. Uh, kiverno kind of conforms to those patterns Leverages those patterns to do some pretty interesting things And make, you know, the administrator and the user You know, both the the administrator's life simpler and of course improves the user experience The developer experience on kubernetes, right? So by the way, one thing we always get asked is what just why kivam? What does the name mean? So it means governance. It's a greek word like kubernetes So we thought it's a nice term for it's a good fit for what it does So that's how we named the project kiverno All right, so how does it I mean, I'll explain a little bit about how it works where it fits I'll it show a very quick introduction To getting started And then you know shooting is going to help us dive into some of the more advanced production topics of if you're running policy managers And kiverno in production, right So I just want to remind everyone To ask any questions that comes up in the chats Whether you're watching this wherever you're watching this asking the chat if It comes up. I'll read it or jim catches it. He'll answer it and also there's the cloud native live channel on the cncf slack You can also ask questions there. Sorry for the interruption jim. Oh, not at all Let's keep it interactive. Just feel free to interrupt anytime, right because as we're going along. So yeah, please Well, we welcome any questions So just a quick explanation on how kiverno works and where it fits in, right? So I think a lot of us know already You know that everything in kubernetes, of course goes through the api server, right? And the api server has certain phases In terms of how it works. It does important things like authentication authorization It also, you know has admission controls and kiverno fits in into that admission controls part Of the api lifecycle Before, you know some your data is either retreat from at cd the database or it's, you know It's updated and inserted back Into at cd. So kiverno installs When as we'll see once you bring up kiverno It becomes a mutating as well as a validating admission controller Which allows it to now receive webbooks on every api request and then based on your policies There's multiple controllers in kiverno, which will take decisions and actions, right? So the whole idea is now you are able to customize some of the behaviors for various api requests And this could be something as simple as, you know, when you're Retrieving objects or you're updating objects Or resources and kubernetes or could be even something as complex as like, you know, there's a connect request which comes in If you're trying to exact into a pod and if you want to validate that or Enforce that in certain namespaces that connection is not you don't want to allow exact all of this now can be controlled Through customizable policies in kiverno, right? so This is, you know the picture here that we're looking at Is showing some of the major components in kiverno So there's a webhook which receives these api requests and then there's multiple background controllers which do processing For, you know, any policy changes So there the policy controller will listen to updates and changes in policies like when you add a new policy It also triggers periodic background scans. So for any of your existing resources It will trigger a scan And of course if there's a real-time request coming in through the webhooks that gets checked against the policy cache Which is you know in memory. So for faster processing faster lookups And then at that point the webhook will respond back to the api server With the appropriate, you know error or allow or whatever the response may be, right? So if it's a validate operation typically either you're returning in You know, you're saying deny and you've given appropriate error message or you're saying i'm going to allow this mess This request to proceed and eventually hit at cd So that's got the basics of how kiverno works and we'll see a lot of this in action And in fact, what you know, we can also do is for some of this I'll actually just you know We'll switch to a live demo and i'm going to install kiverno and we'll take a look at these components, right? So in the docs I went into the installation section There's a you know, also a definition here of how kiverno installs what it looks like So as you can expect, you know, there's a deployment which manages pods There's a service which the api server talks to and kiverno listens to there's a lot of other resources that Either get installed or some of these are automatically created and maintained By kiverno itself one of the things that shooting will cover in detail is if with our you know latest release kiverno now is fully ha Which means you can run multiple instances for availability for scale And you know, kiverno will then run a leader election and coordinate work Across these replicas across the multiple instances That you have in your cluster itself, right? So let's get into it. Let's you know I'm going to just copy this command over here, which as you can see it's just pulling down some yamls from the kiverno repo And we'll see how that works. So switching to my terminal. I'm just going to pull down These yamls. I can see a bunch of things, you know, got created. There's the webbooks. There's the Generate controller. So similar things like we saw in the picture, right? So now let's look at, you know, what exactly happened So if I do By default it goes into a namespace called kiverno. I mean you can customize this with the home chart But if I do get pods here I should see there's the kiverno pod work here running and if I do logs Let's say and we want to Check on the logs for this We'll see what exactly the controller did. So a few things you can see it created It checked for the webbooks it created those if they don't exist And then if we are looking over here, you'll also see that one of the things kiverno Is doing is it it will not say it's ready Until it receives a signal back from the webbook It says that okay now i'm you know kind of ready to start processing requests, right? One other thing that you would see in the logs typically is kiverno will also run a leader election So here it will you know, since we only have one instance. It doesn't really Mean much but it because that instance will always be the leader But if you were running multiple replicas it would automatically run a leader election and one of the instances would you know Register itself as the leader And then it would be the one so all instances will serve api requests But that instance will be the one coordinating some of the background tasks across the replicas So that's how easy it is to get to Go ahead Sorry, there was a question in the chat. I think that can be a good time to break in so If a background scan reveals a policy violation, what can be what actions can be done? by kiverno Yeah, good question right so currently what kiverno does and you know, we'll see um, some of this in shootings demo also Is kiverno produces a policy report which is another resource? And it will you know, it will update that with any violations in background Um, in any background workloads or resources So kiverno doesn't automatically shut down any running, you know workloads right now and we are you know, there's discussions in the community In the kiverno slack channel About whether that should be allowed like mutating existing resources impacting existing workloads But currently what it will do is it reports that it will create the policy report With the violation and it will also in the metrics Which are created and reported through prometheus kiverno will report that there's existing violations So the idea is you can now detect those and you can you know, kind of choose how to enforce those Um, but kiverno will not automatically, you know, it's not intended to just shut down or impact existing workloads in that manner Great. Thanks. Uh, there's another question. Um, Which is maybe a clarification question Uh, it would be nice if you could say how kiverno detects policy violation. That is which mechanism So it's a little bit the abstract question, but maybe you can say a few words about how kiverno works Yeah, no, absolutely, right? So we we talked about some of the here how it installs, etc But what I did not show yet is What exactly a policy looks like right? So it's a great question and that's uh, something we can dive into right now, right? So in kiverno, each policy is a set of rules and you can of course group these policies however you wish but Ideally each policy is a set of rules and um, those rules will have different match and exclude blocks So you can match various resources based on kinds names label selectors Uh, things like namespace, you know, even you can select particular namespaces by labels So or even like user information. Um, that's in the api request, right? So you could write some fairly complex, uh, logic to match and select resources first off Now once you select a resource, you can write, you know, validate, mutate or generate A logic in your policy and you can of course have a single policy that combines some of these for a set of functions So what that looks like and I'll just actually we can look at some, you know, examples So let's say I want to, you know, do some validation, right? And this is a simple validation And let's say for, you know, in my clusters I want to make sure that every namespace has a particular label or a prefix or something like that So this is how simple a kiverno validate rule is right. It's uh, kiverno policies Are kubernetes resources. So they're custom resources They use, you know, yaml for their declaration. Of course, you can create them with json or through the api or any other, you know Where you wish, but ultimately it's a kubernetes resource and you can write them as declarative yaml, right? So this policy what it's saying is If, you know, if I have a namespace which Does not have, you know, this a label with this type of it with a value type and the, you know, some or The label with the key type and the value small medium large I'm going to block this particular request, right? So you're requiring now that every namespace has a label with the key of type and a value of one of these, you know Fields and that's all that this policy. This rule is very simply doing But or you could have much more of course complex rules, right? So if you want to know This is a generator rule, which is kind of Saying I'm going to add resource quotas things like that I could also, you know, if you kind of look at some other Things like for example If I want to let's say let's go to a policy And I want to match, you know, a particular selector in this case So I'm going to check and make sure that the selector is specified And if it's not again, I can block right so several things and you know, you can of course you can Also do if you want more complex operations like in this policy What's happening is we are checking some data from the api You know the api server itself. So this the way this policy It works is it's saying match ingresses and then, you know, go and check and see If you know if based on the configured in ingresses So this is the path and you can of course test all of this through kubectl You know, I'm going to go and store all my configured host names And then I want to make sure that the incoming ingress if a new ingress is created So I'm going to filter on the operation and make sure that that host name is not already used, right? But this is, you know, how simple again writing the policy is with a few, you know, kind of lines of yaml You can, you know, have some fairly powerful logic To be able to look up data as well as to be able to validate data or to mutate, right? So kivarno also allows full mutrate through You know, either a jason patch syntax, or you can also use You know The same strategic merge patch that customize uses you can also use in kivarno Thanks for the intro. Yes, very good Yeah, so I think um, let's let's actually try so I install kivarno and kivarno is running But let's make sure it's working, right? So what I'm going to do You know to make sure it's working one of the most, you know common use cases for kivarno is of course around pod security, right? And I think everyone's heard that probably or hopefully by now that pod security policies psds have been marked for deprecation Um, there is still we have till you know version 1.25 of kubernetes before they are removed But if you haven't implemented psbs and are looking for options right now Um, you know, you could of course use psbs and then look at upgrading to whatever replaces psbs as an entry option So there is work going on in segaught sick security and other groups To come up with the psb Entry replacement but kivarno and opa gatekeeper also provide More flexible more feature rich ways of managing pod security And in fact the nice thing is all of these are based on now the pod security standards Which is something that kubernetes, um, you know, it's in the documentation You can see what controls need to be validated And verified right so this is all now standardized and which allows the use of policy engines like kivarno So let's do this i'm going to actually run this command To install kivarno and you're i'm doing something interesting right i'm using customize And i'm pulling down a bunch of policies and then i'm applying, you know, um, Basically those policies into after the customization i'm going to apply them into my cluster right So you can see it pulled down all of these you know different pod security policies Which are listed over here In terms of what uh, what to do right so everything from Requiring run as non groups Uh to checking for host namespaces the host paths things like that Now behind the scenes what just happened and this is showing the flexibility and the value of using kubernetes resources Internally if I go in here and look at the repo where I pulled this from I see there's a customization and this is very simple. It's just saying okay go into bases restricted But I can also go now further down here and if I look at my customization here It's saying that you know, these are the types of policies I resources I want to pull And i'm going to patch these resources to change my validation failure action to enforce So by default these policies are written to audit to run in the background mode But I just customize them with this one simple command to go replace all of them to enforce right and Customizes fantastic in your you know dev ops pipeline or if you're using githops and kubernetes just works really well with that And kubernetes policies being again kubernetes resources all of this fits in nicely together So now to test those policies. I'm going to use this site, you know We like to use this for demos it called it's called bad pods Because these are pods which are wide open things you should not be running In in your production clusters or even dev test clusters for that matter These are things you should you know kind of secure against so we'll drill down into these manifests I'm going to look go to into everything allowed So the baddest of the bad pods right to it as everything open like you can see all these open doors And i'm going to actually pick a deployment right which is saying allow everything, you know Let's let's see if we can get this Running in my you know local cluster, right so we'll do We'll just grab that raw yamo and we're going to try and run it in my cluster And what kimono just did is it's because I had my you know pod security policies installed it immediately said No, don't run something in a host namespace. Don't use host path. Don't you know run privilege contra Containers and of course you don't want to run containers as root, right? So all of these things got blocked all of these are based on the pod security standards And again, that's what in a few minutes you can go from having a fairly open cluster with no security To a pretty secure cluster Just by grabbing, you know the the standard pod security policies And you can of course again the way kimono is designed There's a lot of flexibility in how you apply these how you roll these out It will not impact your existing workloads of which you know provides quite a lot of value So so that's kind of a very simple, you know Demo of you just installing kimono getting started And you know, I will dive into some of the more advanced topics Uh as well, but I just wanted to you know start out with that Sorry, I was muted. Uh, thank you jim. That was really great Actually two questions that popped up while you were speaking. So one is can we tell kyverna which api version Of resources we want the enforcement to apply to Absolutely. Yeah, so you can you can um have api versions kinds. Um all of that detailed In fact, one of the policies that we just added into our Into our new sample is a policy to check for you know, as you probably also saw recently There were a lot of different apis which got deprecated in In 1.22. So I'll quickly show this even it's a good policy to have in your clusters So this check deprecated apis it shows so all of these api, you know resources as well as versions are marked now For removal in 1.22. So not only are they being deprecated, but they're being removed So you can put this policy in your cluster. You'll immediately get a report for where they're being used And you can you know control again, you can manage then you know replacing these with the allowed api versions Yeah, thanks Um, all right, I guess we can move on. Uh, did we lose a shooting or? Are you still with us? Can you hear me? Oh, yeah All right. Yes, I I guess we can move on Okay, yes, I think shooting will cover, you know, some of the more advanced production topics Yep. Oh, let me share my screen. Can you see my screen now? Yeah Okay, great. So, um, yeah next I'm gonna talk about the details of the high availability of Qverno how you run that in your production cluster And then we'll go to the like how you monitor the Qverno with the permit use matrix And uh, after that I'll switch to the demos I prepared for today I'll cover the basic mutate and validated policies. I'll show you the policy report And then we'll talk about the permit like the policy reporter That is like one major feature Contributed by one of the like a contributor Frank out there in the from the community Okay, so um the first topic I'm gonna Discuss here is the high availability of Qverno Like uh, james showed you like quickly showed you before this is the architecture of Qverno those like Blocks in yellow is the resources that are managed by Qverno directly and the block Those square blocks in gray is how like the Kubernetes default components And all the ones in blue is the Qverno controller here So Qverno runs majorly like majorly three controllers One is the webhook controller and then there is the generate controller and the policy controller The the the ladder to controller actually Like runs in the background process So it is not running as long like along with the mission webhook And here the webhook controller or you call it webhook server Which serves all the admission requests and then apply all the policies like the mutate and validated policies to your incoming resource and then write that to etcd Right, so with ha with like before Qverno release one for all there is really no ha available here. So that um, if you're running Qverno and Large scale of cluster it puts a lot of pressure on the single instance of Qverno So we see a lot of requests for this feature and with one for all we Enabled the ha support With the leader election built in in Qverno controllers and uh to dig into that uh implementation Here the webhook server Like once you spin up multiple instances of Qverno the webhook server will serve all the admission requests That means the Kubernetes service will distribute it all the incoming admission requests to the running instances And uh, that's it like Distribute the the workloads evenly across instances and as for the background controller We sort of enabled this leader election for the general controller and the policy controller because those those two controllers are really Not that the time consumption is not really important there So as the background process we can process the general policy and the policy report like um Asynchronously right so with the ha that um, you can you can depend depends on your scale Cluster scale you can Configure the instance the replica number to either three or five Or even more instances to distribute it all the admission requests Okay, um like before I I I switched to the demo. I'll briefly talk about the Monitor ability of Qverno, then I'll show you how HA works in the live cluster All right, let's go to monitoring Here with one for all as well We have this monitoring ability which Qverno will expose all the metrics to the pometheus to the pometheus and then um, we exposed uh Let me Zoom out a little bit We exposed five metrics to pometheus and here you will have the record of the policy and the rule counts the rule the policy and the rule execution Results and the poll the rule execution latency as well as the admission review latency And you'll also get the policy change counts metric With pometheus and let's let's let's look at the Pick one or two metrics here And take a close look at those Metric like for example, let's say this admission review latency Right, let's say as a cluster administrator You may want to know like the average latency of all your admission requests Here are these metrics admission review latency metric Provides you that kind of information right in this case You um, you can know how fast or slow that all your admission reviews has been For in fact for incoming requests or you can like be alerted that that like a certain policy that takes a lot of Time to process the admission requests so that you can tune your policy either the policy or the resource To reduce that time consumption right and with those labels available You will also be able to Criterate different results based on the like different Labels in this talk. We also provided some useful queries that you can check the average latency By the resource time or I can check the max like latency that The the the an admission request takes and etc right, so this is one of the Metric we exposed the admission review one and then let's go back pick up another one Let's say this policy and rule counts metric here this matrix provides you the like historical data Of are your like previous or the existing policies based on the different metric value And here you have the information Of what kind of policies you have installed in your cluster before and what are the like policies available right now and um, the one interesting Thing about this metric is that with kubikado, you're not able to see what type of the policy or what type of the rule you're Um currently installed in your cluster, but with the different labels. I believe there is a rule type, right? So here you can probably like what are the like what are the mutate policies I have installed in my cluster What are the validated policies and generate as well so you can have like an overview of The current the current status of the policies installed in the cluster Okay, before I move to the demo. Is there any questions? there was one question about um sharing the link to the deprecated api's policy and we will share that Very soon. Just it will appear. All right. There it is just now And yeah, I don't see any other questions. Oh actually, okay Yeah, let's go I think jim has that as the link Maybe has that link and uh, he probably was sure with you As we go All right Okay, let's go to the demo like I mentioned earlier. I want to show you how the HA works in the live cluster just to give you some context today. I have uh, Looks like we lost shooting Okay, um Then she's back Oh It dropped Yes, it looked like that Yeah, yeah, just um like what I'm saying is that uh to give you some context about my cluster I'm running a single node if you could just re-share your screen because we don't see Oh All right now cool Right. So um, I'm running a single node mini cube cluster here And I have already installed kubernetes three instance What you can do here is use customize because I said about the committees or the Dashboard today, so I already have it installed But uh, it's really simple to do right to install kubernetes with him or with install.mo as jim showed before all you have to do is use this ham I Install kubernetes Use this ham command and specify the replica count Like your desired replica count and then Ham will take uh, take care of the rest and one thing to notice To note with the ham install is that kubernetes by default Installs a bunch of like best practice policies for you. Those are all validation policies And if you do kubernetes get cluster policy You will have those are the policies installed by ham Like those are the default built-in policies and all of them are in audit mode So it won't impact on any of your incoming like admission requests Okay back to our agenda. I want to show you how HA works, right before I Start the demo. I want to show how you can determine which kubernetes pod is currently the leader Right to do that. We have you can check the lease resource created in the kubernetes namespace as you can see here We have two lease objects here one is kubernetes another is webhook register If you recall the architecture I showed you before kubernetes runs three major components one is the admission webhook Server or the controller the other two is the background controllers the generate and the policy controller, right? so the kubernetes Object here is really the leader of the background controller And the webhook register Is the component that is used to register the admission webhooks Like if you do kubernetes get mutating and validating webhooks You can see those are the webhooks registered by kubernetes and I have some for making it as well But yeah, that's the idea And uh, if you take a close look at Let me clean this up If you take a close look at this lease object, you will see the holder ID here. It is basically Has the pod information the pod ID and the concatenated with the random UID here, and let's get the qrunom pod again and see Which pod it is for Okay, the third one is currently the leader of the background controller as well as the the webhook register, but um Also, like I mentioned before the webhook server Really, uh, not use this leader because the all the admission requests are evenly distributed to each and every running instance so what i'm going to do here is that I want to install a new false policy and i'm going to create a pod That ballast is in false policy and then um, I'll randomly kill one of the pod It can be leader. It can be like non-leader It doesn't matter in this case because we're talking about the the webhook controller here, right and what I expected to see is that Qvernos should keep like blocking my resource creation no matter like one or two instances done, right? so To do that, let me pull up another terminal And I wrote this little script to continuously create 50 50 pods the image engine x latest and what I have What the cluster policy I have one of the cluster policy is disable latest tag Is that is really that um, I want to disable the pod creation if It's using the latest tag if you take a look at the policy You'll see that it matches the pod and the pod controller say that I want the image hat uses a tag and the tag can be the latest Right. So what I'm going to do is that I want to change this policy To enforce that means I'm going to block that And then I'll start that script and Let me update it to enforce Let's double check the policy. It's set to enforce. Okay, great You can see this policy is now at an enforce mode and uh, let's manually run like one engine next pod using the Latest image and see what that happens Right as expected, it blocks my resource creation, right? So what I'm going to do now is that I want to continuously continuously Create this resource and um, I'm going to kill one of the terminal pod and then The given the rest of the run instance should keep blocking my resource creation Okay, so let me prepare the command Okay, action, you know delete pod Let's choose the leader. Right. This is the leader resolve before and let's start the script See I have this attempt number here. Give or no keeps blocking and let's kill the leader So it is still blocking None of the creation requests pass through Okay, now the pod is deleted. Let's check that again You'll see there's a new pod coming up, but um, this those admission requests are blocked like no matter One or two instances done, right? And the same thing with the background controller You may wonder like the general controller If you kill the leader will that impact the the policy application? The answer here is no and we have the recording of the demo available on our website Like if you're running kubernetes and multiple instances and you are applying some of the General policies and if you kill one of the Looks like we'll ask shooting again Jim, maybe you can pick up sure and I think um Yeah, another pick up but we're back Maybe it's a good time to take I see there are a couple of questions as well or Yeah, if you see them, um Jim you can uh Start answering it. Okay Yeah, we have you back and we're we're gonna take a quick questions break So One policy one question we had is how can the policy affect the performance? Especially when the policy is user generated and the user may make some mistake which can be performance Right impact Yeah, no great question. Right. So policies and admission controls, of course are very critical In a system. So I think the best practice Is you know, there's no easy answer to this, but it's just Treating policies as code and following the best practices we do with any other, you know, kind of code making sure, you know, we have and kubernetes also has some Um integrated testing abilities, right? So you can Of course apply, you know some tests in your cicd pipeline to kerono policies But then also making sure that you are introducing, you know, these in your pipeline Just like with any other like your workload itself I mean if your workload, of course says as bugs as things like that You expect some of those to be caught in qa and staging before it hits production, right? So that's why it was very, you know, one of the design decisions for keverno is Really embrace that policy as code philosophy work really well with get ops other best practices for Managing these policies and then of course there's, you know tools like with, you know, if you're using External policy management tools which integrate with keverno like nirmata and others There's other validation and other checks that can be done But it's those best practices, you know, the policy as code principles, which are important Is it also valid to say that uh, since Keverno is Keverno policies are yamel. It's not If a full programming language, so there's less room for mistakes to be made That is true and keverno also doesn't allow one thing from a security point of view It doesn't allow calling into arbitrary like sites, so you cannot call some external you can make an api server call You can call secure registries that you configure, but you cannot call like just another application, right? So one one interesting, you know activity going on in the cncf security tag is just Sort of looking at the, uh, you know threat model and security posture of admission controllers off this, you know, this whole model to say How do you make sure that your policies themselves cannot, you know, introduce some vulnerabilities and things like that, right? So that that is a you know, that is a concern, of course and The way again we have designed keverno is to reduce that Server that attack surface and make sure that you're only calling trusted systems that are configured and your policy cannot Make some arbitrary call And it it's declarative, right? So some of those things like we you know, like You know, there was a someone filed an issue and this is A fix is in progress where if you're if you end up in any like loop and things like that, you know Kevano itself can validate that right and it can abort Of course the api server has a lot of mechanisms to prevent that it has throttling So it it recovers very well from those type of situations But there's other things going into kevano to help with that as well right Thank you and another question was about managing exceptions to policies if we create a c we don't want it to apply Universally can we do that? Yes, so there is an exclude block and you can in any policy you can exclude based on user names on labels on namespaces on kinds There's all sorts of you know ways to exclude certain So it's one common thing to do is to say I want this policy to apply to everyone except maybe cluster admin Mm-hmm Yeah, just add to that. We also have namespace policies. So what I think maybe we lost shooting again, but uh, yeah, so I think what she was saying is absolutely a great point So if you want policies to only apply to a particular namespace Um, kevano also supports that so you can have policies only configured in certain namespace In fact, one interesting thing you can do is you can use policies to generate policies as well Right. So which kind of you know, there's a little bit of inception But uh, if you want to Yeah, if you want to create policies on the fly For a certain workload you can right because it's it's all kubernetes resources after all and uh, kivano can Detect a trigger that a namespace got generated and you can install a set of policies for that particular namespace based on some label or things like that Interesting Another question, uh, what if you kill all the kivano pods or maybe you can generalize the question. What's the failure policy? Uh strategy of kivano Yeah, that's a good question because um, so admission controls have a failsafe over there Where you when you are configuring an admission control you can you can tell it In the web book, you know configuration whether the api server should Should stop the operation if that admission controller is down or it should still allow it, right? So by default currently kivano You know God says installs with failure ignore But if you have policies that you absolutely want to enforce, you know, the recommendation is to say Change that failure in the admission web book, you know configuration To to actually fail and not allow that request one Something we're building in the community and this gave us, you know is being Discussed for the 1.5 release is how do we dynamically allow on a poor policy basis that failure mode? Right. So kubernetes allows it for every api request for a particular web book server, but The idea, you know, one of our community users came up with to say is Hey, but certain policies I want to fail and other policies. I don't care if they're ignored, right? Like pod security. I don't want to compromise on but okay Maybe if somebody's missed the label, maybe I'll allow that and you know when kivano is back up and running Yeah, I'll I'll go it will produce a violation. So I know So you can now, you know with that feature, you'll be able to make those granular decisions Today you have to configure the web book As either fail or ignore Yeah, very cool High shooting we have you back so I guess we can We don't have a lot of time, but there was another section that you wanted to discuss if you can Present it quickly Yeah, I want to show the pemekius matrix as well as the grand fauna dashboard and So like I described before we exposed this those five pemekius matrix and you can have external tools Consume those pemekius data to have our overview of like each metric data there And here is the detailed blog post I have written before It tells you how to like set up a grand fauna dashboard a bit pemekius and with the very detailed steps and After all the setup is done. You can have this nice view of Nice view dashboard of the overall policies as well as well as the policy applications Sorry policy violations and uh, let me switch back to my terminal and uh, I guess I'll drop Like skip the mutation and validation policies application We have a bunch of simple policies available on our website Feel free to like try it out and what next one what I want to show you is the policy report Right. So let's do kubkato get policy report dash a Since I have all those cluster policy Cluster policies installed and I have few workloads running in my cluster I can see those are the policy reports generated based on like the existing resources We can see that in this namespace. We have 63 rules passed and three rule failed And let's take a close look at each of the policy report Let's do kubkato describe policy report and let's do Give it the name and the namespace Um, so here I grab on the status. You will see the detailed information of each policy application Result or at the rule level here, right? So let's take an example I have a stifle set running and I have this policy to disable privileged containers With this rule to auto privilege auto-disable Privilege container here. It is the pod controller, right? So this is the auto gen feature that is enabled and that captures the state the violation Or here is the like success success result on the stifle set And if you want to see one of the failed results here is that I have a demon set I don't want this demon set to use the host port and here apparently it does not have this Security context defined. So it evaluates one of the policies Right, so we have these these these information are available through the Um, the kubkato command, but you don't have the overall view of like Like the the the policy violations of the application result What I have is thought here is the policy Reporter contributed by uh frank and the policy reporter Has three major functionality One is to send those policy report information To the external targets like grafana elastic surge slack, etc And uh, the second one is that without those external tools installed in your cluster It can also bring up the policy reporter ui locally and that Gives you the dashboards of the policy with reports information and the third is integrated with the permit here's matrix Is that it scripts all the data exposed by permit use? and Give generate the dashboard based the dashboard based on these policy reports information And here is an example of that. I also have it installed in my cluster So what I can do is to show you that The dashboard so I let's let's let's look at the applications I have been stopped here So all the pods are running under listening space cattle dashboards and I have a service deployed What I'm going to do grafana service Deployed what I'm going to do here is I want to forward all the traffic to my local port So that I can see the dashboard from night from my Like local browser Here is the grafana. Let's log into it and uh You can search all the Kivrono related or all the policy related dashboard Here we have three cluster policy reports details policy reports details And policy report, right? Let's take a look at the second one here. It gives you the general information of the policy application results We have 63 rule paths and three rules felt And if you scroll down it gives you like the overall report of uh in in different namespaces And uh, here are the details Of each of the like rule applications out right here is the past resources And we have a lot of paths and here are the failed results Okay, um, I guess that's all I want to show you for today. And if there are any questions regarding Thank you shooting. Uh, that was a great demo. I don't see any Other questions coming up But I think this this was a very good talk about the policies in general in kubernetes, which is like the fundamentals, right? You have to have something and uh, I was personally happy to learn more about kivrono and What's unique about it? Uh, and also some advanced Scenarios There is a question coming up I saw integration with grafana loki. What do you mean by that? Yeah, so, uh, let me share my screen again and go to that policy report documentation Here is the information about grafana loki. I believe um, there's the policy reports generated in your cluster The policy report gathers or collects those information and exposed to the grafana loki targets So you'll get the notification or the alerts of what the like either the valuation Or the pile like success successful policy application results I suspect maybe the question is about what is grafana loki? possibly Locality is more of a log management tool, right? So it uh, it's in the grafana family and it kind of shows more log data Um, so I think that's what you used here um And what's being what policy reporter will stream? Into loki is like policy results as they're occurring So by the way one one thing I just want to call out. So frank, uh, joje lay who wrote this tool He has you know donated it to kivrono. So we are working with frank to bring this into natively into kivrono And just make it part of the kivrono project So, you know with that when once we do that we'll have an integrated sort of ui and policy reporting capability To even send things whether it's the grafana or prometheus slack Other upstreams as required So thank you frank great work and happy to see sort of the community grow with that Yeah, that's great always with open source to have Significant contributions. That's really magical Um Is there any resource to look for on the best practice? To metrics to monitors. I mean, I guess the question is which matrix to monitor So policy execution is of course one Concern that always comes up is how much time are my policies taking like what we were also discussing previously If there's any, you know, kind of lack of efficiency things like that So certainly that comes up and then you also want to watch for policies which are not being So maybe if you missed, you know in in your match and exclude Perhaps if it's too relaxed and if policies are not being applied Those are other things in metrics that would be fairly critical of course in production to always look at and see Through this, of course, yeah And I see the other question which was the integration with slack alert manager things like that So yes, that that is also policy reporter has some of those and you know, you would want to create alerts if you're seeing Your metrics sort of dip below the expected threshold or if again Execution times are starting to grow Things to that nature Awesome, thank you If there aren't any more questions Seems like There aren't I would wrap this up and thank you Jim and shooting For this very interesting talk Thank you our pleasure and thanks for having us Thank you Thank you. It's been a pleasure To the audience I want to say again You can stay in touch in the cncf slack under the cloud native live channel And that will see you again next wednesday All right, thanks. I thank you Jim shooting and everyone