 about why you should listen to us. So again, as I said, I work for IGNW, where a service is only full stack systems integrator, really focused on helping customers across the landscape with things from application modernization, hybrid and multi-cloud, data analytics, DevOps and automation. And as such, we've done this a couple of times. And by a couple, I mean probably hundreds for customers. So we've helped deploy and manage Kubernetes at scale for some of the world's largest companies and have kind of learned a couple of lessons along the way and have gotten some battle scars. And so anytime we can help share some of those learnings with the community and help share some of our tricks, we're happy to do that. So let me go ahead and Kendall, you wanna tell us a little bit about Fairwinds? Yeah, well, and I would just add that's what we're hoping to help you with today is both John's company and my company have learned a lot of things the hard way. You can go learn things the hard way, but you'll save yourself a lot of headache by listening to folks who already have those scars. So about Fairwinds, we are a Kubernetes enablement company. So we offer services, open source, and software in this space. Our services are primarily focused on build and maintain Kubernetes infrastructure. If that's a lot of like, here's my infrastructure problem, make it go away. We can build that and maintain it for you. Open source, because we're managing lots of infrastructure at scale across lots and lots of clients. We are regularly seeing, hey, this is a macro problem that everyone has. So we'll go write a tool that solves that, that helps us with our service and is also beneficial to the community. We're gonna be talking about one of our open source projects here today. And then finally, software, we do have SaaS products that are often leveraging open source underneath. But if you wanna be sure that you're using Kubernetes correctly, we have a software solution in there for you as well. But go ahead. Okay, so let's dive in and just as a level maker, talk about what's new about Kubernetes. So part of the reason we're having this conversation, getting configuration right is very easy if you're using something that you're very familiar with. Part of the reason that it's difficult and part of the reason we're talking about this today is not that Kubernetes is inherently infinitely more complex than other technology out there, although there's some argument that it is sufficiently complex. But a big part of it is just, it's a new paradigm. Everything about Kubernetes is new and sort of different. It's a different way of thinking about the world. It is sort of cloud reaching maturity in some ways, but it's a very different world from when you had a data center and you could walk around the room and you could flip servers off, flip them on. And I think John has been in this industry long enough. We can both look back on times where, as a sysadmin, you had one server that you could go in and look at the uptime and it'd be like, 783 days and you'd be super proud or 15 years and you'd be super proud. And Kubernetes doesn't even attempt to do that. Everything is ephemeral from the get-go, ephemeral meaning built to come and go. And I mean, anything you wanna say about that John before I get into Uniform APIs? Okay, so the other thing is, one of the things that Kubernetes makes really different is it does enable Uniform APIs across clouds, across your data center. So everything below Kubernetes still has to be configured to run on that hardware, be that in AWS, be that in Google Cloud, Azure, your data center, one of the other clouds. But once you've got Kubernetes there, there's a Uniform API for just about everything. And people do run their compute there. There's a way to run serverless in a Uniform way. There's a way to put your databases there and there's questions about all of those different things. But it gives you a Uniform API that people really like having that Uniform API across different locations. So if you're running Kubernetes in lots of different places, it's easy to just go leverage Kubernetes and go from there. So it's all new, it's all different. And that's part of why we're talking about this today. But once you do understand the basics, it's relatively easy from there. Go ahead, John. So yeah, and as Kendall said, right? A lot of organizations, because this is new, they really start off with, I have a Kubernetes, right? I have a single cluster, I have a sandbox, now what? And so with that, it's definitely convenient and it's fairly easy to manage that configuration manually, just like with the servers of old, where I could lock in and make changes to a config file, I can do a lot through the Kubernetes CLI, I can do a lot through the API directly. But that's kind of where it ends from an experimentation perspective. And Kendall, can you kind of tell us some of the areas where you've seen customers start to run into problems with that? Yeah, so I mean, I think it's, the first time you're kicking the tires on Kubernetes, probably the most likely way you're gonna do that is log into one of the clouds, spin up one of the managed clusters, hop into the GUI, click some buttons and try different things, right? But it's one thing if you're kicking the tires with it that way, if you're at your company and you have your first Kubernetes cluster and it's entirely configured through a GUI, again, nothing necessarily inherently wrong with that, but what happens is when you wanna go spin up the next one, you can't remember all the buttons you clicked, you can't remember all the things you had. And so what we see is companies who spun up one Kubernetes cluster and they have one guy who remembers all the configuration they had or maybe wrote it all down to go spin up the second cluster. And then pretty soon, you know, they're three years on and there's one person in the organization who remembers all of the configuration because none of it was documented in this code, it was all done manually, and you have a person as a massive single point of failure, let alone when you have organizations that start that way and maybe even wrote all their configuration down as code, but as they start to spin up their third cluster or their 15th cluster, or they've enabled the company to spin up clusters wherever they want and suddenly there's 150 clusters across the entire organization with no uniform anything and it's all insecure and oh my gosh, you're a financial institution and what are we even thinking? I would admit that scenario and it wasn't a reality but these are things that actually happen. They are, yeah, unfortunately those are problems that real customers have come to us with in the past and the good news is that the community, you know, this isn't an unknown problem to the community. So there's a CNCF project out there called Flux. It's a great project for those who aren't familiar with it. I highly encourage you to take a look at it, install it. It's not the focus of our talk today but I do wanna talk about it because it's a building block of how we see managing configuration across the enterprise from a Kubernetes perspective. So, you know, first of all, you have to buy in to get ups. Ultimately, Git becomes the source of truth and Flux is a tool for the automation of Kubernetes objects from Git at its very core, right? So it does some things really, really well. It solves some of the challenges that Kendall just talked about. Specifically, it allows us to deploy Helm charts, customizations, which is for those who aren't familiar with customize. It's a language for applying patches on top of Kubernetes objects or even just bear case manifests directly to our cluster by committing those as code to Git. And so we have a kind of a walkthrough here, kind of a use case on how we can use Flux and then we'll talk through how that becomes a really good solution for managing configuration but it helps us keep those clusters synchronized but it doesn't really solve for best practices, right? And specifically for highlighting or preventing folks from doing things that violate kind of, you know, call it community best practices around creation of my deployment and creation of my Kubernetes objects. So real quick, I'm gonna go ahead and pull up a demo and kind of walk through how Flux is useful. So in this case, I actually have several Kubernetes clusters already created. I have two staging clusters and a production cluster. Now, historically, if I wanted to say, create a namespace on my staging cluster, I would just do something like kubectl, create NS, you know, testing one and wait a few seconds and then I would do kubectl and I would see my testing one namespace right here, which is super handy. But as you start to scale clusters, whether they're multi-region and you need to have multiple clusters in multiple regions, whether they're different security enclaves, whether they're just multiple clusters for different workloads, right? There's a lot of reasons why you might need more than one cluster. Keeping those things like namespaces and RBAC and different deployments and everything in sync across all of those clusters starts to become daunting. And at a point, right, there's a point of a huge point of diminishing returns where it's almost unmanageable. And so using something like flux, we can, of course, keep all of the configuration of the cluster itself, the deployment of things into the cluster as code, manage it through get. So in this case, I have an info repo here, which you'll see on my screen. And if I just jump into my staging environment, and this is the root that my flux install in my staging clusters uses, I've got a namespaces.emo file here. So if I wanted to create a namespace on my staging clusters and come in here, and this is just a copy paste, so I don't typo this again. This is a, you know, just a standard Kubernetes object, right? Just a standard.emo file. You can get sophisticated, these can be home charts, they can be customizations, et cetera, but you can also just start with bear.emo. And I'm gonna create the testing namespace. So then I'm going to get commit, adding test namespace. I push that up to get hub. Just to stop you for a second here, John. What you're doing is you have an infrastructure repo and you're just making a change to that infrastructure repo and applying it. And that's all that you've done so far, right? Correct. So all I've done so far is I have a repo that defines my infrastructure as code. In this case, I'm using YAML files. And all I've done is make a change to a file, commit that file to get and push up to get, and I'm using GitHub for this, but you can use any, you know, standards compliant get repo. And now I'm gonna rely on the flux tools that are running in my cluster. There's a set of tools that run in the cluster, the operators that will notice the change in get and take action on my clusters. So I can force that with flux, reconcile. Sorry, I should have jumped in there and helped. Yeah, I mean, reconcile is the word because this is a live demo, awesome this here, John. But yeah, it's going to, what's running in the cluster with flux is watching to get repos regularly and saying, okay, any change that's made there, I should apply to the cluster, right? Right, so you'll see here already, because I kicked off that reconciliation, I already have my namespace on my cluster. So, okay, that's kind of, that's great, that's wonderful, it's managed as code, but, you know, okay, I could have done that manually. Where that really starts to pay dividends is when I start to look at how I scale that out. So if I do look at my other staging cluster, you'll see I should already have this namespace here as well, right there. So I didn't have to do any action for these two clusters. All I did was commit my change to code. Now, I'm doing this obviously committing directly to my main branch, you shouldn't do that. You should obviously do pull requests, you should have a solid Git flow or GitOps flow or whatever your flow is for source control, you should be following that, do peer review and all those things. But for demo purposes, I'm just committing directly to main. But the takeaway here is that it's easy to make changes through code and have those changes automatically pushed out to my clusters. Now, I have two separate environments. I have both my production and my staging clusters. My staging clusters use a separate branch or directory structure in that same repo. But that's great. Like I said, this is a really good method for managing configuration across multiple clusters. What this doesn't solve for though is, and I can do deployments. In fact, if I look at my staging environment here, you'll see I have the book info app, I have the microservices demo that I believe Google publishes. I have all of those, I actually have Polaris deployed through Flux. So I'm deploying all of these solutions or all these software packages through Flux. And so, what Flux doesn't do in this case though is make sure that for example, I have requests and limits set or that I'm making sure that containers don't run as rude or that I have read-only file systems. All of these best practices that have been kind of kicked around the industry and everybody kind of, I say everybody kind of knows these things but there's not really like this definitive list of, I can run through this checklist and do that in an automated fashion with Flux. So really that's why Polaris was born and why Fairwinds went through the trouble of building it. So Kendall, you wanna tell us a little bit about where Polaris came from and kind of why the why? Yeah, so, as I mentioned, Fairwinds is building and maintaining lots of infrastructure across lots of clients. We see what the macro problems are. And to John's point, what's being deployed into Kubernetes and is it healthy? And we realized as a company, while we're responsible for building and maintaining that infrastructure, we can't watch all the time to make sure that everything that's deployed into that infrastructure is healthy. And boy, it would sure save us a lot of headache if the things that are deployed into those clusters are configured correctly. And so that's where Polaris comes in. We said, hey, let's build out a bunch of checks so that we can look across things, industry best practices, things that we've learned the hard way, et cetera. And let's put that all together as code. It makes a single place where you can check the health of the things that are being deployed into your cluster and then actually stop unhealthy things from being deployed into your cluster. And it also will give you a score so that you have some feeling for how you're doing and then give you some specific actions to go and improve things in certain examples. But that's Polaris exists because, no matter how great your infrastructure is, if what you're deploying into it is terrible, it's insecure, it's gonna cave in, it's all of those different things. And go ahead, John, if you wanna talk through this diagram. Yep, I appreciate that. So as Kendall said, Polaris was built from industry experience from teams that have been running Kubernetes at lots of different customers. So not just one organization. And some of those organizations are not quite as mature and some are maybe on the bleeding edge and a little bit more mature in their process, but we still need to provide the same quality deployment, the same quality assurance, quality assurance, there we go, to the developers that are interacting with the cluster. So Polaris actually has three deployment methods that we can use to both audit and enforce the best practices in our clusters. So the first is just as a dashboard, I can fire up Polaris inside my cluster. I don't even have to put it into my cluster. I can run it on my desktop if I want and connect it to the API. And it's just gonna go out and scan the CUBE API, look at the objects that are in the cluster and kind of report back on the health of the cluster. And so I'll show that off here in just a minute. It's a really lightweight deployment. It runs in the cluster, it takes very little resources and it just kind of sits in the background and monitors the API to see what's happening. Really good for giving you that score, that health score that Kendall just talked about. In fact, that's actually something they put in the development team puts in the top of the report there. In addition to that though, a lot of folks are saying, well, that's great, but I don't wanna just know about problems, I wanna prevent them. So Polaris has the ability to also act in your cluster and this has to be installed in the cluster, act in the cluster as an admission controller and specifically as a dynamic admission controller webhook, which will allow us to, when a user tries to create say a deployment or a stateful set or a daemon set, it will allow us to compare that against our set of checks. And if the user is trying to do something that violates our best practices guidance, we can prevent that from being run in the cluster. And so we'll demonstrate that here as well. And then lastly, and again, great, right? But we wanna know before it even gets to that, we wanna know before it even gets applied to the cluster, then we can apply this through a command line utility during our CI process to catch best practices, violations during our CI, during our build. So we can give developers early feedback that, hey, this isn't even gonna be deployable because these XYZ reasons. And of course, all that's tunable. There's a score that's tunable and you can turn out different checks on and off. You can even create custom checks. So with that, I'm gonna jump right into what the dashboard itself looks like. So this is the dashboard for one of our staging clusters that we showed you earlier. And really it's gonna tell us in general, hey, grades B, that's not too bad. And I wanna emphasize, and this is in the docs, but I also wanna emphasize this. If you install this and if you install Polaris and you look at the dashboard and the score is a little bit lower than you were thinking or that you were hoping for, that happens. It's designed to be relatively strict from a standards perspective, right? When the Fairwinds team built this, they really designed it to be kind of rigorous, right? Essentially set the bar high. You can always turn some of those checks off if there are things that you just have to live with in your environment or customize the checks as necessary. But if you just install it out of the box and you get like a C, don't feel too bad. That's not an uncommon thing. So if we look at, you know, here's also... Let me just add one thing there, John. Like for some companies, a C is what you're shooting for. And you're fine with that. And it depends a little bit on a number of things. But you can also go down and dig through the checks that are giving flags and lowering that grade and decide how important those things are to you. And a different score is important to everyone. So... Yeah, and so we're gonna, I'm gonna show that here in just a second. So it's gonna tell us a little bit about our cluster. It's gonna kind of give us the summary. And if we scroll down, it's gonna tell us, we have three different categories of checks. So we have our efficiency checks, our reliability checks and our security checks. And there's a little bit of description here, but essentially efficiency checks are exactly that, right? Are you leveraging the Kubernetes resources that you have to their full potential? Are you making good use of the cluster? And are you making good use of the tools that are available to you? The reliability checks are gonna be things like looking for health checks and looking for, you know, those kind of those standard foot guns where if you don't set these, you're setting yourself up for a failure in the future. And then security checks are gonna be things like you're violating security best practices. And I think I saw that in the questions. I apologize, we will get to the questions, but I saw one of those scroll by. So it does support security best practices. There's some out of the box, Polaris has a bunch of security checks in it. We'll talk through those. It also allows you to filter by namespace. So in this case, you know, if I don't care about say CUBE system because I'm using, in this case, I'm using a managed CUBE cluster, then I can't control what goes into CUBE system. I have to rely on my provider. So maybe I don't wanna filter that. Maybe I only wanna look at my microservices demo so I can filter down by namespaces just to look at the dashboard. And then if I look here, it's gonna tell me a list of all the checks, right? So I have my pod spec, right? I'm using host IPC and I'm not gonna run through all the checks but you can see this in this particular case, this deployment has, I don't know what's this about 15 different checks that it's running. And I'm actually violating a few best practices here. So my image pull policy is not always which is kind of the recommended best practice. My file system is not mounted read only and it is allowed to run as root. So if I were to fix all of those, this would go to all teal and this would be essentially contributing to a higher score which if we go back and look here, I think the only thing in the cluster that has a perfect score is actually the Polaris dashboard itself. Yeah. Nice. Way to go. If it didn't, we would probably catch endless amounts of flack for that. So it does have a, obviously it's following all the best practices. Now, that's great. Again, and this is a really good starting point because it's easy to install. You can install this. If you just wanna do, you know, kubectl apply or helm install, you can install this in like five minutes. The containers are public, they're hosted out on Quay. Real easy to use. You can read the docs and again, it's about five minutes to get started but that really just kind of gives you a magnifying glass and helps highlight the problems. If we wanna start preventing this. This is a good place to start. It's a great place to start. That's actually where we would recommend customer start from our perspective, right? Audit, understand where you're at, get an assessment. But the next step there is really, okay, let's start with a little bit of enforcement. How do we start enforcing? And so in that case, we actually have in our production cluster here, I have, you'll see, we actually have the admission controller deployed. And again, this deploys in managed providers, just fine as long as they're running. I think it's 115 or 116, they can support the dynamic admission webhook and give you some task bar, give you some flexibility there. But you'll see here, we have the webhook controller running. Again, small lightweight footprint, the configuration is all passed in via either via config map or via the Helm chart, but the admission controller is running. So in this case, if I go back into my app's directory, I have what I would call a bad app here, just somebody who didn't follow best practices, we're gonna just manually apply that for a second to the cluster. So if we look here, it's just a busy box container and we're gonna say we wanna allow privilege escalation. So I have this. So now if I go in here and I say tube TTL apply dash F deploy that app deployment, and I try to apply that, the Kubernetes API is gonna reject that. It's gonna reject that because in this case, oh, because I don't have the testing names base. Minor detail, minor detail. In this case, it's gonna reject it because it's gonna tell me that Polaris rejected this because privilege escalation should not be allowed. So it's not gonna allow me to install this deployment because I'm using a feature or in this case a security spec that is not permissible in my environment. So if I wanna fix that and go back in here and of course edit my deployment, I'll go back down here and say, yep, okay, you're right. I don't need privilege escalation. I'm gonna turn that off. Thanks for catching that. And if I apply that now, now my application actually gets deployed. You'll see here, it's pending, my cluster needs to scale up, but I've got a container that's pending and we'll get scheduled. So again, Polaris acting as an admission controller gives you another layer of defense against folks applying things to the Kube API that maybe don't meet your requirements. And again, there's a bunch of security checks that are baked into Polaris out of the box, but there's also a ton of flexibility in creating your own checks. So you have the ability, if you go look at the Polaris docs, you have the ability to actually define custom checks and regex matching and things. So for example, if I wanted to block containers from say Quay from running in my cluster, I can do that via Polaris as well. So... Keep going John. Well, and it's worth pointing out that the admission controller is the last step before something's live in the cluster. And the reason that there's two different spots here, the CI pipeline and the admission controller, in theory, most things go through the CI pipeline, but there might be something where somebody's trying to do something manually to apply it to the cluster, where there's some kind of admin reason to need to go into the cluster directly and that admission controller can live outside of the CI pipeline and stop something before it gets deployed in that's misconfigured even if they're going around the normal CI process. But do you wanna go show the CI part too? Is that part of this as well John? Yep, yeah, so I also have a demo of how we would actually integrate this into a CI pipeline and how we could use Polaris as part of a pull request style workflow to validate that my application is, or my deployment manifest, sorry, is compliant before I even start to apply that to the cluster. And real quick, I just, I did see another question. Do you define admission webhook and Polaris? Yes, so Polaris actually uses the admission webhook and if you go out and look at in the GitHub repo, you will see there's an admission webhook definition that then references that admission controller that we have deployed in the cluster. So it is using the standard sort of call it, Kubernetes native way of creating this admission controller. It's nothing fancy or bespoke. So I'm with that real quick, I'm gonna pull up my, so I just have a simple pipeline here set to run in GitHub Actions. So specifically, and I need to go make my code, fix my bad deployment. So ctl, delete, bear with me one here. Will I remove our, your bad deployment and put in your good one? Yeah, exactly. So here we're gonna go back and we're gonna set this back to, to trip our Polaris, Polaris score. So if I wanted to, so let's say I wanted to branch and get branch, I wanted to create a feature branch. We're gonna call it jlw, you know, cncf test. And then I'm gonna check that out. And then if I go in here to my, in my deployment manifest, need to make a change here, I'll insert a new line or something. So I'm gonna go ahead and now I'm gonna push that up to, up to a feature branch in GitHub. I have, I obviously have GitHub Actions already configured to run Polaris, you know, via the CI pipeline. So if we, real quick, we wanna look at what that actually looks like in GitHub. Good news is, is because Polaris is containerized, running this in any CI system that supports containers is relatively easy. But here in my workflow, I have my PR workflow. Sorry, I actually call it in an action. I have my Polaris action. And all it does in this case, and this is all obviously hard-coded, you would wanna pass in environment variables and be a little bit more intelligent about this. But here I'm pulling down the Polaris image from Quay, the one that Fairwinds publishes and maintains for us. And then I'm just gonna run Polaris audit, and I'm gonna pass in some commands. So in this case, I'm going to pass in the location of the YAML manifests that I want Polaris to audit for me. I'm going to pass an exit code on danger, meaning if I get any danger or any essentially critical warnings, I wanna set the exit code, I wanna fail. And then also if my score is below 90, and this again is configurable, you don't have to use my new, you could use 60 or whatever number works. But so that helps avoid those cumulative, tons of best practices violations, but still allows my developers a little bit of wiggle room in this. So with this whole pipeline, then what's gonna happen is on a PR, so I'm gonna go into GitHub and I'm gonna do a PR and I'm gonna create a pull request into main. I'm gonna say create my pull request, and it looks like my check already failed. So we're gonna go back to my action here, my CI test, I'm gonna go down to my, oh, it's running again, because I created a PR. So it ran on my commit as well. We're gonna go into my CI test, we're gonna go to my build step. And here you're gonna see that this step, my Polaris step failed. And it's gonna tell us right here that danger items found in audit. So this, and this is all JSON so that you can parse it and return it back to the CI system accordingly. But essentially this is telling me that Polaris found a critical vulnerability or a danger in my YAML manifest and thus Polaris failed, so my CI system fails to build. So what I wanna do then is go back in now to, to fix that, I go, oh, right. And if, and sorry, just to kind of show where we would find that in here. If we, we can kind of scroll through and see all the checks that are happening. So in this case, this, this check was successful, this check was successful, all these checks were successful. So we really wanna look for where, here we go. We wanna look for these, right? Where the success is false. So in this case, I don't have a liveness probe because it's a busy box container. It doesn't need a liveness probe for testing. But there's another one in, if we keep scrolling here. Oh, there we go. Probable escalation. That was a severity warning. That's why that wasn't the thing that actually triggered it because you haven't set to only trigger on danger. And this one says severity danger. Yes, go ahead. Thank you. I appreciate that, yeah. So for anybody who's, who's used, you know, any kind of sort of auditing or security scanning tools, this, this sort of methodology should look very similar in that we, we rate the way that the owner bill or that the checks, we rate their severity and then we can fail on certain types of checks or aggregate numbers of checks, things like that. But in this case, this one was danger and we failed, right? So in this case, in this case, you know, we'll have to come back in and fix that. So if we look at our app here, I'm going to go into apps, I'm going to go into my bad app, boy, I'm going to edit this. I'm going to go back in here and say, oh, silly, I forgot to, forgot to disable privilege escalation. I don't really need to privilege escalate. Why would I ever do that? And then we're going to just check that in. So we're going to get, push that up to, what do we call our branch? We call it a feature, LWCNCF test. So now that push is going to then kick off another run of GitHub actions for us right here. We'll wait for GitHub to spin up. And so now if we go back, Clarice is still running here. So you'll see now Polaris, the check has succeeded. I still get the same output telling me all the checks that were run, but my code is now clean. And if I go back to my pull request and I look at my CI test, you're going to see all checks have passed. So now I know this code is safe from merging into main and I could potentially deploy this out to the rest of my cluster, the rest of my infrastructure. So again, a standard CI check process, but because Polaris is completely sort of self-contained it makes it very easy to run these kinds of easy checks early on. And it's really designed to be a really low barrier to entry for deploying this in your workflow, getting started quickly and essentially not having to understand a huge complex tool chain for managing a bunch of simple best practices checks. And so with that, kind of talk about steps. So how do I get involved, John? This is such a cool project. Yeah, so if this is interesting, obviously the Fairwinds team and myself and other Polaris users in the community would love for y'all to install it, try it out. And as I said, you don't have to install the admission controller. You don't have to use it all in CI. Just install the dashboard, install it locally, point it at your API on your cluster, let it run and audit. Just try it out, kick the tires, fix issues, sorry, file issues, fix them if you'd like to, but file issues, if you find them, give feedback, let the developers know, the maintainers know how you feel about it, where you think it could evolve, how you find it useful. And then like legitimately, if you find an issue and you'll feel like you're feeling frisky and you wanna fix it, please, right? PLs, PRs are actually welcome, right? Kendall, I think you- It always sounds sarcastic. Yeah, when people say PRs welcome as like, well, up your nose with a rubber hose, if you do want to do better, go for it. But seriously, we would love for the tool to get better and better. This is a very widely adopted tool. There are a lot of people running it in a lot of places. And so because of its community adoption, it is getting better pretty regularly. And we'd love your feedback, your input and pull requests. So, well, thank you, John. I think it's time to dive into questions. Are you ready for these? I can pose them to you and we can knock them out. Yeah, I think so. How would you compare Flux versus Argo? That's a good question. So Argo, and I'm not as familiar with Argo, but I really feel like Argo is meant to be a workflow, more generic workflow engine. CI is obviously one of the workflows that it can support. But as far as handling the actual task executions and orchestrating them via CUBE, Flux is really designed to be a set of get ops tools. And again, I'm not the world's foremost expert on Flux either. There's a whole community around that tool as well. Highly encouraged. If that was interesting, same thing, right? Install it, use it, give feedback to the maintainers. It's an awesome tool as well. Lots of use cases for it. But I think Flux is really more around enabling a get ops workflow and Argo is more of a generic workflow engine. So I think they can even arguably even be used in conjunction with each other because Argo is really good at kind of the GitHub actions portion of what we just showed, right? Orchestrating a CI pipeline, running checks, et cetera. And Flux is really good at taking that artifact that's been run through CI and deploying it. Yep, great. And let's go on to how do you compare Flux versus Terraform, John? So this is probably, I think this is probably a more valid comparison than Flux versus Argo, at least in my mind. And they're both describing a desired state. So Terraform is describing what you want. And you can manage all of the things we just showed in Flux, you can also manage with Terraform. The main difference is that Flux is a pull-based operation. So there's an operator running in the cluster that's continually looking at a get repo and pulling those changes down. Or it doesn't have to be a get repo, I should clarify. It can be get, it can be an object storage bucket, it can be a helm repo. There's lots of different sources we used to get for this demo, but there are other sources that Flux supports. And so Flux is more of a pull workflow, meaning that once that change is pushed into your code repo, Flux will handle deploying that to the cluster. So whereas Terraform, as I'm sure everybody's aware, you have to actually execute it. Now you can do that through CI. There's maybe some reasons why you don't wanna do that, but you can execute Terraform through CI, but sorry, I had to kill my timer there. But again, I think the once push versus pull is probably the easiest way to contrast them. Yeah, and there are, for clarity, there are commercial products for Terraform offered by HashiCorp, as I understand, that do address some of the same things so that there's a way to automatically watch and pull those things in in the way that Flux does. Flux is one of the open source tools. It's one that John recommends. And we're talking about that because this is a CNCF webinar as well. And Flux is a CNCF project, so. Yeah. Okay, two questions. First, really simple, the KubeCTX CLI. Is that a shortcut for KubeCuttle context commands? We're, okay. Yeah, so KubeCTX is just a real quick, it's just a set of packages. It's KubeCTX, KubeNS, and a couple other really nifty command line tools. Brew install, KubeCTX on your Mac, and there's some way to install it on Windows. I don't know what it is. Or, but yeah, super handy for just shortcuts. That's all it is. Okay. And the second part of that is, is everything as code, your template, artifact approach, fitting the much heralded GitOps methodology of handling DevOps tasks related to Kubernetes? Yes, so the answer is a resounding yes. What we just walked through is a very abbreviated, probably butchered GitOps workflow. So Flux is designed to be a GitOps toolkit for Kubernetes, GitOps is the workflow that we were kind of discussing here, so yes. There's, I mean, the very simple bits of GitOps are your entire infrastructure for your operational everything is enshrined in Git as code. That's GitOps. Now are there way more mature versions of GitOps where certain things pull some ways and other things pull others? Yes, there's a million ways to go really far with this, but at the very base, that's what GitOps is, and yes, that's what we're talking about. Question, this is about Polaris to the CI tool, admission controller and dashboard automatically share a single database of custom checks or do customizations have to be synced somehow? The short answer of that is, yes, they automatically share in the same database of custom checks, anything to add there, John? Sort of, the answer is- Oh, am I wrong about that? Oh my gosh, go, yeah, please correct me. So the answer is it depends. The admission controller and the dashboard can share the same configuration because they can pull from the same config map running in the cluster. The CI portion of that, again, in my install, I'm just using the default rules that are provided from the Polaris team. If you wanted to then take those rules that you're using in your cluster and also use them as part of CI, you would have to, arguably, you should be storing them and get anyway so you could just pull them out before you put them in the cluster, but you would have to have some way of keeping that, the CI system, using the same configuration as what's running in your cluster. Oh, I see, yeah, okay. Thank you for that. Yeah, everything running in the cluster can use the same configuration, yeah. And it's not a database directly, it's a configuration file, but yeah. Does Polaris have all the features needed to replace 100% pod security policies and OPA gatekeeper? And I think, I'm gonna try to answer this one, John, and you can correct me again if I get it wrong. The short of that is, I mean, there's no such thing as a tool that's going to 100% have all pod security policy, everything, there's not a tool that's going to do that perfectly because your policy and security needs are going to be unique to you. That said, and Polaris does not have OPA support. That said, Fairwind's commercial product, Fairwind's Insights, which includes Polaris and other open source tools, Trivi, Kubebench, et cetera, is much more heavily focused on the security pieces and does have support for OPA. So if OPA and having an implementation for in the admission controller, CI and a dashboard is a high priority for you as well as those additional security checks, there is a commercial product for that. And for clarity, there's a free version of that commercial product, but it's not open source the way that Polaris is. So yeah, and I'll just echo what Kendall said is, does it have all the features needed to replace 100% pod security and OPA? No, there are features in pod security policies and OPA that are not in Polaris and there are features in Polaris that are not in OPA or pod security policies. It may be, depending on your needs and your implementation, it may be enough coverage for your use case. It may Polaris may not be, right? So you may have to add pod security policies and honestly, our approach is always a layered defense is the best defense. So all of the above as many layers as you can add from a defensive defense in depth perspective is gonna help build out a more resilient infrastructure. Thanks. Are the checks audits customizable by the customer? The short answer that is yes. And there's a way to write custom checks. It is with JSON and a couple of different things in Polaris, it's not leveraging OPA standards or Rego, Rego, whatever. It's something else for writing those checks, but it is customizable. And which checks you're checking are also customizable, right, John? Yes, yeah, which checks you run. Again, I just ran it with the default out of the box configuration, which is a pretty high bar, but also designed to be really useful with minimal configuration. So the team kind of took the, hey, let's make it as useful as possible as verbose as possible and let people turn it down if they don't want it. So that to give you the most information out of the box, but you can absolutely turn it down and use less checks or add custom checks if you need. And I think the example I gave, which is in the documentation is how to prevent, how to have the admission controller prevent quay.io pack containers from running. So it looks like the next one is comments on Kubernetes dropping Docker. Ooh, that's an hour long talk all by itself. Let's skip that one altogether. So yeah, I'm happy to walk right into that one. The takeaway on that is, the Kubernetes maintainers are doing what they think is best for the community and that's all that matters. If there is a Polaris admission controller in place and you still need to do some quick and dirty action on the cluster, is it possible to configure the controller to ignore rules somehow? It is. So because we're using the scaffolding loosely, not scaffold like the tool, but just the scaffolding that Kubernetes has for admission controller web hooks, we have the ability to filter out by namespace or filter out by tags, which objects we are going to apply the admission controller to. Now that being said, that comes with a pretty significant caveat, which is you've now opened up a loophole for people to circumvent your admission controller. So your mileage may vary on how you actually want to use that. If I were in that position, I would probably in that case, just deactivate the admission controller by removing the web hook temporarily. So I would just take a quick, I would just export that object into a YAML file somewhere, delete the admission controller web hook, leave all the tooling running, but just delete the entry that tells Kube to use that web hook, do what I need to do and then put it back. And to be clear, you don't need to run the admission controller. You can run it in CI so that if somebody needs to do something quick and dirty, they can, they don't have admission controller running. Like that's essentially the admission controller exists so that even in a quick and dirty, you're going around CI and you need to make a change, you're not also implementing some other problem. That's why it exists. And you could actually have the admission controller be a little bit less strict than your CI system. So if you wanted CI, for example, to be super high bar, all of the checks, all of the things, and really just wanted to use the admission controller to stop people from doing things that you know are going to hurt your cluster. You could do that as well. You could run them with separate configurations. Right. All right. Next one. Thanks for answering my first question. One more please. Can Polaris, Farouen solutions and insights provide audit policy enforcement monitoring to vendor specific Kubernetes clusters such as VMware Tanzu, Red Hat, OpenShift as long as they are compatible with upstream Kubernetes API? So that's the question is how compatible are they with upstream Kubernetes APIs? In theory, yes. In practice, maybe not. John, I don't know. Have you used Polaris in one of, in either of those? So I haven't used it on Tanzu or OpenShift specifically. We have used it on different flavors of gates. And to your point, it depends on how far from the upstream distro the vendor has deviated. The examples we did today, just to be clear, all of this is running in GKE. So yes, it runs in a vendor distro. You will see Polaris flagging things that, and you may have to do some additional configuration to Polaris if you're going to use the admission controller to stop it from blocking vendor specific things that may not be up to its standard or that may need to run as root, for example. And you may see it flagging things that you have no control over. Like in the case of GKE, I see it flag all kinds of stuff that Google installs in the cluster. I can't fix those things that are provided by the vendor. And there are legitimate reasons why things are running as root and running with the network elevated privileges and things like that because that's part of the cluster operation. So there may just be some additional configuration required. What I would recommend in that case, though, is running, I would use the admission controller sort of as like a last step. I would really focus on using as part of CI because if your deployment process is using 100% upstream compatible deployment artifacts, then yes, it's 100% compatible with your deployment. It may not be compatible with the deployment that the vendor is giving you in CUBE. So let me get through a couple of these relatively quick because we're running out of time. Real quick, how is Polaris resource usage on the cluster? It's very minimal. That's the short answer. It's not doing a whole lot of it. It's not a thing that's running and checking constantly all the time because it just doesn't, it doesn't need to for that. So it's very minimal resource usage. What are the recommended tools for OPA and talk more about OPA? OPA is an open policy agent. You can go look for that. Open policy agent applies to all kinds of things, not just Kubernetes. It is a standard, one of many standards, but seemingly winning some of the standard battles around writing custom policy for different things. We, there are other tools besides Fairwinds Insights for doing OPA enforcement. Some of them are messier than others, but there's a lot of community development of this and it's still continuing to grow and be adopted. Anything to add to that, John, before I move on? It's a great summary of OPA. Thanks, Kendall. Does GitOps model replace traditional CI CD? No, when you think of traditional CI CD, you're thinking of deploying applications into an infrastructure. What GitOps is about is deploying your infrastructure also through CI CD. So making sure that not just your applications, but also all of your infrastructure code is enshrined as code and being deployed in is where that goes. And finally, John, real quick, Kubernetes dropping Docker. Wow, that's news to me. So is Docker being replaced with a what container engine? It is. So there's plenty of chatter about this. If you just go search for kube Docker deprecation, I believe it's 120 or 121 Docker, but will not be used as the default or as a, not as the default, but as a container runtime engine or CRI. It's actually been that way for a while. There's been a shim Docker CRI shim or something, some such thing that handles making Docker compatible with the CRI interface. And I think a container D is probably the most widely accepted replacement for it, but anything that's compatible with CRI can act as the container runtime environment. That doesn't mean you can't use Docker on your workstation to publish the containers, just to be clear. You can still absolutely use Docker build, Docker push, Docker pull on your workstation. It's just, and honestly, a lot of the managed kube distros haven't used Docker for a while anyway. So, a huge deal. Can we deploy flux itself without using, can we deploy flux using? I want to wrap up, John, so I can hand back to Libby to close this out. Thanks for the question. Feel free to email John, email me if you have other questions. Also, we had our Twitter profiles in there. You can get in touch with us if you have other things. But John, you did great. Thank you for the live demos. And everyone, thank you for questions. We appreciate your attendance. We'd love for you to get involved in either of these projects. Thank you for coming. And Libby, if you want to say anything to wrap up, we appreciate everyone coming. Yes, thank you so much. John and Kendall, y'all are awesome. Thank you everyone for attending. We will see you at a webinar next year in 2020. The calendar is open. Communications have been sent out. So be sure to check any of your marketing folks in boxes. And we've got some exciting stuff coming up. So thank you both again. Everyone have a great day and we'll see you soon. Thanks.