 Good morning. Good afternoon. Good evening wherever you're hailing from welcome to a special episode of open shift TV. We are joined by Tyler hour back. We're going to be talking about avoiding deprecations and preparing for upgrades with open policy agent Tyler I'll let you introduce yourself. Tell us what you hear tells we're going to talk about today and we'll dive in. Absolutely. So, hello everybody. Tyler our back architect with red hat open innovation labs focusing mostly on kind of site reliability engineering type work, which is what led me to to a lot of this you know I'm out in the field working with a lot of customers and a lot of times, as much as we'd like to have folks who are, you know, kind of remaining with the latest upgrades latest versions, you realize that that tends not to happen sometimes there's some skipping involved and as we know deprecations roll through with every version and sometimes you don't always catch them. Right. Yeah, they don't always make the news like Dr. Shim or something to that effect. They're not all big shiny things sometimes they you know it's just a small one, you know, release note about about what's going wrong. So we found myself in this position enough that I of course you know as I tend to do reached out to the great blue bird site for help I said, Hey, there's got to be someone, someone working on this there's got to be I don't know how we're capturing this where it's an actionable thing that we can, we can kind of, you know, you leverage as part of like a CI CD process or let's catch this before it becomes a problem right. As it does sometimes good sometimes bad you got got some information back so, you know, read, said SOS someone sent help, what are folks doing to to to take care of this. And as he tends to do, Duffy coley reached out it's like hey, this is a this is a cool website, go or this is a cool repo go check this out. So, at that point, you know, took a look at this little project called deprecate, which was capturing all of the information from the Kubernetes release notes. And it was capturing them at, you know, policy as code is using rego from the open policy agent project. Nice. And it was capturing, you know, that type of information but it was making an actionable you could you could fail. You could use that policy and fail it. If it was a certain version, you know, hey, you're using this API version and it's not available anymore it's gone it's it's disappeared into the ether right we deprecated. So you can use it, you can, you know, just like the rest of your code rather than having to do something at runtime, you can leverage that policy as code to take a look at the manifests, the the pure YAML to take care of that so you know so using rego I was able to start you know taking a look at the things I want to deploy and matching it up against with these policies and seeing what fell out. And I found that a lot fell out because YAML what one one thing that that I've you know, I personally have been victimized by YAML I've lost probably weeks of my life to YAML at this. Oh, yes. I used like I used to call myself a calendar driven YAML engineer right like very much have wasted a lot of time. Exactly. Let's see what the animal. So the, the thing that you know so there's a missing piece that's like, I don't need to run open, I don't want to be a have to run open policy agent as part of my kind of pipelines just to be able to try to enforce these things. So the missing piece kind of fell into place. I think this was maybe cube con two years ago or so now. Weird and in times because I don't remember when things actually the last time I saw real people. Right. Yeah. So it was maybe maybe two years ago. And I saw this tool called Conf test. I was able to use this. I was able to take that com test tool, use, take the the regular policies and just say hey com test use these regular policies against these manifests, these charts, and tell me what goes wrong. And I was able to start pulling them together and I was like, cool, okay, I've got all the pieces now right so I've got, I've got my policies I'm able to capture all my deprecation policies in these in these policy files. I've got my com test tool that allows me to, you know, take action with those policy files and I've got my, my applications my yaml that I wanted to play. So, bringing that all together, I was like cool I'm going to, as one does an open source I showed up with a bunch of duct tape and you know wrapped all these things together and said hey cool now I've got a thing that works. And the great thing about that is that a handful of us were also all looking at the same problem at the same time. So I started looking at it from him when he used this deprecate repo that this guy's already this person's already publishing all these policies for me. And I'll use com test it's already been, you know, put together to be able to take action on that. And I published a little blog on it said hey, this is a cool thing I did I was using get have actions at the time. And look, I can check this anytime I checking code I can check against these policies to see what goes wrong. And at the same time, Steve Wade, you know, shout out Steve Wade to was also kind of crank it on the same thing and he had started building on top of this because that it was a that deprecate original deprecate repo was only available up to a certain version of Kubernetes and it was kind of getting long to that point. Yeah, he kind of took that took that ball and kept running with it and publish these to a repo called deprecation which we'll take a look at here in a little bit. And again the great thing for me was this was all just open source out in the wild I don't need to maintain this myself I just pull that in as a resource. And it's available to me right so I'm getting these things for for free for me. And I'm able to just rely on them I so you know part of me was like, Oh cool so I can use this for my personal project but then you know thinking thinking you know, bigger bigger picture here is. And then rely on, you know, I can set my policies for my team. I can then you know as an enterprise kind of organization level, maybe a bunch of other teams also have their policies that they want in force, hey, you know we don't want anyone to try to use ingress objects we're running on open shift so it has to be a route don't don't go the ingress, you know method right now because it doesn't do all the things we, we like to do with it or that's not the thing we're monitoring so let me block it so it doesn't even need to be just about deprecations right you know your organization rather than it just being some line in a word doc somewhere that says you must follow this role, they can be an actionable thing they can put it into a policy and then as part of a release cycle you know you may get through your release cycle hey everything was good from my perspective now you publish your artifact or whatever to another group, it runs through their policies and then you know you check your gates hey I'm into production because I've checked you know x y and z policies. So you know it kind of like expands into this. Hey, I can, I can really set any kind of roles that I want and I, as long as I can take it you know to kind of inspect the yaml for it and match it up. I'm in good shape. So, you know, it was great because I was, again I just showed up with the duct tape and wrapped all these things together like I didn't need to do a lot of work I just put the pipeline together and pull down all the resources. So it was really cool because then I was able to, I was able to really, you know, really get going because now it wasn't oh you know we're, we're moving from open shift for one to four for something like that because it's always like, no one wants to be on the bleeding edge when when they're doing things like let me just hop scotch a few of these and that's always fun. But between those there are the breakages that we talked about these deprecations so I'm going to rely on the on the stuff from the deprecation repo from Steve, then you know we got excited about it started talking about it in the, you know, in our communities of practice here at Red Hat. I said hey this is a cool thing that I'm seeing over here we should do the open shift version of that because we've also we have our own API version deprecations and hey we're not going to use these things anymore. You know, moved, moved, you know, group, you know API groups and stuff like that around in the early 4.x release cycle. So we started doing a group of folks started you know publishing a bunch and not just deprecation stuff they started doing best practices so again this is all just like using the policy to define your policies, using com test to kind of like check those and really enforcing best practices so things that you want set in your deployment configs and your routes and things like that so it balloons right you can your imagination is kind of like you can set it to do anything. It's really powerful in that way that you can, you know, and enforcing policy as code will always be more effective than any kind of documentation right any kind of run book any kind of anything if you're doing it as code in the environment, it will be done. Exactly. Sometimes whether you like it or not. That was really my big goal is because at the, so at the same time we were looking at all this rego policies stuff. Jordan, what's Jordan Liggett I believe his last name was. Oh yeah, yeah, no Jordan. He was working on, you know, kind of embedding that in the in the cube CTL tool right you know so he you know now when you go to deploy something you know cube CTL apply. You'll get a warning saying hey you're using an update you know this this is this thing's going to be deprecated in version, you know, 122 or something like that. So that was, you know, that was being built into the runtime kind of tooling. And at the, you know, what we wanted to do was kind of make it be able to use for kind of all purposes, not that not just kind of like at runtime or just through the CLI told we wanted you to be able to plug that in. Anywhere you anywhere you wanted and that's what rego kind of allowed us to do. So now you can now nowadays you you have you have multiple ways of doing that whether you know if you're just hacking around you know the it's built into the pool and you can see that. But instead of having to kind of like orchestrate cube CTL commands and your pipeline, you know, you can you can use that the comp test you can use the rego policies and get it done. You know so multiple ways to get the same thing done but you know it gives you some options. Nice. Very nice. So, I'm going to show it off. Yeah, let's let's take a look at it right and that's talking about magic screen share music or some sort. Right. So, what folks if you can't read what's on screen. Let us know. Yes, yes, please let us know this is a best guess right now. So, created a project here we see that we've got nothing running. And the only thing that I've done kind of like you know behind here in the shadows I have tecton up and running so if you look at. We'll see that we've got all of our tecton stuff running. So that's that's all that's kind of like all that's already been done kind of pre cooked. That's that's the, that's the magic right. So, we've got our project project here. The first thing we want to do is, and I will make sure we get these links out here but I've got a repo, it's called deprecono. That'll be easy to find out. So, we've got our, you know, my app that we're going to be taking a look at it's just the usual my usual test HTTP echo from from hashtag or that I use to say, Hey, this thing's running. So we've just it's a simple app, we've got a few things out there, and I've got one that's using the old v1 beta one or back stuff. Just a bunch of nonsense, nonsense the animal for us to test against right now. We'll take a look at those if you stuff here in a little bit. Really kind of the juicy interesting part of this is the tecton stuff. So, in tecton, you have tasks, you know, that allow you to do things you provide the right parameters and it will take the right actions. So we're going to go down or just to get cloned in the comp the comp test to be able to run our comp test pieces as part of our pipeline made the only difference from the comp test that is in this repo with the one that's out there on tecton hub is I just added an additional one because I tend to not want to shove. I want to be able to pull policies from all over the place I don't want to necessarily just after a while that I pulled them into my, my own repo. Sure. There's a way that you can just use the upstream one and do it that way, but that requires some get some module magic and I just get some modules here though. I've lost many, you know, a lot of sleep to get some modules in the past so I avoided them on this one. But that is our kind of our dummy repo and as all good side projects, not a lot in the read me but I'll make sure that I push my actual read me up there. If you want to follow along and play around with us afterwards. So that's the dummy that we're working against. We take, we'll take a look at the pipeline here in a second, which is kind of like I test with the animal so I won't spend too much time there but we'll take a look at that. I mentioned Steve Wade, he's got this deprecation repo out there called deprecation right there. As I mentioned, it's just a bunch of policies. So, for each version of a Kubernetes release, we've got a bunch of kind of some, you'll see deny so there are two types of, well there's probably more than two but the two kind of regular policies you'll see is there's a warm, which is just the warning, and the other ones are deny which is hey I will fail. I'll actually return like an error code for for from Conf test and show you this you know the messages in red versus the messages in yellow types So some of these are in their actual deny policies. If you look at the more recent ones, 122 I think has the ingress one. So, 122 you'll see a bunch of these warn, warn, kind of warning messages. So that's, this is what kind of makes up a a rago policy is you have your action, the message you would like to show if you're, you know, you're looking, you know, this is just a, you know, we're just kind of storing the, the objects that we want in there this could be anything I could call this like ABC equals, you know, x, y, z. This is the, the things that I'm actually looking to take action on, and then it's always input dot something so it's input dot, and then you start creeping down your YAML tree. So you can see here I'm looking for API version. So I see that, you know, API version is, you know, admission registration, I then say, hey, if any of for an API version that matches this, if my kind is any of these kinds up here, show this message. So that's really kind of the, the building blocks of a regular policy so we won't spend too much time on the specific ones because there are a lot of kind of deprecation policies that have been out there. But this repo has all the ones from from 116 the whole way up through 122, at least that will be coming out. And a few other useful ones so think service accounts where they started moving things around things things of that nature. Nice. So this is the the main kind of rago policy repo that I work with and that that will be showing off today. There are a few other ones I mentioned are red hat co p rago policies repo, which is, which is useful. And if we have some extra time left over, I'll show you a cool one that I have cooked up that allows you to not just look at deprecations, but also allows us to work with the open, open shift install config for an API installation. So those are the three kind of rago policies that we will be working with. So if we, again, take a look at our app over here. So we've got tecton, we've got our tasks project just make sure we're in spot. I'll go ahead and apply these two work with the home track way. So we've got our tasks remember tasks are for those not familiar with tecton tasks are the building block of a pipeline and are kind of like the the blueprints if you know so rather than me creating a comp test or get cloned for every single time I want to run it. I can just put a task out there, allow myself to provide some parameters to it and I can just use that over and over again in my pipeline. So we're going to run our pipelines, I've got, you'll see I've got two here one is the pipeline and one is a pipeline run pipeline is the actual content. These are the things that I want to do in the order that I want to do them. You know, go ahead and run it a pipeline run is an instance of a pipeline. And I've got a pipeline run out here because the thing that I wanted to be able to do it which is the dynamic PVC isn't available through the open shift UI yet. So I'll kick things off from from the command line here. But we'll, we'll try to keep, you know, follow it through the UI which is a bit easier on the eyes than, than our pipeline or our email here. So quick look at the actual pipeline though. So at the top I've got some parameters that I would like to use just things like repos that we're going to use what's where I provide the deprecation repo, the application repo that we're working against. So those are, you know, and the versions that we want to work with workspaces are what are how we share the data between each of the, each of the steps in our pipeline. And obviously, you know, I've got something for deprecation where I store all the those radio policies. I've got something for the COP radio policies and our application data. So that's really just, hey, I'm going to run a step and I want to store some information off here so I can use it later later on. And I'd like to break them out into multiple workspaces that way I don't have a bunch of things maybe accidentally clobbering each other. Nice. Yeah, so the you'll see here in tasks, I just have a lot of get something right so this is this is all using that get clone task again you know we're just referencing that and we're passing parameters to it. So here I go off, I get the depth, I get our deprecation policies. I get our COP radio policies and get our app data. So it's all about you know just collecting the resources that we are making sure we have the appropriate versions the latest of a certain tag or branch. And then we actually start doing things. So this is where I, hey, I want to check my check my YAML against a deprecation policy. So you can see that I call out to Conf test I give it the app source and my policy source because I have my app over here I've got the upstream deprecation repositories. And, you know, I pass in a bunch of things. A bunch of things. I pass in the file names. So I pass in the app path to look at in our application application repo. I pass in the policy, which is, it helps build the path to the policy so it's always Kubernetes, Kubernetes version and then dot rago. And how I would like to output it so it just sends it to standard out. That's one thing, because, you know, Steve and I have some differences on this is the arcs. So when you're passing things to Conf test, it will only actually, you know, show a failure exit code if it shows the deny policy. Got it. A lot of the policies that are in that deprecation repo, they just throw warnings, which is, which is good. You know, if you want to be friendly to people, myself, I like to watch the world burn. I want to know if if something goes wrong, I want to know, even if it's a warning, don't just, you know, I need to stop right then. So Conf test has a nice flag that you can pass in so you can pass in any of the Conf test flags. But the one that is important for my use cases fail on one so fail on warning, which, even if a warning thrown it, it bombs out and that causes the pipeline to fail, which is what I needed to do. So it tells, you know, obviously, we need to make sure we get our policies first. So this is just me specifying, hey, run these three first and then run this thing. So it'll run Conf test after those three run and it'll run it with those parameters. Nice. So, with that being said, we can apply our pipeline, which this just makes the pipeline available in our cluster. The thing that actually will kick it off for us is this pipeline run. So here, this is just me supplying the values for those parameters that we had specified. It gives a reference to the pipeline that we're using. And it defines the work spaces that I've specified, because I don't like kind of pre baking pvcs like I don't want to have to create the pvc I just wanted to dynamically spin up every time that I run this. Khan has a concept of a volume claim template. So I can just say, Hey, here's my volume claim template I want to have this access mode. I want it to be this big and give this name. So there, you know, for every instance of a pipeline run. So if I run this pipeline run three times, it'll, it'll spin up with a kind of generated name, and it'll spin up new pvcs for each run of that rather than trying to use the same pvc and potentially overwriting a bunch of my logs, overwriting logs when you're troubleshooting, not that useful I found, which is why I've ended up using volume claim templates instead of the, hey, let me just create three pvcs that I use over and over again. Nice. One thing to mention there is that the nice thing is, is then when you go clean up a pipeline run it like so if I go OC delight OC delete pipeline run, it'll actually clean up those pvcs because it's owned by that pipeline run, instead of me having to man manually go back and clean up all that stuff So, with that being said here, go ahead and apply our pipeline run. You can see that it's created. Cool. We're over here now and are you why I click on our pipelines. Go to the right one. We'll see that we got this. What should be that's this looks like we see this pipeline run is running so it runs fairly quick it's just checking out the repo so those three are running. Oh, wait, no, it's supposed to go wrong. It's broken. That's why. Yeah, so I've got remember I've got our old. It's not supposed to pass the first time I'm showing why we will use this. It failed. Good. Yeah, it says it failed on the Conf test and the log snippet. Hey, here's what happened. So we can see that Conf test is saying hey, you're my apt role dot yaml. It shows that message that we we put in that says hey this is deprecated from Kubernetes 120 use this instead. So, you know, go ahead and pop this. So it shows us the warning message that we specified, and then it tells us how, you know, basically the output of it says hey we had 10 tests, nine of them passed one was a warning and because we passed in the fail on warning flag went ahead and and failed it for us. So, to fix this. Let's go over here to our dummy repo look in my app and kind of just as I mentioned earlier and as our message tells us, hey we're running this old API version deprecated. I want to know to use this instead. It's just going to work magically right get magic. Yeah, so it's just going to work magic. We'll go ahead. Okay, cool. I'm going to fix this. And we'll push, and then we'll go ahead and say, run back up here. Hey, go ahead and rerun this. Again, you know we're saying rerun so it's going to all those checkouts it's going to get the latest version which will pick up our last change. And ideally, if it works as opposed to, we should see everything turn green now. Next up. Come on now. There we go. So we can pop in here we look at our check deprecation policies. Now we see 10 past zero warnings. Oh, good. And now we're nice. So, the great thing about this is again you know we're using this for for API versions, but you could use this to beat up on any YAML that you want. We're using the policies that are specifically tuned for Kubernetes deprecations, but this could be anything you want. And you can make, you know, obviously, Tecton has a bunch of nice built in things to that you can expand these pipelines to, hey, I'm going to run my pipeline from like I mentioned earlier, I can run my pipelines for for my local policies. I want to, you know, then go to some other one I could just have my pipeline that may kick off another pipeline and you know the security teams kind of realm and say hey look my things ready. Here you go. We're running this all kind of manually right now but the way that you would do that is with a piece of Tecton called triggers. You can trigger it based off of, you know, check-ins, you know, things to GitHub. Or, you know, you could just kind of like script out a bunch of trigger templates that says hey, I'm going to go poke and prod this other pipeline that's in a cluster and do it that way as well. So, a bunch of cool things that allow you to really tie all this together within Tecton and with Rego itself. If you're, if you're not familiar with Rego, where would you recommend folks go to learn? So, this is exactly, I had no idea what Rego was. Right. I got pointed there by Duffy and I was like okay cool and a new thing yet another new thing to learn. Right. And the great thing about Rego is on this Open Policy Agent website, they have probably some of my favorite, like, favorite documentation, right? Nice. But this was really good. Like, it shows you what it is, what the purpose of it, and kind of the basics. And I believe the, I don't know if it's on this site or if they kind of like the originators of Open Policy Agent and Rego. I think it's Styra. I think they've got some labs on it as well. I would definitely start with this policy language documentation. If you want to take a look, get a feel for what it looks and feels like. And then there's definitely, there's definitely some labs available from different groups online that you can really, it gives you stuff to play with instead of you kind of having to invent your own. But yeah, this is definitely where I would start is on the docs is where I would go. Absolutely. I've dropped that link in chat so folks can learn that as they need to. Yeah, I know Styra has some labs. I know. CNCF has some labs, maybe, I think. Yep, yeah, that's that's always my good fallback is like, I know, I know if it's like a CNCF project they've always they've got something somewhere I haven't used their stuff but I know they've got it. Right. Yeah. Yeah, so this is definitely where I'd start there you know similar for for tecton if you don't have a real good feel for tecton one thing you don't have to use tecton you can use whatever your CI preference of choices you know it works the same in Jenkins tecton I started off using GitHub actions like this will work anywhere that you can run contest so I really like tecton because one it's kind of built in and it's I don't have to run a lot of extra things that I don't need hey I needed just this pipeline. And these two tasks and I'm running. I didn't need to go kind of work straight and build out my kind of Jenkins image I didn't need to fool around in and GitHub actions and stuff like that it was just all on my cluster all controlled by me. It took a handful of minutes to kind of get the initial workings together to start testing this stuff out. Nice. One other thing I wanted to show us if we had some extra time here is if we go to our tecton pipeline. As I mentioned this doesn't just have to be a doesn't just have to be a deprecation thing this can be any any YAML of our choosing. Another thing yet another thing that I've wasted some time on is I'll go to deploy a an open shift cluster right you know installer provided infrastructure, I can just run open shift install, give it my install config and poof up in AWS comes a cluster. And one thing that I tend to run, tend to have some issues with is sometimes I'll provide I'll just say, Hey, I'm sure this region is is something that I can use. And then I'll find out no open shift wasn't supported in that version and that was that was more like the initial for dot x releases. Because they wouldn't initially support everything in every region. And that's fine. It was new we're figuring it out. But that also meant that the installer ran for good, you know, for a portion of time before I realized that there was a problem for me to a sharp edge for me to run into. So I, again, went back to my favorite YAML of using tool of Comptest and Rego and said, Hey, I'm going to, I'm going to not do that anymore. So go ahead and update pipeline here. And we'll go take a look at what I'm talking about here. So similar to how we have our deprecation repo with those policies. I, you know, up through four dot five, I believe I at least had, and so I've got to get four six and four seven six anything's changed there. And some Rego policies for are for the available regions in a in a in AWS. So if we take a look at that, we can see something similar to to what we saw with our Rego policies and again I like to watch everything burn I want the pain points because I want to know something's gone wrong. So I have a deny message. And I say, Hey, these are all the supported regions for for AWS as of OpenShift four dot one. I then said, Hey, if my if the region in my input file doesn't match anything that I provided up here. I think I was writing this around Halloween I said, I would say that region's haunted and thrown some ghost emojis got to make. Other messages make them fun I always say. So through those out there so you know as we started to add more, you know, they're kind of built off of each other so similar here is where we started I think you north one, the one in Stockholm I believe was the one I was bumped into because we were watching a bunch of stuff and Stockholm at the time. So you can see that hey as of that we see you north somewhere in there. You know, similar thing, I'm just, you know, I am running something against YAML and here are my policies that I'm pulling it. So over in my pipeline. I've similar to how we did our other similar to how we checked out our policy code and then we ran CompTest doing the same thing here with the same exact CompTest task that was already done. No modification needed. I just I'm providing different parameters to it right so instead of the Kubernetes version I'm just saying hey look for OCP AWS and then an OCP version. I can fail on warm but I have nine messages so we don't really need that, but you know just kind of kept kept it safe in case we need it to update that later. And then I just looked for my files under OCP from in my in my app app repo, which again our app repo is just the test one here so if we look at this in Stockholm fig this is just the dummy example that they have out on the website and you'll see that hey I've got EU north one, which wasn't a thing at the time. So, if we want to I want to just go back here and we'll rerun this again. I believe it should pick that up. Yeah, there you go. So we can see that these two are kind of running on their own because it doesn't need some of these other ones. And off these two go and that that's mean that's the I will admit I'm someone who likes to live in live in the terminal. So that failed as expected I'm someone who likes to live in the terminal. So yesterday as I was kind of dumb demoing this out and making sure that everyone's everything still worked before we went live today. So one of the first one of the first times I've used the console that the UI version of what this is usually I'm just over here, you know, using the CLI looking at what's going on in the pod logs. But I decided hey, you know it's talking to people about this it's easier to visually explain what's going on if you can see what's going on. And I will say this is a beautiful, beautiful UI for this so thank you to the folks on the console team through this together the pipelines team. One thing to call out here too because this was a conversation where I was having yesterday I mentioned how I like to volume claim templates. In, in, in the UI, you connect like there's a pipeline editor you can use a visual kind of UI to put together your pipeline. One thing that wasn't available though is you being able to use bought specify volume claim templates in the UI. I talking with the team, that's going to be available in four dot eight run on four dot seven right now so in four dot eight that'll be available so you'll be able to completely do everything I do here and YAML in the UI if that's that's your bag if that's where you prefer to live. Coming soon, I look, look forward to that and four dot eight it sounds like. But back to the output here so we click here we see that hey we fixed we fixed our deprecation stuff so that's still passing, but we now we see our OCP region data is failing. Just as I mentioned, I was using EU North one in four dot one that wasn't a supported region, and I get a new North ones haunted move, move along nothing to see ghosts. So, again, you know that the possibilities here with com test and rego and pull this all together with tecton really kind of endless there's not a lot of code that you need to maintain because you can have the the appropriate people who are specifying your policies, just publish them where you need to go, and you can pull them in that in that manner, you can publish your own put them in your own repo. Put you know tuck it along right with your app to make sure that things remain the same. Test it that way, and it's all done with a pipeline and two tasks, it's just the get the get checkout one or the get clone tasks and the com test task, and that's all the duct tape require to really pull this thing together. That's cool. Which is, you know, from from my perspective like hey, that I'm maintaining almost nothing at this point. I'm just wrapping it in a pipeline. Yeah, yeah. Yeah, I mean, and the fact that you can use kind of any tooling right like just tecton, not just get have actions and everything else right like the, it can be an endless list of possibilities which you can test against and be like, This will fail in the next version. Yeah. It's a separate problem right it's like there's so many options I can do anything now. Some some people tend to like you can overdo it too. Yeah, I can put everything in a rego policy and then it's like, now I can't deploy anything because there's too many policy. Yeah, which policy broke my thing. Yeah, and that's why I like to break it up with different policies and different repos, because then it's not like, Oh, all my policies are in here. One of these hundred broke it. I can say, Okay, it failed from policies in this repo. How many, how many policies are in there. Okay, just this handful of 1020. Okay, which one did the back. What's the message. Let me look for the message. Nice. Also like there's a strategy to making your policies easy enough to troubleshoot as well. Don't try to tie everything. People like to template things. Absolutely. Yeah, so it's like, Okay, I'm just going to have this one policy and I just feed a bunch of stuff into that and it's like, Okay, well which thing that I did I feed that broke it. So we're word of the somewhat wise, don't do that because we I see that a lot and that causes that causes issues. Yeah, like you mentioned, that's one thing that's, it's really easy to go out and talk to folks that we're working with on these things right because Okay, you any toll that you want we can, we can pull these upstream tolls and and use it with your special brand of CI system right you know anything you want we is going to be able to implement it, because we just need a contest CLI, some some bubblegum tape it all together. Yeah, that's pretty awesome. And then the, I mean, the, the way you've implemented, you know, with fail on war and I mean that's going to definitely give some people right like some good feedback before they go to prod, right like that is, I think, wise. Yeah, exactly. And one thing that I've seen before is, when you're just doing your development kind of pipelines hey I'm checking my code triggers will kick things off and put it you know put it into into my repo. Okay, cool. If it was just a warning yet let let the developers keep on moving along let let them get that in there, but then in the logs like you can use Prometheus and your, your log stack so elastic search or anything like that. And then you start pulling in like, Hey, I'm getting a bunch of these things. Hey, you know dev team. I'm just letting you know I'm seeing a lot of this. You're probably skipping over because hey your app is still getting into your dev environment but just so you know if you want to go to prod like, it's got to go fix it. You know, don't don't put up blockers too soon, but definitely before you start getting into like your QA environments your prod environments like something's not in good shape. Don't wait until the last minute don't, don't be don't go back to the old days where it's like you found out when being deployed that it's not going to work. Yeah, don't. Yeah, test first and test often. Yeah, test first test often. And that's what it's what I really like about about tecton right is it's kind of like it's made up of a bunch of pieces like you can bring in the triggers part of it so it's not just I need to go to the command line and kick off a pipeline that I can set up the triggers and anytime that gets checked in like off to the races and I can get informed on those types of things. Then you can get start getting really fancy and go into Prometheus to start setting up alert managers like hey if you see this. You know start sending out messages and alerts and let people know and then like I can go in and look at all the pretty graphs and say like, hey I've got a bunch of deprecated things over here in this namespace I should go probably go talk to this team or where they should you know, ideally they're watching it and they can they can take a look at it right. Yeah, that's I mean. And you know there's the whole process of like which policies we should use right like get your security team involved and like ask them. Like, hey we're thinking about doing this, you know, here's the list of things that it can cover does it cover all the things that we want to test against, if not, what do we need to write ourselves. Yeah. And that's that's what I mean, there's a lot of cool things about this but that's probably the one of the best things is, is it really gets a security team involved early rather than them being reactionary they're kind of involved in process. And it also gets everyone excited about kind of security and nature, because now it's not just a security team right up your policies will use now it's like, hey we're going to write some to like safeguard us, but like give us yours to and we'll use yours and it's just like everyone can help build everybody Everybody jumps that pipeline. Yeah, that like that's the, you know, DevOps get upstream is that everybody's involved in the application delivery. Instead of just being a bunch of silos you get this kind of cross functional collaboration which is really exciting to see. Yeah, that's really cool. I mean this is always like getting security involved early is always going to be a good thing. Right. Like, I know it might seem painful at first for some folks but When you show them that like hey we're actually trying to institute policy as code here right like let's, let's work on this together as a team. There's always like the old wounds that pop up when you first like hey, hey get them involved like invite them into your development process let them know what you're doing it's like, but then they'll see my secrets. They'll know what to look for it's like, that's the good thing. Yeah, what you're doing. This is not bad I get that you think that they're working against you but you're all working for the same people you're working for the same you're all pushing towards the same goal. Like, you know, the one team one dream type thing right like hey we're all trying to accomplish the same thing. We all have some like some side some side quests that we're all on we want to make sure x y and z because that's our goal as a team. But at the end of the day, everyone you're all on the same team act like Yeah, it's, it's one of those things that we have to do and DevOps and any kind of, you know organization is just realize that we're all one big team, right. Like, yes, you might be over here and ops and yes you might be over here and death and yes you might be over here and security but you all have to work together. Yep, exactly and that's like a lot of times won't you know we're when we kick things off is in some of our labs engagements it's like, okay let's bring everyone together. And oftentimes that's the first time everyone's together and then. Yeah, yeah, exactly it's like room like usually you're just a name on the other side of an email right bad news right. So like you get everyone in one room you kind of talk through like, hey, here's the goal. We realize, you know, maybe there's, we've all got, you know, we've all got some scars from from past interactions but we want to do this thing. We are one team, let's work towards this goal as a team let's all get involved. If you've got ideas and you want to kind of collaborate and experiment a little bit. This is this is the place to do it. Awesome. Yeah, I mean, kick the tires like the fires right. Exactly. Yeah. Cool Tyler this is awesome. Thank you so much for bringing this to us. Thank you for giving me a chance like like I said like this is something maybe over a year ago we're maybe even longer than that we're playing around with so it's good to finally be able to get in front of people and show us off and hopefully this sparks some ideas and we can see what else comes out of it. Yeah, that'd be great. As always if you have feedback let me know short at redhead.com. And if you want more info on Rigo and everything else we've dropped the links and chat but I will find them again real quick and redrop them in case you just joined. And anything else you want to recommend before we sign off your Tyler. That's all I got for for today. As you mentioned check check those links out if you have any questions feel free to reach out to myself. Just my name on Twitter is how you can find me. Easy and I love talking about this stuff so feel free to reach out. Nice. Awesome fantastic thank you all for joining thank you Tyler for being here and we will see you all soon. Stay safe out there.