 Good morning. Good afternoon. Good evening. Welcome to another edition of get ops guide to the galaxy I am Chris short you're a host with the most for this expedition that we are going on in the next hour or so Christian Hernandez is here with me Christian the get ups Guide to the stars. I will say yes to the stars. Yeah, I'll be your captain today Yes, our get ops journey of get off guide to the galaxy I don't know about you Chris. It's been a crazy week for I mean it's been like Non-stop right like it's like you wake up it starts and you go to sleep. That's when it ends and it's like Yeah, we not do that all day long. Can I get like a like a show in there on TV? Well, then it's like your day-to-day stuff, right? Just like wait now that now that the meetings are done now I get to go to my yes No, it's funny last week was like a lighter weight week for live streaming and you I used to do this And they have like a karma, you know, they track how many things you complete per week And I actually sent a screenshot of it last week because it was like you completed 34 Oh, look at that. I'm on the air so much not as many meetings. So it was like, oh, that's amazing this week It's like 10 Yeah, exactly. Yeah, so Well, you get it where where you can right? I think With with kube con and and for us get ups con there's a lot to do and it's coming up quick So, right again, like for everyone watching this we got October 12th, right is the day zero for Get ups con and then You can the cloud native con is the rest of that week. So for those of you who are Flying out to Los Angeles say hi to me. I'll be there for those of you who are virtual You know, you know, right say hi to me. I may not say hi back because Well, I'll say generic hi right to the camera because we'll be it'll be a hybrid event as most of you probably already know so So yeah, so today's as I was preparing for today's topic today's topic directory structures, right? It's been It's kind of an interesting a topic because It was about there. Yeah. Yeah, there's it's it's a hotly debate a topic and it's also You know, you're you're you're either on one or the other side, right? You're either a very opinionated or you're like, hey, do whatever you want, right? And it's it's there's a lot of information Not a ways of people doing things and it could kind of get overwhelming, right? Like if if you're starting off in your get-ups journey Oh, you're you know, if you're just starting and then right away you're like Well, what's the best way to you know, like if I'm putting everything in get make how do I configure everything, right? so I Went over briefly long long time ago. So for those of you who haven't Been with us the whole time You know, you can catch past episodes. It's literally like the first episode. I kind of touch on directory structures a little bit But I've only like I really like graze the I guess the top I like the top of the tippy not even the tip of the iceberg But like the part that you can see like the little snowflake on top of it That's how much we've gone into it, but I we did talk about a little bit And so I figured I'll do a show a stream about Best practices, right? So Not necessarily like hey, this is what you need to do but more like hey Suggestion here are some suggestions and some of the things to look out for because Again, just like everything else. It's an answer. You absolutely hate it really depends on what you're doing Everyone always hates that answer. It's like well, it depends like well, it depends on what well, I mean, what are you doing? I don't know. Tell me Okay, I just wanted to play my code so I Kind of set up a kind of best practices and hopefully it'll it'll trigger some some conversations here, so I Know I try to do a lot of keyboard stuff Today is mostly presentation and talk, right? You know, I mean this it's a directory structure conversation. Yeah, we're gonna talk about multiple ways to skin this cat Yeah, yeah, so that's let me share my screen here and it's always the fun dance of You know what there's actually found a setting in zoom that maximizes the screen when you hit the share button Oh, oh, so like you can have like a pre a predestined that'd be cool Like it was like always share this screen. Yeah, it would be nice I don't know if that features in there, but I need to tinker with it Yeah, yeah, so for those if you if anyone out there knows let us know So I have this presentation here. I gave the presentation to you Chris He's gonna load it going to upload it right now. He's gonna share it out, you know, whenever, right? So as we're talking here, so it was processing and now all I have to do is actually Share Yeah, so I'll be jumping in and out of the this presentation That's why I'm not in full-screen mode. So let me know if you guys can't see anything I'll try to zoom in but and you can follow along Yeah, just drop the link in chat. Oh, there we go the links right there. So So, right, so so you've decided to get ops, right? So like this I figured So you've decided to get ops here's a pamphlet to help you out, right? So So for those for those of you who have been following the get ops working group I'm gonna put the link in the chat we have the get ops principles You know the beta release came out we have a release candidate also out, right? Just follow that link open get ops dev all the information is there And the reason I'm dropping me that in is because the one of the get ops principles is you have a single source of truth, right? right, so So, you know Much like everyone else when I started with get ops I basically is like all right cool So like I'm gonna dump all my ammo in a directory and like and then I'll sync that with my repository Yeah, and then and then you know that works great right profit, right? And it actually does work and it's actually is pretty simple But much like everything else is as easy to get started difficult to master sort of thing And now, you know, you probably ran into things like I did where you're running into issues is like, okay Well, that's probably not a good idea, right? So like how do I deploy this? You know, I have this Yeah, how do I not break this right like how do I I have this dump of yaml on a directory and how do I apply it to multiple clusters? And how do I apply this to multiple environments? I don't apply this to multiple clusters in those environments, right? Like you start getting into this like, okay Well, then verse of things. Yeah metaverse of things is like, okay. Well having a directory with all your yaml It's probably not a good idea and you're starting running to this thing is like, okay. Well, how do I? You know, how do I organize my directory structure, right? It's like, all right, you did a POC, right? Hey, this works pretty well It's like the first question is like, okay Well, then how do I that's literally the first like first question is what is get-offs and second? All right, what get-offs controllers are there and third is like, okay, this works. Well, how do I? How do I configure my direct structures like one of the like early questions, right? And a lot of the tools built around get-offs, you know, they've already They already face these similar questions. Yeah, and and they're like, okay Like anyone starting with get-offs automatically lasts like the first thing, right? And so people building tools around it Have kind of have an opinionated approach, right? So I'm gonna go over some of the tools is kind of like, hey You know if you don't really have a strong opinion and you want to just say hey Just give me like a sane default. There's some tools out there that do it. I like how you're like tool tool tool Gerald Yeah Gerald if you're watching that. Yeah, you'll you'll see this here pretty soon here. So there's um, I'll drop this I'll drop this blog here So I have there's all Argo CD Argo autopilot right and Argo autopilot is in the Argo CD project Right. I'm sorry in the Argo project, right? The Argo project has Argo CD Argo workflows and this one here are Argo autopilot and And it's basically kind of like a that as the graphics suggests as a hands-free Argo So if I go back to the presentation here Here's a sample Repository layout they have here and this is basically This is a test someone did with with Argo autopilot and it and it probably kind of breaks it down into Things like my application how do I bootstrap right and I see that William, you know ask how do I You know deploy existing resources and things like that, right? And you know, that's kind of like the how do I deploy Argo, you know, get up friendly way That's kind of like their idea of bootstrapping and they have an idea of projects Not to confuse you even further, right? So we have open ship projects and there's Argo CD projects And then there's Argo autopilot projects. It was something completely different Projects can be seen as environments, right? So like they're that's how they use it. So anyway Argo autopilot environments then yeah, it's one of those things where like they thought about it later. They're like, oh, yeah But now, you know tech debt is real. Um, yeah, no kidding So Argo autopilot creates those directory structures for you So you don't have to like you just it creates it for you and you just need to know where to put your YAML That's that's one thing Red Hat has our own we have Something we spoke about before and I swear one day I'll get bows to come on and talk about cam Cam's another one been on the talk about that yet. We briefly talked about it So cam has a concept Yes, right of Bootstrapping right so it's essentially the same idea Except it's at the day two thing instead of a day, you know one sort of or day even day zero thing and Can name the main space that won't confuse anyone. Yeah. Yeah, exactly And basically See with with cam they do have the the concept of environments and configurations and stuff like that So this is kind of what you get the directory structure And then flux CD flux doesn't actually have a directory structure, but they actually have a pretty good article Flux has like guidelines like what what to do and I actually read this is actually really good I actually do a lot of this and I think as we talk about later Most people start around this kind of path So that's an article you can read and then there's general Gerald moon, right? because you don't know Gerald is a Is an architect here at red hat and you know, he's he started with the get-offs thing right around the same time I did and he actually has he actually went and documented the crap out of this so he has Directory structure and he has You know, you know, what level it is, you know the name of the folder and like what what it's for and he actually goes Down into the weeds here. So there's So good news is that people face this problem before and You know, there's recommendations out there, right? So if you're you know, if you're looking and you're like, well, like what have other people started doing I recommended looking at something like what Gerald or The community of flux has done that they actually just kind of say like, you know, here's an explanation You're kind of like, hey, I don't really care Just give me a directory structure that I can work with something like Argo autopilot or Or cam could be something that That you can use so But if you're like me you're strongly opinionated, right and you want to do your own thing You know, the the the answers really there's no really no magic bullet, right? Because there's no really Really, right? No one's really right in this regard, right? Everyone really does what works for you. It's yeah Works best everyone does what makes sense for them, right? And like I said before the answers that everyone hates is it really depends, right, but That's true. Yeah, it really depends, but there are some best practices, right? So we're you know, I'm gonna kind of go over best practices and unnecessarily directory structures There is like I put in the chat and like we have in the presentation There's loads of information out there if you really really want someone to tell you what to do There's there's plenty of opinions up. There's there's lots of opinions out there 100 opinions so grab a pencil, right? I imagine this is you right looking at it at the computer screen grab a pencil and And And take some notes right because I'm gonna go over my best practices and some of the best practices that I've read out there So the first thing is dry, right? Just like with coding infrastructure as code get ups. It's all the same thing. We're all talking about code And we're all talking about don't repeat yourself, right and you know Basically is avoid duplicating your YAML so you know Like how do I essentially the question is like well, how do I deploy all this YAML across multiple clusters Without copying and pasting YAML everywhere, right? So it's it's like a The trap right? So it's like so again with my example right I dumped everything in one in one YAML Right, it's like well if I want to deploy this deployment to another cluster and I just want to change The the scale right instead of replicas one replicas equals two Do I just copy that directory and just change one line, right? It seems like a big waste, right? It seems like I'm just repeating the same YAML over and over and over again, right? So So in comes customized to the rescue Customized yeah, he's customized right so Customized been around for a while and it's even built into the CLI like, you know Ctl or OC since like 1.14. It's been yeah, it's been there for a little bit. It's been there for a while, right? And what customize is is a patching framework Right, so you it's it's essentially it's like okay. Well, I want to take this YAML and I want to patch it, right? So So you can do environment specific changes by using the same Same YAML right so it's not a templating per se, right? Although you can use it like templates It essentially it produces raw YAML. So the the idea is is that it just gives you raw YAML And you can directly apply it so and The the basic concepts of customize is that you have a series of bases and overlays and The base is essentially what's common Amongst all clusters right so the so if if you if you think about it, right? You're you know, what's common across deployment the differences between Deployments from one environment to another can be it's actually very minimal, right? It could be like the secrets you use the image you use and maybe the scale But like the rest of it like in terms of like a skeleton is essentially the same right so you don't You know, you don't want to be copying all that just to change a few things, right? So this is the concept of an overlay essentially. I want to overlay my changes on Top of the base configurations. So And the thing is it's the base has no knowledge of the overlay. It's just raw YAML. So yeah, it's just YAML So the thing about customize is that you just You just use raw YAML for everything so And then the overlay is essentially just a series of patches against that So how that looks like is Here's an example here. So you have, you know, your application and you have the base So you have things like service route deployment common things. Yeah, basic Kubernetes objects Yeah, basic Kubernetes objects and then there's a special file here called customization Which basically says hey when Customized run read these resources, right? And so basically it's kind of saying read these resources anytime someone calls customized And just displays blank. Yeah, like yep, and then in overlays Essentially, you're like, okay, this is the diff. This is the difference, right? So in this customize you're saying, all right, so as my base Use, you know, go up two directories and use the space And in the dev environment, I'm gonna change my route and then I'm probably in a different name space And this is what the patches are here. So like you basically patch the resource and the patches could be like a JSON patch JSON patch JSON was it six something nine? Oh What's that? What's that? Oh Yeah JSON 6902 Yeah, yeah, 6902 so 6902 just If you don't know what what that is is I'll find it's just yeah common Kubernetes patching right like if you're you're doing cube CTL patching you'll you'll fit right in that. It's not it's nothing crazy, right? And then So why do we use customize right so I use customize for Various reasons right first Again, it goes to the dry principle right don't Don't duplicate it removes a need for duplication of let's just YAML constantly I don't have to copy YAML from one folder one one repo to another one read to another, right? It's just I have a one set of repos And It's you can create a hierarchy, right? And so that gives you kind of some flexibility it could Introduce a bit complexity with that flexibility It's kind of like one of those things is like it's really really flexible, but like you can really It's like a like a ball of twine right like you're trying to follow things But it's kind of it's one of those things for like to an outsider like if you don't know customizers You're like, how do you know what that is? But someone who's been using customize a lot. You're like, hey, yeah Oh, I know that goes to here that goes to there. So it's one of those things like you really have to do it a few times It's easy, but it's it's also easy to mess yourself up a little bit What I really like is overlays can reference remote repositories, right? So here You know going back to the using customize I'm like referencing like the base I have to go up two directories I have to copy that directory there But you can actually reference remote repositories. So here's an example here I have This is Red Hat community of practice another Gerald noon special where he had the what we call the get ops catalog But basically here what you can do is you can say My bases. Oh, hey, my base actually exists on that get repo over there So like you don't really have to copy the YAML, right? You just have to have one file and basically load in what someone else has already saved And even that so like even if you use this what's funny here is if I go to base And I go to customization that even referenced yet another yam another get repo so you can actually start chaining big Start chaining things together. So you don't you don't have to copy YAML everywhere So you don't even have to do You don't even have to do this right like copy the bases you can reference a base from another get repo So that's pretty right. So yeah, that's pretty handy, right? Yeah, security team wants to do some things to the route or maybe you have some kind of Certificate thing that's different in prod compared to Devon stays Yeah, yeah, I could be in stage. Yeah, or like me as a if I want to consume like in a catalog form, right? Like I want to consume, you know, something that my administrators providing for me I don't want to mess around with how to install it, but like that's that's their problem or like that's for them I just need to rename it and yeah, yeah exactly I just need in a different space and so you don't have to copy the entire thing, right? You can just reference other things. So what's pretty cool is that it validates YAML, right? So, you know, it's hard and it does a great job of it. Yeah, it does a great good job of it and Sometimes it's annoying. It's like just apply it. No, there's an error. It's like, but I know what I'm doing Well, I'm like fine fine, I'll let it validate, right? Yeah, and what's really cool It's agnostic, right? Just customizes just right since it's raw YAML It works with Argo CD with flux with ACM, right? With whatever whatever tool you want, right? And so, you know, you keeping your repos agnostic Gives you ultimate flexibility. So that that's what I really like about customize You can use it with pretty much any tool that interfaces with Kubernetes so I can't talk about templating and patching without talking about helm, right? Helm is another way of doing things If you're used to using helm repos or helm charts, you can bring those over As well, right, to kind of solve this problem of duplicating YAML You know, for this kind of a High-level view, right? If you don't know how to helm works, but kind of like review, kind of you have the idea of a chart, right? Which is a template and your values and then it generates your YAML based on those, right? So essentially is a Templating engine essentially where it customized patching framework. There's a templating framework And so, you know, kind of have like the ideas like you have, you know, your template and then you pass parameters to it and then it just produces a YAML file, right? So Who is responsible for other YAML decorations like DNS policy? Cluster first, Inc, and Kate, like Those system layer things are going to be handled by the actual admins, right? Like they'll Yeah, but the actual admins Who's ever in charge of your clustered Clustered or the infrastructure, yeah It's been a long day Words are hard, right? So you have, you know, again, like I said, you know, you just Private parameters and it'll can do that as well. Some things to look out for, right? When using either helm or customized, right? So I have a blog. There's two ways to use helm So I'll put that blog in the chat. Yeah I wrote a blog about helm and Argo CD. So there's two ways to use Argo CD and Both of them are right and but just one of them isn't get ops friendly. So if you If you're if you're doing if you're doing get ops And you want to do things to get ops away. There's a specific way to use that in Argo CD So it's just kind of like things to look out for, right? Because remember get ops is declarative. So doing things imperatively then you Then you have a different source of truth, right? So then you Then you break the principle of get ops. So you have a single source of truth So then you have two instead of one. So So remember if you're going down the path of helm, which is fine. I use helm a lot. I love helm I have a shirt. I wear it sometimes. I love But just remember there's a way to use it that keeps it a get ops friendly. So and to keep it as a single source of truth, right? With customize There's gonna be a lot of customization.yaml files. So I worked with a with a customer of ours And they're like man, there's you know the customization.yaml files like yeah, it saves you from writing have to Copying yaml's back and forth, but you have but the customization.yaml files are like rat droppings. They're just like everywhere They're just like they're like you'll have a folder Why did you have to call it rat droppings? I'm just quoting the customer They're like they're like rat droppings, right? So like you'll have a situation where you have a folder that just has one customization.yaml file and it references like eight things and You know it like I said before you know, you can mess yourself up a little bit. So Just look out for there could be a lot of customization.yaml files. Also, there's there's not a Either or right you don't have to use customize or helm you can use both from in conjunction right and Most tools are flexible with that. I think all tools That I've used like like Argos city ACM flux They let you use both. So it's it's so it's it's not like an either or like you can mix and match What makes sense to you? For instance, I'm deploying an application that I use a database back end. I Just use the bitnami Mariah DB Helm chart to deploy my date DB because it's just easy. I don't have to worry about that Helm take care of that So yeah, you don't need to write your artisanal yaml for things that are yes, yeah And like with the knowledge of like an actual DBA Potentially you're in limited knowledge set right like you have like a known good thing. Yeah. Yeah, exactly. Yeah, I have a known Good values.yaml file. I just use the helm chart so that that And last point right the very very at the bottom Using customize will dictate your get ops repo structure. So if you remember Right here, right? So like if this is customized If you're using customize to not repeat yourself, that's gonna dictate a lot of what you write into your Into your get ops repo so So that's just kind of like one one thing to kind of look out for right and maybe if we have time I'll show you kind of like this little project I'm working on to kind of show you the directory structure that I've been coming up with heavily customize Influence as you'll see but But yeah, so that's kind of like the point there if you're using customize If you start the path down customize you like you're like halfway there, I think right so you're in your directory structure In your directory structure. Let's say okay. Well, at least I know how to you know Not duplicate yaml and I create environments for myself and you know all that's really I just saw that comment that that's that's already got artisanal yaml goes well with avocado toast as it does By the way in LA I'm in LA right we're like the home you're like in the avocado Yeah Yeah, for those of you who are coming to cube con I will show you some nice avocado Multiple places it's like Monday you can go here Tuesday you can go here If you're on the east side of the city, that's the place. Yes, exactly Now if you're on the north so So cool so, you know, that's kind of like what one of the best practices here Okay, so now you kind of figured out the repo structure but so like now what right so like what's what's Where I go from there right now that I'm using customize, you know, I kind of have an idea of how to you know Customize makes me use a particular structure So I'm already kind of halfway there. Where do I go now right so um It's not all unicorns and rainbows right once you start using customize as Some of you might known here Gerald if you if you're watching this I'm sure as some Gerald and I have discovered about Structures right so So there's different ways alright, so now that I have A structure like what do I do with my repos right do I have like a model what we call mono repo do I have a single repository With everything in there right like right like my whole just dump everything in a single repository Do I have separate repos for environments to have a repos specific for clusters like Like now the question becomes do I do model repo do I do poly repo do I do a hybrid of the two So just kind of some of the things to keep in mind Originally Argo CD I Mean I guess even still today they realized this has scale issues with mono repo. So if you're gonna use Argo CD Argo CD deployments buddy. Yeah, let me tell you It has scale issues right so meaning that if you're gonna use mono repo you just kind of have to You know give your Argo CD deployment a little bit more memory maybe scale it out a little bit It has just issues with with mono repo. Yeah, I mean I've I've seen where people are like Argo CD controller and then it's just like scroll scroll scroll So Like I feel like that's an anti-pattern right. Yeah to an extent. So well, what's funny is William who's actually on the chat right now one once Once showed me he goes here take a look at this video and he showed me the repo and like the first five minutes was him scrolling And I'm like and I went back to chat. I'm like hey the first five minutes It's just you scrolling the egos actually the whole video just me scrolling. This is how long this is how big the repo is So yeah, so to do so this is a good question from William here to decide between mono repo versus a poly repo Should it start with the operations team with the operation team looks like is it distributed? Is it one team? These are our questions So like these are some of the questions that and that's kind of like one of one of the bullet points is here The way your organization is laid out is gonna come into play and we'll kind of dictate a lot of this for you, right? So you kind of have to ask these questions, right? You have to say, okay, you know, what is my operations team look like? Where are those demarcations? Right? What is operations team? SRE teams development team? What what are those teams look like and where are those responsibilities? Marked in so Gotta ask those questions to kind of come to a conclusion of whether you're doing a mono repo or a separate repo poly repo And then So Again multiple repositories you kind of get into the the idea of having a I wouldn't do it this way, but I think this illustrates this point is that you have a repo like per environment for for instance or a repo like if you're using Microservices, right? Maybe you have a repo per service not necessarily one repo with all services in it, right? Maybe you have a repo with all that That's kind of one way to do it. That's kind of the way I do it. I actually see things The way the question that I asked is that is the cluster I'm on is it Multi-tenant right or is it not meaning like okay how many teams like yeah, how many teams right like so like if I'm a if I'm an admin Maybe I'll control the cluster in my repo, but then I'll have a multi-tenant system where it's like, okay, you know These developers have their own instance of Argo CD that then they you know Deploy their application using the specific namespace. Yeah, exactly 100% So this is kind of like what the multiple repositories come into play and how your organization is laid out You can also go full DevOps is what I like to call full DevOps right and have a single repository and And the single repository meaning like alright, like I have a one-to-one relationship between a repository and a cluster And so I have my whole definition of a cluster in a get repo and that just might be DevOps style right just one big repository per cluster And changes and all that go into one get ops work get workflow for your get ops environment so So yeah, so you have this single repository Which is fine right and remember I I'll put this in the In the in the chat if you're using Argo CD just remember there is scaling considerations if you're going to do a mono repo right so If you guys Again, if you haven't seen past episodes the intuit guys came in the guys who actually wrote Argo CD and they've actually said like hey Yeah, like we know this issues here and You know we're working on it so they already know right and so yeah, but just something to keep in mind See here I don't have a few slides left here. Let's talk about Questions to ask yourself right I didn't realize this Big number one. Yeah Well like I was I was trying to think like all right Things to think of question mark or question mark could have been yeah You know Well, I was trying to decide between maybe you face emoji. Thank you face emoji. I'm a big fan of that. Yeah There's probably I don't know how you made that that big and didn't realize it I don't know. Yeah. Well, this is the PDF. Maybe it doesn't show On the Google slide So So I have a A few tips and things to ask yourself right and I think William put in the chat very I was very Some of the questions they need to ask I like yeah, so I like some of those The question mark thinking face right there were Another another good comment. I'll I'll I'll get to the comment in a bit here So some of the questions here is about So some of the tips so utilize customized and utilize to refer to other get refills, right? So if you're not doing a mono repo if you're doing poly repo You can aggregate a lot of those repositories In a single repository so you can do something like hey for this deployment I'm taking these get re get ops repos and you know making one enough deploying that to a cluster you can do something like that Make sure it's easily repeatable. I have a cool workflow It's like hey I have another cluster. I just copy my overlay to another name change one or two things and then off I go so So make sure you're you're it's easily repeatable right And then I didn't say this before It's one of the first things I always say when people starting off in this journey keep your application repo Separate keep your code repo separate from your deployment repo like you have different life over here infrastructure Over there you got to keep those separate keep those separate because they have different life cycles. So You know you don't want a scaling, you know Application code hasn't changed and I want to scale it You don't want like a build of that application to kick off because you changed the replicas from one to two, right? So or vice versa you upgrade or vice versa right version and now we got to rebuild the application or something like that. Yeah. Yeah, so there's there's a They're just different life cycles. It's just not you know, that's just it just makes your life easier So things to ask yourself, right? I think One of the question one of the things William says do you want job security is one of the questions you want job security Get get into get ops. Um That was funny So You know do you have an environment specific configuration Or do you have a cluster specific configuration? You have a mixture of both Mean Gerald we were we're butting heads for a long time I for the longest for the longest time have the opinion that you have Cluster specific consideration Configurations or you have environment, but you never have both because one is a subset of another, right? So, you know if you have Two clusters in a dev environment and one of them slightly different holding you actually have a cluster specific configuration and not an environment specific Configuration But there is so after speaking to a lot of customers and after you know you have a cluster specific configuration Speaking to a lot of customers and after you know going over a lot of use cases Going on this get ops journey, right? We're going to the galaxy, right? There are some some instances where you're gonna mix both where it's And my I guess my last tip or last thing I want to kind of want to go over in the slide dump here Yeah, so so William very good point in in when when Gerald noon Watches this later. He's gonna he's gonna chat me he goes I would argue it's is you have three you have cluster environment and team right and Gerald actually so if you look at Gerald's repo, he actually has a concept of tenants, right? And that's what a team is right so like a team within so you actually do have like three So there is So I actually acquiesce. I actually thought okay. Hey. Yeah Gerald. You're right I jokingly say Gerald ops. I succumbed to Gerald ops where yeah, you do actually do have three I mean, I like Gerald ops. I like Gerald ops. I've been using it for a while So I have Yeah seems to be working okay for me so Another way to approach this and this is something that I kind of did with one of the customers I was working with is that you kind of have a fan out or a cascade approach Right. So for you engineers fan out, right or case cascade same same idea, right? The you have a there is such a thing as especially if you're working with your security teams Which you should be working with security teams a lot especially with literally everything, please. I'm not everything right DevSecOps is Is a thing, right? Yeah, it's the way now. It is the way. Yeah, that's right in the headlines lately. Yeah, sorry Yeah, exactly. Yeah, Naveena asked for tenants. Do you mean namespaces? So yeah, so this is kind of what what The idea I agree with Gerald but the naming I kind of go differently Yeah, and he and he kind of says they if you look at the his repo he kind of mentions it as well When I think of tenants I think of namespaces. So I actually say teams right instead of cluster environment team So kind of like what will you put in the chat? That's kind of what you mean When he says tenant he actually means teams so So yes, so That is probably a poor choice of language. I'm sure it doesn't translate well, you know, yeah, yeah, exactly. Yeah, so Yeah, so yeah a team can have multiple clusters as a function right exactly. Yeah. Yeah, so you can have So yeah, so I think I think team is is a good a good analogy I like the name tenant because like a team can be a tenant on a cluster right but Like you said it's probably a bad it's a bad choice of words because it confuses with namespace. Yeah, so Which I agree with I mean Like like we said at the beginning there's more than one way to get off this cat Yeah, yeah, there's only one way to get off this cat so So you have a so like if your security team or operations team you're gonna have a base configuration right so no matter Especially in an organization no matter what cluster I install it's gonna have to have these basic sets of cluster configuration right and Like I mentioned before in the other you can reference other repos from customized Then you can branch out to say okay, and now I have a dev stage and production Right within that dev I have yet another repo of cluster one cluster two cluster three ad nauseam And then that eventually makes it on to a cluster right and the same for stage and prod right where it fans out In this particular way I So you're talking about you know three four five repos deep So if you have a change that needs to go into this one particular cluster you just change this repo And you know it makes it in there if you are If there's a security patch that needs to happen You can do it in this repo right the the base configuration repo and then it just basically cascades out to all the other ones Same for the environment you you kind of get the idea here so So here the question good question here so Basically is it better to have different repos on an application level or an environment level like pod and like prod and dev so You have different yeah great question right so this is another one of those One of those things where it's Again it depends right so I think the application level would be like in the environment layer I would I would say I agree with that That's where I think most of that Stuff where that stuff would go so and then someone said again the follow up question was what's the meaning of cluster repo so cluster repo is the repository where your cluster specific configurations go so for things like Like security policies would go in a cluster repo or things like You know connecting to an identity provider right your login information would go into that repo so I like the they come I'm holding for dev third ops right so it's We're gonna have dev something ops is that's like the new thing right so yeah dev get ops dev give ups there's Oh my god, don't even get me started Coming to a theater near you yeah coming to it coming to a theater near you So um, so yeah, so that's that's pretty much it of my of my tips It is a good tip friend like yeah, yeah, so there's it's kind of just some of the things you have to ask for questions I again just to kind of Wrap it up and then A little bow is the get familiar with customize get that power under you because that's going to dictate a lot ask yourself questions like we know you know what is my environment look like what is my organization look like who's controlling what that will then beyond customize then we'll tell you what what kind of repos are you going to be using so So one of the to be solved issues is one of the things that coming up from William here one of the to be solved issues we have in telco if I update the base on a on a cascade or layered set up how to prevent immediate remediation to make sure remediation happens on an authorized maintenance windows but that's but that's all the But also that's an old versus new mentality correct right so um you you never commit so the the so there's there's kind of multiple layers here you never commit to something that you don't want live right away so right just you know like if you know think again remember You can stage that thing yeah and wait to merge it until your maintenance window and then just merge it yeah correct and then just then merge it into your or you can use some other tool to kick it off during your maintenance window or you can use the same thing Also in rgo CD there's called sync windows I think it's called yes I can find it yeah sync waves sync windows rgo cr figure it's called it's called sink windows here we go okay maybe you better move it or you can use a CI CD or something like that. Yeah, anything. There's also in Argo CD, there's called sync windows. I think it's called. Let me see if I can find it. Yeah, sync waves. Sync windows, Argo CR, I figured it's called, it's called sync windows, here we go. Okay, yeah. So sync, oops, sync windows here allow you to sync things at a certain time. Nice. So it's like- It's cron job format, we all- Yeah, cron job format. It's like, hey, only sync this at this time or during this windows, right? Like Sundays at 8 p.m. or whatever. So you can do that as well, right? So you have a sync window to set up on Argo CD. So that's another way of doing it. But again, I kind of agree with William here. It's kind of like the old hat versus new hat mentality. It's just like, well, if you kind of have to change, switch the mentality is like, if I don't want it in production right now, then don't merge it because that branch you're tracking is your source of truth. I think of it as a database of the live or what's going on right now. So yeah, so yeah, so William, yeah. So if you weren't aware of the sync windows- There you go. 100%, there you go. You use those, you can say, hey, I don't even want- Balance the old school with the new school, man. Yeah, yeah, now we're doing both, right? So like if you're doing, where's that? Oh, wait, it was in my slide right here. So that's on the application level, right? So you can have it pretty much on any level of where you want it, right? So on the cluster first. So essentially it's like, hey, for this application, sync it Saturday at midnight or whatever. Oh my gosh. Yeah. All right, let's answer the virtual Cates clusters inside name spaces. I'm assuming this is for a test case, not an actual- Yeah, so you can, there is a project that actually lets you do that, but it's a project that Chris doesn't like the name. So maybe I can just like put it- What project? It's, I'll just type- Oh no, don't put that on my fucking channel. Yeah, so the problem with Kubernetes in Kubernetes is that you can't really Google for it. It's like a infinite loop kind of thing. Yeah, yeah. Anyways, we'll find, I'll tweet it out later. I think we retweeted it and we talked about how the names should be changed. We talked extensively about how the names should be changed. But none of our tweets were answered. I wanna file an issue, but I don't wanna file an issue at the same time. It's like, you should know better than this. Don't call something in Kubernetes land. Like it just violates the code of conduct. Yeah, exactly. Yeah. But yeah, so for who asks this question here? Oh, Naveen, yeah. So you can run Kubernetes inside of a sign up Kubernetes namespace using kind. So I'll just let you Google for that. It's a brand new project, but it's possible. So let's see here. Yeah, kind. Kind's the name of Kubernetes in Docker. And then there's... Now you can use Podman 4. You can use Podman 4, yes, correct. I haven't gotten that working yet, but I'm gonna... I haven't gotten that working yet. Yeah, I haven't read the documentation yet, but... I haven't dived that deep into it, but I do have Podman machine here on the Mac. Oh, Podman machine, yeah. Yeah, you can do it. Yeah, you can do OCP over OCP vert. You can also probably do Snow, which is single node OpenShift. As a virtual machine. So you can put that in your CICD pipeline to test some of these things out. Yeah, I highly recommend kind in your pipelines, for sure. Yeah, for sure, for sure. Yeah, so for like nine times out of 10, if it works... So if it works in kind, it'll work in OpenShift. That's easy. If it works in OpenShift, a mayor... No, wait, it's one way or the other, right? I think it needs to be like, it needs to be hardened more to work. Yeah, if it works in OpenShift, it'll work in kind. Yeah. Sometimes if you don't build your containers right, it won't work in OpenShift because that's... If you're running like as root or something, OpenShift will kick that back. So maybe one day I'll do a stream about pipelining with GitOps because I think that's a very... How to test something. Yeah, bring it from that test to QA to prod process. Especially for the clusters, right? Because that's kind of what we're talking about right now. Because we're talking about... Does this cluster even work? Yes or no? Yeah, exactly. If I make this change, how is that going to look like in my cluster? Yeah. You have to fire off, you have to spin up a cluster and test that change. So I'll do some... I think I may be booked up for the rest of this year. So maybe it's something early next year. It might be something we want to get then elder for the actual line developer. That'd be cool. Yeah, that'd be cool. Oh man, having him on, that'd be great. It has so much praise for that guy. He single handedly changed how we test, right? He just had... I mean, he made everything easier. Just Kubernetes contributor workshops, the whole nine yard just got easier by one project. Yeah, it's amazing. This guy's one idea, kind of just for everyone, right? Even here at Red Hat, a lot of us was kind. So cool. Last four minutes. I don't know if you guys have any questions or any other comments. Hopefully you enjoyed this kind of like the overview of kind of the best practices. This has been insightful when I opened into, I think, our audience and myself, right? Like, I always think it's great when I learn something on our shows, right? So we got... Jay, that would be a good show. So I will definitely put on the list. We have a backlog of, you know, so I can put that on the list. And it'll... We'll eventually get that. So that's a lot of cool things. So I think that's it. Do we have another show before KubeCon? Oh, we do. We do on the seventh? Yeah. Yeah, on the seventh. And I think that's when... Oh, okay, cool. So next show. So October 7th. So it's the show right before KubeCon. We'll have Michael Foster on. So Michael Foster is the TMM for ACS. So for those of you who are using... Who are using StackRocks, or now that you guys are using, you know, Red Hat customers using ACS, Advanced Cluster Security. So we'll be talking about the... The Application Delivery Pipeline, right? So it's kind of like that. The idea of building security into your delivery system. So that's kind of... I like talking about some of these things that are kind of like ACM, the cluster management, where like, get-offs is kind of like the core of something bigger. ACS is kind of like the same idea. He'll be on talk about that. That'll be a good show. Yeah. So how do I stop sharing? Gotta find the bar. Gotta find the... Oh, there's a big ol' red button. Yeah, stop sharing. So yeah, so that's it. See if there's anything other in the chat here. No, I just dropped the link to the calendar for folks. I haven't updated it for next episode yet, but now that this one's done, I can. So, you know, it's a timing thing. It's always about timing. It's always about timing. Timing's always... So, yeah, folks, this is actually our last stream for the week, which is awesome. Have a good weekend. Have a great weekend. Well, I get to crunch numbers tomorrow. So, yeah, that'll be fun. Yeah, Stephanie was pinging me. We're trying to crunch some numbers back and forth here. So, yeah. You know, math is involved, and Lord knows I'm not good at it. So it's always fun. Yeah. That's why I have a computer do the math for me. Ta-da. Nice shirts. Thank you. Yes. Yes, cool shirts. So, yeah, stay safe out there, folks. Yep. Stay safe, everyone. We'll see you next week. Yep. Cheers.