 Hello, hello, and welcome to another Stack Rocks Office Hours, the second of the year. I'm super excited to have a special guest, Christian Hernandez on. We're gonna be talking Argo CD, security best practices, supply chain security, all the good stuff, all the hot topics, and we're coming off of some pretty exciting news, actually. So thanks everybody for joining. Before I get into it, Christian, introduce yourself, welcome to the show, and let us know what's going on in the Argo CD side of the world. Yeah, yeah, so thank you, thank you, Mike, Michael. I never know what to call you, actually. I've asked you before, but you said either one, so I'll stick with Mike because it's less syllables. Yes, so thank you for having me on. Yes, my name is Christian Hernandez. I am a technical marketing here at Red Hat. Focus on OpenShift GitOps, which is our supported offering of Argo CD. So yeah, so I've been here at Red Hat for almost eight years now, so it's been a while. So kind of funny background how I got here at Red Hat. I was like some Red Hatters, I was a customer and I was poached. I was actually the, so fun fact for everyone, so only for your show, because I haven't said it on my show yet. I was actually the second OpenShift SA hired in North America, so if you believe that, so for your Red Hatters, I'll give you a moment to process that. There was only two OpenShift SA's in North America at the time, so I was very busy. My background's basically mostly operations, systems management, that's kind of where, special place in my heart, right? So Andrew Sullivan, I talked with him a lot because that's kind of, I see Aida Aida with him a lot of things because he's in the operations. Ask of it, later on in my career, I actually moved into more like SRE work and so that's where kind of the first taste I got of CI CD and application delivery and keeping the lights on sort of thing, so. Yeah, and so going from being the second SA in North America to now, basically helping run GitOps to the point where Argo CD, 115% uptake in production in the last year according to the CNCF survey, so that's one of the main reasons why I want to have you on. You know, it's such a big background that you have and honestly that project is moving extremely quickly so I would love to, for anybody out there who doesn't know Argo, maybe break down the Argo project a little bit into its subgroups and what's going on in the space. Yeah, so when we say, so when people say Argo, they mainly mean Argo CD, but Argo is actually the project name because Argo CD, so the actual Argo CD, the actual CD part is, that's the actual name of what we use. So Argo is a project, right? An umbrella housing project for a tooling that was created add into it. So the Argo project has this history add into it. Into it actually, they were consulting with the company, Appalachics, and then they basically said, these guys are great, we're just gonna buy the company and move them in house. And they ended up doing that. And so, and from there, they birthed a bunch of projects, right? So one project being Argo CD, another one, very, very popular is Argo workflows. And there's other like Argo events and I'm missing one, the one that works with Istio, I completely forget the name. Oh man, they're gonna beat me up if I forget the name. Rollouts, Argo rollouts. Argo rollouts, right? And so Argo is basically an umbrella project created by Intuit, open source by Intuit. It was kind of their strategic plan to enter the open source community to basically take all the tooling they were using in house and open source it. And so, so yeah, rollouts, thank you. I knew I was gonna, someone was gonna help me there. So yeah, so basically Intuit said, hey, we've done a lot of cool things that helped us deploy our applications out. Let's just open source this stuff. And of course, Red Hat being Red Hat jumped on it saying this is great software. So I am, so being technical marketing, I am a member of the marketing committee at the Argo project. So that's kind of where I spend most of my time doing things like ArgoCon and their marketing type of thing. So we are planning another ArgoCon later of the year. So we'll look for that announcement. And again, I'm a big fan of the software. And so there's been a lot of, I do have one commit with Argo CD, very small commit, but I do have a commit. So as an end user, I do try to improve the software as much as I can as time permits. Yeah, that's awesome. And especially it is such an important space. And you mentioned it was donated by Intuit and it's going to be a graduated project soon. I think that's the goal, obviously. Yeah, the goal is. It's the one that looked like. Yeah, so the goal, so it was donated to the CNCF, right? So it is an incubating project. And they're actually looking for graduation. There's actually one more hurdle that they need to clear for graduation. But that, the expectation is graduating by the end of this year to be a fully graduated project in the CNCF. We'd love to see that announcement come. Was it October in Detroit? For a cute moment. Hopefully October in Detroit. Yeah, North America, yeah. Detroit, we can, you know, have fun in Detroit. That would be a great, great announcement. So we're, you know, touch wood, knock on wood. However you say, I know you're Canadian. I don't know which one you say. The American version or the English version. Yeah, we just do it with a little bit of an accent, you know? But yes, that would be amazing. And with such an uptick and obviously supply chain being so hot, I did obviously want to bring you on because there are some anti-patterns that I see with Kubernetes, especially, you know, in deployments, right? So I figured it was a good use case to bring you on and chat. Argo CD had to configure it securely. Yeah. Maybe some example pipelines that are good for demonstration to say, hey, like these are hardened pipelines that are sort of best practice, right? Yeah, yeah. And I do like the fact that you brought up supply chain because I think, you know, talking about Argo CD, you talk about GitOps a lot. And like GitOps is essentially the cornerstone, right? Of the software delivery pipelines, right? Because the whole automation that takes place, you can use things like Argo CD for some of that, you know, supply chain, you know, building security first in that whole process is very important. And so, yeah, I'd like to kind of go over some of the patterns that you can use Argo CD with. You know, I messaged you saying like, I kind of want to go over like this Goofis Gallant, the sort of idea. So I don't know if any of you remember Goofis Gallant, this was, maybe I'm aging myself, I'm not sure. But it's kind of like the kid that does the bad thing versus the kid that does the good thing, like be like this guy, don't be like this guy, sort of thing. So I do kind of want to, so, you know, full disclosure, everyone, I actually set up kind of like a little demo, but I didn't actually really test it. Risky. So risky, always risky, it should work, right? Cause I've done, you know, things like this before, but we'll see if the demo gods are happy with me. But I do kind of want to go over kind of like, when someone gets started with GetOps, some of the, with OpenShift GetOps, which is Argo CD, some of the things that people automatically do, and maybe you should probably think twice about before doing. So I'll share the screen there. Yeah, that'd be awesome. Let's share it out and let's start with the basics. Oh, starting right on the command line, eh? Okay. Yeah, I don't mess around. I just go to command line right away. Let me, do I need to make it bigger? Is that fine? Yeah, I think it's okay, unless somebody in the chat wants to comment. Yeah, just, yeah, if somebody in the chat just let me know. Yeah, and welcome all the chat comments to everybody who's watching any questions, Argo CD related, security related, more than happy to call out live. Well, yeah, or if I'm doing something wrong, let me know, we like it all. So I have actually already installed, if I do OC GetOperators, I think for a, I already saw the, I already saw the OpenShift GetOps operator, right? And so normally what people would do, get routes on this guy, let's do, what was the name? Oh yeah, I almost forgot. But the OpenShift GetOps, right? And then I can get that route and go over here. And sorry, real quick for anybody, warning, potential security risk ahead, that's. Yeah, I use self-sign certificate. I trust my own self-sign certificate, so that's fine. You probably shouldn't. Anybody in the chat, we can see all the chats. So wherever, whatever platform you're on, throw it in there and we'll call it out. Thanks, Bjorn. What, the, so one of the things that OpenShift GetOps with you is like that integration with OpenShift, right? And so you can either use the Argo CD, ODIC connections, right? Or the OpenShift authentication, right? Obviously I recommend using the OpenShift authentication. So you have that single source of where people can log in, right? And so, which makes it convenient, right? And so here, I created a user. Let's see if I can remember the name. That's not it. All right, so this is the standard ODIC, right? And so they'll come in, they'll log in here, and they'll try to create a new app. So let's create a new app, my test, let's. So here's another thing. I should make this a little bigger. And this will come into play later, but. The default project. So there's a project, right? So this is where, especially with OpenShift customers, they get confused here. They see project here, and they immediately associate it with projects in OpenShift or Kubernetes namespaces. They're completely different. So talk about an overloaded term. I think projects are overloaded term, but this is, so here, Argos CD has its own concept of projects. So whereas in Kubernetes slash OpenShift, the project and namespaces, right? One project associates to one namespace, whereas in Argos CD, a project could spend many namespaces, right? A project is basically a housing for your application, your application can spend multiple namespaces, but for the sake of this here, we'll do project, we'll do a default sync policy, right? You can do automatic, you can do manual. We'll do manual. And you can change that after as well, right? The sync policy? Yeah, the sync policy, right? So you can do manual, you can actually do, let's do automatic. Actually, I think if I do actually have a YAML for this here so that makes it a little easier to read. Yeah, which is nice because all of that configuration that you just went through, you can save, correct, for Argos. So even though you're going through in the user, in the UI, you can have everything as code as well, right? Yeah, so here, this is the application, right? So this is kind of the YAML view of what I was doing in the UI, but to kind of say, hey, you know, I'm gonna deploy some Quarkus app, right? In this namespace, you know, this is where it trips up people, namespace, but then you have project, these are different things. So like in the namespace inside of Kubernetes, and these are my parameters and these are the versions, right? Kind of the standard thing for a helm chart, right? And if I can jump in, the reason for the project is also so that admins can help group developers that are working on multiple applications. Really like that's the use case for it because you logged in as a developer but there is sort of an admin view as well, right? Yeah, so the, and so that's kind of like where I was getting with projects is yeah, so like you'll have to set up some sort of RBAC because when someone comes in and wants to deploy this application, for example, I do an OC apply, if I spell it right, right here. Oh, look at that. Here it'll do, it'll try to sync. This will probably, this will fail because by default, OpenShift GetOps doesn't give the Argo CD service account access to do anything outside its own namespace. So. Which is the hardened OpenShift platform getting in the way a little bit, right? Yeah, yeah, yeah. So this is, and this is kind of like the OpenShift default for almost everything. And which is, it's kind of like the argument you always have with people is like, OpenShift is hard, I like, no, OpenShift is secure. You know, you're used to like being able to do everything. We don't allow, so then what people end up doing, they go here and they say, okay, well, you know, screw that. If I do OC, just get SA, OpenShift GetOps. BNF user brought up a good point too about the projects. So that you don't have an overload when syncing. So you can break up your applications in the smaller. And it's a smaller, yeah, finite. So you don't have these massive microservices that will crash if you try to sync them all at once. So the instinct, right? And if you do OCADM, add, oops, I see in policy, I see in policy, add a cluster role to user cluster admin. This guy here, well, you don't need a namespace because you're doing cluster admin, right? Oh wait, I do need to provide those. So now if I, you know, I do that and I do terminate. So this is, because it's never gonna finish here. Oh, now I have to go to the sync, sorry. All right, terminate that. So if I design the, to terminate instead of, like refreshing the sync status? Yeah, so you have to terminate or else it'll never, it'll. Gotcha. I mean, you could technically wait for three minutes and it'll do it again. So, so this, so this way here, so notice two things, like one, I logged in as a developer, right? But since I gave cluster admin access to Argo CD, it doesn't matter what OpenShift thinks I am, right? So like just because I logged in as developer, it has nothing to do with the access that Argo CD has. So this is, so I wouldn't say this is necessarily bad, right? So like going back to the supply chain conversation, it's like if you're doing a supply chain delivery on an application, that may be fine to give cluster admin to your Argo CD or whatever get ops controller you have, but then you need to go in, you need to either trust it or don't trust it, right? So you need to either absolutely trust it and not let anyone log in to OpenShift, right? Or Argo CD because Argo CD has that access. And you can give Argo CD access per namespace basically, right? Into the Detroit to deploy and sync. So yeah, it's not necessarily insecure. It's just more of now there's another layer of a possible misconfiguration if somebody's not paying attention. And you give developer admin permission by accident through Argo, right? I think that's the worry. That's the worry, right? And then Argo CD has its own RBAC configuration. So let's go to my second cluster, the Mr. Gallant here. So this is why you see the multiples. So here, let's go developer, except of the same account. Here, like the same thing, right? Another like, I guess not bad or another anti-pattern I see is that like here I can go here and I can go to app details and I can do edit. Oops, not their parameters, edit. And I can say, hey, I want to change this from two to like three. Then save this here. So it'll actually deploy that pod. So within the UI. So I think this is what they call click ops, right? So like for me, from like being a get ops guy, like I wouldn't do this. Like I would actually check in this YAML and keep this in version control. So even if I edit this in the UI, it'll just revert it back to the known state, right? To the state that I said. So that's like another kind of anti-pattern that I see. If you're doing get ops, if you're not doing get ops, then it's fine. It's just automation. But if you want to go towards get ops, it's an anti-get ops pattern. Yeah, it's, you want to get to the point where everything is checked in as code and then configured into your environment, especially when you start working with developers. You don't want developers going in and clicking changes in Argo, correct? I think that's probably the biggest pattern you want to stay away from. So this is what this cluster, this user is going to do, right? So they're going to deploy an application here. So this is an application. This is not a Helm chart, but this application actually, if I do a tree of, let me clear this, that of this guy here. I have that, is this it? Is this it? Yes. I have the same Helm chart, but then it's checked in, right? Also another thing, which you brought up, like you can give Argo CD access specifically to specific namespaces, is that I annotated this namespace, right? And so by default, Argo doesn't have, like I said, doesn't have access to deploy anything to other namespaces. You can actually go and granularly add those in after the fact. Yeah, that actually ties in really well because Rockhound had a comment and they asked, so where would I basically limit our backs for devs in Argo, right? And so you would, go ahead. Yeah, so you would do that. So like I have here, so I have other applications, like the price list application, which is an app actually when you came on to my show, you scan that image. It's still bad by the way, but I haven't gone back and updated it, so don't scan this with the Stack Rocks. But this is an application that I'm deploying into an app project, right? So here, so this is kind of, this is where you define your Argo CD application. So you have, I'm sorry, Argo CD project. So I gave it a name, what resources it's allowed to use. So you can actually even say this project can only deploy secrets, for example, right? Right here, since I want to deploy everything, I just kind of put asterisks there. Destinations, right? So you can then say only to this server, because you can manage multiple clusters with Argo, to this specific server, to this specific namespace is the only project, right? Source repos meaning like if you have different Git repos, you can list them like so like this app. So you can get pretty granular as you see here, just within this repo, only deploy it to this specific cluster in this specific namespace and only these specific resources, right? So you can get pretty granular and on top of that, you can assign roles to them, right? Yeah, that's awesome. And so just a kind of like quick background on roles and this always trips up people. There's only two roles in Argo CD, full admin and read only. And so like you can either give people the world or nothing, right? And then the reason for that is that everything else is granular, right? Let me make this a little bigger. So everything else, you have to define yourself. So for example, and then this is basic crud. So like it's kind of like a ACL crud, create, update, delete sort of permissions. So for example here, I'm gonna give the role developer. So whoever is the role developer, the ability, this is, it kind of like I'm speaking backwards. So the object is applications, you're able, you can get that. And within, for all applications under the price list, application, the app project, you can, sorry, in the namespace, in the project price list, you can get all applications, right, if you're allowed. So I always read it the other way around. I'm allowed at all applications in the price list namespace is essentially what, and then you basically have to do it for each one, right? So it's really granular. You have to say, I can get it, I can sync it, and I can list them, right? And so you're saying, what's nice about this is, and you gotta think that probably operations isn't doing all of this work. What's nice is you can go and say, hey, here is admin for this namespace, let's say, complete control, and then you give it to maybe your head developer or head operation for the application, and then they can go and decide based on their team structure, what makes the most sense, right? But overall, I know that they can't break out of that one namespace because I've carved out that project, right? And this is like their little island, right? And that's their bubble. And so like you have your main operations that's kind of like, you know what, here, I'll let, because you don't want to be sending all these emails to developers on applications, I just kind of carve out those bubbles, and then you have different teams with different access and they can coexist and get as fine-grained as they want to get, right, with permission. Yeah, yeah. Which is really useful. It's really useful, right? And it has that aspect of delegation. Yeah, you give the power to the developer and the development teams, right? Yeah, yeah. And there's different ways of doing that, right? Like this is one way of doing it. Another way of doing it is have multiple Argo instances running because you can just, because it's an operator, right? So you can install the Argo CD operator to like the cluster operator to like manage the cluster, but then say, hey, developer, here's your own private version of Argo CD. That's another way of doing it. A lot of teams do it that way. Yeah, and you can, if there's an issue with having Argo on the cluster, you can also tell the operator to basically with a taint or toleration that Argo was not gonna run on this cluster too, right? That's just one of the built-in great functionalities of operators. Before you move on, BNF user had a question. How do you handle the state change with image name? I know not using latest tag isn't best practice, but using say, shop becomes impossible if you want to use it all as infrastructure as code. Yeah, so for GitOps, it's, and I don't know for what your opinion on this is Mike with respect to security, but for GitOps using floating tags as an anti-pattern because someone can always just update the tag, right? So like, because of GitOps, the idea is that what's in Git is reflected on my cluster. So if I'm using a floating tag like dev and someone changes, just force pushes an update on the image, like then there's a, then the application updated without me knowing, right? And so there is a project. It's within Argo CD project, they have like Argo CD labs. So it's guys like their sandbox way. There's a project called Argo CD image updater, which the idea behind it is it takes a lot of that guesswork out of how to update, manage, maintain those images. Yeah, to me, it also depends on how you're going to implement it if you're implementing continuous deployment, right? Where it's an automatic update, just understand that there's security implications behind having any sort of automation make changes like that. For me, I personally would rather there be some sort of manual process, automate everything up into the point where somebody's saying, yes, this is the tag that we want to go with, but really there should be some sort of, even though you can't just automate. Yeah, you need some sort of gate and you need some sort of tagging process that people follow and documented well. Yeah, I'm a fan of versioning. So I may be like stepping back for what I said. I personally don't use the Shaw, I use versioning, right? And so it's more like a process sort of thing where it's like, I use V1 and then when I deploy again, I use V2, when I play again, I use V3. That could technically be considered floating tags. When I say floating tags, I don't mean like environment tags, like dev, this one's tagged as prod. I mean like actual versions. Sometimes I'll like the get commit, I'll chop off like the last six digits or whatever and use that as a tag. So it corresponds to a commit that I did. But yeah, so a lot, so part of what the Argo CD image updater does is you can have it do a PR, right? So like you can have it like, either write back directly to get or do a PR for you. So I think a lot of people have done the PR as a gating mechanism. That's awesome. Because yeah, you wanna update to the newest version so it creates a PR, basically just updates the tag to pull the newest image version and then goes through that reconciliation process. Yeah. So, cool. Which is cool. Sorry, I disrupted the flow a little bit. Yeah, yeah, no, this is all great conversation. No, I love it. I do wanna point out though here. So, I did say there's only two roles, read only in admin. You might be saying, hey, Christian, I see you have role developer here. So like what constitutes a developer? Like how, where did that come from, right? So they say, this is policy. This is what the policy applies to. This is the object. This is what you can do to the object within whatever project and whatever your, so what is, you know, where does that come from? So, I once said that get ops is kind of like rat droppings, right? So there's, they're like all the configurations everywhere. I think that's Kubernetes in general. So they actually, let me pull that back up here. If I do, there we go, let's do an edit while we're, so that way I can use VI. So here in the operator, this is where I define it, right? So here in the operator, it says, all right, I'm gonna define a group. What's the name of your group? Well, the name of my group is developer and the role I'm assigning. So this here, this is basically, I'm just creating that definition. So whoever is part of the group developer gets assigned this role. What can the, what can this role do is then define within the project? So I'll let that simmer a little bit because it's really, really confusing. It took me a while. I can barely explain it now, but it's the recap here. I'm defining whoever is part of the developer group gets assigned a brand new role that I'm creating. So I'm creating this right now called role developer. What can that developer do? It depends how you define it in your project. So just kind of two ends of it. How that, so then how does that come through, right? How does that, let me go back here. If I go to user info, you'll see that I have that this user info is came from developer. This actually comes from OpenShift. So if I do a, oops, but OC get groups. That's really, you can, it says that I'm part of the developer group, right? So that's kind of how you tie it all together. You have three things, right? So you have the OAuth, SAML, whatever you want to do, right? I'm using HT Password. You can use SSO, whatever, right? How that group comes in inside of OpenShift, you can then tie it to Argo CD here. And then from there, you can tie that into your RBAC. And so, and then you can do this multiple times, right? So that you can, if you have an admin, you can create an admin group and they can do whatever you tell it to. So you can get mix and match that sort of thing. Yeah, and having just, having the two sort of separated RBACs, let's say, like having the ability to create a role in Argo and then the roles on the cluster itself allows you to get such a fine grade approach because Kubernetes RBAC is notoriously very annoying to work with, especially if you're getting into the weeds on YAMLs. So it's nice to be able to group it all together and then just say, here, Argo, you handle deployment within this and then we'll handle everything role-wise within a user interface that can then also be declarative too, right? Which, to me, that's awesome because, like, it's a year operations team. You can go and work with one team to create a template that then can be shared out to say, hey, here's a default of, let's say, senior engineers and junior developers or something like that. Default permissions in Argo, take it and expand on it to fit your use case a little bit more, right? So you have that capability, which I think is awesome. Yeah, yeah. Actually, so this is proof here that I didn't, I didn't test this. I know what the error is, right? It says resource quota is forbidden. So like, I'm not actually allowed to set a resource quota here. So this is, actually, I might file a bug on this, I'm not sure. Hold on, desired minus if you do. Oh, I know what the problem is. Cool, we can do this live. We'll do this live. So remember when I said, let's go back to, where was I? Yeah, there we go. Remember when I said that, that I can specify, I can annotate specific namespaces for it to allow to deploy to. So it looks like I forgot to annotate this namespace here. Notice I do open ship get ops. Look at, I push it, get status. Let's do this here. It might be just how they are back is set. And so for those who are watching, in order for our go to have permissions over the namespace, you add the annotation, right? And then it will, our back will automatically update so that our capability to push to that namespace, push changes for that namespace, I should say. Oh wait, I have to go over here and do this. So I have what they call an app of apps pattern. Unable to deploy revision. Yeah, so this is my R back, right? So I'm logged in as developer. This is my R back coming into play because I can't actually sync this here. It won't let me, because I'm a developer, right? So. Awesome, and it tells you that, right? That it's a permission issue? Yeah, it says permission, yeah. So if you were to log into it, min and hit sync, it would sync. Yeah, so let me log out. Let's do that. Put me on the spot, let's do it. I don't know what my password is. So I'm gonna destroy the cluster anyway, so. Okay. Keep admin. So let's actually look at the R back. So I see here admin. Oh, I forgot to mention, you can set a default policy, right? So default policy, I set to read only. So if it can't find the user, so if I didn't define it here on lines 83 and 84, I'm pointing at the screen like as if you can see me. If you can't, if you don't define like lines here 83 and 84, if you don't define it, you can have a default policy to say we'll read only, right? So you could be a catch-all. Yeah. So I do have admin set there. So let's do this here, keep admin. So you find some of the top mistakes most of the time are misconfigurations between the two R back policies. Because so that's one that comes to mind. Another one that comes to mind is get repository syncing. That can be challenging. I think it's more of what you were writing about at Conway's Law, right? It's maybe trying to force the application to mirror your organization, but you might be trying to force the functionality too far. Yeah, yeah, it's, yeah. So it is Conway's Law. So it's almost like it's, you know, if you're having trouble with like with your Git repo or like with your Git structure, it's actually not, it's actually, the problem is you. Like it honestly is, it's like your organization isn't allowing for these processes, right? And so. Sometimes for a reason, right? Like it's not. Yeah, sometimes for a reason, yeah. Yeah. Because I've seen this happen a lot, especially with GitHub Actions, for example, when they first came out, right? They were so powerful in a sense to just be able to push and that everybody was giving access to GitHub to like push directly into a Kubernetes cluster, which it can be a little bit scary. It could be a little scary. So I know what this error is. I know it's basically, if I do OC Git, I thought I was going to run into this problem. If I do OC Git, it's role bindings, I think it is. GitOps. Oh yeah, I can do it in the namespace. Let's do in priceless, that's what it is. We have this guy here, Argo CD server. So the problem is, I'll just do this live. This guy here, the role is. Ah, the classic role and role binding dilemma. Yeah, that's right, I messed that up. This is exactly what I was saying about mess around with RBAC in YAML format, right? So anyways, the reason why my app isn't syncing, even though I have the annotation, is because the annotation is a pre-described set of, it's basically a role, right? It's a pre-described role. So when you annotate that, the backend controller sets all this up for you so you don't have to worry about. One of the things that it doesn't involve is actually, if you notice here, if you look at the resources, there's no resource quota, right? So like if I wanted to, I can go here and I can add resource quota, that would be a part of core, do you know? Questions for the audience. Oh, that's good. Yeah, we're putting the audience on the spot and I gotta go look at the Kubernetes documentation now. Okay, resources. Sometimes it's good to see other people debug. Okay, plus, no, so it is part of core, resource quota is part of the, here we go. So anyways, that's how you find out. I don't know if all you know that. If you do, oh, see API resources and then like you grep for, let's say pod, oops, that's prod. You'll see that pod is part of the core API. Horizontal pod autoscalers is actually part of autoscaling API group. Anyways. Cool. There's a whole fact there for you here. So here, API groups. It says all though, doesn't it? Maybe it groups. API groups get patch delete or secrets and config maps. Why is API groups there twice? There's one, so one is for, you get patch delete but not create, looks like. Gotcha. Let's just do this. That works. I'm not, I always have to look our backup so this may not work. I'd actually be surprised. Again, let's go back here. I'm admin, so I can sync it. Remember when it told me I couldn't sync. It can't sync now. Let's try this again. Let's see if this works or it errors out again. I may be putting it in the wrong place. In the namespace price list. I may be putting, let's try one more time. Oh, I know it's from operators. It removed the, it's helping you. It is helping you. That'll never go away. So the way around this would be to create a custom resource and attach it to that. So. This is kind of a good feature though because obviously if you have a hardened cluster with set features and you're trying to push something with no resource quotas, right? That is insecure. That is a good catch. That's one of the biggest issues I find sometimes. One of the last things people get too is resource quotas on their application. Because they want their applications to run and then all of a sudden it's a big memory hog. That's what I call an umphest. Yeah, umphest. You get an umphest. Kubernetes has done a lot of work on memory and actually auto scaling based off memory. There was some features that came out in the last couple of releases. But I remember what two or three years ago when you just get the out of memory all the time and the cleanup for those was not really succinct in Kubernetes. It's a huge challenge. I'm very proud of this fact. If you Google GitOps dash examples I'm the first hit. Oh wow. So there you go. Google celebrity over here. Yeah, I'm gonna try to find another app here that'll... So let's do this live. Let's do this from scratch here. So this one looks pretty good deployment. This has the namespace though. I don't want to, that's always a problem. I could use customize but we literally be here an hour for me to do this correctly. So I'm trying to find something that is not on as a route. This one is overlays. Oh man. I did find it slightly funny that a lot of the conversation has been about RBAC but misconfigurations. I don't know if anybody watched the OOSP top 10 that just came out in 2021. I think the number one excuse was misconfigurations just in general in the cloud because you have so many applications, so many different permissions now that getting the cluster user project permissions correct without allowing for too much permission is a big challenge because you get into these situations where sometimes you're not quite sure exactly how to debug it and so you just log in as admin and hit sync, right? Yeah, yeah. And that's exactly what every human would do and so you need to have these processes in order that are actually, well one, useful, two, easy. There's the whole level of security of if an update means Chrome comes down and it comes down for five minutes, well I'm less likely to hit the update button, right? Yeah. So we need to get rid of those anti-patterns, get it set up properly and make sure that people are aware of their permissions, right? Yeah, and this is actually, like you said, an actually like a good thing that like even me as admin can't sync this, you know, good news bad news with operators, right? Like the operator, you know, switches that back, you know, whether or not you're able to set resource quota, that should be up to your organization. I think the operator should allow it, but resource quota is I can see the argument for not letting I understand why it's the default. The default is to not allow people to update resource quotas. Eventually you can get to the point obviously where you can say, hey, this namespace has this total amount of resources allowed, right? You can set that at the namespace level, but what's really cool I found is, like we're talking about Argo CD for developers, but Argo CD for your operations teams would be awesome too, right? You have a good repository, you have, you know, your operator declarations and yamls and stuff like that and you can make changes and then sync and update the cluster in a version controlled manner too, right? Which is a lot, so I, you know, you and I both deal a lot with our field here at Red Hat as well as customers and I've seen that pattern a lot where kind of like where I mentioned before where there is a admin Argo CD instance and that's kind of like the overarching instance that sets up the cluster for multi-tenancy. So you can still have multi-tenancy. I think that's where OpenShift shines a lot. There's a lot of tools for multi-tenancy and so we having, being able to still leverage the multi-tenancy and still have like total control is something that's really powerful, something you can do with Argo CD, right? So like with Argo, you have the cluster Argo and then you have maybe tenant Argos, I guess is what, you know, you know, you have those default, let's say cluster setups, maybe some of them are devs, some of them are tasks, some of them are greenfield deployments or something like that and you can use ACM or like ACM as a policy engine to go and then deploy those new clusters too, right? So everything has code for operators and developers and then- Yeah, common patterns. Yeah, common patterns and so, and then it's nice too because you get your security experts to go in and say, okay, well, here are the hardened defaults, like these are the things we can change on, these are the things we're not gonna compromise on. You know, maybe in this cluster we can allow a little bit more flexibility because maybe it's all stateless, right? Yeah, yeah, what another, I think like I'm thinking more of also just like the process so like Argos City will keep your cluster in sync, right? But like how things get into the cluster is also an important thing. You know, you came on my stream and talk about, you know, you scan the images, right? Like how did those images, you know, that with all those vulnerabilities, how did they end up on my cluster, right? And essentially it's just like my process is broken, right? The process of, you know, you can have, you know, the drift detection is fine, but like what if the declared state is bad, right? Like what if I'm running an image? Like how do I, you know, how did that get into my cluster? And I think a lot of, you know, we, at least I think more time should be spent on the CI part, right? On the pipeline part where it's like, hey, you know, you can do a cube linter, right? For your YAML and then you can do an image scanning and depending on your criteria, you can, you know, allow to promote that. Like you talked about gating, gating I think is very important as well. Yeah, the, there is a lot of time spent on runtime detection, drift prevention, right? Those are sort of your brake glass things, right? If Argo, let's say is automating something and it gets to the point where all of a sudden something, you know, crashes or there's a vulnerability that was introduced because of an automatic update, it's nice to have a tool to automatically notify you to say, hey, by the way, there is a serious vulnerability like a lock for shell that's in your, in your cluster in a container and something that is close to something important, right? Maybe a storage device. There's also the difference between something that's stateless that doesn't really have high impact, right? And it has maybe a bash capabilities. So that's a little bit of a different situation, right? Cause maybe it's not publicly available. It's very, it's a, maybe a developer tool to diagnose things, low impact, especially a dev cluster. So there's how we adjust and look at things in the cluster and then how we adjust upstream is the biggest part. And if you're, if they're the security team that just says, hey, by the way, this container is not, it's too vulnerable for us to run the cluster. Like you have to go and fix it. That's not really useful for the developer, right? Yeah. Realistically, it should be sort of cordoned off. You should go into the process upstream with Argo CD or some, or your CI process, whatever you have set up and say, hey, these are the packages that we're not gonna let through. So you just need to update and, and we'll notify you if anything like that happens, right? And honestly, if you can do it in an automated way, that's awesome. Like something like ACS does this, imagine having eight teams and you need to push log for J. I don't need, I can send out one email to all the teams and I can just enforce the policy to say, hey, you have to update anything with log for J to the new extension. Yeah, and it doesn't, it doesn't pass, right? And so, you know, then that's, I hold the supply chain developer. Like I always think about it, like, I think, I don't know who said this, like Henry Ford, right? It's like an assembly line. Something is bad. I just knock it off the assembly line, right? And, you know, in, you know, the cars don't get built with some, with a bad part. It's essentially, that's, that's the idea. Someone in chat actually, Luis said, better one Argo CD per cluster. I actually agree with that. So you'll never get, hear me fighting over that as a get-ups purist. If I can get up, get on my get-ups. So box for a second, like, so Kubernetes was the idea of like, I'll take away your SSH keys away, right? That's Kubernetes. So get-ups is the idea of I'm taking your cube CTL away. Like I'm taking OC, cube CTL away. I'm installing Argo CD. And if you want any changes, submit a PR, right? And so, and then, you know, it's kind of like that, that, that, if you think of a pipe, the faucet, right? That has multiple points where you can control the water flow, right? So you either control the flow, you know, in your get repo or you control it in Argo CD. That's me, but I understand not everyone has the same ability to do that or even the same, you know, even wants to do that. But then anyways, Luis, I agree with you. I'm all about putting Argo in and then controlling everything in the get workflow. Yep. We had this discussion actually because we were doing some commentary on education. And one of them was, you know, configure your application securely, but then deploy your application securely and somebody commented, well, aren't those kind of the same? And I'm like, well, are you giving somebody the ability to cube CTL deploy? Because that also means they can cube CTL delete. And so that's not necessarily, right? And how is those permissions working? And by the way, like, are you giving cube config files out to everybody? Like, how are you ingesting that? How are you getting your developers up and running? There's so much more operationally that it's involved. So, you know, setting that up where, hey, you have this, just here's your developer environment, check everything in to get. And maybe you do an automated scan, check configuration in dev and we leave it alone because it's in a private cluster. But by the way, you need to fix these three things before the next release in six weeks or we're breaking it, right? Like it's just not gonna get done in six weeks. And also your security team can have a little bit of give and say, okay, well, there's six top things, but these are the three things we need to focus on because we all know that it's engineering cycles. You can't be too strict, right? Yeah, yeah, exactly. We'll get to it to the next sprint. Someone mentioned in the chat, is there a CIS benchmark which includes Argo? I'm not sure, but that does put a good segue about the vulnerability that was on in Argo like a couple of weeks ago. Yeah, we do have five minutes left, so I would love to talk about that. It was fun. Yeah. So this company actually was hired by Intuit. I believe it either Intuit or just like the Argo project as a community to do a, what do you call it? Security audit, like an assessment security audit. And they found the CVE. So actually, can I put this in the chat? Can I chat here? No, I can't chat here. Anyways. I can put it in the chat. I have it, I'll put it up. Oh, you have it? All right, cool. What I found funny most about everything is they used the Red Hat logo and just turned it black to indicate a hacker, but okay. That's funny. That is pretty funny. So anyways, a too long didn't read version of the CVE is that there was a way to traverse the directory structure within inside of Argo. So specifically when you were using a Helm chart, there was the ability to, so the way Argo works, there is a, I can like quickly say OCD get pods. I'm going too much into detail. Argo CD has what they call the, is made up of different controllers. One of the controllers is, oh, I just noticed that you can't see it. There we go. Here, one of the controllers is called the repo server and it's job is to basically do the get clone and the like get, essentially does a get clone, get fetch, does everything with the repos. Even when you're using a Helm chart, it saves it in the repo server, right? It's just like a generic repo server. So this here, they found a way to hop from one application to another. So here, here Argo CD, so each one of these little cards here is an application. And they're supposed to be self-contained, right? So the vulnerability is that they found a way to hop from this application to another application and see that application's data. Regardless of- The metadata, the secrets, what's injecting. The metadata, secrets, yeah, all that stuff. So me in a multi-tenant system, that's a big problem, right? Me, my app, I can go into Priceless app and look at the secrets, right? Database access, right? Especially namespaces, namespaces being such a significant security containment unit, let's say. Anytime you can hop namespaces like that, it's scary. Yeah, so when you can hop from one thing to another, it's definitely scary. Read that article, it goes into detail how it worked. So what can you do is essentially update, right? So- Yeah, it was fixed extremely quickly, right? I believe a full five days. Yeah, like within the week, they pushed a patch update. The community pushed an update and then Red Hat then not only pushed that same update, we actually back ported an update- To all the previous versions? Of OpenShift, right? So OpenShift 4.6 has a version of OpenShift GitOps, which is kind of older version of Argo, but they back ported that fix for that. So it's one of the, you know, it's a Red Hat show. So I'm telling you, it's one of the many great things you get from Red Hat is the back porting fixes. Yeah, and we're honestly like a lot of companies will just say, hey, we're not supporting this version anymore, you have to upgrade or get off, right? Yeah, essentially it's upgrade would be here. But we went back and we back ported that fix. So if you're running OpenShift GitOps, just make sure you're running the latest version of whatever version that you're on. So actually, I had this up here, where was it? It was on, for OpenShift 4.6, you need to be on 1.3.3 or newer or 1.4.2 for any other version. So, awesome. So this is cool, this is kind of, you know, showing, and this is kind of cool, it actually goes into the code. But it shows that the Argo CD community has essentially security built into the culture, right? So they hire people to say, hey, find something wrong, they find something bad, all right, cool, we'll patch it. It's kind of like their mantra for is security first. And honestly, this is, we do have to go, this is a great wrap up actually, because I did want to touch on two things. Is one, the speed at which open source community comes around to fix security vulnerabilities when they're there, I think is impressive. I think people are used to having their security applications gated behind some sort of subscription service. But then, how secure is your security platform? You normally are giving your security platform admin permissions in a lot of your clusters and environments, right? So it's nice to be able to have eyes on these things, to trust them. And yeah, that's a great segue to basically just saying, everybody come back in a month, we're gonna have some Stack Rocks open source news for y'all that are watching. And you can join Christian, get ops guide to the galaxy, is it every two weeks? Yeah, every two weeks, every other Thursday. So not this Thursday, but next Thursday. Next Thursday, I'll be talking about Argo events. Nice, yes, and there's a previous, a bunch of previous stuff, it's all on YouTube. If you wanna go check it out, I did a talk about Stack Rocks showcasing ACS when we first got acquired and brought over Red Hat. So you can check all that stuff out in the library. Christian, thanks for joining, thanks for your talk. Oh, thank you, I appreciate it. Appreciate the time and look to see you back here next month, third Tuesday of the month for some Stack Rocks news. Take care, everyone.