 Good morning. Good afternoon. Good evening and welcome to another developer experience office hours here on red hat live streaming I am Chris short host with the most and showrunner for this great thing. We call red hat live streaming I'm joined by two of my favorite technical people here in the in the world DeWon Ahmed and Carlos Santana not with the guitar with the Kubernetes clusters DeWon do you want to tell us what we're talking about today Absolutely. Absolutely. So I'm dialed in from the beautiful new Brunswick on a relatively cloudy day That's why I have multiple lights on so today We'll be talking about Kubernetes admission controllers and before I pass into Carlos, I wanted to ask the audience who's using Kubernetes admission controllers. Oh good question And there's a slight delay. So maybe Carlos and or and introduce yourself. Yes, please Folks, let us know if you're using an admission controller. It's a good Yes, a delay Yep, my name is Carlos Santana. I work at IPM as a cloud architect Mostly working with clients on DevOps and cloud native Helping them with OpenShift. So I work a lot with OpenShift Helping them get into the cloud basically and I also do open source. So I'm a technical lead 4k native I run the working group for user experience and docs there So happy to be here. Yeah, we're happy to have you Carlos. This isn't your first time on the channel. Is it Carlos? Line presenting. Yes. Wow. Okay. I can't believe it's taking this long because you've always like been around I feel Yeah, in the chat. Yeah, you're always watching and that's cool. We appreciate that But yeah, it's great to have you on the first and Carlos is actually one of my ex colleagues at IBM Yeah Just to work together in IBM. Yeah. Yeah, you're still in IBM. What do you mean? You never left I see I see that was a trick question So to comment coming back to the admission controller, please that was a trick question because by default I think Kubernetes ships 30 plus admission controllers with a yeah and The the specific controllers will be so you're using it by default even if you don't know it. So today Carlos will be talking about admission webhooks and how He can use these to to manage large set of our go deployments It's it's one thing to deploy a single app on a single cluster But when you have many apps and you're dealing with many cluster What are the challenges that that we're facing before that Carlos? Would you be able to give us some some idea about like what these admission controllers or webhooks are? Just from the very general point of view Yeah, so let's use the official Kubernetes stocks because open shift is Kubernetes, right? And Kubernetes open shift. There's actually a top people stories about Open shift that there's things that started in open shift and then there were like upstream to Kubernetes And some people are confused saying all that's in Kubernetes, but it was in open chip first, right? And then show up in Kubernetes. So they're they're like siblings So let me share my screen and go over a little bit about you see my screen. Yes so we'll go over the admission controllers and admission controllers We're going to talk about the dynamic one. So there's many of them, but there's two specific ones that are very Useful, so one of them is the validation that you can see here a validation webhook and the mutation webhook and I don't know if they have the picture in this in this but essentially is when Something and it could be a one of the admins of the or a developer Working with open shift. It makes a request to the QP API server open chip API server It goes to the different life cycle Checks you can say like authentication. You can see here identification Authorization and then comes validation and mutation webhooks. So if you take the validation webhook Is your opportunity you as a programmer or admin or this is also very very Use in operators when you're writing operators There's a chance the API has Registered you have registered with the API saying Ken is this allowed or not allowed am I allowed to create a namespace? Am I allowed to create a deployment? with a image that is from Docker or or Or the Amazon registry or it must be for quake And you can decide am I going to allow it or not allowed and then and then the API service asking you and asking like All the people that have put validation webhooks. I think that's what I'm saying like I'm using open shift there's already validation webhooks that you can see in there and If one of them say deny forget it, you're not getting to allow access. So this is very different from our back You're familiar with Robics access control So our back even though maybe you are an admin and you can create a namespace Let's say the simple example that it will use a lot if let's say you are an admin straight up this open shift cluster and for some reason I Managing this cluster for for a long time and I want every namespace to have an owner like a name And an email of someone because maybe three months from now I Wonder if this namespace still used right and we can delete it or not. So who do we ask? Usually by that time we cannot find who was the person that created this So maybe I required that that namespace Has a label maybe owner and an email right or an a I Added a link in the chat Carlos if you would like to open that link. I think it might be easier There's the picture which goes from out Yeah, so this is the picture I was looking for. Thank you. Yeah, I would have the other one Yeah, so It was so mean into here It's asking it's asking so evaluation is here, but I was just saying at first as validation and Basically validation is like asking, you know, all these web hooks and web looks at People think that are like very complex type of things. They're just HTTP requests with post. It's a web API Similar similar ones that you use in GitHub when you push something to get help you get called by a web hook Basically doesn't HTTP post request and typically they have application gaysum So what I was saying was if I'm a Atmine and I want to have in my cluster like every namespace might have a Label with the owner I can create a policy. I can create my own validation web hook I create my own code. So I like it. I write a little web web code that says if the namespace to be created Doesn't have a label then deny it. I don't want to I don't want to be allowed Then mutation web hooks it could be something mutating the request. So this one is read-only So you are allowed not allowed. This one can allow it and not allow but on top of that can mutate that request So maybe I'm the lazy developers like And I own the cluster. So I'm not going to like Edit every namespace to add my email address as an owner So maybe I create a mutation web hook and the mutation web hook a skull saying do you want the api server? Ask your web hook saying I And you register like what type of objects do you want to listen? So maybe I only want to listen to namespaces. For example, you don't want to listen to everything And then you say I want to listen to namespaces and if the label owner is not present Maybe I want to add it. So I'm mutating that request So when you create a namespace, I check that it doesn't have an owner And I put like the owner is Carlos because Carlos the only one using his open cluster and I add that Label and owner back to the api saying hey go ahead and patch So the programming model is you're going to patch that resource that typically you see in yaml But it's actually In api world is Jason. You're going to patch it with a json patch saying add the label owner With my name Carlos in there and then let it through so by the time when it hits that's why validation is after So when validation sees that that request it only sees a namespace with the owners are like Very good. The namespace has an owner. So That's a high level kind of a story of like how did it work and Before I learned this I did learn about this recently in terms of like the details behind it Um, I didn't get my my head around this. So how do you for you to you were you knew this or maybe it's it's useful I think I think I didn't have much idea about this before and this five minutes overview got me got me the fundamental So so thank you for that. Yeah. Thank you so much And and crd so I want to add one more thing It's like the reason that is uh using operators is for crd So sometimes you install things like the open she's serverless. For example, the one I'm familiar with which is based on k-native Or tecton. I was working with tecton And the the version of the crd. Let's say from last year was v1. I thought one But now you install the new version of the operator so what What about if I want to upgrade those resources to vbeta or v1 like in k-native where Where v1 and we want every resource to be in v1 So one thing that you can do is if you give me a A yaml, you know, you do keep ctl apply or oc apply with a yaml with a k-native of v1 alpha 1 When you write operators, um, you create a crd conversion Mutation webbook saying I'm going to convert this resource that now has v1 alpha 1 And I'm going to convert it to v1. So when it's written into xcd It has v1. So my webbook can expect the resource saying well this field we change the name from A template to something else now This field needs to be specified. So I'm going to specify default value And and the api version now we can change to v1. So I just upgraded The operator was helpful enough to upgrade my yaml To that version and maybe give you back a message and you collect messages saying, hey, you should like upgrade this this yaml that you happen get to the v1 But we'll try our best to upgrade you on the fly um, but that's the usefulness of the crd and this was It could run it this years on open shift years will say reasons and it's like recent as two years Right, uh, so it's very recent right because I've been alone for a long time before we there was a time that was no crd Um, right if people remember programming those but uh mutation webbook is something that you can specify when you write in your operator and the operator sdk the framework operator framework sdk When you start working with the with that framework, it also helps you You know We we creating these weapons like templating that like the the metadata around it So I just wanted to add the the crd as a as a A notation there because a lot of people ask or see them when they start an operator and they check For validation mutation weapon. They start only see all these Mutation weapons. I wonder like where they're coming from. What are they useful, right? So us a cluster administrator should be aware about these things because they can Break your cluster for example So these validation weapons can have something around The failure mode so we can talk about the failure mode We want to go deeper. Um, the failure mark is like if if let's say the weapon is down Uh, do do I deny the request or do I allow it? Right because this is a thought running in your cluster. So maybe you're doing maintenance or something or it cannot be reached or times out Uh, so you can do fail. Um, I think it would do The failure policies call You can you can fail or you can ignore If it's not here So the failure policy you can say, yeah, um, if I cannot reach that web hook go ahead and Yeah, keep it going, right? It's not that important. Maybe it's a death cluster, right? We're doing testing Uh, and then maybe in production you configure the yaml saying nope, uh, that that means I want that check. Um, I want to fail if if the if the web hook cannot be reached, right? Because it could be a security validation weapons things like uh, opa open policy agents Gatekeeper kiver no type of like they do security policies. They implement validation weapons that you can configure with uh configuration like daml or or regal Um, and you and those sometimes are policies that do need to be compliant So in production that cluster needs to be in compliance. So you don't want to Have it like ignore like if do your best to be in compliance, right? We never heard auditor saying Well, do do your best, but if you can't like that's okay You're either in compliance or not compliance. There's no middle ground. Uh with that Uh any questions from the chat? Uh, no, but we just get a hey from kaleed. So hey kaleed. Nice So these are the documentation in the upstream kubernetes, um about um validation web hook. So um, let's see how we can create one. Um, what do you think? Sounds good and we have a specific use case uh here today. So for those of you who are familiar with argocd You probably know that project for those of you who are not so familiar with argocd. I added a link in the chat So argocd is a github's continuous delivery tool for kubernetes Or when you're deploying your application to your cluster, you probably want to follow a github's model Which basically says you use gith as the single source of truth You make changes in gith rather than making direct changes on the cluster, which is very dangerous You don't do kubernetes apply. You don't do scripts Even worse, you don't do kubernetes apply of a bunch of scripts You rather make changes on gith and then those gets reflected on the cluster and the last thing is All the changes you or your team members are making are very viable and auditable So that's what argocd, which is an open source tool provides in short and Carlos will will explain the specific use case he has When dealing with large volume of argocd apps and how admission webhooks can play a role there Yep, so um, let's take a look at a validation web look The for that you have to create a a yaml file. So if we look at the at the yaml file specification This is this is the one that I talk about read only validation. So You register your webhook And these are the things the rules the rules are the things like I want to only be concerned with these objects If your service That you're going to implement your code basically the the web servers that listens for that http polls request It's going to be sitting in the cluster You you need to specify this the namespace and the name So the api server can conform the URL https The name of the service dot Example namespace dot svc like that service and it will call you you can also have the option instead of putting service in here And and replace this section for URL Um, so I I have a question for you. So if I put a URL, what what can I put in that URL? What do you think I can put in that URL? I think it's it's down here in the specification. So I can put um A a url I can put a url in that section. So what what can I put in there? What do you think I can put there? Do you see here? I mean, it would be the path to Whatever pod you want really, right? And if that's a fully quality, no It can be in my web hook example.com. It can be something outside your cluster. Oh, okay So if you convince your friend To install this yaml file that you put at the bottom bottom bottom on this yaml file Right and you put https and it goes to carlos.com I now have access to I be able to modify or validate Your your request so I could be modifying your image pod on the fly and you will never notice because The deployment says that he has one version And I put in my my hack version. So this also could be used for For malicious use. So that's why you have to be Careful about so the the web hook not necessarily needs to live in a pod in that cluster It could be living in another cluster. It could be a Like a k native and serverless function living in some cloud And it could be useful right because it could be used by many many clusters Or it could be some Exposure that you're like calling something outside the cluster That you shouldn't so that's where like you put things like egress policies You put things like istio to make sure that nothing leaves the cluster you put network policies Um, well for the qp api network policy will not work, but you will put something in the nodes right in the master nodes So you're trying to highlight that because uh, that's some some something that People are not aware that you can put a url and that you'll be calling something Outside your your cluster and your cluster may be broken. You don't know why it's because there's somebody else like messing with you, right? So that was an interesting one. So Fun fun tip there, right always check And that's why validation and mutation what we say cluster scope Resource so usually and that means the one that have access like this. There's it's not namespace So this is an example of a node.js validation a mission controller I want to show a little bit of that example So it's also good to see a specific example. I took this one from a from a blog post. I saw the other day Um, and basically this is the the yamo file. So I have the webhook, right? So I have a webhook validation mutation webhook and I'm going to listen Um, and you can be very specific of like, what do you want to listen? So in case that you're doing development, maybe you don't want to listen to everything in the cluster you want to specific So I I put this one like because I was like experimenting that it was a cluster that my team was using I put like maybe only want to listen to objects that live in a namespace That matched expression CNTK experiments So this is like telling the team like I'm doing an experiment in this this namespace if you want to You know Use my validation webhook go ahead and label Your namespace with experiment equals CNTK and then The kubei-pi server will start sending me events from that namespace. So It's it's very granular what you can listen on right? I don't and then what things why I want to listen I want to listen to pods. So anytime in deployments or We say deployment, but it's a replica set read the bombing case replica set replica set creates a pod When a pod gets created so you have things here like when it gets created deleted Updated like you can listen to anything In this case, I'm listening for pods But it could be config maps. It could be namespaces. It could be any resource In kubei neti that this is why it's very powerful, right? This is What I found out is like this is the the I would say the easiest way to extend kubei netis Right kubei netis giving you a webhook Into here. Yeah, and and then the scope is it's namespace. So these resources say nesca scope and then what I want what I want the The kubei pi to call right is is this pod With a service sitting in the default namespace and then I'm going to allow this registry and the path is slash notice that The kubei pi server always calls with https So my code needs to have Oh, this is something very cool that I found out about open shift is When you're developing this type of things either your framework or the person that is starting this or sometimes even as a bad practice Your same code is creating the certificates the tls certificates So if the the api server is calling an https endpoint and it's sitting inside your cluster Then your program needs to have an https or ssl certificates and then you the The kubei pi server which is serving as a client needs to see a who am I going to trust? So as you can see here, um, I think I saw ca bundle something ca bundle Uh, where was it? I think it was Uh, you look for ca bundle ca bundle the ca bundle It's a base 64 included certificate ca And I was like, uh, do I create a Shell script? I don't know if I have it here somewhere. Um, a a shell script that like looks like this Like everybody have done open ssl Can be familiar with this right? Uh, open ssl credit For the address of like white list register default theme Uh, create a csr Generate the ca don't have the ca now have the my tele-certificate Right and then you have to give this to security and we have to rotate them and all that Um, so the beauty of this for people I appreciate appreciate not doing this is you go into your, um Uh Webhook and you add this annotation at the top I don't know you can see this You say service the beta the open shift the i inject ca bundle So when you send these to kubernetes or open shift Open shift is going to insert in here the ca bundle Of the open shift ca. There's an operator ca in open shift So I don't have to do that piece for the ca The other thing is like, how do I get The the certificate and the key for my app and it could be go python or node. Yes, right? Um, and you go into the deployment Um, and then in the in the deployment you say Huh You create the service of the of my deployment is pointing to my service. But for this service I use a different annotation use another annotation saying service beta open shift Serving cert secret name. Basically, I'm taking open shift. Hey open shift create me a secret In this namespace with this service is with this name And in that secret I should be able to find my service certificate and my key that I'm going to load into my application Once I found out this I'm just testing your open shift like I don't I don't want to deal anymore with open ssl. Um, so by doing this, um, Suddenly a secret would show up in my deployment that I can use so now I can mount it So if you're familiar with deployments, right? You will have a volume that you can mount so you will mount mounted And then you have a value months and certs and in that search folder inside my application. I will see that Um and I just joined this, um, and then Uh, let's see the code. So it's the specific, uh, webhook what it does is, uh, having In no jet. So the thing is this is a webhook So web, uh, especially creating a web server. You can use python node java Apache htpd anything that does the web request with bash cgi um And and basically this is i'm using node jas. Um, this is where i'm loading the the certificates that are created by open shift um, and then i'm inspecting the The the jason that I get from the api service. So he sends me a a review object I can't um, so this is the git or I think I was showing the other one Uh for the node jas, uh, let me see the server. So this is a a little bit more, uh, simple one So this one is basically what i'm doing is i'm going to configure this validation webhook To allowed or not allowed? Through a false very easy And basically what i'm going to say is i'm listening for pods And if the pod starts with a certain registry or it doesn't start with a certain registry I'm not going to allow it So in this case this example is around Uh, let me see if I can find it. It should be here So i'm have environment borrow with my phone saying if the image in the pod Starts with docker.io or gc r.io Good if it starts with I don't know the hack registry.com It's not going to be allowed, right? So this is very useful for air gap environments where customers have an air gap environments They have their own registry usually they they use quay From rehab right and load their own registry and they have it in their like acame.com internal ca And for security reasons, they don't allow any any image to come from anywhere else So they will mirror images from the web They will mirror images from the open ship marketplace and they will have that Image here and they they can implement a validation webbook With this small amount of code, right where i'm checking If it starts with this do not allow the pod to be created or to be created Quick question carlos would that allowed registry url be that mirror registry url then because There will be mirroring from whatever allowed registry to that mirror registry And are they trusting only that mirror registry which they have? Yeah, so quay has a feature called Syncing or mirror so I can have a server called carlos registry You know like registry.carlos.com And I have quay running there in my inside my vpc in the cloud public outcome premise And i'm i'm fetching all those images mirroring those images there and my open ship cluster only takes images from there So that's a very common scenario in enterprise where they want to have And and that's the opportunity to scan images right for cvs constantly That's another thing that you can do nothing that is Very efficient, but it's possible right what happens if in this code I want to take the the registry Of that the the image url of that of that Of that pod and inside my my app. I want to download it And I want to run 3d or I want to run a scanning Right there and if I find something like critical, I say like nope The pod would not be allowed to be created. So that's just an example of like So you can see the code so if we go to the code Basically the these webhooks what they expect back is a A jason a response and they the the document goes into the details. This is document was written Um, I forgot his name, uh, but he's from from google And I told him the other day on twitter, uh, that this is an amazing document and I went back to the I'm a gig I went back to the get history. I did a gameplay And I figure he wrote most of this code I saw other people help but the first version, uh, he wrote this and I said like he explained this very very well So there's a section here called response Uh section and then it goes over, um, You know, what is the response the request and the response of that api? And and he goes around like why am I what information am I getting and what things that I need to reply So I need to reply with something very simple Is a a a jason object that has a mission review and then I'm going to allow it or not allow it but on top of that I can mutate it. Um, so Let's look at the at the art any questions so far. You have any questions? No, you're good. Go ahead. Yeah Or or your free to ask questions folks that are watching. Yeah, um So, um, so that was like a way of like I'm going to deny things. Um, right. I cannot change the pod But what about if I want to change the pod on the fly? Like, uh, then I will need to create a mutation webhook um, so that goes into like, uh, the example here that I have that that I wrote the other day We say it's a mutation webhook that I want to listen to, um, Argo, um, like a get ups definition of an application and projects. Um, and I want to listen the same thing like the namespace that has this experiment um and what I want to do here is Uh, listen to them and the reason I want to listen to them is because I want to run in my code and uh, the the things that I want to do is because I'm doing development with a large number of uh, get repo get repositories. Um Uh, they have many argo apps so I can show a little bit of a diagram that I'm dealing with So, um, you see my screen? Yeah, with it. So we have, um, typically the clusters. Um, this is not Like, uh, uh, what everybody does right or specific to everybody does But most likely these are the things that we test. Um, so you can start with a single cluster that has the Let's say for example, the namespace is c, i, c, d, dev, staging and prod And then you can go into like, well, I cannot do everything with one cluster I need two clusters because in an enterprise you start like dividing the the resources Um, in in theory, you should be able to have this right with the tenancy and open chip with a big Cluster with a lot of worker nodes in practice a lot of customers want to have their own Own cluster for different purposes so they can have one for production only Uh, or a shared one or they only have one for production and the rest Or they can't even go to the extent if they're a large corporation They can have multiple ones for production meaning like active active or multiple regions And then they have one for staging Dev and c, i, c, d So all these clusters, uh, definitions everything that it gets deployed doesn't get deployed by a human That's kind of like the practice in the enterprise when you go get ups only It's like we call it like no sneaky person. It's niki sre is allowed to go and you run all see apply Um, everything is in git. Um, so then we have multiple git repos and the reason we have multiple git repos is We have different teams In in real world We don't we don't have one only person knowing about everything. We have people that are in charge of managing the Uh, the cluster infrastructure So resources like machine sets machine conflicts in that cluster Because we don't want to bother the development team like, uh namespaces cluster roles Like those are global things that should be done once Um, and maybe managed by by we call it the infrastructure team, right? How many machines you run the development team really don't Care about that. Um, then we have things like services or shared services Like operators So if you're going to have a shared service like let's say service mesh is still maybe you want one control plane And to manage that control plane the short services is the thing is in charge of that or jager or Again in idm some club pack for integration kind of like uh middleware You have one instance and it's managed by the services team and that person are in charge of that And then you have the rest of the you know population of of apps Where they deploy that so you have different git get repos that you need to manage with with argocity. So you will have Different different argo apps In different git repos. So for example Um, I have this example here. So in argo you start with a single let's say you have A very very simple starting point And this is the pattern of Apps of apps. I think you were saying uh asking me before Yeah, you create you create an argo app and you put the yaml for that argo app and This is kind of the seed like Um, you have to apply this manually or you apply this one from cross plane data form ansible Maybe the the whatever automation you have for creating a cluster That automation usually installs argo using like Red hat advanced cluster management or just puts the argo in the cluster And then say and for that argo instance, here's your your starting point. So this is called the bootstrap um, then you have Layers So you have an argo app that defines the argo apps that are only for services The argo app that is for infra and the argo app that is for apps So you're already starting with a parent and child relationship Um, and then you start with well the apps team Um, want to have different environments the ci the dev the stage and prod And then maybe they want to have this in in the github repo and they in their different teams repo They have they want to have another argo app that deals with the ones with the folders inside that repo Uh, for example, the deployables. So let me enable the deployables here And you have the deployables here. So as you can see it gets um and this i'm just describing one cluster So I said before that I have the the ci cd the dev the stage in the prop prop one prop two And a lot of people I even have preprop one preprop two They come with all the certain names. So they have you have many apps, uh, like this And you have the infra also the infra will have um services and infra will have their own argo apps So this one has like the cluster roles for the different So the idea is that The clusters Will reuse the same github repo. You don't want to have different github repo for different clusters. So if you want to follow the 12 factor, you know methodology of having like similar environments if you're using Istio for example, uh 1.6, you don't want istio 1.9 in staging 1.6 in prod You want consistency. So the way you have consistency is by reducing the ripples. So you have to put these ripples in in git So you create git ripples. Uh, so this is an example on how many git ripples you will have Maybe you have the main one that has these argo apps Then you have the the The github repo that has these folders that are shared. They're like libraries this first two are like Like libraries that are shared and then the last one this one is like for the different teams They will have their own Kind of like namespaces assigned and they have their own cycle of using like course customized for their product and staging um So it gets a little bit a little bit complex with so many argo apps everywhere But once you deploy this is static uh to the point that we have argo projects So an argo project is a resource that says these argo apps are only allowed to deploy From this git repo. So you put kind of a governance Right, so if I if if the apps team tried to create a cluster role Um, that is defined over here. Um It will say no no no no like you're not in infra during apps So have some type of control of like in the project. So in the argo project you can do kind of like a Wireless allow list saying I'm allowing Argo apps associated with this project only to deploy to these namespaces and only to deploy from these git repos And the same thing for services like only from the services Library and only these apps. So if you um want to make a change You will be making a change for the whole team across all the clusters So that's where I came with the With the idea of um creating kind of like for development purposes I'm I'm developing this architecture. Um, and it's open source. So this is open source on their project That I work on is called the cloud native toolkit Um, so these I'm creating patterns and best practices on how to do DevOps with git ops And how to structure this in a way that can scale at a large corporations So, um I'm not I'm going I'm using this as a developer of git ops. I don't if that has the role The person that develops uh git ops and and DevOps for other things to use But the idea is like this this is our um, our I'm doing automation Setting them up, but then I want to change maybe I want to tweak or debug something in one Argo apps For example, this one So if I change the the yaml for here to point to a fork of my repo that I have like I'm changing something here um Argo will complain saying this uh Argo app is not allowed to deploy on my fork Because it's not in the Argo project and then um, I want to change Sort of like one thing. I don't want to change the whole thing So I need to change now and then the other thing is like if I try to change this This Argo app would revert it because this is the parent So I need to change this one and this one and when I change this one gets reverted Because the definition of this one is controlled by this Argo app So if I want to do like development on something that I say is is running as as desired in development And I want to make a change just on my fork It's because very difficult of like Argo wants to like the state of truth To be in git so I need to change this I need to change this I need to change the Argo app to add my repo Change this one change this one and then change this one on the cluster And and do it every time that I have a new environment because I bring out clusters like, you know Every day I bring a pretty good cluster start them down Winging that up. So what I did one quick point. Sorry, Carlos is uh, just emphasizing on the point of this is something A use case on development because in production You want to follow that proper procedure of making git changes and going the entire route You wouldn't make direct changes on the cluster This is something what Carlos is mentioning that if you have large set of clusters You want to check something uh using your own personal branch or something on a specific cluster on resources You don't probably have the time or effort of going through the entire change And it's not even intended So how you're using webhooks to make change only for your development uh environment Yep, so um, I had this uh, grand dishes uh, bad idea of creating a mutation webhook and learning how to create one So I said, um, what about if I create a mutation webhook that every time, you know One of these Argo apps gets updated. I overwrite it with my fork The project and the and the Argo app to point to my uh, thing so nothing gets Reverted so I created this mutation webhook That I have in this other people that we can share On the chat, uh, call the git override Uh, if I can find it here And this is one I was explaining that I want to listen to Argo apps and Argo projects Um, and I want to modify them on the fly. So I write a little bit of so I made a configurable So this deployment webhook Will load a config map. So you as a user, um, so Let me see how this Argo apps will look like. Um, and maybe let me do this thing of GitHub now. I know people, uh Tell me lately about this, uh, you feature pressing the dot. Yeah pressing the dot The pressing the dot thing is mind blowing, but yes yeah so For testing I have a testing folder here and it's show kind of like the example what a department that I'm trying to fix So if I have an Argo project That says I want to only allowed, um, you know, um Argo apps to deploy from this git repo So if I try to create my fork or you try to your fork The Argo project will say like you cannot deploy because you you're only allowed to deploy from this git repo So this is uh, some people ask me like what is the usefulness of Argo projects like Argo projects are very useful Uh, a lot of people use the default one and then like well, I got something working. I don't care, but they are very useful Um, they're also they're like you can only deploy to these namespaces right to more governance and then, um, there's a default set of resources you can deploy but Uh, uh, you can add even more But for this example is like I want to override this this line of source repo on the fly Um, and this is the Argo app parent So this this one points to the same repo on this folder called the test child So this is the parent Argo app Uh that deploys this shout app um And I want to override the the shout app to point to my to my own fork I mean and even a different uh in a different branch So to do that, I made it like configurable one one validation webhook configurable for my team For development purposes and I told them as far for automation We deployed this webhook and uh, you based on the your personal fork. So Carlos to increase Um, you will create the config map in your namespace Or in the cluster that you have a all of us after The own cluster by using the same repo and a question on this Who can create this config map because this config map would allow you to deploy your own personal fork So, uh, would you need admin privilege to create that config map? So this this config map would need to live in the same place where this deployment please So you can you can create this deployment in your namespace You don't have to have admin to create a deployment And create the config map what you need admin access is the webhook. Remember like the webhook is like the admin configuration Yeah, uh, right points. It doesn't matter that much. Um, but yeah the the config map So my team will create this as when they get a big cluster like each of us We have so many clusters for testing like we deploy cluster and you're working with your own cluster to do your changes Right to validate that you make the change to debug around Um, and then you have Your that I say like I don't want to use main or maybe you're using the same get repo and you want to use main Uh, you want to use a fork? So I put just fork here. So, uh, the the config map is basically this upstream URL That maybe is used by carlos Um, I want to like change it to my fork Um, and then the branch to my fork branch And I put this get URL Uh, this config mapping there my code Um, not my code, but like some some little example code We'll do the checking saying if if it's an application Uh object, I'm going to check for the the source and and the repo URL And if I have a match, I'm going to reply back. Remember I said like you you give back the api json patches So I'm going to patch the resource mutated Uh with saying no, uh replace the repo URL with my original URL the upstream and replace also the the branch and and if it's a project go ahead and replace it um The the repo source URL right and replace it with my my uh, the original one the original repo URL So that way, uh, you respond back with this uh small um A mission review, but and and that's what we talk about validation. So validation only returned allowed Um and something like this. So this is a mutation weapon I can also pass a patch type and a patch So I'm giving back like hey mutate the argo app and the argo project with my fork URLs Um, and then send it back. Um, so this code can be written in python java Go Different operators written in go that will use a call the go client from Kubernetes So have strong types to the objects that you want to create The objects of the crds But in my case was a simple like json objects that I I'm not doing anything with crds I'm not doing operators and I was able to to do this Uh and patch The the definition that gets into xcd into the cluster But the problem that I then that you will suffer is Uh, what happens if I go notice That that doesn't match the source of truth In the git repo doesn't match the one that is in the cluster. What is the typical thing that argo does? It it reconciles Say it says like no, no, that's not the the the source of truth. I want you to be the the one in git The app goes out of sync Yeah, and it goes out of sync. I was like, ah So, how do I configure argo to say like ignore this small amount of fields, right? Um, so basically what you do is you you configure argo So I have it in the read me here as one of the things that you do um is You configure the argo config map, uh, but in in case of open shift We have an operator called the open shift get ups operator And you will configure a the the instance that operand of argo cd And then use you add this to the spec section in there saying Hey, if if if it's um, I'm going to customize and for applications and projects Please ignore these fields in those like take into account everything but ignore the things that I'm like changing And this way argo will not try to revert you back To the source of truth And and that's that's the way we We deploy we apply this to the argo cd We create the config map to your fork And then we deploy the the deployments, uh, which I have in quake Uh, the image the yaml file Um, and they create the weapon. So this piece is the one that you need like At me because you're going to be affecting your your cluster Um, and then if you want to be affected like this is what me like be more paranoid Um, put a label on it, right? Um, uh to activate it That that was like the main namespace selector, uh, deputy Any questions or so far so good so far so good I was told that this was going to be a validation and mission a discussion so I think it's an open discussion because uh, it's developer experience office hour So we're not limited to any particular agenda. It's a free flowing discussion Especially since we have only nine minutes remaining you mentioned about Cicd best practices with using our city and tecton I'm really intrigued to learn more about that and maybe if we can cover it within nine minutes Maybe point us to that cloud native toolkit Uh site where our audience could could learn more Yeah, so um one thing that I did, uh, so let's go and cover one of those two things. Uh, the first one is um best practices on get ups on this type of thing. So the version that I had um One point and the links. I'm going to be in the show notes is this other one that I have here Uh, the cloud native toolkit. This is a template. Um, so if we do This let's do the github. I'm getting addicted to this um Keep without everywhere. Yeah, the old version onto last night, right? I talk about developers like Having a deadline. Um I had this version that was you seen Uh customize. Um, so I had here. I don't know if this is big enough But this is the kind of the framework that I'm using. Um, this is the main repo that I was talking about get ups Um, I have like the different Profiles where you have like a shared cluster and each cluster would have their own Way of deploying. So I have using customize to apply the the services or like a customized to apply like I want I want, um Grafana refactories, you know the service k native All the things that I want to deploy just on common this and we customize I can deploy The problem that we had was we had so many argo apps That all of them had this section here Source, so this is kind of the thing that was fixing right and the next argo app has this thing here And the next argo app has this thing here So we wrote a little script to mutate all the argo apps with the fork that you were doing And it was kind of like leaving the the github 30. So if you wanted to do a pull request back was like all these changes Uh, we also had the project, right? So we had the project in here with these two things only these two things so, um Last night I did a an update and put into master With a better option of like actually using customized properly because i'm I'm learning all this cognitive as we go. So if you use customize properly, uh, I reduce I reduce it to to this. So now if we take the the single cluster the customization Now i'm using patches in customize. So what i'm doing here is The argo the argo projects no longer have The github url i'm patching. So i'm using a feature in customers called patches With a target. So i'm targeting application argo applications with the label Getups and i'm going to add The repo url and the target revision i'm going to target up app projects and i'm going to If they're infra they get the infra url if their services they get service url and they're up So i'm tagging the the resources with the right thing Um, and for example, if you go now to the instances There's no there's no source url. So Actually the editor will tell you that you have a bad argo But the thing is i want i don't want people to start to continue putting URLs in their argo apps. Basically, they're going to go into the customization File in here at the bottom and this is the new section. So now you go to one file To update the git url under branch. So you put your branch in here saying Main and then in here you put um https github.com slash my fork such github's Uh, this is services right this is tagged services And that's it. It will take all the argo apps when you run customized from argo This is not manually like argo supports customized It would apply this and then all the argo apps will have this same information. So you avoid duplication. You avoid error um And there's a little script in here that you run destruction say to run this and basically it will take The difference all these four repos and put them in the right place for the right argo apps in the argo project. So um, that's that's the thing that i was telling you like i have a better way now To do the kind of the git override with you know proper customized and this is kind of the the new version of the framework um For tecton let's see tecton and argo together. So I have an app in here. I have a repo I have this This example running on it on a cluster. Um, so I have my my repos here and I have uh these um This app so the idea is that I'm going to have uh different deployments. Um um So if I modified so if I modified this file, for example, let me do the dot again Go back to the dot So let's say the app thing One of the app things are going to make a change in their The docker file or the application that they have. So this is a customer details app Um, and I make a change to the docker file And I save it say like remove dmv full bar and and I apply it and um Actually it's already applied. So I apply it. I go to argo Which I have here um, oh no, sorry, uh, I go to The pipeline so you can see here. I have a tecton pipeline that kicked off This pipeline what it does is Builds the image. Maybe those docker lanes like things that tecton are good at Uh, check your yaml file do scanning. I remove the ones we in the toolkit We have a long one, but this one is shorter for demo purposes So it's going to build the image and everything but the trick is this is the ce in a space Nothing runs in here just tecton and the the last step. Um, if we see one of the previous runs uh No, if we see that the previous loans from uh Let's see uh promotes Let me see this one. So I think I guess this one. Let's look at this one So basically what this does is builds an image puts it in a registry There's open shift and then the last step it goes into the The repository of the github's repo That defines the dev image and updates the image Then what's happening with that? So if in a github uh repo gets updated The repo tells argl that there's a change or argl checks every four minutes, but um, then argl will Deploy the new version into dev. So you will see the new version in dev Um, so if we these are all the argl apps, so if we go to applications And we go to dev and we go here Argo will deploy the application in dev And then the next thing that we want to do is running another tecton pipeline when the application is running in in in dev And that tecton pipeline will do a functional test We run something like newman we run like integration test So with the way we do that with argl is we add a pulse sync job And that pulse sync job is located here this job that you see here So I think since we're almost few seconds away from the from the from the end line We can we can resume probably in a in a separate session. I think for the audience. We have all the show notes in the chat and I think most of will be more than happy to invite carlos over For another session But I wanted to thank carlos again for coming to to developer experience office hours And thanks chris for hosting the show and for the audience for tuning in Yeah, thank you chris and there for the invitation Yeah, it's great having you carlos and thank you dawan for bringing them on Up next is the one and only open shift commons briefing We'll be talking about cloud native disaster recovery for stateful workloads So stay tuned because very good will happen to you All right folks, if I don't see you again, stay safe out there and uh catch you next time yep