 This is Michelle DePama, Principal Solutions Architect, and your host of the Data Services Open Data Foundation Office Hour. I am here with Daniel Park. Thanks for coming back. And special guest, Stephen Wilson. Hello, everyone. Hey, so this show's a little different than some of the other things we've talked about. So you guys wanna introduce the concept for today's show? Yeah, so the idea that we had is just sharing a little bit, how can we deploy ODF, no? That Data Foundation using a little bit like the GitOps approach, no? With the help of our GoCD as the tool to do the deployment and just compare a little bit the traditional way of deploying ODF with this new approach that could be using GitOps. And the idea was for Stephen to give a brief introduction into how we have applied GitOps to deploying ODF and doing also data operations. And then we have a demo where we can show a little bit how it works and give an explanation on the things that we have made, not just as an example really of using GitOps with ODF. Yeah, yeah, sure. Talk about the manual deployment method that we've traditionally used as well as will be infrastructure's code deployment method where we're using the GitOps methodology to produce the infrastructure that we're gonna run these workloads on, especially inside of OpenShift. Just recapping, most times the manual deployment involves deploying the ODF operator from operator hub and then we mainly go and we label and sometimes we tank the ODF nodes that we're gonna use. And then we also go and we either use the UI or the command line to add storage cluster to the OpenShift cluster. Well, with the GitOps method, we establish a desired state and then we use OpenShift GitOps or a GitOps controller, some type of controller to take that desired state and make it so in the cluster and we reach our final state is what we want. And that's typically managed by PRs and merges from different branches to main branch that the cluster uses to deploy the infrastructure that it's composed of. What that typically looks like is you usually bootstrap an OpenShift cluster and then we add in OpenShift GitOps more commonly called Argo CD. And we point that to a Git repository. And then from there, the Argo CD instance monitors the applications that we've deployed within the Argo CD deployment. And it monitors those and watches for changes and syncs the cluster with the changes that have been pushed into the Git repo. And management of the cluster from then on is either through pull or push PRs and making the cluster the desired state. And really that's it. And the good thing about that is everything's logged and tracked and it really helps with the security posture of the environment. So quick question. So the motivation for approaching it this way would be you would capture absolutely all the customizations that you've done and that you want done it repeatable. You can do this on any. So you've got it all maintained in your kind of GitOps Argo CD pipeline. You don't have to go back in and do anything later. So if you lose a cluster or you're migrating clusters or whatever you're doing with something fresh or you can go back to and restore this state. Is that correct? Because my initial thought was like, why would we do it this way? And I'm like, well, I guess you've got a million little things that you've actually changed to your ODF installation that you may want to keep track of. So, sorry. Go ahead, Jen. That's true. And then also at the same time from a security perspective, it allows you to see if the cluster is in line with what your desired state. So Argo CD allows you to see that, hey, this cluster is not in sync. And then you ask your questions, why is it not in sync? And then that's when you should have checks to constantly make sure that the cluster is in sync. And then if it's not somebody has done something and then you get security to work at it, to look at it and things like that. So that's another tracking mechanism as well as what you had said earlier about just managing the cluster through GitOps. Yeah, that's what I was going to mention. Because when you have this GitOps approach, really you have a repo, your GitHub repo that is the source of truth for the desired state of the cluster. And it can't be any other way. It has to be as you have it in that repo. And then automatically you are taking advantage of all the auditing and trailing that you have in Git, because you have a pull request where somebody has to say, okay, I want to make a change. And then somebody has to approve it and you're going to have the time that when it was approved, so you have a perfect audit trail of all the changes in your cluster. Now that is really a big advantage. Even as you said, Michel, because maybe there are not so many things that you can touch or configure when you're deploying ODF, but then with data operations, maybe you want to add more OSD, there's no more space to your cluster. You want to enable, I don't know, for example, Radar Skateway or certain changes, know that you want to do, then you can perfectly follow the trail of who has modified it, when and why, you know? How many times have we, something happened in the network or in the cluster and everybody's asking, what do we change? When do we change it? Who did this and who did that? And in the course, everybody's saying, not me, I didn't change anything. Well, that's why we got that get history to say, now you did increase the size of the ODF cluster and it wasn't approved. With a comment, no less. A reason why you did it. Yeah, as long as you got a good reason, you're good to go. Okay. Yeah. So, I think, yeah, go ahead. I was going to say, typically one thing going further with it, typically, when we implemented the solution, there's two really popular methods, using customized and using Helm charts to add these repos and manage the state of the cluster. And Daniel's going to go through and show you an example of those. And also the kind of structure, a recommend structure or one way of doing things for managing the cluster with the ODF cluster with the get-optimistic. Daniel, are you going to share your screen? Do you want to? Yeah. Okay. I will share my screen just a second. The full, this entire screen. Can you see that? Okay. Yep. Coming through Claire. Yes, perfect. Well, I just have a couple of slides, but it's really to explain a little bit what we're trying to achieve in the demo, that we're going to show today and that we're going to go through just to give a little bit of context to get us going. So the idea or the goal that we want to achieve in the demo today is get ODF deploy using the get-ops approach, as we mentioned, with the help of Argo CD that is going to sync the desired state of our ODF deployment. And then we want, as a destination, no, we want to deploy in two different clusters. We want to deploy ODF in a cluster that is production and in a cluster that is development. And we also want to have different configurations in each of these clusters, so that really when we want to apply those different configurations, we want to think or make different configurations is when, as Stephen was mentioning before, you have different ways in which you can render or generate or modify these objects that you want to apply to your cluster, not the desired state. In this case, we can see here in the left that we are going to have our source of truth as we explained, a Git repo, that is the one for Argo CD. And that Git repo, the contents of that Git repo is going to be a HELLM ODF chart, a HELLM application that we're going to go through in a second and I can show you a little bit how it is structured. And once we have our HELLM application, we are going to use Argo CD. And in this case, Argo CD application sets to get automatically deployed the desired state of our ODF deployments in both of our clusters, not the production cluster and the pre-production cluster. Just very briefly to mention, in Argo CD, we have the concept of an application, but an application has a source and a destination, but you can only configure one application per one Argo CD cluster. Now with application sets, the advantage that it has is that with an application set, you can configure several clusters. So you are going to deploy the same application on X number of clusters, using a template that we are also going to check in a moment. So it's very powerful and very easy to use and it's a really good tool to, from one single place, be able to deploy X number of clusters. In this case, we're going to deploy ODF. The last one, the last slide that I have here, and it's just to get you through a little bit what would be the workflow of the demo that we're going to follow, is first, we need to have Argo CD running, as you can imagine. No, we need to have the OpenShift KTOX instance up and running. I have already completed these steps for the sake of time. No, otherwise the demo will be too long, but I'm going to show you anyway, how, what it looks when it's running on the cluster. So we go through that. This is the part that we have here in the left and the steps that I have come through is really deploying the OpenShift KTOX operator. That automatically deploys an Argo CD instance. And when that Argo CD instance is running, we have configured a repo and a project that I will show you now. Now, the repo is going to be our GitHub repo, our source of truth, and a project is an Argo CD project where we define or we set certain rules for our source and destinations. I will show you that in a moment. And then I have also added the production cluster and the pre-production cluster to our Argo CD as clusters. So we can manage them and we can also configure applications on them. I will show you this in a moment. And then what we are going to actually do now during the demo is the actual ODF deployment. And we're going to go through these steps that we have here that I'm going to briefly mention as we go along. So do you have any questions or shall we begin going through the demo? No, dive right in. Yeah, let's jump right on in. Okay. So because we work with Git and Git is our source of truth, we need a Git repo. And I have already cloned the Git repo that we would also leave on the link, on the notes of this recording of this video. But you can see here that I have already cloned the repo and I have here... Can you make your font a little larger? Yeah. The visual paradigm. Thank you. Yes, that's a good point. Is this better or...? Yeah, I like that. Okay. Great. Thank you. Thank you for that. So let me just clear this. So here we have the ODF help chart or help application. We are going to use... And let me just show you a little bit the structure of this folder that is our actual application. Here we have a chart. Jamel file. This is a standard for Helm applications. In the chart you normally set metadata like who's the maintainer of the Helm application. If it maybe has any dependencies, prerequisites, some things like that. Then we have the templates. This is another folder. And inside the templates folder is where you have all of the objects that we want to apply to deploy ODF. So you can see here that, for example, we have the namespace, we have the operator. So all of these are the objects that are going to get applied. And then we have values files. These values files that I'm going to show you just in a moment and we're going to modify are where we specify the parameters that we want to set for each of our clusters. So let me go into one of these value files, for example, for development. Sorry, let's go. Set here ODF. And as I said, this is just an example chart that we have put together for the demo. To give an idea of what you can do, but it doesn't mean that you have to do something exactly like this or maybe even close to this. You can do whatever you like, depending on your necessities. But this is just an example that maybe can get people not going with this kind of example. So with this chart, the thing that we can configure, for example, is our version of ODF. We're going to use ODF 49 because our OCP is still configured in an OCP version 49 that I will show you right now just in a moment. We can also configure the requests and the limits for all of our ODF services that we're going to leave them as they are. We can also configure here if we want to have Radius Gateway enabled or disabled, Multi-Cloud Gateway, our Chef Tools. So let's, for example, because we have to make some changes now, let's, for example, enable the Chef Tools. Sorry, let's set this to true. And the Chef Tools is just a small pod that gets deployed. When ODF gets deployed, you get this small Chef Toolbox that you can exit into and it will give your shell into Chef that is really nice for troubleshooting issues. So let's say that we want to have that in development deployed. Also, in this chart that we have written for ODF, if you are running in AWS, you can select if you want to have dedicated ODF nodes. So if you set this to true, that we're going to do it in production, you will get worker nodes deployed for you. So using a matching set, an OCP matching set, you will get three worker nodes deployed and they're going to be used as dedicated worker nodes for ODF. Now, when this is enabled, let's say that for pre-production, we want to save resources. So we're just going to use our standard worker nodes that we have running and we don't want to enable dedicated nodes. And here we have the configuration of the storage cluster. So the number of OSDs, if encryption is going to be enabled, the number of replicas, this is the storage class that we're going to use. So this is a storage class from which our OSDs and our persistent volumes are going to be created from and also the disk size. Let's say that we are running a development cluster. We want to save resources and we're going to use only 512 gigs for our OSDs, not as a size. So that would be the configuration that we have for our development cluster. And let's go a moment into production. Can I ask you a quick question? So are there, I mean, that's from what I remember, that pretty much looks like everything you do on install when you go through the operator kind of UI path. Is there anything missing? Are there any limitations to doing it this way? No, not really. That's a good question. There's no really limitation. It depends on how much effort you put into your hands, let's say. So this is something that we have for the example. It's quite, you have quite a lot of options. Maybe there are some things missing here, like for example, deploying with a logical storage operator, deploying with LSO that could be added. But you can get as creative as you like and everything is going to work so that you are not going to miss anything. The only thing is depending on the effort that you put on your Helm chart, to be as rich as possible. Yeah, and I would say that's the benefit that Helm has over the customized version, is that within the Helm chart you're able to make decisions based on what the current cluster looks like, as well as decisions that you're able to make through the variables that you instantiated, like Daniel said, the decision could be if you're on AWS, then use these variables, if you're on vSphere, use these variables, and all of that is in the Helm chart. And Michelle, there's also a great benefit also that the CLI has over the UI, and that is there are some parameters that we're able to change on the CLI that you're not able to change within the UI. Right, right. The toolbox, I think maybe made my information all the things like the toolbox are always CLI, right, and what have you. The toolbox, the different sizes of the disk that you use, you can change that from the CLI, whereas in the UI you have the three different options that you have, but yeah, there's a lot of changes that you can make just like Daniel's showing here, the memory request and the limits that we have, you can change that on the CLI, and then also some of the components that you do not want to deploy like he had enabled is true for some things like he's showing here now the multi-cloud gateway being enabled or not enabled, those are things that you would change on the command line in your storage cluster CR that eventually gets deployed by the Helm simply engine and usually what that looks like at the end of the day is simply an OC apply. Okay, so quick question around this and I don't want to get too much into it because I know you have good stuff to show us. So let's say is there a convenient way to dump this state so that I can then put it in getups and follow this? Is there an easy way to do that or are you going to end up going through the command line looking at this and looking at that and those parameters? Is there any transition path there? For you know what I mean? If you were looking at something you didn't know what was going on because you didn't set it up so you want to know what's the fast path to this? Yeah, you have like I think ways of going around that. No, you could go and and look the actual chart that we are looking at here and then you have all the options and you have the latest version because you're going to go to get and you're going to get the latest commit that is available and you can also render the file. So I will show you that afterwards but what we are doing here is we are setting a certain parameter certain values that then they're going to be pushed into the actual object to deploy to deploy multi-cloud gate whatever. So even if someone gets a little bit lost with this parameters there's a way that you can do a dry run and you can render all of the parameters and you can take a look and see what they actually look like and then you could take a look at that and in the other hand we're also going to show you afterwards you can take a look on the different parameters and you could also check from there that we can see in a moment also. Okay, awesome. And also OC Describe is your best friend. I was just thinking because the ODF it has a lot of parts you have to get into all the parts so continue please. So I went into the production I will just at the moment we did pre-production so we have let's say that we want to apply for our development cluster already now we are going to modify production now that we want to let's say change certain things like for example let's say that because it's in production we have more resources and we want to set the limit for our OSDs higher because they're going to give us better performance not just as an example let's say again let's enable the test tools that it's always a good idea and then here I wanted to use the dedicated nodes again this what it's going to do is going to get a matching set and it's going to deploy three nodes for us in AWS and it's going to label them and it's going to get them ready for use with ODF the only thing that I need to do here just a moment is get the cluster ID from the cluster so let's just run this command are you just going to call it yeah here we go okay so we go back into our file and I just have to change this cluster ID because it needs it for the for the matching set to work the region and the AMA file it's exactly the same so we don't need to change that let's say that encryption is true we can leave that set and as you can see we are using bigger drives because we are working in production and let's say that we need more space here okay so we have modified our files let me clear this for example if we do a get status we can see what has been modified for the moment now that we have modified these two files the development and the production cluster so now what we need to do is actually push them to our our get repo so I'm going to this is a standard get commands now I'm going to do where we can set a commit with a comment like you just mentioned before where we say values or obf for example and now let's push this to our upstream repo so this is going into into get hub and into the upstream repo okay so we have made our modifications they are in the upstream now the next thing that we want to do it's actually work with Argo and create the Argo CD applications now with this Helm chart that we have just let me show you a moment where are we running so you can see here I'll see get cluster version so as you can see this is like a 49 OCP cluster and we have an instance of Argo CD already deployed so the only thing that has happened here is that the operator has been deployed and it has configured an Argo CD instance as you can see here we have a controller for the application and also for the application set so we have Argo CD running and then I have also I'm going to use for this part the Argo CD CLI but you can we are also going to see how you can do it on the UI but just let me show you this Argo CD that we have created a project this as we mentioned before when we are going through the slides an Argo CD project is not an open shift project this is a part of Argo CD and really what you define in a project is like you are trying to set limits on the on the destinations that the destinations can be on which clusters on which namespaces I can deploy an application and also the sources that are available or that are permitted for this project and in this case you can see that we are using our github repo okay so that's the project that is ready we can check that we don't have any applications deployed for at the moment and the other thing that we did is add the clusters so as I mentioned before you can see here that we have the development cluster where we want to deploy ODF and we have the production cluster with the URL not to the API already configured into Argo CD next what we need to do is use an application set to actually get our applications no Argo CD applications deployed for that normally when we run this with automation it will normally all of it get deployed but I wanted to automatically but I wanted to run it manually so you can also take a look at it looks like no so going into what we were mentioning before when we said that we could render the files no this is the command with Helm that you can use to render all the variables that we have in values into the actual objects that we have inside the templates so what I'm going to do right now is render our Argo CD and you're going to be able to see it because this really generates files for us in this folder what we're going to need because there's some strange space there so let's go manually through it here it is and really what I wanted to show is the application set that gets created no so this is the our application set and this is the application set already rendered for us no so when you were mentioning before Michelle if there's a way to see what the state looks like you can use the Helm template command that I just run here and you will get the render version as I said normally this will be done also through Argo CD the creation of the application set but I wanted to just run it manually so we can do an OC create and see it will be instantiated just like regular template in general templates okay here with application sets how more or less the idea behind application sets is that you have a generator and you have different types of generators here we're using the the simplest one the easiest one that is really a list of elements that we want to get deployed or defined in our template but I will go through that now in a moment but if you take a look we have a list of elements and what we have in our list is the clusters to which we want to deploy our application so this is just a list where we would enumerate all of the clusters where we want to deploy as you can see here we have cluster pro and cluster def that are the same clusters that we have configured already in Argo CD as we mentioned before so what you have is a generator this list of elements and then that list of elements is going to be applied on this template so really what is going to go through is in a loop and it's going to apply wherever you have the cluster variable is going to apply what is defined in your list of elements so really here we are generating Argo CD applications on the fly so let's say that from the template onwards we are generating Argo CD applications on the fly and what's an Argo CD application as we explained before is a source of truth in kit and a destination where Argo CD has to automatically sink the state and as you can see here I need to jump real quick how do you want to handle it will you be able to come back I will you can jump off and then come back when you're ready okay same place for you perfect yeah so what I was mentioning also you have the source really and a destination the source is where is our github repo what type of application that we are using so as you can see here we have source the kind of application that we are using is Helm what is going to be our value files that as you saw before our value files is really where we are configuring all of the parameters that we want to use where you can see here that we have values and then the variable is going to be applied for each of the ones that we have in this list that if you remember and we take a look here I can just very quickly show you here are go github f so as you can see here the value files that we have here is going to convert it into this one and this one so that's really why we have that defined in so so quick question so just make sure that for everyone listening to get this straight so we are using Helm to then we're going to deploy a Helm chart which is going to configure our go CD to have a project that's going to have a destination of whatever we put in the list there and then go and apply all of the values that we like to hold all of this information we have that in a github repository yep that's all right just checking because I'm not a github person yet so I just want to make sure I have like the right the whole workflow then I'm like okay so go on yeah you can do this in many different ways because there are let's say that there are there are people that use different repos one for our go CD and the management the same repo but then use folders you can have everything in the same repo but then use branches for different things so there are many ways of going around this in this case we're using the same repo but we're using different paths so we have a path that is ODF where we have our Helm chart our ODF Helm chart but then we have another path that is bootstrap where we have all the information about our go CD from the other because by the path but there's not there's not a good way of doing this now everyone can choose how they prefer to go around this and it's perfectly fine so as we were mentioning we have a source that is our source of truth with our repo our path as you can see here and also if we have a branch that is the target revision in our case we also have the application set branch but this is our the state of the application that we want to define and all of this is in the source and then we have destination that is what you were just mentioning before Michelle that is where do you want this apply you have to tell me where do you want to apply this and as you can see here we are going to go through the list of elements and we are going to change the destination name to cluster there one in the development cluster the only thing is that because it has specific values the the advantage that we are going to get is that we are going to get different configurations for each of our clusters okay so let's just do a standard let me make sure that we are in the right project in the right name space open shift go the application set and what is going to go what is going to happen here if we maybe to take a look we can check as you can see we have a controller for applications and a controller for application set so we can take a look at the logs for the application set and see if actually did this job and as you can see I just wanted to mention in case people have issues when trying to debug this that we talk to the application set controller you get quite a lot of information as an error here everything went fine and as you can see we have a cluster pro ODF application deployed and a cluster ODF I'm going to show it with the actual algo cd case command where you're going to see it much better so what happened really with our application set basically it deployed two applications one is called def ODF and the other one pro ODF the big difference that it has is that it's going to be deployed to two different clusters not to different destinations that I'm going to show you just in a moment the other thing that I wanted to mention is as you can see the state of the application is out of sync why is it out of sync because we have a state ready and waiting for us on a helm chart and it hasn't yet been applied to our destination clusters so our destination cluster and the source of truth our github repo are out of sync you can configure this to as soon as the application gets created you can configure it to start syncing but here maybe because in production for caution you want to leave this disabled and do it manually maybe not automatically set here we have it we have to press the button automatically so I could also show you the UI when it starts the sync so let's move into the UI now I will make it bigger let me get the password just a moment here what I'm doing is I'm extracting the secret from the github cluster secret that is inside the namespace and I'm going to start with the open shift github you have the deck server that is able to authenticate using open shift user so we could perfectly login via open shift and use a user from open shift from our actual ocp cluster let me make this bigger otherwise is that better can you see that okay so again this is just the same view that we had in the CLI we have our dev application and our production application as you can see the big difference the destination one is dev and the other one is pro so we have our two clusters let's go ahead and sync this one thing that I wanted to show you before I started syncing is what we were speaking before about the audit trail as you can see here is mentioning that it's out of sync and it's saying the source of truth is this commit from our github repo if I press here it will take us directly to our github repo and we can perfectly see that it was me if something goes wrong it was my fault that 14 minutes ago I made two changes to the development cluster values file and to the production cluster and we can also see what we actually change here in this part so as you can see it's great also for seeing who made changes when and what did they change okay so let's give this a sync here we have syncing actually says go apply these differences right that you're actually kicking it off okay and you mentioned that could be done automatically but in this case because you want to show us we're doing it manually okay that's yeah that's completely right this the step that I'm doing now that I'm pressing synchronize you can do you can set that when you're defining the application to be automatic and then you wouldn't have to do this step so as soon as the application gets created or as soon there is a change on your github repo it's going to be applied to your application no okay and once I fire it off as you can see here where it's really starting the the deployment no the first thing is going to go and start the ODF operator and it's going to go through all the steps no that you will normally go to deploy to deploy so just I just remembered from before oh maybe it was it has to grow is that the production cluster yeah that's a good that's a really good point yeah I wanted to mention that so it's a really good question and as you can see here that's the main difference no that you can see that we have here the matching sets objects that we didn't have in in development no so we have that ready to to get deployed here so we have that listed as a dependency somewhere like wait until these are I don't remember okay that's another that's another really good question and you're really we're going to speak about that because that's a little bit and if we or a little bit complicated way of doing things and I will go let me sink it and once we start sinking because it takes a while we can speak about about that because it's a really good question what you just mentioned I wanted to take a little bit deeper but as you said you can see here that the first thing that needs to get deployed also our machines know that are going to be the workers that we're going to use for our dedicated ODF workers so going into the topic that you just brought up that is a great topic how do we orchestrate all of this because really if we have a small application where you have I don't know a deployment and some ports and it's something easy to manage okay that's fine you really don't have to go into all of these if you're working with something complex like it can be like ODF for example where you have an operator we're also deploying infrastructure as you can see here that we're deploying new machines before deploying our actual ODF so there are a lot of steps that need to follow an order otherwise they're not going to work and let's say that Argo CD provided us with tools to work with this so you can give an order to which we apply and I'm going to show you that straight away how we do that while that gets deployed so we have this is the same repo that we were working for before now and we have ODF that I mentioned before and if we go just show you this let me just do it here so you can see a little bit better so what we have is sync waves they're called sync waves and it's just a way inside Argo CD to give an order on how our objects are applied so how it really goes is that is an annotation that you set in each of your objects and then you give it an order no the negative numbers are the ones that go first and then we go onwards so for example here you can see that with minus one we have the matching sets no they have to go first then we also see for example with number zero that we have our operator so we have the the actual ODF operator that I was looking at the subscription to get the operator deployed then on an extra step for example we will get actually our storage cluster now that this is when Seth gets deployed so as you can see there is a way to give an order to how you want all of these objects applied so the order is kept in sync wave in AWS this is how I make a I'm just being sad that was I know you showed it already but I just want to no I didn't it was brought up by Stephen that he mentioned that and let me show you an example because I just let me show you where we have an if we deploy with a gateway maybe here I have an if yeah so just as an example because I'm not sure it because this hand chart that I have right here is I yes I know where I'm going to show you but as you say how you work with these templates is you have your normal object no like your this is a normal Kubernetes or OCP object and then like you will do with your ginger templates you have all kind of ifs when else so you can modify or do all the things that you could need to give it a little bit of logic inside the templates no okay so that but that's your helm templating part making those decisions helping you okay I'm just trying to separate it in my mind like so for decisions you're going to work the helm templates and then for order you've got the sink wave piece and okay all right okay yeah that's perfectly done I'm using the actual object and as you can see what I'm really have here no in my helm chart is if in our values file we have dedicated enable then go ahead and deploy the matching sets no so it's really how this is this is done at the helm level so Argo CD doesn't take part in this Argo CD the only thing that it does is that it's able to render all of these values that to understand them and then do the rendering to actually apply the object in the end but as you can see here for example the cluster ID that we got no and that we set on the variable is also set here so it's really kind of a ginger template approach that you will follow okay thanks and then you're going through stuff that I think I've done manually more times than I can count you know it is because I because as you know we have like our HPDS here in Red Hat where you get an AWS lab for example but you get it for four days five days and then you have to give it back and deploy another one and in the end you're deploying all the time around so that's also what I use I normally use this to deploy ODF and I don't have to touch anything on you get as you said no your matching sets you get your labels everything in place also a good fit for that as well the last thing that I wanted to mention is even if we have sync waves everything is not as pretty as it seems with sync waves because let's say that Argo CD is going to give the object as a health okay once it has been applied and it has responded with this health check of okay no so for example you have health checks where the board is running but for different let's say CRD you know that you want to use like for example the matching sets let's say that Argo CD once it has applied and the matching set is created it's going to say okay I have finished moving to the next into the next step but your machines are still getting deployed they're being provisioned they have to be added to your OCP cluster so there are still some steps that is missing and what I have done that is not pretty to get around that is using what we call called like wait-bots so let me show you one just a second so here for example so this is a wait-bots that make sure that the ODF operator is running and what I do is using sync waves I put the wait-bot in the middle of let's say the deployment of the operator and the deployment of the cluster we want to make sure that the operator is ready and really what I have here is an OC wait that is checking that all of the bots that are needed for the ODF operator are up and running before continuing into the next step so that's why we're using wait-bots as I say maybe this is not the prettiest thing to do there are other ways to work around this for example you can use custom ROCD health checks where you could say the actual health check and say for this object don't move forward until this and this condition is met but that's a little bit more complicated you can perfectly do it but it's using Lua scripting that is quite complex in that sense so that's another way of going around it in this case we're using wait-bots and the other thing that maybe we could do that I haven't explored yet is really using like a pipeline or looking at ROCD workflows where you could say okay first in the first step deploy all my infrastructure deploy all my machine sets now when that's finished send an event send a notification or maybe if it's a pipeline just go into the next step and then deploy the ODF operator and go like that and give it an order using maybe decked on pipelines for example or something like that but I haven't gone or explored that deeply but I would like to check if it will fit no to use it like that it's interesting okay so that's definitely going to be I could see using something like that when you've got like big pieces of infrastructure that you actually have to deploy is there do you have wait-pods down the line anywhere is it just for the machine set I have two of them said no I have one for the machine set that we can see here no for machines let me just make this a little bit so we can see it better that's not good okay I have this one for machines and I have this one for the operator those are the only two that I have no because afterwards afterwards once you have the ODF operator installed then everything else you have already your CRDs and everything else can be deployed at the same time we don't have to actually have to wait for anything else okay I will just bring it up just in case somebody also has this issue it's this is going a little bit deeper than more deeper than we wanted no but I will bring it up anyway let's say that Pargo CD when you do the first sync we press the button it's going to check all of your objects and it's going to do like a dry run and see if it can actually apply all of the things that you do before it actually starts syncing and if it can't do something because maybe an object doesn't exist it's going to give you an error and it won't move forward okay and that gives you suddenly you have an issue because let's say that when you deploy the ODF operator that is going to create the custom resources for you but if I'm doing the sync before I have a plate operator those custom resources don't exist and I'm going to get an error and then when this option comes in when you say a skip dry run it's going to do so really you're telling him if this resource is not available just skip it during the testing at the beginning and continue with the sync so this is what we are mentioning here okay so let's take a look and see how this is the production this is cluster pro right and then okay yeah where are we let's let's check on the development that should go a little bit faster as you can see here there's a good thing that we have right now that it says that it has been able to sync what it means when it has been able to sync is okay I have got the desired state that I have in my github repo I have now that state on the destination cluster no it's done sync here means finished right there's no more work for it to do okay the only thing is that for him he has finished let's say you are still waiting for all these to go up no or something like that he has already applied that object so for him he's synced and everything is ready but we can check very quickly and see how the development cluster is going so let's do this and let's let's set cluster it has to be open shift oh yeah so this one has finished now our pre-production cluster created successfully it's okay okay yeah let me just make sure that we did things right and we have a 512 gig oh I always forget about the main space so as you can see here for the pre-production we have 512 no that is what we selected so everything is looking good let's go into production this one takes a little bit longer because we're using the matching sets that we mentioned but let's see how it's going so it's configuring safe months no so it's still it's still working let's just a pod so we can see how it's being deployed okay so it's looking good it's doing its work slowly but steadily it's going along so it's working with monitors after monitors we have the OSDs and then it's going to finish I wanted to show you I don't know how we're going with time because I wanted to show you if we had time how to do let's say a day-to-operation that is for example adding rados gateway can you do it in like 8 minutes to be quick yeah yes I think I think I can let's try anyway if we we will leave it running anyway so let's say that we want to make a change again to production because for example we have a new requirement that we need to have powerful signals and we want to deploy rados gateway that the rados gateway is sets S3 compatible endpoint no and we also want to use SSL because I see that there's production and also let's go ahead and say Okay I'm running out of space let's add OSBs to our production cluster so we have to make these changes again this is on my local repo I have to get that in to our up a stream repository and also I have to again add commit, let's say here adding RGW just so we do it quickly and we do a push. Okay, so this is a new push, we are going to have a new commit, a new version and if we go into the production cluster, we can see this takes a little bit of time, we can hit refresh or if not, it will actually check by himself that there's something different, but what has happened here? I have made a change in my GitHub repo and then the cluster is saying, okay, I'm out of sync, there's something different going on here and as you can see the new object that is out of sync is for example our safe object storage that we just mentioned before. So now if I press sync, it's going to go ahead and deploy our radius gateway and it's also going to deploy our new OSDs, not that we selected. The only thing is that before I actually apply this change, we need for the cluster to finish, so let me make it work. Yeah, I can ask you if we can do that. Okay. Yeah, so this is production. No, let me just check that we're in production. Yeah, so everything is running fine. Let me just very quickly. I'm going to go into our safe toolbox now. So we can actually take a look. Well, let me run the sync and then we can go into the toolbox. So this, let's fire this. So it starts deploying radius gateway and our OSDs and now let's go into the toolbox again. Open G. All right. Okay, so at the moment, we have a new, the cluster now, we can see that the health is okay. And for the time being, radius gateway hasn't been applied here, no? So we can imagine that it's here it goes. So we can see now the difference that now we have radius gateway deployed and it would be available now for object storage use. This was before. Okay. Did you add an OSD? What else? You did something else too, right? The radius gateway and something. Yeah, I didn't, as you mentioned, I didn't know it's these. No, so let's say that we were running out of space and that we needed more more OSDs. I really, adding more OSDs is adding more raw space to our cluster. No, so as you can see here, what's going on is that currently we have three OSDs. So in each of our hosts, we have one OSD. But here you see that with the change that we applied, no, the stuff is working or let's say that Rook or ODF is working on adding three new OSDs. This, the only thing that this takes a little bit longer. Let me just check with the status. Okay. But your dependency set up in sync ways still have to be obeyed, right? No matter if you change something that then everything gets followed. Okay. Yeah, yeah, yeah, that's right. Each time that you press sync, it will go through the full tree to check everything. So the only thing that the things that are in sync won't be modified, but it always follows the full tree of the wave sync that you just mentioned. Okay. And how does the, what is the Argo CD, the dashboard show, oh wait, they're up. Now they're done already, right? Okay. I was just wondering like it shows a different icon or something in the UI when something's progressing, right? Yeah, you can see that it's blue. So for example, we were a little bit late, but when something is out of sync, it goes yellow. When something is, when it's working, knowing and getting something ready, you can see that the lines are blue. When something is in an error state, it goes red, no? So when you, if you get an error that you can also perfectly get an error, it will go red. So with the colors you have here, no? In this course, you can see what's going on. There's many nice things in the UI, no? We don't have time to explore, but for example, you can have a look at the location of all of the boards that have been deployed or the objects in the different worker nodes. So there's some really cool things to check here. You can also filter to whatever you want to actually see. No, because there are loads of objects, but maybe you want to see X, Y and Z that are more important to you. You can also go through that. But yes, as you mentioned, our OSPs are up and ready. We can see that they are ready to use. So we would already have more space for our developers to use and everything will be fine and ready. So that's more or less what we had and more or less on time, no? Yeah, perfect. So that's amazing, actually. I'm really impressed. Like, and certainly how you would manage a real production cluster and make your life easier. So are there some links that I need to get so I can drop them in the description of this video? Maybe templates and things like that? Yeah, I think I will send you the repo that I'm using. Again, this repo is just an example. It doesn't mean that people have to follow it, no, as it is just, but maybe to get it started is a good point. So we can certainly share, no, in the link, the GitHub repo that will get you started. All right. OK, so it's the top of the hour. I just thank you so much for coming back and I hope to have you again. We'll come up with something new. And I want to thank Stephen Wilson if he couldn't join us again, but that's OK. He'll forget all this going. And I'll drop the links in the description and take any questions from chat when this goes live. Thanks so much. Thank you. Thanks so much, Jen. Bye, bye. Bye.