 Good morning. Good afternoon. Good evening and welcome to another episode of get ops guide to the galaxy I am Chris short host of redhead live streaming and showrunner. I am joined by Joshua Packer and Christian Hernandez the get ops guy and we're gonna be talking about multi-cluster stuff today Christian Yeah, yeah, I'm actually Really excited about this. I've been meaning to get the ACM team on On the stream and actually this happened completely by accident Josh and I are wearing the same shirt We're so We're so tightly knit here. No, I'm really excited to get to get Joshua From the ACM team down here. Josh. Can you give kind of a brief introduction? Introduction of you know what I mean who you are what you do at red hat and you know what your role is with the ACM team Absolutely, so ACM is advanced cluster management or open cluster management if you're looking in the community for us We are all about cluster management at fleet scale and under that purview. I work on own develop Cohesively connect and and lead the teams that do both our cluster provisioning and life-cycling So everything from creating clusters to destroying clusters to all the pieces that go in between and then the Application life cycle which is where the get ops comes in and that allows me not only to Sort of work with applications and work with you know, what goes out into your fleet But get into that space of you know, how do we define what the infrastructure looks like this? We'll talk about it many times the infrastructure as code the single source of truth And so, you know being attached to these different pillars within the the product sort of allows a lot of this What we'll see today these sort of cross referencing where we can use the single source of truth get ops for managing everything in your fleet That's awesome. Yeah, yeah, it's what I what I like to call Josh's what I like to call The person I ask anytime I have questions about ACM. I'm the go-to guy Yeah, the go-to guy is these the one I asked questions to is what I what I see it here And you might have seen Scott on another show called advanced cluster management presents on the channel So I will drop that playlist in chat if you're curious about the topic That's right. Right. So if you catch definitely catch that stream if you want to know all things ACM I invited Josh really to To the stream because we you know, we hear on the show. We talk a lot about get ops, right? We talk a lot about application delivery and get ops as a practice and you know, a lot of the time so, you know quick quick story This is an old story because this is back when we can travel And this is back like around 2000 2018 where I actually did Red Hat user group up in is in Minnesota and You know, I was talking about Argo CD and the the you know, one of the questions is That I got we know that I used to get a lot was it's like, okay, you know, like this this is great Like, you know, but but what about like my day zero stuff like how does How do I provision my clusters right in a declarative way? And you know, I think You know get ops as a practice, right? We all love get ops get ops as a practice but I think ACM brings a very interesting aspect of You know going beyond just like get ops and having multi cluster management, right fleet management and How to manage that with get ops, right? Because get ops as a practice is great But you know now now we're talking going up a level right going up into like, okay Like, you know, I'm doing the get ops practices. I have, you know, thousands of clusters, you know, you know spread across, you know multiple environments, how do I manage that and You know, I think You know having a tool like ACM having an offering like, you know Something that manages the cluster with your get ops practices is a very very powerful thing So now you have an end to end right now you have an end to end of a of you know, I provision my cluster You know, I manage it with get ops and I then I control a life cycle that that cluster I can destroy it I can you know upgraded I can apply policies to it So don't want to see too much thunder, but I think it's very important to talk about You know, when we're thinking about get ops, we're not only thinking about like the application delivery But the whole infrastructure platform your whole end-to-end solution as a whole so So yeah, so, you know, I'd love to get your thoughts on Josh on like the whole end-to-end aspect of Get ops and in multi-cluster management Absolutely. Well, so I'm gonna touch back though Just on one thing you mentioned and it was a question we get all the time And I'm sure it's a question some of the viewers on this have as well is red hat and we're wearing the shirts That's a open-shift get open shift get ops is red hats Productized version of the Argo piece and so the question in advanced cluster management We always get is advanced cluster management had a way of provisioning apps and doing get off and helm You know, which do I use do I use Argo? Do I use the ACM subscriptions? You know, how do I make the choice and you know, the answer used to be well Depended what you were doing that answer is still the same. It depends on what you're doing But in both cases, we want you to use both We want you to use them together and what we're gonna show you today is how ACM and Argo are like this They're fully integrated the ACM brings the fleet management that was sort of lacking from the Argo space Argo is very much the, you know contained for the the cluster that is working on or if you've gone out And you have a bunch of clusters. You still have to bring those in ACM We're gonna she see brings all those pieces together and gives you that that visibility Into Argo that one number one we never had before but number two allows you to operate Argo and Through open shift get ops at scale across the fleet. So without further ado, maybe I'll grab a screen share and we'll we'll get rolling so Hopefully it pops up what we're looking at here is the ACM. This is the 2.3 release It literally went out the door like three days ago. I think it was fresh fresh. Yeah, yeah exactly new new here Right out the pipeline. This is exact as new as it gets. So what we're gonna do is we trough the vine Absolutely new no And so, you know as Christian mentioned, you know doing everything from day zero on through day infinity with your cluster I if those of you familiar with ACM I just navigated in from the welcome screen down to applications And so what we see here you see some tight some labels that say Argo. We see some that don't have it I'm not gonna get into that just yet What I want a target on here is there's this application that's deployed in ACM called infrastructure build out And so the backstory in this is that I got an open shift cluster I ran an open shift install to be honest internally I just requested one and it came up but you deploy an open shift cluster you go to hub So you go into open shifts operator hub you say I want ACM You hit the install button in seven minutes. You have an ACM cluster at this point. I'm at this stage I have no applications. I have no other clusters I want to create my first application in ACM and so I go through I click the create and What I do is I choose get and I pick my infrastructure get repo and in this case It's labeled get ops fleet samples So I pick my repo that has all the definitions for my security for my Configuration for the clusters. I want to provision for the applications that need to go out to it So all that with just this single Infrastructure build out and let's take a look quickly in git here. We'll get it to pop out I was able to build out a bunch of policies provision some clusters Apply policy to those clusters in this case We actually go out and we deploy open shift get offs Which is our go on all of those clusters as well. So what does that look like? So we see there's a couple of different directories The one we're going to be interested in seed and we were we were having a little taking a little liberty with naming So this is the directory that seeds all of the infrastructure pieces for your environment But here it starts to get pretty simple anything under common is something is Configuration that is going to be common to all of my clusters. So one example is the ACM policies So things like I wanted open shift get ops or Argo on all of my clusters So I have a policy that's defined in git that does that I wanted to enable at CD encryption across all of my open shift clusters So again, I have a policy and I'll click in one It's a bunch of yaml but again if you've seen any of the ACM stuff or you can go back and watch the twitch channels on it when I go into the UI and say Generate a policy. I choose all the settings I want in the UI I click that yaml button and I've got that yaml that I can just paste directly into git and Then that becomes part of my source of truth part of my configuration that it gets applied by ACM to my entire fleet But so again, you've got a split up of these seeds We've got common configuration and policies where we've got those specific to our ACM hub So configuration apps special cron jobs again, because this is a bit of a Demonstration cluster to show a little bit of everything. We've got specialized cron jobs that in the evening put the managed clusters So portions of my fleet to sleep so that I'm not paying AWS fee GCP fees Azur fees Down those dev environments when you're eggs exactly when we have a lot We were just talking about a long weekend this weekend It's the same idea as we are able to turn those things off And you know if it doesn't happen from the GitOps it can always be Overridden from the UI and so again in 2.4 We finally bring and I'm diverging here from GitOps for just a second I promise but I selected a cluster and you see we have the hibernate and resume now as Part of the user interface that you have access to to To control power states on your provision clusters again but so back to the the original discussion is I created a single subscription it brought in all the configurations and It brought in all of these other applications. So it deployed ACM we use this term called subscription is our version of an application in Argo that is able to go to Git and Pull in the hello world app Here again, it did a mortgage app and we pushed it to one core one system But it also again, I remember I mentioned it deployed GitOps everywhere So I have our GitOps open shift GitOps, which is Argo So all of my managed clusters have a copy of Argo on it So if I click into this so number one ACM knows hey, there's an Argo app So not only does it see the ACM mortgage app the hello world app, but it finds apps that are deployed by Argo Argo introduced with from the Red Hat team has this concept of an application set All an application set is is grouping a single application to many clusters similar concept that ACM has with our Application which is why we're able to display it this way So we click into it and so we see we've got this standard topology We have some warning symbols and we'll talk about that in a second, but so this is an Argo app. It's deployed by Argo It's deployed with an application set from Argo. So if we click on this it'll just take a second to refresh hopefully and We get the application set we can see I know it might be a little small. Let me It's not zooming in but you can see the Argo project reference there So again ACM building that topology But this is strictly for all intents and purposes and application coming out of Argo Now I just want to show for comparison purposes I'm gonna click on the mortgage app and again we can see exact same topology Everything looks the same except that little reference link I had for application set that's missing now Let's take it a step further. So we go back. We're gonna click into the engine example again We have the application so we can take a look here Again, we see all of the different clusters where it's deployed and we also see why there's a warning There is a problem with the deployment on a single cluster But if I'm going to click this you can see it says launch the Argo editor It's gonna push me out and this is the actual application definition in Argo that is running Running on that same OpenShift hub. So all that data that Argo has about the applications It's brought in to me about the deployments. So let's hop back for one second. So again That's actually pretty cool for like so you can imagine an administrator You know Administrating, you know, you know fleet of clusters and the information of what the the developers are doing right there See I CD process the the the deployment process Right in Argo CD bubbles up to the admin so you can actually see kind of like like a like a single pane of glass So, you know, so as an admin you don't lose that information, right as your developers are doing things with Argo CD All that stuff is coming up. That's and and exactly and and this is the piece We always push about ACM is that this topology looks the same Whether it's the seven clusters here or a hundred clusters or a thousand clusters But you'll see where you're having a problem and so let's click it We click on it. It takes us to the Argo and this is the one that is having the problem But let's the problem exactly and actually the problem is is I'm trying to deploy it on my hub cluster in a namespace that I don't have permission to but I needed an I wanted an example to be able to show how we can dig down into it But let me just quickly show a comparison. So this is that Same I pushed enter too quick This is that same list and so it's it's readable at seven so you can see there is the problem But then let's say we get the 10 the page is full. Let's say we get to 100 You're now scrolling through now. Yeah, you can start to play with synced and unsynced But again, the point is is that ACM brings that fleet view when you're talking about hundreds of clusters even 50 cluster even 20 clusters You know, you're not scanning through that pages and as Christian said It's that that much easier to just look at it and see where the problem now This was an entire deployment problem where it didn't deploy any of the pieces to the cluster But imagine it was just a route. You'll see just that one little icon You'll know it's a route problem. You'll click on it and then list on the right You're gonna be able to tell exactly which cluster it has the problem on and then click that and you get the Argo editor Same thing as ACM the last piece though I want to show here is we've also demonstrated this for ACM before is an ACM app You can click on this without leaving ACM at all to see the yaml and the logs that go with it That still applies to Argo applications as well So this is this we're still in it opened a new tab. That's why the screen flash, but we're This is the yaml definition for that deployment there And so you can scroll down and take a look at the status Etc and see if there's a problem as well as you can see the real live logs from the system as well And that's without leaving the pane of glass of ACM You didn't have to jump down into Argo You get that information immediately and then if you need to do something deeper Then you click on that application icon you choose Take me to Argo and boom you're in there And so this is that scenario where you've got Argo running on the hub And it's deploying applications to all of your well two sets of your managed cluster And so the question is well, how do we make that easier? And one of the ways is is if you've ever done it before you need to Run a run a command on the Argo CLI to import clusters into Argo ACM Doubt in 2.4 does that for you automatically in 2.3. We had a we had a slightly different approach We would import them as well But you were limited to If you had only a single if you had more than one Argo running it would only work with one Argo on your hub But so now we use and we've talked about this on some of the other streams But we have this placement concept, which is really I'll keep it civil It's just label matching is if your cluster has the right label Then ACM will import that cluster into a given Argo And so you can have two Argos for example You can have one for configuration And that could get all clusters you create doesn't like all openshift clusters you create Doesn't matter whether it was for production for tests or whatever It goes to you know, that one gets imported to the Argo that is for configuration But you could have two other Argos one for development and one for production And in those cases only the ones with the usage label production For example go to the production Argo and all that happens seamlessly when you provision You don't have to you have to do anything special. There's a resource we introduced Here let me pull it up quickly There's a resource we introduced that allows you to specify that So if I go back here to the hub under configuration We see this get ops cluster Kind this is just that it You specify the Argo namespace where the Argo lives and you specify again I said this placement rule which if we scroll down here is just the label match So any cluster I provision that have the label usage and it's either production Or development Argo gets installed on So again that import step that used to be the manual piece That's all automatic if you have a bunch of open shift clusters And you have Argo and you want them imported into a new Argo because you just deployed ACM And you're making a new hub as soon as you import those in If with this resource created those get imported into Argo and can now be targets for Argo So again any cluster you provision you import into ACM can be automatically can automatically become a target for Argo And that's a big gain changer from a fleet perspective because that was a manual process Up until this point with a bunch of cli command you have to find the kube configs, etc Now that's automatic. That's really cool. That's really really cool. Now. It's like one button Um, you know push button get ops, right? It's just like one button is essentially one button and it's it's all there for you. So that's really really cool exactly Yeah, no, it's it's this it the Argo integration It's it gives us a lot of different possibilities And it seems to be at least for the time being resonating with a lot of folks as well Which is which is what we always want to see So now maybe we'll dive a little more into Argo integrations because it didn't it doesn't just stop with the basic We wanted to bring these things. There's two more points sort of major points I want to make and then we'll do some cool provisioning etc doing get up through get ops But one is application sets. We were bringing acm together with application with Argo application sets fits the mold of sort of the what you create one application set to many cluster concepts And so but again, it was limited to clusters and having a knowledge of clusters that are imported into the Argo server and so what we did was we worked with the Argo community and we integrated Our acm placement rules into application sets as well in Argo. So that's an example here. So this is a Which is really cool by the way the you're talking about the the generator, right the Absolutely. Yep. Yeah. Yeah, so I don't want to see your thunder there, but that was actually cool I saw I saw well because you know, like I was a really big fan, right of application sets and you know as as it's evolving I saw I saw on git I saw on git hub I saw like a bunch of commits from josh. Well, I was like, I wonder what I wonder what josh is doing in You know doing all these commits application sets. So this is actually really cool of how The integration between Argo cd and acm is happening through these generators. So I'm sure you're gonna you're gonna go into that As well. So absolutely. Well, that was what I yeah just pulled up So we talked about placement rules again placement rules is this automatic always reconciling way of Defining a set or a group of clusters that you want to target for an application in acm for a policy But now for an application set in Argo as well. And so we got that in I want to say a couple months ago So it'll be in the next release of ocm Or sorry open shift git ops, which is the brand naming for Argo But it's already available in the 0.2 community release of application sets But what this allows you to do is we introduce this cluster decision resource So before they had they had their own labels So if you had specialized labels on their on an Argo secret you could match it But again, that doesn't fit well with or too well It doesn't fit that well with the I've got a bunch of managed clusters I want to target and I'm already using placement in policies and other spaces And so we introduced and it's called cluster decision resource and not placement because we were trying to you know Not just be acm or ocm pushing our way in somewhere This is actually a generalized approach. And so you see it has this concept this idea of a config map reference You define a config map that kind of says what resource Should I read for a list of servers? What parameters within that resource should I read for that list of servers? And then it takes the name of that actual resource And so what that allowed us to do is number one It may meant it could work with our placement rule But it also meant that you know the folks at flux the folks at any other integrator that builds a Fleet management solution would actually be able to Take advantage of this change in application sets and use their version of placement And so it was kind of our well our homage to you know, trying to help the community along a little bit while also Building an integration point for us And so what we do with this to ease the ease the work along is you don't have to do any additional work when you're using Argo with acm ACM builds these config maps. There's actually two because we have two types of placement rules They build them automatically So literally all you have to do is create an application set resource Tell it whether you want to use an acm placement rule or the new acm placement There's so we have two types give it the name and you don't need the recue is sort of how often it checks It's technically optional But so just two parameters and the placement rule created and boom you're deploying to your clusters those same clusters That were imported into argo using our other resource You can even share the same placement rule between the two objects And so for us this was big and we're not stopping there And I don't have it to I don't have it to show you today But I will plug a future is that soon enough we're going to have a application set Create ui within the acm space because we're so behind it as a capability and there is no Application set ui anywhere else at this point either. So we're hoping to be the first we'll see It's coming in the it's coming up in the october time frame. So we'll probably be back to show that as well Yeah, there you go. Yeah Make our show planning easier. Thank you And so but so yeah, so at the point being though as you can see it's a deep integration We really anything we do with an acm subscription, which was the native capability we brought and that we have in our community We also wanted to extend because argo is now such a part of the red hat strategy We wanted to make a part of red hat and be able to do whatever we do in ACM be able to do through acm in a fleet with argo Which will lead me to the last piece at least around the argo pie that i'm going to talk about and that is We've talked about acm in the past our application strategy and i'm going to shrink down here a little bit It was always a pull strategy in that you define a subscription on the hub That subscription is propagated to your managed clusters based on the placement And then each of those subscriptions on those managed clusters reached out to get and pulled it down the idea being The hub goes down the subscription is able to keep processing you can replicate that with argo and and some of the Recommended configurations are actually in that format and to replicate that what that means is you put argo on each of those managed clusters But then the question becomes number one How do I get my application? definitions for argo to those clusters to do that pull mentality as well as How do I see that because argo doesn't have a roll up across argos today? And that's where acm again comes to we'll say it comes to the rescue for today But yeah ties to tie that tries that all together. It's the peanut butter to the chocolate, right? It's it goes it goes together. It's the peanut butter to the chocolate. It's the it makes the natella and no It's the natella to the two pieces of bread Exactly and so that's this last one and so here again because i'm acm and we're working with argo And this is a demonstration platform as part of this infrastructure build out It created an acm subscription, which is again the subscription is able to go to get and bring things down It creates an acm subscription using placement so that again means to all my managed clusters that brings an argo app So an argo application definition Is being propagated to all my managed clusters using an acm subscription But that could have easily just been an argo on the hub pushing the an argo application object to all those managed clusters But what happens is is when that argo application object is created on those managed clusters The local argo that again at the very beginning you guys saw me add a policy that said put argo everywhere So i've got argo on every single one of my open shift clusters That argo on that remote cluster says oh There's an application object for argo. I am going to deploy it and so in this case It's again another ng and x app and this is the special the special piece the the piece to resist on someone We might say is that because acm has these managed clusters and we've got this search collector that's out there grabbing data It's able to detect that there is an argo application deployed on a remote argo So there's no other There's no other information that there's an argo remotely there except for that fact that a policy put it there But our search is able to detect There's an argo application that argo application deployed and hey look that same argo application It's actually deployed across all six of these clusters And so it builds a group and puts even though they're individual argo applications It builds a single topology because it knows it's from the same source. It's from the same get repo It's from the same branch in that get repo or tag and it's from the same folder in that get repo in its definition So even though these are six distinct argo applications on remote clusters acm is able to detect that information and bring it back And display it just as we saw with the other one so we can see All of the different clusters it's on we see the deployment Under the replica set we get the access again to look at the yaml and the logs And so this is also when our go we talk about argo everywhere Or that's the way I like to call it when we have argo everywhere you may have developers that are using it So as what you can take away from this is as they're off there creating apps. They're not under the covers It's not going under any radar ACm is going to be displaying that for your operations for your sre folks In its application list, so you'll be able to tell that. Hey, you know christians over there and you know He built the miss pac-man app instead of the pac-man app that I showed earlier On the remote cluster and he thought well I was on the dev cluster and nobody was going to notice kind of acm Is going to bring those details back so all again this application although it's being Realized and shown as christian said on a single pane of glass in the acm It's actually only deployed in argos on remote systems It is not in any way on the hub instance itself and that one is really the kind of cool piece So it covers all of our possible scenarios. It's argo. We've got acm works with argo running on the hub ACm works with argo everywhere. ACm can help you put argo everywhere ACm will detect what argo is doing what argo is doing. Yeah, exactly exactly and bring that back But also in presented in a meaningful way because you know those again, let's say You know christian and I both had access to three of those six the different three of those six clusters So christian had half. I had half if I had put pac-man and I again You same github same branch same piece on what those three and and christian uses on the other ACm's going to say hey look we've got six copies of the exact same thing Deployed across all six clusters. Is that really what I want to do the sre is going to be like why is it on six? You know knock knock christian. Why are you doing this knock knock josh? What are you doing over here? I see you guys are deploying You know, so there's a bunch of visualization help you get here too And I say six, but this works for a hundred or you know when we start to talk edge scenarios We start talking about a thousand and that's really where you expect to find not argo on the hub targeting a thousand clusters But argo on each of those single node clusters everywhere and acm again is going to bring back your application status The fact that they're you know, there are argo applications there and and so you know that makes sort of the trifecta of We've got argo on the hub multiple argos on the hub or argos out in your managed fleet Whichever way you want to slice and dice it and it can be any combination of that As well as using acm subscriptions in there. We're going to display it. They're all going to look the same It's going to be the same look and feel there's no new learning curve, etc It's it's all just there and so as an example It's it's kind of like what uh to steal what kind of andrew solovan Sometimes says it's it's not a or it's a yes and right like yes And you know, you can do all this other other other cool things and I like that you brought up the the edge kind of use case as well and to take that kind of like a little further is that You know leveraging that multi-tenancy that that you get with open shift with argo cd of multiple Developer teams being able to bubble that up Right into acm to like you said, you know to have that that that policy driven to have the sre's or or have the ops guys Be able to look and see like hey, like, you know, we're having all these applications bubbling up And oh and by the way, I can see that and you know take action if I need to so it's it's really it's really cool It's a yes and right so it's like, you know, you can have it on the hub. You can have it You know everywhere you can bring it, you know, you can bring it in from From other clusters and see that information meaningful information as well Yeah, and the cool part is there's nothing special like you don't there's no weird like I don't you don't have to switch flip Flip switches anything fancy if somebody goes out and deploys argo on a remote managed cluster and starts deploying apps We're going to detect that and bring it back. It didn't have to be a policy that I used just to put it everywhere I just used the policy so I had a single source of truth that said, you know, I want to put argo everywhere So I defined a policy and my placement rules said put it everywhere and therefore it is and so, you know There's that detection mechanism. There's nothing there's no no special switches anybody that's running acm We'll get this and we'll it you know gets this for free and and we'll start to see it coming up And so just a because you know, I like to prove that I'm really doing what I'm doing I'll click on this which is again, we're looking at the remote one. So it took us to the argo This is the remote argo running on the other cluster So if I click back here before remember we saw I had lots of apps and it was even in a different filter mode There is only the one app because this is a completely different argo on the remote cluster I'm not going to show you or else since we're recording this although they'll be blown away tonight anyways, but But you'll have to but you know, I figure it's enough to see that there are no other apps like I'm clicking here We're not filtering it's it's literally it's on the remote system versus if we scroll back through my tabs enough Maybe we find the other one I had maybe it was this one Yeah, and here you see we've got all of them This is so this is the argo that was running on the hub itself And so again, it's you know, it's all these interactions and we're just creating You know creating more and more of these pieces and all of the data details you see here again I mentioned it came through search. It's available and you can You can deal with it through the acm search, which is just amazing for remote management and sort of keeping an eye on your cluster Oh, yeah, yeah, definitely Cool See here. There is a question that um, you know that chris answered but maybe you can expand on it a little bit Josh about The difference kind of like the history between mcm and acm Um, I actually still get those questions at all. Like well, it's a cm mcm Is mcm a replacement for like they they're they're a little confused on on that history and you know, you coming from ibm Um, you can probably give us just a little brief Uh history of and the differences if there's any Absolutely, so I just seems like ages ago now, but I guess it was really only about two years ago Then it was so mcm was multi cluster manager And that is actually what became the open source code base that is now advanced cluster management Although I think as always to confuse things a cms or sorry ibm still has an mcm product But it is now a cm based And it's really it's I think the they call it a the cloud pack includes mcm, which is really just a cm They use our community builds and that is why we've actually you know, not the plug here But we've expanded a cm onto power and the the z the z platforms as well So if you're running open shift z or open shift on power You can take advantage of acm as management platform or manage to platform as well at this point in the game with 2.4 Which is amazing So basically acm is mcm Yes, a cm ACm has replaced is the is the child of and has now become the new mcm Cool. Cool. Thank you. Um Sweet here, uh, there's no questions people like my metaphors, which is always cool Well chocolate and peanut butter Then maybe I will expand a little and we'll do a little more get opsie stuff because this is a get ops call So we've talked about the argo implementation argo very big on get up But also does helm acm does get does helm, you know some of the different targets Let's just talk about for a second. You can toward bit bucket get tea All of these can be get enterprise These are on-premise Ca or versions of get that you can run as well that you can target both these platforms to I guess the piece the takeaway before I get into some of the cluster provisioning cool stuff is that To take away from this is they're very interchangeable. We're highly integrated with both We fully support both and will continue to develop both because there are different places for each of them to To be used and they work great interchangeably as well, which I hope I demonstrated here Especially we did, you know, the remote argo apps. They were deployed by an acm subscription So there's all kinds of different possibilities And you know, we'd love to see what people come up with as well But you know, we try to show at least that there are a number of different ways to do it One that hopefully fits everybody's need that they'll be able to pick and But they'll get a similar experience regardless, which is our our goal Yes, yes, exactly. All right. Are they following the same versioning? ACM with a mcm kind of that transition I want to say they do And they should be documenting as well under the covers where which version of acm they specifically Are pulling in which again would be the standard releases So they'll be picking up at 2.3 shortly as part of it and in the cloud pack for an IBM They use the the acm releases as well. Cool. Good to know All right, so let's switch gears a little and we're going to talk a little bit about get ops provisioning Of clusters and so that's great. I've done some demos I think they're mostly they've been internal although we've talked about it At least a little bit on some of the previous twitch streams with the provisioning as well But provisioning a cluster and get is a pain and the main reason it's a pain is you got secrets Galore that you need to be able to make this thing happen You've got a big set of configurations and anytime you're using the word big That doesn't actually do well with get ops because Somebody needs to review big and the more yaml There is the more the chance that there is a sticky finger or an nefarious set of fingers That are trying to get something by and so what you really really want is what is the simplest way We can provision a cluster and something you get for free and what i'm about to show you is it's fast as well And so I think I actually talked about this with gurney on a previous Twitch the acm integration of cluster pools that come from high So the background there is a cluster pool is literally a pool One or more clusters that you've preconfigured the administrator preconfigures that is sleeping in hibernated state And that as you need you are able to provision and so in 2.4 I shouldn't say provision you're able to claim because it's already provisioned is the idea And so in 2.4 we introduced ui's and so here you see I've got a couple of different cluster pools created across a Couple of different cloud providers one in amazon one in azure one in gcp You can see that each one of those has a cluster in it asleep This is a demonstration. So I'm using single node clusters, but you can have Three node master clusters that host workload as well as six node clusters or 12 node clusters The sky is kind of the limit The cool part is is you can have a bunch of different pools as well And the only cost is the storage that are assigned to some of the drives in the in the providers Because otherwise they're off and they're asleep But what that also means is within seven Maybe well five to ten minutes. I'll use that number within five to ten minutes You can have a functional cluster with snow It's a little it's a little faster when you've got six or 12 It takes a little longer for all the nodes to sync together But literally it's the time it takes for the thing to power up and resynchronize back together is how long it takes to get a cluster But so why is this big news for get ops? Well, number one is it's fast again five to ten minutes for a cluster pretty cool What else cluster pool does though is if I don't have a cluster in waiting It will start to provision another one. So even though I've got one in waiting I can request two I can request three I can request four and the pool will just provision those additional clusters plus one more that it'll just put the sleep for the next person That comes to order and so there's no limit Except if I hit a quota, which is we won't get into today. It's a whole other discussion, but um, but quota Yeah, you want to limit somebody access But you can claim a cluster and so the key here again, I keep going on and on about simplicity So we look here. We see there's a cluster that can be claimed in aws one in azure one in gcp I think we'll try claiming one in gcp. So we're going to do it from git So i'm going to come back here get the list up Uh, I don't know if i'm going to be able to do it as a true split screen So i'm just gonna we'll do the change and get and then we'll flip back and we'll see how quick we can We can get the screens to sync I thought I had to get there it is that that's one of the things with with get ups is that um Sometimes like it works too well. I have always like I've always it's almost a complaint Right with with us that like that do a lot of demos and talk with customers Is that I'll do a demo? I'm like, oh wait like it already did it but like I promise already did it It actually just works too well like I couldn't pause it. So yeah, exactly So I was going to split screen this but my screen's pretty small that I'm doing the demo So I'm not sure how we'll flip back and forth But anyways, so I'm back in my same folder that I was before Except this time I'm going to go to a clusters directory And I'm actually going to switch the branch also to cluster management And so again, I'll get ops terms all things you do in get ops you define yaml You define it in different branches in a repo and then the get ops story is you've got argo Or you've got an acm subscription it points to that branch it points to that folder it points to that repo It's always syncing you can choose when it syncs. You can manually sync it You can time window the syncing and that's key Especially when you talk about provisioning because maybe I want to define it. I want to make the change So here you can see we've got a bunch of clusters But if we look at the customized we've actually only deployed the zero one The for aws the zero one for azure and the zero one for gcp And if we flip back i'm just going to close as I go so that This becomes quicker as we do it You get carried away you get too many of these windows All right, we can see always happens on a live stream all We've got the aws the azure the gcp So these were all provisioned via get ops these three I built up myself outside But these three came from the repo from the get ops because as we saw a second ago and we'll go right back These were the ones uncommented in the repository. So i'm going to do an edit I'm going to do this live just straight in git So we're going to do another gcp So i'm going to just comment it out here or take the comment out I should say and i'm going to put in my comment that says except I have to use my bluetooth keyboard because my other keyboard died on the provision jmp Gcp zero two and so i'm going to commit this so this is now committed So the customize has now activated this additional piece of yaml So if we go over here and we look again, I was talking about simplicity This is the simplicity. I have the name of the claim which becomes the name of the cluster that we start to see in a cm We have the namespace where the claim is being created and so From an administrative point of view, this is a namespace where I've been granted access to create claim objects So you can see exactly who's got what where and then also I got a some labels ACM any label you put on here will transfer to the imported clusters So if I want to define it again as development as production as it needs open shift Git ops installed I put the labels on it here and that becomes part of my managed cluster that that becomes my possession But then get pre-configured by all the git ops. I already set up And then the last one is I picked the pool and again, this is going to be based on the namespace You've been given access to there may be one or more pools I like to think of pools in this this regard really is templates of a cluster So from an administrator perspective the administrator took care of What this looks like and just gave it a name So he defined it or she defined if it's going to have one node Is it going to be an extra large or an 2x large all those kind of things and then they you know And so then you commit this to get I just committed it directly In theory if you're doing true git ops, I would have had to I would have had this forked I would have done a pull request and my administrator would have approved it which merged it in And so once it's merged in The ACM git ops is going to apply it automatically now We don't see it here because I have it on a subscription and we talked about Synchronizations and time windows If we go back to the subscription I'm using for this which is the snow cluster provisioning because there's snow Which stands for single node open shift I have synchronization off and so if we look here we can Reconcile off. Yeah, we have the reconcile rate off because I didn't want it to automatically apply But again subscriptions argo automatic reconcile you have the option of both whether it's acm or argo Time windows that we call them time windows So even if it's committed to git we can have it only apply on sundays at one in the morning or Any day of the week at one in the morning when maybe nobody's using it that's or using other systems And that's when we want the provisioning to happen because it's going to hit a bunch of repos to deploy a bunch of apps So again, you can use time windows sync controls on top of the fact that you've got The git as that source of truth that you you know, you're monitoring and you're controlling and remember there were like Three things we had to set in that resource So it's very easy to see is the person making the submission choosing What we really want them to choose and should I be letting them and you know Did they use a name that makes sense and they didn't have access to try and sneak in You know, I wanted to go to forex large because I was going to run some Netflix servers on there as well Or Netflix containers on there as well. So, you know, it really simplifies that process So anyways, I am going to hit the manual sync button here For it Which should then give us a time update If everything is working Come on So we've got a question chat here. Yep Can you do transformations to application like if you move them from one cluster to another Do the routes change the storage classes get updated? How does that get managed here? So It can and it sort of depends on what you're doing. Yeah, so You can do it based on so our placement determines where it goes So you would use place you'd be able to use placement to move it from location a to location b And then you've got access to things like customize Etc to modify names pvs etc When we get into pvs I don't want to give away the soup of some of the stuff that's coming But we've got a bunch of stuff around replication within the app so that you you know You're able to move an app from location a stateful app from location to location But in the most most cases today if you're if you want to deal with stateful It actually has to come from a pv in a container that is Separate to the main application if you're going to move it And so coming but coming are there's a number of different replication Strategies both from odf which is red hats storage as well as the it's called volume sync from the kube back organization that We're integrating with so if you stay tuned there is going to be soon We're going to have a bunch of examples where we've got databases flipping from cluster to cluster type of thing And keeping the physical volume for the the pvs in sync between them as we do that I wanted that 10 years ago Yeah, exactly go back in time and give you that I was uh anytime. Yeah, actually chris is funny how he mentioned that like anytime, uh, um, you know, I talk about Getups with someone is like well like getups is not really anything like like new per se It's like now we just have the ability to do the things we've always been wanting to do Now we actually can do it right so it's not like this I like the ideas newer anything we've been wanting to do this, but now we can so it's really cool Yep, and so again talking about things that happened So I flip back to my cluster list and we can see that I'm getting there's my gcp zero two This is the actual underlying cluster that was sitting there hibernated We see that it's resuming and coming back But again, we got that usage development pieces pulled over So once it comes online and the manager goes to ready state It's going to get my policies at cd. He's going to be encrypted It's going to get open shift get ops installed on it again Via the policy after that deploys all the applications that open shift get ops knows about And is referencing those also get applied So, you know, if we come back in about I don't know about 15 minutes or so I will have all of these things that we looked at before in application that said six plus local will now be seven plus local and You know all we did was define. We wanted a new cluster Yeah, we've been talking about turning off environments for Decades to save money and yeah Now it's just you know, you didn't want to commit Well, that's exactly what's really cool is like just taking it back step back is like you have a um Uh a get ops way to not only to provision a cluster but to apply everything you want to apply to this cluster Right. So then I you know, I you know, either as like an administrator or like, you know A project, you know manager, right or or even a developer. I can go and make a pull request to not only give me a cluster, but then Apply all the manifests that I want to apply And then also that that whole cluster is managed, you know Has a life cycle management aspect to it all via get ops workflows, which I think it's just It's like you said chris. It's just blows my mind and that like how How far we've come to where like now like we're like where we've always been wanting to be right for the past like 10 years But I mean even the past year which is hard And everything else right like the advancements that have come so quickly have brought us to this point where it's like We can do all that now Yeah Yep. No the biggest challenge I think is now for us as the dev team, especially in the get ops space is Is the simplification piece of it is You know, we don't want it as six yamls or and we have to deal with how do you deal with, you know There's a whole different discussion around how do you deal with secrets with get ops? You know The cluster pool for us is that perfect way of easy provisioning obfuscate All of the mess of what the configuration needs to be it's all predefined up front Which is what we're usually asked for the developer doesn't really care They just want by the the large the medium or the small and and they wanted it yesterday Well, now there is the large the medium and the small t-shirt size and you get it in five minutes Which is as close to yesterday as I can get At least today Yeah, yeah, I don't but so it's already it's the clusters now online It's in the pending import, which means it's being imported as it came out of sleep into acm And then soon enough the applications start to we'll start to roll down onto it and Before long it'll be configured and so it's it's This for me is the real game changer and just I I know we've talked about it on this before but pools For certain type of development scenarios where you need clusters all the time It really is a game changer and I can't say that enough And if it sounds like I'm excited, but you know, I'm pushing it it I use it in my daily It's it's changed my workflow like I come in on a monday and it takes me About 15 minutes and I've got the latest and greatest acm deployed It takes me under 10 minutes to get a six node cluster three worker three master And then takes me under 10 minutes to deploy the latest acm build snapshot on it And then I'm good and you know what if I find a problem on wednesday And I need to try the latest done out it takes me another 15 minutes And I've got a complete I start from scratch I get a completely new cluster in less than 10 minutes I deploy acm to it so any kind of development even and we're now applying it and we're going off a get-offs a little bit here But uh, you know we use it in our build strategies, etc. Like it's Getting that clean cluster is like the nirvana and getting it quick It's it's it's a game changing experience for developers that at least that work in Multicluster spaces all the time and you're doing dev or you're doing test and you you know You need to spin things up and down at different versions or you just need a bunch of clusters Having that on demand is is huge and there's really no other way around it Well, yeah, well and you talk about like changing your day-to-day workflow or like fundamentally change your work like that's kind of um where uh what get-offs did with me Whereas, you know, I used to be an SA and you know, and you know, I build demos and things like that Whereas like now I just need to bear kubernetes cluster and then I can you know apply I can basically go from like kubernetes install to my demo working like in a few commands now This is actually taking that even a step further now like you just Now you automated the building Of the of the cluster now. I don't have to build it And also by the way in the application of like my stuff right like a like my actual Absolutely. Yeah, so that's like you you take that, you know, like an extra step further right now I don't have to you know run openshift create cluster now. That's all that's done with me I don't have to do the oc apply dash k To apply my demo like you've already done that um in No, you're you're absolutely right and it's and then I mean and that I don't even include that in my calculation But it's actually there is after I deployed acm. I logged in I clicked into application Just like I showed at the you know, we come for a circle at the very beginning I you know, I write in my infrastructure build out. I choose get I paste in the url I choose the branch as so here. I'll even I'll go through it. I literally do this I choose get off fleet sample It populates the branch which I choose main it populates and I so this is just pulling it out a get I pick seeds I choose so this has been created already, which is why I was able I didn't have to paste it I was able to pull it from the drop down Usually I use I'll use the low reconcile rate because I don't need it to check that often I just needed to check the first time and then ever so often it's every hour It'll go out and look I hit save and that's it I walk away and you know, I go and do my daily development But then when I need a cluster, I'll have three You know part of that get ops flow as it went out and it pushed out the three Three clusters from a pool it deployed a bunch of applications You know my p.m. Or christian comes to me and says josh, I need you to do a demo. I didn't set anything up This is the environment because it builds As you said I build it out every time and you know as we get new stuff. I need to patch something I you can add things in it's you know You just I check it in to get and my source of truth just you know for my infrastructure code just keeps growing But it's it's there and it's always the same it makes it the same every single time and I need to make a change I commit it if it doesn't work and I see something. I'm like How did that get there? I go back and look at the commit history and go, you know Why did my buddy here john check this in you know make this change that that source of truth That's the whole that's the get ops mantra and you know, it's it's it's wonderful when it when it's working It's wonderful and it it definitely improves Set up configuration time and there's a whole discussion on the side of like Recoverability if I need to rebuild an infrastructure All I need is that single piece to start and then once we start to bring in christis You were talking about with with the pvs and stuff. It's the whole solution It's like, you know built from sketch one command type of thing on a new You know, I just provisioned a new cluster. I installed a cm from the hub with three clicks It's literally three clicks maybe four and then I apply a subscription or I you know I or I click in the hub and add argo and then I add an application set that points to that get that that is my seed Like argo could be pointing to that same seed directory as the acm application set It builds everything out and that could be you know, my six clusters my 60 clusters acm will attempt to do re imports It the sky's the limit like it's it's the infrastructure is code When it's realized and especially as we try to sort of keep it simple so it's readable It's it's it's game changing in a lot of ways. No, it's yeah, it's it's it's definitely game changing and it's also um I mean a while ago it changed my mind the The you know now I'm the preacher of like let's keep your your um your clusters as cattle, right? So like long I think long is the days where people thought like you can install kubernetes and treat it like like a v-sphere instance, right? And it's like that's not what it was made for and now like now we have the tools and the abilities to like treat them as treat them as cattle, right like clusters as cattle like we we um, you know, uh washari here asked about um You know moving Applications and it's like well, it's not this, you know, it your applications will move with everything and it's not really necessarily Uh a move in in the literal sense because it's actually it's it's a it's a copy and destroy, right? It's kind of like the mv command on linux It actually what it's actually doing is just copy delete versus An actual like moving of of your application. So Exactly. There was actually a question about support for v-sphere for the v-sphere ipi I know we show a lot of cloud stuff um the v so Um, what where are you guys at with the the v-sphere ipi support? So you definitely can provision with v-sphere. Um, we don't have the pools yet in v-sphere So okay the mware and open stack are the two that are in the works for the cluster pooling it's there so open stack and v-sphere both provision a little faster than the cloud providers and Sort of storing on local is not always it's more of a just the pool just becomes a speed thing at that point There really isn't the same kind of cost savings because even if you're using a cloud provider for v-sphere like packet or something like that You pay a fixed fee for the esxi whether it's empty or it's full. So You know Suspending your cluster doesn't actually do you any good But but we're bringing it and it's more so just because it makes These flows so much simpler and you get this idea of sort of templating and being able to define what you need your You know your administrator defines what the cluster some developer just says I want the big one or I want the small one You know, I want the one that's in v-sphere versus the one that's in aws, etc But so yeah, if you looked at well, so number one I just wanted to point out like we can see things are already starting to float over with the numbers It did actually provision But if we go back to the clusters and we do the create so this is the current Current stack of provisionable pieces. So we have the aws the gcp microsoft azure v-sphere Open stack bare metal. There's also tech preview for the assisted installer Which is the iso based for bare metal, but also I think can be used in virtualization and then You we also support the import so I had click they create but you can import your ARO your rosa your AKS eks iks I know I messed one Anyways, all of them All of this we I guess we called start ks you could or splat ks. There you go Yeah, it's it's um, you know talk about hybrid cloud, right? We always talk about hybrid cloud And you know all the challenges that come with hybrid cloud Um, you know, it's great to have like the single um A single interface right to manage your hybrid cloud. I think that's um, that's the The christy said the get ops is a holy grail. I think this is like the nirvana of yeah Of dev ops, right? It's like, yeah, I don't to have truly true hybrid cloud Um and be able to see everything with a single pane of glass It's yeah, and it's it's yeah, and it's not just like even if it's you can do a combination You can have get ops and get ops for your configuration definitions definitions, but you can manage all of your clusters Manually in a cm, but as they come in the get ops apply You know all the pieces of get ops apply because it's all just labeling and so you can import those star ks This is another thing that was added in the 2.4 discovered clusters You can you know if so if you're in and you already have five or six open shift clusters today That are not under fleet management You can add a cm to one of those clusters and then you use this page You define your ocm credentials and you'll get the list and you can pick which ones you want to import um You can provision a new cluster put a cm on it as I described you provision ocp go to the hub a cm goes in in five to eight minutes and Then you can discover all of your clusters from the remote ocm open cluster management Online and then pick which ones you want to import into a cm So there's a number of different ways to put it all together But then once it's together You can start leveraging get ops and you can do that for everything if you want like we demonstrated and this demonstration showed today Which was all the way from provisioning these things to configuring them to loading out the applications on them Or you can just use it for configuration or you can just use it for the application management Um, and you can manually manage each of those pieces from the uis from the clis as well There's a bunch of different ways to slice the pie all of them You know all of them supported all of them working together I mean we did everything in get ops today But we viewed it all and looked at the topology through the single pane of glass And that's kind of the key is sort of bringing these two types of technologies together that manual and they get It get in such a way that it's it it allows you to manage your entire fleet at scale Perfect. That's that's that's a that's that is amazing. So we're here at the top of the hour I don't see any more questions coming in So So yeah, so josh, I don't know if you have any any any final thoughts any final words You know before we close out here I was just gonna say if you haven't given us to try give it a try If you're using argo, you know, don't be afraid of a cm anymore. It's not in this or that It's a together we are better And so, you know, it's everything we talked about today is in operator hub If you want to find the get ops samples look me up on git hub, which is jn packer And it's everything we do is in the public here in red hat For the community and for the open source. And so you can see how it was done. You can try it out yourself Cool. Cool. Thanks So, yeah, so cool. Thank you everyone for for joining in chris. I don't know if there's another another show coming up There we have There's not so tomorrow red hat. We have a recharge day, which is really great So Yeah, so we it's you know, we have tomorrow off. So it's a long weekend for us red hatters There's other companies that do something similar. I think amazon has has something like that. So, um, you know For you for those of you have our recharge day, um, please enjoy it And uh, yeah, so next week, I don't know what we have next coming up next week, chris. I'll let you take that Um, something on the spot like that. Yeah, yes I mean, it's okay. I'm ready After this you have recharge. Yeah, there you go. So, um, so yeah next next next show, um Um, I talk about our back with with uh with argo cd We tried that once we failed failures not an option. I'm gonna get back on the keyboard again Yeah, we're gonna get on again. So, um, so look out look out for that in the next couple weeks. So, um, so yeah, other than that That's it. Thank you for everyone for uh showing up. Yeah, thank you so much, josh And thank you to the audience and stay safe out there folks. All right. Cheers everyone