 Cornelia on. She's, you know, if you if you took a look at that resume, she's been a kind of a heavy hitter. And, you know, like going and, you know, coming from Pivotal, coming from Adele, EMC is it's been, you know, very exciting to have, you know, have someone with with that breadth of knowledge, you know, coming DevOps for many, many of years. And then also Scott Helm, heavy hitter, I've always pinged Scott when I've had issues, because I just started writing Helm charts. I've always been pinging them with with some with some help. And Scott, I see that you have like a junior associate back there eating lunch or something. Yes, we're a good friend. Nice. Nice. And I have a two-legged, well, teen associate. Wow. Yeah. Going back and forth in the background. Pretty wild. Yeah, crazy times nowadays, right? So, so yeah, so I like to essentially just turn it over to you guys to talk about flux, open shift, get off anything, anything you guys want to show us what you guys are doing over at Weave Works. Sounds great. Thank you so much. And so, I mean, the TLDR is flux and open shift. You did this yesterday, Scott, when we were talking, you did the little heart, right? So, yep. So, I mean, that's the TLDR. But I have some stuff that I definitely want to show here. I'm going to go ahead and share my screen. And I have some slides because I believe that pictures are really helpful to set the context. But I promise you that this is not about sliding you to death. I'm going to spend most of the time on the command prompt in GitHub, in the GitHub UI. I think that's really helpful. And so, we'll show you some stuff. So, let me go ahead and share my screen. That's what Scott would call slide ops, right? We're not going to do slide ops today. Yeah, no slide ops. So, I assume you can see my screen. Yep. Yeah, we can see it. All right. I'm going to go ahead and go into present mode. So, you can see what the title here is, is that it's application DevOps and something called that I'm calling GitOps flows. Now, I should tell you that this is an official term, GitOps flows. But you'll see what I mean. I'll explain what I mean by that in just a moment. By the way, I noticed just a moment before we started that it says confidential on this slide. It's not confidential. Yeah, we have the same things. Yeah, we have the same issue at Red Hat. Our master deck says confidential. Oh, I forgot to take that off. Yep, exactly. And I was trying to figure out how to do it in Google Slides right before we started and I was unsuccessful. So it's pretty hidden. They hide that setting very well. Yeah, they do. Yeah. Yeah. So cool. So when I talk about application DevOps, I want to just take a moment to clarify that there is there's a number of different DevOps doesn't have like one clear, crisp definition. And so I want to give you a context of what I mean by application DevOps here. So I believe that DevOps fundamentally is about streamlining and eliminating the barriers between what has historically been, you know, development and then operations. We've historically had this boundary and what DevOps is about is about reducing or eliminating those boundaries. Now, those DevOps is a concept that exists on so many different layers. So there's this notion of doing DevOps for the applications, like the consumer based applications. And then there's also the notion of doing DevOps for infrastructure or platforms, etc, etc. And so I do not subscribe to the definition of developers do applications and ops do platforms. That is not the way that I'm talking about this. I'm talking about teams that are responsible for doing application development and operations. Makes sense. Makes sense. Yeah, makes total sense. That's it's in the name, right, DevOps, right? And you're talking about more of an application stack, right? So like kind of a clarification, you're talking about an app, the team that takes care of an application in the sense. Yep. And it turns out, I mean, I understand why people will say dev, you know, the dev part of dev is the application team and then ops is the infrastructure team, because we used to generally draw like operations was often infrastructure and operations. So we have these organizational structures that I don't want to carry forward into this DevOps world. I guess that's that's the way that I would sum it up. So so then to clarify that a little bit more, what I'm really talking about here is enabling that application DevOps team to operate at the speed that they need to operate in these days, release more frequently, reduce lead time, and then operate operate their applications. They are responsible for meantime to recovery for their applications. They're not responsible necessarily for the meantime to recovery for some server that's gone up, you know, belly up, but they're responsible for the meantime to recovery for their application. And then I was making this distinction that the platform team is very often in an enterprise responsible for maintaining security compliance, resilience and cost management. And so today I'm really going to focus on that first persona, the top persona there. Now I'll leave this slide here and folks can take a look at it. But what it comes down to is one of the things that has happened as we've moved into this Kubernetes space is that when we take this idea of DevOps and we say, Oh, okay, well, we're doing DevOps now in the Kubernetes setting. What that what we've done initially is we've placed a tremendous burden on the application application teams and that burden being, well, you've got Kubernetes now. So now you've got to know all this Kubernetes to be able to run your applications because remember, you're also responsible for running them. And so that's one of the pain points that give ops definitely addresses. And we'll see that as we go through through the demo. So then let me tell you a little bit about what it is that I'm going to do in the demo. So this is a really simplified kind of DevOps flow. We have on the left hand side, more of the development stages of the DevOps flow. And on the right hand side is when we're actually starting to apply the operations, we're running this thing in production. But it's the application team who's responsible for this. And at a very simplified view, we've got on the far left hand side, the developers working in their IDE, they're writing code, they're running unit tests, they're not touching Kubernetes yet at all. At some point, when all the unit tests pass, and they say, Yep, there's enough here, they go ahead and they build an image. That image will then get deployed to let's say some staging environment or a test environment. I'm just for simple sake here, have two environments. So something pre prod and prod. And I'll get to that, that parenthetical remark in just a moment, the auto deploy part, we'll get to that in just a bit. Then we're going to do some integration testing and staging. So there might be another set of test suites and things like that. And then we're ready to quote unquote, hand this over to operations. And I say quote unquote, because again, and this isn't a throwing it over the wall, it's a dotted line, it's not a solid wall. Right. As a part of that, you know, going into prod, we are first of all going to have to go through an approval process. And then we do a deployment into the production environment. Okay, so far so good. I'm following you. Alright, so now let's start to pull a little bit of GitOps into this. So you'll notice that I've introduced a couple of things here, I've introduced get. So there's definitely a get element in here. But then there's also and it's not totally visible. It's kind of implicit. There is the notion of reconciliation. So GitOps is about get. And it's also about this new fangled way of doing operations, which is all in the Kubernetes world, reconciliation based, right, controllers, reconcilers. And that's what we're going to talk about in this GitOps setting. Yeah, it's always pretty cool. I always like to say, you know, operations are now borrowing from what developers just have been doing this entire time. I say, well, they have this reaction really cool process that we should probably use. And now that we have the platform to do it in. It just seems like I, you know, you said Scott did the heart thing, I always say, we go together like peanut butter and chocolate, right? Like that, right? Then then now it goes together and creates this whole new thing that's that's really cool. Yeah. And but the interesting thing is that developers didn't necessarily do those steps in the development process, using this reconciler based approach. Yes, correct. That's where GitOps actually brings more than just get. If you're just bringing Git into the equation, then I assert that you're not doing GitOps, you're doing Git automation, right? Which is something developers have been doing for a long time, but operations has moved into this, this constantly adapting mode. And we do that with reconcilers, right? Yeah, which is, which is the power of GitOps, right? Exactly. It's responsive. It's proactive. It can put things back to the way it was when things go bumping the night, right? Like, and I've said this before, I feel like GitOps is kind of the holy grail of DevOps, right? Where it's going to give you the automation you want while enforcing those practices from DevOps that are hard to instantiate at first, right? Like this kind of forces a culture change just by doing it, right? Yeah, it's pretty cool. It definitely does. Yeah. Well, especially like, you know, I come from the upside is like, well, now I don't have, now I don't have to write all these scripts to do a lot of this, you know, Kubernetes does a lot of this for me. And like, there's a process for it now. You know, I'm not the type person to hold on to my scripts. There are definitely, I've known people have done that. I'm the complete opposite. Like, I'm like, no, I don't want to run my crappy script anymore. If the platform can do it for me, I'm all on board. I'm definitely on board. Yep, exactly. So then this is what I want to demo. So what we're going to have here is we're going to have the the left hand side of the flow. I'm actually going to start at the right hand side of the flow. So I already have something that has been there's a PR for something that I want to deploy into into prod. Now how that PR gets created is kind of out of band for today. I'll we'll say a little bit more about that as we go along. But I'm going to start here on the right hand side, which is, well, how do I do this deployment into prod? Well, you do it by approving a pull request. That's I'm going to start with number three here. And then we'll see that simply doing that, that approval is going to cause things to happen in the production environment. Now here's where I'm going to introduce flux for the first time. And by the way, I should warn everybody who's listening that today's talk is not a flux tutorial. It is not a getting started. I'm not going to go through the getting started the quick start guide and say here's how you install flux. I am going to show you some elements of, you know, options that you can do in installing, you know, getting flux installed. But I'm going to do this more from the perspective of, okay, what is the value that flux brings? What is the the experience for these DevOps teams once flux has been established? Flux is often established by the platform teams. So I'm going to for today assume that flux has been established, although like I said, I'll show you some options that you have. Okay, so I'm going to start there and then I'm going to work backwards. So I'll explain what happens on the left hand side when I get to that part in the demo. And I think with that, I'm going to jump over to demo. And so let's see here, I think I want to go, I'm going to leave my, hang on, leave my, there we go. And I'm going to come over to here. And I'm going to come into the, as soon as that pop up goes away, I'm going to come into the OpenShift UI. Because after all, I want to show you all how flux works beautifully on top of OpenShift. Could you, just real quick, could you like blow up your font size a little bit so folks can see it. Thank you. Is the OpenShift UI good enough? Yep, I'm just, There you go. There it is. So here's the OpenShift UI. So what I have, by the way, and I'll jump back into this, this slide, is that I have an OpenShift cluster here. And that's the one we're looking at right now is the OpenShift prod cluster. I have a second OpenShift cluster, which will be touching in just a little bit, which is this integration cluster. And I have a third OpenShift cluster. And I just created separate clusters for these things. And actually, that's the royal eye. I want to call out my colleague Chanwit, who did all of this configuration for me. So thank you, Chanwit. Chanwit set up all of these clusters for me and got them configured. So thank you. I have a third OpenShift cluster that is dealing with this spot right here. It's dealing with the building of the images. And I'll show that to you in just a moment. So right now I'm over here in the OpenShift cluster that is for this part. So if I go to that OpenShift cluster, I can't do it justice. I don't know all of these elements. But what I want to do is I want to come over here into the Operator Hub. And I want to show you that in the Operator Hub available today, and I'll do a little tap dancing here while it refreshes. It's a little slow. I imagine my machines might be a bit under provisioned. Either that or did I lose my login? Let's see. Yeah, I think tokens expire after a little bit. So you might. Yeah, I did it right before. So let's go back here. Okay, so it should be fine. Let's see. I'm just thinking, well, let me go to here. Demo gods. We were just talking about this, right? This is how you know it's that it's real. It's that things don't always work. It says waiting for console. Oh, there we go. Okay, it looks like I'm still connected. So let's go to Operators. What happened to my second? Yeah, it looks like the console's having an issue. Yeah, something just. Okay, there we go. Weird. Okay. So let's go to the Operator Hub again. And now I'm at least I'm getting my little progress bar. And what I'm going to show you when it pops up is that there is an operator in Operator Hub for flux. And that is in fact exactly how I've deployed flux. And I'll show you in just a moment when this does decide to update. It's extraordinarily slow. Well, let me show you in the meantime while we're waiting for that update. It's not your local machine, is it? Right? Like, yeah. It's not on my local machine. No, no. These are, I think they're running up on Equinax, if I remember correctly. I think that's where Chanwood put them. So, but what you can see here is, and I'll explain the windows on the left-hand side, but what you can see here on the right-hand side is exactly this command that I just executed. So let's see. Yep, I'm still connected. And so what I did here is I am showing you the pods that are running in the flux system namespace. These are all the pods. And then there's a whole host of other things like service accounts and a number of other things. And I'll show you what that definition looks like in just a bit. There's a whole host of things that got installed by that flux operator. So in this particular case, I have installed flux using the operator that is available in Operator Hub. Full stop. I did not do any other magic. I just installed the flux operator from Operator Hub. Awesome. So let's see. There we go. So if I go to flux, there you go. You can see it is in there. And if I go to installed operators at the risk of it taking, there we go. There we go. You can see that flux is installed. And it's a community operator. So if we go back here, you can see that it's a community operator. It lost my search context. So it's a community operator. So I would consider this kind of alpha beta stage. But this is what I'm demoing live today. Okay. So that's even tempting the demo guys even further, right? Even further. We made a stuff, yeah. Exactly, exactly. So what I want to do now is, in fact, let me come back here. And you'll notice that there is a Git repo in this picture. That Git repo is the config repo. The source code is over here in this other Git repo. And we'll touch that in just a bit. I'm in this config repo. So it is taking the image that's already been created. And by the way, the image repository that I'm using for my demo is key. So you can see here that I have versions. Let me refresh this because I'm pretty sure I have a 5.09. Yep. So I have a 5.09, a 5.08. And what I'm currently running, and let me go over into my other, let me get my other OpenShift demo window, is right here running on, I have a port forwarded. So running on port 9000 is what I'm running in prod. And I'll just refresh that again. You can see here that I'm running 5.08. So I'm running this version that I had pulled down. And that is being configured by the, is being deployed through the configuration repository that is here. So for super simplicity sake, I do not recommend this for the enterprise and production. I have the config for staging and the config for prod in my same GitHub repository. So you can see here that I've got my prod cluster and I've got the configuration for my web app. So here my web app has a back end, a front end, and some common stuff. If I go to the front end, for example, you can see that I've got a deployment. And right now this deployment is pointing to 5.0.8. Okay, so that's what we've got running in production. Now to shorten the cycle a little bit, I already staged a, and let me go over into the demo repository here, I already have a number of changes that I've made. So you can see here that I've modified the front end deployment. And if I do a git diff, uh sorry, git diff, you can see, uh, git diff, cached. Staged. Staged, thank you. Here we go. You can see that I have bumped the version number. Oh, hang on. This is, sorry, I'm gonna go, this is how we know it's real. I had already issued that pull request. I decided last night when I was setting up, I had already issued the pull request and I hadn't done it locally on my machine. I had done it just through the git UI. So if we take a look at the pull request that I have already created, what we can do here is we can take a look at the files that are changed, and we can see that in the in the back end, I've bumped from 5.0.8 to 5.0.9, and in the front end, I've done the same. Okay. So that's all I'm doing. So at some point, somebody decided my new changes to my image are cool, and I'm going to issue this pull request. And so from a get this thing deployed into production, all I need to do is merge that pull request. Now, before I merge the pull request, I want to draw your attention to a couple of elements over here. You'll notice that I have two windows here, one which is running a watch on flux get sources and another one that's running a watch on flux get customizations. I will explain those two boxes in just a moment. Those are beginning to give you insight into that concept that I talked about called the get ops flow. So this is what's allowing us, these are the components that we're using to program how things get from the get repository out into the running deployment. Okay, so we have a programming model. Flux represents a programming model for programming get ops. Nice. Makes sense? Yes, all right. Yeah, I see it. Yeah. All right. So you'll notice if you take a look at this. So by the way, there's two components in this particular flow. One, which says I need to go look at get. And then the second of which says once I've gotten something from get, what am I going to do with it? And in this particular case, I'm going to apply a customization to it. It's just straight yaml. So it's actually an identity. So I'm just going to take this yaml and apply it to the cluster. I'm just going to do a kubectl apply if you will. So flux provides these and Scott, actually I'm wondering if this is one of those areas. In fact, why don't I do that? And I'm going to ask you to guide me because you spend a lot more time in this. Where's what's the diagram that I want to show here, Scott? That shows the kind of different, different, you know, the source customization. If you click on getting started or learn more down below, it'll take you to what right now is in the toolkit website. And where do there's a diagram? I want a diagram somewhere. Here's a diagram. I believe it is under, I believe it's under a micro. Let me take a look and see if I can bring your memory this around. Yeah, it's more muscle memory than anything I think. It's possible. Yep. Thought I thought I could eyeball memory about one second. Yeah, we use this three times. While you're bringing that up, I'll show you this is the flow. This is the flow that is implemented in prod here. So there's two components. There's two steps to this flow. There's retrieve it, retrieve from get and deliver to Kubernetes. Super simple. The way that this is showing up in the CLI here is that the get from get is happening via this flux source reconciler. So remember we talked to Chris, you were saying it's all about reconciliation. It's all about adaptation. We have this source reconciler that's constantly watching get. And it's saying anything changed yet? Anything changed yet? Anything changed yet? And when it does, it grabs that. Yeah, and that's using the, that's leveraging what's already built in Kubernetes, right? Like you have a reconciliation. The loop. The loop is definitely implemented using the customer resource. It's KubeBuilder. It's using KubeBuilder to create the reconciler. Yes, there's a custom resource. So for example, we can come over here and I can do a KubeCuttle API resources and I'll grep that to flux. And by the way, Cornelia, in the zoom chat, I posted a link to the okay, very cool stuff on me. Yeah, my goodness. So slow. So what you can see here, you got it here. Yeah, I put it in the twitch. Yeah, thank you. So what you can see here is you can see all of the different CRs, CRDs, that are created as a part of flux. So right here, you can see that there's this Git repositories CR and that Git repository CR. I have one of those in existence already and that's what we're watching up here is we have a customer resource that's a Git repository and there's a reconciler that's running in Kubernetes that is taking the values that are in that Git repository object and constantly checking it. So and those values are what's my Git URL, what's my branch, that type of stuff. Makes sense? Yep. Yeah. Okay. Now, the status that it reports on that, what you can see here is you can see that it's reporting the actual Git SHA that it has most recently reconciled from Git. Nice. Okay. So if I go back over here and I take a look at the SHA, it should match. So what we've got is A4F and if we go back to the command line, sure enough, A4F. Yep. Okay. Now, you'll notice that it's not only the reconciler that's watching Git, there's the downstream step. Remember, there's the apply to Kubernetes. That is what's happening through this customization object. And notice that it's also reporting ultimately what I've just applied maps all the way back to this Git repository, this Git SHA. Makes sense? Yeah. So I mean, this is a great way, like go ahead, Christian, sorry. Well, no, I only had one question when the customizations, are you actually talking about customize in this? Can I describe this one, Cornelia? Yes, please. So this is, I have a fun couple of slides on this that I don't have prepped for now, but it's basically helping people to understand why we actually use the customized controller to apply in YAML. And the answer is, it seemed like a good idea at the time to name it that. That was it. In retrospect, everyone agrees that that may not be the most clear name for end users who are specifically looking to use customize. The thing is, since customize is built into the kube control CLI at this point, its packages were available. And it also does just, it's essentially a wrapper right now for kube cuddle apply. So we use the same thing because under the hood it's all the same, but it doesn't really make a lot of sense for end users thinking which controller should I use so that there is a proposal to modify that. But for now, that's what we're using. All right, so I'm going to do the magic thing. I'm going to come back over here. I'm going to go into my pull requests. I'm going to bump that application and I'm going to jump back very, very quickly. So for those of you who might be listening who are new to GitOps, notice that I'm not doing, other than showing you, I did kube cuddle apply to show you things like the API resources or I didn't do kube cuddle apply, I did kube cuddle get. So I'm using kube cuddle to give you some visualizations, but I'm not applying anything into the cluster by hand. So I want to apply something to my cluster and the way that I do that is I do a acceptable request. So I've just merged the pull request and if I come over here, watch the Git Shah. So the way that I've configured this and up there it just changed. Oh, yeah, the customized one changed. The customized one changed and I think it's just a matter of there's a delay somewhere because it should always be downstream. This one should be updating as well. And I'm a little surprised that it hasn't, but you could also see down here where I'm showing you what's going on with the web app that I've got a container creating. Yeah, the rollout's happening. I see it there. So I'm going to control C out of this and I'm going to restart that watch. Yeah, I don't know what was happening with my watch, but what happened was the source controller did its thing and notice that these two steps, there's something really important here is that these are happening. These reconcilers are independent reconcilers and so they're happening on their own schedules. So you have the control. Remember what I'm talking about here is I'm talking about programming your GitOps flows. You get to decide, you know what, I'm not going to, I'm not going to overburden my Git repository. I'm only going to do a reconcilers. I'm only going to look at my Git repository every 10 minutes or so. We do in Flux also have an eventing mechanism that I consider an optimization so that you can set it up so that when something happens, you can set up, for example, a GitHub Action, which will trigger Flux. So it isn't that only the reconcilers are running. You can also kick Flux and say, hey, I have a GitHub Action that is going to kick the reconciliation to go off. Or even just a simple webhook too that's also supported. It's pretty flexible. Yeah. Yeah. And so now you can see here that we have my new version of my back-end and my front-end. They're 102 seconds old. And just to show this, if I go back over into here and do a refresh, if the demo gods will have it, that should bump too. Let's see what happened in my other window. Hang on. I think my port forward must have stopped. So I'm going to restart my port forward. Oh, it stopped, of course, because I'm in a different pod. That's why. Yep. So if I come back over here and I refresh this, ta-da. There you go. New version. 5.09. Okay. So this is pretty vanilla stuff. And right now, Chris Short is thinking to himself, why on earth did I invite these folks onto the show? I can't get that at all. I don't know. I know it's worth it. So it is pretty simple, but there are a couple of things just to go back and summarize on this picture that we aren't assuming that we know how you would like to apply things into your runtime environment. We are giving you a programming model in which you can implement the flow the way that you want it to. Okay. And so that's why I showed you the source controller separate from the customized controller. You get to decide how those things are wired. And in fact, let me show you what that looks like. If I come back into this repository, if I go into the prod cluster and I go into flux system, you can see here, here's the components. So here is the source object. This is exactly the source object that we were looking at. And you can see that the interval is a one-minute interval. Let me blow this up a little bit for you. Okay. Then there's a customization object which is tying off of that source object. Sorry, it's the wrong one. Where's my pipeline? There was a question in chat, and Scott, you might be able to answer this. Let me pull it back up here. That's why I have the whole thing in front of me. Is there any web UI for managing flux instances or app deployments at scale, something similar to what ArdorCV has today? What's the preferred way of managing flux deployments at scale? And one last question, does flux have some sort of RBAC integration with CUBE? So it's kind of three questions. It's like three questions. It's tackling one at a time, right? Right. So web UI. Scott, actually, Scott is so deeply ingrained in the whole flux. He's part of the Flux open-source team. So Scott, I'll let you take all those while I poke around and look for the other object that I want to show you. Great, sure. Yeah. So last question first, RBAC. Flux specifically uses RBAC. It doesn't have some type of separate access control system. That is somewhat different from some other projects in this space. You simply use the RBAC you have and as everyone who's set up or anyone who's set up or thinking about even setting up, even simply plain YAML manifests in some type of version control system knows that the way you structure your repo in the week or repos generally mirrors, if you're successful anyway, mirrors your organizational structure how you want people to be able to access certain things. So anyway, it's slightly aside from RBAC, but the point is that we don't have conventions that you must follow for that. We do have a way of impersonating using, excuse me, delegating access. Yeah. Yeah, that's what I was talking about. Those are two different things. Yeah, two different things. RBAC style in Kubernetes and we have a kind of interesting way of going about it, if you ever want to see it. It's given a really great, a really good talk on this too that we could always link to. But yes, quick answer is yes, follows RBAC rules. It does use impersonation inside of Kubernetes to do that delegation. So it's kind of an interesting way about it. You use that run as this, okay, got you. Yeah, and it allows name space separation in a way that works well. There are some really good demos to show different use cases of how flux uses RBAC. And then the UI question, we do have, that is part of our roadmap. And it is being worked on pretty seriously by some members who do work for WeWorks, but it is for the open source project. And so other people are very welcome to become more involved in that with us. Yeah. And the other thing- As we say, PRs are welcome. Yeah, exactly. PRs are welcome. And by the way, it's not only roadmap. I mean, it's definitely roadmap. But if you go into the Flux CD repose, you can find some of the wireframes that our colleague Jordan has put together that show you, so I'm showing it in the CLI, but it'll show you that view. And it actually goes a bit beyond, in some ways, well, in some ways, not a bit beyond. So if you take a look at the way that I strung things together, and I said, check it out, we've got the SHA here, same SHA down here. And there's a third step, which is then we have the running application. That is what the wireframes are showing. They're showing a GUI that links together not just these first two pieces that are about the delivery part, but links the delivery to the running application, the operations part as well. So if you will, it's linking this, this all the way down to the running instance. Exactly. Yeah, and I think one thing that's really interest or important to note about Flux UI, that's out of scope of the Flux UI, there is no intention of a UI for Flux making imperative changes into your cluster for you. Ever. There is absolutely no intention of that because that goes against the GitOps model as many of us understand it. So what it likely will do is allow you to do the same kinds of things through, I don't want to be pejorative when I say ClickOps, but let's say through clicking, right, through UI. But what that would do under the hood is use the controllers that already are able to write back to Git to do so on your behalf as long as you authorize it properly. So it won't actually make imperative changes in your cluster. It will however update your configurations in order for those to then move to your cluster. There is one other side of this is that Flux allows you to pause and resume reconciliation on a per delivery basis. Let's say whether it's a helm release or customization. And I know that it's not the only tool that does that, but it does that through annotations. It doesn't do that by telling the cluster, hey, stop working. So that's how we do it and that's how the UI would do it under the hood. Or the plan is it will cool at some point. I'd just like to share in the chat, someone says be Flux, not Flummoxed, which I thought was amazing. That was brilliant. Brilliant. Okay, so I'm going to pull us back because I want to make sure we don't run out of time to show the cooler stuff. So both of those components, so these two components that you see in this diagram, Retrieve from Git and Deliver to Kubernetes are in the same file here. So you can see here, here's the Git repository. This is the source object. So this is, I'm pulling from this Git URL on a one minute interval. And then down here, I'm applying the customization on a, I'll come back to this 10 minute interval in just a moment. I'm tying, here's the link. So I'm going to this Git repository object and I'm going to apply specifically within that Git repository. I'm going to apply everything that's in prod cluster. Interesting. So this is kind of a way of getting, so your Git repository is your main repository and your customization object is your overlay, essentially in this case. Yeah, that's a good way of thinking of it and it's also the area where I scope things. Yes. So notice that this is scoped to just the prod cluster and I can use branches. I can use semvers. I can use all sorts of different ways to control my GitOps flow. So that's part of the programming model that we give you through these components. So it makes sense why then now you are separating your prod manifests from your dev manifest instead of having the overlays in the same repo. You have individual repos and you just load them in almost like a plugin system. I'm using this and I'm going to plug in this, this, and this. Yep, exactly. Now I'll come back to this 10-minute thing in just a minute because I'm looking at Git every minute but I'm only reconciling here every 10 minutes. Did we just get lucky? And that's where I was saying that we also have an eventing mechanism. But I'll come back to that. So if there's an eventing mechanism then what is happening on this 10-minute interval? I'll come back to that in just a second. Okay, so I was here. We've just demoed what's on the right-hand side. What's on the left-hand side is a lot more sophisticated and it's super interesting in a couple of different ways. So what I'm going to demo to you now is that I'm going to make a change. I'm going to go into this cycle right here. I'm going to assume that the unit test passed. I'm going to commit a change into the repository and then the rest of this is going to happen. I've got a config, which is the same repository but it's the staging folder. So it's scoped differently. And then we're going to do this deployment into the integration environment. So let me switch context here for a moment because I want to show you my build server. Where's my cursor? There we go. So woman behind the curtain. Bear with me one second. I'm doing things on this window so that you can't see things like... Yeah. Ignore passwords. Ignore passwords. Yeah, I don't want any of that on stream here. Yeah. Yeah, I was once watching a stream with Stephen Augustus. You guys probably know him. He's in the chairs. Oh, yeah. I love Stephen. And yeah, he's a great guy. And I was watching his stream and once he accidentally showed his Azure token, whatever. So yeah. So he said it was such a pain to just change everything because I had to delete the token now. Yeah. No, I've actually done that at a meetup before where it's like, oh, dang. There's my token and I'm burning it now and watching all the alerts pop up and I'll get back to it. In a minute. Yeah. In demos, I really try to use read silent just so that it doesn't... Yes. Yeah. Good point. Yeah. But I don't... I sometimes forget to put that in the gist of for demo. So hopefully people know that when they do it themselves. Okay. So here, I'm going to come back over into... I'm now on my other, my build server. And if I come into pipelines, you can see here that I've installed open shift pipelines. So we've got Tecton running behind the scenes here. And here's my build pipeline for the application that I'm deploying. So you can see here that I have had quite a number of pipeline runs. And you can see that the last pipeline run was yesterday. So I'll prove to you that this is all going to be live. So this is where I'm going to actually use my command line. Because I don't think I staged this yet. And I want to show you a number of things. So I'm going to go into pod info. And let's see. What have I done here? Get status. Get log. I don't think I've made the change. Okay. Good. So I'm going to go in and I'm going to bump my version number. So here's my massive code change. Is I'm going to go from version 5.0.9 to 5.0.10. I'm going to go ahead and save that. Now I do a get status. Get add. I'm going to do a get commit. Release new version. Then I'm going to tag this thing. I'm going to do a get tag so that I get the right tag. Because I'm keying off of tags. Some of my automation keys off of the tags that are showing up in key. Where my image is going to show up. So I'm going to do a get tag. And then I'm going to do a get push. I'm going to push the tag up. And I'm going to do a get push of the bits. And now what we're going to see is we are going to go back into here. There you go. You can see the pipeline started running. Now this is one where Chanwit, my colleague Chanwit, used exactly, and I dare say he probably learned about it, from an earlier GitOps happy hour that you did, where somebody showed he's using this polling operator that was written by somebody else. Who was it that did this? Mario. You had Mario on the show quite a number of months ago. And so Mario took advantage of this polling operator that was polling to kick off these pipeline runs. Nice. So we're using that same component. Cool. Nice. And I see you're using Tecton as well. So that's really cool. Exactly. Yeah, this is Tecton. We wanted, because what we wanted to do in putting this demo together is that we wanted to say, okay, you're an OpenShift user. You've embraced things like Tecton OpenShift pipelines. And so we really wanted to use this. Of course, you could be using CircleCI. You could be using GitHub Actions, those types of things. But I assume that we're in this OpenShift context that that would make sense. Yeah. So I am then going to go back over here into this repository. And let's see if I can be fast enough I am not sure that I was fast enough. But let me switch contexts. I need to change my KubeKettle context. That's always a challenge with GitOps is if you're trying to show something to someone, by the time you show them, you go, it already reconciled, you're like, oh, trust me, it did change. Like that's how magic it is. There's no magic in the background. I wanted to show you the sync. But the sync happened so fast that, you know, here we are. But the cool thing about GitOps is that it's all auditable. You can see it all in Git itself. Yeah, you can see the audit trail in Git. Yeah, that was one of my points I was meaning to make during the show earlier was that I love GitOps for its auditing capability. Right? Like you want to know what changed and who approved it. Here you go. Right? Like here's the shaws. Here's everything. All the information you need to audit the trail, any change is right there. You can do the same with, say, storage buckets, S3-competible storage buckets with Flux. And I'm mentioning this only in the context of audit trails because it's highly recommended to enable versioning for your buckets if you do this. Yes. Okay. So you got to encrypt those buckets, though, too. There's a lot of people who don't. And it's like, you always get the news. It's like, oh, someone stole a bunch of credit cards to encrypt down the bucket. Yeah. So I don't end up in a Corey Quincy email, folks. Exactly. So I think I was fast enough because this was the shot that we had before. And to verify that, let's come over here. If we go back to demo, the latest shot, yep. And this was 20 minutes ago. So we're in good shape. But what you're going to see here is we're going to see that shot change. And I'm not going to have done any kind of a commit. That's kind of the magic that I want to show you here. Now, you might have noticed that there was a third window here. You know, I have four windows on the left and I was showing you all of those. So I'm now in the context of this. There's another watch that I'm going to put here. And I'm going to watch on an image update. Okay. And so I'll put the image update watch up there. And then I'll show you how these things all link together in just a moment. Now, this image update automation is doing this. It is watching the image registry. And based on a certain policy, it is saying if there are changes in the image registry that this policy applies to, then I am going to automatically update my configuration to reflect those image changes. Nice. Okay. So what we're effectively doing, we're going from this type of a flow, which is the flow that we did in production, to this flow. Okay. And so this flow says I can be triggered off of this or I can be triggered off of this. Right. And in fact, that is where I'm going to get back to that comment about the 10 minutes. A getups flow, and the reason that I'm using flow, and flow isn't made the right word either, is that it's not, to me, a pipeline as something that has a start and a direction. Getups flows do not have a single starting point and a single direction. What's happening here is that this flow that is happening in the staging environment can be triggered off of a change that this image update policy applies to. Or if I had gone in and just changed my configuration, it would have updated up, it would have started the getups flow based on that. Or if somebody did the modern day equivalent of SSH'ing into a box and causing drift, i.e. they did a kubectl apply, this would have also triggered some activity in this getups flow. And so you remember that 10 minute interval? That 10 minute interval wasn't waiting for an event from the get repository object. It's saying that on a 10 minute interval, I'm going to constantly reconcile. And if a change happened from something that I don't have an event on, I'm still going to reconcile it for you. So there's many, I guess, points of entry, right? Exactly. Where you can do the triggering. And there are points of entry that are not, that are cleaned up after. Like, for example, when Cornelia said, if you did like a kubectl edit or a kubectl apply, if that's not also represented in your get config, that's considered a drift. And that will be overwritten with your config. Exactly. Nice. Exactly. So that's what I mean by these number ones are intentional. It's not ordered. Any one of these could be the starting point of some activity in this flow. So it's kind of, you know, I always go back to, I remember Kelsey Hightower did a talk a long time ago, back when he was in CoroS. So this is a long time ago. He said that, you know, Kubernetes is how you design a system if you don't have SSH. And I always kind of just take that and expand on it for GitOps is like, GitOps is how you design a system when I take away kubectl from you. Right. And, you know, that's, and that's kind of like, you know, the direction we're going to say, you know, eventually we want to take kubectl away because we have all this reconciling going on. Yep. Exactly. Okay. So it takes about five minutes to build that image. And sure enough, while we were chatting, and I was showing you these things, notice that the shot changed. And so we now also have, so my image update automation has not triggered yet. And I wonder why that is. Oh, it must have just triggered. Yep. There we go. It just triggered in that moment. It just triggered because you can see here that the new containers are getting created. Okay. So what we have is, again, that flow that I showed you. Now, let me show you what the effect of that was in the Git repository. So this time we are in the staging configuration. So just like we had before in Flux system, I had this flow. I had the Git repository object, which is going off of the Config repository. And here I'm scoped to the staging clock. I'm scoped to the staging cluster. But where am I representing the other steps in this GitOps flow? Well, those other steps are being represented up here. Should have put them in the same, I maybe should have put them down in this folder, but don't worry about that. Is that I have this image update automation that says, okay, I'm going to watch based on certain policies, I'm going to watch this, and then I'm going to make updates to the configuration. Let me show you what that update looked like. So in fact, let me go back here to the commits and because there's a few interesting things to show here. Notice that here was my merge, my pull request merge 26 minutes ago. But the more recent update was this thing that was done by Flux. Flux updated the repository, not me. And so if I take a look at what this particular change was, that notice that it bumped it from 5.09 to 5.010. But there's something super interesting here, and I need to move this because I just want to show the file now so that you can see a little bit better. This is all roads lead to get. So if I go into, let's say my front end deployment, and I just want to show it to you with a wider screen. So if I go down to that, so this is the change that was made. How did that image update automation controller know to watch this particular image tag in my deployment? Well, it's through this annotation. And this annotation is pointing to that image update policy. Okay, does that make sense? Yeah, no, definitely. And so let me show you the policy now. So if I go back to the policy, the policy says, I am going to update any time that I see a change in the patch release. So if I had updated to 5.1, it wouldn't have done this. Yeah, I'll say it wouldn't have triggered, yeah. So here you're, I guess, you're upsticking or you're pinning yourself to a Zstream, essentially. Exactly. Setting a policy, I want version 5.0, whatever that is. Exactly. And it's really important to note that this is the thing that differentiates, say, this type of get-ups with, or excuse me, from other ways of doing this, say, with a latest tag and the image pull policy always, you know what I mean? Yeah, yeah. Other tools that store, that do pinning and store that within, let's say, a secret or something in your cluster. The main reason that it's not stored purely in cluster is get-ops is also, amongst its other values, allows for disaster recovery. So if you want to be able to set up an entire cluster exactly the way you had it, not just within a specific range and hopefully that's right, no, exactly the way you had it, it could do that. Mm-hmm. Yeah. Yeah, definitely. I'm also not a fan of floating tags, specifically for the reason you mentioned, Scott, is I want a specific version. I don't want to just, whatever dev is, who knows what that is, right? Yeah. Definitely. And notice that it's scoped. So if you remember when we looked at the deployment.yaml for the prod cluster, that annotation didn't exist. And furthermore, that those image update objects, those CRs, they also didn't exist. So over here in the prod cluster, I don't have any of the image update automation. But over in the staging cluster, I have all of the update logic that I've programmed in. And I've programmed it through these series of flux objects. And so like I said at the beginning, I'm not giving you a flux getting started. What I wanted to really help folks understand is that flux is presenting to you a programming model for programming your GitOps flows. And that I think is super cool. Right. Yeah. Almost like a library where it's like how do I implement. Or toolkit. GitOps. Toolkit. Exactly. Yeah. Which is cool. We got a question in chat here on the versioning thing. Is there a way to automate the versioning change from like 50X to 51X kind of deal? I'm not sure. A list like 50X and 51X. I don't know, right? So I don't remember all of the details, but the regular expressions, if you will, and I'm putting air quotes around that. The regular expressions that you have, the language for specifying these policy differences is quite rich. And I don't know all of the details on it, but I'm pretty sure that you'll be able to do that. You could express ranges. Right. There's all sorts of things that you can do there. Yeah. Because I was actually thinking about Regex, right? Like I could do like 5.0 through 1, right? Right. So either one would work. Yeah. That'd be interesting. There is an issue I'll paste in the chat where Michael Bridgen, who is leading the Image Update Automation project, has consolidated all of the open issues and proposals into one so that there can be progress on that level. We just talked about it in the Flux Dev meeting today. And that would probably be the best place to follow. Because there's a redesign. There's some redesign elements being described in that too. But the goal is definitely to have policy ranges. I just don't know if it's multiples of those we'd have to ask Michael probably. Cool. Yeah. Excellent. So that, if you will, is roughly what I wanted to show today. I do want to come back and just emphasize it in one last picture. Is, and in fact, I'll go into present mode. I want to summarize what GitOps is. So what GitOps is, we've been talking about it. We've been talking about the fact that what we've brought is Git. And we've brought this reconciliation-based approach to the bridging the gap between Git and the running system. So the way that I sometimes speak to this slide is that Kubernetes brought us the bookends. Kubernetes brought us declarative configuration and reconciliation of the containers. And of course, containers was just use case one for Kubernetes. We do so for other things as well. What GitOps does is it brings in, well, if you have that declarative configuration, you need to manage it in a sensible way that supports things like collaboration around around deployments into prod, like we saw, we had a merge request or a pull request. So it brings that management of the declarative configuration, collaborative management of declarative configuration. And then it also allows you a, and what I wanted to show to you today is the model, the programming model, for bridging the gap between Git and what is running in the final system at the end. And so that's what we've been doing today is really talking about that programming model. Now, I want to tee up a future conversation, which is what if everything that I showed you today, when I did, for example, all the way back here in this diagram, you'll notice that I had a parenthetical remark that said progressive deployment into prod. Now, Kubernetes does a little bit of a progressive deployment. It will, if you've got a deployment, if you have multiple instances, it would take down just one of them and incrementally bring you over to the new instances. And if something goes wrong with the first one, it will pause. But what if you wanted something more sophisticated? What if you wanted a canary style deployment? Or you wanted to do a blue-green deployment? Or you wanted to do something more sophisticated in your prod deployment? Well, you can carry forward programming your GitOps flow so that I don't just deliver to Kubernetes directly. I am going to deliver to Kubernetes, but then I'm also going to program in something like Flagger. And Flagger, it doesn't replace the deployment that is in Kubernetes. It dovetails on top of that. It says, okay, while this deployment is happening, so once it sees the deployment happening, it steps in and says, all right, let me do a little bit more control for you. And so what I wanted to do here was bring up this idea that GitOps flows are not just about delivery flows, they're delivery and operational flows. And what we want to do is provide you a programming model for bringing those all together. Nice, nice. All running on OpenShift. All running on OpenShift. That is definitely the peanut butter and chocolate. Yeah, that's definitely it. Cool, cool. Carl Henrich Lund says here, Flagger looks super awesome. I just tried it last week. Awesome. Folks are wondering, I'll drop a link for Flagger here in chat. Yeah, big plus ones from people for Flagger. And Flagger, by the way, Flagger is part of Flux. So Flux is a CNCF open source project that just reached incubation status. Just saw that. Well done, Scott and team. And Flagger is part of that. Flagger is part of that project. Awesome. Cool. Yeah. Cool. All right, all right. Great, awesome. Yeah, this is awesome. Let me see here. There might be a couple more things you want to do. Oh, yeah, so Hilary just asked us a nice question here. You want me to, I'll go ahead. You want to read it out? Who reads it out? I'll do it. A question from an SRE point of view. Let's say I've got something that's come up in production and we want to apply a patch, but for whatever reason, I may not want to patch. I may not want that patch to roll back in and deploy the rest of the fleet. Can I tell Flux to ignore things? Perhaps I'm temporarily shifting resource configs for a migration as an example. Yeah, can you like say, hey, Flux, take a break for a little bit? You can. When I mentioned earlier that you can pause and resume reconciliation on a per delivery type basis. I don't know if the language is really, it's fairly precise, but we could maybe come up with some better terms than that. But there are different delivery controllers. Right now we have the customized controller, which can either do customized overlays or customized smart sophisticated customized things or just playing YAML. And we have the helm controller. And so you can tell each of those either customizations or helm releases to just go ahead and take a break. Nice. It's important for break glass as well. It's very important for, you know, to note that human operators aren't having their jobs taken away from them. They're just having more of that time freed up to do more creative things. Yeah, that's right. More problem solving capabilities being handled for you, right? And when push comes up, you can still be like, oh wait, hang in a second. I'm not going to have the machine run wild on me. And I'm like, ah, it's Hal. Yeah, exactly. So to some extent, Hilary was saying, what if I'm not nefariously going in and creating drift accidentally or intentionally, but I just have to do a kubectl apply? How can I, and I need that to live longer because it's going to be the thing that brings my application back up into a running state because something was dorked up somewhere else. You can essentially pause this. You can pause this part and say, you know what, those changes right now, remember that 10-minute mark? You can pause that 10-minute reconciler and say, yep, nope, don't do it right now at all. Not even 10 minutes. That's right. Yeah, awesome. Very good details on Hal. You know, so I won't get into some of the tips and tricks right now, but yeah. Sorry, go ahead. I was going to say, production's down. I need to do something real quick. Let's just pause and then figure it out and then you can resume the reconciliation once you figure things out. So that's really powerful. You get how the API is down. I mean, yeah, exactly. Yeah, it could be anything. Yeah, a lot of your control is here. I saw somebody recently doing a demo of Flux where they didn't know about the pause thing. And so what they did was they scaled the number of instances of that particular reconciler, like the customized controller. They scaled it down to zero. The problem with that is that that's pretty global scope. So it's any GitOps flow that that customization controller is controlling just got paused. But I want to scope it to a particular flow. I want to say that, you know what? This particular microservice I want to pause. I want to stop all of the reconciliation. I want to stop the GitOps flow because I have something that I need to do urgently on the server. But all of my other microservices should continue running using their GitOps, continue being managed by their GitOps flows. So that pause is extremely powerful. It's doing something that you couldn't do by just issuing commands against the Kubernetes objects. Oh, definitely. That's one reason why people should migrate from Flux 1 to Flux 2 as well because that was the old way. Got it. All right. So yeah, we are a few minutes over, which is fine. I told for those of you watching, I told Cornelia, you can go as long as you want. All this stuff is really cool. But yeah, is there anything else you want to show us? Something you want to tease? I know we got you guys back for the next round. Is there any cliffhangers you want to leave out for the viewers or anything? This was one teaser. If y'all want, we can show you how Flagger folds into this whole picture. So we can go all the way back to this picture and actually do a little bit more around that parenthetical progressive. The other thing that I think that would be really valuable is Scott and I had talked about having Scott do a bit more of a, all right. So Cornelia showed you Flux was already installed. She was showing you some of these components that were already created, showing them to you in the repository. How do you actually get all that set up? So that's certainly something that we can do as well. You know what, there's actually a good question, which I think I will modify slightly, right? So the question was actually, how much work would it be to add an Opusheff route support to Flagger? But I think that extends more to how does Flagger handle things like CRDs and can it be extended to do things with CRDs? So that might be another teaser, right? When you guys get into... Yeah, I think talking about Flagger is a really great idea. Yeah, definitely. Super cool stuff. I have one announcement to make. Awesome. Oh, yeah, the announcement. Sorry, Scott. Yes. We definitely have to do the announcement. This is why we have Scott on, because this is the second thing he reminded me to do today. So go ahead, Scott. I'll let you have the floor there. Okay, sure. I mean, we can flip for it. But yeah, so in Opscon EU 2021 is a day zero co-located KubeCon and CloudNativeCon event. So those who've registered already may have seen it and may have signed up for it or maybe you didn't. Those who haven't, please do that as soon as you can. And the most important thing I'm mentioning is not to say come to this, but I mean, of course, if you're here, you're probably interested. But the CFPs, the calls for proposal or for participation proposals, whatever, are open through April 16th. So it's open now. It was open as of several days ago. We announced it on Twitter. We blessed it on the mailing list and it's Slack and on GitHub. But this is one other channel to let all of you fine folks know. So I pasted the link in- Yeah, we covered it over. So we just shared it. Yeah, so we're pretty excited. We'll be there, right? The fine folks at WeaveWorks will be there. CFPs, we're collecting CFPs. This is a GetOps working group in the CNCF. We're putting this on. But the fine folks at WeaveWorks and Red Hat is co-sponsoring it, putting this together. But it's open for everyone who's coming to QCon doing a DayZero event. There we go, Chris says it. Actually, sorry, Cornelia has it up on the screen. So yeah, so if you guys, if you have an idea, feel free. You know, we have lightning talks. We have regular tracks. And it's going to be a great event. It's going to be really cool. So I hope to- Yeah, I'm excited. Yeah, it's going to be really cool. It's going to be hosted by none other than Chris Short. So you get the same voice. Yeah, the same voice. Thank you, Chris. You're quite welcome. I'm happy to do it. Yeah. So cool. So yeah, where can folks find you at Twitter or, you know, any social media, any GitHub, anything you guys want to share out there? Twitter, GetOps, WG. That's right, GetOps, WG. Yeah, let me grab that. Putting that in the chat. In the context of the CFP, but I know we also have other Twitters. Yep, which we're already tweeted out. So my Twitter handle is cdavisafc. And Scott, you are found at- That's B-Y. It's one of those old school elite names. It's for my last name, but it's all switched together with a number in there. Cool. There we go. We just shared it out. Yep. All right, cool. So thank you very much for coming on. This is very informative. The audience loves it. And, you know, the questions coming in were great. Thank you very much, audience, for participating. And this, we can't wait to have you back. And also, obviously, GetOps.com. Yep, very good. All right, thanks all. Thank you. Everybody stay safe out there, and we will see you later. Or do you have anything, Scott, before I re-jump? No, no, no. Awesome. That was a clap. That was a clap. Got it. Thank you. That wasn't a handout. That was a clap. This clap thing for Cornelius demo. But, and also. Yes, great work, Cornelius. Thank you. Stay safe out there, everybody. See you soon.