 And we're gonna do Q&A and a panel right now. So I'm gonna let Christian do all this and track down the other remaining speakers. But Christian, just take it away. And C-MAC, thanks for joining us today. Are you still in Sweden? C-MAC, are you back in Canada? I am still in Sweden. Yeah, Canada was too cold for me, so I came to this side. You also came to somewhere else as well. To Sweden. Sweden is nice, okay? Canada's pretty nice too. But yeah, and Pinky has joined us, so wonderful. Pinky, where are you hiding out these days? I'm in Arizona, so yeah. It's pretty pleasant here. Awesome, guys. Well, well done, C-MAC. And I will see if I can trace down Jamie Forrest in the backstage. I'm gonna leave you all in the capable hands of Christian. So take it away. Yeah, yeah, so actually, you know, Pinky kind of joined us. We kind of added her last minute. So, you know, we kind of were like, okay, well, she can do the panel. We had room on the panel. So real quick, Pinky, if you can introduce yourself. I know we heard C-MAC and Cornelia earlier, but if you just quickly introduce yourself. Yeah. Yeah, so my name is Pryon Karavi. I also go by Pinky. And I have been a user of Flux and GitOps for I think two and a half years now. I actually started, I started with GitOps at State Farm, where I was on the platform, the GitOps platform team. And I recently just moved to WeWorks where I'm a DX engineer now. So thank you. Awesome, thank you. Yeah, yeah, it was awesome for those of you who weren't at GitOpsCon in North America. She made an amazing presentation. One of my favorite presentations there. So if you catch the playlist there, it's really, I always love hearing and user stories. So, you know, here at the end, we kind of try to get everyone together. I think, I think we couldn't get Dan because of his schedule, I'm not sure. I'm still trying to track him down, see hopefully he can make it, but we'll see here. To have kind of like a Q&A session, I'll kind of, for those of you who are here, feel free to drop a question either in the chat on the event tab or the stage tab. I keep switching back and forth. I'm not 100% sure where I should be, but I'm switching back and forth. So don't worry about that. I'll see you guys the questions. So I think I'd like to start since we gave all the other presenters time to speak. I think I'll start with Pinky, with my first question here, because you know what, let's just start with Pinky because she wasn't able to present. No, I'm your favorite. Yes, exactly. Exactly, yeah. Well, I mean, if we're gonna be honest now. So, you know, actually this question came to mind because I actually attended one of your talks. You did a WeaveWorks presentation and you know, you did a meetup, right? A GetOps meetup with WeaveWorks. And this question pops to mind and it relates to kind of the presentation I did. So GetOps seems really like a fully automated operation model for your entire platform, right? So do you think we can start getting the benefits of GetOps, even if like someone isn't 100% ready to implement all the principles or all the components? You know, if you're not really ready to implement maybe all of it, can you still kind of get the benefits from GetOps? Yeah, I actually got this question a lot during my time at State Farm. And so like some teams are just not ready. They're still using certain pipelines. There's Jenkins, because at that time we were using GetLab CI in our process. So yes, yes. Basically the first step is to obviously define all your deployments or your infrastructure, whatever you're doing, declaratively as code. So that in of itself comes with so many benefits, right? Because one, you know what your desired state is by just going and looking at it instead of all these manual like processes. And then also, it's something that's repeatable and reusable as well, right? So there's a lot of benefits to just doing that one step. And then the next step is just going and creating something like a config repo where you can actually store code there. And even if you're just doing manual deployments from there, like for the time being, that's still valuable. And then you can kind of get a pattern of which ones you're more doing like deployments for manually. And from there, you can actually take those projects first and then start like applying an operator such as my experiences with Flux, right? And so you can easily set an operator like that up and then just have it listening to your repo. And then, you know, from there, it's kind of like magic. Yeah, from there it rolls downhill, right? Like, oh, can't hold up. Yeah, and then you kind of just start hitting all of the principles, exactly. Yeah, it's natural after that, yeah. Yeah, it's almost like a natural progression. It's like, oh, once I start doing this and then the next thing becomes easier and easier and easier, right? Right. I'm kind of curious, Jamie, if you kind of had the similar, you know, she was at State Farm right here at University of Michigan, were you, did you do that same thing, right? Did you have that same kind of like, we're gonna do this one small thing first and then, you know, it kind of just rolled after that? Yeah, exactly. I think that it's helpful to start out with those small pieces. Look for things that the DevOps or devs are repeating. Look for things that they do over and over. Look for things that just require a small change like per environment, right? Like, you just need this variable change or you just need whatever, you know? And build it out bit by bit. And that's the technical aspect. The other thing is the cultural stuff. And we've touched on that a little bit here today, but you have to work with developers. If you're the one just doing the DevOps stuff, work with developers, find out what their workflow is. Find out how you can nurture this transition by automating things that will make it easier for them. Make it, try to find ways in which you can show them benefit if you're doing a big culture change, basically. Yeah, right, exactly. And I think doing that and then the things that I touched on at the end of my presentation, you know, folks can check back on that as well as there's some great tips in there in terms of some of the technical things that you can do. Cool, cool. There's actually a question coming in. I'm actually gonna pause that question because I think I originally had this question for Dan, but I think I'm gonna ask it to Cornelia because, so sorry to put you on the spot here for a question that was meant for Dan. That's what a panel is. That's what a panel is. But since you were involved in the process of the GitOps principles, you and I were in that meeting, ton of those meetings together. I'm kind of curious. So like what considerations were taken into account when the GitOps principles were being drafted, right? I kind of know the answer to this question, but I kind of wanna get your input, right? Like we'll kind of, I'm kind of think, so there were a lot of opinions, which means a lot of obstacles. You know, what were some of those obstacles that you saw that we were trying to overcome when developing those principles? Yeah, that's a really great question to dovetail from the one that you were just talking about, which is what consistently comes up is, hey, do I have to get all the way to the Holy Grail? And so I just wanna add on to what the other two participants already said and that is, even if you're not doing GitOps, adding automation, hugely valuable, do it. Yeah, doesn't mean don't automate it, right? And so I think the most contentious thing when we started building the principles was, do we really need to specify, for example, that it has to be convergent? Or do we really need to specify the principle that Git is the only way for you to affect things? And the tension that we fell into is that it's easy to in the beginning say, okay, well, okay, we'll dilute the definition of GitOps so that people don't feel, so people feel like they have a way of getting started. And in the end, we decided that there were a certain set of principles that were so essential that we wanted to be that prescriptive. Now, again, that doesn't mean you can't get on a path to getting to full GitOps and still realize value. So I think that was the biggest fundamental tension was do I really need a convergent system? Can't I just check things into Git and have a script run and call that GitOps? And the answer is no, let's not call it GitOps, but yes, please do it. If what you're doing today is clicky-clicky interfaces by all means create a script that triggers off of a Git check-in. But we really want you to get the ultimate values, which comes only when you embrace those four principles, or maybe it's five by now, but we really tried to stay very kind of axiomatic so that we weren't adding things that didn't provide additional value. We tied those things to value. Yeah, I think you touched on a very interesting point. This is like the value, right? The value you get from a lot of this automation, I think very, very early on in the GitOps working group is like, well, we don't wanna exclude people from the club, but also we do want to define what GitOps is. So I think distilling that into principles of version controlled and immutability doesn't necessarily mean Git, even though it's GitOps, right? Like we talk about GitOps, that's because we didn't wanna exclude other methods of doing, like for example, like object storage, right? That is immutable and that is version. You can version those. So I think that's very, very great. Siamak, I know it's late for you and I don't wanna ignore you, but there's questions coming up. So I think maybe I'll start this question with you. So these are, I'm not putting you in a weird position. This is just the way the dice rolled, right? The way the dice rolled in the questions, right? So maybe you don't know. The question comes from Jeffrey too about using GitOps at the edge, right? And so what, first of all, like how does that fit very well in terms of using GitOps at the edge? And second, are there concerns about using GitOps at the edge? Yes, yes. So there are like a couple of aspects when it comes to edge-rich effects, pretty much any application that is running there, any service that is running there, including how GitOps work for. The first piece of that is a matter of scale. Why so? Like I spoke about GitOps picking up because of the number of clusters are growing and this isn't the numbers of 10, 50, 100. When we're talking about edge, then we are speaking about thousands, multiple, 10,000, 100,000 endpoints and devices. We talk about in-vehicle services and that changes the perspective of when we talk about GitOps, there is GitOps on the Kubernetes clusters, there is always the pull and push model conversation. There's a choice that customers have depending on their security and compliance or some of the other aspect that choose one or other when it comes to edge that choice really disappears. There's only one way to think about this. The connectivity is not always there. So the solution is to become more resilient of when that device cannot reach back home or the near edge centers of service that doesn't disrupt the service that is running on it and it doesn't prevent whatever engine or software or Argo CD or Fox or anything else that is running there, a small version of it and enabling the GitOps workflow for that device to pull the configuration and that doesn't get disrupted. We have seen similar cases with not age but from the connectivity perspective it has been similar with Argo CD being on cruise ships around the world. So they come in and out of network and that they have seen some of the issues that it can create and address in Argo CD. The mere fact of being so many reaching about back to a Git repo that can create other problems for the Git provider that is not easily surfaced in regular applications that are being deployed, right? If you have 10,000 instances of a device coming to a particular Git repo to pull their configuration that's not something that necessarily the Git providers are ready for or would be ready for. So the way that this works also might need to change on the GitOps part, on the engine that is doing this. So there are multiple aspects of that flaking infrastructure that an age device is and then the nature of the limited compute and spotting network that exists on this device is that changes some of the assumptions that we have within the existing GitOps tooling and those needs to change a little bit. Yeah, I would. Oh, go ahead, go ahead Cornelius. I would love to pile on a little bit on this. Just a moment ago, we were talking about are all the principles essential? And we said, no, yes, that we had a minimal set and we tied it to value. So what is one of the values that we were tying these principles to? Well, it's autonomy. It's independent system autonomy which is absolutely critical on the edge. We can't have an edge network where there's all sorts of dependencies between nodes and the edges, between the central hub and the edges. So GitOps is absolutely essential when it comes to doing things at scale. And Cia Mack, you talked about scale and size, but it's also obviously that distributed nature. And that's my last point to add onto this. We keep talking about Git being versioned immutable. There's another attribute of Git that is also very interesting, which is it's a versioned immutable distributed system. And so Cia Mack, you're absolutely right. We have to solve problems of how often are we cloning down to the edges and there needs to be throttling and things like that. But Git by its very nature already has a whole bunch of capabilities in it around disconnected, distributed, all of that stuff. And so Git being versioned immutable and distributed is a way that it serves these edge use cases. Yeah, and I was actually gonna... By the way, it relates to what I was gonna say, so perfect. Because I remember we did a panel at KubeCon and we kind of touched on a little bit about this, about Git, but first of all, I just like to say having Argo being deployed on a ship is just like perfect, right? Because it's like, it's like that's what he's named after a ship. But some of the things that we've been, in Jamie, you've been part of some of these conversations in the GitOps working group talking about the idea of Git mirrors and using payloads with respect to the source of truth, right? Because you can always just tar up your Git repo and it'd be cool to have that as your payload, right? To have for edge use cases, for disconnected use cases. So that's, you know, again, if you wanna be involved, the Open GitOps group, we meet every other Wednesday, OpenGitOps.dev, if you wanna learn more about that. So this next question, I love this next question because it's gonna go to both Jamie and Pinky because you guys are end users. So I'll pick on you, whoever wants to chime in first, go ahead, feel free. So there are, so I think maybe I should, this is by Anonymous, which makes it a little eerie. Someone anonymously asked this. So there are certain instances where configuring the cluster are not possible to declaratively or in this person's opinion, declaratively, right? Things like host subnets, ciders, machine and node get deleted, right? So some tasks are almost like you can't do them with declaratively, right? Like VMs, right? Like nodes, like networks. So, you know, I'm curious about, you know, both you Pinky and your time at Safe Arm and Jamie, how did you guys manage those, the things that weren't get-opsible, let's just say. So whoever wants to go first, go ahead. I'll let you guys step on each other. Who wants to go first? Do you wanna go ahead? I actually cheated a little bit and I had to ask Kingdon about this one. I was checking if Flux can do this and I'm pretty sure it can do exactly what they're saying. So you can patch resources and stuff like that. I don't know, Argo, obviously I'm limited to my experience with Flux, but yeah, I think this actually can be done and there are maybe some things you'd have to kind of an angle to do it, but yeah. So that was an interesting question because I had to get. So from our perspective, anything that can be pulled down in a manifest can be modified and applied back up, right? Customize is great for doing stuff like that. So check out the customize tool. The other thing is there are some tasks that we've found where maybe it isn't all declarative. Maybe there needs to be a tool that takes the declarative and applies it imperatively and there's just some instances like that where you're not gonna be reading directly from Git to and applying the manifest directly to the cluster and there will be something going in between. You just work with those scenarios, right? And you put in as much as you can and then have a pipeline that will run that tool to take the declarative resource and apply it in an imperative way if it needs to be done so. Basically as much as that can be done declaratively, it should be done declaratively for sure, but yeah, there definitely are some situations. I agree. Yeah, definitely. And there's also like other tools, right? Like we talk a lot about Argo, we talk about Flux, but there's things like Terraform, there's things like Ansible, that are like really good for that sort of thing, right? And so part of the question, they talk about like networking, well, you can technically manage your network configuration. So like I'm an old Cisco guy, right? Like you can have like your configuration of your Cisco switches, you can technically have that in Git and have Ansible apply those. Like it's not, you know. You mentioned Terraform and that's actually what we're doing right now is we have snippets of Terraform saved in the repositories with the respective apps that we're rolling out or for our cluster configuration repositories and then using the Terraform container to actually apply those in. So there's ways of doing a lot of that stuff. Yeah, there's a tool for pretty much everything. There's a tool for pretty much everything, that's right there. We actually have a Terraform controller coming soon too. I know a lot of users. I heard about that. Yeah, did you? Yeah, I did. So that'll be really cool. Of course it's right after we got done with all of the work we've been doing. Right? Yeah. And like this always works, right? I like this conversation about imperative versus eventually consistent. Realize that the work that say Kubernetes does to actually recover the state from, some divergence from actual and desired state. The code that's on the inside of that is imperative. For the most part, that code is Golang or it's Java or something like that. What we're trying to do with GitOps is we're trying to elevate the interface to the developer, the DevOps persona, so that they don't have to deal with that imperative, the hard imperative stuff. And so there is no rule that says you can't do some imperative as a part of your automation of the system. It's just that you're taking on that burden. And I know to some people who are used to imperative, going declarative feels like a burden, but in the end it ends up being so much easier because you're deferring that responsibility for the hairy hard imperative stuff. And imperative stuff is the hairy hard stuff relative to declarative. You're deferring that to a system that's good at that that will do it consistently, that won't make mistakes and all of that stuff. And that's a system like Kubernetes or GitOps or something like that. So this next question is again, also another great question because I honestly don't have an opinion on it. So let's just see what you guys think about it. So there's talk about pull versus push, right? So pulling a configuration versus pushing a configuration out, right? I know that Flux can do both. I know Argosity can do both the push and pull, but with respect to GitOps and with like respect to, this is general to everyone, right? So whoever wants to chime in, go ahead. What's your opinion between non-push versus pull? It doesn't have to be either or is this just something that is like whatever works for you? What's the consensus on that? Who wants to go first here? Who should I pick on? CMock, do you have an opinion? I can pick on CMock. I can start, definitely. So what we're seeing, like to put it like very explicitly, we don't have an opinion on it. Like what we're seeing that they're both models gets practiced a lot because each of them come with some advantages and some disadvantages. And there are some customers that can live with this advantage of either and go that way, right? So with the push model, what we hear from customers that they do like that central model of the single pane of glass that is from central place, they can see all the stages of everything that is happening across their field, what stages they have. It's one place that gives them like very simple visibility and they can live with having access from outside to all those clusters, right? And they have customers that this is an absolute no-no. There's no way that they can define a single piece of network somewhere in the infrastructure that can reach everything. So they cannot live with that and the pull model is the only way that they would work with. And we talked about edge where some use cases is just impossible to go the other way around. So from, I personally, I have no opinions that I just see. I see advantages and advantages for each of them and each of them fits certain use cases of the same. Yeah, I would just add to that. So in a very specific sense, there are limitations, physical network and also compliance. So if you're an organization that deals with compliance, there are some barriers that will define for you whether you're gonna do push or pull based on access and whatnot. So that's another thing to consider. So I'm in the camp of right tool for the situation, the right way for the situation. Yeah, Cornelia, Pinky, I don't know if you guys have any additional thoughts on that. I was gonna say patience too, I guess. Like, are you gonna be patient enough to wait for like, in terms of flux, like the sink and our bull to hit, right? Or are you, do you need it like out there like that? So that's another thing, depends. I think, yeah, I think either one. Right. My fellow panelists expressed everything that was in my head, so. Yeah, there you go. Yeah, so I think that is a consensus. I don't think I've ever seen strong opinions one way or another, I think since most tools can do both. It's one of those things where it's like, well, you know, whatever works for you. It's one of the, the one thing that I will add is, remember we talked about patterns earlier. It's a pattern that you need to decide on deliberately. And different patterns are gonna give you different characteristics. So just understand the pattern and then decide which way you wanna apply it. Yeah, for those that know me, I can hold very strong opinions, but sometimes my opinions will, those strong opinions will change depending on what you're doing, right? Sometimes I get asked a question, I go, well, tell me what you're doing. My opinion might change depending on what you're doing and how you're delivering it. Cause I, you know, it'll be a strong opinion, but it might change. So to, you know, to be cognizant of time, right? So, you know, to respect everyone's time, you know, we, you know, we, we don't want everyone sitting in front of the camera all day. Oh, that's just that's question. And I like to get a thought for each one of you. I don't know where you guys are with respect to in the screen, I may be pointing at myself, but I like to ask the question. Actually, this question was originally meant for CMock, but I'm gonna ask it to all of you. So you guys take, you know, a minute or two to answer. I will start with CMock and we'll go. So what's, you know, get off to so new, but I can't help to think about what's the future, right? So like, what's next? Where does get ups go? Where does, where does it take us? What's the next, you know, splat ops, you know, for a marketing phrase that's gonna come up. So where do you see, you know, we see the benefits of get ups. Where do you, you know, what's the future? What's that? If you can, you know, looking at your crystal ball, what do you, where do you think it's going? Hey, I wish I had that with me. But what I, what I think would happen to be frank is that on one side of thing, this adoption and boom around get ups is like related a little bit to the question you ask of what happens when things are not declared to you. I think that would, that is causing a little bit of change and pushing projects and products toward becoming more declarative. So that's one aspect of how I see the coming year would play out that because of the interest in this model of working, the products have to morph themselves. Projects have to morph themselves into something that plays a lot better with the, with the get ups workflow. And this is something that we at Red Hat have gone through. We work has gone through and I expect a lot of other products that do the same. There's a lot of talks about security as code and compliance and these technologies quite often are not really compatible with the get ups model. So that's one aspect that I see that things would change with projects adapting themselves to get ups, which is an interesting thing. And the other thing is that get ups is really despite being around for a couple of years, it is at in the very beginning of its adoption cycle or the hype cycle, whatever you call it, right? So it's still at the very beginning of that steep like uphill. And I think like throughout this coming year, we will start like seeing use cases that we have not thought of it as adoption is picking up within the type of customers that are like extremely enterprise ready, enterprise like their environments are completely air gap disconnected. Like really the type of application that you generally don't see on the SaaS services, right? And those use cases, I don't think are discussed or seen much within the get ups space yet. So I expect to hear a lot of those kinds of things from that type of practitioner, those are starting to adopt some of the principles that were discussed. Yeah, I sort of think it almost forces your hand, right? I think if I'm looking at my crystal ball, I'm saying that there's gonna be reemergence of DevOps because things like Kubernetes and get ups forces your hand into that. So Cornelia real quick, so take a couple of minutes. What do you think is next? Yeah, I'll make it even quicker than a couple of minutes. So first of all, I wanna clarify that I would call it the adoption cycle, not the hype cycle. I do think that we are still in the skinny part of the tail on crossing the chasm. I don't think we've crossed the chasm yet. I think that we, because we spend so much time in this space, we kind of dilute ourselves into thinking that this is more widely adopted than it is. I think we haven't crossed the chasm yet. So that's the first point. And that's the part B of that point is what is our, I'm gonna call it Kubernetes moment. We had convergence systems, container-based convergence systems before Kubernetes. We had Cloud Foundry. We had... Masosphere, we had the other one that I'm thinking of that Red Hat bought. Makara, you're talking about Makara? No, no, no, no. Passable. No, there's all kinds of tools, by the way, everyone. Yeah, yeah. And so we had those systems before, but like I said, I used to have to teach people about convergence systems. And now I don't have to do that anymore. So that happened with Kubernetes. That was, you know, Kubernetes obviously is a hugely successful open source project and we can spend a lot of time thinking about what were the elements that made it successful. But the fact is that we had a Kubernetes moment where there was a tipping point and we haven't definitely not hit that tipping point yet. So the future of GitOps is still looking for that tipping point. And then I think we'll see a whole lot more innovation in so many different ways. You know what, that other name's gonna come to you in the middle of the night and you're gonna be like, that's what I meant. I'm gonna Google it right now. Yeah, it always happens later, right? So next, Jamie, what do you think? So what do you think is? Core OS. Core OS, there we go. There we go. I didn't even have to Google it, I finally remembered it. Right, which is the foundation of Red Hat Core OS which is underneath OpenShift and Fedora Core OS which is underneath OKD for the nodes. I think the intelligence part is gonna come in. So right now we have people putting changes into Git, automated reconciliation between Git and the services. The intelligence, their machine learning artificial intelligence is gonna fill that in so it's not people, it's actually the machine intelligence making those changes to Git and then the reconciliation. So that's the next step. Yeah, pretty soon we'll all be out of a job. So like you really touched a nerve there with Cornelia cause she's now in the Alexa team so she's like, yes, exactly, right? Oh, I know. Yeah, exactly. It'd be like, hey, Alexa scale that cluster for me. It'll be like, she does it on her own because you know, she like, oh, notice is a... Right, and that's the same. I think you're right. Yeah, you can start cause machine learning can like predict when you'll have spikes in your cluster and it'll start preemptively scaling your cluster, let's say. So that's really interesting as there's that AI ML ops, you know, that... So that's really, really cool. I also see something like that coming up. So Pinky, I'll give you the floor at the end. Now it's yours, what do you predict? It's actually funny. I just wanna say my old teammate, Russ, he actually texted me the other day and he's like, oh, so, okay. So is Flux now implemented with Alexa yet? Can I tell? No, it's really funny. Oh, people have Pinky. Yeah, there you go. That's the future is what I see. No, I just think it's, for me, it's really hard to see like an end game, I guess, because there's just so many things that keep changing. When I first started, I don't know what just happened. Can you all still hear me? Yes. Okay, cool. So when I first started with GitOps, like progressive delivery wasn't even a thing that was really implemented yet for the whole process. And then, you know, things like Flagger came along. And so there's still things that are so cool and innovative that are still changing like what's possible with GitOps. And so,