 All right, all right. So yeah, so quick introductions, right? So Christian, my new Christian Hernandez again, so I am a product manager at Red Hat. I'm also a maintainer of open GitOps. So if you see, you know, you see that QR code quote. Hello? There we go. If you see that QR code of getting involved, like if you wanna ask someone directly, you can ask me, you know, any questions about that as well. I'm also a contributor to the Argo project. And Nick, Nick's here. Hi, I'm Nick Schutz. I'm a field engineer at solo.io. My specialty is in application networking. Previous to this, I was at Red Hat for quite a few years practicing Kubernetes and OpenShift, helping enterprises all over the place. Now I do that with application networking. Sweet, sweet. Yeah, so take it away, talking about Service Mesh. So Istio, Istio is an open source Service Mesh. It's also a CNCF incubating project as of recently. So the common Service Mesh functions are things like enforced MTLS, application resilience, observability, granular policy access control. Those are just some things that you can allow via Service Mesh with people's applications without having them change their application at all easily. So one of the challenges behind Service Mesh, or Istio as a Service Mesh is traditionally the deployment. And it's fine if you wanna do a Istio deployment on a single cluster perhaps in multiple namespaces and use their low level configurations to do so. But once you start getting into multi-cluster, you start to get some challenges. Things like communication between those clusters. Root trust between those clusters. And when that starts to happen, what's missing and lacking is that MTLS between the clusters like I was saying, an API that actually understands multi-cluster comprehensively and something that people consume at a higher level, a more comprehensive level. And that's where the GNU Mesh open source and enterprise product comes in. It allows for that abstraction at the multi-cluster layer or for multi-cluster deployments to a point where it makes it very easy to consume and also everything is done via declarative configs via CRDs. And that makes it perfect for things like GitOps. Yeah, and like I said, since all these configurations to manage your service mesh are all just like CRDs, YAML, that sort of thing, right? It's all Kubernetes native. It makes GitOps an Istio. I like to call it like peanut butter and chocolate, right? Goes better together, right? So as you can tell I'm a big candy guy. It just goes better together, right? And so if you are looking at your desired state, I'm gonna kind of use this term loosely. So if any security guys are here, don't get too mad at me, I'll get into that. But you can look at your desired state as almost like your policy, right? And when I say policy, I don't necessarily mean security. Obviously that's gonna be a big part of it. Part of the larger picture is security, but what I'm talking about policy is like your organizational policies, right? Like if you're thinking about, for instance, I'm just gonna get really, really simple, like your corporate standard build of your Linux VMs, right? Or whatever server that you're setting up. Those are actually policies, right? You have a policy that you want to apply to a system. So bringing that back up to GitOps is that you can view that source of truth, right? That Git that you want to apply, that you don't want it to drift as policy, right? And so that in kind of almost like a policy governor, almost right, you're thinking more, now you're thinking like kind of like more, as you can talk kind of like enterprise architect background, but like you think about more of it being like a generic governance model of like how you manage your system, right? So it doesn't, policy doesn't necessarily mean security, right? I'm not talking about like policy with respect to like authorization and authentication, although that's a bigger part of it. I'm talking about it mean more like holistically about your systems, right? And Service Smash is a powerful tool, right? And so as Nick was talking about, it's really there to help you scale and it's really, really powerful, but it gets really, really complex when you want to do it massively at scale as trying to look at it at a single pane of glass, right? And so that's kind of like where GitOps comes into play where you kind of want to like distribute that power and make sure you have that power and make sure you have ultimate control across all your systems there. And so which is why I kind of have that graphic is like GitOps and Istio, they work pretty well together, right? And this is, and when you're managing a large systems, right, you know, both Nick and I have like, we work with really, really large enterprises to try to like get some of this stuff out and observability becomes paramount very, it becomes really, especially with distributed systems, right? Especially with like really, really large enterprises and you have thousands and thousands of applications across thousands and thousands of clusters, you know, you get data with that with service mesh, right? And you get, you have to use that data. Now that you have that data, you know, the someone once said that what's the difference between data and information, right? And just information to just process data. Now that you have the data, now you need to have the information you need in order to do things like policy enforcement, right? And so I try to get a better graphic than this. I don't know if that makes sense, right? You have an infinite loop going between your configuration and your clusters, but the idea there is that your GitOps controller is kind of like your policy enforcer. Again, I'm kind of using these terms a little loosely because if there's any security guy there, it's like, well, it's not really enforcing any policy. I'll get to that in a second, but if you think about it, your GitOps controller, whether you're using Flux, or Argo, or whatever, you kind of think of it as like, I have this policy that I want to be able to deploy massively across my cluster, right? Like I want to be able to, and then what I mean policy, I kind of like am eluding to some of the Istio primitives of how to traffic controlling, right? And so, where GitOps really comes into play with respect to power is that it's preventing that policy from drifting, right? It's one thing to, and I'm kind of, I'm kind of skip bullet points right here. It's one thing to use things like Kyverno and OPA and other security aspects of it, right? Because when you're using stuff like Kyverno or OPA, that's really more on the like, traditional policy side aspect of it, where it's like you set something up and that controller is making sure that that is happening. However, there's nothing controlling if someone, some bad actor, some of that fingers are configured, someone changes the Kyverno or OPA configuration on your cluster, and now all of a sudden, it doesn't matter if you're running Kyverno, right? Like if someone's there changing the policy to something you don't want it to be, then how good is that policy, right? And so kind of jumping back up is like preventing that drift, right? You're gonna be using your GitOps controller to making sure that drift doesn't happen. You wanna set your policies and you wanna make sure that policy is always what you want it to be, right? And basically, also you wanna automate those policy changes, right? So if you ever change a policy, whether you put in a new constraint or maybe you widen it a little bit, you want that to automatically propagate massively to your entire cluster. And so another aspect of GitOps is what I really like about Istio and what I really like about Service Mesh is that it puts in things like progressive rollouts into, it codifies them, right? So there's a challenge with that. So I find that if things are dynamic in a Service Mesh world, right? I want, you want to be able to do something like canary deployments, right? I always talk about canary deployments. Those are really, really dynamic, right? You wanna release a new version of your application. You wanna release maybe some like 10% of it. Okay, that's looking good. Maybe release another 10%, so now you're at 20%. Okay, that looks looking good. How does that look like in the GitOps world, right? Because is that a series of, series of commits to a branch, right? And, you know, or a PR, you know, a bunch of, you know, commits to that branch or if you're, you know, really running GitOps, a bunch of PRs and like every PR someone has to look at that PR and merge it, right? And approving, you have to go through that approval process. For every time you wanna increase that canary, that canary's gonna take forever. And another way to do this kind of ignore those differences, but then that kind of goes against the spirit of GitOps. You know, you really want to have like a policy around that, right? And so if your source of truth is static, how do you do like a dynamic rollout, right? Really like embrace the power of a service mesh, right? And a lot of controllers, a lot of projects now have a controller for that. So for instance, in the Argo world, that's Argo rollouts, and I know Flux has Flagger to do some of those things as well. But you have another controller, progressive, that is GitOps friendly, right? That is basically saying, hey, given these parameters, right? I want you to start at 10% and increase progressively to 100% and until that rollout is successful, right? And it works with your GitOps controller. What's really cool about using something like Argo rollouts or Flagger is that it's also policy-based with respect to metrics, right? So you can plug into things like Prometheus, right? So like, I use this deal a lot, so I use Prometheus. You can tap into Prometheus and just run whatever query you want to in order to do the rollout. So now you kind of have like a policy, what's the word I'm looking for? You kind of have like a low-end, high-end sort of thing. I'm like, okay, well, given these parameters, do the rollout for me. And then it becomes automatic, right? And GitOps friendly. And it has things like rollback and, yeah, so basically like rollback, whereas if something goes wrong, Prometheus is catching errors like 500s or whatever, you're able to do that as well. So the idea behind using something like Istio service mesh and GitOps together is like you really wanna scale, right? So the idea is scaling. The idea is that, okay, we have, organizations are always talking about like, oh, you need to do less with more, we need to scale rapidly. You wanna be able to handle anything. Now you have like the tool sets to do it and now you have the tool sets to do it automatically, right? And so we have like things like federated meshes or multi-cluster, I forget, I don't know what the right term is, so it's called federated message cluster, right? So, and you manage complexity exponentially, right? So once you add like another cluster, like an Istio cluster, the problem grows exponentially, not just incrementally. So your apps could be really anywhere, right? And so GitOps basically just helps you manage that. Not only you manage that complexity, it lets you do other cool things like progressive delivery in a GitOps friendly way, right? And so one of the things, so I have, I don't know if you've seen the back of my shirt, if you wanna see it later, you can. Back of my shirt says, unless it's in Git, it's only a rumor. Another thing I like to say, so that's kind of like one of my cash phrases, well another thing I like saying is that automation does not mean automatic either, so everything basically, that phrase just means that you are ultimately in control of how that rollout happens is that, but what actually does the rollout is automatic. So that's kind of like what I mean from that aspect, right? And then, so you saw the image before, right? That kind of how it looks like here where you are not only using the glue mesh to help your federated clusters, right? Well now you have like a controller there as well to help you with the GitOps thing. I don't know if you have any. Yeah, so this isn't just theory. We have people, lots of companies in production today implementing these patterns very effectively for hundreds of clusters at a time. So it's a proven methodology that large scale companies use on a daily basis, and it's very effective and cost efficient. Sweet, sweet, yeah. And by the way, these slides, I just uploaded them to Sketch, so I see you taking pictures, which is fine, but just let you know we uploaded that there. So I guess with that, do we have time for, yeah, we have time for some questions here if anyone has any questions? Anybody? Anyone? Go back a slide. Yes. That is a question, so I, I understand. So I'll allow it, yes. Oh, there's a question back there. So if you do wanna learn more about Istio, just raw Istio, or even glue mesh, you can go to academy.solo.io, and we have some self-paced classes, or you can actually take a webinar-style class where it's interactive and you can have hands on the technology. So please do come and join us. You do them quite frequently, and thank you guys. Oh yeah, got a question right here? Yeah, no, so please. Is it on? Yeah. Oh, sorry. So a quick question. We're also using Istio with Flagger, and one interesting problem we had to solve, I think, is when you have Flagger, or RGCD rollout, you have this internal state of the rollout which managed by Flagger. It is GitOps-friendly, but you still have, like, failure, you still need to handle failure states and failure conditions, and it can a little, so it goes a little bit in a different direction as a GitOps, so your GitOps repository drift from the source of truth. So you can have different workload on different version, deploy it on your system, and a different version defined in the GitOps repository. So do you have any good cookbook solutions to solve it, or do you need to approach it yourself like we did? We had callbacks, automatic rollouts, and making sure we change the state inside the repository to reflect it correctly. Yeah, yeah, so I don't know, so for Flagger, I'm not sure, right? So I know for rollouts, specifically, that since it works hand-in-hand with RGCD, it'll mark the deployment as degraded, and so meaning that just your deployment is just degraded, so it's no longer in sync with Git, it's no longer, it's basically marking your app as it's not working, right? And so, which I actually talked to someone once, it's like, well, that's not technically true, because it rolled it back, so it is healthy, but technically, but it does show it. I do think, though, that there's room for improvement because to recover for something like that is very imperative, like right now, even with the Argo rollouts, you have to do some imperative work, right? So to automate that, you may need another controller, but with... Also, in GlueMesh, there's a validation webhook as well that won't allow you to apply configs that are bad, and you can set that to strict, and you can also use OPOP policies to check your configs, et cetera, in tandem with that. So, not sure if we answered your question, other than I agree that there needs to be a better way to handle either rollbacks or remediation of that. Just a quick question about the diagram, I guess, and maybe it kind of follows that question about whether things are really get-ups or not, but so the top part of the diagram shows GlueMesh being synced with Argo CD, I guess being synced with Git, those GlueMesh CRDs, I guess being synced with Git, but then there's the arrow about GlueMesh creating or translating Istio resources and the individual clusters, right? Are those Istio resources? Does their configuration ever show up in Git, or is it only in the mind of GlueMesh, if you understand what I mean? It would be GlueMesh's abstraction of the Istio resources, yes. Okay. Yeah, and also it's very similar with Argo rollouts, right? So you see part of Argo rollouts, and again, I'm ignorant on Flagger, so I'm not sure if it's the same with Flagger, but with Argo rollouts, you're not actually even using a deployment either because you're using the Argo rollout specification which has the same spec as a deployment. So it is layers of abstraction because you have a rollout that controls your replica set versus a deployment. So very similar to Glue, whereas you're using the Glue translation of Glue, will then create those resources. It's all about making the user experience easy and very readable in the end, to reduce human error, to allow replication easily, et cetera. So if I go back to an old version of the GlueMesh CRDs, it should put back the Istio resources I had before. So in general, that's the get-offs idea, right? Yes. Okay, thanks. Any other questions? Hi, it's pretty much related to the last question. It's about the sync or the drift detection on the Istio clusters. Because in this case, if I understand, we are going to synchronize the Glue resources through Argo CD in this case, and GlueMesh is going to create the Istio cluster, right? But if there is some drift on the Istio clusters, like at the global failover or the WAF configuration or whatever, that drift is going to be picked up by the GlueMesh or the Argo CD sync. The Argo CD will detect that, the drift, right? I'm not sure if there's a tool in Glue that detects drift, but I think... So in this case, it wouldn't definitely be Argo looking after the resources that have been deployed, et cetera, for changes. Okay, yeah, my question was because I really don't... I haven't used this combination, so I haven't seen how it looks on the Argo CD UI. Yeah, so in just to kind of pull the curtain back a little bit, in Argo CD, so for Istio, I'm gonna use some Istio nomenclature. You have your virtual service, right? Usually you'll have your virtual service like pointed certain percentages. You actually point at the stable, 100% at the stable release. So in Git, it's more of your storing, like your end state in Git for the virtual service. Now the Argo controller takes care of, sorry, the Argo rollout controller takes care of what points the stable, right? The stable, and I'm using stable in quotes because that's just like, they call it the stable release versus the canary release, right? It's an actual specification, it's actually called that. So, but the, yeah, so the Argo rollout itself is the one that looks at what your end state wants to be and then shifts the traffic accordingly, right? So it'll mark individual replica sets as stable versus canary, and you kind of, you're kind of giving a lot more control over to the rollout controller, right? Versus, and in Argo, you just have your end state defined, like what do I wanna see as my end state? Yeah, and really GitOps is about, you know, having a process behind it, right? Having that gateway into your configuration. So in an ideal world, you would have, checks and balances before that drift can even happen. So it's a theoretical question. Doesn't this break the principles of GitOps in that Gluemesh translates the source of truth? Yeah, so it's, this is actually, and funny enough, there's a conversation I had with Christian Posta, who's at solo as well. So getting into just like the get-offs aspect of it, the, it's points of demarcation, as I always say, right? Cause if you technically think about it, a deployment isn't really your, right? Like isn't really what runs, right? It's the pods, right? So is storing pod definitions more get-offs than storing a deployment? Or are you just storing what your end state, what your end state wants to be, right? And so it's really your points of demarcation, right? Like so, and then there's like a balance between that, right? And do it like, if you write like a Kubernetes controller that is like three lines of YAML, but like has a massive deployment, like that's technically get-offs because it's like you just have a, you know, you're just selling the controller what you want to do or do you store those individual pod YAMLs and the individual service YAMLs, all of that and get, it's something we've actually discussed at the get-offs working group, right? This concept of like soft get-offs and hard get-offs where, and where that point of demarcation is, where I come, where my point of view is like your point of demarcation and where the controller lets go of that and where another controller picks it up, right? And where do you want those to be? Yeah, I mean, where I'm coming from is more in terms of repeatability and consistency. So to me, if Gloomesh changes in a future release, it might change your end service. True, but that would be the same with Istio in general. That abstraction is needed. I mean, really it comes down to, are people going to use it, right? Are they going to be able to consume it? And in the end, we're trying to make that easier. Cool, we have about two more minutes, so if there's any more questions, awesome. Thank you. Cool, thank you everyone.