 All right, all right. So, yeah, so quick introductions, right? So, Christian, my new Christian Hernandez again. So I am a product manager at Red Hat. I'm also a maintainer of open get-ups. So, if you see, you know, you see that QR code quote. Hello? There we go. If you see that QR code of getting involved, like if you wanna ask someone directly, you can ask me, you know, any questions about that as well. I'm also a contributor to the Argo project. And Nick, Nick's here. Hi, I'm Nick Schutz. I'm a field engineer at solo.io. My specialty is in application networking. Previous to this, I was at Red Hat for quite a few years practicing Kubernetes and OpenShift, helping enterprises all over the place. Now I do that with application networking. Sweet, sweet. And yeah, so take it away, talking about Service Mesh. So Istio, Istio is an open source Service Mesh. It's also a CNCF incubating project as of recently. So the common Service Mesh functions are things like enforced MTLS, application resilience, observability, granular policy access control. Those are just some things that you can allow via Service Mesh with people's applications without having them change their application at all, easily. So one of the challenges behind Service Mesh, or Istio as a Service Mesh is traditionally the deployment. And it's fine if you wanna do a Istio deployment on a single cluster perhaps in multiple namespaces and use their low level configurations to do so. But once you start getting into multi-cluster, you start to get some challenges. Things like communication between those clusters, root trust between those clusters. And when that starts to happen, what's missing in lacking is that MTLS between the clusters, like I was saying, an API that actually understands multi-cluster comprehensively and something that people consume at a higher level, a more comprehensive level. And that's where the GNU Mesh open source and enterprise product comes in. It allows for that abstraction at the multi-cluster layer or for multi-cluster deployments to a point where it makes it very easy to consume and also everything is done via declarative configs via CRDs. And that makes it perfect for things like GitOps. Yeah, and like you said, since all these configurations to manage your service mesh are all just like CRDs, YAML, that sort of thing, right? It's all Kubernetes native. It makes GitOps an Istio. I like to call it like peanut butter and chocolate, right? Goes better together, right? So as you can tell I'm a big candy guy. It just goes better together, right? And so if you are looking at your desired state, I'm gonna kind of use this term loosely. So if any security guys are here, don't get too mad at me, I'll get into that. But you can look at your desired state as almost like your policy, right? And when I say policy, I don't necessarily mean security. Obviously that's gonna be a big part of it. Part of the larger picture is security, but what I'm talking about policy is like your organizational policies, right? Like if you're thinking about, for instance, I'm just gonna get really, really simple, like your corporate standard build of your Linux VMs, right? Or whatever server that you're setting up. Those are actually policies, right? You have a policy that you want to apply to a system. So bringing that back up to GitOps is that you can view that source of truth, right? That Git that you want to apply, that you don't want it to drift as policy, right? And so that in kind of almost like a policy governor, almost right, you're thinking more, now you're thinking like kind of like more, as you can talk kind of like enterprise architect background, but like you think about more of it being like a generic governance model of like how you manage your system, right? So policy doesn't necessarily mean security, right? I'm not talking about like policy with respect to like authorization and authentication, although that's a bigger part of it. I'm talking about it being more like holistically about your systems, right? And service mesh is a powerful tool, right? And so as Nick was talking about, it's really there to help you scale and it's really, really powerful, but it gets really, really complex when you want to do it massively at scale is trying to look at it at a single pane of glass, right? And so that's kind of like where GitOps comes into play where you kind of want to like distribute that power and make sure you have that power and make sure you have ultimate control across all your systems there. And so which is why I kind of have that graphic is like GitOps and NSTO, you know, you, they work pretty well together, right? And this is, and when you're managing a large systems, right, you know, both Nick and I have like, we work with really, really large enterprises to try to like get some of this stuff out and observability becomes paramount. Very, it becomes really, especially with the distributed systems, right? Especially with like really, really large enterprises and you have, you know, thousands and thousands of applications across thousands and thousands of clusters, you know, you get, you get data with that with service mesh, right? And you get, you have to use that data. Now that you have that data, you know, the, you know, the, someone once said that what's the difference between data and information, right, and just information to just process data. Now that you have the data, now you need to have the information you need in order to do things like policy enforcement, right? And so I try to get a better graphic than this. I don't know if that makes sense, right? You have an infinite loop going between your configuration and your, your clusters, but the idea there is that your Github's controller is kind of like your policy enforcer. Again, I'm kind of using these terms a little loosely because if there's any security guy there, it's like, well, it's not really enforcing any policy. I'll get to that in a second, but, but if you think about it, your Github's controller, whether you're using Flux or Argo or, you know, whatever, you, you know, you kind of think of it as like, you know, I have this policy that I want to be able to deploy massively across my cluster, right? Like I want to be able to, and then when I mean policy, I kind of like am eluding to some of the, the Istio primitives of how to, you know, traffic controlling, right? And so where Github's really comes into, comes into, into play with respect to like power is that it's, it's preventing that policy from drifting, right? It's one thing to, and I'm kind of, I'm kind of, kind of skip bullet points right here. It's one thing to use things like Kyverno and, and OPA and, you know, other security aspects of it, right? Because like, you know, when you're using stuff like Kyverno or OPA, that's really more on the, like traditional policy side aspect of it, where it's like you set something up and that controller is making sure that that is happening. However, there's nothing controlling if someone, some bad actor, someone that fingers a config, someone changes the Kyverno or OPA configuration on your cluster. Now all of a sudden it doesn't matter if you're running Kyverno, right? Like if someone's there changing the policy to something you don't want it to be, then, you know, how good is that policy, right? And so, you know, kind of jumping back up is like preventing that, that drift, right? You're gonna be using your get-offs controller to making sure that drift doesn't happen. You wanna set your policies and you wanna make sure that policy is always what you want it to be, right? And basically also you wanna automate those policy changes, right? So if you ever change a policy, whether you put in a new constraint or maybe you widen it a little bit, you want that to automatically propagate massively to your entire cluster. And so another aspect of get-offs is what I really like about Istio and what I really like about Service Mesh is that it puts in things like progressive rollouts into, it codifies them, right? So there's a challenge with that. So I find that if things are dynamic in a Service Mesh world, right? I want, you know, you want to be able to do something like canary deployments, right? I always talk about canary deployments. Those are really, really dynamic, right? You wanna, you know, release a new version of your application. You wanna release maybe some, like 10% of it. Okay, that's looking good. Maybe release another 10%, so now you're at 20%. Okay, that looks looking good. How does that look like in a get-offs world, right? Because is that a series of commits to a branch, right? And, you know, or a PR, you know, a bunch of, you know, commits to that branch or if you're, you know, really running get-offs, a bunch of PRs and like every PR someone has to look at that PR and merge it, right? And approving, you have to go through that approval process. For every time you wanna increase that canary, that canary's gonna take forever. And another way to do this kind of ignore those differences, but then that kind of goes against the spirit of get-offs. You know, you really want to have, like, a policy around that, right? And so, if your source of truth is static, how do you do, like, a dynamic rollout, right? Really, like, embrace the power of a service mesh, right? And a lot of controllers, a lot of projects now have a controller for that. So, for instance, in the Argo world, that's Argo rollouts, and I know Flux has Flagger to do some of those things as well, but you have another controller, progressive, that is get-offs friendly, right? That is basically saying, hey, given these parameters, right, I want you to start at 10% and increase progressively to 100% and until that rollout is successful, right? And it works with your get-offs controller. What's really cool about using something like Argo rollouts or Flagger is that it's also policy-based with respect to metrics, right? So you can plug into things like Prometheus, right? So, like, I use this deal a lot, so I use, you know, Prometheus. You can tap into Prometheus and just run whatever query you want to in order to do the rollout. So now you kind of have, like, a policy, what's the word I'm looking for? You kind of have, like, a low-end, high-end sort of thing. I'm like, okay, well, given these parameters, do the rollout for me. And then it becomes, like, automatic, right? And get-offs friendly. And it has things like rollback and, yeah, so basically, like rollback, whereas, like, if something goes wrong, Prometheus is catching errors, like, 500s or whatever, you're able to do that as well. So the idea behind using something like Istio, service mesh, and get-offs together is, like, you really want to scale, right? So the idea is scaling. The idea is that, okay, you know, we have, you know, organizations are always talking about, like, oh, we need to do less with more, we need to scale rapidly, you want to be able to handle anything. You have, now you have, like, the tool sets to do it, and now you have the tool sets to do it automatically, right? And so we have, like, things like federated meshes, or multi-cluster, I forget, I don't know what the right term is, so it's called, federated message cluster, right? So, and, you know, you manage complexity exponentially, right? So once you add, like, another cluster, like an Istio cluster, the problem grows exponentially, not just, you know, incrementally. So your apps could be really anywhere, right? And so, get-offs basically just helps you manage that. Not only manage that complexity, it lets you do other cool things, like progressive delivery, in a get-offs-friendly way, right? And so, one of the things, so I have, I don't know if you've seen the back of my shirt, if you want to see it later, you can. The back of my shirt says, unless it's in git, it's only a rumor. Another thing I like to say, so that's kind of like one of my cash phrases, well, another thing I like saying is that automation does not mean automatic, either. So, you know, everything basically, that phrase just means that you are ultimately in control of how that rollout happens, is that, but what actually does the rollout is automatic. So that's kind of like what I mean from that aspect, right? And then, so you saw the image before, right? That kind of how it looks like here, where you are not only using the glue mesh to help your federated clusters, right? But now you have like a controller there, as well, to help you with the get-offs thing. I don't know if you have any. Yeah, so this isn't just theory. We have people, lots of companies in production today, implementing these patterns very effectively for hundreds of clusters at a time. So it's a proven methodology that large-scale companies use on a daily basis, and it's very effective and cost efficient. Sweet, sweet, yeah. And by the way, these slides, I just uploaded them to Sketch, so I see you taking pictures, which is fine, but just let you know we uploaded that there. So I guess with that, do we have time for, yeah, we have time for some questions here, if anyone has any questions? Anybody? Anyone? Go back a slide. Yes. That is a question, so I, let's do that. So I'll allow it, yes. Oh, there's a question back there. So if you do wanna learn more about Istio, just raw Istio, or even glue mesh, you can go to academy.solo.io, and we have some self-paced classes, or you can actually take a webinar-style class where it's interactive and you can have hands-on the technology. So please do come and join us. You do them quite frequently, and thank you guys. Oh yeah, got a question right here? Yeah, no, so please. Is it on? Yeah. Oh, sorry. So a quick question, we're also using Istio with Flagger, and one interesting problem we had to solve, I think, is when you have Flagger, or RGCD rollout, you have this internal state of the rollout which managed by Flagger, it is GitOps-friendly, but you still have, like, failure, you still need to handle failure states and failure conditions, and it can a little, so it goes a little bit in a different direction as a GitOps, so your GitOps repository drift from the source of truth, so you can have different workload and different version, and deploy it on your system, and a different version defined in the GitOps repository, so do you have any good cookbook solutions to solve it, or do you need to approach it yourself like we did, we had callbacks, automatic rollouts, and making sure we change the state inside the repository to reflect it correctly? Yeah, yeah, so I don't know for Flagger, I'm not sure, right, so I know for rollouts specifically that since it works hand in hand with RGCD, it'll mark the deployment as degraded, and so meaning that just your deployment is just degraded, so it's no longer in sync with Git, it's no longer, it's basically marking your app as it's not working, right, and so I should actually talk to someone once, I was like, well, that's not technically true because it rolled it back, so it is healthy, but technically, but it does show it, I do think though that there's room for improvement because to recover for something like that is very imperative, like right now, even with Argo rollouts, you have to do some imperative work, right, so to automate that, you may need another controller, but with... Also in GlueMesh, there's a validation webhook as well that won't allow you to apply configs that are bad, and you can set that to strict, and you can also use OPA policies to check your configs, et cetera, in tandem with that. So not sure if we answered your question other than I agree that there needs to be a better way to handle either rollbacks or remediation of that. Thank you. Just a quick question about the diagram, I guess, and maybe it kind of follows that question about whether things are really get-ups or not, but so the top part of the diagram shows GlueMesh being synced with Argo CD, I guess being synced with Git, those GlueMesh CRDs, I guess being synced with Git, but then there's the arrow about GlueMesh creating or translating Istia resources and the individual clusters, right? Are those Istia resources? Does their configuration ever show up in Git or is it only in the mind of GlueMesh, if you understand what I mean? It would be GlueMesh's abstraction of the Istia resources, yes. Okay. Yeah, and also it's very similar with Argo rollouts, right? So you see part of Argo rollouts, and again, I'm ignorant on Flagger, so I'm not sure if it's the same with Flagger, but with Argo rollouts, you're not actually even using the deployment either because you're using the aren't on Argo rollout specification, which has the same spec as a deployment, so it is like layers of abstraction because you have a rollout that controls your replica set versus a deployment, so very similar to Glue, whereas you're using the Glue translation of Glue will then create those resources. It's all about making the user experience easy and very readable in the end, to reduce human error, to allow replication easily, et cetera. So if I go back to an old version of the GlueMesh CRDs, it should put back the Istia resources I had before, but so in general, that's the GitOps idea, right? Yes, okay, thanks. Any other questions? Hi, it's pretty much related to the last question. It's about the sync or the drift detection on the Istio clusters, because in this case, if I understand, we are going to synchronize the Glue resources through Argo CD in this case, and GlueMesh is going to create the Istio cluster, right? But if there is some drift on the Istio clusters, like at the global failover or the WAF configuration or whatever, that drift is going to be picked up by the GlueMesh or the Argo CD sync. The Argo CD will detect that, the drift, right? I'm not sure if there's a tool in Glue that detects drift, but I think- So in this case, it wouldn't definitely be Argo looking after the resources that have been deployed, et cetera, for changes. Yeah, my question was because I really don't, I haven't used this combination, so I haven't seen how it looks on the Argo CD UI. Yeah, yeah, so in just to kind of pull the curtain back a little bit, in Argo CD, so I'll like for Istio, I'm gonna use some Istio nomenclature. You have like your virtual service, right? Usually you'll have like your virtual service like pointed certain percentages. You actually point at the stable, 100% at the stable release. So like in Git, it's more of like you're storing like your end state in Git for the virtual service. Now the Argo controller takes care of, sorry, the Argo rollout controller takes care of what points the stable, right? The stable, and I'm using stable in quotes because that's just like, they call it the stable release versus the canary release, right? It's an actual specification, it's actually called that. So, but the, yeah, so the Argo rollout itself is the one that looks at what your end state wants to be and then shifts the traffic accordingly, right? So it'll mark individual replicasets as stable versus canary, and you're kind of giving a lot more control over to the rollout controller, right? Versus, and in Argo, you just have your end state defined, like how, what do I wanna see as my end state? Yeah, and really GitOps is about, having a process behind it, right? Having that gateway into your configuration. So in an ideal world, you would have checks and balances before that drift can even happen. So theoretical question, doesn't this break the principles of GitOps in that Gluimesh translates the source of truth? Yeah, so it's, this is actually, and funny enough, there's a conversation I had with Christian Posta, who's at Solo as well. It's, so getting into just like the get-offs aspect of it, the, it's points of demarcation, as I always say, right? Because like if you technically think about it, a deployment isn't really your, right? Like isn't really what runs, right? It's the pods, right? So is storing pod definitions more get-offs than storing a deployment? Or are you just storing what your end state, what your end state wants to be, right? And so it's really your points of demarcation, right? Like so, and then there's like a balance between that, right, dude, like if you write like a Kubernetes controller that is like three lines of YAML, but like has a massive deployment, like that's technically get-offs because it's like, you just have a, you know, you're just selling the controller what you wanna do, or do you store those individual pod YAMLs and the individual service YAMLs, all of that can get. It's something we've actually discussed at the get-offs working group, right? The, this concept of like soft get-offs and hard get-offs where, and where that point of demarcation is, where I come, where my point of view is like your point of demarcation and where the controller lets go of that and where another controller picks it up, right? And where do you want those to be? Yeah, I mean, where I'm coming from is more in terms of repeatability and consistency. So to me, if glumash changes in a future release, it might change your end service. True, but that would be the same with Istio in general. That abstraction is needed. I mean, really it comes down to, are people gonna use it, right? Are they gonna be able to consume it? And in the end, we're trying to make that easier. Cool, we have about two more minutes. So if there's any more questions, comments, concerns. Awesome. Thank you. Cool, thank you everyone.