 I'd like to introduce myself first. It's actually the first time I speak on the Istio Day here in North America. So I work at Solo. I am one of the founding member in the Istio community, made a lot of contribution to the Istio project, and I wrote two books about Istio. I'm also one of the CNCF ambassador. A fun fact about me is I actually own 207 patents while I was working at IBM. With that, I would like to pass the mic to each of our panelists. Eric, can you lead the way? Good afternoon. My name is Eric Van Arman. How long have I been on the TOC now? Maybe two years? As I say, two years. I work for IBM. And I'm on the TOC, and I'm also the workgroup lead for the test and release workgroup, as well as a doc maintainer. John, can you go next? Hey, everyone. I'm John Howard. I'm a software engineer at Google. I've been working at Istio for about five years now. Go ahead, Mitch. Hey, everybody. I'm Mitch Connors. I'm that black sort of square up there because I didn't do what Lynn asked me to do. Sorry, Lynn. I just remembered you asked me to put a photo there. Well, your face is here live. That's the most important thing. I'm a senior principal software engineer at Aviatrix, where I lead container networking. And I've been on the Istio project also for about five years, TOC. And this year, I get to serve as a CNCF ambassador, which is a lot of fun. Yeah, awesome. Last but not the least, Niraj. Hey, everyone. Is the mic on? Yeah, I think so. I'm Niraj Poddar. I'm VP of engineering at Solo. Prior to Solo, I'd co-founded my company called Aspen Mesh. I've been a long time in the collaborative community working with Istio, leading the community. Currently, part of the TOC was former steering. He wrote a book on Istio, which didn't sell as well as Christian Posters in Linsden, but still wrote a book. Apart from that, folks, you guys can all come forward. It'll be good to see the faces, at least when you ask me questions, right? So come on in. Yeah, that's a good point. If you guys want to ask a question, actually there's a mic there. So if you guys want to ask a question, just get on the mic, ask a question because that way we do have virtual recording so they'll be able to hear your question. Before any question comes in, do we have anybody lines up for questions? I can't ask some seeding questions. So I guess the first question I have would be, you guys learned about ambient, right? John talked about how we are designing ambient for a very, very large scale this morning. Why would someone want a sidecar service mesh? So anyone wants to take that? Lynn, I think you're asking the wrong question there. I think the question is, why would anyone want a sidecar-based service mesh? When you put your app out on Kubernetes and you're designing your deployment to your stateful set, how many of you stick your application in a sidecar? Anybody? Anybody? No. Oh, there's one guy. I'm sorry for your loss. It completely defies the way that Kubernetes is meant to work. It was an awesome magic trick for getting service mesh bootstrapped and off the ground. And now that we have a better idea of what we're doing, I think we have much more efficient and effective ways to operate a service mesh. Any other TLC member wants to add anything? Yeah, I mean, Mitch gave a very valid technical answer. Let me give a business reasons, right? So I just spoke to a bunch of customers and they were asking the same thing. Why would you run ambient? And the answer is three-folds, right? One, operational simplification. As we spoke with customers, with users of Istio, we found out that running Istio in production with sidecar ain't easy, right? I just met with three of them who told me the story that how they started running with Istio two years ago didn't go to prod and now they're coming back because of ambient, right? So there's a lot of operational complexity running Istio with sidecars in production. It's coupled with how do you onboard your applications to the mesh, right? You have to restart your applications. Upgrading requires restart. So there's a lot of inertia there on an option. Second is the operational cost. So since the sidecars are running in the form factor next to your applications and every pod, there's a lot of memory consumption. There's a lot of CPU being spent. In fact, I spoke with a large customer just yesterday and they said 30% of their money was spent on sidecars. That's unacceptable, right? Like you can't go to your director of VP and say, let me buy a new product which is gonna cost you 40% more. And the last thing that we have seen is there are also some performance benefits you can get of running in the sidecarless mode, particularly when you just want layer four, MTLS and encryption. So these are the three pillars I say from a business point of view. Why you should focus on running sidecarless or ambient architecture? Yeah, I think that's really well said. Anyone wants to add anything? Dingle points are pretty much hidden. You guys are really shy. Any questions from the audience? All right, okay. If you do have questions, don't be shy. Stand in the middle. There's a mic there. I can also pass the mic around, but it might be easy if you just stand there and ask questions. Lin only has so many questions prepared. And they get a lot worse as they go to the end. I mean, it's gonna be a very short AMA so that we can spare the rest of the people from. Yeah, but anyways, so like my journey sort of from the last year, I think last year in the QCon, I saw you, we were talking about Ambient Mesh, kind of like, you know, getting traction and stuff. Kind of pretty new. So I think it's been in one year, kind of, we are still playing around with Istio and Anonvoi and watching videos in the YouTube and all of you guys and seeing the same thing. So I think there's one, like it's most of a, like, you know, comments is like for the multi-cluster setup, it seems to be still very clunky and it requires a lot of, you know, broiler pad stuff. And most importantly, though the documentation is sort of like, you know, clears out, you know, so there are like various things that you could, the various topologies you could essentially use, right? So it does not quite clearly says where exactly you're losing stuff. For instance, I just really have to go through and dig a little bit deeper to understand that what is really the bottleneck when we are really talking about, you know, when we have cluster across multiple availability zones and kind of having one single, maybe an endpoint to the end users without worrying really. So what are you guys thinking about the multi-cluster side? What are the new things? I know Ambient Mesh is like the new traction, but what about these things, existing that you're coming up if you guys want to describe something? Anyone wants to start? I can start. So basically in the traditional sidecar world, there is a very challenging problem, right? With multi-cluster, with all the endpoints available to each of the other cluster. I think John touched a little bit early on today how we are looking at design Ambient for better scale. For instance, with the Waypoint proxy, you are not aware of everything going on within the entire mesh. You are actually focusing on what are your potentially destinations, and then only focus on those endpoints. That would actually really help as you're thinking about beyond more than one cluster, right? Because your Waypoint could be your central destination focus regardless of whether your destination is one or multiple cluster. So instead of having that sidecar, which you used to have visibility of everything running in the mesh across multiple clusters, the way we are having designed the Waypoint, I think it's going to really help in those scenarios. Is it still working? Yeah. So the Waypoint, that endpoint, could still stay across the different Kubernetes cluster? Right now it's only one cluster, but we have some thoughts on designing Waypoint to be able to cross multiple clusters. Yeah. We need to get a single cluster up to beta before we start talking about multi-cluster for Ambient. That's going to be step one. But I do think that maybe the questions you're asking need a solution outside the Istio project. It's important for a project like Istio as it grows to know what it is and what it isn't. Looking back on our history, we came out as a way to manage multiple Envoy proxies because Envoy was amazing, but no one in the world needs one. Everyone needs a fleet of Envoy proxies to really achieve most of the benefits they want to get from that technology, and that's what Istio does. It operates a fleet of Envoy proxies. In multi-cluster, what you're beginning to need is something to operate a fleet of Istios. A fleet of Istio control planes across tens or dozens or hundreds of clusters. There are projects out there that are starting to try to do that for you. I think Admiral is one to have a look at, but I wouldn't expect to see a solution to that problem from within the Istio project. We're not experts on it, and you should find someone who is to help you solve it. Yeah, that's a very good point. I was going to mention the same thing around Istio is a toolbox, right? We should provide the tools so that you can get multi-cluster setup for communication. But once you have a multi-cluster environment, there's a lot more to it. You need a single pane of glass for visibility, probably. You need to have a right conduit for telemetry. You need to have a management solution so at scale, if you have hundreds of clusters, you can install and upgrade Istio uniformly. So, Mitch is absolutely correct in that we should provide the tools, but we should leave space for other projects if you are interested to contribute and bring ideas in the ecosystem or even vendors to participate there. This is where our community will grow further. Given that, I would say there are gaps in the current documentation that we should not elide, and we should definitely fix them. But even with ambient multi-cluster landing, I would say a complete solution that a customer or a user can take to production. I think it will be rather dangerous to just bake it out of the open source Istio yourself. You should look for some other projects or help elsewhere. If you don't mind. Yeah, so... Actually, we have a couple of people behind you. Let me do a time tag. We may have to come back to you because I think our session is only 25 minutes. Thank you so much for that question. Yes. Go ahead. Yeah, no worries. I'll be quick. I'm very excited about Ambient. I actually used it off the experimental branch quite a while ago. I'm super excited about the progress you guys made over the last year with Alpha Now. I'm very interested in understanding what is the path to beta, what is currently missing, and obviously when can we actually use it in production. My company is very much interested in the cost savings. Yeah, I can probably take that. Yeah, I would say, you know, the Ambient for, I don't know how long, at least last coupon we were talking about it. So it's been quite a while. You may wonder what's taking so long, right? I think if we were building Ambient from scratch as a startup new project, we would have called it 1.0, GA, go run it and prod, and then have a bunch of outages, right? But with Istio, it's like this existing enterprise software deployed by many people here, thousands of highly regulated environments, et cetera. So the bar is, I'd say, very high. So it's not necessarily that ambience like this, you know, terrible thing that's taking forever to get to beta. It's just the bar for beta quality at Istio's maturity is extremely high. So what we've been working on is largely about reliability. So making sure that there is no period of time where you lose traffic for a couple of seconds or the security policies don't work for, you know, when the pod's starting up. That may be perfectly fine for a demo, but it's not acceptable in a banking environment or something, right? So there's a large list of things like that and then there's all the integration. So things like multi-cluster, multi-network, expanding to VMs. C&I? Yeah, C&I, all these like CA integrations that we talked about. So there's all these areas because Istio is such an established project that we want to make sure the integrations are seamless and all that sort of thing. Probably missing a lot of other stuff. I think that's kind of, I think it's called the Ambient Path Debater or something that kind of lists out these things and we're starting to track that. So I would encourage you to check that out. Can I ask him two questions? Yeah, please. First, are you an existing Istio user with sidecars? Anyone a migrate to Ambient? Yes, that's correct. Fantastic. And secondly, will you be a beta tester once we release it? I would love to. Thank you. Yeah, I would say one of... Mitch was probably going to say the same thing, but one of the best things that you guys can do to help out is to try out Ambient where it is now and give us feedback. Or even if you don't have time to try it out but you have some additional information, one of the hardest things about running an open-source project is getting user feedback from people. A lot of people use it in production. We have no idea. And they have all these issues that they work around and do all these crazy things to fix the issue or anything like that. So the number one thing you can do is try it out and tell us. We have a user survey going around. Yes, we do have a user survey. It's probably at the end, we'll have a QR code. If you haven't taken that, please take it. Find us in the hallways, chat, open up the issues, whatever you need to do. But feedback is the best thing. Yeah, just to add to that, because we talk about it a lot within the community, it's a chicken and an egg problem, right? We try to get something out. Because we don't want to make changes in the APIs and stuff once you get to beta, right? So if we can't get the feedback and get the thing right to begin with, it's hard for us to say, hey, it's beta, please try it. So we're looking for feedback even with the alpha that it's at. Sounds good to me. Once it's beta, it's too late for feedback. For the most part. Bug fixes, yes. The API is the wrong shape. Sorry, we needed that feedback last year. So get your feedback in while you can. Thank you. Yeah, thanks for that great question. Hi. I think my question is going to be asked in the end of the day still. So I've been working with the Istio for the last five years, and I've been a great opponent of it. Discouraging teams from Integrate and Istio with sidecars for any project I've met in my life. And I think that's one of the most diverse reasons of course. And now with this burst of Selium popularity and much overlap between Istio and new features implemented in Selium, do you still that implementing the new Datapath as mBand mesh is like still viable thing in the long run? Because I actually do like the Selium project the most, and this is something Selium as a project lacks the ability to robustly configure Envoys that they still utilize. Don't you think that one possible path forward is like adopting Istio as a control plane for Datapath provided by Selium project as one of the options? So we're constantly looking at Istio integrates really, really well with Selium, when you use Selium as a CNI, so you can have Selium network policy enforced along with your sidecar function, perfect fine. For Ambient, we're looking at doing the same thing. We do expect your network policy to continue to be enforced by Selium but as any of the security I was saying we do expect you pick the right tools for your job. Like Istio is designed to do mutual TLS, secure your application communication within the mesh and if you're looking for fifth compliance, Istio is designed for that along with layer seven processing wasm integration traffic shifting canary base that I do expect Istio will continue to be shine there, especially with Ambient to have the right architecture in the first place with the two layer slicing the layer approach. So we do expect we continue to work with Selium but mostly on the CNI layer Yeah, I'll just say one thing you said you'll be working with Istio for five years? Thank you you're a warrior, think you have lots of burns Yeah, I've been actually running it as ingress gateway at scale for millions of our past four years Fantastic Thank you so much So we want to be able to interview you in that case See, that's what I'll add one thing on Selium though See, for an open source project like Istio which is already being run in production for large scale environments it's important to understand what we can be good at what can be securely and reliably do instead of ever expanding our scope So that's why what Lynn said we want to integrate with as a CNI layer Now there is marketing fluff saying sometimes how particular service mesh based on EBPF can solve all your problems That's not really true It has to have strong technological foundational stuff to say that you're doing mutual TLS I feel very strongly that Envoy as a layer seven and layer four proxy is the right choice and that's what Envoy that's what Istio will always configure I think for CNI if you're using EBPF you should continue to use that you should continue to use Selium I really don't see Istio configuring Selium data path helping in a way because the set of functionality there is not just not there for what we want to use It actually is because the Selium agent itself incorporates Envoy binary inside so the EBPF directly directs the traffic So that's a layer seven proxy which is deployed in a modulated way which inherently is unsafe That's the problem It's a per node Envoy proxy But there is an ongoing proposal to split the threading models for listeners so that each listener can implement them That's a big, big refactor of a very stable proxy that is running in production for the last six years And EnvoyCon I think 2019 Exactly Don't you think that this is a way forward? No, it's Is it fair to call it vaporware? I don't know that there's any I've spent last 10 years creating proxies The number of mistakes you make creating stable proxies that people can use is the service area is huge Envoy is there where people trust it changing the fundamental guts right now So if there's a use case that has no legs I don't think it's the direction that the community should go I think we need to agree to disagree because Enginx was following the same model for years until it became completely obsolete by Envoy I think we need to be careful that we don't turn this into another project bashing session There are some amazing service mesh technologies out there beside Istio and you should approach them with a requirements perspective Tomorrow Liz Rice is going to be giving a talk from Isovalent on why sometimes you don't need MTLS That's a little surprising to me, I feel like generally you need MTLS You should go to that talk and listen and bring your security requirements in mind and pick whatever meets your requirements I'm in complete agreement that you don't always need MTLS for any case You should definitely chat because you are running it in production for the last 5 years I want to understand your use cases better but thank you If I could add something, I think I have a bit of a different perspective from some folks here We talk a lot about how we split the proxy and we have the Z-Tenno layer and the Waypoint layer In my mind, the real service mesh functionality that we want is all about the Waypoint I don't want to be in the business of operating a Z-Tenno I don't want to be doing this node networking stuff The CNIs already do that but what they don't do is give us the secure, transport, encryption and identity layer If they did, I would happily use that I really don't want to be in this business of no networking The fun stuff is all the great functionality we can put in the Waypoint proxy WASM, rich telemetry, cool routing features failover, reliability, all this stuff I want the networking layer to be better but it's not quite there yet We have to bridge the gap as part of Istio that our users can have that secure end-to-end encryption If eventually the rest of the ecosystem picks up and can provide that for us, I'd be happy to integrate personally I totally agree with you You see the big scale of those functionalities We should follow up on your opinion I think encryption might be optional because whenever it comes to encryption, there must be cost So it could be optional to the customer For example, a customer could trust the household infrastructure and they trust they think it's fine that the data could be plain text on the wire Yeah, there's definitely some customers that don't require encryption a lot tend to, or at least will eventually require it if they don't today It could certainly be viable to have a mode that doesn't require it By the way I got a couple questions about Ambient Mesh So first is when Z tunnel will be production ready? Yeah, so we're first targeting beta which is, I don't know what you'd call it production ready if you're extremely brave I wouldn't call it really production ready at that point So we're kind of heads down on that and once we hit that, we'll start planning for a GA I think it's pretty plausible, we've been talking about splitting up exactly what we promote to beta and GA promoting Z tunnel, which is a much smaller component to beta and or GA earlier than the rest of all of Ambient which has a lot of other things that we need to work out with but that hasn't really been finalized, so it would be hard to give you a concrete timeline there Okay, sounds good The other question is like for Ambient Mesh by design I see there's a waypoint proxy and this waypoint proxy is tied to a destination workload that means each destination workload probably has only one waypoint proxy and in terms of reliability what if this waypoint proxy fails, does that mean that my entire destination workload or service goes dark? Yeah, I can take that So it's not one to one it's actually, I would say any to any So we offer today in current version because we want to keep it simple you can either deploy a waypoint for the namespace or a service account we could expand that to be completely arbitrary set of workloads, you could say this app and this other app and this other namespace they all share one waypoint and that waypoint can have one instance to 10, 100, it can scale up independently, so based on your reliability and isolation constraints you can pick and choose your model basically, so it's fairly flexible architecturally When we say you can scale it up this is not like scaling up the sidecar where you go edit some config map and reinstall your control plane and cross your fingers and restart your workloads and eventually maybe you get the scaled up sidecar that you wanted it's a deployment, if you want two of it you go and say replicas too That sounds great to me, but how about in current Zetano I see for the waypoint proxy address in that field there's only just a single value it's an address of the service so then we look up the service and the servants can have multiple instances behind it that makes sense alright, we still have more people so thank you so much for that question thank you so much for this question if anyone by the way doesn't get their questions answered or all of us I'm sure it'll be at the east view of Buu throughout the week come by chat, we'd love to hear from everyone let me start by saying Istio is amazing thanks we've been mainly using Istio for the ingress gateway part so we don't use sidecars as of yet waiting for ambient mesh the question mainly I wanted to ask was so we run into situations where we have Istio running on Kubernetes which is running on bare metal hardware and there are situations where you want to make a major change in the hardware Linux upgrade or something like that and we want to just take the entire cluster offline a nice way would be that if we can flip a switch and say Istio, don't forward any traffic to any of the services within the cluster but when we are running some smoke tests let these services be alive so do you have any suggestions of how that can be achieved or is Istio not the right way to do it when you talk about changes in the cluster are you referring to configuration changes to say virtual services or policy or are you talking about changes like upgrading your Istio installation or something else something beyond Istio I'm upgrading cube, I'm upgrading Linux kernel or I'm taking out cluster and upgrading hardware something like that yeah so normally when you do cluster level upgrades or rotation right you need something in front of it relying on Istio can work if you want to have a cluster in front of this which is deployed with Istio Ingress Gateway otherwise you're going to have a lot of problems with mirroring configuration right the better way is you have another cluster that is set up already which will have the same amount of services and then you can fail over in front of it that's the recommended pattern I would say we do that, what I want to get to is taking out the entire cluster when we run our service is fine but when we run our smoke test when we go with that approach where whatever load balancer whatever we have we just switch the entire cluster off now we can't even test our normal test cases that's why I was asking we looked into in services you have fault injections but that's a service level not at a cluster level let's chat more we have done stuff like based on headers some routing especially like dev token headers maybe that will help you yeah I would imagine with the subsets you can match on the cluster so normally we match on labels but we inject a bunch of synthetic labels and clusters one of them it's hard to know exactly what you need for your exact setup but I would imagine with some combination of that definitely worth following up what is the feature you mentioned destination rule subset I think we are over the time if I'm reading correctly Facila is there right or do we have time for one more question oh we have five more minutes oh thank you so much Zach yeah go ahead sorry can you hear me? I have one question regarding the protocol supported by ISD or Mesh we are trying to do some tests can you speak a little bit louder can you speak a little bit louder maybe to the microphone some tests with ISD Mesh with the protocol Infini-Span and we want to know if that is supported in the ISD Mesh can you say the name of that protocol again? Infini-Span and is it a TCP based protocol? no it's not based on I still didn't get the name I've never heard of the name so I'm going to guess means we don't support it more or less if it's not TCP based then we more or less don't support it well in the same for example Redis I know we support it in Istio but it's only the inbound or the outbound yeah so in general I would say for protocol support we do have just any TCP that works fine HTTP is the number one thing we focus on there is some small amount of support for things like Redis, MySQL Mongo maybe a few others but it's I would say a lot less mature than the HTTP yes primarily it's for telemetry it's not for advanced routing or security enforcement there is this project that I can't pronounce but you can probably find it Iraqi? yeah they do they kind of layer on top of Istio and add a bunch of different protocol support you might want to check that out protocol is UDP based you might look at Cisco has a sandbox project called media streaming mesh it's sandbox it's still not mature yet but they're looking at RTSP and other UDP based protocols in a mesh sort of architecture Istio may get there eventually with ambient but it's going to be a little bit thank you thanks for asking yeah thanks for that question question I wrote what we can look into all right thanks for that all right I guess it's time for us to wrap up thank you everyone so much and thank you for all the panelists we will be around if you guys have more questions I wasn't kidding about turning the tables on you if you've been an Istio user for some time and want to talk I would love to interview you and better get to understand what's going on and what you think about ambient thanks everyone