 Hello, everyone. Good morning, good afternoon, or good evening, wherever you are. This is the Cloud Networking State of the Union panel discussion. My name is Raghavan Srinivas, and I'm going to be the moderator for the panel. I really appreciate the rest of the panelists who are up at different times of the day. But we really wanted to do this as a live panel, and hopefully the next conference we can all be face to face. Having said that, welcome again, and I will go through, do a quick round of introductions, and the rest of it is going to be driven by you, the audience. So you're going to be asking questions in the tab, and we are going to be looking at it. If you like a particular question, you can go ahead and start it, and that way I can determine which question is probably more popular. So having said that again, thanks to the panelists, and let me go through the introductions. My name is Raghavan Srinivas. I go by rags. I represent in 4Q in this panel. My interests are several, you know, in the communities ecosystem, particularly I come from a developer perspective. I still feel like development and communities is kind of a pain. So I like scaffold draft and so on. So big fan of all of those. I like to kind of keep this minimal. So let's go to the next panelist. Hey, thanks for everyone. That's Tim. Go ahead, Tim. That's all my name. Yeah, sorry, we jumped right over. Christian, we'll jump back. Hi, I'm Tim. I work at Google. I've been working on Kubernetes for a few years now. Mostly I spend my time in networking and cluster, multi cluster node and infrastructural topics. Perfect. Sorry, Christian didn't mean to skip you. Yeah, thank you all for joining or joining live and watching this course for panelists, but I'm Christian. I work at Solo. My interests are on the Kubernetes level and application level and then up from there. So how application networking, how service mesh, how these types of things improve one's platform. All right, back to Tim. Hi everyone. Hi everyone, thanks for joining us. I am very new to Solo. I joined Solo about two months ago. I'm currently the director of Open Source at Solo. Prior to joining Solo, I worked for IBM for a long time, contributor and speaker to KubeCon. I've been contributing to Istio project for a long time. My primary interest is Istio service mesh and helpful user adopting service mesh. I'm currently a maintainer on the project, also a TSE member. Perfect. My name is Alyssa. I am also at Google, where I've been working on front end proxying for about 14 years on the DfE team. I launched H2B2 and H2B3 at Google, so worldwide. And then I switched over to working on Envoy, where I'm now a senior maintainer. So I'm very focused on the Envoy project, both as a server and as a client now Envoy mobile and kind of network scalability in general. Perfect. Thanks everyone. So feel free again, you know, as a participating audience, you know, thanks for watching and, you know, feel free to start some questions. I already see a few, you know, in the Q&A. But let me start with some questions that I kind of lined up. So one question, this is, you know, for everyone on the panel, and we can we can just go, you know, Christian first and Tim, then Lynn and then Alyssa, if you don't mind. So some requirements and assumptions for network design for Kubernetes really too simplistic. And it's becoming increasingly hard for Kubernetes clusters to scale to, you know, hundreds of thousands. Is that, you know, what do you have to say about that? So let's go, you know, with Christian first. Sure. So I guess I'm going to have a frame of reference that might not appreciate the assumptions in that question. So first of all, I do think that the Kubernetes did an amazing job at two things. One, trying to be as simple, you know, upfront, so that, you know, you can bring your applications to Kubernetes. And there's not going to be a lot of complexity and hoops to jump through to get it to Kubernetes. The second thing is building the APIs in the right place so that people can extend it. So, and then I guess, in so far that we need to scale to hundreds of thousands of nodes. So the opinion or I'm working with organizations in an enterprise where some of them have deployed maybe not hundreds of thousands, but thousands and thousands of nodes. And they are, you know, we're almost trying to redesign that and go toward maybe smaller, smaller Kubernetes clusters. But that's, that's, that's my point of view here. As someone whose fingerprints are on some of those early design decisions. I thank you for for calling out the main goal, which was to keep it simple. The assumption at the beginning of the whole Kubernetes experience was to make it not really surprising for people who are coming to it from a VM ish world and so wanted to feel sort of the same but be different. You know, here we are now six, seven years later, and we've got a lot of new workloads that we're trying to bring into Kubernetes. And honestly, if you told me seven years ago that we'd be talking about the things we're talking about now, I would not have believed you. So, the requirements have shifted. I do think that Kubernetes is struggling a little bit to provide the primitives that we need for some of the higher level primitives that we want to build things like telecom are bringing a lot of really interesting and complicated requirements that we're we're slowly understanding and being able to adjust, but it but it is a slow process. Yeah, I would add from an application developer perspective right when they start to look at Kubernetes, they would immediately see a bunch of challenges for them. Because all they care about is going to be, you know, I have multiple services as I'm moving from my monolithic to multiple micro services. How am I going to, you know, handle my service connectivity as my services going to be distributed to multiple nodes in Kubernetes and maybe multiple zones and regions. So the network is not always reliable as monolithic from their perspective. So what does Kubernetes do for them in that perspective. I think today it's it's not clear. That's why people starts rely on a higher level technology like service mesh to help to tackle this problem from a framework perspective because people knows they didn't want to solve this problem in their application code they could really costly. Yeah, I'm, I'm a big fan of launch and iterate so to me the fact that we're hitting these scaling boundaries is a sign of Kubernetes success. You know, there's the iconic, you know, 640 chaos to be enough for anyone I don't think that scaling up in that particular direction was expected. You know, it was like trying to create a successful ecosystem and then you never know what you know what payloads and what infrastructure infrastructure people are going to be using it for in years. So, so again with kind of the launch and iterate mindset I think that you're always going to hit scalability, you know blocks and you just have to figure out how to iterate to get around them. Okay, so I know questions are starting to trickle in, you know, feel free to add questions and you know we are here live to answer those questions that you have. So let's start with the top one. What is the status of IP V6 only now mature in Kubernetes. And what about direct connectivity parts and our containers in parts instead of connecting through the IP address of the host. This is from Jim Hugo Jim. Thanks for asking the question. Where do we want to start who wants to take this. There's a lot of context I can speak to the state of Kubernetes. It might even still be premature to say that IP V6 support is mature dual stack support is beta now so it's more mature than it has been in the past. It's not going GA in the next release but hopefully the release after that. And to the rest of the question, you know, direct connectivity Kubernetes was actually built to assume direct connectivity. It was only worked into overlays and host based proxying out of necessity. So there are environments. And I'll pick on my favorite one like Google Cloud, which build pod connectivity into the VPC network automatically. That's not always practical not for all customers and not all environments. So, you know, we do allow other models. I'm hoping that the rise of IP V6 20 years later, the rise of IP V6 will enable more people to embrace that model. The IPv4, you know, IP address limitations have been have been a real problem for people. Okay, anybody else wants to add anything. Okay, let's move on to the next one. Oh, I skipped over one. It looks like at least a dozen sync network enhancements are slated already for 1.22. Can you address the continued fast pace of change and where you see Kubernetes networking going and and I think this is a classic, you know, too much happening versus, you know, too little happening right you know where is the happy medium I guess right. So who wants to take this question I know that you know many of you are kind of part of the significant work right so should we start that somebody else other than Tim. I can take an initial crack. So, so, so, first of all, I do think it's, there are a lot of networking initiatives going on a lot of things happening in the networking working group but I think that's good. You know, there's a lot of inner iteration and a lot of innovation happening in those spaces and like, like Alyssa said, when you get to those points where you hit bottlenecks you hit scalability issues and so on, things become really interesting. One of the ones that I'm particularly interested in is the is the gateway API, which will, I guess the plant, maybe the ingress be one. And that's that's one of those things that's kind of been a long time coming. But yeah, there's there's a lot of really good stuff happening in the working group. And I would add that you know there are interesting things in addition to the gateway API, it's like the multi cluster service API, you know the is your community is looking at implement that. We know there are challenges with running services in multi cluster where we want to have ability for match admin or the service owner to be able to control whatever services they want to expose outside of their cluster services they want to import into their cluster. So that's really interesting. And I know team you are also driving effort to replace the sidecar cap, which we had a huge interest from the is your community perspective, because there are a lot of what user wants to be able to precisely control the proxy start sequence. The psychoproxy corresponding to the application container in the service machine environment. So that's super interesting to us. From our perspective, you know, being working on is still for a long time. We realize certainly an item on the roadmap for a particular release sometimes doesn't mean it's actually committed to the release. It actually means that you know making enough progress and hopefully reach some stage for that release, but not necessarily you know it's a hard thing you know that release is not going to hold up for the completion. So I'm actually really curious to hear from Tim on this too. Alyssa, do you have anything to say or, you know, I think, yeah, I think it's covered there. Okay, Tim, you want to final thoughts. Sure. I'll be quick on this one. SIG network has been super super active. The last three or four cycles, which is great. There's a lot of caps listed in say this next release. They are huge like dual stack actually dual stack isn't fundamentally changing in the next release but some of the some of the in progress work is huge like dual stack touches every part of Kubernetes. And therefore it's taken a very long time for us to build our confidence in both the API and the implementation. Many of these caps though are much smaller in scope right they're adding a field to the API which allows load balancer control from the cloud provider in a in a very nuanced way. I wouldn't be overwhelmed by the sheer number of them. You know, but we're taking we're tackling a bunch of different issues, sort of all at the same time and in different places. That includes things like topology and topology aware routing of traffic. I saw another question in the in the Q&A sort of about that. So we're adding more capabilities there and that'll become default eventually. Things like gateway I think is a huge effort, but actually kind of outside the core of Kubernetes right it's all happening as as custom resources. So that's sort of an interesting proof of the of the viability of custom resources honestly. We're also looking at things like major IPAM overhaul so that we can be more functional in restricted environments like on premises. Yes, Lynn. I'm very interested in the container lifecycle problem. It is a very complicated problem that works across a couple of the biggest and most busy sigs so it's been a little bit difficult to get traction on on how to proceed with that. Somebody took the idea that we wrote and started writing a cap for it. I'm a little wary because we don't have sort of alternate ideas explored yet. But I definitely want to see the sidecar lifecycle problem tackled in a holistic way. So there's a lot going on, but don't just look at the numbers also spend a little time to understand the scope of each because they're not that big. Not the classic, it's not the quantity, it's the quality. All right. Ryan is very popular or this question is, you know, is really, you know, very critical. What are your thoughts on BPF, EBP in relation to Kubernetes networking. Good, bad, ugly, scary. One pass on that I'm going to go with good and scary. You know, when you're adding something like this, it's, it's really hard to get it right, especially get it right at first pass to make sure that you're not setting up your filters wrong or you're not allowing the permissions to get screwed up and traffic you could have the wrong place. But again, you know, in a prior life I was, you know, TL of Google's each to be three effort or quick as it was at the time and we're getting quick support and unboy hopefully later today. I don't want to use each to be three without BPF is just that the kernel doesn't doesn't throw packets around fast enough so I'm hoping that that soon this will spread out to the entire Kubernetes ecosystem. And you're going to want BPF to get it right and really have kind of the next generation of high speed networking. Yeah, I think that what Alyssa says really resonate to me as well when I first look at the question I was like, Okay, it's great. I love, you know, all the momentum behind the BPF and all the powerful thing they provide, but it's really scary, you know, as individual developer. I don't think I want to touch BPF right I really would rather have a, you know, a trust was vendor to provide solution for BPF, and then for average developer to consume because it's hard of messing up the kernel you know it's going to be super hard to debug I think it's going to be even harder to debug than envoy proxy which many of you probably agree it could be very hard to debug also. Christian Tim any thoughts. I'll just throw in that it. I think there's a lot of excitement around it. A lot of potential power. I'm not seeing too many enterprises go down that path just yet a lot of people are asking questions, but I'm not seeing that much in the wild at least people going all in with it. I, you know, it's, it's funny to call BPF young because it's been around for years and years but in the sort of overall adoption curve of these sorts of things it is pretty young so I'm super excited about BPF, not from an end user point of view I don't think end users should ever see or know about it. I'm more excited in the opportunities that it opens sort of the, the adjacency of the possible right now that we've got this, we can open a whole bunch of new doors and explore a whole bunch of new things that we haven't been able to explore before because they weren't really pragmatic in terms of implementation right the floor has been raised in terms of what we can expect out of the capabilities of a Kubernetes cluster or at least it's on its way up. And so, so I'm just excited at sort of what is the next round of ideas and API is that we can add that we just couldn't even consider last year. Okay. The next question is from Ricardo. How do you see the current multi cluster or hybrid cluster options. And do you think there should be more support for this use case directly in communities. Obviously I think he's referring to the networking aspects of this right. We have a multi cluster or hybrid hybrid cluster. And, and what should Kubernetes networking, or how does it need to be enhanced. If it should be. Is my reading of the question is that, is that accurate. That's how I read it. Yeah. All right, go ahead. Yeah. I'll start but then I'll hand off quickly because I'm actually really curious to hear everybody else's thoughts. Kubernetes is a cluster oriented solution it is the scope of the, the boundaries are growing a little bit. I, I do think that some of the capabilities to enable multi cluster are going to be or are starting to be ingested into the sort of scope of Kubernetes. There are some ports and service exports now which is a way to publish services across clusters. But it's explicitly an API and not an implementation because the, the variety of sorts of ways to make this happen are so broad. And I think gateway also plays to this it's a more neutral way to describe the sort of doors between services. I do think that multi cluster is sort of going to become the norm. Everybody is going to be a multi cluster user at some point. Whether that's just in terms of, you know, upgrading a cluster by creating a new one and flipping the traffic over or by people who do geographic distribution or blast radius or whatever reasons. I do think multi cluster is becoming the norm. I want to be careful that Kubernetes doesn't try to be everything to everyone and try to absorb every single API out there. So service import service export was a very tactical small step to cover one use case that just kept coming up over and over and over again. The rest we're going to be super careful about this is where I think service mesh becomes a really interesting way to explore it because you have a lot more freedom as you move up the stack. This is where I want to hear everybody else. I think there is another question by Gary which kind of feeds into the same thing as well which is kind of like many cloud providers offer multi zone deployment within a region and you know some offer multi cluster networking for clusters deployed across regions. Is there something from a networking perspective again which stops people from doing that. So, just factor that into, you know, when you're answering the original question. Yeah, if I could just build a little bit on what Tim said. At least we've been looking at it from the application level how do applications communicate with each other that might be deployed across different clusters for various, you know, regulatory compliance type reasons or, you know, scaling reasons and isolation reasons and so on. So I do agree that it is, or at least we'll become the norm if it's not right now. But like I said, we've been looking at it from the application perspective, and we're heavily involved in service mesh and maybe it'll say a few words about what, you know, how Istio plays into that as well but at sort of a high level. We see applications can be deployed on Kubernetes, they can be deployed on VMs, they can be deployed on prem, they can be deployed in public cloud. And the way we try to solve that is at an abstraction. So what is multi cluster, what is multi infrastructure, what is that networking look like. Well, the applications need to be able to exchange messages with each other. And so what are the solutions that support that service mesh as one and there's others but Yeah, just to add on to that certainly multi cluster service API, I think it's super interesting for service mesh because I don't believe you would using Kubernetes along with implementation of multi cluster service API without service mesh. The reason is you could use that assuming there's an implementation out there. But the problem is, you have to solve all that problem that service mesh is going to solve for you. I mean, when you talk about the connectivity service connectivity among multi cluster, how are you going to secure, you know, the communication, how are you going to, you know, do like low balance of fail over, you know, pre for local and fail over to remote. How are you doing traffic shifting so a set of challenges going to come to you as a service owner, and then it makes sense just to build on top of what service mesh already provides. So team, to your point, I, I, I almost feel you know that API is really behind with service machine mind. Yeah, lens point I think I think it's basically natural that people are going to want to have heterogeneous setups like home, you're going to end up having a much cleaner solution and infrastructure if you can be homogeneous. But it's just not a reality. But but when you have these mixed sort of deployments it does get hard managing it and making sure that like you understand the behavioral changes of like what traffic is going where and why and how things you know how this health checks versus that health checks it gets really complicated and so it's always interesting to see how the ecosystem evolves from like everything kind of looks like this to the reality of everything is mixed and complicated to the reality of, but I just want to manage it one way how do I do that. Great. I was going to bubble this question up anyway, because this is a little bit more tactical but but seems like it's getting more rewards, which is with security policy PSP being deprecated in communities 1.2.1. There seems like there are some security gaps with communities networking. Are we seeing more clients adopt, you know, open policy agent to fill these gaps. I don't mind starting. Sorry, I can't stand avoid. I personally I've seen a ton more uptake of OPA and using that for much more sophisticated policies in the last year. It's been a very noticeable uptake from what I've seen. There are I mean security is such a complicated and broad problem it's hard to to talk about what even network security means I see there are some other questions in the Q&A that talk about identity and I think that's an important part that you know Kubernetes doesn't address very well currently anyway. And so yes I'm seeing a lot more people uptake on OPA for locking down things like the Kubernetes APIs who can do what with which resources and those sorts of capabilities. And in fact we found a handful of things that were added to Kubernetes with best intentions but turn out to be very sharp knives and that most people don't need and so we're recommending that a lot of people just turn those facets of those APIs off or put them behind site specific OPA policies to control those sorts of APIs. Okay, anybody else want to add. Otherwise we can actually go to this question which is will ingress or gateway ever ever become identity aware. This from Martin wants to take a stab at that. Then you want to go. Well, I would say gosh, I would say the ingress is kind of becoming identity or well at least within the service mesh environment right because within the service mesh environment we actually give the identity to the ingress gateway from like we the control the playing of the service mesh provisions the identity for all the services within the mesh and the gateway at least the in the I would say the in the portion of the gateway you know it has the identity whereas it's interact with the rest of the services in the mesh. And I would just add that I when I thought about identity I thought about, I guess I immediately thought end user like the requests are coming in to to the gateway in ingress setting. And what I would expect is that those those types of things would be delegated to, you know, I am an identity provider solutions that, but that it would be able to provide identity for for those end users. Okay. It's very yes it's very difficult, because Kubernetes is so broad, and it is on so many platforms it's difficult to make assumptions about identity and what's carried on the wire and what isn't, which makes it sort of a limiting factor for apis like ingress which try to be very compatible. And identity doesn't map very well in there I do hope that we can make it better and make it more identity aware. This comes back to something like BPF where you know maybe we can do really creative things at the network level without doing full encapsulations that give us a sense of identity so that we can at least enable some increased sense of policy around identities. Everything else at Kubernetes is at layer three and layer four which is really just not identity aware. I want to be very careful how we open those doors. But I do hope that we can find some way to do that better. I know that we are coming up to less than four minutes. I really want to bring up this last question, which is kind of a little bit forward thinking. And basically what it is is, you know, again, going back to the original simplistic design, Kubernetes handles simple networking things pretty well. What is it keeping up with the capabilities people need in the 2020s, you know, 2030s, whatever. Where is the line from what capabilities Kubernetes should grow to support, and what should remain out of bounds. How far are we from that line. Maybe we'll start with Alyssa and kind of go. How close we are to done. I will say, I don't think it's ever going to be done, right. I think that there's always going to be some evolution of, oh, and now we have this new type of deployment and we have this new setup and such. So, I think that that's going to continue to evolve. And I think, you know, for, for what's added next, I think it's just going to be based on demand, largely especially with open source projects like this. There isn't always an easy roadmap, right. It's based on what people use as deployments and what innovation there is in the entire space, not just the project. So, I feel like there's maybe like a feel for a roadmap for like a year or so out but beyond that I don't know. I don't know if other people have a sense for that. Christian, you want to go next. Yeah, I'll just say one thing I absolutely love about Kubernetes is that they, the community, the project they drew a line and they said this is this is where we're going to live. And we're going to allow other things to sprout off or extend in and, you know, with the API's customer resources all that stuff I think we'll probably see more optimizations in communities to continue to support that model. But I think that is what will enable the drive to all these various, either niche or not originally thought of networking models. Okay. Lynn, you want to go next. I would say, you know, some of the challenges with service owner running their services in Kubernetes really open up the doors for other solutions like service mesh to shine in this space, right. So it's actually really nice Nick Christian said, Kubernetes are not going after those set of problems and leave to another solution or architecture to solve that problem so I think that's really nice and you know we have so many vendors in service mesh community today is trying to compete in that space. So that's really interesting what I would really love to see you know I actually really love Kubernetes become so boring that people really view Kubernetes as infrastructure now. But it's not filling the gap where the service underneath which is on the layer seven on the service level of connecting services so I'm, I really love you know service mesh would get there one day becoming so boring as Kubernetes that people doesn't have to worry about debugging that because you know, worry about all the performance input, you know, that's like, yeah, look forward to that too. Thank you. I get the last word. All right, well, I, you know, a year and a half ago at cube con in San Diego. I made a statement that Kubernetes is already a service mesh. It was a little bit controversial then I think it's a little bit controversial still, you know, we're doing inside of communities we're doing a lot of the things that service mesh does but we're doing them at a lower level with a lower level of fidelity. And I think that's intentional like Christian said right we want to draw the box and say this is how big we're going to get. In an API like gateway, you can see where we built intentionally extension points all through the API it's much more complicated API than ingress was because it's tackles a lot bigger problem. But it's designed to be extended because we want these implementations of things like service mesh to be able to plug in at various points and say, you know, if I'm a user and I need to pop out to implementation specific capabilities here's where I can do it here and I think it's really important because the otherwise the abstraction isn't useful. Right. The, there are other things that I think Kubernetes is not keeping up on and this is an area where I think we need to explore the next year or two. Things like multi network have been a persistent problem for customers like telco. Kubernetes initially, you know, we put our fingers in our ears and hum because we didn't want to deal with those use cases because they're really hard. Well, Kubernetes is pretty successful now and we have to hear those users in those use cases that they're bringing to us. Community came up with some solutions. I think we need to normalize those and make them a regular sort of capability. And I also think we need to look at things like virtual network functions and you know bump in the wire packet processing even within a cluster. I'm not 100% sure how we want to go in and implement those things. But I think there's enough demand for it now that we need to start thinking about what weather and how we're going to try to tackle it. That's about, you know, time to wrap up. I really want to thank all the panelists, but but really wanted to thank the audience for such insightful questions. There are so many more. I wish we had like much more time to talk about. But that's kind of the leading to please come to Slack channel where, you know, the channel is number two dash to a number as a number to dash to on networking. So, you know, I urge the panelists also to please join. You know, it's going to be chaotic, but you know, that's fun, right? I mean, this is the hallway conversation that we're going to have. I mean, I close us to the hallway conversation. So I want to thank again all the panelists and really want to thank the audience for the questions again. And hope to see you all in person in North America. Thank you very much. Thanks everyone.