 Hello, you guys say is Ken with us this morning. Yeah, Ken isn't on just yet, is he? No, he's not on the purchase of it all right, but So some folks from Microsoft are gonna present today Do we have those folks on the phone? Yeah, this is the book. Can you hear me? Hey deep up? Yeah You know we I'm running around a conference trying to find a place to Park for a moment. I think I found one Oh, so we've got Brian to Sean Deepak and she or Yeah, and she's yeah, and Mike Mike and Brian I got to say your names were easier than the others If I can Well fantastic, you know our fearless leader Ken is As a busy calendar, I I suspect we're about five after now six after he'd probably want us to Get going in his stead. Let me let me ping him One more minute. Let me ping him and just see if He's gonna make it. Okay, can everyone see the screen that I've shared Messaging can Okay, hey fair enough, let's call it. So it was we're eight minutes in Ken has been duly warned I'm pumped to To hear about this Okay, thanks. So today I'm going to talk about How we are thinking about containers in Azure networking and how we are leveraging C&I and some of the Scenarios that we are facing that we think we need to To add support for NC and I Okay, so Azure Network Is a very rich hyperscalable Reliable virtual network that we've offered for many years now For virtual machines and that we have brought to containers recently It offers lots of capabilities right Starting here on the left Your virtual machines or containers can be accessed through public IP addresses directly from the Internet We provide load balancing. We provide ACLs Your front-end access gets a DNS name so customers can access your service over the Internet Using DNS names we provide DDoS protection then moving here on the right on the virtual network We offer a very rich virtual network for for your virtual machines and containers in the cloud So basically customers can specify their own IP address ranges their own subnets Then they can specify Security groups by which they can specify isolation policies like this this security group is not allowed to that talk to that Security group and and so forth We support rich service chaining through a construct that we call user-defined routes Which basically lets customer control how traffic from one virtual machine to another virtual machine flows And whether that needs to be sent over through a virtual appliance before it gets routed to the destination So so we have a rich suite of capabilities in in the private network that customers can can set up in in the in the cloud environment We have a very rich ecosystem of network appliances in the Azure marketplace that leverage these capabilities to insert themselves into customers virtual networks We offer back-end connectivity to customers on-premise So this may be through point-to-site VPN or VPN gateways We also offer private peering through various ISPs and providers that we have relationship with we call that solution express route So this is a brief primer of on on the virtual network that we offer to our customers today Right now. Let's look at Azure CNI, right? So so when we started looking at see CNI and how we Do container networking for Azure, right? There were a few things that that we observed right one is We want to leverage the Azure SDN which already is a very evolved ecosystem and Bring those same SDN capabilities to containers because when we talk to customers They essentially need similar kind of network and So even more but some of the core constructs around virtual networks load balancing DNS The need for containers as well We realize that that in the container world. There are multiple orchestrators that Exist today to manage those containers. So we have to integrate with with these various orchestrators Of course, we are running in the in the guest environment for us VM is equivalent to a guest So the operating system there can be the Linux or Windows. So we need to support both Scale is a is a big deal for containers Creating a million VMs is much harder than creating a million containers. So so that's why Scaling the virtual network is is essential for container scenarios On that actually if I can interrupt for a moment and curious can you define a bit? What is a vNet like sure a vNet is a virtual network that a customer specifies Through a template that hey I want to deploy these 10 VMs in in these two subnets Here is the IP address space that I want to use for for my my my workload in the cloud so so they define Subnets they define address ranges they define Their security policies they define their routing policies So it's basically customers own private network for their their compute workloads in the cloud Right. So in Amazon's term Amazon calls it VPC Azure calls it vNet, right? So so they're basically the same Okay, so okay Okay, I think that actually it helps some so so it's not necessarily an an overlay network provisioned all CNI or your CNI plug-in Yeah, so so right now VNet so far what I've described to you is Underlay in the sense that it is implemented at the host and exposed to the VMs Now when we start talking about containers, right? There are Of course overlays can exist inside the VMs as well as we have an underlay at the host level and Both can coexist But at the same time we want to leverage the capabilities that are already available through the underlay as well as as Where it makes sense to to provide additional capabilities via an overlay on the top So so VNet by itself is not overlay, but it does not preclude overlays overlays can coexist with underlays Okay Provisioning time of less than a second is is is a requirement that again containers drive in the virtual machine world The provisioning times can be longer The one one key thing here is As we talk to our customers customers are not purely container customers or purely VM customers, right? They they may have some workloads that are containerized and then some Other part of the infrastructure that is not containerized so so what we've noticed for a lot of our customers is is they have a mix of both container and and and VM based workloads and They don't want to to build two separate networks that are completely Oblivious of each other For for for for these scenarios They want a seamless single network that spans across both where containers can talk to VMs and VMs can talk to containers They want to specify their policies in a consistent way so one of the Motivation for us to leverage as your SDN has been to make sure Customers get a consistent experience No matter in what form they deploy their compute workload in the cloud whether it be Azure functions or lambdas or containers or VMs, right? We want to give a consistent network and a consistent way of managing policies for for the network to our customers so Let me move on to next slide here Okay, so now I start digging more into into how we leverage C&I and and how we've built this functionality right so at the top level I show ACS which is Azure Container Service Engine. This is the service that today Creates orchestrators or launches orchestrators on behalf of customers. So customer comes to ACS and says hey create a cobernator's cluster for me or create a DCOS cluster for me So ACS engine creates that but then That The way we plug SDN Into that orchestrator is by leveraging C&I right and for that we've written Few plugins, right? There is a network plugin which basically attaches Container interfaces to the underlying Azure Virtual Network that is implemented at the host level, right and It it routes traffic on the virtual network, right? Then there is the IPAM plugin which is responsible for allocating IP addresses For these for the containers through C&I One key thing I want to call out here, right? So the the experience that we have offered to the customers is it's one IP address space That they can manage for VMs and containers whereby a subset of that address space is delegated for container use But the other space may be used for VMs that way they have a single network and VMs and containers can freely communicate with each other Rather than then building the two networks and they're using their address space completely in isolation in which case Yes, they become islands. So Deepak a couple quick questions. I actually am I Wireless cut out for moments. You might have said this but the the way in which that designation is done between IPs for containers and IPs for VMs. That's sounds like that's within your IPAM plugin. That's more of a DHCP reservation or not not reservation, but Yeah, yeah, it's it's it's along the same lines. It's done in our IPAM plugin and DHCP is the protocol we use to assign IP addresses But how we make the reservation is through our APIs where customers can say hey This address space is reserved for VMs or this address space is delegated for containers And the tech that drives the that runs the DHCP server. This is What is that? So that's part of the Azure platform already So we we run DHCP server for all the virtual machines and for containers in the cloud It's running on the host And it's talking to our network controller, which is telling hey on this VM Here is the IP address that you should respond to when you get a DHCP request from the VM or from the container So our network controllers is is provisioning our DHCP server to respond with the right IP address Okay, so so this this is a brief overview of what we have currently in Available for containers through the Azure CNI plugin In in the public cloud now There are new scenarios that are coming up And this is what is driving us for For proposing extensions to CNI One is Single tenant container clusters, right? So so a customer comes and says hey, I want to To create my container cluster and We we provide a managed orchestrator experience. What that that means is What I call it here a KS as your governor service What it does is rather than Customer running their own orchestrator V as your platform is running Orchestrator on behalf of them, right? So customer does not need to deal with managing and running their own Own orchestrator. So think of it as orchestrator as a service or container Yeah, so so we we deploy an orchestrator on behalf of them and Customer just has to deal with the containers that they they they get right So So in this scenario the containers will also be connected to vNet via Azure CNI and then vNet policy will be available to containers Right, so the second scenario, which is the next step of the previous scenario here We are taking the next step where we are offering container as a service customer doesn't even need to pick What orchestrator? They they get to use they just say hey, I need X number of containers and as your Container service is the one that is responsible for Creating these multiple containers on behalf of various customers So so really the key difference here is there's not an orchestrator per customer per tenant, right? the orchestrator is shared across multiple customers and Customers just get the the individual containers that they they want right question on that it Yeah, I don't know if you're You're able to say but what's the what container orchestrator is used behind the scenes for your cats? So today we have offering based on cober or we will have offering based on cobernators and we will have offering based on service fabric so So we will use the container orchestrator behind the scene Customer doesn't need to know about it Okay, yeah, that's kind of my question is which one is that it is cobernators Okay, so so similar model is there for Azure functions where customer doesn't care about what Function orchestrator is being used. They just care about deploying and and and running their functions Azure web apps Which is similar along the lines of hey customer just cares about deploying their website The the orchestration and management of those websites is left to a central coordinator right, so then the the next scenario which I've alluded to before right so these containers that get created right in the Azure vNet customer wants rich policies to to connect and protect These containers and the workload that is running on those containers and they're asking us for for similar constructs that they're used to for their virtual machines And these are pretty standard constructs out there in the past right there's the security group notion There's the notion of service chaining VPN and private peering capabilities There's another interesting trend that we are seeing This is service endpoints whereby Customers like if they even when they are running share on a shared past service like storage or sequel DB as a service Which is a multi-tenant service. They still want those endpoints to be isolated and protected from other customers So they want even the endpoints of those shared services to be projected inside their vNet and to be locked down to To their virtual network, right? So this is I would say even the next step of containers where the service may not be containerized But still it has to provide isolation at the network level on a per Perk customer basis by by isolating them into different vNets A load balancer DNS NAT right customers want all all those those capabilities and they want the policies to be common for VM and container workloads right so based on these scenarios here here are the the Requirements that that we would like to to work with CNI and see how we can address these these requirements, right? One is around the policies So I know right now Kubernetes has a way to specify policies and and maybe that's that's okay that's totally fine, but We would like to be able to bridge the capabilities of the underlying platform with the the orchestrator specific policies so that So that a customer may be able to use any number of orchestrators and may be able to specify those policies in an orchestrator specific Language, but down below we are able to translate that into a common way of rationalizing those policies and implementing those policies And managing those policies Otherwise as a public cloud platform provider We we have to write a plug-in for each orchestrator for the implementation of policies what CNI has done awesome at is in terms of network virtualization where We are able to plug into multiple orchestrators in a standard way To implement virtual networks. I think the same capability needs to be extended to policies so that we can implement policies in a consistent way regardless of of what the the orchestrator is Multi-tenant networks on containers Sorry, do you do you want to discuss these one by one or do you want to read them all out and then discuss afterwards? We can take it either way We can we can discuss them here itself Why don't we do that? Okay, so they this is Brian Borum here I Happened to be a CNI maintainer I think I'm the only one on this call and I Also work on the weave network So how would you anticipate so the thing with policies is is that the rules Generally talk about groupings of things For instance the way Kubernetes does it is the groupings are specified by namespace and by label How how do you envisage that would work cross-orchestrator? I think this notion of grouping and tagging is fairly generic So if we defined a standard way to group and tag and then specify Accels based on those groups and tags in a standard way in CNI then I think that will Meet the requirements of of various orchestrators now Different orchestrators may choose to do more advanced stuff But at a bare minimal level, I think if we define the notion of grouping and tagging That will that will be awesome I'm really finding that difficult to imagine Okay, well we can move on I guess see Let me try to extend what what Deepak said here is is basically if you have a Basic way of describing tags and memberships to those tags for workloads And then policies can be specified for those tags Themselves like this this tag means you should have access to internet or you should have Put this this load should be behind a load balancer on this web and and this is the endpoint I want exposed to so all those things can be put Specified on the tags and then when the container itself is our part itself is instantiated it just Subscribes to membership to attack. That's that's one way to look at it and that's that's That's kind of most of the cloud providers and platforms already supports some similar notion along these lines so Really just jumping it so as we look at this this item, right? I think the From the CNI standpoint and Brian keep me on today. I don't believe this is something we would extend CNI to manage But this might be an area that This is not not the forum where we take decisions about what CNI do That's all right, I'm prepared to you know listen and try and help out but Yeah, I I mean I'm really surprised that as far as I can tell there's no issue or PR Submitted to the CNI project So we are in the process we can submit the PR to discuss the proposal in a little bit detail on that part What what we want to like what we are trying to propose but this this call What we are trying to do is is basically establish the common requirements that we are trying to address If you can agree to or have some Commonality in this that it makes sense to extend a discussion to actually submit a formal PR on the CNI project itself From the spec point of view, how do we want to extend it? Yeah We want to bring these topics to this forum so that we can all discuss and debate and and decide what is the best path forward and If PR you think is the best way to move forward we can submit a PR if you think Hey, we need more discussion around the merits of it or where to do it if it's better to do in in Networks sake we can take it there So so we are really looking to partner with all of you to to decide How to take this forward because these are requirements that our customers are putting on us and we can Meet those requirements in a very Azure specific way and maybe Amazon solve them in Amazon specific way What we are trying to do here is is in the spirit of openness We want to bring it to this forum so that we can all have have open discussions around. Hey customers are putting these requirements What do we do? Yeah It's I mean the one thing we could do in this forum is is decide that we want another thing Which is like the common policy interface or something like that When you say SIG network, I'm guessing you mean the Kubernetes group. Yes So that's not gonna help you with any cross orchestrator questions And do you know that you're exactly right that that is why we thought that will bring it to CNI because that's more This is not CNI. This is the CNCF network working group. Yes. Yes You're right to the CNCF working group so that we can take an approach that is orchestrator independent But again, we all realize Kubernetes is making a lot of progress here. So so if We would love for CNCF working group to take this on and define these capabilities in an orchestrator independently Yeah Yeah, I You know very briefly. I think that's very difficult and I think What you said about tagging I I really find it hard to imagine how to map the Kubernetes semantics But yeah, we probably shouldn't go into all the the detail I mean, I might suggest that you give that give that a go for instance to figure out what you think is a generic scheme and then demonstrate exactly how you would map the Kubernetes semantics onto that Yep, Brian. Brian. This is this is Christopher, you know having, you know, like you You know have to support multiple orchestrators in a security policy model I would say that the differences once you get into the details between how Various orchestrators model security groups versus labels versus tags versus profiles um You know, I I I it may be doable but the way I look at it it would require major surgery on any number of orchestrators to try and get to a common a common model, but That just my you know, we've been living this dream for the last couple of years and and I would say that there's not a once you get down to the what not only what the data model is but what the people who use that orchestrator expect that data model to drive There are differences that are not just cosmetic But let me throw this out then How do you see cni itself? Evolving or succeeding because even though we write a cni plugin and we are orchestrated independent But if for policies we have to implement an orchestrator dependent Implementation then Do you expect customers to pick orchestrator independent portion for their virtual network and an orchestrator dependent portion for their policies? um it it seems like like either we provide a Good comprehensive networking capability that is orchestrator independent But if we are going to to leave some big portions of it Which are specific to orchestrator then it it becomes challenging for customers You know, I understand first of all, I guess You know, everyone who has tried to build um a generic networking and security Model as a single thing and I'll paint nova neutron daylight any number of other of these Platforms have tried to make the all singing old dancing Model of networking have have have died under the weight of their complexity, right? So first thing, you know, let's separate does this belong in cni? Versus should we have a common policy model? Sort of like what what ken did, you know, we can you know, does that belong in cni does that belong in something else? um, it is one discussion, right and and Simple keep it, you know There's different things and each one of those have a different data model, etc. I don't trying to cram them all into one big nasty I guess I'm showing where my opinion is on that one Big nasty plug-in um, the second one though is The thing that we have to realize is in some orchestrators things like security groups, etc are Specifically network security things in other orchestrators labels tags, etc Are used for a myriad of other functions other than just network? show So That that's the first disconnect right is in some platforms those things are Dedicated and they have dedicated semantics around them like a security group is sort of like a vlan in a lot of cloud orchestrators, whereas labels You know role equals database server or stage equals prod or Application equals Customer front end have many different semantic meanings Outside of just the network policy. Yeah, and I think That's another big You know some of these things have very semantically tied some of them are just metadata that other things can act on Sure I think the scope of what we do in networking will not be to associate semantics with the label itself But it will be more to to be able to apply policies with those labels From a networking perspective now those labels can have other semantics outside of of networking and that's totally fine, but Uh, like you may say front end back end. That's fine, right? But then when you want to apply network policies using the same labels is where Where networking comes in? Yeah Go ahead. Sorry There's a there's a honking big rattle there, but what's Basically, I think what what we're trying to say here is is that there will always be More semantic to tags labels in whatever way orchestra is is expressing their functionality in features to customers What we're looking for is how does orchestrator express those? Those functionality that he needs to run the workload from networking point of view to the network controller Which is a public cloud network controller in case when you bring that run that workload in in azure or aws for that matter If we we want to enforce those policies so in kubernetes world There's a policy controller that integrates with kubernetes and and and enforces those policies Even though they are described in kubernetes, um data store wherever it stores it It's expressed to policy control and policy controller takes it and enforces it So if we can come come to a is standard specification on how orchestrator expresses those policies to network controller That will help us create an orchestrator independent Way of expression of that to to controllers and then we can independently evolve No, I understand that but I'm thinking specifically to show about orchestrators There are orchestrators that Have a very strong semantic type around say security groups Things you can talk to other things in security group a If we map say security groups are just another word for labels Um that all falls apart say in kubernetes world There might be 15 different labels on there each of which can have a different No, completely agree. That might be worse. If somebody thinks You know, I okay, you know How you deal with that discontinuity in not the policy model But the way we've trained people to think about these different constructs and these different orchestrators Uh, it is That's what this continuity is, right? Sure. Sure. I agree with that I think there are a few patterns in what different orchestrators used to describe this commonality of different workloads They run we we can look at those patterns and come through how to Come to a common language specification for policy in that Um, if we if we come to a conclusion that there are so many different variations. There is no way to standardize. That's a different Um, uh observation to be had but I think there's not too many ways There there are few ways people are doing it in in in like we can look at kubernetes or or mesosphere or other network controllers even how they are Or describing policies and and see if we have commonalities that we can take We we really don't need to debate it here. I think I think the next step on the Discussion is for someone to come up with a candidate I agree Brian Yeah, I think we should definitely have offline discussion Again, we can come up with a candidate that the question will come down to Do we even think that this is uh, something that that should be Standardized across I think that definitely would be a good Is it to me? It's a yes. We do Some of us do believe that some of us have in terms of how we would do that but I think it's something we should if if you can do it. I'd be in favor of it and and Okay, uh, specifically. I I would not I would quit a new thing Which which I might call the common policy interface or I don't know what I might call it but I wouldn't uh Well, there's a separate discussion whether what are the benefits or this benefits of actually putting it inside P and I yeah Of the cpi is kind of model myself Brian and and I'm not you know guys don't think I'm against this I'm just saying that this is this is a non trivial exercise Yeah, there's another aspect to what's being you know to one of the challenges that Deepak and crew are speaking to and that's um kind of intermingling of Uh, you know in some respects making vm's first class in In a container network Yeah, I think that's an important aspect of what we are We are bringing yeah, and in part because cpi is um There's a smaller subset of those doing and running containers that That would really you know would truly be In need of c and i being multi container orchestrator certainly Any anyone who's providing containers as a service providing container orchestration And then you know large enterprises that have gotten multiple teams that have just spun up various things and And it's almost like solving that problem is almost sounds like a decent startup idea or Uh, and and I'm sure something that Uh Mr. Marvin, which is he's not Marvin It's Marvin the Martian. It's uh the the account that all our zoom rooms are dialed into Press defer from from tiger But um Marvin the Martian Actually, no Marvin the robot sorry from uh from uh hitch hikers So I know um because you had a few um You're going to talk about some ipv6 but before we transition over to that Um, I think in terms of like the kind of the cpi or network policy groups some of the server Low balancing stuff and service chaining aspects you have on here depot I know i'm interested in working with you to kind of come up with sort of some proposals in terms of how that would maybe be mapped And are there others on this color want to work on that maybe I can set like a second Like meeting with just the smaller group to kind of start working through some of these areas and come back to this team with proposals Yeah, I I'm hesitant. I'm interested. I'm hesitating can about the amount of time I've got but yeah, I'd be interested in at least participating to a certain degree I'm specifically interested in service chaining Uh what people are asking for in service chaining because I'm hearing sort of different things as people move into containers about is Classical service chaining still relevant or not. I'd be interested in hearing this as well Before we move to the ipv6 so any other I didn't see any emails with people who were prepared to talk about any other services that they wanted to discuss with the group are there any they might come prepared with a network service discussion topic Okay, we will I will start with this list um for microsoft and we will we can always add to it Okay, sounds great. Can we'll reach out to you? Thank you Yeah, definitely, please do All right, um, then we only have about 15 minutes left. Sorry, uh curse, but if you want to kind of start with uh quick overview of what you think um is missing for ipv6 and container networking Sure, I can do that Am I muted? Okay, I'm not muted. Okay, probably everyone probably wishes I was but yeah Okay, uh, let's go ahead and Uh present Okay So I didn't have a whole lot of time to put stuff on slides. I apologize but I'm giving this a little bit of a thought um for those of you who Don't know I think probably one here knows me but I've also done a couple of v6 transitions on fairly large networks in the past and Have the scars of doing ops area in the itf networking group co-chairs. So I've lived the dream of getting the world to v6 It will happen next year. We promise this time. Um So there are some talking to folks who are actively interested in in v6 You know the the initial Thinking around v6 and containers and I know a lot of people have thought about that as I will just put a slash 64 on each container host and then Slack how many people here are familiar with v6 things like slack and rd or should I do a real quick? Couple man tutorial on v6 I think I'm very familiar, but if anyone wants to hear you please speak up So the idea of just having containers slack off of a 64 per host Um, it at least initially sounds like a rational thing Um When I've talked to customers about that a couple things come up or potential users One there's a class of folks out there who Believe that a slide and are probably correct that a slash 64 per host Will probably blow out their v6 allocations pretty quickly At the size of of cluster every time some folks who are have tens of thousands of hosts Uh, and all of a sudden that that infinite address space in v6 starts disappearing pretty quickly um the other The other issue that you need to think about is routes longer than a slash 64 Consume four slots in most hardware routing tables so If your v4 routing table space is is 256 000 routes Your v4 if you're carrying greater than slash 64 It's going to be 64 000 routes. So that's a fairly big hit and and slash 64 is or less or shorter Are two x right so the I We can't get away with carrying less than a slash 64 I mean We can't get away with carrying things less than a slash 32 But we should try and limit broad announcements of of greater than slash 64 is in an infrastructure Um, most containers do not have nor should they require an rd demon I don't think it would be a wise idea to say that Every container would have to have a sidecar every pod would have to have a sidecar that loads a a router discovery demon On board with it In order to do slack And classical global ipan mechanisms Will probably not scale with the incredibly sparse address space in v6 right v6 was designed to be slapped If we try and maintain a global address map of say a slash 48 That will probably not scale particularly well in a lot of ipan mechanisms Some of the things that we're seeing from customers as they're looking at v6 automatic configuration of infrastructure They would really prefer not to have to have basically the infrastructure auto Uh configure itself auto discover itself Uh set up periods if necessary, etc Um, and we're also seeing a strong request for v6 only infrastructure People want to preserve their v4 addresses for external services or for legacy workloads So they would really like to run a v6 only infrastructure but That exposes v4 services and supports legacy before workloads given this set of Requirements we've been giving this a little bit of thought Uh, this is one message. This is the only way to solve it. This is one way we've been thinking about this um Each node to give an l2 network is slacked from a slash 64 In a private cloud environment, that could be a top of a rack, you know, the top of rack is an l3 switch It could be an az or the equivalent in a public cloud Etc, but you know every l2 physical infrastructure or a thing that maps to physical infrastructure is a is a single slash 64 each node Would then use dhcp v6 to request a prefix delegation of a larger than slash 64 block for that node This would be within a given l2 network. All blocks are come out of one or more slash 64s So this means that the announcement of the route north of the switch would be a slash 64 in keeping with reducing uh longer prefixes in the core um You know each of those greater than slash 64 blocks could be a slash 80 a slash 96 something along those lines For that node. This does raise an interesting thing We've got now a huge amount of addresses per node Do we ever recycle the addresses or do we use addresses once and and throw them away at least until uh, you know At some point at the heat death of the universe And that gives us some traceability for workloads, right every time you get a new workload You get a new ip address and log events to a given ip address for a given ip address are therefore traceable to a specific instance of a workload rather than An ephemeral thing at some point in time if you're going back and looking at logs later Possible interesting side effect um Each node would then implement host local addressing for its containers out of that block And as I said each of the l2 border routers is carrying its local slash 60 greater than slash 64 blocks But only announcing these slash 64 aggregates Which reduces the address uh table explosion in the core services should be available on v4 and v6 um And We should design service pools, etc. So the external announcements are limited to to slash 64s or less rather than Greater than slash 64s. We don't want to be announcing slash 128 out of services. We can avoid them out to the core um And legacy v4 workloads and containers. Do we do run dual stack or do we do some kind of v4 as a service? uh mechanism um to Again further reduce v4 address utilization and save it for things that are need to be publicly accessed um And some talks around that as well, but earlier in in thinking So and this is just one um sort of straw man You know, is this a rational way of handling v6 in a uh In an at scale uh containerized environment um You know, there's more of a okay, you know, I put something out there. Well, you know You know, what else do what other what other ideas do people have? about dealing with v6 Infrastructure, etc. I think it goes without saying that until we have parity in any orchestration system of v4 and v6 We're not going to be able to really move forward. I know we're working on that pretty hard in in kubernetes, but um, you know We're not there yet. And but once we are how do we actually build Rational infrastructure to support v6 and make use of v6 So with that I'll open it up for questions comments pop shots Whatever I'm trying here. I I have to confess. I'm I'm not a great, uh ipv6 expert, but I was wondering I sat through the talk about using kubernetes with ipv6 at the last cube con in austin and I was wondering To what extent what you just described is compatible with that or different um It is somewhat It is somewhat different but I have some concerns about If people want to do v6 only infrastructure How we're going to manage that at large scale so um, I'm you know, I think um What people have been talking about before and blogging about before have been sort of yeah, we've got v6 up and running on kubernetes, but um You know how we're going to manage this at scale um, the People have been talking about v6 per host. I think is a great idea up to a certain point but there's definitely community out there who Are concerned that they're not going to actually have the address space to to do that with And you mean you mean a you mean a 64 per host a 64 per host. Yeah Yeah, you know so, um And you know What i'm trying to do here with this design is scope um Do a couple things not require networking stack software in pods i.e. do you know pods should not have to slack um and Trying to scope Uh longer than 64 is too a limited blast radius So, yeah, sure. That's where that's where this sort of design came out of Yeah if it it feels like Someone should should try it in real life and report back. I mean, I guess I guess uh, there's a danger that that um What i'm trying to say if uh Let's suppose the 64 per host is is a really bad thing like you say and let's also suppose that that becomes wildly popular amongst early adopters, you know that right so there's a danger to Standardizing too soon, which is you don't really know whether what you're saying is is a good idea And there's a danger in standardizing too late because everyone's gone off and done the wrong thing Right, so I think the the the issue is the vast majority of people slash 64 per host Will work, you know, because most of the cluster sizes you see are in the hundreds 2000 and that's that that's perfectly fine for I'd say a slash 48 block that the the issue is going to be You know the larger style deployments, but if we start like you said brine if we start baking things like Slack for containers pods in That's going to make Things more complicated later if we discover that why why I mean, I don't understand why That would like why why wouldn't they be the cni plugin that would be doing slack? so So in order to You could yeah, you could possibly do that brine. You'll have to generate a mac address Pseudo mac address for everything that you're Proxying that you know that that you are going to do Sort of a proxied rd4 but You could Do that the question is also going to be though Scoping that correctly, right? So rd is normally go to If we think about that if let's say I'm going to rd to something from the cni plugin it could be the real router That's upstream of me or it could be just something that is You know, that's not really slack. It's it's a it's an ipam request and I think that's when we run into managing very very sparse ipam tables Um But if the local cni plugin could then do slack To the top of rack that might be another that might be another option and then the slash 64 is coming out of um The top of rack switch or the border router whatever is doing the rd within that l2 zone That's another way of doing that's another possible way of doing it brine Yeah, so I guess we're running out of time. Where would What where do we go forward? Where would something like this live? Like if this is like a document like best practice or for v6 and containers Yeah, we started talking about this can what at the beginning of the year sometime last year of you know Is v6 one of the things that we want to? uh sort of Help provide some guidance. I think I remember Memory being faulty the ken I had been talking about you know, is this is our best current practice or should we should we document something? Yeah, right? You know that might be um You know the this might be the starting point of a bcp um for v6 I'm not sure there's a standard to be promulgated here as far as you know, this is the way You must do it. You know, this is Your you know people are going to do v6 here cns the cncs Suggested mechanisms to do that both for adopters and and for people writing Infrastructure that goes into cncf You know into cloud native infrastructures um My view is more of a document than a promulgated standard suspensions. We're not a standards organization right So, you know, we could start a document um You know, we could See about doing testing some of this in real life and seeing what works. What doesn't I think The problem is going to be the the problems are going to pop up at scale I think anything work at small scale, right? It's going to be testing into at big enough scale where we can possibly start dividing issues Yeah I think it'd be worth getting in front of people like dan williamson kc calandrello Who are Other cni maintainers that that do more ipv6 than I do Yeah, so I've I've obviously I've discussed this with rkc uh here at tigera a little bit already as well, but um Yeah, getting this in front of some of the the more v6 Savvy cni guys is would be maybe a next step Yeah, so, you know, you could you could do that via an issue on the the cni github or or the email list or Right because as it stands, I don't think this drives any specific cni Changes No, that's fine. I just you know, you could file an issue that said I just want to bring this to your attention for a comment You know, yeah Or just you know, there is a mailing list. I can't remember what it is off the top of my head Yeah, I don't believe anyone's ever sent any emails to So I guess it was a sort of 10 to you and the rest of the group What do we want to do is next steps for something like this? Have an opinion about like to hear from others first I guess are we even interested in waiting into the v6 pool in this group? You know, I was just finding guidance. I'm okay with it You know, I've been surprised by the lack of v6 support and a lot of the things I've been doing recently. So I think it'd be good to have some sort of statements about v6 coming out of the work group so maybe a v6 sort of position position position statement and then Start drafting up some kind of bcp Yeah, that would be sort of a living document that would grow as people start filling in chunks of it Yeah, I think that would make sense Anybody like finally opposed to that Does anyone's still awake? I think so let's let's start with I know it over time. So we'll start with them and in two weeks I'll send some emails up. I hope in two weeks. We've made a little bit of progress on the services piece and on the the v6 idea around You know kind of a position statement some kind So ken do you just want to do that document in the and get help? Yeah, that'd be a good idea All right guys, thanks everyone. Thanks for your time. See you guys in a few weeks. Bye. All right