 Hey, guys. So I'm Abhishek. I'm an engineer with cloud scaling. So OK. There you go. If you don't know cloud scaling, we are an enterprise-grade OpenStack company. We do OpenStack for our customers and end-to-end solution based on OpenStack. So starting from planning the data center to training, deploying OpenStack to support everything. So I started with a focus on software-defined networking in my career. And then when I joined cloud scaling, I ended up as a jack of all trades and doing everything from, sorry? Oh. And slower. I ended up doing everything for cloud scaling, starting from deployment to support to sales to marketing, basically everything. I have to say I'm still pretty bad at configuring actual network devices. I can barely manage to write the code. I'm Randy Bias. I'm the CEO and founder of cloud scaling. I am or have been in the past a hands-on router jock in building ISPs and large-scale data centers. So I'm just here to help Abhishek out. He's more the SDN software side of the networking part of the equation. And I'm representing our network architecture team, which is probably about three, four people strong. Sure. So we are going to explore our network architecture in detail today. And we will start with the motivation, the design goals that we have for the architecture. We will describe the physical topology in detail. We will look at the software components that actually make this happen. We will look under the hood how each component works. And we will end with our future directions based on SDN and moving to Neutron. By the way, our networking model is mainly based on Nova Network. We run as a Nova Network plugin. So let's start. But let's first look at the vanilla OpenStack networking modes without the Nova Network. First, let me ask a question before we jump into that. How many people are using Neutron? And how many people are using Nova Network? Oh, OK. More Neutron users. OK. How many of you are finding problems with Neutron? OK. I don't know. How many of you are having problems with Nova Networking? Oh. All right. Oh, but you're probably using a default networking mode, which Abhishek's about to talk about. So yeah, Nova Network supports three basic networking modes. The single host mode in which all the traffic goes through a single x86 box running Nova Network. The multi-host mode distributes the network manager into the compute nodes. And then the flat networking model. And on the right-hand side, we have the two networking modes that we have in OCS. The OCS L3 networking mode that we are discussing today. And we also have an integration with OpenContrail, that is a VPC-type networking mode in OCS. OCS is our product open cloud system that's a cloud operating system based off of OpenStack. Right. So the single host mode. All traffic goes through a single instance running Nova Network, an x86 box. It's pretty easy to use. How many people are running this? All right. That's fine. It's pretty easy to see that at some point of time this will run into scaling constraints. And when that box goes down, the whole network goes down. Not a very good situation to be at. So this also necessitates VLANs to support multi-tenancy. And that comes with all the problems with VLANs. All VLANs run out of possible tags at some point of time. And if the tenant requires multiple mode nodes and there are tags, then you're out of luck. The second mode is multi-host mode in which the network manager runs on the end boxes, the compute nodes, and is distributed throughout the whole network. The problem with this is that the network manager is responsible for providing NAT services. So in this case, NAT has to come down to the compute nodes. That's not cool. You don't want to expose your compute nodes to the internet in any case. And the third mode is the flat networking mode in which, again, no one network is centralized. And that runs into scaling issues on the network plane. So we didn't like these networking models in OpenStack. We had to come up with something of our own and which scales pretty well and meets our customer needs. So we started looking at what others have done or had to solve the same problem. It turns out that a lot of other people have actually come up with their networking models on OpenStack, on top of OpenStack. It's also turned out that a lot of them use Neutron and not Nova. I'm sorry, it's the other way. They use Nova Network and not Neutron. So CERN uses a custom driver for Nova Network that has a flat database of IP to map mappings for all of the networks. And when a VM boots up, it requests an IP address from that database. Nova Network goes and grabs an IP from that database and assigns to the VM. ANL, on the other hand, added infinite bands and VXL and support to Nova Network to increase the throughput. Given that information, we were ready to go and design our own network architecture on top of Nova Network. And here's what we have. Let's look at the architecture in detail. So what are the design goals for this? At Cloud Scaling, we believe in hybrid clouds. We believe that OpenStack's future is in working as a hybrid cloud to EC2 and GC. That makes us support hybrid cloud better. And we have to have EC2 style networking in OpenStack. That was our primary goal. We wanted to use pure L3 as much as possible because pure L3 is fast and it's well understood by people. People can just go and debug it. You don't need any extra expertise to debug a pure L3. We did not want VLANs because we did not want to run into all the scaling issues. And we did not want to run any STP because spanning tree protocol because STP requires a single interface to be on and be the primary to eliminate routing loops in the network. And that kind of kills HA in a way. Who's had a massive spanning tree protocol failure in their network? Oh, come on. I usually get like almost everybody. We did not want to have any single points of failure in the network. And that rules out single host and flat mode in an over network. We wanted to do distributed DHCP at scale for all the VMs. And we have a very clever way of doing that, which we are going to see in some time. We did not want to have any virtual routers in the whole packet path in the data plane and the control plane. So why L3 and not why anything else? That's because L3 scales. And that's pretty much a known fact. Layer 3 networks are hierarchical. That enables route aggregation. As you go up, you can always aggregate routes and announce aggregate and not individual routes. The best example of that is the internet. The internet scales like hell. Large cloud operators do not want VLANs. VLANs limit scalability in a number of ways. And a pure L3 model always forms a great underlay for an overlay SDN, which is where the whole networking industry is moving towards. And why not L2? This slide shows a breakdown of all the problems. L2 is where the topology is flat and the addressing is flat. So each router has to maintain a database of all the entries in the network that obviously does not scale for a huge cloud provider. And then we run into the same problems with STP and VLAN, spanning trees and VLANs. We wanted to have small failure domains. So if a zone node fails, it takes down one part of the network and as small a part as possible so that the whole network does not go down. And we have some throughput from the network. So an example there is if you're running single host mode in open stack and you have HA with Linux HA and you have a failover event, then your networking basically stops for 60 seconds for your entire cloud. I mean, that's not really very acceptable. If you look at the larger scale cloud operators, somebody like Google, they're running four, five, nine uptime environments. And they have regular outages all the time, hardware fails all the time. But it only takes down a very small piece of the system. So it just tends to make a lot more sense to have a networking model and overall cloud model that encourages failures that happen in isolation. I mean, we've all run production environments. And the worst thing that could happen in a production environment is you get a cascade failure where A takes down, B takes down, C. And then it's a complete nightmare to sequence it all back up. So I think this is a principle that still gets lost a lot of the time. You see people betting big. We're going to get a double pair of EMC Vmaxes in, and that's everything. That's the entire system. And that works great until it doesn't. And then when it doesn't, you basically have lost the entire system. And so we really felt that that's an anti-pattern really these days. Sure. So our physical network topology is derived from the spine and leaf model that Arista popularized a few years back. The spine and leaf model has a network of routers which forms the spine. And then top of the rack switches on the other end of it which forms the leaves. The most awesome thing about this model is that you can increase scalability by horizontally adding more routers and more top of the rack switches to the whole system. And nothing breaks. Yeah, spine and leaf is what allows us to focus on maximizing the amount of bandwidth that you can get with East to West traffic, right? Because we're getting rid of spanning trade. We don't have any valence across racks. So we get all that capacity back, and we want to use it all. And as you probably know, I mean, if you look at most modern data centers, even if it's not a web scale data center, and the amount of bandwidth coming out of the front end is some fraction of the overall bandwidth in the back end of the system. And so you want most traffic in most networks is East to West. And so a spine and leaf allows us to build a very big network with very cheap gear. You can even use one rack unit, 10 gig switches for that spine initially. And you can scale it out pretty far and you can get tremendous amounts of East-West traffic, which makes all kinds of problems kind of go away. So another awesome feature of this architecture is the ability to use of using OSPF or BGP on both sides. And then we can do ECMP so that we get a type of HA. Basically, a zone node can go round Robin and try each of the links using ECMP. And then whatever works, it uses that. How many people are familiar with ECMP and our network folks equal cost multipathing? Oh, that's fantastic. That's awesome. So just to clarify just a little bit on what Abhishek was talking about, we use ECMP obviously between the spine and leaf so that we can have n number of links between the spine and the top rack switches. But we also do some other clever things with it where we have physical servers or virtual machines participating in the OSPF routing fabric and basically being nodes on that. And that's one of the ways that we get load balancing for like our API endpoints. Rather than running HA, we run load balancing and we actually have each of the API endpoints participate in OSPF. A little bit more complicated than that, but basically participate in OSPF so that the switches are doing the load balancing across the API endpoints. That also removes any overhead of using static routes and leads to a fewer number of problems in the network, debugging problems. So given that background, let's look at the physical topology the network has. So this shows schematic diagram. At the top, we have the data center switches, whatever the data center provides us, which comes down to an edge network which forms the edge of the cloud, the customer cloud. There is an optional NAT service that I'm going to describe in some time that sits in between the edge network and the core network. And then we have the core network which forms the spline network. That comes down to the leaves, basically, which are the zone racks or the resource racks, all of the racks, which actually hosts the zone nodes, the compute nodes, the storage nodes, all of the nodes. So let's look at what are the software components in the whole network architecture and how they work. We obviously have NOVA network because our network plug-in runs as a plug-in in NOVA network. That forms the network manager. And the network manager is distributed and synchronized across all the zone nodes. We use MySQL and Galera for HA on my top of MySQL. That helps us to achieve seamless load balancing across all the multiple databases, multiple zone nodes. And whenever a zone node goes away, we can always have a multi-master failover automatically. We don't have any downtime for failovers. And this also means we can run as many zone nodes as we want to. And when we want to add a new zone node, we can just plug it in and configure MySQL to it. And we are up and running in no time. So just a little background. And we built our system a little bit differently than some of the other folks who built products were on OpenStack. When Obeshack is talking about his own node, he's referring to the control plane. So the control plane for our system is scale out. But it's centralized into a pair of racks, typically, so that we can do security enforcement around it. Because the security policy you want to enforce on the availability zone control system is different than the resource nodes. So we don't run a hyper-converged system where all software is running in all systems. And it's actually different. So it's scale out, but it's on a set of fixed hardware and network infrastructure that's designed for the control plane, which is separate from the data plane. The control plane runs on a set of zone nodes, as many as you want. Sure. So the other component of it is the virtual interface driver that sits on the compute nodes. And it's actually responsible for provisioning the network for the VM. It creates the network interfaces for the virtual network interfaces for the VMs, assign the slash 30 addresses to the VMs. And it also provides DHCP service for the VMs. And Nova Network actually provides the EC2 style metadata service for the VMs. OK, it also configures routing, obviously. So we have an optional NATR component that can sit between the Edge network and the Core network to provide floating IP services for the VMs. And we have a custom utility that we call L3 AdNet that is used to pin slash 30 addresses at slash 30 networks to each of the compute hosts so that the VMs in those compute hosts can use those slash 30 networks. How does the NATR work? The NATR is actually very simple. Can you go back for a sec? Sure. I just want to clarify another thing. So it may not be immediately obvious. When we put down a rack, that rack has a network routed to it statically. And then that network is chopped into subnets, and each of those are statically routed to the hypervisors in the rack. And that allows us to have a relatively stable network that doesn't have a lot of things moving around, keeps OSPF very simple because we can do route aggregation at the top of rack switch, obviously. And it also is what allows us to have a very clean layout. You know which network block is assigned to every hypervisor system. And then out of that, we carve out slash 30 networks for each of the VMs that are running there. And then that allows us to actually know deterministically on a given hypervisor which IP is mapped to which virtual machines. And then to derive the MAC addresses from the IP address. So it's kind of a relatively static hierarchical layout. Like you would expect kind of ISP days with dial-up modems or even broadband DSL type kind of system. So how does the NATR work? The NATR is very simple. The NATR sits between the Edge network and the Core network. And it actually has a config file which has Nova's credentials for accessing the database and so that the NATR can actually access the Nova database. The NATR keeps on polling the Nova database for new floating IP and private IP pairs. And whenever a cloud operator goes and assigns a floating IP to a VM, that goes into the database. The NATR sees that new entry. And the NATR knows that the entry has not been processed yet. NATR then uses DC rules to install one-to-one NAT rules on its outgoing interface. And so that whenever a packet goes out, it can be written. And when it comes back, it can be written again and goes back to the VM it wanted to go to. This actually shows inside of the NATR in three possible cases. The first case is when a traffic from a VM tries to go to the internet. When it goes out, its source destination IP and the port are rewritten. When it comes back, it's rewritten again so that the packet goes back to the VM. The second case is VM to VM NAT. It doesn't really need to go all the way out, out of the Edge network. So it goes to the Edge network and comes back again. The third case is cell-to-cell VM NAT. And in this case, also, it comes back again from the Edge network. So the key with the NATR layer is that we designed it to be between the core and the edge so that any traffic inside the cloud basically doesn't go through it. It's when you're egressing the cloud that you actually transit through the NATR layer. And then a typical NAT system that you would see in a load bouncer or a firewall is sort of stateful. And you have to do synchronization. But we did the same trick where we're basically using ECMP to load bounce across in NAT devices. And those NAT devices basically have the same configuration because they're pulling the NOVA database, pulling it out, and basically setting up the same rules on every system. So they don't need to be synchronized or synchronized through the NOVA database. And so we can run any of those at scale. And when we tested it, we get about five gigs a second out of each Linux box for 50,000 one-to-one NATs. And we haven't really done any performance optimizations on it. So you run 10 of those, and you can do 50 gigs a second out of the front end of the cloud for NAT, which is very good, considering you're using very cheap and expensive 5K Linux boxes. And the NATR piggybacks on the zone nodes for maintaining the database state, which is again HA. So ultimately, the NATR is totally stateless. The other side is the WIFT driver and the NOVA network working with the NOVA network manager to build the VMs and the networks for the VMs. So whenever an administrator launches a VM, it gets scheduled to a particular host. And the WIFT driver in that host gets called, and it builds the Linux bridge for that VM. The network manager gets which host the VM went to. The VM got scheduled to, and lists all the available networks on that host. It finds the first unused IP that can be used for the VM and calls the WIFT driver again. The WIFT driver then goes and creates a virtual interface for the VM to be used by the VM and the bridge, a Linux bridge. It then sets up a UDHCPD process for that VM so that the VM can get DHCP services. The Mac to be used by the VM is calculated based on that IP address, and DHCP uses it. Then the virtual interface driver creates a slash 13 network for the VM and assigns an IP to the bridge and one IP to the VM, and then starts up the DHCP process. It then adds routes so that the VM can talk to the gateway and adds, we actually have two interesting features. The feature, the inability to block a particular set of networks, and then the ability to whitelist particular hosts in those networks. These are enabled by adding IP table rules on the compute nodes. What that's used for is to protect the control plane from the data plane. I mean, you need multi-layered defense, right? You don't want to depend on any one mechanism, but we want to make sure that from the hypervisor layer, we're blocking any packets that try to get out to the actual control plane in the system. And because it's all fully automated, when you set up the control plane, the IP address ranges that are in there are basically saved in a centralized metadata store. And then when we do the automated provisioning of a host, the hypervisor node, those whitelists and block networks are just automatically added. So you don't really have to manually manage that. So the dual is the event where the VM is removed from the compute host. The drive goes and stops all the DHCP processes. It cleans up all the config files. And the network manager then deletes all the IPs associated with the virtual interfaces and cleans up the Linux bridges in the host. So the L3 AdNet utility that is used by cloud operators to pin networks to hosts, it takes in a cider, a host, IP, or host name, and a pair of DNS addresses to be used. It loops over the input cider and breaks it into its slash 30 chunks. And then loops over each of those chunks and uses the lower managed API to create an entry into the database. So that gets used by the scheduler because typically in OpenStack you've got a sort of a global pool of IP addresses when you're doing dynamic allocation. And the assumption is that you just cherry pick out of that pool for all VMs everywhere. So this basically wraps it so that you say this suddenly it goes to this hypervisor node and then a bunch of slash 30s get created in the database. And when the scheduler picks a node that the VM is supposed to be allocated on, it's got the smaller pool of slash 30s that basically pick the VM out of whatever is not used and allocate that IP address to that virtual machine. Let's look at the challenges that we faced while writing the Custom Nova Network Driver. So OpenStack does two releases per year. It used to do one. So that has always been a moving target for us. More often than not, the plugin interface has changed, the library dependency has changed. That brings in the extra overhead of making sure that the thing works and all the regression tests and testing it extensively. Initially when we started out, the database API wasn't rich enough. The SQL Alchemy API wasn't rich enough. So we had to actually write straight SQL queries in the matter. Well, Nova Network is supposed to be deprecated in favor of Neutron long back, but that never actually happened. So we haven't really thought of moving the whole network architecture to Neutron. So let's see where that takes us. The most common question that we always get asked is, why not Neutron? And why not Neutron right now? Primarily because when we started during Diablo, Neutron wasn't around. So there's no question of using Neutron. We don't really think Neutron is stable right now for use in large clouds. It has a few problems around API changes and interfaces not being very stable. The command line interfaces and the procedures are not very intuitive and not very operator friendly. And then not all Neutron plugins are equally usable and equally good around the network architecture. The other thing I'd add there is it, it still feels like Neutron is being used to try to be the network as a service cloud system. But I mean, does anybody really believe that we're gonna like model layer two through layer seven semantics and everything that the network industry's done for the last 30 years? Load balancers, caching appliances, wide area, acceleration products, firewalls. Like, are we gonna build all of that into a single unified API? Like I just, I can't wrap my head around that. It seems like a boondoggle. Like, that's one of the things here that I think has to get sort of a rethought around Neutron. We need to keep the scope down a little bit tighter and then maybe have a set of services that provide us different API models for firewalls and load balancers and sort of a higher order network services above layer two, layer three. So our problems with Neutron brought us to like trying out different plugins and then we ran into OpenContrail. That actually proved to be a very useful thing for us. OpenContrail met all of our requirements. Primarily because OpenContrail uses all commodity technology and nothing custom on top of it. It's just layer three with MPLS, GRE, and I don't know, XMPP, XMPP, some BGP, all of these. So our network underlay, that network architecture forms a perfect overlay for, perfect underlay for OpenContrail to run on since it's pure L3 and it can just work on top of it. So right now we have two network modes in OpenStack or in OCS. One is the default L3 networking mode and another is VPC with OpenContrail. Generally customers who want to have seamless scalability for its tenants who don't want to use VLANs, they will offer the VPC specific deployment with OCS. Yeah, so sometimes I get asked why did you choose OpenContrail and it's kind of a long story but the short answer is that I engaged with people like Nasirah in 2008 while I was at GoGrid around SDN and OpenFlow and it just felt a little experimental and I know there's a lot of network people in here and we always get nervous about putting kind of experimental stuff in the network. At least I certainly do. And you know, Contrail at the time, now OpenContrail was interesting because it was basically well-known protocols, MPLS, BGP and all that good stuff and it wasn't too experimental. I mean it was reconfigured in a different way and we're doing L3 all the way down the host and doing L3 over L3 encapsulation. Those are a little bit new but I felt like that was a lot less experimental and a lot less proprietary because a lot of the OpenFlow vendors were actually putting extensions onto the OpenFlow protocol to kind of get it to where they needed to be. And so that's kind of why we started there. We're not married to OpenContrail but it felt like the least experimental approach with a lot of the right kind of architecture with scale well. Same could be said of Nuage which also takes kind of a similar kind of BGP type based approach for scaling. The second ties down to our belief that the internet scales awesome level so we should just model the internet and use something like BGP that can aggregate routes up. So we also get asked what's the packet path for our L3 model and the VPC model and let's look at that. In our L3 model when a VM tries to communicate with a VM in the same network, it goes up to the spline and then comes back. It does not go up to the edge network at all. But when a VM tries to communicate with the internet, it obviously goes up to the edge router and then goes out to the internet. Right, so here we're getting the advantage of using that ECMP mechanism between the top of rack and the spine for that sort of scale out model. So you can design a network where you've got racks of compute that maybe have 20 gigs out of the top and are designed for sort of web server instances and then you can have other racks which maybe are running block storage which have 160 gigs out of the top of the rack. And so you can sort of actually vary the bandwidth to the spine and then maximize that east-west traffic as you might say, okay, for every block storage rack, that's gonna support somewhere in the order of four to 10 compute racks and you can appropriately set up the network bandwidth to sort of meet that need and get that load bouncing between the layers because you're just doing direct routing straight from the hypervisor node to whatever the other endpoint is inside the cloud. Also notice that this does not use encapsulation in any way, reducing the overhead of encapsulation. In contrast, open control does use encapsulation using GRE, that forms pretty nice L3 over L3. Open control layer three over our underlay network which is again layer three. So every traffic that comes out of the virtual router running on the compute nodes is encapsulated using GRE and so when a node wants to talk to a node in the same network, it goes through the spline router and goes back to the node. When a node wants to talk to the outside internet using a floating IP, it gets decapsulated at the edge router and goes up to the internet. So in this case, the edge router has to be a device which can actually read and write GRE. Future directions. So we know that we have to probably move to neutron at some point of time when over network is totally deprecated. We don't really know when that is and let's see what happens with that. We are looking forward to adding more SDN type capabilities using OpenContrail to the OCS product and we are most interested in the service training thing that they do. You can place arbitrary services in the network using the service training feature. That's pretty awesome and that's very useful for NFV type services for service carriers, for example. And that's pretty much what we have for today. That was Abhishek's very first presentation ever. Excellent. I didn't even have to ask who's got questions. Just write up to him, Mike. I've got a couple questions. Number one, so just trying to dive deeper into the model, it sounds like you're still running your DNS mask IP tables. No DNS masks. OK. So what service are you running for? I'm just scaling out DICP. It's UDICP from Busybox, UDICPD. And are you still using IP tables on the compute node? We are using IP tables on compute node. It scales well enough. Yeah. But you're just not running NAT on those IP tables since you're running it on that NATR, right? And is that basically what the NATR is running? Is IP tables just with NATR? It's using IP route 2, which is in the Linux kernel and it uses TC to configure the IP route 2 in the kernel. So basically, think of it as a very fast way to rewrite packet headers as the packets go in and out of the external network interface. And we just have a one-to-one mapping of what the NAT is. And basically, as it goes out, the packet gets rewritten. Last question I have is you brought up the point around some of the limitations or challenges Neutron has. You helped me understand why this work wasn't, or why didn't you focus your effort on bringing this work into, I mean, because it's really good. I mean, it looks like awesome stuff you guys are doing. Totally bulletproof. Yeah. So help me understand why you didn't work with the Neutron project and get this stuff in there as opposed to doing it in Nova Network. And the reason I asked that is just I feel like from a user and just a general OpenStack perspective, it causes confusion within the user community of, OK, maybe we shouldn't go to Neutron. You know what I'm saying? Especially when you say the roadmap where it's like, hey, we're going to have to move this over to Neutron. Totally. So first of all, Neutron or Quantum at the time was not even an incubation when we built this stuff. And how it sort of happened is we had a bunch of people who had worked at ISPs and carriers. We were doing stuff, building clouds before OpenStack even launched and existed. And I went to Chris Brown, who was the lead developer for EC2. I said, how'd you guys do it? And he basically told me. And I was like, that is freaking smart. And I said, we're just going to do that. And so we started down the path but weren't complete when we did the big cloud stack deployment for CREA Telecom in 2010. And then when OpenStack came out, we immediately saw that it made sense to build it in there. And that's where we started doing it. There's just no option to go into Quantum or Neutron, right? And then as things have evolved, it's just been challenging to figure out how to get it back into the community for a couple of reasons. One is, I'm a small startup. I'm resource constrained. Two is that this is a network-centric approach, unlike some of the default OpenStack networking models, which kind of were made by systems guys who don't really understand networking. And so there's a level of sophistication there that can be a little bit difficult to explain, like using ECMP to load balance across a bunch of Linux for NAT at the edge. I mean, that's just a little challenging for people to get. Now, we're open to pushing it back in. If I could get some help, we'd do that. And we're open to getting it in Neutron. We're not trying to hold it back. It's just sort of a resource and sophistication kind of problem. Yeah, I mean, have you filed any blueprints and working with a Neutron group, getting any of their feedback yet? No. No. Give me a couple of engineers full time for about three months, and I'll get that done for you. Well, I mean, even if you just file the blueprints and then socialize it, I mean, that's part of what the summit is for is to get people behind it, right? Because, again, I mean, it's good work. And I think people will get that done. I think that's fair. So you're signaling us for time? Right. Four minutes? All right. This is from Keshav from HP. Is there any data you try to do on L2 with the ISS and running with Trill? Because you're saying you don't like L2 and you only want to use L3. Is there any data available using the same thing with ISS on L2? Using ISSS instead of OSPF on L2 itself? No, but it shouldn't be any reason that ISS wouldn't work. I mean, I assume you're not asking about Trill, but just ISS, right? Right. Yeah, I mean, why not use ISS? It's perfectly legitimate. We just like OSPF. One of the reasons that we really like it is it's got a really good link state detection failure on the interfaces for ECMP. And it converges really fast if you've got a relatively small number of networks and you do aggressive route aggregation throughout the network. It's pretty bulletproof and we know it well. It probably doesn't scale as well as ISS, ultimately. But for any size, style, cloud, and enterprise is going to build, it's going to be fine. You can jump in here. It's a great model, I think. It's good to know a different model of provisioning through network. No one network, but I think, again, going back to the previous point, integrating with Neutron would be awesome as well. So one question I had was, so you use slash 30, you use per VM, actually. So that's consuming more addresses than just one. So is it V4, V6, and? It's V4. This is probably the weakest part of this. And again, it's been a resource issue. What you could do is instead of allocating, instead of subnetting out the network after you route it to the hypervisor, because the hypervisor isn't consuming any of those IP addresses directly, you could just have that entire network on the hypervisor and you could do network enforcement so that none of the VMs on the node would see a slash 26 or whatever was appropriate, but they wouldn't be able to see any of their neighbors. They'd just be able to see their gateway. So you're actually doing the partitioning of the switch fabric, basically, at the switch levels. That makes sense. I mean, we could make it more efficient just having, due to time constraints. And IPv4 is challenging because you're running out of space. Trust me, I got this gigantic enterprise right now and they literally can't give me any RC 1918 space. They're stealing public ranges that aren't routed in order to get around it. So it's definitely a challenge. We need to fix that and address it. And it would be something where we'd get the advantage if we could push it back out to the community of having other people help us fix that. Because it's not that much code. It's just one of those things where it falls off the priority list. Right, and the model of deployment matches pretty much how V6 deployment would happen with each rack having their own subnets and further dividing it to host. I mean, to be just totally honest, we were kind of shocked that people had this level of interest and there were 300 and some odd people there were going to come to this presentation. I didn't actually realize that people would be that interested in how we solved these network problems. It was a little bit of a surprise to me. So I'll take that as feedback. One question my friend had here was overlapping IP addresses. How do you handle that? We don't, not with the layer three stuff. That's what VPC is for. How are we supporting multi-tenancy? If you want multi-tenancy and you want to do basically. Yeah, overlapping is not supported then. Well, so multi-tenancy can be a number of things. They're signaling me to get out but you guys still ask me questions so I'm gonna keep going. And they can cut my mic, I'm loud. But multi-tenancy can mean kind of like how do I enforce security between two tenants? So like Amazon does that with security groups which are supported in OpenStack which we fully support for layer three so you would use that for network partitioning. But if you want to actually do isolation, full isolation of the IP, then you need to use virtual private cloud, our SDN based product uses ConTrail as an option on top of it. And then you get encapsulation in virtual networks and tunneling in classic enterprise networking model. Okay, thank you very much.