 Good afternoon. This is my first OpenStack Summit, and I'm presenting. So it's quite an exciting moment for me. And based on some of the session that I've seen so far on the networking track, it seems like networking is still quite important for many of the folks sitting here. So we do care about the network. It's just quite exciting time, actually, for the folks who are doing the networking because of the IoT internet of technology or internet of things. And especially, the NFB use cases puts a lot of demand on the network. So my talk is going to be primarily focused on how do we address the scale, how we are going to achieve the sort of like the modularity with the right kind of performance and right kind of security. My name is Milo Hoda. I'm a lead architect part of Cisco InterCloud Services Group. So in this session, I'll be talking about how we're going to achieve the scale of the network, both the overlay as well as underlay. And what I would like to call an integrated overlay. Integrated overlay is all about bringing the virtual world and the physical world together. At the end of the day, that's quite critical. And everything that we do from a networking standpoint has to be exposed through OpenStack. That's our unified API. So the focus of the talk is also going to be making sure that anything and everything that we do to address the scale of the network is exposed through the API of the OpenStack. So here is a quick agenda. Before I dive into the InterCloud network architecture and scale, I'd like to give you a 30,000-feet overview of what the InterCloud is all about. Then I'll go over the tenant network connectivity model because that's quite important for us to understand. We are not defining our use cases or our requirement sitting in some kind of lab. These are driven by our tenants, our customers. Those customers are internal SaaS application provider. Also, we are talking to different allies, partners who are also feeding us those information. And then I'll talk about the network architecture. Once we understand the different connectivity model and the different requirements, then I'm going to discuss how we are addressing all of those requirements through underlay as well as overlay. At the end of the day, your underlay foundation has to be very strong. Underlay network has to scale itself. It needs to be performant. It needs to give you the kind of flexibility that you are looking for. And at the end of the day, you need to be able to manage it from a day to operational standpoint also. So I'll talk about all of that. Scale without the right kind of performance optimization is not going to give you anything. You might be able to achieve a large number of scale, but at the end of the day, your system needs to be performant as well. So I'm going to talk about some of the performance optimization that we are doing to meet our tenant connectivity requirements. Then I'll go into L4 plus services as well and how we can achieve the scale for L4 plus services. And everything that we do, we have to be exposing through the unified OpenStack API. So I'll talk about the tenant consumption model as well. What are the different approaches that we are taking there? And some of the future direction. And towards the end of the session, I'd like to have some Q&A from you and some feedback. So what is InterCloud? InterCloud model is really based on the Cloud Federation. At the end of the day, it's about interconnecting and integrating the Clouds, primarily OpenStack-based Clouds, operated by Cisco and our partners. We need to be able to deliver infrastructure as a service or pass and sass across all of this Cloud in a centralized manner. Any service operated by anyone, deployed anywhere, should be able to be consumed by anyone. So that's the goal. So here's a quick example of how a tenant can go ahead and consume the services. The tenant is sitting probably somewhere in Europe, should be able to consume the services across the globe. The idea here is to be able to interconnect all of those Clouds and then be able to expose those services in a unified manner so that the tenant can come in any geographic location. But they should be able to consume the services from our Cloud platform. To make this happen, there's a lot of network-related capabilities that we need to offer. As you can imagine here, there's a lot of Federation that needs to happen. We need to be able to offer consistent networking services on each of those Clouds. So that will give us the ubiquity. Federation of the network services so that the users can view all this collection of inter-Cloud services in a single resource. So that's quite critical for us. We need to be globally consistent. That's the key thing that I would like to highlight here. At the end of the day, if you want to build such a massive Cloud that needs to come together in a unified manner, you need to make sure that all those Clouds that you're building are unified. I mean, it's repeatable. You can build it at scale. So that's another thing that's quite important. And then we need to be able to be consistent on the identity front, on the security front, as well as the network addressing scheme. These are all important stuff for the Federation. So interconnecting all of these, the virtual private networking is another key element for us. When you create this virtual private network, you've got to make sure that those network can span across the different Cloud as well. So that would require you some level of segmentation or isolation between the tenants. So how are we going to achieve all of these? At the end of the day, all those Cloud providers need to come together and interconnect them through some kind of private or public internet action points. So that's how they're going to be interconnected too. Now, as I mentioned, we are trying to create that reference architecture. At the end of the day, all the small boxes that you see there, let's say CIS, those are the individual fabric that we need to build. And that fabric needs to be built in a very repeatable manner so that we can scale. Now, the way we are trying to come up with this architecture is based on Cisco ACI fabric network technology using OpenStack. So let's quickly zoom into what it means. But before I do that, I would like to quickly discuss the network, the tenant connectivity model. I think it's quite important, as I mentioned. I mean, at the end of the day, you need to be able to expose those networking capabilities to your tenant. And different tenant has a different level of connectivity requirements. So I'd like to quickly walk you through what are some of those connectivity requirements. Once we understand that, then we can define the requirements for both underlay as well as the overlay. At the end of the day, we are taking a very top-down centric approach. We are not really defining the use cases by ourselves. These are getting, we are getting them from our tenants and from our customers. So we are taking this top-down approach. We are looking at what are the services that we're going to be exposing. And then we are designing our underlay as well as the overlay network based on that. So in this picture on your right, this is one of the CIS, sort of like the fabric or the pod or the region, however way you want to call it. At the end of the day, you want to go ahead and you want to build that fabric. And as far as the different kind of connection goes, so through the IXPs, some of the enterprise customers who has a requirement for low latency application, they need to be able to connect to that pod through the IXP, through the internet exchange provider. So the idea here is to extend the enterprise customer network, which is tenant A, that network needs to be extended, although up to this CIS pod. So that's what we call a private link direct. This is a direct connection. Now the second type of tenant might be like the tenant B. These are our alliance partner customers. These are the customers that our alliance partners already has. Telstra, for example, they have a large number of MPLS L3 VPN customers. And if the Telstra wants to offer the cloud services, then how are we going to bring the Telstra MPLS L3 VPN customers into our cloud, in a fully isolated manner? So there is a notion of this MPLS L3 VPN mapping down to our cloud. And then you have the other type of customers, which might be the MPLS L3 VPN customers that are offered by some kind of SPS. And they have the internet exchange point connectivity there. So they can also extend their MPLS L3 VPN network connectivity to our cloud as well. And the other type of connections are like the overlay, VPN as a service. This is what typically what if you want to just use the internet, then VPN as a service is sort of like an overlay, VPN services that can be offered from the pod. So this generates a lot of demand on the network, as you can imagine, both on the underlay side as well as on the overlay. So I'm going to quickly go through that. One other thing I would like to quickly talk about is the tenant profile. In CIS, every tenant gets their own dedicated sort of network name space. What that means is that from a secured standpoint, what those tenants can do is they can create their own private network, assign the IP address, bring their own IPs. They can go ahead and route those IPs into their private network. They can use the floating IP to go out to the internet. So basically essentially what we are offering here is every tenant is fully isolated from each other. Now this puts a lot of pressure from a scalability standpoint on the network, both on the underlay as well as on the overlay. So at the end of the day, the tenant consumes the overlay network, the network that you expose to OpenStack. But your underlay foundation is to be very, very strong. This is where we are leveraging the Cisco technology, Cisco application-centric infrastructure fabric to form or give that underlay foundation. So some of the key requirements for us for the underlay, we always talk about the overlay scale. But at the end of the day, the overlay scale would matter really much if your underlay doesn't scale also. So from a requirement standpoint, the list of things that I have highlighted here, automation is one of the key things for us. Automation, because when we started our cloud journey, we started with the standalone devices. With the standalone, it takes literally days to really to bring up the fabric. But we need to do much better than that. So we need a lot of automation really to bring up those part of the fabric. We need to be able to linearly scale or horizontally scale. The idea behind there is that in day one, we'll not be having two or hundreds of racks deployed per fabric. What we would like to do is we would like to start small and we would like to scale out down the road. And we need to have that kind of flexibility. The network needs to be very highly available. Availability needs to be there in every layer. At the controller layer, at the top of the rack switch, at the spine layer, at every layer, we need to make sure that we build a very highly available network. It's just to be modular also. Modular because we want to use the same fabric and we would like to use that fabric to bring the different kind of walk loads. So that's the reason the modularity aspect is also quite important. Security compliance and ease of data operation is one of the critical requirement for us. And application-centric infrastructure meets all of this requirement. So let me quickly show you how we are building this shared fabric. So here, as you can see, I mentioned about the automation. Automation is one of the critical things that I mentioned. So those of you who build the data center fabric, you know how many hours you have to spend just to stand up the fabric itself. With the ACI, what it does for us is as soon as you connect your leaf and spine switches and as long as you connect your controller, which is the epic controller, the topology gets discovered automatically for you. Not only that, not only the topology gets discovered, it will also make sure that all the physical wearing and image and everything is in the right order. If it finds out that one of the top of the switch is not connected properly or appropriately, then it will tell you right then, right there. So that's quite critical for us, right? So essentially, we can bring our fabric stand-up time from days into minutes, basically, into hours or into minutes. So and from a scalability standpoint, the fabric doesn't have to be stand up with all of the racks for the entire fabric, right? If you want to grow up to, let's say, 70 or 80 racks, you can start with, let's say, 3 to 5 racks. And then you can horizontal scale out. You can just keep adding more and more racks into the fabric. The fabric should be able to accommodate that for you. If you want to scale your spine switches, or if you want to have more poor density, all you have to do is just add more line cuts into your spine switches. And then you should be able to scale that way. If you need more control plane scale, you can add more epic controllers. If you need more hypervisors, you should be able to just add that into the fabric. One other key benefit that I would like to highlight with the fabric is that, as I mentioned, it's highly modular. So if you want to bring different kind of workload, let's say, in this picture, I've shown three different boxes in the control plane side, right? So if you want to bring, let's say, ML to plug-in in one of the domain. And if you want to bring the GBP group-based policy plug-in, or if you want to bring, let's say, Kubernetes, and if you want to offer, let's say, Hadoop as a service, you should be able to slice that fabric into multiple segments or multiple domains. And then as soon as you add the different hypervisors into the fabric, the fabric should be able to automatically place those hypervisors into the appropriate domain. This is very powerful from an automation standpoint. You don't have to manually do a lot of those configurations, which is typically error-prone. So it gives you a lot of automation. As far as the HA goes, as I mentioned, the HA is built into the entire fabric. If you look at on the spine, even if you lose one of the spine switches there, you still have 70% to 75% capacity there. You can lose two of the spine, no issues there. You can upgrade your spine without any problem as well. We have the redundancy building of the TOR into the leaf switches. So two of the leaf switches are functioned as a BPC pair. So from each and every UCS servers that we are connecting into those TORs, if one of the TOR dies, no issues. You still have connectivity. And then from the leaf to the spine, we have four links that is connected from the TOR to the spine, basically, or the leaf to the spine. So if you lose one of the link there, there are four links, if you lose one of the link, then reconvergence time is around 100 microseconds. It's not even 100 milliseconds, it's 100 microseconds. So redundancy is built into each and every layer of this fabric. So once you have this very strong foundation, then what you can do is this is just a simple, very simple example. You can install. So there's a lot of concerns in terms of how you're going to scale the neutron. So if your fabric can scale up to, let's say, thousands of tenants or tens of thousands of tenant network, then what essentially you can do is you can slice your fabric and you can allocate the resources for each and every controller instance. Let's say in this example, we have three open stack controller sharing the same fabric. And then any hypervisor that you associate with this controller, you should be able to create this logical data center, if you will. So within the ACI fabric, we call that VMM domain. So using the construct called VMM domain, you can automatically figure out which hypervisor goes to which controller. And this is how you can achieve the scale of your neutron as well. And that's exactly what we're planning to do. So now that I talked about the underlay, I mean underlay scale from an operational standpoint, also ACI fabric gives you quite a lot of simplicity and ease of use. And it brings a lot of automation. But at the end of the day, overlay is what the tenant really consumes. It is this neutron router or the segment or ports on the open stack side is what the tenant typically consumes. So overlay to provide the right kind of isolation is one of the critical aspect for any cloud. And some people take pure overlay approach. Some people think that the problem can be solved with the underlay network. What we are saying is that why not integrate overlay? Take the best of both wall basically. Underlay gives you a certain scale, performance capability. And with the overlay technology also, if you can merge both of them together, you can get a very robust and performance system that will eventually serve your tenants. So what is that integrated overlay? Before I talk about that integrated overlay, here are the list of things that needs to be offered by the overlay network. And for us, as I mentioned, for each and every tenant, we have a separate namespace for the security reason, as well as to offer the premium services and the flexibility. All of these features that I have listed there needs to be offered at scale with the right kind of performance. And then you also need to make sure that your data operation is very simple. Because you can build a massive scale cloud. But at the end of the day, if you cannot operate it very efficiently, then it's not going to cut it for you. So what is that integrated overlay? Let us quickly go through this. So here, as you can see, we are, first of all, we are moving away from the VLAN base network into VXLAN base network. It's a two separate domain here. So our fabric, the ACA fabric, is based on VXLAN. So it's a two different domain that we are talking about. And then in between your top of the switch into the hyperbaser, it is also we are extending the VXLAN. So this is how we are achieving the scale of the network or the network segment. Now we are also going to be using Oplex proxy and the agent. So the idea there is the Oplex proxy and agent. What it does for you is it gives you the scale from a configuration standpoint. No longer you have to maintain all the D by specific configuration details on the neutral side or on your network controller side. What essentially happens is you define your intent. Let's say on your open stack side, and then that intent gets passed down into the epic controller. And then epic controller tells the intent to the top of the switch. And the top of the switch instructs the agent that is sitting on the hyperbaser to do the actual configuration. So by doing that, you are delegating your configuration responsibility just like the way we are doing that on the data plane networking. We are distributing our functions into different devices. We are also distributing this configuration function into different devices. So this would be the reason why we can scale up to such a large number of the tenant. Because if you don't do that, all of those configuration details needs to be there either in the neutron or it needs to be there on the network controller. And we are taking this distributed kind of approach here. So this gives you a lot of benefits. Let me just quickly walk you through to some of them. So here is a quick snapshot. So one other benefit that I didn't mention in the previous slide. So think of it, right? I mean, if one of your attendants is complaining that my application is not performing very well, right? Or one of the VMs is down, right? Where to start? Is it an underlay problem? Is it an overlay problem? You want to go to OpenStack first. You want to go to your network site first. Where exactly do you start, right? Wouldn't it be nice to bring all of this information together? This is exactly what the Oplex does for you. It brings the virtual and the physical networking stats under the same pane of glass so that at any given time you would know what is the status of my VNIC, what is the status of my actual physical network interface. So as I mentioned, I mean, you can achieve all the scale and everything, but at the end of the day, because of all the NFB use cases or that requires a lot of network performance or the optimization, we need to make sure that our network is performing very, very well. So there are four different areas where we are looking at to optimize the performance. One is, of course, the L2, L3 packet forwarding behavior. We want to make sure that we have the optimal behavior of L2, L3. ARP suppression is another one. ARP is a good thing for the networking stuff, right? There is a good reason why ARP exists. But at the same time, all the L2-related security vulnerabilities that we face today is due to ARP as well, right? So we want to make sure that we suppress those ARPs at the right place. And then we are also looking at the VXLan offload to be done on the NIC and the smart scheduler. Those are the four different areas where we are looking to optimize. So as I mentioned, we want to leverage both the hardware as well as the software to make sure that we have the optimal packet forwarding behavior. So here is an example of what we are doing here. We are bringing the full switching support all the way down to the OBS. So if the packet needs to be handled within the same host, if you have two different VMs sitting on the same host, there is no reason why the packet needs to go to the top of the X switch. If the VM is sitting on a different host, then it makes sense to punt the packet to the tor. So this is one of the things that we are doing. We are also, this would be the behavior, if you don't suppress your ARP. If the ARP is spread across all the tor, then this is going to be the behavior. We want to go ahead and suppress the ARP right over here on the OBS. So the ARP doesn't really spread across the entire fabric. We are also for our L3 forwarding also. If you have two different VM sitting on two different, let's say VXL and ID or VLAN segment, but those VMs are sitting on the same host, right? You don't have to necessarily punt the packet into some kind of gateway. We like to make sure that we handle it locally. So both L2 as well as the L3 forwarding happens locally on the host if the VM is sitting on the same host, right? If the VM is, let's say, going into different host, then of course, you need to forward the packet to the top of the switch or some top of the switch. So by doing that, we are essentially delegating the responsibility where it makes more sense, right? For the locally switched or routed packet, it's handled on the OBS. For the packet that needs to go outside, the packet goes to the top of the switch. We are also evaluating a couple of NIC card. Since we moved from VLAN to VXLAN, we want to make sure the list of features that I've listed there, these are the features that we are looking to support using a couple of NIC cards that I've listed there. This is going through engineering analysis as we speak. So hopefully in the next summit, we should be able to share some numbers, some of the performance numbers on this. One other performance optimization that we are doing is around the scheduler. Because right now, if you look at the NOVA scheduler, it doesn't take any network constraint into consideration. So we want to make sure that the network constraints, such as your bandwidth, your latency, if one of your service VM is pretty busy, we want to make sure that all those things are taken into consideration before the NOVA plays the different VMs into the network. So here is the solver scheduler. This is, again, something that Cisco is leading and driving. And all of you can use these to more intelligent to replace the workload. So I've talked about the performance optimization. Now let's quickly go through the scale. The number of scaling parameters that we are looking at, I didn't put any concrete number here, simply because this is going through some validation or the engineering analysis right now. Hopefully in the next summit, I should be able to share some concrete numbers. But from a scale standpoint, we are looking for thousands of tenancy scale. We are looking for tens of thousands of tenant network scale. So these are the kind of scale that we are looking for. And then tens of thousands of VM scale. And we need to be able to scale. So when I'm saying scaling those numbers, the list of components that we have to scale also, right? It's not just you have to scale each and every of these components to achieve the kind of scale that we are looking for here. So netpad scale is one of the key element for us. So in order to scale the netpad, we are distributing the netpad into individual host. So it's no longer centralized. So this is how, even for the pad and the net, net is for your floating IPs. So since we are distributing that into individual host, we can achieve millions of pad sessions and thousands of net sessions by taking this approach. So those of you who are familiar with the DVR, DVR is taking the very similar kind of approach. Even though for the pad, pad is centralized in the network node. But the net is fully distributed into individual host. So once we scale our L2, L3, and once we scale our netpad, we also need to make sure that we scale our DHCP and DNS as well. So this is where we are introducing the PNR. We deployed DNS mask, a based approach, but that didn't really scale for us. Neither did it provide us a kind of stability that we were looking for. So PNR is a Cisco product that can scale up to tens of thousands of network or the segments. So we are introducing the PNR and with the relay agent that will help us to achieve the tens of thousands of tenant network scale that we are looking for. So this is how it will be done, basically. So as far as the scaling standpoint, every DHCP relay will be mapped to a PNR server that will give us the HA, as well as it will give us the scale. Let's quickly talk about L4 plus services. So I talked about scaling the L2 segment, scaling the L3, scaling DHCP and DNS. So now we also need to make sure that we can offer L4 plus services as well. This is where we deployed the HA proxy. HA proxy didn't really was not stable enough, and it didn't really scale very well for us. So what we're doing is we are working with our partners because Cisco doesn't have any load balancer, right? So we are working with our partners working with our partners to provide us the LBS solution. But at the end of the day, the plugin piece is going to be a standard plugin. There are a few things that we are looking for from a scaling standpoint. We are looking for the autoscale functionality based on the service VM. And the idea there would be like you will have a neutron sort of like LBS service plugin that will talk to some kind of controller. And the controller is going to manage the different kind of service VM. So that's how we are looking to achieve the scale. From a high availability standpoint, we are looking for active, active kind of high availability. And then we are also going to be offering the SSL offload. For the VPN as a service, it's going to be one arm mode kind of deployment. So if you look at the opens one, you know that your VPN as a service implementation is the reference implementation does it based on your L3. So your neutron router and the VPN as a service are tied together. So that brings some challenges for us from a performance as well as the scale standpoint. So what we are trying to do here is really decouple the L3 gateway function from the VPN as a service function. So we are instantiating the CSR1000V router just to offer the VPN as a service function. And L3 forwarding is going to be done through top of the switch or the open virtual switch. So this is how we are getting the optimal behavior for the L3 forwarding. But for the VPN as a service, we will have dedicated appliances for that. There is a dependency on NCS plus ESC controller. So there will be plug-in that will be upstream on that as well. So let me quickly touch upon on the tenant consumption. So from a scale performance standpoint, I've gone through some of those approaches that we are taking. But let's quickly talk about the tenant consumption at the end of the day. That really matters. That is the most critical thing. In calendar year 15, we are going to be going with the Neutron ML2 base route. What that means is that we're going to be scaling the Neutron router. We'll be scaling the traditional sort of like Altu network, the ports, DHCP DNS, all the cool stuff that I talked about. But in calendar year 16, we are exploring the group-based policy plug-in. Our plan is to move into the group-based policy plug-in in calendar year 16. The idea there is really to offer a very simplified service. It's based on intern driven. So you can express your L base offering in a much more granular format. It will help you to consume the services in a much more simpler way. So let me quickly introduce you the group-based policy. What does it look like? So if you look at this one, you see that if you want to go ahead and bring the web tier or your DB tier application, and if you want to apply some kind of load balance or the firewall, it's going to be very, very easy for you as an application architect to express that. And then all the details or the implementation details are going to be hidden from you. So this is a bunch of other companies are contributing to the group-based policy plug-in. But where the differentiation comes in is when you use the group-based policy plug-in with the ACI. Because the ACI gives you the integrated policy model. So think of ACI as a policy-driven network fabric, and the group-based policy plug-in is its API layer. So all the cool stuff that the ACI has to offer can be expressed in the group-based policy plug-in. So here are the list of things, list of differentiation, list of capabilities that the group-based policy plug-in can offer in ABAB. And we wind off whatever the ML2 can offer today. Intent-driven consumption model, private link orchestration is another one. Some people call it like one VPN. We can do that with the group-based policy plug-in. We can offer the service training. These are quite critical for the NFB use cases, where you want to subject the tenant traffic to go through. Let's say load balancer firewall and things like that. So if you want to offer that service training or service teaching, you should be able to do that with the group-based policy plug-in. It will allow you to offer the flexible QS implementation. This is quite important for the noisy neighbor problem. If you want to rate limit your traffic from one VM to another VM or from one tenant to another tenant, the group-based policy plug-in should be able to help you in that. The critical and the most important thing is the fault in the performance management aspect of it. At the end of the day, as I mentioned, once you bring the underlay and overlay together with the ACF fabric and the group-based policy plug-in, then you can have a lot more visibility into why an application is not performing very well. Or if there is something wrong with one of the connection, you should be able to just go to a single pane of glass to narrow down why you have a problem. So what are some of the future directions? All of these things that I've talked to you about are something that we are doing today. But as I mentioned, the group-based policy plug-in is something that we are looking to do in calendar year 16. The reason we are delaying this a little bit is because we want to make sure ML2 plug-in and the group-based policy plug-in can be offered from the same OpenStack instance. We also want to make sure that we offer the NFP services at scale with the group-based policy plug-in. And as far as the IPv6 goes, we are starting to deploy IPv6 this year. But next year, we're going to be looking at a larger scale. And we'll be also offering the SLB66 services. So in summary, ACI with the group-based policy plug-in and ML2 plug-in will simplify our tenant conception model quite a bit. We want to build a highly-performance system that scales. We also want to make sure that our data operation is very seamless. So with that, I'm open for Q&A. Thank you.