 Okay, we have a few more people coming in. Okay, good afternoon everyone. Thanks, good afternoon, thanks. Thanks for coming here just after the lunch. I'll try to keep it a little entertaining, try to keep you guys awake. So I'm Praveen Yalagandhula from Aave Networks. So the talks, title, good, bad, and ugly of the OpenStack APIs, this is our experience as we build our network service on top of OpenStack. So as we built our application, it's a network service on top of OpenStack. Using the OpenStack APIs, our experiences in building our application. Okay. Okay, so let me start off to say what we are building. What we are building is an enterprise grade scalable network service. So any enterprise grade scalable service, you have a bunch of requirements that we should be able to elastically scale out, scale in the resources. We should have high availability that the service should work, even if some hosts go down and things go bad in the network. And we are talking about a network service. So we need to have tenant isolation and that will allow provisioning, self provisioning so that users can go allocate whatever network service that they want and do whatever they want to and it shouldn't affect any other tenants. And we are talking about a high performance network service. So I'll try to spend some time on what application that we are building to give you an idea of what OpenStack APIs you would need to build such a network service and the goals that we have. So let me tell you one first goal about what the application that we are building. The first goal that we had was to build it on top of OpenStack instead of in OpenStack or alongside OpenStack. So as I mentioned, however, the application that we are looking at is a network service. So you may think that, oh, it should go with Neutron or something run inside OpenStack. But our goal is to build it on top of OpenStack, something like build it and use only the OpenStack APIs and run in some sense like user space. So it's a layered design so that you have it underneath the OpenStack controllers providing the basic framework for allocating resources and like NOAA with VM and the CPU memory resources and Neutron for allocating the network resources. And the application should simply work by just using the API calls. That's our goal. And what I mean by in or along OpenStack is like if you build some service inside OpenStack, you'll be basically saying, say, put it as part of the Neutron or run some service on one of the network nodes or on your controller nodes and maybe you're plugged into the message queue in the OpenStack core and getting information from that kind of a service there. And along OpenStack is where you build something outside OpenStack but plugs into the same underlay, say if you are using VXLAN or VLAN, basically that can understand that and provide the network service outside in some sense like along the OpenStack and providing that service outside OpenStack. So why do you want to build it on top of OpenStack? So it's a nice layer design. So the main thing is that you will have then very flexible deployment models. I'll give some examples later. But it also makes it the management of the service becomes easy. And if you think about the way like the network stacks have built like you have L2 layer, L3 layer, L4 layer and so on and having these nice kind of layers allows you to do innovation at the higher layers easily if you have the bottom APIs defined very well. So this is at a very high level abstract what I'm talking about. So let me put a little bit more concreteness to say what's the exact application that we are talking about. So to give you a sense of what does it entail to build such an application on top of OpenStack. So the application that we are targeting to build is load balancer as a service. It's an enterprise grade load balancer as a service. So many of you may know what load balancer is but I'll quickly go through what load balancer is. So if you have a web server you're running and if you have just one web server connected to internet and if your users are coming through it and if something happens then obviously they will lose the connectivity. So you want to have multiple web servers to basically get the high availability and typically people put what's called a load balancer in front of it. And then so if you have traffic coming in the load balancer will then distribute the topic across web servers. And if you have one of the observers go down the load balancer basically detects it it continuously monitors and then doesn't forward the request to that web server. So this is just to give you an example of what network service that we are building and in a typical deployment you have these multiple layers of the tiers in your application and you put load balancers in front of everything. So in the enterprise applications you typically have lots of these load balancer middle boxes sitting in between different tiers of applications. And in this case the traffic coming from the outside is referred to as not so traffic and going across your applications within data centers east-west. So the goals, main goal for the load balancer is that it should be highly available and should have high performance. But as is shown it's at the basic level you can think of load balancers just as the requests are coming in it's distributing it out to the backend members. So you may ask, okay at this point it just looks like very fast packet sprayer so why can't we do it in the network router or some network service already there in the neutron or inside open stack. The main reason is that over time the load balancer there is more expected out of load balancer in typical enterprise applications. So there are lots of advanced features that you have in load balancers what's called the term has changed to be called application delivery controller or ADC that's one of the terms I will use. But few things that you have are like the session persistence where users should be pinned to the same server. So something like user one comes all his request or her request should go to the same server. So that means these load balancers are not stateless anymore they need to be stateful. So you need to keep some information around for every session that these load balancers are maintaining. And the second one is the intelligent load balancing so that's again you are not just blindly spraying packets to the backend members. You are to look at what's the HTTP request that user is trying to do and based on that you may have to do some other work. So for example some cases application may say like any user that's currently browsing my inventory send them to some of these set of regular VMs. But once they add something to the cart then bump their session up to premium and so that they go to like premium servers. So you have PCI compliance when they have the credit card entering credit card and so on. So those kind of like rule based context switching. What's the rule based forwarding the request to backend members based on rules. And that basically adds more work to these load balancer services. So this request is P termination and looking into the things and so on. The other thing that people expect out of it is SSL termination. I mean this is another very CPU intensive. So all this is to show that the network services are not a simple one that we are talking about. It's basically CPU intensive, memory intensive and can require a lot of resources. And in this case SSL termination is like you are moving the SSL handshakes and SSL encryption decryption into the load balancer and the backend members they just simply receive the regular traffic. And that's again good in many cases so that you have better performance in the load balancer and also you can easily apply enterprise level security policies. Okay, so I have given you an idea that this is the load balancer as a services, the application that we are trying to build. Now let me tell you what's the architecture that we are coming in. Like as I said, it's our goal is to build it on top of OpenStack. So we didn't want to use, we don't run anything inside OpenStack. So this in the left hand side is how a legacy way of doing load balancers like you have load balancers sitting outside OpenStack and you have to manage. In contrast, what we do which I show on the right hand side is like a service VM based architecture. So the work, the network service work that we wanted to do it in the service VM and leverage the Nova Neutron to allocate the resources. So that's at a basic level that's the idea. So let me show you a high level architecture picture. So we have, it's like a software defined networking architecture. So you have central controller, which is the, like in this case, RV controller. It's a logically centralized multiple VMs together implementing it. It's the one that is managing the configuration. It's the one that is the brain for orchestrating the system. And at the bottom we have service engines. Those are the data plane engines. Like in SDN you have simple switches and an SDN controller managing those switches. It's something similar. So the service engines are the one that are doing the real network service work, that SSL termination and all those things. And the good, cool thing about this architecture is the RV controller essentially talks to Nova Neutron to create these service engines on demand. As you get more load, you can create more and once the load goes away, you can destroy it. So that's the power of building on top of the infrastructure of the service APIs, right? So that you can dynamically allocate resources and you can dynamically reduce the resource usage when there isn't much use. OK, so let me go through what using the OpenStack APIs and building it on OpenStack gives us. So let me go through how it looks at a high level what we are trying to achieve in terms of deploying the solution. So we wanted to run our controllers also as virtual machines so that everything is run as virtual machines. So we have controller running as virtual machine and we have created one management network. And basically, the controller is set up to be able to talk to Nova Neutron so that, again, it can create these service engine VMs those data plane engines as needed. So let's suppose a user has two backend servers to which they want to load balance. So when they set up a load balancer, they create, basically, call some API to create a load balancer and say that load balance to these two backend servers. The AV controller can then talk to Nova Neutron, essential in this case, Nova, to instantiate those service engines. We will instantiate multiple so that you have high availability and it can instantiate more or less based on what the load and so on. And then the other thing that it will do is it will basically plug them into the right networks so that when there is traffic coming, the service engine can receive the traffic and load balance it to the backend members. So that gives an idea of how the service VM-based architecture would work on top of OpenStack. Now, in this case, if you have, say, for high availability, if one of the service engines goes down, then we need to be able to quickly figure out that, oh, that has gone down and be able to move the traffic to the next service engine. So again, I'm showing these to give you an idea of what kind of APS that we would need as we go along to be able to do these kind of things. So I'll summarize it at the end to go through each of these. So the other thing that we can do is that we create different service engines for different tenants so you can get tenant isolation and OpenStack is great for that. So yeah, basically, we can then isolate the traffic for each tenant. Then we can let users do whatever they want to with their load balancers, do run scripts, or do very high CPU intensive network service processing, and all those kind of things are possible with this architecture. OK, so let me tell you the other thing about this building on top of OpenStack is now you get flexible deployments. So you can have, as I've shown, I didn't show you where these VMs are instantiated and so on. And we can have different kinds of deployment models. So you can have the avicontroller VM running in a different tenant, service tenant, and the service engines could be created in the tenant's context. We just need to plug them into the right networks. We create them in the right tenant context so the resources can be counted against the tenant. Or you can do them all in a provider context so that the tenants don't see the service VMs at all. So in these cases, I'm assuming that the avicontroller has, in some sense, some kind of admin credentials to the NOAA neutron to be able to do all this magic. But again, even though it has admin credentials, remember that there is nothing running inside the OpenStack. It's all on top of OpenStack. So you can do even more things like share the ICs across tenants in case you don't want to have too many service engines. You can do this kind of thing too. OK, so that gives and hopefully gives a good flavor of what's network service that we are trying to build. And then I can motivate the APIs that we are trying to use and what's the reason for expecting certain APIs. So as I said, one of the things is elasticity. That is we should be able to create and delete VMs as the traffic increases and decreases. And we want to attach and move the VMs, move the interfaces to different networks as the traffic over time depending on the requirements. And the second thing is high availability, that we should be able to detect VMs or network disconnections as soon as they happen. And we should be able to switch the IPA like the networks from one network to another and so on. The other thing is multi-tenancy. That is basically we should be able to create different VMs in different tenants and be able to provide that isolation. And the last thing is high performance and efficiency. Obviously, we are building a network service, so we want to have very high performance. And we should be able to support multiple IPs on a single interface. OK, so let me switch to the OpenStack APIs and talk about what worked and what didn't work for us as we done this work. OK, so the great thing about OpenStack APIs is that as we have seen and we probably all have used that it's very easy to create VMs and plug them into the right networks. So basic VM creation and connecting it to the networks is pretty solid APIs. And this also allows us to scale out and scale in the VMs because you can basically create more VMs when there is resource requirements and delete them. The placement options, as you've seen, like we want to be able to place these network service VMs, we want to place closer to the applications as possible because you don't want to have too many hops in the network as the traffic comes. So one of the things that we wanted from Nova APIs when booting up is the flexibility to deploy it in the right place. Now, if you have admin access to the Nova APIs, then you can actually see where each of these application VMs are which host they are on and where exactly they are all. But if you don't have the admin access, if the API controller doesn't have the admin access, all it can see is that, OK, those VMs are in so and so availability zone. That's it. So you have a little bit of limited support to be able to really place the VMs that we are creating. So there is the anti-affinity server group API that's helpful, but it doesn't still guarantee that the VM that I'm trying to place will go on to a specific host. It just says that if you create two VMs in one single anti-affinity server group, then the API just guarantees the API that says that it will be placed on two different hosts in the same availability zone. But it doesn't guarantee that it will go on to any specific host or any of those guarantees. So overall, Nova VM creation perspective, it's pretty good. Like if you have admin privileges, you have a lot more insight, and you can do a lot more things. On the side of multi-tenancy, like because of the Keystone support for all these tenants, users, and the Nova Neutron being built on top of those abstractions, we have very good, very solid support for multi-tenancy in OpenStack. So that side, it's really cool compared to other infrastructure-as-a-service products like this OpenStack. The multi-tenancy concept is very well-baked in. The Neutron's CRUD API at the basic level is, again, very good. So we have Neutron network supports, routers, and subnets, and makes it easy. You can create Neutron ports in any network that you want, connect that to your VMs, and it makes it easy to create these multiple tiers of VMs and have proper isolation between these tenants and so on. So the basic Neutron APIs are pretty solid. So let me start to then go on to what didn't work. Again, remember that I'm trying to mainly focus on what didn't work to tell you what are the places where the APIs can be better. But I don't mean to say bash on a lot of APIs. As you can see, it's even possible to build that network service application on OpenStack APIs because it's really like we have really good set of APIs. Now let me go through what kind of things didn't work for us as we built our network service application. So one of the things we found very lacking is that no notification APIs from core OpenStack services. So whenever we do, say, create the Nova VM and if the VM dies or something happens to it, there is no way to get notified about that VM that it has disappeared. Similarly, for Neutron, there is no way to know if the port got deleted or something bad happened. We won't know about it until we do period. We have to check it explicitly. Similarly, Keystone also, we don't have. Essentially, the thing is that we don't have any notification services that these applications can subscribe to, to these individual services to get information on when the resource that they have created has something has happened to it. Now the only option that we have is to periodically check. That's not so great because if you are built a really scalable system and there are like 1,000 VMs that our controller creates, then we have to basically do lots of API calls to figure out if a particular service engine that I created is still exist or not. So that's not good. And Silometer alerts is a possibility, but we have seen that it's not so popular and when we started working with many of our customers, we have found that many installations are not really using it. And again, that will give alerts from a different service and what we are looking for is basically for every service to tell about the objects that we have created in that service. Now, periodically checking is a good thing, is a possibility, so that's our workaround, but the thing with it is, as I said, it's not scalable. So we have to do it over time and that doesn't result in a very fast detection of failures. So we do have to do lots of heartbeats between a distributed heartbeat mechanism to figure out when something goes down and then quickly act on it. So it is, in some sense, possible to avert it, but it's not something that is nice, that's easy to do. Like, we had to build more infrastructure to get around that. Okay, so the next thing that I'm going to talk about is the NOVA interfaces and neutron ports. So this is one of the things that happened because as you guys might have seen today morning, that when Kyle was talking at the Keystone, neutron or the quantum evolved out of NOVA, like initially NOVA and networking was part of NOVA and then slowly it got moved out into quantum and neutron, right? But as they tried to move it out, I think that there are some of the concerns that haven't been properly separated out. Like, so if you see in the physical world, if you have two switches and a physical server and you have ports and the physical server you have typically nicks that you take a wire and connect those nicks to the ports to get connectivity. But in case of neutron, like you have NOVA VMs and you have neutron networks which are equivalent to switches, but in the API viewpoint, interfaces of these VMs like the virtual nicks are not exposed out in the API calls. So all NOVA has at that point is just these IP addresses and there is no concept of having two virtual nicks on the NOVA so that you can talk about, okay, this IP address on this vNIC and this has this IP address on this vNIC. So on the neutron side, it's good that you have these two different ports that get created when you're connecting those VMs. So neutron ports, from the implementation perspective, neutron ports essentially becomes the virtual nicks on the NOVA VM and there is no easy way to, like because of that, there is no easy way to move. In a physical world, I can connect this, take the wire from nicks zero and connect it to another port on another switch so that if I want to move it from one VLAN to another VLAN or one network to another network, I can do that. But in case of open stack world, you can't do that. The only way you can do that is you essentially destroy that interface and create a new interface and basically create a new interface in a different network. Now, think about it, in the case of physical servers, like you can do one API call to say like movies from this wire from this network to another network. Whereas in case of open stack, you will have to essentially do multiple operations and those are pretty heavy weight too. Like basically when you're taking out the interface, the Liberty gets called and you're doing all this magic to take out the VNIC in the Liberty for that VM in a KVM implementation. Whereas if you had that nice separation of abstractions, we could have easily just changed the labels on the existing VNICs that are already there on the neutral, right? So this basically was one of the issues that we had run into and we had to again get around to it by doing different things. The other issue that we have seen is that with the security group APIs. So the semantics of the security group APIs depends on who implements it and the open stack, both NOVA and Neutron implements the security groups. So from the API perspective, if I'm going and deploying my application on some customer's open stack deployment, I need to know beforehand which they're using NOVA security groups or the Neutron security groups. And that's because the way the security groups work is different. Like if you use the NOVA, they are like security group is applied across all VMs. I'm assuming that all of you know about security groups. Just essentially think of it as firewall rules, right? And in case of Neutron, they are applied per port. So any, you can say for this, on this Neutron port allow so and so packets or basically drops so and so packets. That's how you basically when security groups are associated with Neutron ports in case of Neutron. And as I was saying, we need to know beforehand where who's implementing it because otherwise when we are using the security group APIs, the behavior could be different. And unfortunately, the information about who's implementing the security groups is buried in the NOVA.com. So there is again, from an application that's just getting deployed on top of open stack, there is no way for it to know who's implementing the security groups, really. And as I mentioned, there is inconsistency in the behavior also as how finally the security groups get applied. This is a bit of like issues with the reference implementation, but it's also coming because of the semantics not being defined well and not set in stone. So assume that you have Neutron that's implementing the security groups. So you have Neutron net one and Neutron net two. And as a user, I come and say, okay, create a VM with one interface in network two using the security group I created, my security group. So we get the NOVA VM. And since Neutron is implementing it, implementing the security groups, so what happens is that the my security group that security groups gets associated with that Neutron port, all's good. Now, if I do a NOVA API call to say, attach an interface again to my existing NOVA VM. Now, it's nice and good that there is an interface that gets created in Neutron and gets plugged into NOVA. NOVA knows that there is a new IP address for that VM. And as a user, I would expect because I have done saying that my security group is done through NOVA APIs, I would expect that the security group of that new Neutron port would also be that security group, but no. If you do that today with your reference implementation, you most likely, what you will see is that the default security groups gets associated. And this mess happens because I mean, it's not, there is, because of the contention between what the semantics of security groups for NOVA and Neutron, these corner cases fall into, like nobody really goes and checks on those corner cases. So if you have these, you better, basically the expectation is that if the Neutron is implementing security groups, anyone developing anything should be using the Neutron security groups. Otherwise, you will basically run into these kind of issues. So as I said, it's a bit of reference implementation issue, but it's also because there are two different services supporting the same set of APIs, we have this possibility of incoherency in the semantics of the APIs. And that's leading to my next point is that overall, we don't have none of our API documentation anywhere that we don't define the semantics of API to say, when you call this API, this is what will happen, exactly. So I can tell you one example, like the lot of Neutron APIs, because the way that it's not, these semantics are not set in stone, plugins can implement whatever way they wish. So in a physical world, if you have a switch and if I get two servers and plug them into the switch, I'm guaranteed to be able to talk to, between these two VMs. I can use whatever IP addresses, as long I can run Zerocon for whatever, and they should be able to communicate using any MAC address. I mean, you have the guarantees that the switch is essentially a L2 connectivity, right? But if you do the same with OpenStack, where you have, we said that these are similar abstractions where switch is a Neutron NAND. But if you create two VMs and connect them together, guaranteed that they won't be able to talk to each other. And that's, again, because most plugins, they won't allow you to be able to have these VMs be able to talk to each other, until you have some IP address assigned or you have to do certain other things. And the thing about it is that, not only that they don't allow, there is too much restriction on network connectivity. The other thing is that there is no APS, I can expose to say that, okay, don't do source IP spoofing check or don't do this and so on. So for example, as I was mentioning, like source IP addressing spoofing is not allowed, even in a local network, in Neutron. Like all implementations will do that. However, it's one of the essential primitives if you're building a highly available service. Like if I have, as I mentioned in my service, I have highly available two service engines trying to implement the, like receive traffic for the same IP address so that they can basically quickly move the traffic from one service engine to another service engine. To be able to do that, they'll be able to like detect that there is something going on and be able to quickly move it. Now, those kind of things, if you want to do that, then you need to be able to have knobs to say that, okay, these service engines are special. I want to be able to almost run as if they are connected to a real switch and real L2 network. Now, there are some Neutron extension APIs, like allowed address pairs and port security APIs that we have defined over time to allow some of these kind of roundabouts to get around these defaults that are baked into the API semantics. However, they're not part of core APIs. That means that it's not guaranteed when we walk into any OpenStack deployment. It's not guaranteed that they will be there. Depending on plugins, some plugins may implement it, some plugins don't implement it. If you use some of the advanced features in Neutron, then suddenly there is no guarantee that that advanced feature in Neutron would still work with the allowed address pair extension because these extensions are essentially created out of necessity. Oh yeah, this reason came up, so we'll define this extension API and somebody implemented it and doing that. So the main point here is that if we start making sure that we have proper semantics on the APIs and make sure that we have proper layering of the APIs so that when you have the L2 APIs that say what real L2 network would work and L3 APIs to reflect how a real L3 network would work, then it becomes easy for the applications to really utilize those APIs and expect the behavior as you would expect in a real world. So I'll jump to one more thing which is to talk about network performance. Again, this is not really the APIs but it's more reference implementation restriction. But I thought it will be good to again present what kind of things that we have run into as we developed the high performance network service. So one, as you guys might have seen if you're using ML2 plus with OVS VLN, you'll have this kind of multiple layers in the hypervisor before a packet reaches. When packet leaves the nick and before it can reach the VM, it has to go through a bunch of things. And there are multiple layers and depending on the plugin, this implementation will be different. But we have seen that when we initially started one and a half year ago, the performance was pretty bad on the KVM based and the reference implementation versus say ESX or some other hypervisors. And many of these things got fixed over time. Like VEAT pair was a bottleneck in who went to 12, 4, 12, 10 time frame but that got fixed in the later versions of the kernel. And then the older OVS performance wasn't that great. There are some issues and the newer versions fixed it. And then there is a net filter connection, basically net filter contract issues where every TCP connection is maintained to again to do these come some kind of these to implement these security group rules. And because of the limited number of entries and so on, there is the limit on the number of packets per second that can pass through the contract NF contract that's associated with the Linux bridge. So overall, most of these are all fixable issues. So in the sense that they're all getting good over time and I think they will continue to improve. But the previous slide as I mentioned, there are no APIs to say for my service engine, I want to be able to say that it's a special VM and I have admin privileges and I want to disable it. The only way I can do it is today to go SSH to the compute node, change the driver, the compute agent to say that don't put the Linux, this extra take out some of the layers. I mean, the new extension APIs like the port security APIs that are getting there to be able to do those things. But as I said, like again, these are extension APIs. So we are not guaranteed that those will work in every plugin. So that's all my rant about what worked and what didn't work. So summarize enterprise grade scalable services, network services, we are trying to build it on the virtualized platform. So we want it to be elastic, highly available, multi-tenancy, self-provisioning, and high performance. These are all our requirements. And OpenStack APIs are really good for if you're doing like the first implementation of our system. In one month, we were able to get it up and running. But it took one and a half years to really make it running in customers' deployments. And that's mainly because support for these advanced APIs that you want to build, like if you are trying to build real enterprise grade applications, they are limited. So we had to get around those to realize this. So that's the end of my talk. And I'm open to questions now. Thank you.