 No takers on the doors. That's fine. All right. Thank you all very much for coming. I appreciate it. So what we're going to talk about today is the OVN project and kind of give an update on what we've been working on. My name is Russell Bryant. Hi, I'm Kyle Mastry. Justin Pettit. And I'm Ben Faff. So let's just get started here. So let's start out with a little bit on what OVN is actually aimed at. So the idea here is that OVN implements network virtualization. So what do I mean by that? Well, I mean that we're separating the structure of the physical network from the virtual network. If you have a legacy physical network in one of your environments and you want to move that into a cloud, then as you can see on the left, you probably have some sort of a network topology. It might involve switches. It might have routers in there. And then when you move your physical machines into a cloud, the VMs and the containers, they're not necessarily placed in a physical topology that has anything to do with the physical topology you had before. So if you want to retain the properties of the network, you have to introduce that in a virtual manner because you can't constrain the physical network in the same way that you did the virtual or the physical network in the same way that you did when everything was physical. So OVN provides abstractions that allow you to recapitulate your physical network in your virtual environment with most of the same features and allows you to do it in a self-service manner. So why did we start OVN? Well, so we'd looked around at a lot of the networking projects and we felt like there was a lot of opportunity for improvement. So Justin and I here have been involved in OpenV switch since the very beginning and we have a lot of experience with it therefore. And we felt like there was a lot of opportunity to implement things in a way that would be simpler, more efficient and easier to understand. In addition, we felt like it was a natural evolution of OpenV switch. OpenV switch up to this point had only implemented some pretty low-level abstractions. You could build other things on top of it, but we hadn't actually done that. And over the years we'd had a lot of people ask us, you know, where's the rest of it? Where's this distributed switch? And we felt like it was time to start building that. The other thing is that if you look at, say, the existing OpenV switch integration in Neutron, it was pretty tied to Neutron and we felt like it made sense to have something that was a little more separable so that you could apply it to other cloud management systems as well. Right, so I guess the next question is, now that we know the motivations around OVN, is why should OpenStack care about OVN, right? And I've been saying this for a while. I think that Neutron is more of a platform. So its job should be to provide cloud networking, but it should also be an API and a database server. I think that's ultimately what it should be and it should be a platform to enable a lot of other things as well. And Neutron already uses OpenV switch, obviously, the default implementation, the L2 agent-based approach already uses OpenV switch. There's actually a lot of other both open source and proprietary things that use OpenV switch as well. So OpenV switch is, it's not a part of Neutron, but it's used by a lot of different plugins and drivers for that as well. So, you know, OVN is definitely in scope as well to increase OVS implementation there. And as well, and I think the other critical thing to think about here is, should OVN succeed in its mission, it's actually going to simplify some things within Neutron I think as well, because it kind of removes the work that's being done by like the L2 agent right now, that kind of moves into OVN as well. And so it simplifies what Neutron is and the scope for Neutron there as well. All right, so now that we've kind of got a little bit of background of like why we're doing this, why OpenStack should care, you know, what is it? So this slide is sort of a bit of a feature list in a sense, so it provides logical switches. That's your Neutron networks. It has this concept of layer 2 through 4 ACLs, which we use to implement security groups for OpenStack. It implements distributed logical routers. It supports multiple tunnel overlay protocols and also supports top of rack based logical to physical gateways. So physical switches that support being a VTEP endpoint as a gateway into physical networks. We have support for that. OVN is a part of the OVS project, so it ends up working everywhere that OVS works today. So most of our development has been on Linux using KVM and Zen. It also works with containers. There's people working on integrating this stuff with containers with Docker. The use of DPDK is the data path for OVS is transparent to OVN, so it works with that. It should work with Hyper-V. We actually haven't really done any dev or testing against it, but I don't think there won't be an enormous amount of work to make this work with Hyper-V as well. So at least we'll work there eventually. In terms of integration, so where OVN is used, we've been developing integration with Neutron basically in parallel with the development of OVN itself in the last six to nine months. But it can also be integrated elsewhere, other cloud management systems. There's people looking into integration with things like Kubernetes, for example, as well. So that's also being worked on. So some of the particulars about the project itself. So it's being developed by the same community as OpenVSwitch. So that means that it's all the developments happening out in the open on the OVS Dev mailing list. The project is vendor neutral. As you can tell, since we have IBM, Red Hat, and VMware here, there's also a number of contributors from other companies as well. And as I mentioned, all the development is occurring in public. You can go to the OVS Dev mailing list and you can see all of the code coming in and being reviewed there. And it's developed under an Apache license, which means that if people can bring the project in and can make modifications to the project without having to share themselves, which sometimes makes it a little easy for some companies to deploy it. And that's nice for OpenStack, because all of OpenStack is distributed under the same license, the same Apache license. So some of the goals of the OVS project. One, it should be production quality. I think OVS has a pretty good reputation generally for the switching component itself for performance and for stability. The design is pretty straightforward. It's fairly easy to reason about. We plan to scale to thousands of hypervisors. And that's just the connection to the hypervisors. That doesn't include if there are multiple containers or VMs running on those hypervisors. And we've really looked at trying to improve the performance and the stability over some of the other alternatives that we've seen out there that provided some of these features. So now we'll talk about how OVN is different. So here's an overview of how the OVN architecture looks. So you see in the middle there, there's a couple of databases that are involved. And the databases are the core to the architecture. OVN doesn't really have a controller in the same sense as a lot of similar systems. Instead, the way it works is that the cloud management system, or in this case, a neutron, pushes the desired configuration of the system down to what's labeled there as the northbound DB. And so that DB is expressed in terms of semantics like, say, logical switches and logical routers. It doesn't say anything about the physical layout of the system. So the component that's closest to being a central controller there, you see that's OVN North D. That takes that northbound database and it translates it into a simpler lower level format in terms of logical flows. And I'll talk a little bit more about logical flows later. And it puts that into the southbound database. Then on each hypervisor, we have a local controller running. So that local controller, it talks to the southbound database to see what the system should look like. And it translates that into specific terms for OpenVSwitch on its own particular hypervisor and pushes that down. So the system is quite distributed in the sense that most of the logic for a particular hypervisor is calculated on that hypervisor. So the question is the difference between the OVS agent and OVN controller. So it's true that in both cases you need some logic on each hypervisor. And yes, that does provide the local logic on the hypervisor. The implementation is completely different and the architecture is completely different. So in OVN, the configuration is, as I said, coordinated through these two databases. The southbound database is in terms of what we call logical flows. So a logical flow looks a lot like an open flow flow or you can even think of it as being similar to an ACL. But in terms of being, but it's in terms of logical entities like logical switches and logical routers instead of physical entities. Like the logical flow tables don't say anything about where things are located physically. So for example, if a VM migrates from one hypervisor to another, the logical flow table doesn't change at all. And it's the local controller that's responsible for taking these logical flows and figuring out how they relate to physical entities. And this architecture is one that we've tested in other systems before and we're confident that it provides value. And so I want to talk a little bit more about what the neutron plugin does. So what exactly is consistent and what consists of the neutron integration versus OVN itself? And so the neutron plugin, it speaks the OVSDB protocol to that northbound database and that's the integration point. So when you create a neutron network, it then talks to OVN's northbound database to create a logical network there. And the goal effectively of what we're getting to is that the only part of neutron you're running is the neutron API server. So you've got the API server running with the OVN plugin and then the rest of it is effectively taken care of by OVN as opposed to the various neutron agents that would be used today. So there's no rabbit MQ in this case except for notifications. So if you're using notifications consumed by Solometer or some custom listener, there may be rabbit in that case. But there's not the use of rabbit amongst a bunch of neutron agents. All of that's replaced by OVN and its database architecture. And okay, so let's, you know, getting into a little bit more details of how OVN works. So we're going to do a little bit of a comparison, maybe a review of what it works today. So with a neutron today to implement security groups, if you're using OVS support, you have to have some additional layers between your VM and the OVS bridge. You have to have, we have to stick a Linux bridge there in the middle and an extra vEath pair so that we have a place to apply IP tables rules. So that's how it works today. And then let's talk about how it works in OVN. Okay, yeah, I can talk about this. So the way it works today is that this has actually been a hotly anticipated feature of OVS for quite a while and it's just recently merged. But OVS now has integration with the kernel connection tracker. And that allows us to do stateful, implement stateful policies in OVS itself. So now instead of having these extra layers, we can hook the VM directly up to the OVS bridge and then do everything that we need for security groups using OVS flows. There was a, I have a blog post there that's the link at the bottom if you want to like try to race me and see if you can write it down by the time we change slides. But it talks about this in more detail and that blog post also has a link to a talk from the previous OpenStack summit. This integration was in progress at that point, the OVS contract integration. And they did a lot of performance testing of like the way that security groups are implemented today versus doing it this way. And the performance impact was significant in terms of like 30% improvement in terms of the CPU consumption to implement security groups. So that's a great talk if you want to dive into the more details of the performance benefits. But also the pipeline is simplified quite a bit, sure. Do we get this without the rest of OVS? You can, yeah. So there's sort of layers here, right? So there's, to implement this, first you need OVS integration with the kernels connection tracker. The existing OVS support, there's actually been prototyping of this already. That can also be updated to implement security groups this way. They'll have to figure out, you know, the right way to build the flows whereas OVN is doing that itself, right? So it can be done. I don't think that's available yet, but it can be. Yeah, and one thing that I think is important to mention is that all of the work that we're doing, we're not trying to give OVN special preference over any other projects in OVS. So all of the integration that we're doing is not tied to OVN. So there's a very generic interface to programming the connection tracking flows. And so Neutron's existing plugin could easily be written. Yeah, and actually, I mean, I know people have already been working on this and I expect that it'll probably get worked on in this cycle. So this will probably just become the way you would do security groups with OVS. I mean, there's no reason you wouldn't do this way once it's more widely available. I mean, the limiting factor is effectively just that it's so brand new. I mean, it requires kernel changes and changes to OVS. Yeah, but certainly get there. But all that's coming along. All of the needed pieces are finally all wrapped up and all merged into their respective places including upstream kernel and the OVS project. So I think the next thing that the team is kind of looking at implementing after security groups has been L3. So the layer 3 stuff and how can we do that with OVN as well? So I think it's worth looking at the existing L3 stuff and how that's implemented with the agent-based approach right now. And so it is agent-based just like the L2 implementation as well. The L3 functionality with the L3 agent is done using the Linux IP stack and IP tables as well. That's for both L3 routing as well as NAT as well. And this does allow, I think the key thing that this allows is for this overlapping IP address support. And that using network namespaces as well. So this is all the technology that's there, right? And so this is from the networking guide, right? This is the actual diagram that explains, and Carl is smiling over there. This is like what it looks like right now with all the different bridges and ports and everything. And I think the goal is to try to simplify this as well with OVN as well. So I think that's what Ben's going to talk about is what that looks like. So the OVN L3 design is considerably simpler from an architectural standpoint. We're able to distribute all of the L3 flows through the logical flow table that I was talking about. So if you have a L3 router, then that translates in the southbound database into what we call a logical data path. And that has flows that implement the routing table, that implement ARP, static and dynamic. And so we already have a lot of this checked in. There are patches in various states of review for other parts like the IPv6 support, for example. And in general, what we expect is that the flow caching that OpenVSwitch does should lead to improved performance here. In the way that you do it without OVN, each routing step requires a hop through a namespace. So you have a cost to get in and out, and you also have to go through the kernel routing table. So you have a fairly, not a huge, but a significant per packet cost with OVN and the way that OVS does caching. Primarily, you calculate all of that once on the first trip through the kernel, and then after that it's cached. And you immediately know that you need to just set the desk back and decrement the TTL and go on your merry way. And just as a final note, this won't require the Neutron L3 agent. We're going to talk about gateways in just a moment here. So the question is about the cache. So I'm talking about the caching that OpenVSwitch does in a general manner and without needing any assistance from the OpenFlow programmer. We wrote a paper about this that was the best paper at NSDI this year. So anybody who has interest in that cache, I'd recommend that you go read our NSDI paper. So now we're going to talk about gateways. So we've been including in the OVS distribution for quite a while this VTEP extension to the OVS-DV. It's a particular schema that we've put together to work with integration. So this is actually what NSX uses to integrate with, which is a VMware's product for network virtualization to integrate with hardware VTEPs. But since it's part of the OVS distribution, we've also been able to leverage that as well. And so that means that with the VTEP schema, you're able to control a number of physical gateways, including ones from Arista, Brocade, Keamless, Dell, HP, Juniper, and Lenovo. And actually, we've did some work with Brocade. And if you're interested in seeing OVN work with a hardware gateway, you can go to the Brocade booth and they're demonstrating it down there. There's also software implementations that we do, and we've done a port to DPDK. So if you want to see sort of a reference implementation of how to write a DPDK application using OVS, that code is available as well. So I'll talk a little bit about the capabilities of the VTEP. But right now they're fairly slim. I mean, they're basically about getting traffic in and out from physical space to logical space and vice versa. But I think we'll probably have to move beyond that going forward because there are going to be requirements around L3. Actually, L3 is in there, but we need to do scale out and failover and additional stateful services. And we need to continue to work with vendors to add support for those features as well. But that's underway. So here's just an explanation of what I was talking about with the physical workload integration. So in blue you've got the hypervisors there, and then they've got the yellow VMs. And then you've got the two pink boxes that are the physical workloads. And so you can now use a VTEP top of rack switch so you can have a 48 port switch and have different VLANs for the different physical workloads. And that will be part of the same logical network. And so we use VXLAN tunnels between the hypervisors and the VTEPs because a lot of ASICs for the top of racks support the VXLAN protocol. And so we use that to get those physical workloads into the logical space. And so you can have much higher fan out than is possible with software switches. Okay, and another topic. I wanted to talk about rolling upgrades for a minute. Because this is a topic that's near and dear to every open stack player's heart. And so I wanted to mention that this is something we've thought about for OBN and confident that this will work fine. And some of the key points about this. So if you think about what does it take to do rolling upgrades in Neutron today if you're using the existing OBS support. There's rules around how we manage the database. There's also rules about how we use rabbit or the message queue, whatever your message bus of choice is. So versioning the interfaces between the components that are used in the message bus, versioning the data. So if you've heard of Oslo versioned objects if you're working on development. So there's a ton of work there, still a ton of work ongoing there to make that work. To do this with OBN, the interface among all the components is the OBS DB protocol. That's a well-defined protocol. There's an RFC. And the real work is in how we manage the schema, the database schema. That's how the data is transferred on all the components. So the schema's versioned and we will be very carefully managing our changes to that schema so that they're backwards compatible. I mean this is something the OBS project has a lot of experience doing. That's how for the existing OpenVSwitch schema for each switch today, they're very careful about making sure that the changes there are backwards compatible. So we will do the same thing. And what that lets us do is support rolling upgrades in a way that is sort of expected for OpenStack. So you update the databases first in your controllers, not work controllers but controller nodes and OpenStack speak. So you upgrade those databases first and then you can roll through the upgrades of each of the hypervisors. In the OVN case, we're OVN controllers running. So I think we have a solid plan for how that's going to work for OVN. It's not really any additional work. Alright, so talk a little bit more about the status of things. So on the neutron side, so we've been developing things in parallel. Naturally the neutron integration trails OVN itself just a little bit. But what we have support is basically all the core services. So we support L2 networks, we support provider networks. So actually just a bit ago we were talking about gateways as a way to bridge your logical networks to your physical networks. We also support provider networks if you're familiar with existing OVS support. So you can have sort of direct connectivity to a physical network available on every hypervisor. And then you can either connect your VMs directly to that or create logical routers to those networks. So anyways, we support that. We support security groups. We've got a couple of things that there's not really well-defined neutron APIs for yet. One of them is special support for containers running in VMs. So a lot of people running container applications in VMs don't want to create another layer of overlay networks for their container applications. And so we have a way to have those containers sort of independently connected to networks managed by neutron or OVN. But there's not really an API to express that in neutron. So we've got a special binding profile you can use to do it. The other thing is the VTEP gateways that Justin was talking about. There's not really a neutron API to do that. So we're using a special binding profile there for connecting an existing neutron network to a VTEP gateway. So it's exposed but not by an official neutron API yet. I mean it's exposed but through an OVN specific API effectively. So if you were to go spin up our DevStack support today, it's going to enable the L3 agent. That's what we've been using historically while OVN was working on its own distributed L3 support. We've got several patches in the works integrating with the OVN's L3. And that's something that as soon as we get back, we're jumping right back on and that'll be integrated very shortly. So we also use the, or we have been using neutrons DHCP agent historically. But there's patches already up for review for native DHCP support. So effectively what you get is no more neutron DHCP agent, no more DNS mask, and distributed DHCP sort of managed by each local controller and using flows and the local controller. So that's up for review now so hopefully we'll have that fairly soon. There's some specific neutron APIs that are being worked on that are of interest for us. There's this thing called VLAN of where VMs. It was not originally targeted for this use case of containers running in VMs. But it turns out to be exactly what we needed. We needed a way to express that. You have this port that's like a VIF on a VM. But then logically there are other things inside of that VM and that's what VLAN of where VMs is expressing. As you're saying that the port on a VM is a trunk port and then there's multiple VLANs inside of that. And that actually provides what we needed. So we're interested in helping getting that done this cycle. A couple other projects. The networking L2 gateway project is aiming to provide an official neutron API for the VTEP gateway stuff we need. And the networking SFC project is also quite interesting because we've been thinking about SFC and OVN and we needed to do that for real. First we need an official neutron API to do it. So those are some of the things we'll be looking at in the integration coming up. So a few more things that we're working on or starting to think about. So first of all we're pretty close to being ready to get NAT support in. Open vSwitch has been integrating support for NAT in sort of the same way it's been integrating support for the connection tracker. This means that we'll be able to do NAT through the logical flow table without having to add any specific calls say to IP tables. And just I should point out that the other OVS agent for neutron should be able to take advantage of that as well as long as the work gets done there. Let's see as Russell mentioned native DHCP support is on its way. Patches have been posted and there's been a start on review there. We've started to think a lot about service function chaining. It seems like it may not actually be that difficult to get it in and there is some work in progress on that. I don't believe any of that's been posted yet but I assume that will happen fairly soon. And finally we're interested in integrating with Kubernetes. It seems that the basic requirement there is that we have some sort of a basic form of load balancer. So we're going to work on that. We have someone working on that and possibly that would be useful for other integrations as well. Okay so we've talked about all kinds of different things but if you want to go off and look closer and find out more this slide has a bunch of references. So if you want to dive into the OVN architecture there's a man page and it's published on the OpenVSwitch website as well in various forums called OVN architecture. And this goes through and quite a bit of good detail about the architecture of the system. And that's really the document to start with if you want to get a real good understanding of how everything is built. And then that document has several references to others that cover details like details of the database schema or details of a particular service that runs as a part of the system. All of this code is in the master branch of the main OVS repo. So it's developed right on site. I mean it's fairly decoupled but it's in the same Git repo. And it's been in master for quite a while now but not in an OVS release yet. And there's all the documentation I mentioned is there on the OpenVSwitch site where it's linked there. The Neutron plugin is in its own repo. I mean the way that Neutron plugins are effectively managed these days is that everyone is sort of separated off in its own repo. So we have one that's called Networking OVN. That's sort of the naming convention of the Neutron plugins these days or the repos for them. And then we have some docs associated with that. Like there's especially one if you're interested in trying this out there's a doc in the Networking OVN docs about testing and tells you how to use our DevStack integration. It's actually quite simple. You grab DevStack, you grab our sample local.conf for DevStack out of our Git repo and you run Stack.sh like it just kind of works out of the box and installs all the right stuff. So that's all the super important references you need to find out more. So if you're interested in the project right now we're developing pretty quickly for it. But there's a lot of opportunity to still contribute and we have a lot of people who have been joining the project. So like the DHCP work is from some new contributors that just jumped in in the project recently. So there's plenty of opportunity and we welcome people to try developing for it. Also if you want to try using it that would be great. Let us know if they're features that are missing or that would be useful for deployment. And in particular we want to know about scaling. And so as we said our goal is to scale to thousands of hypervisors and so we want to have some people who run fairly large clouds to try it out and let us know if there are any issues that they see. So as you mentioned the core OVN is being developed on the OBS Dev mailing list. There's a link there to the archives and you can subscribe if you're interested. And then we also have discussions on the Pound OpenV switch IRC channel and we have a weekly meeting on Thursdays at 10.15 a.m. Pacific where we discuss the current status of OVN. Yeah so in the Neutron plugin like I mentioned it's in its own Git repo all the development the development process for the Neutron plugin is the same as the rest of OpenStack. So it uses Garrett just like everything else in OpenStack so the process is the same. We use the OpenStack Dev mailing list to discuss details of the integration as needed. We also have an IRC channel where the people work on this integration hang out and chat throughout the day about the things we're working on or any problems that people run into. And that's the Pound OpenStack Neutron OVN channel so come hang out with us. And as a quick little tease. Yeah so you know I think we're going to the keynote tomorrow there may be a mention of OVN just kind of a tease I guess. I think I was supposed to dance there but I didn't. All right so thank you very much and I believe we have maybe about five minutes for questions. Can I ask Jackson. So the question was did we go with a core plugin versus ML2. Right now it's a core plugin. Question there at the mic. It'd probably be easiest if you can line up at the mic but if not then we'll try to grab you. So let me get this one next. Thanks a lot. Really appreciate to see all this advanced feature. We've been waiting for a long time right. All this connection tracking. All stuff so it's really good to see that. I don't know. We can hear you fine. So I can see the potential benefit of using this OVN structure where there is a southbound DB and a northbound DB. Basically effectively separate out from the open stack side with this kind of neutral side plugin. One concern though is. Sorry one second. Hey guys if you want to talk it'd be great if you want to go ahead and step out. It's actually really hard to hear the question and probably hard for people to hear our answer. So thanks. Thank you. So if you look at the structure right now one of the practical potential issue we're going to run into is that. When when something goes wrong let's say you know south south you know southbound DB somehow does not respond anymore northbound DB something screwed up or. So when there is something really goes wrong and the or is the in terms of actual operational perspective is very hard to track down and what's going on. You know why why you know you know neutral does not respond or something. So and also in terms of dealing with really large scale as you mentioned right. You know in fact you know anyway we're going to come down to OK how we're going to really deploy the truly scalable structure of OVN that has truly scalable DB you know component to there. Yeah so I think what are those you guys had at this point. So I think this this question comes down to where's the where's the scale and the the reliability of the database implementation itself. Well so there's a few ways to look at this. So first we're we are working on adding a scale out and H.A. to the OBS DB itself. And so that's one way to look at it. The other is that OBS DB is what we're using because we have really good integration with it and we're really familiar with it. But there's nothing in the design that says it has to be a particular database. And so if you suppose that we fail at making OBS DB fast enough reliable enough and scale well enough then we're perfectly open to picking a different database that already has all those features. It wouldn't be it wouldn't be traumatic to switch to another database. And we'll just do that if it proves necessary. I asked a stupid question. The L3 agents are the L3 support you're adding. You added dynamic app support was added to OBS or to OVN. And then the second part is who can work with to get support for deeply not burn interfaces in OVN for the host user etc. So let me start with the the first question about dynamic app support. So the this this the patches that I'm preparing to get out for review on dynamic app did require a small enhancement to open V switch itself. But it's generic and it might prove useful to to other users of open V switch. I'm really reluctant to add anything to OBS that would be OVN specific that that would be unfair I think to to other users. But what I'm what I'm planning to work on is probably going to be useful to other people. My guess is that when they see it they'll say oh yeah that's useful. Let's use that. And I've already forgotten your other question. The second question was who can I work with on exposing DPDK northbound interfaces. So that came up on the mailing list at one point and we need to revisit it just because like we've been super busy with like a sort of existing task list. And the answer was effectively like why does OVN need to expose that and it seemed overly complicated and it was like we need to revisit this. And I think it's not so much on the OVN side unless I kind of want to revisit this in the open stack side. I think open stack requiring OVN to expose that to the northbound seems overly complex. We need to revisit it. So that's kind of my answer. I pulled the engineer who kicked off that discussion on the mailing list. It came to me originally I was basically we need to know it so we can tell Nova what interface type it's going to need. Yeah I understand. We don't actually need to tell OVN about it. I know but like to me it's like Nova can pretty easily figure out the capabilities itself. It's the thing plugging the thing in the switch it can look it out. So anyway we probably don't have time to go into details but let's talk more. I want to revisit that and just haven't had time. I think we got a couple of minutes. So currently a requirement is OVS support and there isn't a port of OVS to ESX available. Any other questions? Cool thank you all very much.