 Good afternoon, everybody. It's 3.40. The microphone is on, and I guess it's time to start. My name is Salvador Orlando, and I have been a regular contributor to OpenStack Networking for a few releases cycle. In the past, I also used to be part of what is called the core developer team of Neutron, means the people that basically can approve patches, and I'm now not any more part of the team, more on the way to retirement, but still contributing. And today we are going to talk about writing Neutron plugins when you really need to write a Neutron plugin. And this talk is specifically aimed at discussing how to write Neutron plugins for the Neutron stadium, which is the sort of revolution which has happened in the past year, which on the one hand has empowered developers, integrators for developing their plugin and drivers in a completely independent way, but on the other hand has introduced a new set of processes which in some cases might be a little bit challenging. Probably you are already aware of most of these processes. And the aim of this talk is to put all the things together in a single place. And you know, I know that it's 3.40pm, we are all a bit tired, and so I'll try to make it as most entertaining as possible. I mean, I don't know, I know that a technical talk is probably never going to be really entertaining in any way, but I mean, I'll try to make my best, worst case I can start dancing around. So let's start with a bit of a summary. So in the first part of the talk I will start confusing you with definitions of plugins, service plugins and drivers, but I also try to put a little bit of order in when it comes to taking a decision whether you really need a Neutron plugin or more like, you know, especially when you're doing ML2 and ML2 driver, because that's a decision that often, you know, if you also look at the mailing list, that's a decision that it's often not really obvious, and people tend to think that whatever they do they have to do an ML2 driver, which is not necessarily true. And then just to make things more confusing, we'll introduce the Neutron Stadium. I mean, probably most of you are already aware of what this Neutron Stadium means, so it's not going to be big news for you, but the last part of the presentation and possibly the longest part of the presentation is about driving the Neutron Stadium. So how to deal with the new set of procedures and rules and technical differences that you have to deal with for having your plugin or your driver living in this stadium. So what is a Neutron plugin? A Neutron plugin is basically the engine that implements the whole Neutron API. So Neutron itself is that tiny layer that you see in the blue box with the API on top of it. That's just a tiny layer. It doesn't do a lot of stuff. It does authentication. It actually Neutron doesn't do authentication. It talks to Keystone for doing authentication. So, you know, it just have the hooks for doing authentication. It does authorization, validates the request, and then dispatches the request to the plugin. Now, all the important stuff happens in that layer in the plugin. And the plugin, for instance, if you look at the reference control print implementation, deals with communication with the agents of our PC channels, agents, you know, like layer 2 agent, layer 3 agent, the DHCP agent. Instead, if you look at commercial plugins or, I mean, other plugins, it just acts as a proxy towards a third-party controller, storing anyway data in the Neutron database. And in some cases, like, for instance, the open-contract plugin, it's even simpler. It's just a pure proxy towards a third-party. It doesn't even... It's completely stateless, too. So this is the basic architecture. And the plugin, it's maybe a single component. Or, as it is in this picture, it may be multiple components. And this really depends on whether you have just a standalone plugin implementing all the APIs, or whether you have a set of core and services plugin. So what do we mean by core here? Even if you are... We are now in 2015, and Neutron has been around for five years. We still call core functionality just basic layer 2 networking plus IPAM. So, you know, in Neutron, it might be surprising, but layer 3 networking, layer 3 capabilities, and even security groups, which are basic functionalities required for integration with Nova, are still considered extensions. But the way it works... So when it comes to dispatching requests in Neutron, you have on the API side, you have the core API and the various extensions. Here, for instance, we have just the core extension, layer 3 extensions, and the security group extension that for some reason I've called it firewall in this slide. And you have different options. It sort of depends on what you deploy in your backend. You can have either a single plugin implementing all your APIs, as is in the picture on the left. You can have, for instance, a plugin doing the core functionality and the layer 3 functionality, and then another plugin doing firewall. Or, for instance, you can have a distinct plugin for every functionality. What are the differences? Where you have a standalone plugin, obviously we are probably more looking at the integrated solution. On the other hand, when you have multiple plugins for different services, you are probably looking at a solution where you have different technology providing different services. And the additional challenge here is that you have also to consider interaction among the various plugins. So this is the basic picture, hopefully not too confusing. To give you some examples, when you run the default stuff, which is what is tested by Neutron upstream in the gate, you're running a plugin for the basic functionality, the core functionality, which is the ML2 plugin. And then a layer 3 reference implementation plugin, which does communication with the layer 3 agent. In the basic implementation, basically, sorry, there are too many, basically, in the reference implementation you have the core ML2 plugin, which implements the core API and the security group API. And then you have the layer 3 reference plugin, which is called the L3 router plugin or something like that, which implements the layer 3 API. So the question is what about DHCP? This is a good question because the DHCP agent is a key component of the Neutron reference control plan. But it's not in that picture for a very simple reason that we don't have a DHCP API and we don't have a DHCP plugin. Communication with the DHCP agent is entirely performed by the ML2 plugin. As a part, for instance, when you react to a create subnet operation, that's where you have a command sent to the DHCP agent. So the picture that you see here is just what happens in Neutron server. What happens in Neutron server is that you have APIs dispatching call to plugins. But then there is another part of the picture, which is the integration between the plugins implementing the reference control plane and the various agents. So for instance, there will be the core plugin in which this case is ML2 which interacts with the layer 2 agent for doing VIF wiring and ensuring layer 2 connectivity and securing VIFs by implementing security groups. But it also talks with the DHCP agent to program the DNS mask instances running in network name species that then distributes addresses to virtual machines. And similarly, the layer 3 plugin communicates with the layer 3 agent. So in order to avoid confusion, this picture here is just Neutron server. The agents are here on the bottom. You don't see them, but they are all here. Anyway, some plugins like ML2 are a bit particular because they have not just drivers too. So that's kind of two-level indirection, where you have a Neutron server that dispatches call to a plugin and then the ML2 plugin which dispatches call to a driver. The difference here in the advantage of having two levels of indirection is that a lot of boilerplate code which is required specifically for doing database operation sits in the ML2 plugin and the driver developer only has to implement the actions which are performed at specific moments during request processing. For instance, in the case of the ML2 plugin, you have the pre-commit operation and the post-commit operations which are respectfully performed before the database operation is performed and after the database operation is performed. And also another characteristic of the ML2 plugin is that you can have multiple mechanism drivers. You can have hierarchical port binding that allows you also to manage heterogeneous fabrics in your data center. But all, I mean, it's worth keeping in mind that this is entirely exclusively pertaining the layer-to-layer and therefore simply with binding and ensuring basic layer-to-connectivity. A similar architecture is also implemented by the reference plugin for the advanced services. As we discussed in the previous slides, there were different services for Neutron. Some services are the so-called advanced services which have been split out in different repositories as part of the Neutron Stadium operation and they have different plugins which adopt the same driver architecture. So now the question for you is, should I implement a plugin or an ML2 driver? Let's start with a real case. And the real case is the case of an OVN plugin. I mean, you probably are aware of what OVN is. Yesterday, Ben, Justin, and the other OVN folks gave a talk exactly in this room and so what happened with the OVN plugin was that its development started as an ML2 driver. At some point, you realize that ML2 does more than ML2. Sorry, that OVN does more than layer-to. There are other services. Just because of this, the ML2 driver alone is not sufficient. But also, you realize that for the very architecture of OVN, it's not like you can have multiple drivers that interact together and cooperate. So what happened was that people at some point, developers at some point realize that the ML2 architecture really didn't make a lot of sense for OVN and OVN has now switched to a standalone plugin which is a single plugin implementing all the network services exposed by Neutron. But in general, the workflow could be something like this. You start asking yourself a question. Do I have only layer-to to implement? If you are just implementing only layer-to, for instance, what you want to integrate with Neutron is the switch that's running your data center which exposes an API, which you can configure somehow, then you need to ask yourself, the device that I want to control, does it interact with other devices in my data center? For instance, is that top of rack switch and then you have leaf switches that are part of the same network so that maybe you need some sort of... You want to have Neutron doing some sort of orchestration, configuring both switches and ensuring that you have a single layer-to-connectivity across all these devices. If the answer to this question is yes as well, then you definitely need an ML2 driver. If the answer to this question is no, you can just implement a plugin for the core Neutron API and be good with it. But I have to tell the truth, but the thing is that even if the answer to this question is no, you can still implement an ML2 driver because even if you don't really need ML2, you will still have the advantage of not having to deal with re-implementing... I mean, doing all the DB support code, you just can leverage whatever ML2 already provides for you. But on the other hand, let's go back to the first question. If you don't have just Layer-2, but for instance, you have other services, just Layer-3, for instance, if you want, and you need to ask yourself whether you are able to mix and match those services, what I mean by mix and match. For instance, can I run my Layer-3 services on top of somebody else Layer-2 solution? For instance, if you are... if you have a wonderful integration with some Layer-3 product, can it run on top of the ML2 plugin with open-v-switch? Can it run on top of, let's say, Layer-2 with mid-onnet? If the answer is yes, then you definitely need to go for a modular approach. A modular approach where you have distinct plugins for... when you have distinct plugins for distinct services. And so we are going on this box. Among the various services, are you providing Layer-2? Yes. If you are providing Layer-2, you need to ask yourself the same question as before. If you need to interact with other drivers, then you need an ML2 driver. If you don't need to interact with other drivers, you don't need an ML2 driver, but since you don't do only Layer-2, you definitely need to provide service plugins for every service that you provide. Finally, if you are in a condition where you are driving a third-party controller, which completely manages your virtualized network, as it is the case of... of OVN, or as it is the case for instance of mid-onnet, or it could be the case of commercial controllers like the plugins that we maintain at VMware, definitely the best solution for you is to do a standalone plugin. So I have this diagram, which is very clear, just like a tube map. But hopefully, it kind of provides some guidance for people that are approaching the task of writing a plugin. So now that we have a very clear picture of what the plugin is, what the service plugin is, what the driver is, and how they interact, we will introduce the Neutron Stadium. As you can see, there is actually a Neutron Stadium, and people are all cheering for Neutron, and what means the Neutron Stadium? How many of you are aware, actually, of what the Neutron Stadium is? Well, I need to talk about the Neutron Stadium then. So how many of you are actually aware of the OpenStack Big Tent? Fine. So the OpenStack Big Tent is a concept that we can assume that we are fairly comfortable with. The thing is that within Neutron, we have decided to build a stadium within the Big Tent. I know it doesn't make any sense having a stadium in a tent. It should be the other way around. But in Neutron, we are a bunch of people that used to do things that don't make a lot of sense, so let's not be too much worried about it. The concept here is that within Neutron, we have the ability of having multiple projects with a sort of independent governance from a technical perspective. So every project gets its own repository with their own core team. And this means that you have a lot of freedom. You have a lot of freedom of developing the code in the way that you see best fit for your project. You don't need to go and ask the Neutron core team for approval of Blueprints, which pertain to your project. You don't have to ask the Neutron core team for approval of patches that pertain to your code. For the integration with your backend, you are fully in control. From an OpenStack governance perspective, this does not change a lot. It doesn't mean that you are an OpenStack first level project with your own PTL and whatever with the elections. You are just, from an OpenStack perspective, you're still part of the Neutron project, of the OpenStack networking project. But within Neutron, you are, from a technical perspective, you are completely independent. However, this comes with a cost. So these are the advantages here. So the advantages are that you are self-managed, each plugin or driver has distinct development and core team. And each plugin or driver lives in its own code repository. And there is a limited oversight from the Neutron core team, so pretty much you can merge whatever you want in your plugin. And you just, you have to, optionally, you can decide to coordinate with the rest, with the Neutron for doing a release. But if you want, you can release whenever you want. So these are the good aspects, which means if you remember how it was Neutron, let's say, until the end of Icehouse, does anybody remember what it meant for a vendor to merge for a vendor or anybody doing a plugin, which was not ML2, merging a patch into Neutron for a plugin-specific patch until Icehouse, it meant that you push the patch, like at the beginning of the release cycle, and then perhaps a year ago, a year later, that patch will be reviewed and merged. This led to a situation where there was a queue of about 10 plugins waiting to be merged in the Neutron core tree, and there was a backlog of, I think, over 50 blueprints, plugins-specific blueprints that were not even reviewed or approved nor targeted to any release. So basically, plugin development was pretty much stuck, and the Neutron core team was really busy with dealing with the plugin code, which often people in the Neutron core team didn't have any clue of what that code did, and so that's why it was decided to move all the plugin code out of tree. This decision has paid off very well, but there are some caveats, something that a few challenges in doing these that we are going to address in the last 20 minutes of this talk. And what best way of addressing this caveat than, you know, doing it with a concrete example, and for this reason that we are introducing here, I think, a chance in this talk to introducing a new plugin, the plugin for human-defined networking. What is human-defined networking? It's a chance for you all to discover the human side of IT. I mean, aren't you bored of network automation? Don't you think software-defined networking has failed their promises? I mean, it's been like how many years? Like four years they've been promising you to have fully visualized networking in your environments, and that's never happened, right? It's time to stop and go back to the past and rediscover the human side of IT. And this is why we are introducing here a human-defined networking plugin for Neutron, which completely redefines how virtual networking works. So the main concept, you have a REST API for doing request, which is the Neutron API, and all the requests are converted into emails which are sent to the networking team in your office. Stop doing, like, RPCs, stop using these agents. You don't need these agents. I mean, if you consider how much a single instance on a Layer 2 agent is costing you, you'll find out that hiding somebody, it's much cheaper for you. Then the process, you don't have to worry about scaling or not scaling because it's implicitly asynchronous and eventually consistent. Also, you don't need to worry about quality of service, you don't need to worry about time processing requests because it's karma-based. There is karma-based request prioritization. If you are nice to an IT guy, or probably you're buying some chocolate or a beer, you can rest assured that your request will be served in a very quick time. So these are the concepts of the HDN plugin, and this is very simple architecture. As you can see, there is the REST interface, the message bus, which is based on email. At the moment, I'm sorry, I have only email, but I swear to you, I'm planning to implement also phone and fax, and somehow also post, really. You can send a letter, put a stamp on it, and eventually you'll be given to the IT team that will cheerfully proceed to satisfy your request. So when you have to think about creating a neutron port or create a neutron network, and you try to automate across all the switches where you can have somebody going there, I mean, I don't want even to get started on scaling out. I mean, scaling out, what it means that you deploy more agents, and you make the agents multi-traded, and then you say, okay, we can use monitoring of the OBS DB to get notifications and immediately proceed all the changes. Come on, you can just hire more people. The more people you are, the more the system scale. And I swear to you, I mean, it might be actually true that with all the money that you spend in maintaining your network of virtualization infrastructure, actually hiring people will be cheaper. Anyway, let's look at the... Now, let's get serious, sorry guys, let's get serious, let's stop playing. Let's look at the architecture of the ultimate plugin that's going to solve all the problems of your data centers. So we are implementing the core APIs, layer-through API, layer-through APIs, security group APIs, and then we have an additional API extension, which is specific to the HDN plugin, which is for administrative tasks. And our architecture is that we have decided to go for a one-to-one mapping between services and plugins because we want to progressively roll out, roll this out in your data center. For instance, if you are already deploying ML2 with some random layer-through stuff, you can start by checking away your layer-through stuff and introducing ML2. And over time, you can roll in also, you know, core plugin and the security group and get rid of those also of ML2 with whatever SDN thing you're using, completely useless stuff. So anyway, if you want to implement now this thing in the neutron stadium, you need to keep a few things in mind. It's not that easy as it was before. Just do the code and drop it in the neutron code tree. You are running outside of the neutron code tree. So there are a few things to keep in mind. One, how to successfully do development with this stuff and therefore integrate with DevStack. The second thing is that you likely need some changes in the neutron database schema. And since you are not anymore in the neutron source code tree, we need a solution for doing from an external plugin changes in the neutron database schema. And then we, since we have multiple plugins which interact each other, we need another solution for ensuring that this integration works correctly. And finally, we'll have to deal with testing all this stuff. So let's start with DevStack integration, which is probably the simplest bit, because DevStack, since a year so far, has a plugin structure. So rather than going into DevStack and doing changes for explicitly supporting HDN, you can just declare that you are using a DevStack plugin and drop the DevStack plugin in the HDN source code tree, which is exactly what you are doing here. The DevStack plugin, it's very simple. It's just a script. This script gets loaded when DevStack is running. You can do three things. One is to execute something at specific stage during DevStack's execution, which are, these stages are before service configuration, before service, sorry, before service, before software installation, before service configuration, and after service configuration. Then you can also override default values that are DevStack default values. And in the specific case of Neutron, you can redefine some of the plugin-specific Neutron procedure to have them specific to your plugin. So let's have a look about how it's done here. I need to zoom in a little bit. So this is the DevStack plugin. I hope you can see it. Probably need to make it slightly bigger. So this is an example of DevStack plugin. As you can see, the code for the HDN plugin has been shared as Available on GitHub. So you have a bunch of functions which are basically override of a plugin-specific Neutron function. For instance, you know, is Neutron OVS-based plugin? We return one, which means no, because as the comment says, OVS is totally evil. And all the other functions which are for installing the agent packages or configuring the HTTP agent or configuring the HTTP agent are just empty functions because we don't need all that stuff. And we just need some functions for configuring Neutron for running with the HDN plugin. And here our configured HDN plugin is the most important function because basically it configures all the parameters which we need to run this plugin, which are pretty much just email settings, and that's it. But the most important thing is that this is how the DevStack plugin actually works. This is the skeleton of every DevStack plugin that you may want to implement. You have basically, if your service is enabled, where your service is your specific plugin, then you have three phases which are install, post-config, and post-extra. Typically, all the plugin configuration for a Neutron plugin is done in this phase, the post-config phase. And basically you just have to plug there your shell script, which does the plugin configuration. So let's go back to our discussion of the HDN plugin. Let's see how we solve the problem for the database schema. The HDN plugin just adds a little table for managing these operator's tasks. You know, when you create a network, it adds a task for an operator where you kindly ask the operator to create a network, to set up a network for you in the data center. But the problem is that since the HDN plugin is living in a distinct source code tree, we can't alter the Neutron schema directly. So we add a schema migration which is specific to our plugin. Just like Neutron migrations, also plugin-specific migrations are managed using Alimbic. And each sub-project can get its own distinct Alimbic environment. This is something that you have to keep in mind that each sub-project has a distinct Alimbic environment. And you can have multiple branches just like for Neutron. So if you're aware, Neutron has multiple Alimbic branches now. And this is for making sure that you can do online migrations without having to restart your service. You can do the same thing in your plugin because basically you will inherit the same environment that Neutron uses. And you can also specify if you want dependencies between your branches. But be careful, you can't specify a dependency with a migration which is in Neutron. So basically if you want to say, add my table for tasks, not before the table x, y, z is added to Neutron, that's something that you cannot do that, you cannot yet do, right because these migrations are separated. I don't want to go into the technical details because this is very, very boring. But here on the slides you have all the references. The first reference is an example of how it's done for the HDN plugin. And the other two references here are links to the Neutron developer documentation where all these details, all these things are discussed in detail. So one important remark that I have is that the Neutron developer documentation has all that you need for making sure that you can successfully implement and maintain a plugin out of three. When you go to the Neutron developer documentation you have two sections. One is policies and one is Neutron dev ref. In policies you will find all the process aspects which are required for you as a maintainer of external plugins and all the technical details are instead in the Neutron dev ref side. So as we said we had several plugins which can be unaware of each other. So what happens, for instance, when you create a network using the HDN plugin and as a result of creating a network you have to add a task for an HDN operator to go and configure that network in the data center. So you do this using the Neutron callbacks mechanism. This is a mechanism which has been introduced in the key release of Neutron and it's very comfortable for making sure that you can react to events which are triggered in other plugins rather than going manually to alter the code of the plugin to interact with your plugin. So this way plugins do not need to be aware of each other and in your plugin you can just subscribe for events. So for instance if you want to run a callback after a port is created you just have to subscribe, register a callback and subscribe it to the event dot after create to the after create event for a port resource. And that callback will be executed after a port is created for instance. A typical example of this case where you have a service plugin which makes use of ports as is for instance a layer 3 service plugin which uses ports for router interfaces and you want to add a callback to react to a delete port event. Yet in case if the call plugin tries to delete a port using your L3 plugin you can register this callback to prevent this from happening. And plugins basically just generate events as you can see here in the slides by calling notify and when you call notify the registry basically takes all the callbacks that are registered to that particular pair of resource and event and simply calls the callbacks, invokes the callbacks. And for the hdn plugin this means that whenever resource is created we trigger an after create event and then the hdn task extensions register the callbacks there is similar behavior for update and delete events. What happens is that you create a network and there is a task which is a the callback automatically creates a task for an hdn operator sends you an email. When the hdn operator reads the email it gets the task ID that pertains to that specific operation it goes to the hdn plugin gets the details of the task goes to the data center bugs cable, configuration switches has a cup of coffee then comes back to the computer task completed okay and that's how the workflow works and I mean that's pretty much it that's very simple there's really not much to say and now one thing that you have to keep in mind and that will annoy you a lot is that since the now you're living in a separate repository what happens is that actually few people that are doing changes in neutron are not able to know that they might be breaking your plugin this actually happens more often than you wish for because the great majority of neutron plugins are actually inherited the so-called the database mixings which are a bunch of base classes which do common database operations so often it happens that somebody does a change in the database class that change upstream is only tested with the ML2 plugin and it doesn't even realize that it's breaking your code so you have to be prepared for failures that you can't just avoid them from happening I believe that for the plugin that my team is maintaining we have them on a weekly basis I mean on a good week we have one failure a week on a bad week we have one failure a day so you have to prepare it for it I mean constantly test your plugin and in the vast majority of cases I'll say 95% of cases you just have to run a unit test to find out that your plugin is broken to pinpoint root cause and fix it most cases if you don't let this failure accumulated therefore causing bit rotting most cases if you interact promptly it's just a matter of few minutes to push a patch to fix your plugin but the bottom line is that there's nothing that you can do that this is a consequences of the introduction of the Neutron Stadium and you just have to be prepared with it and deal with it so testing is very important for your plugin I don't think we have a lot of time to discuss about testing because we are three minutes from time but the rule of thumb we are whatever you are whatever you're doing in your plugin if you Neutron test it your subproject should do that too so does Neutron run an API job? yes, you should run an API job too does Neutron run integration test? yes, you must run integration test too do you have to report the votes from your CI upstream? probably not, that's not important that's for two reasons is that maintaining a CI that votes upstream requires you to do an awful lot of work because if you consider how many patches are submitted everyday upstream this means that you need to have a CI infrastructure which is ready to test hundreds of patches a day and that's probably asking too much to a plugin maintainer the second reason for not voting upstream is that unfortunately developers are going to ignore the vote of the third party CI you can put a minus one because it's breaking your plugin but that minus one is probably going to be overridden and in the worst case it's going to annoy people but it's an important information for you not for the other developers there's important information for you that a patch that creates you a problem is about to be merged or is already merged and you need to react quickly to fix your plugin perfect to conclude this discussion and I see that just a few of you are sleeping I see just one guy sleeping he just woke up I'm going to beat you after the talk you're mine and basically other challenges that you have to deal with is release management release management basically it's the only thing that you can do on your own for your plugin is to coordinate release management with a member of the neutral release team but the neutral release team did a very good job of describing the process in detail in the document which is linked here and then the requirements for your project and dealing with the requirements is difficult it's tiring it's boring so the best thing that you could do is just subscribe to the requirements contract and you'll find more details at that address and when you subscribe to the requirement contract you will not have to deal anymore with the requirements you'll receive periodic updates from the global requirements OpenStack project and that would be pretty much everything you need you don't need to do anything else I mean there is a caveat here is that if you have special requirements for your project you need to push that special requirement into OpenStack requirements so that's the only caveat one note about Oslo incubator we've seen that many plugins are importing stuff for Neutron.openstack.com that's not such a good practice you need to import stuff directly from the Oslo incubator and finally we already discussed what you need to do for your CI but if you want to look at more details that's your link and well just to conclude don't forget about documenting your plugin about providing admin guide for your plugin providing a developer guide for your plugin and now it's really everything from my side I would like to thank you for your time and any question that you have you are welcome to ask I hope I did not bother you too much and please install and spread the word about the hdm plugin thank you if you have a question it's a quick question on the hdm reference I didn't see any mention of LBAS yes I'm looking for contributors I'm looking for contributors and the way I'm going to implement load balancer I'm not really sure how to implement I will not really I mean probably we can use we can use like hardware load balancer but I was actually considering giving people you know plugin cable run in a random way and but at some point I probably need some sort of hardware load balancer but I'm finding a way to make it more human as well yeah maybe for next summit because I mean we are tired of sdm we need to implement this hdm stuff and make it the reference implementation any other question seems that we don't have any other questions so thank you very much for your time and spread the word about hdm