 You give you that by PowerPoint. So actually since we are all pretty a bit tired We'll try to keep it a bit light and possibly funny and Hopefully try to explain something valuable as well. So I am Salvatore There is Armando We are before we dive in I mean, I just wanted to mention obviously. Thank you for coming but for the Previewers in the audience Anyone who like reviewed no VMware's code. There are gift card Because Starbucks gift card for you at the VMware booths to go and you know have a coffee on us Well, not today Over? Yeah, maybe too late. Sorry. No, you perhaps maybe if you grab like our Anyway, so as you understood VMware is the one that piece the bill So I am Salvatore the fat guy is Armando. He used to be fit Yeah, but now probably is getting gold, you know, especially after a week of binge eating So what do we do in our life? We both are core open-stack neutron developers I'm usually the one that breaks thing that he comes in and he fixes my code So for this reason since I'm the one that breaks stuff, I'll I don't understand anything I'll start talking and speaking you don't really listen to me Then our Manda will come in and tell the serious things about how to write What are the things that you need to consider when writing a neutron plug-in today's talk is? About writing neutron plugins The title if you've been if you think that you get out of this talk And you'll be able to write a neutron plug-in and understand they've been thinking about neutron Well, the title of the talk is very very misleading in that case for you I'm afraid but hopefully we'll be able to give you at least the beats that will enable you to understand Whether you really need to write a neutron plug-in That's the first thing that we consider and if you really need to write a neutron plug-in where you should start from so We can start move on the first part will just try and be a bit of recap of what is a neutron plug-in What is an extension? What is a service plug-in just to put a little bit of an order on some? Terminology which especially for newcomers may be really really confusing then Armando will Talk a little bit about the various design choices that you have to face when writing a neutron plug-in the problems that you might face This is pretty much based on our experience both with the Open source plugins like the open this which plug-in which to which we have contributed as well or to end to the plugin that we maintain which is The nice here MVP plug-in and then there will be the first part which will if there is time left We will look at some sample code for an example educational plug-in. Okay. Let's move on The word of neutron plug-ins So this is probably a concept which most of you are already familiar with a neutron plug-in is Basically just let's say in its simplest form It's a collection of Python modules that implement a standard interface They receive API code they receive a comma and operations directly from the API layers and Then they can interface With either with agents like the layer 2 agent layer 3 agent the ACP agent Or they can control third-party controllers as it is for instance in our source code base with the radio plug-in Or in other cases they can be a little bit more complex like they can have a driver and then that driver is specific for driving a Particular class of devices like it happened for instance with the ML to plug-in where you can have a driver for a Specific network type or you can have a driver which for a specific network device A plug-in is about Implementing cooperation. So if a plug-in receives a create a network operation It's it's responsibilities to ensure that the all the operations required for creating that network are performed But other things like authentication authorization Validation of request it's things which are already performed in the API layer So if you are doing a gain authorization in the plug-in You're probably not doing something extremely exactly correct because it might be that your plug-in will end up So doing something that the operator the user is not expecting. So Sometime you Here talking about service plug-ins Especially when it comes to advanced service. So we have for this load balancing or firewall VPN What's the difference between a core and a service plug-in the core plug-in as Minimum requirement of implementing what's defined as a core neutron API and this is basically layer-to-networking an IPAM a Service plug-in instead provides additional network services Currently if you look at the code base, there are a set of service plug-in Which are already defined the simplest service possible service plug-in is the layer 3 plug-in now with avana as ml2 is a kind of diva become the new Let's say default plug-in. We have a split layer-to-layer 3 services So you are able to run ml2 which does only the layer 2 API and then a service plug-in which does Implements the layer 3 API So the difference here is explained probably in the diagrams that you see below We have an example on the left where we have a single monolithic plug-in So it is a core plug-in which implements not just the core API But also the layer 3 and the firewall API and all the remaining APIs And this is the case of a monolithic plug-in as I was saying like for instance the MVP plug-in In this middle diagram you have a core plug-in which implements the core API plus some extension in this case the layer 3 extension Plus another plug-in which is a service plug-in which just implements the firewall The firewall API this is the case of for instance of running The open v-switch plug-in With the firewall service plug-in in avana The right most case instead is a more the more modular possible Where you have three distinct plug-in each plug-in for a different extension and this will be for instance the case where With avana you run ml2 the layer 3 reference plug-in that we currently have and the firewall reference plug-in for the firewall API so this is about plug-ins and Core plug-ins and service plug-in but gets even more complex because we want really to make things more complex for you So that you You'll get a big headache when approaching it run some plug-ins have drivers What's the difference between a plug-in and the driver? The plug-in is the interface with the API layer the driver is a An actuator is what performs the operation on a specific back-end We have several examples of plug-in with drivers the ml2 is probably the best example when you have drivers for Open v-switch and Linux bridge, but also driver for upper v tail f are instant networks so And basically this driver mechanism has allowed us to take what was previously the open v-switch plug-in The linux bridge plug-in and bring them together in the ml2 plug-in because basically we had the two plug-ins That were performing exactly the same operation exactly in the same way with the only exception at the end of the day a command was performed Calling all of us vc8 sctl for instance for the open v-switch plug-in a calling Brctl for the linux bridge plug-in since there was just a difference in the Actuation of the command you can bring them together have just two distinct drivers and use exactly the same plug-in Why we should do why it's better at driver than a plug-in because a driver of course doesn't have all the boilerplate Code that you need to replug in so the boilerplate plate code stays in the plug-in And you just have to implement a smaller subset of the code so moving forward My mom may have to make the right decision for instance either implementing a driver or a plug-in implementing a service plug-in or Implementing a whole new plug-in. It's all about trade-offs. So sometimes for It might be difficult to interact with the all the different service plug-ins The drivers and so you might decide to go for a new Mololytic plug-in in other cases you might be interested in the flexibility of Interoperating with a lot of different systems and that will be the case where for example You decide to develop a driver a new service plug-in or an extension that Plug-in implementers cannot do their own plug-in. So those are the choices that we Have to consider that we have at the end of the day For instance, if you have some kind of device like you are a vendor you have a switch You have a firewall you have a load balancer and you want to integrate your device into Neutron Do not think about having a new plug-in that will be probably an overkill for you In that case you should think about having a driver if the device you want to integrate is a switch Think about an ML to driver if you have a load balancer think about the load balancing driver if you have I don't know A coffee machine thing. Yeah, you need to develop a plug-in for a coffee machine. Yeah Yeah, we are thinking about having Neutron doing coffee nights house, especially when it's late at night, you know and Instead if you have a new features that applies to existing API resources you have mainly mostly two choices And an example is a feature which was introduced in Havana, which is the bandwidth metering feature So you can either have an extension and then out of the support for this extension in several plugins or Have a new service plug-in in the case of the Metering extension since it was like a standalone thing orthogonal feature the choice was adding a new service plug-in if you consider instead another extension like the extension for providing the logical routers without not So without the features with the ability of enabling a disabling source nothing That's kind of a something which applies to plug-in which are already existing And so in that case instead of having a service plug-in you will prefer to have an API extension and Relevant a plug-in support Finally when it's the case where you need to implement a new plug-in There's a very probably a very limited subset of cases either if you have a new integrated solution with a vertical stack like you know an SDN controller or Let's not call this SDN controller a controller that does a layer 2 to layer 7 features or if you for instance decide that you want to implement a Let's say the same features like Layer 2 layer 3 network in perhaps also with open source components But you want to go for a new paradigm because for instance you think that the current paradigm which is used by You open the ML to plug-in is wrong You think you can do something really better than that case We won't use a different paradigm you will have to implement the best choice for you is to implement a new plug-in So I think this concludes the available choices Let's say that we are now in a stage where we decided that everything that already exists It's not good for us and we want to implement a new plug-in. So since you know, I break code I break stuff I can't do anything, right? Armando is going to talk to us And as well, I mean I was wondering whether you guys have any questions so far Everything is like crystal clear and we can move on We have one two three Yeah, let's try to give it interactive because at this time of the day Yeah, we should give a gift people asking questions What what can you say about if you want to reuse the plug-in? But you need a driver So is it a type driver or when to write a type driver versus a when can you get away with just doing a mechanism driver? Yeah, type driver versus a mechanism driver is a question that should probably It's specific to ml2. So basically if you look at the existing type drivers type drivers are for transport logic For different transport mechanism. So you have the fake. Yeah, sorry. You have a VLAN type driver GRI type driver VXLAN type driver. So that's for implementing strategy strategies for For doing the transport network. So do I have to allocate the GRI tunnels? Do I have to allocate to put a VLAN? Sorry a VLAN tag on the pocket or do I have to do VXLAN as a transport layer instead of mechanism driver is Now I have decided to use GRI as a type driver How can I implement the GRI tunnel using this particular mechanism? which could be open with which linux bridge the particular arista switch or the X Y Z the network device or the ABC Intelligent supers which layer to layer 99. So that's the difference between a type a mechanism driver Mechanism is about the device type is about to transport I wanted to have a flow chart which is a very good point. The thing is if I had that flow chart on these slides We will develop discussing just the flow chart in this session So what do we what I'm what I'm starting to do with with me in presenting in this session is Collecting all this stuff perhaps either on the OpenStack wiki or some blogs which could be a reference for Developers which are approaching a neutron, but yes, that's definitely a great idea Make sure that I understood what you said so Given that the choices that you laid out, I mean it seems that writing a new monolithic plug-in That's not a you know, doesn't fall the ML 2 model is still okay going forward in ice house and definitely It's still okay if you if you have let's say your if you are Want to integrate neutron with something which is super cool yard that it's very hard for you to Integrate into the ML 2 framework or if you think that for some reason Your plug-in will be a lot less efficient if implemented as an ML 2 driver And you think that the best route for you will be having a new monolithic plug-in Well, definitely go for it and I would suggest that probably in that case I would advise having both an ML 2 driver for flexibility and a monolithic plug-in Which is possible of course you'll need to do more work even more work But that's probably even a better strategy because if you have a user which wants to be flexible and for instance Use your particular solution just for layer 2 and somebody has solution for layer 3 you'll have layer 2 driver But if somebody wants to go vertical with your solution, they could use your your plug-in Okay, the deprecation of OBS and Linux bridge. It's something which is a problem You should not really be worried about why is that it's that because actually the ML 2 plug-in is not introducing a new Paradigma which was not already implemented with open-v-switch and Linux bridge is when you will migrate migrate for a van advice Your deployment you will automatically switch from open-v-switch or Linux bridge to ML 2 There will be no data plane outage and there will be also an automatic database migration to make sure Yeah, I know people are a little bit laughing because then there will be problems. Okay We know there will be problems, but let's say that at least in theory. They should on paper. They should not be problems Yes, the agents for instance are exactly the same Yes, the agents are still the same. We can talk about This is a little bit of topic so I will go add with the next section so next topic Is there any more question? No, I think we So okay, so once we made like the decision on what kind of architectural choice We want to go for obviously Let's let's dive into like the nuts and bolts on how we can like plan and develop for a new neutron plug-in so one thing to to keep in mind is that obviously, you know neutrons first consumer is open stack and especially Nova and so when when thinking about Developing a new plug-in we gotta make sure that if you want Nova to use neutron Correctly we need at least to implement the L3 and then security group extensions So that again things like security groups and you know floating IPs Can be you know can be dealt with correctly Also another another thing to take into account while dealing with the development and new plug-in is What what to do with with the agents so the DHCP agent and and the L3 agent and the metadata proxy agent Do do we want to like reuse those or do we want to rely like on third-party services that are going to That are gonna be used to again to provide services like DHCP and and you know routing and so on There are plugins that like for example the net that the MVP one that Do not use the DHCP one, but use the L3 There are others that use none of them And in ice house for example the MVP plug-in There is work in progress that would basically allow us to Make no longer use of the HCP agent so again this is something that that needs to be taken into account I mean here. There is a flowchart, but obviously it's not the one that you had in mind It's more like more of a comic and then something to laugh on more than anything else But actually that's very serious. That's what pretty much happens every day in open stuff. Yeah, I guess So point of this line being is there are a number of considerations to be made while dealing with the actual like coding of the plug-in I mean the way the code is structured today there is an awful lot of use of class your class mixings and Things like, you know the API model and the DB model are leveraged using the derivation. So inheritance Those are obviously are you know issues that impact the way, you know develop the you know the the actual code, but other considerations relate to how you know what what is the basically the Relationship between the neutron server and and the effectively the effective like plug-in back-end So say for example that that you rely on a controller Things to to be taken into account is how how you make that relationship work work. So Where is where I know do you keep the two data stores in sync? Do you make sure that the resources? Statuses are are kept in sync by using the poll versus a push approach Also in terms of scalability and availability. I mean the way the server is designed Obviously, you can scale it out, you know, you can run it you can run multiple instances of the server behind a lot of balancer But you know what happens to the way you deal with with with the plug-in back-end Well, so these are some of the issues again that that relate to to the design of the plug-in itself But in the end when Okay, you're good So, okay when I meant the back-end, I mean look a specific sample that your plug-in uses some you know some controller That is for me like What I mean as a back-end in the end? Neutral do you don't server lies on a database to store like resources like networks subnets router and so on you have like Databases that lie like in the in the open this which is on the hyperbisers that also represents state and You also have the the data store that belongs to the controller So how you basically make sure that all these resources represent the consistent state This is something that you need to you need to take into account Also when your program when you're programming your your fabric there may be failures So how do you deal with things like again transactionality of the operations that are intrinsically distributed? So obviously these things need to be taken into account to make sure that the game there is consistent eventual consistency at least Yeah, and when we talk about back-end that we are just generally referring to control slash data plane when you ever kind of a Centralized controller. This is somewhat easier even if not trivial. This becomes really important where you are Managing a distributed control plane like a set of switches for instance so, I mean one of the bullet on this light is mentioning unit and functional test so one of the you know biggest problems that We as reviewers face is that when new code is submitted for review We have actually, you know, no No possibility of Trying to go down because maybe we don't you know, we don't have a controller at our disposal We don't have like the actual infrastructure to To try and play with with the new you know with the new plug-in. So again unit and functional tests are Especially unit tests are a good way for us to really understand how things are wired up together And you know how things really, you know work together and some you know Some what it guarantees that there is some you know some integrity in the code There are discussions that we are having we had that the summit and we intend to like work on in the other size time frame To make sure that every like every plug-in submitter not only You know provide code that that's you know covered again, you know by unit and functional test, but also we're thinking of having like Tempest like this that are done by some some sort of downstream CIA infrastructure So, okay There is some some boy break that needs to be like Britain and then the plug-in like sub three But the way the framework is implemented today Quite a bit of stuff, you know can be leveraged. So things like some other mentioned previously, you know Authorization and authentication And the way you basically can provide can implement extensions and way you can add resources or New attributes to existing resources. Actually, this is one of the points on the slide today I mean adding like a new resource to the to the neutron Concepts all model is as easy as riding the Jason dictionary And effectively if you can specify for every new resource, you know What kind of attributes compose their resource? What kind of operations you can do on it was like create, you know the crowd Operations that you can do on their resource and you can also like specify validators And those are all things that again are provided in the core framework and then can be leveraged with very little code Also the way a new plan, you know a new plug-in a third-party plugin leverages that the data model So like the DDB it's by by inheritance, which man, which means that effectively like, you know inherit from from from from the DDB DDB code and you have access to to the DDB schema and you can do operation but obviously A new plug-in may need to rely on new resources or may need to provide new concepts, which means that You not only need to again Use the the core The bee schema But you need to introduce new like the bee tables and so on which means that When you think about developing a new plug-in you also need to provide the beep at migration scripts to introduce these new You know relational seems to be the bee which basically it's done by using tools like a limbic Which is what we use to deal with the bee migrations Any other question You mean obviously So obviously be something that obviously, you know, it's it's local to to the open V switch on the you know on the hypervisor and You know, new drone has no actual knowledge of the state on on on that open V switch if not for like the kind of ports That are then I exist on that node and the state of the board, you know The state of those ports, but we don't have or things like, you know performance counters and so on there is like a Basically, there is a snapshot picture of the important state on that the bee in neutron But there is a neutron has no way to like to get up to get access to that Beyond the open V switch and you know, there is no There is no real intention of like doing something like that for scalability reasons Okay, so I guess we'll you know for the sake of timing. Let's let's go ahead So well these are more like Things that need to be taken into account, you know, when it comes to like the process of contributing a new plugin to to to the new to the to the neutron like I Stream project so obviously there are certain sensor that they need to be met when we will need a fair degree of unit test coverage it would be important to again provide documentation so that users and other developers can can you know can get can get and have a play with with With the new plugin being submit submitted also something that easy the pain of Again getting you know and rolling this stuff out is that stuck so for examples some plugins Do have support in that stuck for automating the like the deployment and the distillation of like third part 30 part third part libraries third part packages and also They provide like Is you know installation scripts to again automating the configuration of computer files? and the automation of like launching the Service instance on so that basically once you get like the whole open stack in a box Deployment you can basically verify that things work the way they should You mean you meant Oslo so I mean okay Oslo is like okay, it's an umbrella Like utility libraries and one of which is like Oslo config I'm not necessarily sure whether I see the relationship with Oslo and then that's okay Basically Oslo our collection of libraries which are of wide interest for the whole open stack project If there is something which is of interest just for neutron that does not make sense put it in Oslo However, as Armando said the implementation of a neutron plug-in is based of a derivation approach So basically you derive your plug-in for the base class which already has plenty of utility methods and functions which you can use for Implementing the plug-in especially when it comes with interactivity database and managing IP allocation for instance That's provided entirely by neutron. It won't make a lot of sense to put it in Oslo since it's not Shaded by all of us that project. Yeah, I mean Oslo. It's more like you know cross-functional Reuse and cross-project reuse So what other aspect to take into you know to take into account is that again provide Functional testing via tempest tempest is like the the API functional test suite for for open stack and You know every project like glance Nova Swift and and so on they Again along with the with the actual you know code that implement features they provide You know test test cases for doing the old you know the end-to-end functional testing Another aspect to take into account again as I mentioned before Smoke start so smoke stack. It's something that it's kind of a you know a cop that Patrol the the open stack Ecosystem so what happens is that every every commit as that's pushed for review. It's validated by by this like this Cop called smoke stack. So the idea is that for providing end-to-end testing We would like to have some sort of CI downstream CI infrastructure to again to validate the changes being made And the reason why we're thinking about see downstream CI infrastructure is because the new vendor plug-in I mean current vendor plugins or new vendor plugins sometimes may need to rely on Resources that may be difficult to visualize. They may difficult to like to scale out. So by Basically giving flexibility in saying, okay, you know Do you know do something like in your lab in your environment to make sure that you can run some sort of functional test suite Based on tempest or based on something else But at the same time they can provide feedback on the upstream review. It will give us like a sense of confidence that what's being submitted there's some you know, it's sound and Functional That said, I mean this this link kind of you know outline The process a little better and I mean we're working in the house time frame to make sure that this process like gets You know gets gets more mature and then more Thorough so I guess that we really know kind of run I'm running out of time and I'd like to move on to the next part Which is probably gonna be the most interesting one so I'm gonna hand back over to Salvatore and I mean if you have any questions, maybe not after this session because you're getting late But definitely like tomorrow we we have a very little time Luckily, we don't need you don't need me for the last part of the session You don't need anybody because it's all everything. It's on github. So the first part is writing a new plugin So on github the link is on the slides. There is the code for these educational plugin Let's say educational plugin. It's actually a real very important plugin. It's the hdm plugin Stop with software defined networking. We will introduce human defined networking So we need to rediscover the human side of information technology stop with automation. We don't need We don't need all these infrastructure automation infrastructure. Who needs cloud? How this losing joy thanks to do to clouds. We are losing jobs We need to rediscover the human face of a T So we have a rest API interface and this list is the tests are Transforming to emails which are sent to networking guy in your it department if you have more than one networking guy you can configure all the emails of your networking guy or if you want you can just even mail it and I am also implementing an extension for supporting phone and fax if you want and It's an asynchronous eventually consistent Request processing so you don't need to implement rubby temp you when it's rubby temp you when you can pick up the phone and It has also karma based requests prior priorization Basically the nicer you are to a it guy the sooner your request will be processed This means for instance if you have a really important request for a provisioning a new network by 12 doughnuts to the it guy and you will do process your request How does that work that's to work you have the rest interface with a little night with a cute resting bear And then there is the message bus which is an email then we'll go to the our back end the human powered the plug-in engine so Implementing the plug-in we have just the core API the moment just for make it simpler actually I've done also the layer 3 extension There is support for networks ports and subnet so we can do routers we can do floating at peace Well other neutral extensions at the moment are outside scope because at the end of the day You really don't need the extension you just pick up the phone or you're just going go meet your it Tell the it guy directly perhaps you don't even need that extension. So where is the code? The source code for the HEDN plug-in is available on github That's the address the slides are available on slide shares. You should be able to access them and so I Tested with Gmail, but I think it should work with all the SMTP servers. So to cut the long story short that's where the code is and Basically all the code that you need is the HDN directory and The HDN directory basically contains a main file, which is the HDN plug-in.py, which is the plug-in implementation I have added comments throughout the file to explain Why there is a particular a particular thing? We can just look at one detail for instance the create a network operation So will on the create a network operation. It's important to note that there is in every operation actually There is always this context variable the context variable is really important if you are implementing a plug-in because it already Contains a lot of information which might be really useful for you. There is the tenant in the context Object you have the tenant ID, which is basically the identifier of the tenant performing the request You have a Flub which is called is admin is underscore admin Which you can use to test if the current tenant has admin privileges or not because sometimes the same Operation is different and perform and perform with that admin privileges or not to be honest with you You should use the is admin only real critical cases because usually you should let the API layer perform or the authorization operations and Finally the context object the context object as a database session object So instead of having to worry about the graph connecting the database grabbing a session for each operation You just rely on that context object, which is a ready to use a database session object So basically the code for this plug-in. I think we could we are kind of out of time So you can just check out on GitHub and it actually works unless the other neutron plug-ins. It sends you an email, you know, and You can what does this plug-in it just per course the base class for performing the database operation puts the status You know puts the since it's asynchronous puts the status of the resource you are creating a impending create status calls the space class to perform the database operation so that the record is entered by the base notifies the notifies the plug-in and Sending an email notifies the back end and then it returns Now there are already two bugs in this operation. You can spot them Raise a file the bugs on github and if you want you can pull requests I mean the two bugs one is regarding rest standards And so it's about the return code and the second bug is about data consistency Because we are not probably ending full tolerance, but that's up to you to discover so you can go and get the github File a bug if you find it if you think you find it and you can also file a pull request with the patch And I would be more than happy to accept it for you actually maintaining this plug-in. Yeah I'm actually maintaining it. Yeah, that's going to become the default neutron plug-in because yeah We are all a little bit sick of all these SDN high-perm thing. Let's talk about human defined networking It's so nice. So I think that's all for me and it's all from us. Actually One minute to spare. We are we still have a few things. I think we have a few minutes for all your questions Sorry, what do you mean? Yeah In the database basically you store the status of the resource in the database as pending create which is not created The paradigm is that you have an admin extension that the it guy will come back and say oh the resources being created And now it's active in a real world where you have this ugly thing called automation This will mean that you are running an asynchronous task when the task completes It calls a callback that updates the database which is pretty much what happens with the open-v-switch plug-in So the ML 2 plugin so that at the moment they are still not properly managing the operational status I think so basically here, you know, you there is a state transition between like, you know It results not being there to just being created an impending state now if the operation would you know where to be Synchronous then before exiting this method you will move from pending create to create and you return the resource, you know to the user in this specific case Supposedly was saying, you know that the operation is asynchronous. So there is you know, you need to make this the state transition outside the scope of the Dismetting like for example, yeah, and it and it mean and it mean Well this I'm pretty sure this will never pass tempest s because tempest s actually need that you can be able to provision a virtual machine With this test and to be able to provision a virtual machine to this test You will need the network right to implement it and unfortunately tempest is a ridiculous time out like four minutes Come on give the guy a break. You can provide all the networking for minutes Yeah, you can wait until tomorrow like or to the after tomorrow that would be normal tempest should wait two days For a test to complete Yeah, I mean you can make this plug-in fast. I mean it also scales horizontally just hire more people Okay, so they're not yet kicking us out. So just in case you have any more question. Otherwise, thanks Thanks for listening. Thanks to