 Hi everybody, I am David Espiner, resource engineer at Orange Labs France and today I'm going to talk to you about how to provide on-demand multi-site connectivity for edge infrastructure tools with Diminit, a distributed model for inter-site networking resources management. So let's begin. So here we have a simple representation of how the backbone of a telecom operator can be seen. We have a few data centers where infrastructure as a service services are proposed using virtual infrastructure managers like OpenStack and we have a series of regional and local point of presence that interconnects different access networks. Today we have needs and requirements from IoT network function virtualization and the 5G that are challenging this traditional centralized approach. In a traditional deployment as the one of the example, there are several problems related to the centralized management. The OpenStack instance by itself will be a single point of failure and a bottleneck. Moreover, some studies have shown the problems of using a single centralized OpenStack entity to manage an distributed infrastructure being the bottleneck, the communication bus. So we need the beam to be closer to the user, in some cases to respect delay constraints, to avoid traffic overhead at the backbone also and in other cases to respect legal requirements where data needs to be stored at a precise site and not at a data center location. For this reason we were interested in a different approach. We asked ourselves what will happen if you use every point of presence as a micro data center deploying one instance of OpenStack per pop with computational and storage capabilities. We referred to this as a distributed IaaS management where resources are managed in a collaborative way among OpenStack instances. This way we can respect the delay constraint, we can avoid the traffic overhead and we can meet the legal requirements. This kind of deployment has been proposed at the edge computing next step in architecture design and testing what paper of the OpenStack edge computing group. But even if this approach could better fit the requirement and constraints, it raises several questions for a management point of view. We focus especially in how to manage and interconnect networking constructions belonging to several independent beams in a distributed native way. So software stacks like OpenStack have been conceived to work in a pretty standalone way, managing a single deployment. So we need a solution capable to come called VOTE, intra and inter pop infrastructure connectivity. And of course we need this connectivity solution to be provided while guaranteeing the scalability, resiliency, locality awareness and automation of the entire system. The scalability means that the infrastructure should not be restricted by design to a certain amount of beams deployed. For the resiliency, all parts of the infrastructure should be able to survive to network partitioning issues. So the isolated sites should remain locally operative. The locality awareness implies that locally created data should remain as local as possible. If you have resources created at the top of the Paris region, you will expect these resources to be stored in the closest location of Paris and not at least for instance. And finally the configuration and instantiation of intersite resources shall be kept as simple as possible to allow the deployment of complex scenarios. So how to provide the networking resources, proposed in the centralized model, but in a distributed way? With this I mean the network, subnetworks, routers, service function chaining and other networking objects proposed by a single neutral instance. So one of the most important challenges is to identify which networking information should be shared among beams. If we take the example of a network and its correspondence of network being shared between the two sites as in the picture, it is necessary to find out which information should be exchanged regarding the IP addresses provided in the segment. One possibility could be to communicate every time an IP is assigned or to logically divide the IP allocation pool among the two sites. So this information will depend on the type of resource because as you know it's not the same thing the verification that we need to do to do routing than the verification to do the extension of a network. So secondly we wanted to define the scope of a request. Should we broadcast information at every site? Should we use a huge replicated database? The idea was to mitigate as much as possible the overhead of management traffic. The tier challenge is related to information availability. In case of network disconnection, each event should be able to provide its service locally. Of course the non-disconnected sites should still propose inter-site resources management. We also considered two technological challenges. The first one is related to the implementation of well-defined, automatized interfaces to allow the communication among beings and the second one is related to the networking technologies actually allowing us to interconnect the network constructions at the data plane level. So to answer these challenges and to provide a distributed management we look at a field that has been heavily studied in the last year, software defined network. If you look at the territorial propositions and industrial solutions there are a lot of SDN controllers used in a distributed fashion. So to take inspiration we analyzed several SDN controllers composed by multiple instances and we found two of these controllers disco and open the light of federation mode that use a physically and logically distributed architecture leveraging a collaboration among controllers. So inspired in these two SDN controllers we designate DEMINET, a distributed model for inter-site networking resources management. So DEMINET relies in a fully distributed architecture where independent models are deployed beside the networking VIM service, in this case neutral. Since VIM are independent among them and also independent among modules new DEMINET instances can join the deployment without affecting others behavior. Thanks to this architecture single point of failure and bottleneck issues are mitigated because we do not depend on a single point of entry to create inter-site resources but instead they can be created at all DEMINET instances. If we do a comparison with the tricycle project when you use a tricycle deployment even to create local elements you must address to the central neutral. In a DEMINET deployment this is not an issue since you communicate directly with every neutron instance. DEMINET instances will also share only the necessary inter-site resource related information because by contacting only the relevant sites for a request we will mitigate network communication overhead. So DEMINET takes really serious the scope of a request. So how DEMINET modules are composed following the approach of the two picket SDN controllers we have at database local to every instance. To communicate among modules every instance has an horizontal interface inspired by the east west interface of SDN controllers and a user face api allowing the user to request the creation of inter-site resources. We have implemented the layer 2 extension and the layer 3 routing resource I will talk more about these resources just in a minute and every module has a logic curve which is in charge of the actual management and coordination among modules so because every type of resource needs a different management a pair resource type logic has been implemented in the logic curve in order to provide the correct management for these resources. So what this really means when a user wants to manage inter-site resources let's take an example of the layer 3 3 routing resource so this is a logical router used to interconnect subnetworks present in independent sites so a request to DEMINET is addressed to any model providing the following information the resource type in this case layer 3 routing possibly a name and a list of network identifiers and the sites where they belong so we're relying keystone to find the information of remote endpoints and we use the remote region's name to locate sites in this example the user addresses to DEMINET 1 of region 1 requesting the creation of a layer 3 routing resource among network A of region 1, network B of region 2 and network C of region 3 so we know that to do routing among subnetworks they need not to overlap so the step number one is to query remote modules and the local neutron about the network subnetwork prefix once the module 1 receives this information it proceeds to do the verification in step 2 and once this is okay internally it creates and store the DEMINET objects in the database and then in step 3 we send an horizontal communication to the two remote modules requesting the creation of a layer 3 resource and announcing that DEMINET 1 is the master of the resource why we did that because we didn't want to pay the price in terms of delay and complexity of a consensus algorithm so we prefer to use a peer resource master to act as the current leader for the resource information this master model also creates a global identifier for the resource that will be announced to remote sites at this point every module has created the local DEMINET objects related to the inter-site resources and we are ready to proceed to the interconnection that I will explain in a few minutes so as you can see a logical inter-site resource is composed by several local sub-resources that exist in every one of the requested sites so now let's take a look to the second kind of resource the layer 2 extension resource in this case the user is able to expand a network and its sub-network to several locations this means that we have a sub-network spanning several open stack deployments and how this is provided so the user will address a request to a DEMINET module with resource type later to extension and providing the identifier of one of the local networks in the case of our example this network A identifier and the list of remote regions where this network would be extended in our example it is region 2 and region 3 in the step number one DEMINET 1 queries its local neutron about network A information to do the management one this information is retrieved in the step 2 the module does an IP allocation pool splitting to provide independent allocation pools to remote regions in this way we do a logical division of the allocation pool so every site can allocate IPs without affecting the other site's behavior in a step 3 the module sends the request to remote modules to create an intersite resource of type later to extension providing the list of regions and identifiers at this point it's it only provides region one network identifier and all the parameters such as the sub-network prefix and IP allocation pool that remote regions will use to create the corresponding network and sub-network at the step 4 each DEMINET will request to its local neutron the creation of a network and sub-network with all the parameters provided by the master module and also to do the corresponding DHCP settings so they will respond with the network identifiers to the master module that at the final step will do an update of the list of sub-resources in his database and will also send the remote request to the modules 2 and 3 to update the intersite resource with all the necessary remote identifiers so while we are sending these identifiers to every site composing the intersite resource what about the way to interconnect all these subjects at the data point level so we didn't want to remain in the wheel so instead we look at all all the possibilities inside the neutron plugins to use the best technology for our module and actually we found that the neutron to neutron interconnection plugin was pretty close to what we needed to interconnect all our deployments once the logical information of the intersite resource is was successfully instantiated so what is neutron interconnections the neutron interconnection plugin proposes to leverage bgp vpn route exchange among two neutron instances by giving to neutron the information of the local and remote resources that will be connected through a bgp vpn so each site will allocate independently their root target identifiers and will exchange them by using any kind of bgp implementation at the overlay level using for instance govgp or at the underlying level using physical equipments so one interconnection object is created for each pair of local resource and remote resource so in the case of a laser tree routing resource composed by three sites at each site will have two interconnection objects for for an interconnection to be in active state the users need to create the initial interconnection and the symmetrical one in the remote neutron so the service plugin will check if the symmetric interconnection exists and once it is verified they will exchange the root target identifiers so let's take a look of how an neutron deployment leveraging the neutron interconnection plugin can be seen so we have the plugin neutron interconnection activated in blue and using the bgp vpn driver we also have the bgp vpn service plugin which uses the backpipe bgp vpn driver and in compute notes we have the backpipe extension activated when we create an interconnection object the service plugin we will do all the necessary calls to the bgp vpn api of neutron to create the bgp vpn object and do an update to attach the local network to it in this case bgp vpn uses a list of import of import and export targets export targets are used to announce the roots of the attached objects and import targets are used to express interest in the routes announced with that target in our example for the network 8 we will use the root target 64 512 to 0.4 thousand to announce its routes and we will import routes announced with the root target 64 512 to 0.6 thousand this is done by using the backpipe extension that with all the necessary rules inside the switch once the different bgp vpn routes are exchanged after that the traffic is encapsulated and sending directly to the remote switch so now let's take a look to our implementation so we implemented dminit as a python server deployed besides neutron and for our proof of concept we used the french testbed grid 5000 using in total 21 depth stack stain deployments with ml2 obs type driver and l2 population and networking backpipe extension in compute notes and for the service plugins we used the neutron interconnection service plugin and the networking bgp vpn service plug for the bgp implementation needed to exchange the routes we used go bgp routing instances deployed as the local gateway routers for the depth stack deployments and to increase the scalability of the bgp route exchange we deployed three route reflectors so the goals of our experiment were to verify that intersite resources were correctly created at each site and to measure the time it took to create each kind of intersite resource among two three four up to 21 different sites so we can say that the distributed approach actually did a pretty good work and we were capable to instantiate the intersite resources in the first campaign of test that we did there was a problem related to the remote module skills so the time started to increase more or less exponentially we fixed that and in the second camp and the second campaign show it as likely better performance most of the time needed for the intersite resources instantiation is consumed by neutron when creating objects or doing requests so in the case of the ledger 2 extension since more exchange are needed with the neutron api it adds extra delay for the total time of creation however even if our model works it does it outside the scope of neutron so we take the presumption that the user will do not further changes directly to the neutron objects as a consequence if our model is implemented as a service plugin some changes will be needed inside neutron code to assure or guarantee that the service plugin is the only capable to touch some created intersite sub resources we also consider implementing more complex networking resources such as intersite service function changes and we are going to optimize our code because the little augmentation of almost 200 milliseconds each time a site is added is a little bizarre so to summarize we have presented diminut a module capable to provide networking connectivity in a multi-deployment open stack scenario by realizing in a fully distributed architecture intersite networking resources are created on demand and exchanging only the necessary information diminishing the quantity of management traffic among instances the diminut contribution actually goes beyond the technical contribution on the network scenario we are investigating how can the diminut proposal be generalized to other services to make them collaborative with minimal efforts in terms of development such a generic pieces of code may represent a huge contribution to the living to deliver building blocks for collaboration between independent systems so if you have questions do not hesitate to ask thank you very much for your attention