 Hello, everyone. So I'm Tom Morin working for Orange Labs. And the topic I will talk to you about today is the question of how we can provide a solution to allow tenants to interconnect multiple open stack deployments. So between data centers in traditional clouds or between NFV pops, you have cases where you will want, as an end user, to interconnect networks or, perhaps, virtual routers, networks or virtual routers with specific properties. So sometimes you will want this to be on demand. Well, in fact, we want always this to be on demand. That can sound obvious. But when you involve multiple open stack deployments, it's not because some solutions can be available to provide interconnections, but requiring the intervention of the admins to build interconnections. You will want some cases to be the interconnections to be on demand and providing private addressing and isolation. That's a pretty common need. These two things, you can do them today with a solution such as a neutral VPN as a service. But behind the scene, it involves using IPsec, which has a significant performance overhead. And you could want a solution that would be both on demand, provide private addressing and isolation, and also avoid the overhead of a packet encryption. So what I'm going to discuss afterwards is a solution with these three properties. So one solution which you can use today involves adding an orchestrator on top of the different clouds in which you have resources that you want to interconnect. So this is something that can be valuable, but it's not always possible or it's not always wanted. There are various reasons. First of all, this orchestrator may need admin rights to set up networking. And this can be complicated to achieve in context where different organizations are involved to handle these different clouds. You may also, well, you will also want for this to be in demand to expose an API to the projects. So that's an additional requirement put on the orchestrator. And whatever you do, you will have a solution which is with the extra complexity brought by this additional orchestrator. So with that in mind, we wanted to explore a solution that would work without having to introduce an orchestrator in the picture. The solution that we came up with, the proposal that we came up with, is a proposal in which we extend Neutron's API with two face sets. The first face set is the user-facing API. It's a simple API that lets a tenant project define that a local resource, network A, for instance, will be interconnected, will need to be interconnected to a remote resource, network B, on another open stack. The way that this API will be used is that the end user, the project here depicted as a monkey, will do symmetrical calls on both open stack clouds. So on one side, you say, I want to connect network A here with network B on the other side. And on the other side, you say, I want to interconnect network B here with the network A on the other side. And then the other face set is introduced. It's a Neutron to Neutron API, which explains the title of the presentation. And this API will allow each Neutron to check if the symmetrical interconnection has been defined on the other side. So here, after call one, well, the symmetric interconnection definition has not been defined on the other side. So the test will fail. But after call two, the test will succeed. And we will be able to proceed. I will explain a bit more afterwards. The rest of the process will be ready for the technical parameters to be exchanged so that an actual interconnection is set up between the two clouds. So the parameters that will be exchanged will depend on the actual technique that you will use to interconnect, that will be used to interconnect the two clouds. But in the end, at the end of the exchange, everything has been exchanged. And each Neutron has been able to do the local work required to set up the interconnection. That can sound a bit abstract because we don't talk here about the technique that's actually used. And it's actually on purpose. And I will explain this a bit afterward. But before jumping on the topic of the actual interconnection technique, it's important to explain why we have this symmetry of API calls. And the reason is that as soon as we talk about multi-tenancy and needs for network isolation, it implies that we address trust questions and security questions. And here, the requirement is that, well, we need to trust the overall system that isolation will be preserved. Said differently, the goal is that no interconnection should be set up unless it was explicitly asked by each tenant. A tenant should not end up with his network resources interconnected with something else that he doesn't know or care about. And how it's done is that the interconnect will be set up only if both sides agrees. And that's the reason for this symmetricity of the definitions. And of the neutron to neutron check, at steps 1, b, and 2, b here, by each neutron, that the other side has received the same symmetrical definition. So the implication behind this is that, of course, each open stack instance has to trust that the other open stack instance will implement the same contract and also has to trust the packets coming from the other side. If you want to propose all that will work in cases where you don't have this trust relationship, well, then you need to rely on something closer to IPsec VPNs, which is not the scope of this presentation. The other thing that you need to care about in terms of security and trust question is the authentication of the API exchanges between the two neutrons. And here the design is simple enough because the only requirement that we have is that each neutron simply needs credentials to talk to the other side. It doesn't need to have right access to the other side. It only needs read-only access to the interconnection info, so it's very scoped and restricted. And for this to be feasible, you just need one user existing on the other side. In practice, to make the configuration comfortable to set up, we believe that the Keystone generation will be interesting to add to the picture, so that in cases where you don't have two clouds but, let's say, a dozen, you don't have to configure too many credentials there. But this is not inherent to the proposal. So let's go back to interconnection techniques. So the reason why the first slide in this talk don't talk much about the details of how things are interconnected is that the design is really agnostic to interconnection techniques. So what I call an interconnection technique is what we end up using, so that packets can actually flow between what we have interconnected with the API. So the goal is here to allow a given technique, let's say, I don't know, GRE tunnel stitching or VXLan stitching, to allow a given technique to be used. We will just need to write a simple driver for it. And in this case, you can see the neutral-neutral API exchange as a simple conduit to carry whatever information is needed to establish an interconnection, whether it is a data plane ID, a VXLan ID, IPs or authentication parameters, and so on. So the design is really agnostic to interconnection techniques. And the one thing that we can say is that if you are in a case of a deployment where more than one technique is supported, we simply need to define how a given technique is selected for a given interconnection. And this will typically be done via configuration, which is pretty straightforward. Use a technique full with CloudX and a technique bar with CloudY. We consider the possibility of introducing a negotiation to the picture so that the thing could be more finely-grained negotiated. But for now, we kept it out of the picture. It's something that we could revisit. But today, we see more benefits in keeping the solution simple. So in the end, for a given technique to be applicable in the context of the solution, you need a technique that will be able to provide isolated network connectivity. The network connectivity could be L2 and or L3 if you have a solution that can do both. Of course, it's interesting. But if you only need L3, well, a solution that only does L3 is fine as well. Interoperability is preferred because it will make a solution applicable between two open stacks that do not necessarily use the same SDN control solution. So examples that we can mention, one of them is VLAN handoff, another is VXLAN gateways. We could also consider leveraging the work done in the networking L2 gateway project. BGP VPNs is another possibility, which I will talk more about this in the next slide. Simple GRE could work as well. It's basically pick your poison. You have to find a solution that matches your needs. And many could work. So I mentioned the BGP VPN as a solution, and then we'll explain more about this one. It's actually a good fit in this context for different reasons. So first of all, you may or may not be familiar with BGP VPNs. BGP VPNs is a technique that's been used for quite a while now, in particular by telcos, but by many network operating teams, to provide a isolated network connectivity over an IP and PLS core. But it's also a technique that's evolved significantly. And in the pastures, it's been extended to do not only IP VPNs, but also EVPNs. It relies on exchanges of BGP routes of a specific type associated with the use of data planning and calculations, such as MPLS or VXLAN. So in this context, this technology has a few strengths. One strength is that, given how the technology is designed, we can have a local allocation of the IDs that will be exchanged. You don't need to have a space of IDs, like data plan IDs, that would have to be coordinated between the different clouds. You can have each side allocates the ID that it will advertise to the other side and that it will use to receive traffic. So this independence provides a good isolation. And in the demo, you will see that the configuration that we need to do to use this is actually pretty short. The other strength is that we actually already have in the Neutron Stadium a project that provides an API to control BGP VPN interconnections. So we can actually write a driver that will be simply reusing this API. And it means that the solution will be usable on day one with all the different SDN controllers that already have a driver for BGP VPNs. So this is neat, in fact. It's really a service composition, one of the holy grails of SDN in some kind of way. Because by just reusing an existing API, we are able to build a new service. A new service compared to BGP VPN. The BGP VPN API alone cannot really provide this service because it does provide the interconnections. But these require admin intervention. And these are not really on demand. Another strength is that you have flexible modalities for deploying this across the one. You can actually have this routing setup as an overlay on top of IP1 connectivity. But you can also have this setup as P-rings with one IPM-PLS BGP VPN routing. Each open stack would be, for instance, a given IPM-PLS autonomous system, BGP autonomous system. And routing would be propagated hop-by-hop. And the traffic could be carried just as plain IPM-PLS VPN traffic over the one. And last, it's a technology that initially was providing IP connectivity. But that's been extended with EVPN for quite a few years now. So you can do both IP and Ethernet interconnects with this solution. So now a demo. So in this demo, well, it's a pretty lightweight demo. We will see two clouds. Well, in fact, these are very simple clouds. Each cloud is actually a Dev Stack. And that's why it's interesting. Neutron using the ML2 OVS standard drivers. The Neutron-Nutron India Connection Service plugin that we actually add to the picture. It's really an implementation of this proposal that we show. It's configured to use the BGP VPN interconnection driver. And for this to be usable, of course, we need to activate the networking BGP VPN service inside Neutron as well. To have the two open stack here with the BGP protocol, we use Go BGP, which is an independent BGP implementation supporting the BGP VPN extensions. We could have used something else. For instance, a free-run routing. There are various implementations. We could have also used the commercial implementations of this stack. There are plenty. And well, to make the demo complete, we have simply a user with an open stack CLI configured to be able to use both clouds in the same environment. So what we will try to achieve, what we will see in this demo actually, is that we will interconnect network A in the Mars open stack with network B on the Pluto open stack. And we will actually check that VM1 can ping VM2 and vice versa, which is kind of if the first one worked, the other one will work as well. So OK. So initially, we simply list the different VMs that we have in this cloud. So we have VM1 and VM2. And we know there are IPs. We can just start pings toward these two IPs. So VM1 has started a ping to VM2 and vice versa. Of course, we haven't done anything to create an interconnection. So nothing is happening much. So next, we just look at the network list on both sides. So here we actually specify on each common line which cloud we are talking to. So we have listed the network list on Cloud Mars. This is just to retrieve the UUIDs. And based on these UUIDs, we will be able to actually start creating the interconnections. So because it's easier, we actually store these UUIDs in shell variables. So it's just to make the rest of the common line readable. But it's not more important. And here is the interesting part. Here, the common line is actually not finished. We are creating on the Mars open stack. So it's the first row on the left. A request for creating an interconnection between local resource network A. And as a remote resource, we specify that we want to connect the network B. So we give the UUID of network B. But we also need to explain which is the remote open stack. And for this purpose, we provide the Keystone API entry point URL. We also have other parameters that are less interesting to explain. We actually, since we interconnect network and we want IP connectivity, we specify that we want an interconnection of type network R3. And we also specify the region that we need to use, which in this case is region 1, the different one, because we only have one. So after this first step, we only have created the interconnection on the one side. The one thing that you can see here is that the interconnection is in a two-validate state. It means that until we do something else, the Mars open stack, since there is no symmetric interconnection, it won't do anything with it. So what's interesting is to create a second interconnection. We can actually check that on the other side, we don't have any interconnection yet. We can check that even a few seconds afterwards, we don't have an interconnection. Sorry, we still have the same state for the interconnection on the Mars side. And now we will create an interconnection on the other side. So here, we are creating the symmetric interconnection. And this time, we are talking to the Pluto open stack. And we are specifying that the local resource is Pluto network B. And that the remote resource is the UID of network A. And we also, of course, specify that the remote keystone this time is Mars. And once we do this, the actual API code returns reasonably fast. But the work behind the scene has not been fully done. So we need to wait, so it's still paused, perhaps. So now the interconnection has been created on the Pluto side. And it's already in state validated, as you can see. Because the Pluto side was able to actually see that the symmetric interconnection was existing on the Mars side. And it was able to already progress towards setting of the interconnection. It actually allocated the BGPVP and identifiers that will be used for this interconnection. But if we look at what's on the Mars side, we see that on the Mars side, the interconnection is not yet fully validated. But a few seconds afterwards, the interconnection is fully validated. And the interconnection is actually set up. We can see the pink flows. So that's the key result of this demo. But what's interesting is also to see that for the solution to be complete, we need to handle the full lifecycle of interconnection. And it means that at some point, we'll destroy a connection on one side or the other. And it's interesting to see what happens. In this case, if we delete the interconnection on the Pluto side, then at once the traffic will stop flowing. But since we can recreate it and we have a properly defined state machine for the state of the interconnections, if we recreate it, it will be the interconnection will be usable again. And I think that's it. I don't have much more to show. That's when you're supposed to applause and so on. You could have asked if it was faked first. So it's interesting to show what happened behind the scene. So it's focusing on this BDPN driver. We like it because it's easy to implement, as I was explaining, with this form of a service composition. But it's not the only one that you could conceive. But still, let me show you how this one worked. So to make this usable, the only thing that we have to configure is to enable this BDPN driver in the configuration. This is the only one that we have today. We have actually another one that's called the dummy driver that doesn't do anything that's used for debugging. And we also configure the ranges of IDs that each side can use. And as you can see, they don't have LAP. You don't have to coordinate the use of the IDs in this setup. So what happens when the symmetricity of the interconnection definition has been confirmed is that each side will advertise to the other side, will provide to the other side the BDPN root target. This is the technical name of the identifiers used for BDPNs that it will use to advertise its own BDPN routes. And to send traffic, the other side will simply import the routes carrying this route target into the relevant network, the network that has been interconnected. And how this is done is that we simply create, the driver simply creates via the existing neutral BGPVN API, the BGPVNs with these IDs, and associate these BGPVNs to the network. So when preparing specification for the proposals and getting into the details, and when starting to work on the code, we noticed that there were a few things that are interesting to detail and actually think that are quite important. First of all, as I was saying, we need to handle the lifecycle of interconnections correctly. It means that when an interconnection is deleted on the one side, we need to actually cheer it down and let the other side know that the interconnection is in a different state. It's not ready anymore. So we pretty quickly came to the conclusion that we needed an explicit state machine, explicit and robust. We also understood that we needed to have something that would be robust, including in the case where the neutral on the other side is not reachable. So we need things to converge again to a usable state when it comes back up or when the connectivity to it comes back up. So we need some form of a paradigmatized. This is pretty obvious, but still useful to mention. And there's something that we need to get right as well, is that the work to actually handle the exchanges between the two neutrons need to happen behind the scene. The typical contract when we build REST APIs is that these REST APIs should return instantly and successfully as soon as the request that was made is, let's say, legitimate. And if there is work to do behind the scene, if there is more work to do, typically when it requires interconnections, when it requires exchanges with backends, it would need to be done offline behind the scene. And to actually do this right and be able to handle concurrency between API calls and the behind the scene work, we needed to have a proper locking so that two workers doing behind the scene work do not work on the same interconnection at the same time or to avoid the case where you have an API call, for instance, to delete an interconnection and at the same time behind the scene work to actually set it up. So we actually introduced into the design interminged states in the state machine, which actually act as locks. So we end up with a state machine that's not overly complex, but still complex enough. And in the end, we have to realize that what we are doing is actually a global state distribution. But we think we are managing to keep it simple, maybe not stupid, but well, simple enough, because the local state machine is a typo in there. The local state machine doesn't need to know the state of the other state machine. And the interactions between the two clouds is restricted to simple operations, refresh, that is, read-only. Refresh and get that are both read-only. So that is stuff that you care about if you want to know more about the project and get to contributing to it. But as a user, you shouldn't need to know much about this. So one thing that is interesting to discuss is that, well, with this proposed solution, there are a few things that the end user still needs to take care about on its own. One thing is that the end user have to choose IP address ranges and pools consistently across clouds. There's nothing in this solution preventing a user from using the same IP twice on both clouds. And of course, if it does, well, you won't have connectivity between the overlapping IPs. The other thing that is not covered is security groups. What's convenient with security groups is to say, well, I want this security group to have connectivity to port 80 to the other security group. That's the convenient way of using security groups. But for this to work in such a scenario involving multiple clouds or multiple regions, we would need one cloud to know about which IP is a member of a security group on the other side. And for now, this proposition doesn't cover this. So what we are left with is the solution where a user will say, this security group will have connectivity to port 80 on these IP addresses. And it will have to give these IP addresses via the APIs. So more work will have to be done by end users. It is acceptable in particular because it can be automated in, for instance, a hit template, sensible paybooks, or, I don't know, terraform. But still, it's not fantastic. So what we have in mind is keep this topic 29 and see if we could prevent users from shooting themselves in the foot with overlapping IP addresses, or both, ideally, be able to make security groups work seamlessly across clouds. But for this to work, we need to have a mechanism where we distribute security group membership between clouds and regions. So before concluding, I will just give simple examples where we believe that the solution can be interesting. The use case that I used when explaining this proposal is the use case where we have two open stacks. So as I was explaining earlier, the solution is relevant in the context where we have two open stack clouds between entities that have a trust relationship between one another. This is the only prerequisite, but it's a significant one. We can actually see another case where this proposal can be relevant. It's the case where you have multiple regions of a single cloud where you need to do such interconnections. This is a pretty frequent case, sorry. Something that I didn't really mention or that's worth insisting on, at least, is that the solution, even though we explain it with two clouds, generalize easily with more than one. In this case, the end user will have to talk to more than two clouds. But in the end, the exchanges between pairs of neutrons will be the same. And the other case that's interesting to mention is that it's also a solution that will allow to address cases where we have some form of heterogeneity between the different clouds in terms of which SDN control solution is used. So typically, in some cases, you want to interconnect the clouds or regions, maybe. That, for design reasons or historical reasons, use different SDN controllers. Or you may want to migrate from one SDN control solution to another. For instance, you need to move from A to B and you want to preserve connectivity for workloads until you have fully phased out SDN A. And in this case, you could rely on this solution to actually create this connectivity between the different regions or between the different open stack using different controllers. So implementation status. So the specs were proposed in last spring. They were merged a few months afterwards in the Neutron Specs repo. If you want to have a look at the specs, they are linked right here. The project to actually develop the solution was recently created under the Neutron umbrella. So it's an official Neutron Stadium project. We plan to start contributing code and doing reviews very soon. So if you're interested, it's really the right time to jump in, because we will actively work on the actual code behind this in the next few weeks. So as a wrap up, so it's a solution that's meant to allow on-demand interconnections without the need for an accelerator in the picture and without some of the drawbacks of solutions of such as a Neutron deep end service in terms of a data plane overhead. It's meant to be relevant when you have two or more open stack instances, whether these are different open stack clouds or different regions of one cloud, including in the case where they use a different SDN solution. The first driver that we will work on is a driver relying on a BDP VPNs. And it will allow the solution to be usable with an interesting set of SDN control solutions, including Neutron, on day one. If BDP VPN is not a good fit for you, well, since the solution is agnostic, drivers for other solutions can be developed. And we will actually welcome it greatly. The next steps, while as I was saying, are code submission and reviews in the context of the open stack Neutron interconnection project. And perhaps also, we'll find the time to actually do demos with heterogeneous SDN controllers. It's just a matter of deploying them and tweaking them. As I was saying, the code is not specific to an SDN controller. And before concluding, I want to thank Yannick Thomas and Przemek Yadek, who are colleagues who have greatly contributed to the maturing and development of this proposal, including the code that's behind the demo that you've seen. I'm done. Perhaps there are questions in the room. Yeah, you've got a mic over there and another one. Yeah, hi. I've got a pretty simple question, but still. So is it possible to interconnect a bunch of networks using one interconnection just to share not two networks? If I have several networks inside one cloud and they want to interconnect them to the several networks in the other cloud, is it possible, yeah? Oh, yeah, yeah, sorry. I should actually have mentioned it. Yeah, so it works with two cloud, but it also works inside one cloud between different tenants. Is this what you mean? Or even for one tenant, if you want to interconnection without relying on Neutron routers, you could use it. But in the case of one tenant, it's not the only solution. But between tenants of a given region or of a given cloud, it's something that's usable. Yeah, I meant one tenant, but this one tenant have several networks just on one cloud and several networks on another cloud. And all of these networks should be reachable between these clouds. OK, so you want to interconnect network A1 and A2 on cloud Mars with network B1 and B2 on cloud Pluto? Yeah, yeah, yeah, just like four networks like in. Yeah, so you will have as many interconnections to define, but yes, it would work. Yeah. Hello, first, thank you for working on this. This looks very promising. Thank you. In what form is the proposed code? Is it just an ML2 plug-in, or do we need to modify Neutron's code itself? Yeah, so like many extensions to the base Neutron API, it comes as the form of a Neutron service plug-in, just like networking SFC, networking BGVPN, or Neutron VPN as a service, for instance. So it's a Python module that has to be present on the Neutron controller. And you just declare in Neutron configuration that you will enable this service plug-in. So to be usable, you need a deployment where Neutron has been packaged to include this Python module. But then you don't need any modification to Neutron itself? No, Neutron has had a plug-in architecture for a while, and it's wonderful. So I've seen the Go BGP speaker there. Have you added support within networking BGVPN, or you have configured the pairing manually between both instances? Yeah, so in this demo, we used the reference driver for networking BGVPN, networking backpipe. So it has its own lightweight, I shouldn't say else, but it uses the exact BGP implementation. So we just configure it to peer with Go BGP locally, and we'll use Go BGP just to do the pairing between one cloud and the other, so that there is no Go BGP-specific thing in Neutron or the Neutron agent. Thanks. Hi. In GVN Trality, you want to interconnect a lot of open stacks together. Is that possible to establish something like a hub and spoke architecture with root registry version in a middle open stack or something like this? That's a good point. So first of all, the answer would depend on the technique being used. If you take the specific example of BGP-VPNs, it would then depend on the backend and how the route that you import are available for traffic that you receive. I don't know if you follow me. So the answer is kind of a solution specific. You could do it. But yeah, in terms of API contract, it's probably not in the scope of this API, or we would have to modify it to make it clear in the contract, because in the contract, when you define a connection, you define it between two networks, you're not saying that you want this interconnection to be transitive to other networks connected to the one in the middle. So to make this clean, you would have to evolve the API to say that, yeah, wow, I want this interconnection, but I would like connectivity to be transitive, so. Hi, so you have to give the credentials of Cloud B to users of Cloud A, right? Not users. Do you only need a neutron to have credentials for one user on the other side that will have read access to interconnection objects? Only to the user. So it would typically be a user which would be in a service tenant which would have a role scoped to only be able to read interconnections if you want to really secure things. So you can have the credential so that users are written in the configuration file. If you have more than one cloud, sorry, more than two clouds, and you don't want to have a complex configuration, these credentials could be credentials to a user that would be handled in a federated keystone. Got it, thank you. Okay. Okay, thank you. Thank you very much.