 Hello, everybody. Thank you for attending this presentation about Pushing Open Sack to the Edge, a duel between Rabbit and Q and Cupid dispatch router. So who's going to win? Stay tuned. So who's here? Well, who am I? I'm just 28, I'm still figuring out. My name is Alisson Agi, I'm a research engineer at Orange, France in the beautiful region of Brittany. And please have your. Hello, everybody. I'm Javier Rojas-Baldirama. I'm working at INRIAD Research Lab in computer science in France along with Matthew that couldn't make it today. So what are we going to see today? Well, we are going to see what are the challenges of bringing Open Sack to the edge. We are going to show you how deployments of Rabbit and Q and Cupid dispatch router for Open Sack are made over the one. Then we are going to show you our results of our performance evaluation of both message buses in this context. And we will leave you with some conclusions from our work. And the next steps we need to see. So what's up with the Edge? Well, a telco usually has a big network. When you are in the core, you might see big data centers where you have big resources. But the more you go outside the network at the edges, the less smaller resources you have per site, but the more sites you have. So the problem scalability comes very quickly. You also have the problem of locality, meaning that if you have two edge sites that want to communicate one with the other, well, you don't want them to go way up to the core. And back to the Edge. So these are the two main problems we are going to address in this presentation. Of course, there are other problems, but we will focus on these two. So for a telco like Orange, Pushing Open Sack to the Edge is a key why. Well, a telco can always deploy Open Sack like any other cloud provider in big data centers. It works very well. But should it do just that? Well, we don't think so. Why? Because it has the privilege of being the closest as possible to its customers. So it should leverage this privilege and use its network to deploy and deploy Open Sack on the Edge. But how do we deploy Open Sack on the Edge? Well, first we can think of having per site the control plane and compute nodes. But you will have too many control planes to manage. You will need to synchronize between them and all that might be very costly. So the deployment we are considering in this presentation is when you have your control plane, your APIs, your front-ends centralized, and on the Edge nodes you have your compute nodes. And only your compute nodes, which are connected remotely to your control plane, which is centralized. So we can take advantage of Open Sack's native scalability. All processes, almost all the processes are stateless. We have still the database and the message bus to see how they respond to distributed context. And we need to see how Open Sack behaves overall over one. So the deployment we are considering here is where you have your control plane centralized. You have Keystone. You have control parts of Nova and Neutron. And on the Edge sites, you have your compute nodes, meaning the agent sites of Nova and Neutron. So Nova compute and Neutron agent, whether it's Open V Switch, Linux Bridge, 007. And between your compute nodes and your control sites, you have a wide area network connecting them. So what kind of traffic do we see between the compute nodes and the control nodes? Well, you have RPC traffic. For example, Nova compute sending information to Nova conductor. And you have REST API calls, for example, from Nova to Glance or from Nova to Neutron. But we are going to focus in this presentation about the RPC traffic, which goes through the message bus. So what's up with the message bus? Well, it is one of the most critical components of Open Sack because it carries all the inter-process communication. So if Nova wants to communicate with Nova conductor, it goes to the message bus. If Nova conductor wants to know where's the schedule of VM, it asks Nova scheduler over the message bus, et cetera. You have also a Neutron server, which communicates with Neutron agents through the message bus. So these two message bus can be the same. But theoretically, they are separate. They are decorrelated. So they can be two separate buses. But what are the kinds of messages we see? Well, we have RPCs, remote procedure calls. There are some kinds we see. The RPC call, which is a request from a client to a server. So basically, it's client saying to the server, I want something. And the server responds, here you are. You have also the RPC cast, which is the client just sending some information to the server saying, here I have some news for you. And the server doesn't even say thank you. And then the fan out is where you have the client sending a message, broadcasting message to a bunch of servers. So it's the client saying to everybody, hey, listen, but without being annoying. So of course, here the client on the server can be, all the processes can play the client on the server part. So for example, Nova Conductor can be client to Nova Compute. For example, when sending boot request and Nova Compute can be client to Nova Conductor when sending information about to the VMs, running on the compute node. So more precisely, the message bus is, first of all, hostile messaging, which is an open stack library for sending RPCs. And there are multiple drivers under hostile messaging. The one most of the open stack users know is RabbitMQ. It's an implementation of MQP 0.9. So it's a broker. The client connects to the broker. It sends a message of the broker, stores the message in a queue, so it stores the messages. It has a message retention. Then the server comes and gets the, consumes the messages from the broker, from the queues. There are other implementations and drivers under hostile messaging. But the one we are considering here today is QP dispatch router. It's an implementation of MQP 1.0. MQP 1.0 is not a direct evolution of 0.9. It's another protocol. So it's a routing protocol. It's a QP dispatch router, as it seems to just, it's a router. So we have a router topology. The client sends a message to a router and the QP dispatch router figures out which is the best way, the shortest way, using a shortest path algorithm to reach the server. And it just gets the message from here and sends it there. There is no message retention, just buffers. It gets the message and passes them there. That's what we call message passing. So why are we interested in QP, the QP dispatch router? Well, if we get back to our edge topology, you have a central site. You have regional sites and under that you have small edge sites. You might have your clients, your processes, clients and servers on these sites. When you are using a broker, even if it's a cluster, it's basically one entity, one chunk of software. So you might have it in your central site. So if client three wants to communicate to the server three on the bottom left, they have to go way up to the central site to connect to the broker. So that is not very effective. But if you are using QP dispatch router, which is a router topology, you might have a router instance on every one of your sites. So if client three wants to communicate with server three, it just has to stay in edge site one. If client one wants to communicate with server four on the bottom right, they don't have to go above regional site two. So it routes messages. So you see why we are interested in QP dispatch router in this context. We are not going to see in our performance evaluation this deployment to our focus in on, as we said earlier, on centralized, centralized topology control plane. But it is very interesting in the edge context. So what are we going to see in this presentation, which is the goal of this presentation, well to evaluate the performance of a rabid QP dispatch router over one. So does it resist one constraints and hazards, for example, packet loss, latency, and network dropouts where you have compute nodes disconnected from the control plane, and reachable messages pop out like crazy. Does a router fit better, does it really fit better a decentralized environment in open stack, do the messages in the open stack still resilient, even if the router doesn't retain messages, isn't a broker maybe a safer option because it retains messages. So we need to see all of that. And also, we need to know how overall RPC communications, whether it's rabid MQ or QP dispatch router behave over one. So because what can go wrong in one? So for example, you might have RPC client disconnected from or you might have loss between the RPC client and the broker or the router. So the Nova conductor message to boot a VM to Nova compute is lost in wilderness. You might have a problem between the broker and the server. If you have a latency, so for example, for RPC calls, it's a request and response so you get at least twice the latency. If you have an RPC cast, well, QP dispatch router just gets the message and sends it. So it does well, but rabid MQ waits for a kind of an act so you still have twice the latency. So all hell can break loose over one. But let's get enough of speculations and let's see what the experiments say. And I'll leave you with Javier for this. Thank you. Thank you, Ali. So right now I will present part of the results we described in the test plan of massively distributed RPCs that are proposed to the OpenStack community and available in this link. We organized our test in two categories, synthetic where we only test the bus, rabid MQ or QDR and decentralized configuration and operational test with OpenStack in a centralized way like Ali described it previously. For the operational experiments, we have two use cases, the network dropout and latency loss. So for all our tests, we use a tool with a homemade developed called Enos for easing the deployment of OpenStack in virtualization of Bermuda that is a layer over cola that eases the deployment of OpenStack and to give you an idea, we can deploy 400 computers in one hour with this tool. All the experiments were performed on a great 5,000, a dedicated testbed for experimentation that is located in France. In Vancouver this year, we presented the results of the synthetic experiments. So we worked with three patterns of RPC implementations that are available on OpenStack and we identified that brokers and routers are similar even though we identified the routers are lightweight and achieves latency message delivery especially under high load. Also we identified that routers offers locality of the message in the centralized deployments. However, to fully deploy the OpenStack in the centralized way, we also have to work with the API and the database. If you want more details about this presentation, please refer to the presentation on the website of OpenStack. To give you an idea of these results, here I will present a summary, partial summary of the synthetic results where we can observe some metrics for three configurations, one bus, three buses or five buses with a high load of messages going from 1,000 clients to 10,000 clients. We can observe that in terms of memory, for example, one broker uses at least 7,000 megabytes of RAM compared to only 500 megabytes in the case of router. In terms of course, to use 1,000 messages, in the same way the brokers use at least 24 cores compared to only one router. If we push this experiment a little bit and we work with 10,000 agents, we can observe that in terms of memory, the routers consume 4,000 megabytes of RAM compared to only 5,000 of the router. It means we have a ratio going from 9 to 17 between the use of memory between brokers and routers. And in terms of course used for this same experiment, we have a ratio between the broker and router going from 8 to 27. For synthesis, we can see here a group of box plots on this side from one bus, Rabbit queue and Cupid dispatch, where the latency is higher in all the cases. This is verified for the configurations of three Rabbit n queues or three routers or five Rabbit n queues and five routers. If you have more details about these experiments, please download the document that is linked in this page. So let's go now to the operational experiments we performed with OpenStack. In this configuration, we deployed on the core node three control nodes, one network node in the core site, and in the opposite of the one, we have the edge nodes going from in configuration of 100 nodes and 400 nodes. We deployed OpenStack in a visualization mode on 20 physical nodes containing each one 32 cores and 192 gigabytes of RAM. So for the network dropout, we use EP tables to generate the dropout with a frequency going from 5 to 10 minutes and a duration of this dropout from 30, 60, and 120 seconds. We performed 10 benchmarks of rally with a fixed duration of 30 minutes. And after each combination of parameters, we redeployed it completely OpenStack. So the first thing we can observe here is that in general OpenStack managed pretty well the dropout. We can see the bars, the clear bars corresponding to the dropouts, and on top we can observe the results of Rabbit n queues and down the results of QDR for the case of boot and delete servers. However, we can identify three kinds of errors, mostly on this side for QDR. First on top of the Y axis are errors related exclusively to rally and the timeout of rally. It means rally couldn't identify the status of the virtual machine after the timeout. The second category include that we can say more natural errors in case of dropouts. In case of dropouts, when we started an action of boot and delete server, then the communication is broken and OpenStack identifies this error and we cannot have a successful benchmark. Finally, we have, especially in this case, some errors down next to the X axis that are associated to the computers. Computers are not reachable because of the dropout, so they are not available for schedule. If we are more aggressive with this, we observe the same results for a frequency of 300 seconds. Another interesting thing to show is that these results are very predictable. In case of, again, the Rabbit n queues apparently works better than QPD dispatch because of the retention of messages because routers don't have this feature. Because they don't have this feature. Now concerning the latency and loss, we use three parameters, latency, loss, and compute deployed on the infrastructure. So for latency, latency is going from 5 to 200 seconds and loss going from no loss to 2%. And with two configurations of 100 and 400 computers, again, redeploying OpenStack all the time after each combination of parameters. There are two situations. The first situations where apparently we can expect a result that we can interpret. Like is the case for the Nova, for benchmark of Nova, for server boot and associated float and IPs where we can identify the direct impact of loss and latency for the duration of the benchmark. And this effect is equal for 100 computers or 400 computers. The differences when we increase the latency and loss is associated to the complex traffic of messages between the service of Nova and Neutron and eventually case time. On the other side, we have a different behavior for cases of Neutron benchmark, benchmark like is the case of create and delete subnets. It's difficult to interpret these kind of results because Rally cannot provide all the information we require. Rally, the only thing that evaluates is the response time between the Neutron API and execution. But behind these scenes, there are a lot of things that are happening, especially in terms of traffic messages. To give an idea, we have here the timeline of a set of benchmarks for multicast in a 400 computer configuration where we can observe that the duration of the benchmark is different for each test. We observe yellow bars for each test and the number of messages in terms of traffic are also different. Easily we can go up to 1,000 messages for some cases like the boot server and attach interface. But there are some extreme cases like create and delete port where we can observe more than 2,000 messages per second. The thing is, during the benchmark, there are a lot of messages that are going from client, from the core and the age and between all the services. In this case, for example, for the multicast queries, we can observe that the conductor for the case of NOVA sends a lot of messages. And in case of the fan out queues in the buses, we can observe that Neutron uses a lot of fan outs. It means that there are potential reasons, actually there are reasonable reasons of doing that. One is we want the update of the status of some resources as quick as possible. And another reason to send fan outs to everybody is to avoid, to pollute, request the database to update the resources. However, these fan outs are not consumable for most of clients. So in these situations, we can reduce the traffic of messages in a distributed deployment. For that, more experiments are going on and we will conclude the... So what do we conclude from this study? Well, that in front of one latency and loss, Cupid dispatch router, which has no message retention, still does as well as a rabbit in queue, which has a message retention. Of course, we've seen that in case of no truck network dropouts, we have more errors because of the same setup because of message retention. But overall, Cupid dispatch router consumes way less resources than rabbit in queue and still does basically the same job. So what are the next steps we need to see? Well, we deployed QDR Cupid dispatch router in centralized, which is not using its router capabilities. So we would like to see how it behaves when we really use it as a router and deploy it in near compute nodes, for example, on edge sites. We would also want to see a bigger scale of computes, so not the hundreds, but the thousands. And finally, we think of how open stack might be if we had more components which are distributed. So we've seen that we can distribute, for example, the message bus, which is about database. It's very interesting, too. So if you have one message to take away from this presentation, is that Cupid dispatch router is a real, real good alternative to rabbit in queue. So get it, try it on your deployments, and let us know what you think about it. This is it. Thank you for your attention. And if you have any questions, well, we are here to at least listen to them. No questions? Is there a dependency between the size of router mesh, which you used, and the performance of the cluster? I mean, does it mean how many routers run? We've run three instances of Cupid dispatch router, because we had also three instances of rabbit in queue. So it's kind of clustered, yeah, it's clustered. I don't know if I responded to the question. Just to complete the answer, during the synthetic experiments, we see that the effect of going beyond a configuration of three buses, the results are not very different. And it's a realistic deployment using three, only three. I would like to ask, did you have any problems with Cupid integration and open stack services? I think rabbit in queue is the preferred one, and more supported one. And the second question is, do you also recommend Cupid router for more centralized deployments? So I'm going to respond. So yes, we had, we are using call ansible to deploy open stack. So we rely on, we relied on the container of Cupid dispatch router. There were some problems with it. So I think Mathieu addressed that, so maybe if he can answer better than me, the question about the problem facing deploying Cupid. And the other question is, when you are using, sorry, can you repeat the question? If you use it in a more centralized way, would you still recommend it? Yes, we would recommend it just because of the resource consumption. Because it does as well as rabbit in queue, and it still consumes way less resources in terms of CPU and RAM. So maybe you can explain what our... To give you an idea, the case of the synthetic experiments, we also verify the consumption of rabbit in queue compared to Cupid dispatch is bigger. So it's a really good alternative to that if you have some constraints about consumption in terms of CPU and memory. And for the deployment, it was mostly associated to the older version that is packaged out of the box with the containers. But when you update, you don't have any difference working with rabbit in queue or Cupid. Thank you. Sorry, another question. You know, with rabbit in queue, there are certain operational patterns you should use when you, for example, restart rabbit in queue. If you have, say, three rabbit in queue, and there are mirrored queues, then you should not just randomly kill one of your rabbit in queue. You should restart them in order. So from an operational standpoint, what do you think about Cupid routers? Are there certain requirements to restarting, starting, stopping Cupid routers? Is certain orchestration required to operate Cupid? So, specifically, I don't have anything in mind right now. So maybe if I, we can get after this presentation and maybe speak more in detail about this. Okay, thank you. No more questions? Thank you very much. Yes, thank you.