 Hello. So I think it's about the time to start. So hello everyone. So my name is Li Ma and I'm from AW Cloud. So I think most of you don't know AW Cloud. So we are a pure OpenStack player in China from 2012. And we help our enterprise customers to design, build and operate OpenStack clouds. So I'm the architect in this company and I will study with the focus on software defined networking and distributed systems. So today I'm going to talk about messaging and the RPC messaging patterns in OpenStack and why not RabbitMQ are occupied for us. And I will introduce an asynchronous messaging library called ZRMQ and also the distributed messaging driver in OpenStack, which is developed based upon ZRMQ. So finally, I will introduce some real world deployments that we have been done. So RPC messaging. As most of you guys know, it is a very critical component in any single OpenStack installation. So when your messaging system has problems and when it doesn't work with you and it doesn't scale with you as you continue to grow, so you may run into major issues with the rest of your OpenStack installation. So the other reason why is because most of the OpenStack systems depends on messaging to be reliable and to work with you all the time. So if your messaging system has problems, for example, like compute and conducts and all neutral agents if your messaging system has problems, all these systems is to work properly for us. So you can see that RPC messaging is very critical for us. So when we set out to deploy OpenStack in 2012, we chose RabbitMQ as it is official reference. But however, we found numerous problems with reliability, resiliency, and speed. So I know it's not just us have RPC issues. There are various topics that have been presented in every submit, so you can review them from YouTube and I'm not going to discuss with them in detail. So in conclusion, we really faced these problems. So when we deployed OpenStack in 2012, and our OpenStack distribution is based on PySX, and we deployed a single RabbitMQ instance at the early stage, and we relied on monitoring and operations to make it stable. And as time went by, we switched to cloud stream technology. But however, problems were still there because the deployment became much more complicated for us. So for example, RabbitMQ itself is a black box for us, really. And moreover, when you deploy in a production system, you need to deploy HAProxy as an external heartbeat method along with SpaceMaker, which makes everything highly available. And you also need to identify the ratio of the RAM nodes and disk nodes without any best practice guide. So moreover, we also kept pace with the community to continue to upgrade our RPC libraries to get rid of these bugs, which is reported in Launchpad. And finally, the monitoring and operations are still needed to check if the whole cluster is overloaded or malfunctioned. So the bottom line is here. We found that scouting a broker is not practical for us because we have a lot of customers and partners in the ITC industry. So they need to start small and scale online. So scouting a broker is really difficult. So we need to find some other solutions available for us. So let's review our requirements for RPC messaging. So firstly, there should be no single partner failure. And it should be something that be horizontally scouting. And it also should be reliable. So finally, we found that the RAM QDriver is the answer and it met all of our requirements. So in 2014, we launched our OpenStack distribution based upon S-House. And we totally switched to the RAM QDriver in OpenStack. And it has a distributed message architecture. It is horizontally scouting. And it has fast delivery. But unfortunately, in S-House, some of the objects don't switch to Oslo.messaging. So we need to develop it for each project. And also in this year, we stick on the RAM QDriver. And also, we have some improvements on performance and stability. And also, the RPC library totally switched to Oslo.messaging. So the RAM QDriver means zero broker, zero latency, and zero cost. So the RAM QDriver is a high-performance asynchronous messaging library and not used in scalable distributed or concurrent applications. So it really provides a messaging queue, but without a dedicated message broker. So the RAM Q library is considered as an intelligent socket library, which gives you sockets and that carries atomic messages through various transports like in-process, inter-process, TCP, multicast group communication. So it doesn't have these rigid defined structures like MQP does, like queues and exchangers. Instead, it has a single interface. A single socket interface is like bind, connect, send, and receive. So it is clear that anyone who is familiar with socket programming is likely to work with Zero Pro very well. And the most powerful stuff is the patents. So Zero MQ patents is implemented through a pair of sockets that has matching types like request and reply. It is a classical request and reply patents. It is synchronous communication because one client sends a request, it is blocked immediately, and until it receives the reply from the server. So in Zero MQ, it doesn't rely on the sequence of service that, and it is usually used in RPC and CSCon applications. So and the next pattern is pub and sub. So it is a published and a subscribe model and it is asynchronous. When pub socket sends a message, it returns immediately and the IO threads at the background does the rest of the things. And the pub socket cannot receive and sub socket cannot send. And the next pattern is push and pull. So it is a pipeline pattern and it is also asynchronous. So it implements a fair queueing to sort the request from the upstream. So and the Zero MQ also has some advanced socket patterns like dealer and router, x pub, x sub and you can also define your own. So I'm not to expand this part and you can review Zero MQ guide in this official upside. So here, what has Zero MQ brought to us? So firstly, Zero MQ is a messaging library. It is not a messaging system. So it allows us to define our own messaging models. So finally, we found that Zero MQ library can bring brokerage architecture. So it becomes possible to have a messaging layer for OpenStack without an essential broker. So let's see the Zero MQ driver in OpenStack. So it was introduced in Fossum. So really it has been there for years. But I know some of you don't know Zero MQ driver because it lacks of test coverage. So it is mostly unmountable in upstream for years. So there is a summit topic taken in Hong Kong summit going broccoli as the transition from QPL to Zero MQ which is presented by Bluehost. It is a huge service provider in United States. And actually I was on spot in Hong Kong. So I had a conversation with engineers in Bluehost after the, in the summit. So and at that time I had a feeling that Zero MQ driver is really worth trying. So when I returned from Hong Kong and we started to experiment and do some evaluations and it really great. So finally in upstream it has been switched to Oslo.messaging. So actually theoretically all the OpenStack projects can take advantage of this driver. So as I stated before, Zero MQ driver implements blockless architecture for OpenStack RPC. So let's see how it works. Firstly I will introduce messaging topology. Zero MQ driver has a distinct message topology with traditional messaging systems. So let's see the left part. It is a start topology which brought by a broker solution. So all the messengers needs to went through the central broker when some OpenStack components needs to communicate with others. And the left side is Zero MQ. Zero MQ is considered as a peer to peer distributed system. It forms a partially connected match. So you can see that any OpenStack components can directly send a message to the final destinations. So there is no central broker. So theoretically it can scale out very well. So let's see some components in the Zero MQ driver first RPC client. So here are five OpenStack services which are running in five hosts. So each OpenStack service can consider it as an RPC client. So it sends messengers through a message queue to the others. So the next component is Zero MQ receiver. It is a new component which brought by Zero MQ driver. So it is a distributed broker running on each host. So let's see how it works. So here's still five OpenStack components and each with Zero MQ receiver. So there's no single broker and the messengers directly send from OpenStack services to another. So here Zero MQ receiver receives the message from RPC client via TCP and then it redistributes messengers to local processes via IPC. So that each receiver managers all the messengers within the local host. So you can see that here are the five hosts and each have a Zero MQ receiver. So when the message goes through from OpenStack service to another from OpenStack service to the destination Zero MQ receiver, then there will be the redistributing message to the destination OpenStack service. So let's see in a single host. Here is the Zero MQ receiver. When it receives the message, then it manages the set of IPC channels. Then the messengers, the messengers topic is scheduler. So it is going through the IPC channels to the Nova scheduler. So it is inside a single host. So that's why I say that it is a distributed broker running on each host. And the final component is matchmaker. It implements the out-to-host discovery. It is designed for bare topic. So in OpenStack, there are two, generally there are two types of message topic. The first one is direct topic. So you can see it is the topic type. Here is the compute means that this message will send to the Nova compute host. And there's a dot, here is the host name, host one. It means so I know that I will send this message to the Nova computer, which will set on the host one. So it is a direct topic. And then the next is the bare topic. It's like you can in OpenStack, there are the find out messengers. So when you send a find out messengers, it doesn't specify any destinations because so I need to broadcast this message to all the Nova computes. So the matchmaker in Xerium Q implements was implemented to support bare topic. So let's see. So here is a Neutron server. And here we have two Neutron agents on each host. So we have each has a Xerium Q receiver. So here we have a matchmaker server. So when Neutron agents agent restart, it automatically adjusts its host name to the matchmaker. So it has the host name ALB agent one, has the host name ALB agent two. So when Neutron server need to find out a message with this topic, and it initially it goes to matchmaker to ask. Here I need to send a message. So where should I send to? So the matchmaker returns a list of host names of the Neutron agents. So then the Neutron server send the messengers one by one. So it simulates broadcast. So the matchmaker in Xerium Q driver is implemented based upon a radius server. So in radius, the key value is topic and host name. So you can see that I send a topic and Neutron server can find where I should go, where the messengers should go. So here is a typical deployment scenario. So we have a controller which runs Nova Neutron Cilometer Keystone Glance Horizon and Database. So we need to start a new process called Oslo Xerium Q receiver. And it manages all the messengers related with this process. So I need to run a radius server in our network. And also on computer node, there's Nova, maybe Nova compute, and Neutron L2 agents and a Cilometer computer agents. And you also need to run Xerium Q receiver to manage the messengers inside the host. And also on network node, there are Neutron agents so you also need to run Xerium Q receiver. So there's no any central brokers like RabbitMQ resizing this network. So all the local brokers are Xerium Q receiver. So Xerium Q driver in Oslo.messaging, which is a decentralized PC solution. So it implements multi-host mode for messaging broker. And really it is supported in Oslo messaging kilo, so we can use this. And also it is supported in DevStack in kilo. So if someone need to have a try, you can run DevStack. And here is a simple configuration for Xerium Q, just disable RabbitMQ, disable QPID, and enable Xerium Q. And also you can specify Xerium Matchmaker to radius. So here it is only supported all in one deployment in DevStack. So here some future directions in upstream we are discussing in the mailing list to improve the Xerium Q driver. So first is outbound socket pull. Actually this function has been implemented in Liberty and by Chronicle. So outbound socket pull is very important for Xerium Q. So in kilo Xerium Q driver, when some OpenStack service needs to send a message, you then need to create a new Xerium Q socket to send it and close it. So every time you need to send a message you need to create a new Xerium Q and close it. So it is not very efficient. So here we implement socket pull to reuse and recycle Xerium Q sockets in the lifecycle. So it is very important. And another part is some, we need to do some reflection work. So this reflection work are proposed by Marantis. So firstly, currently all the logic was implemented in implicit Xerium Q. That PYT is only one Python source file. So when some newcomer needs to learn Xerium Q driver in detail and you need to read the code and maybe find some problems because everything is in one Python file so we need to separate it. And another stuff is to improve its socket pipeline by using some advanced patterns which I stated before like DLR router. So it is more technical detail. And another important part is supporting the multi backend. So currently Xerium Q can run with most other open stack projects. But unfortunately when Cinder implements multi backend feature it changes the directed topic format. So as I stated before the directed topic usually forms as a topic name dot house name. But however in Cinder multi, when you enable Cinder multi backend it follows by a backend name. So usually the host in Xerium Q driver, the host one at staff is considered as a real host name. So actually usually the host string is not resolved. So we need to fix it. And also the very important stuff is that if you are really interested in Xerium Q try to use it and give feedbacks to the upstream. So Xerium Q driver really needs more and more real world deployment cases. Actually we have some real world deployment cases but when I discuss with the community I find it is really like, we really like some other deployment cases. So if you are interested in it try it and give feedback. So we can get this message and can discuss with other developers in Xerium Q driver to see how to move on. So here are some real world deployment cases from AW Cloud which we have done so far. So actually during the last year we deployed more than 20 open stack in production. So all the cases are using Xerium Q. So first one is Beijing Telechrom Public Cloud and the total physical machine has more than 300 and the running instance more than 8,000 and this project we started from 2013. This is a public cloud. And another one is China Education Television. So it has more than 5,000, 500 physical machines which run in open stack and this project was started from 2014. So and here is another public cloud which is in development stages. It is at the early stage and it had more than 1,000 physical machines inside the data center and I'm not sure how it is because this project is at the initial stage. But actually I have a feeling that there is no problem in RPC plans because Xerium Q is a fully distributed solution and it can scale out. So we don't need to care much about it. So I think I will spend more on database part. So here are some deployment tips when you try Xerium Q. The first one is a host name should be resolvable. So either in static part or static file or you can run a DNS server in the cluster. And the next one is sometimes Xerium Q when using Xerium Q message may be disappeared without any exceptions. This problem is discovered in our production system but usually happens in larger scale deployments and the solution is very simple to increase and receive socket buff of kernel. Because Xerium Q is a low level socket library so its performance mostly rely on the operating system, the TCP IP stack inside the kernel. So maybe in some situation you need to optimize kernel stack. And here are two references maybe you are interested in. The first one is Xerium Q deployment guide which I was, I written it and put it in the of doc-style-openstair.com. So if you want to deploy Xerium Q manually and maybe this is an important reference. And another is upstream status which I will maintain in this wiki page and all the Xerium Q, all the bugs which is found in Xerium Q driver and its review status and all the blueprints and all the references and I will put them on this wiki page. So you can check it. So thank you very much and if you have any problems you can send an email to me so I'm willing to help. So any questions? I have a question. Okay. Since you're going from a start topology to a peer-to-peer mesh, does it change the way you want to deploy your networking on your management layer? Do you want to use a different way? Do you need 10 gig on your management layer or is it going to change the way you would do that? Oh, it is okay. In all deployments, we always put Xerium Q network within the management network but you can, let me see. So okay, you can read this deployment guide and it will help you to, if you have a dedicated network, just running for the RPC, you can set it up. So it is easy to just specify the network in the Xerium Q options in the configuration file. Hey, it was good to see the real world use cases. Thanks for that. So the question I have is like, do you guys care about like, I know at least from when I looked at Xerium Q that it didn't have any security support, so like you could not use SSL or things like that. So does that matter or for now you guys are just not tackling that problem? You mean the security, the encryption or decryption of the Xerium Q message? Currently, I finally know that currently Xerium Q library supports the message, message encryption and decryption but the driver in upstream doesn't implement it yet. So we will find some time to discuss how to implement it in the upstream. The message disappears in a large scale deployment. Is there any retirement mechanism without, because the risk disappears without any errors or exceptions, right? Uh-huh. Are there any try mechanisms built in? How does the high-level application work? Currently, when I discovered this problem in our production system, it happened for the Neutron Security Group because Neutron Security Group has a very large payload of the message and when Xerium Q send this message and the message is discarded by the operating system. So we find out that we need to increase the socket buffer of the kernel to make it work. Do you have any numbers to show us the difference in performance between the rabbit and Xerium Q? Oh, let me see, I just, let me, let me, okay. Check, so, so I just mentioned that there is, there is some of the topic taken in Hong Kong and maybe, if I don't know, there is, let me check. Oh, so here it is, so this topic, this topic and Bluehost provides some data for performance. So you can check this, this presentation. Right, so I have probably some basic question. So how do clients talk to, can actually identify themselves? Well, find this matchmaker, right? So they're- Oh, the matchmaker is a radio server, so you need to specify the radio server's IP and the port and username and password in the configuration file. So in terms of high availability, you still need to have this radio server somehow make high available itself, right? And for example, the every new client receiver, how do you call this? In the first place is to auto discover itself, auto discover all the cues and topics using the radio server. When the, okay, okay, I got it. Sorry, let me see. Oh, sorry, so Xerium Q receiver. Okay, so when you enable Xerium Q in your open cell cluster and all the open stack services, when it started itself and it will dynamically adjust the topic and its host name into the matchmaker and we implemented it. So any like an implementation guides how to make this matchmaker HAA basically because all these clients are very nice because they're distributed, okay? And independent, you don't have the central point by this matchmaker server seems to be kind of again, much make, well, it's like a single point of failure for me. It is a radio server, so you can make it highly available according to the some radius guide. So it is not a single failure point, single point of failure. Thank you. Yeah. Hi, I came in a little late, so I may have missed this. Hi. So this architecture doesn't involve queuing and eliminates queuing completely. Have you had any problems with notification-based services? If there are no listeners for a notification, I imagine it would get dropped, or? There is a listener inside the Xerium Q driver, so you mean a centimeter? Yeah, I'm thinking about accounting, for example. RPC, it makes sense not to go queue with a queue, in my opinion. There's nobody listening, the RPC will fail. But if you're setting notifications, that can't get lost. If there's, say, a reset of the system and the servers are listening, it may drop notifications. I'm just wondering if you've seen any application misbehavior or something like that, or is it actually working quite well? Okay, currently Xerium Q driver just supports the RPC notification. Oh, just, okay. Yeah. All right, thank you. The salamander event data doesn't go over rabbit and queue the first place. Isn't that true? Mm-hmm. Use the moment. Isn't it true that the salamander event data doesn't go over rabbit and queue in the first place, right? It's on event database, is that correct? So this wouldn't apply in terms of salamander event data. Or does the salamander event data go how does this affect, I guess I understand his question now, how does this affect the way that salamander would communicate with this event data that each service sends to salamander? Each server can directly send a salamander, currently salamander also can run with Xerium Q. So currently the service will send, directly send the notifications to the salamander host. Yeah. Yeah, it works. Yeah, you can, you can, okay. I, oh, okay, okay. I just, in my, in my, let me, let me see this one. Okay, in this deployment guide, and you can specify which network you use for the salamander queue. Yeah. You can specify the salamander queue as the listening address actually. So usually we deploy Zerium Q, Zerium Q network within the management network, but you can, if you have a redundant NIC for just for RPC, so you can do that. So, okay, thank you very much.