 Hi, we're here to talk about firehose a unified message bus for info services. I'm Matt Trinich. Yeah, I'm His faithful sidekick Jeremy Stanley So traditionally open-stack presentations always start out with a big complex diagram And open-stack infra is no different. We have a very large collection of diverse systems that we manage But you've all seen complex diagrams before so I just sort of punted and Let you use your imagination And that's where firehose fits in so there are all of those really different services that the infor team runs Some of them admit events some of them don't But they all have their own way of admitting events to other services and end users and firehose was our attempt to Unify that to a single place for users and other software We decided to use MQTT, which is a protocol that's very popular in the IOT space, but is really lightweight and Really low bandwidth so it was a good well suited for our application here where we have a lot of services sending a lot of data It's also quite reliable So our firehose is actually called firehose.openstack.org is its host name We have anonymous read-only access to the MQTT bus over 1883 on TCP We also support SSL at slash TLS on 8883 Which is the standard ports for both for the protocol. There is also web sockets support so MQTT is a protocol can be encapsulated in web sockets and you can expose that and it It's temporarily disabled on our system, but our hope is to re-enable it in the near future I think there was a web sockets bug that Needed addressing. There's a lib web sockets bug which is causing Denial of service attacks when you connect over web sockets Not not very pretty So talk a little bit more about MQTT since especially in this community might not be something You're super familiar with unless you do home automation or you know have a connected toaster MQTT is a pub sub message messaging protocol It was formerly called the MQ telemetry transport or the message queuing telemetry transport But there was an extensive argument in the technical committee for the MQTT Specification and they decided that MQTT stands for nothing It is an ISO standard For people who know ISO standards 29 22 And the protocol dates back to the late 90s Collaboration between IBM and I can't remember the name of the other company, but that's okay. I work for IBM And it's lightweight by design and low bandwidth it's actually and designed for flaky network connections It's designed to be a reliable lightweight connection for edge devices to send telemetry back Flaky network connections that sounds like the internet. Yeah, and specifically public clouds And One of the coolest things with MQTT are their topic system Topics and MQTT are dynamic and hierarchical And they support wild carding which means that Something that's publishing data to an MQTT broker can build up a hierarchy of what that information it's sending is And then users who want to listen to pieces of that Can filter on just those pieces and it's all done dynamically And while that's abstract and confusing I have a simple example So let's say my laptop was talking to an MQTT broker It would send top it would send the payload with the Hard drive temperature to the topic sensors synana joe, which is the host name slash temperature slash device name and Then if I was you know the administrator wanting to monitor everyone's hard drive temperature at on whatever broker I could wild card on just sensors and temperature or I could monitor all of the temperature sensors on my laptop Or I could just monitor all of the sensors for my laptop and you can see how this dynamic wild carding will Gives you a lot of flexibility in what kind of messages you receive from the broker as a Consumer of events which is something that other messaging systems don't really have They tend to have static topics and you subscribe to the topic and you get all of the events, but having this dynamic wild cardable System gives it a lot of flexibility and let you get just the data you need right so filtering at the server side Critical in this case because we named it fire hose for a reason We expect it to be providing large volumes of data and you're not going to want to have to filter it at the client The other thing that MQTT provides is levels of quality of service So by default when you send a message, there's no guarantee You're actually going to get it the broker just sends it and this is done for The lightweight aspect of it without providing any guarantees. It's very simple and low resource utilization, but there are also methods of Specifying more Guaranteed message delivery. So there's QoS one which guarantees the message will be delivered at least once But no guarantee that it won't be delivered again and QoS to goes To all available effort to guarantee that the message is delivered once and only once and it does a four-step Handshaking process between the client and the broker to make sure the message was delivered only once and the tradeoff Is as you increase the level of quality of service it increases resource utilization on both the client and the broker And and also network traffic flying a little bit on the handshake as well So for MQTT clients one of the biggest advantages of this protocol is that it's used everywhere for IOT at this point Which means that there are bindings available for every language You could or at least I could think of and then some I've never even heard of we tried to think of some at Random and yeah, they were there. Yeah, my favorite from that wiki page is lotus script to come back to the IBM employer and Hurrah But uh, yeah, they I mean lotus script We have examples on the documentation for Haskell and Ruby and you name a language There's probably a client binding for it somewhere And in particular I wanted to call out the eclipse PAHO project Which is an effort from Eclipse to have a standardized interface between multiple languages Obviously the exact semantics are going to be different because each language has different syntax But the way they behave and operate are very similar across multiple languages And if you're writing client code for different languages It's useful to use those clients because then it'll be similar in behavior between all of the languages you're writing clients for And then the other aspect of MQTT is the broker it has a centralized broker as opposed to something like zero MQ and it Enables it's where all of the Message passing goes through There are a lot of different options out there some open source some not I spent a lot of time looking at all of the options a popular choice is Erlang that trade-offs with Erlang are it's Gives you Distributed brokers that are HA and clustered, but there's a big CPU and memory penalty for it There are two popular Erlang implementations Vern MQ and MQE MQTD Which are both written in Erlang and very similar and both of their documentation go to great length to explain how they're different from each other But we ended up going with mosquito It's a very lightweight MQTD broker implemented in C and a very popular choice with a lot of open source home automation software It's an eclipse IoT project and it supports the MQTD 3.1 and 3.1 one protocols We picked it mostly because it's C and very lightweight. We know how to in the info community We know how to maintain C software Erlang is not so much and the lightweight aspect the resource utilization as you'll see in a bit is Kind of staggering when you think about how much data we're passing through it So just to start out from the infrastructure side of this effort Like most of our infrastructure that we run We've got a virtual machine in rack space that we have deployed the software on through configuration management as a puppet module that Matt did the Majority of the leg work on that does the mosquito broker deployment so mosquito broker on it Just running one instance right now We've got some ideas on how to do a high availability a high availability setup with it But for our current use cases, it's not necessary. So we're kind of gonna cross that bridge when we when we come to it Hardware specifications we tried to start out fairly minimal while still targeting the expected Publisher and subscriber capacities that that we have for for our initial use cases and we may we may scale the system up more as we As we get further into to other use cases down the road, but it's really just two virtual CPUs Not really super fast virtual CPUs at that a couple gigs of RAM 40 gigs of root file system and The flavor we're using is basically capped at about 200 megabits of bandwidth inbound and 200 outbound So pretty minimal. It's worth pointing out that my laptop is significantly faster than this instance. Yeah, yeah The services that we've got deployed on it now For anyone who wants to go in there and subscribe and start, you know looking We've got the ansible wheel that we use to Deploy our configuration management to all of the infra Community infrastructure servers Matt wrote a callback that he's working on getting upstreamed into to ansible itself That can basically emit MQTT Notification directly into our broker So you can you can kind of watch things getting notifications of things getting deployed on our servers through the fire hose Pretty much anything that is in the Garrett event stream is also showing up in the fire hose now We've got again Matt wrote a little Feeder demon called Gary MQTT that just listens to the SSH based event stream coming out of Garrett and Translates those messages as it receives them into MQTT notifications that go out on the broker We've got a launch pad client Matt wrote that is Effectively a mail client Because about the only way to get notifications out of launch pad Is via email so it's subscribed to official open stack projects I think yeah, the open stack project group in in launch pad and it gets emails about every bug that Except for you know like private embargoed bugs and stuff it's not going to hear about but pretty much everything else it gets all the emails for all bugs on all open stack projects and checks its email Every few seconds and then turns those Messages into MQTT notifications so you can you can sniff on bug updates new bugs bugs closed whatever on the fly as they're happening he also wrote a Link for our gearman worker that does the subunit processing for the subunit to sequel Effort that is at the back end right now for like health.opensdack.org and so on And so you can see when logs are getting processed and subunit data being Written to the the back end. It's worth pointing out on the launch pad one We have the first actual event stream from launch pad anywhere. Yes. Yes We invented the launch pad event stream actually the launch pad admins probably have an event stream of some sort I just not exposed to the public. I've asked Robert. He said no Maybe they want to use ours. Yeah, and we we did a fair amount of load testing of the the server So again, this is this is a load testing of the production to gig server as it's deployed This is just and and so for a baseline This is prior to our load tests just more or less what the current services that are deployed on there are doing from a message rate perspective, so we've got Two and a half yes like spiking up to around two and a half thousand messages per minute right now and Actually, if you can go back to the slide You can see the the green line is is basically the the number of messages We're sending the yellow line is higher because we have multiple subscribers listening to different parts usually of The fire hose but you know in in total the amount of traffic that they're receiving an aggregate The number of messages they're receiving is going to be higher because of duplication multiple subscribers listening to the same messages, etc and This is the actual data throughput for the for the same server over the same period of time. You can see where we're basically spiking up to Two megabytes a minute. Yeah, roughly We did some some ad hoc load testing at first just to kind of see how far we could push the system running some four loops and while loops and shell and Trying to spin up lots of brokers and not brokers lots of publishers and lots of subscribers and We pushed message rates up to around two million messages a minute. So pretty significant throughput before we stopped with that effort and during that time CPU Really didn't break about 35. I guess percent utilization and memory Really most of the available memory was getting used for for caching and buffers. We got a little over 50% utilization for mostly for mosquito Demon itself as as we pushed in there Though we also think we might have had a memory leak Yeah, I think we hit a memory leak because if you notice on the graph at two o'clock When it's at that dip that's where the big spike is at two o'clock Yeah, so we had done a restart at one point in there and that's kind of where the memory dropped off So evidence of probably some leakage or some really long-term caching that wasn't yeah But what the restarts did show us was the network Reliability because while the broker was down the clients were seamless and didn't even notice that we had lost the connection Yeah, and so after the ad hoc benchmarking we tried to do some Slightly more structured benchmarking. So these are Basically graphs of where we varied the number of publishers and number of subscribers with a small tool Matt wrote that The URL is on there on the slide But basically it allows you to specify how many subscribers and how many publishers you want and it connects them all as separate Threads or sub processes To the the broker and then charts the throughput that it's getting so we we ranged from Anywhere from one to a thousand twenty four subscribers and publishers in an array and Stepped through that exponentially just because the the data points in between the exponential steps weren't really very interesting, but this was effectively showing that at low numbers of Subscribers and publishers as you would expect we were able to to sustain a fair amount more We actually limited the the product of Subscribers and publishers to no more than about ten twenty four mainly so that we didn't overrun the available bandwidth on the server But we were effectively pushing it, you know at the the peak possible bandwidth for that flavor without Particularly breaking a sweat So now that we have a reliable broker and we've got data publishing into it It's useful to see how simple it is as an end user to start playing around with the data that we're getting The mosquito broker also publishes a number of C clients Publishes a C command line client called mosquito sub on Ubuntu. That's specifically the mosquito clients package I don't know why they split it out, but If you run that command just tell it to log into the host firehose.opensdack.org and subscribe to the topic pound That'll give you all of the message because pound is the wild card, which means Accept everything after this and if you don't specify anything earlier in the hierarchy. That's everything You'll be you'll be gulping from the fire hose. Yeah, like to say or sticking it straight into your mouth It's there's a lot of stuff that goes through especially maybe a little bit less now It'll just be the ansible but during a normal work day You have all of the Garrett traffic plus all of the ansible plus all of the bugs plus everything else But if you're you know want to get more targeted I You can listen to all of the comments on Garrett reviews on Nova Specifically I showed how to do it from the CLI here, which is the same command But you specify the topic for Garrett OpenStack Nova comment added But there's also as an example if you wanted to listen to all of the OpenStack projects in the OpenStack namespace You would replace Nova with a plus sign and that would wildcard it or if you wanted all Nova related events you would replace comment added with a plus it's worth noting that That is basically what Zool and also a lot of third-party CI systems do they're listening for comments and patch set created and so on events coming off the Garrett event stream and right now we've got Somewhere on the order of 200 I think roughly third-party CI systems connected to Garrett's SSH interface Independently all listening to everything coming out of the Garrett event stream So this is this is an interesting way that that some of that could possibly be backhauled. Yeah And then I showed how to do it in Python using the eclipse PAHO library Which you can see it's the first import line and it's actually very compact code It does the exact same thing as the CLI so when it gets the incoming message on that topic It will just print it to standard out, but you could replace that bit to do whatever you wanted with and Thought that was a useful case or if you're In infrastructure developer you're writing a package and you want to listen for all ansible events Or you're just curious like I am because I have services deployed on health.openstack.org And I have no way of knowing if puppet ran or not You can subscribe to health.openstack.org Ansible task completion which would be the puppet runs or get clones as well and I thought it would be useful to show examples of how to do those kind of Subscriptions as well In this case I used Ruby because I figured why use the same language all the time if you can use everything And the Ruby is actually more compact than the Python code. I kind of like that You had to up the font size just fill out the slide. Yes Well, I did that partially because of the next language example, which is in go which is far more verbose And Does the same exact thing as the Ruby so you have you know, what if eight lines versus that? If it was in C it'd probably be a few more lines than that. Yes, it would be So and the go is using Paho while Ruby was using something else on gems. I don't think it was Paho So we've talked about where we can go from here Obviously, we've already got a pretty interesting working system There's a number of possible use cases that are satisfied now, but you know, we'd like to Maybe start targeting third-party CI as a specific use case for for this as I mentioned earlier You can use it for things like desktop notifications if you like Matt was saying watching the bug Updates coming from Launchpad. Maybe you want to subscribe to some specific bugs and you can Yeah, sure, you could probably filter your bug notifications into a mailbox and have that notify you on your desktop But you could also just run a MQTT client and have desktop notifications for specific things that you're interested in in that regard also Possible use for interservice communication within the scope of infra right now we've got a number of different systems that are interdependent and This gives us one possible way for them to trigger actions off of one another's completion messages and on the desktop notification front It's also interesting to point out there are list bindings for all of those emacs faithful So they could get emacs events triggered based off of MQTT if their operating system is emacs Yeah One thing we have support for but aren't using quite yet is Garrett bought the IRC bought Supports MQTT. We just haven't flipped the switch to use that instead of the native SSH. Yep, that's that's true So we had a few future plans For starters for the Gary MQTT Subscriber we would like to try to get some integration with Zool after Zool v3 has arrived and We were also talking about Again after Zool v3 probably getting publishers in node pool and Zool and Really anything anybody can can think of that Infra is doing now that might make sense to add notifications for we'd love input You know just give give Matt or anybody on the infrateam a Heads up and and we can kind of try to figure out what's what's feasible to start exposing through this service Yeah, the one we have pending right now is the log stash workers getting those to emit notifications Yeah into the MQTT stream, right? Yep, that's that's probably very soon to arrive Yeah, we just need to review those patches a little longer Yeah, we did play with the log stash output plug-in for MQTT would there's a someone who wrote a plug-in for log stash itself But we quickly overrode the bandwidth limitations and was crashing the log stash server Yeah, as you can imagine all of the logs out of all of our CI jobs Not really a good fit as content for a message bus So with that I put a lot of links here for where you can get more information about firehose MQTT and all of the things We're using we've got the docs there as well as the schema docs which Detail all of the specifics of how topics are generated dynamically as well as the payloads for each of the services And then the original SPAC and then MQTT and mosquito And with that open the floor to questions. There are mics over there and can also hand you on Yeah, looks like we've still got at least a few more minutes. So Anyone with curiosity about anything? Here's teary to heckle us So I arrived late. So maybe you covered this The question is should we use firehose to feed events to staccatidics. I Don't think we mentioned that no that would that that's an interesting use case Um, I don't know. I think staccatidics probably wants to be able to reanalyze historical data and But it could maintain its own database. I mean you worry about sink. It could yeah We'd we'd have to catch it up first and then yeah, I don't know. That's a it's an interesting proposition It's worth thinking about And if we thought we were being heckled by teary here's Jim What's the What's the plan with the stability of the schema as the underlying services change So I've all of the things that are generating messages now are things I wrote I've been I haven't changed anything because I work on other things and it's stable in that way But if we need to change things we can The payloads aren't versioned right now nor the topics But if we needed to make that switch right now, there aren't that many consumers So it's easy to handle but at some point we'll have to think of a way of Do we want a stability contract on that schema and how we evolve things over time because our needs on the system are going to change as We gain users and I think the the first step toward it at least is that we do have some Fairly extensive schema documentation now that yeah, because Matt did everything I'm just standing here in support of Matt apparently I reviewed a few changes Yeah, so how do we know the list of the topics? Yes, there was there was a slide. I don't think I'm on the Wi-Fi, but this link Yes, I'm not on the Wi-Fi And this resolution is far So yeah, it was on one of the slides, but we've also definitely got documentation for that which Possibly Matt will be able to access right here Should have done this beforehand. I I did not but it's always fun to show off all of my bad English writing ability Me speak English bad Yeah, so you can see here for Garrett for example It documents the how the topics are generated so and then all of the choices right there And for the actual payload it points you to the Garrett docs because the payloads are literally copied directly from Garrett Same for launchpad Subunit workers With example payloads and then Ansible which is the complicated one because it depends on the internal state of Ansible and the event It's a minute But it's all documented and if there are any gaps in here Please tell me or please push a patch because I tried to be as thorough as I could but I might have missed something Yeah, this is all documentation In restructured text format in the open stack in for a system config repository As indicated there that it's awesome first first of all So this morning there was a fishbowl called achieving resiliency at scales of 10,000 plus nodes and there was some One of the outcomes from that was that as a community we'd like more Performance regression testing so that like every time there's a tempest run in the gate for example You could potentially and I think some of this is already happening like you could extract some metrics Maybe just a few simple ones or maybe later some a lot more in-depth ones things like How well I think we already capture how long the runtime of the tempest runners, right? But other things like how many bytes were put out on the message queue Just like performance metrics in general where you could over time look out for regressions in is this particular code Change having an impact is so is that kind of metric something that could potentially be Published on this and then something you know could could gather subscribe and gather Metrics and see trends over time So I don't think this is particularly suited for that application But you're talking to the right person about doing data analysis off of gate results because I've spent a lot of time on that problem So the first thing we do right now is we have the open stack health dashboard which tracks everything down to the test level and does Database doesn't it keeps a database of all of the tests run history so you can see the performance of an individual test over an individual run over all time There's also an effort that has been currently abandoned But was in progress to do the performance counter aggregation that you were talking about and leverage some of our existing infrastructure to aggregate performance counters from systems on on the machine like my sequel and rabbit and Export those counters into stats D or a different database where we could also do the same kind of aggregation That we're doing at the test level right now Yeah, I'm spacing on the name But what was what was that called in case anybody wanted to like get involved and pick that back up? I can't remember either Well, yeah hit us up an open stack QA or open stack in for IRC channels and we'll dig up what the project name was Yeah, yeah, right Yeah, it was it was it was a great effort It's just it currently is lacking for actual people to work on it So two questions. Okay. Um, the first one is when you talk about influx they be using with this Can you get into that a little bit more? Do you use by who says aggregation to? So we're not doing any aggregation on this What I was talking about just now we have test result aggregation Using a project I wrote a while back called subunit to sequel Which takes the machine parsable output from test runs and puts them into my sequel database And then we have tooling on top of that database to do the analytics and the visualization to show performance regressions and General trends and our test running I see Second question if I were to set this up in my infrastructure The question is how long did it take you and what you think, you know, how long with the next person you try to So for me the primary gate was review bandwidth, but the actual writing of the code is very simple And the deployment was actually pretty straightforward as well with the once I had the puppet modules fixed But the puppet module that we're using for this is is a very lightweight wrapper around install some system packages and put a couple of configuration files in place and Reload a service when those change So for example like what you have in OpenStack there should be like you talk about a week two weeks Yeah, probably that sounds reasonable great. Thank you Yeah, it's worth noting since there are language bindings for basically every language under the Sun you know if if what you're wanting to integrate and and emit messages from is Source code that's under your control then it's it's fairly trivial to add in 10 20 30 lines of a function call to emit whatever you want from wherever you want in the Applications of execution Are any other questions? Well, then thank you. Yes