 And if not, I'll just make it up as I go along. So, we're running a bit late, but I'm Jeremy. I work on Fedora Infrastructure at Red Hat. I'm gonna talk a little bit about the potential future of Fed Message. So everything I say is just things I've thought about and I welcome feedback and criticism and all that stuff. So, Fed Message, if you're not aware is what we use to send messages between applications. And it has a lot of painful aspects. So, we're gonna quickly cover a lot of those pain points and then we're gonna talk about publishing messages specifically and how that will solve some of these pain points and message schemas. And if you have a question along the way, just stop me. I also have a habit of speaking too fast. So, if I start doing that, just yell at me. All right, so the pain points in Fed Message. I think the first one I always notice is configuration. The configuration is very complicated. It doesn't work out of the box. You have to adjust several configuration options and it will load configuration from all over the place. So, I've got a list here of all the places. There's probably more places that I have not documented here. So, it makes it daunting to initial users. The other issue is publishing messages is actually pretty difficult, which is a problem for a library for sending and receiving messages. The main issues are each thread, each Python thread needs an endpoint and you have to define these endpoints before using Fed Message and you have to distribute this configuration to anybody who wants to listen to your messages. These are not dynamically generated so you have a fixed number of endpoints, which means you can't spin up threads and do some IO and then kill the threads and have those publish messages and it generally makes it difficult and confusing to use. When you start up a thread, it binds to a socket and because network latency is not zero, it takes a bit of time before a potential consumer will connect to you and this often leads to mysterious message loss and this is also very confusing. One thing people want to do with it is publish to 0MQ, which is what Fedora does, and also AMQP or ActiveMQ or Stomp or choose your message technology here and it does do that, but you can't do both at the same time. So, if you want to have 0MQ and AMQP, that's not currently possible. The final and most common pain point, I think, is that message schemas change regularly and this leads to a lot of key errors when programs look in just plain Python dictionaries and suddenly keys have changed. So having some tools to enforce that for developers I think will make it much less painful for everybody. Subscribing is, there's two different ways to subscribe to messages. There's a simple for loop basically. It just receives all messages and you can act on them one at a time. The second way is using a library called Moksha, or Moksha, I'm not really sure. Moksha, okay, Moksha. And you define a class and then you have to add an entry point and then you start this other service and hope everything goes well. But it has a fairly rigid API. It doesn't let you define a topic when you start it, when you start running your consumer. And this is something a lot of people want to do. They might want to test in stage and in production and not alter and be able to provide the message topic that the consumer uses at run time. And this is very difficult currently. Often this is, you can do this, but only in initialization and you have to do it in a very specific order and if you don't you'll get weird errors and nobody knows what to do then. And finally, it's a mixture of twisted asynchronous code and threads and that is risky at best and often runs into trouble. The last thing that FedMessage offers is cryptography message authentication and authorization. And at the time this was implemented, ZeroMQ did not have any feature to do this, but now it does. So we have a lot of, we have two different cryptography stacks. We use M2 Crypto or Cryptography and we prefer Cryptography but it requires a fairly new version. And this leads to a lot of problems, especially when you install from PyPy. So I think we can simplify that greatly. The last thing, I think it would be really nice is if it was easy for developers to write tests, record messages they publish, replay them, like VCR for most HTTP libraries. That makes it a lot easier on developers to write reliable consumers. So I've come up with a potential new publishing architecture. Very much, I just have a proof of concept and I'm still working on it, but I definitely welcome any feedback, any suggestions and hopefully it will solve a lot of the problems I just covered. So the current architecture for FedMessage publishing and subscribing is this sort of complicated diagram over here which I realize now that the black arrows on the dark background aren't great but it's okay, it is a tangled mess. Basically every subscriber has to connect to every publisher, which I described earlier. So you can then give each publisher at the top as a Python thread or a Python process and each subscriber at the bottom is an interested party. But when you wanna add a new publisher, you have to tell all the subscribers about it and each publisher has to have a connection from each subscriber and this quickly does not scale. So for example, Bodie has 60 endpoints and each subscriber has to connect to all 60 of these and when you have 60 times, say you have 250 interested parties, that's tens of thousands of sockets. So that's not very efficient and it's very difficult to track down where messages get lost. So yeah, so it has four servers and each thread needs its own endpoint. So each web request handler has its own endpoint, the backend has a couple of endpoints. I don't know if they're all used but because it's a static list, you have to define at least as many as you're gonna use and maybe more. So you also have a lot of subscribers who are just sitting there trying to connect to things that it'll never be able to connect to if you don't use them all. Yeah, so you'll have port 9,000, 9,001, 9,002, 9,000 so on and so forth. Yeah, so obviously it would be nice if we didn't have to do that. So the other approach is to create basically a broker with 0MQ. So at the top you have the publishers again and they talk to this service which will publish for them and the subscribers also talk to this service and so the publishers actually submit their messages to the, you can call this the broker, I think I call it the published service but broker's probably better. So they submit it to the published service and the published service then publishes it and in this diagram we're doing 0MQ but the nice thing about this design is that the published service can just publish it however it wants and it can publish it in many ways at the same time and if you're familiar with 0MQ which you don't have to be for this but it uses a request and response sockets so that happens synchronously and you can be sure that messages either did get published or did not get published and combining this with something like AMQP can let you reliably publish messages or not publish messages, you'll at least know because in the current design you publish a message and you have no idea what happens. It may get lost, there may have been nobody listening so there's a lot of uncertainty which is fine and there's good reasons to do that but many people are using FedMessage in a way that requires more certainty. So I think I've talked a bit about the benefits but we have a single point for all the subscribers to talk to so you don't have to let them know when you bring up more publishers so you can add a new web front end devotee without having to add more end points. The race condition of binding to a socket and having a subscriber connect to it before you publish is no longer a problem because they connect to the service that should be running well before you try to publish anything and then it'll just continue to run whether you restart the publishing, if you restart Bode for example in its current form all the subscribers will have to quickly reconnect before it publishes anything. Whereas with this service you don't have to do that. Another nice thing is that this will work out of the box. Currently one of the things you have to configure with FedMessage is all the end points. It doesn't come with any default ones and so if you try to publish a message it just won't work. And so this will use an inter-process communication socket for default submissions so you would run this service on every host basically and it'll bind by default to a TCP socket. You can also use this publish service to forward to other published services if you want to scale out a little bit or combine your messages, you can do both. So instead of having, I'll go back, instead of having subscribers talking to the published service you could have more published services. So if you needed to deal with a whole bunch of subscribers you could have many services running that all publish the same message. Another cool thing that I mentioned is you can publish to 0MQ, AMQP, whatever publisher you wanna implement all at the same time. You can publish multiple variations on the 0MQ format so if you want to change the wire format that becomes easy, you just add a new socket let everybody connect to the new one and once you're satisfied that everybody's updated you just kill the old one. It gives us the ability to have reliable messages if we want that and it's simpler to debug because there's only one place messages are getting sent. So I'm gonna try a live demo which I know is very dangerous. So it may not work but hopefully it will. Just drag this somewhere. Aha. All right, so I made the font really big hopefully everybody can see that. I can't see it but that's okay. So I've made a short little demo script. So at the top I just call, well, see here. So I import a multi-processing pool so I'm just going to call a function in a whole bunch of different processes and all it does is publish a short message and down here I'll just map this into the multi-process pool. So when I run this, we'll have a whole bunch of people checking it basically. So this is the new design I have written a really, really bad implementation of this published service so it may crash. Hopefully not. So the other thing I need to do is subscribe and actually hear about these messages. So I think I subscribed to the right port but we'll see in a second. All right, so I'm gonna run this script and so you can see at the top it sent four messages and I can't see if it actually received them. It looks like it did, all four of them, yeah? Cool. All right, so the other thing this did, which I'm gonna have to do some more dragging here, is publish this to RabbitMQ. So you can see I've got a RabbitMQ broker and here came our four messages and they landed in the same message queue. It got published to the leader.standby queue and we can go down here if I can see and get the same message that was published to ZeroMQ. So this lets you do both and this is nice if you have existing AMQP infrastructure or ActiveMQ and you also wanna use ZeroMQ apps that we've written. So actually I had one more bit to my demo and it's how it works currently or in the old way. So you'll notice here I have to subscribe to four different endpoints. This is because I wanna send four messages in four different processes and I've configured this already to use these endpoints. So I'm gonna have to connect to all of these endpoints and then I'm gonna go back and send these messages in the old way, which uses the script looks basically the same, it just uses the old publish function. All right, so it published four messages or should have and I don't know how many showed up. It varies. Sometimes all four do arrive. I think two arrived, but there should have been four. So you'll see that messages got lost because of this startup time, each process spins up, binds to one of those sockets and then the subscriber has to connect quickly before it sends the message and you can never know how fast the network's gonna be so it makes it very, very unreliable. So, I'm gonna find my slides again. So yeah, that's publishing. Yeah, Mike? You could set it up that way. The nice thing about this is it's very flexible. If you wanna have a separate published service, you can have one or you can have half a dozen. You can combine them all into a single endpoint, kind of like hub.fidoraproject.org today and it's kinda how you wanna deploy it. Obviously, if you combine them all into one and everybody talks to that, it's if one of them, if that service goes down, for example, nobody's gonna get any messages, but you may decide that that's worth the simplicity. Yeah, so this is why I called this talk the future of FedMessage, because I think that this would be a great way to get us to use a broker, which everybody's using messages in a way where they want it to be reliable. And so this way, you can just add AMQP in the back and then migrate however fast you want. So I definitely agree this is more of a migration strategy or potential migration strategy. Yeah, Adam, I wasn't there at the time. I do know that RabbitMQ, oh yeah, so the question was, was FedMessage not written in the way it is because all the brokers fell over from the number of messages? And that's a paraphrase, I did. But yeah, the answer is I was not there at the time. Yeah, yeah, so the question is, has anything changed since the initial investigation into brokers to make us believe they would be a better solution today? Not that I'm aware of, I didn't do the initial investigation. I will say that brokers like RabbitMQ are deployed in many large production areas and run at massive scales, far beyond what we do. Hundreds of thousands of messages a second. Yeah, so I mean, yeah, and you don't even have to use AMQP, so. It's true, so it's true. So the comment was, I guess basically that reliable message buses are appealing, but rarely work, is that a good summary? I don't know, it's true. I think there are plenty of reasons to not want a reliable message service. And there are plenty of things we do in Fedora infrastructure where we don't care whether a message gets lost. The problem is that many people want to guarantee that a message either made it or didn't make it, and there's no way to answer that right now. And there are plenty of patterns in 0MQ to make this possible where you can send a message and have something on the other side that checks back in and if it doesn't check back in, you assume you lost it. But yeah, distributed computing is really hard and messages get lost no matter what you do. And I don't really have a good answer other than it's tough. Yeah, so Adam, I have looked at GIL message. There are also very common, like in 0MQ has RFCs and very common patterns. And so, yeah, reliable messaging via 0MQ is... Yeah, I mean, we could use that. It's the thing that really spurred this is that we had all this CI, CD stuff and people were running all these tests in CentOS infrastructure and we want to get the results to the Fedora results DB and people wanted to send reliable messages and there are plenty of other times people have said, well, I expect this to make it or know that it didn't make it and we can't answer that. So Mike, yeah, Tim, I think first. I don't really, like I don't have off the top of my head how many people want it to be reliable. I have heard people say that it is reliable which is just not true. And when you make those assumptions, you have problems because you've assumed that the message will either get there, you'll be told about it. It's true and you don't have to use AMQP as a backend. You can still use 0MQ and the messages will either get there or won't and that's fine. But what you get from having this service is that you don't have to write down all your endpoints and not use threads or have to distribute this massive list or have everybody have to connect to hundreds of endpoints. So you can still use 0MQ, Randy? Yeah, so I think it can be fixed in other ways and it's just a matter about being honest. Is it gonna be reliable or not? And if it's not reliable, application developers need to deal with that and if it is, they need to know about that and you just need to go one way or the other and the problem is, yeah, yeah. So to summarize, no, we don't need this. You can build reliable messaging techniques on top of 0MQ using unreliable techniques and then you do that if you want to know that your message got there or didn't and if you don't care about that, then that's fine. But people have cared about that using FedMessage. So this is just a happy side effect of solving a different problem, which is that you don't need to have 60 endpoints for an application. No, that's fair, that's fair. It's just, mostly this is already a feature in FedMessage via Mocsha and I don't wanna have to support Mocsha and so I wanna have feature parity and I know that people use this feature to publish to AMQP and to stomp. So providing this is basically feature parity and we don't have to use it, but it'll be there. So, see, I don't know how much time I have left. I'm way over time, actually. So I don't know if there's something after this talk. We can skip the end of the talk if you want or we can keep going. Okay, well then I guess we're done but that was the interesting part of the talk anyway and I guess just to summarize, we got to the questions. Whoops, I went too far. So there's a couple issues on FedMessage. If you have opinions, please chime in. I definitely like to hear what people have to say. Thank you very much.