 Communication is complicated. If you're inviting friends and family to a party, you might ask them to reply to the invitation so that you can learn who's going to attend in order to plan accordingly. Unfortunately, it requires quite some effort to process all those replies, including ones from an anonymous person telling you they're bringing an unknown number of kids to the party, or your friend Ada being a bit on shredding aside by accepting and regretting at the same time. So finally, you might be better off by not asking for a reply, simply sending out the invitation and dealing with the people that show up anyways. And machine-to-machine communication isn't that much different. If your HTTP server tells you, hey, I'm a teapot, this is totally RFC compliant, but nevertheless, it's probably pretty useless unless you're trying to use your server for making coffee. So again, you might be better off by not asking for a reply or a response and go on with other tasks. For the rest of the talk, I'm trying to show you how to improve the maintainability and stability of your overall system by choosing the right means of communication for each job, which very often means to add a synchronous communication to the mix. I'm going to highlight some disadvantages of synchronous communication introduced to the advanced message queuing protocol and its two main use cases, which are published subscribe pattern and the task queue. If time allows, I have some example code for you and a quick live demo, and I'll finally wrap things up. So let's look at synchronous communication. A very common pattern in machine-to-machine communication is what's called a master data update. Look at this setup. You have a master data application that is in charge of the data about your business partners. And it's surrounded by several independent components that are dealing with their orders and their support contracts. And somehow those two client applications also have to work with data about your business partners. If a business partner goes out of business, this important information is going to result in an update to their master data record. But unfortunately, this is not sufficient because if a business partner goes out of business, you want to cancel their pending orders and put their support contracts on hold. So somehow the master data application has to spread the knowledge about the update it just received so that the orders and the support contracts app learn about that. So I'm mainly doing web development. And if I think about machine-to-machine communication, the first thing that comes to my mind is REST. And of course, you can build a REST API in order to spread the information about the master data update. But unfortunately, this has some downsides. If your order's application is temporarily down, the master data app is going to make a REST request to inform it about your customer being broke. But this request is going to hit a timeout. And if the order's application is back again, it has missed all the important information. If you add a new application to your setup, maybe one that's in charge of invoices, this app is never going to learn about your business partner being broke unless you modify your master data applications code so that the master data app is making a third, an additional REST request to this new invoices application. This is basically because HTTP is routing one-to-one, while what you would prefer to have here is a one-to-end routing. So you end up with a rather tight coupling of your components. Master data updates are not restricted to the world of e-commerce. I'm working for a social networking site, that's xing.com, and we have similar issues. We have a core application that is in charge of the main user data, and it is surrounded by a number of other applications that are powering different verticals of our platform, like the event section or the job section. Now, if a user is interested in information about a particular event, like a conference, their request is going to hit the events app, and this app can immediately serve data-like event location, event description, and so on. But in order to render a proper list of attendees, it has to know their names, their job titles, their companies, and so on, and this is core user data, which is owned by the core user application. So our event app has to work with that data room, and to keep performance high, it's going to cache that data, which also implies that if a user updates their core data, for example, the job title, this results in an update to the core user data in the core application, but the other applications have to learn that this update has happened because they need to invalidate their caches. We've already seen that REST has some downsides in this setup, which is why we don't use it here. We are using asynchronous messaging instead to be more specific, messaging based on the advanced message queuing protocol. And if you type this into the search engine of your choice, you're probably going to end up on mqp.org, which tells you about capable, commoditized multi-vendor communications ecosystem, creating opportunities for commerce and innovation, transforming the way business is done, so it's basically a tool for world domination on the internet and even in the cloud. So bingo. Wikipedia has a much friendlier and more useful message for us. It's talking about an open standard for a protocol for a message-oriented middleware. So this middleware is a message broker, which is basically a demon that is connecting message producer and message consumer. MQP is a wire-level protocol, so all it does is define a data format. And every client that's able to provide that format is compatible irrespective of its implementation. So this allows you to glue together components that are based on totally different technology stacks. Client implementations for sending and receiving messages are available in basically all programming languages. And you can also choose from a number of broker implementations. We're using RabbitMQ, which is a very popular open source implementation, but you have other options as well. And there are even some cloud services that are offering MQP infrastructure for you. Let's have a look at the message flow. There is a producer or publisher that is sending a message, and it connects to the message broker and drops this message into a so-called exchange. The exchange is in charge of the routing. It copies the message into zero, one, or several method queues. And from there, it can be consumed by the consumer application. This can be push or pull dependent how you set things up. In our example, the producer is a master data application that has received an update. And the consumer is the client app that wants to learn about master data updates that have happened. The producer is dropping the message into the exchange, and it doesn't need to know anything about the consumer receiving the message. If the consumer has a hard time to invalidate the cash, the producer cannot help anyways. And most of the time, it doesn't even care. So it's sending the message, it's not waiting for any kind of response, and immediately moves on to other tasks. In reality, you do not have one producer and one consumer, but you're probably having several producers sending messages to several exchanges. Each exchange can be bound to several method queues, and even the single method queue can be consumed from several consumer instances. But let's have a look at the messages first. The main ingredients of a message are payload and routing key. So the routing key is usually a string, it doesn't have to, and it's used for routing the message into the correct message queues. The payload is your application data, and you can actually put there whatever you want. We are putting text into the payload, and we format it as a JSON. Simply because we're dealing with web applications, and most web applications love to work with a JSON format, but you can choose whatever is best for your setup. You can add some structured app-specific data, like properties or attributes, messages have headers, and they can receive annotations along the way. But the payload is immutable. So that means you can have end-to-end signing or end-to-end encryption or run some validity checks on your incoming messages. If a producer is connecting to the message broker, and it finds that the exchange it wants to use is not present, it can simply create it. So far, this is not very useful because the exchange has nothing to route the messages to, so it has to drop all the messages. But the producer can as well create a message queue and bind it to the exchange, and now the producer can start sending messages. Those messages are going to be in the queue until the consumer appears and starts to process them. If a consumer connects to the broker and finds that the queue it wants to consume is not present, it can create this queue, bind it to the exchange, and wait for messages, which are going to appear as soon as a producer connects to the broker and starts to send them. So creating exchanges and queues and bindings between them is nothing that you do in your broker configuration. It's your application code that is in charge of that. So all the power lies with your application. Now, with great power comes great responsibilities. In our case, that means that your application code is also in charge of cleaning up. So it has to remove unused queues or unused exchanges. If you have a queue that is still receiving messages but those messages are not consumed because the consumer went away. Those messages are going to pile up in the queue up to the point that your whole broker is getting into trouble. If a consumer is temporarily down, the messages are staying in the queue until the consumer is back and can continue to consume those messages. So no information is lost. The consumer is going to learn, for example, about the master data update about your business partner being out of business, even if it has a temporary downtime. If you add a new consumer to this setting, the consumer connects to the broker, creates its own queue and binds it to the exchange. Now, the exchange is going to copy the message about the master data update into both queues so that both consumers learn about this update. They are both receiving the same messages. This is happening without you having to touch the producer code. So your producer doesn't have to be aware of the consumers that are consuming those messages. You cannot do this with HTTP because of its one-to-one routing, but MQP gives you a proper one-to-n routing here. Now, consumers basically can add themselves to the system, create a queue and decide which messages they want to consume, which basically means they can subscribe to messages. So what we have here is a nice published subscribe pattern. Let's look at the master data example again. So we have the master data application that has to inform some other client apps about updates it receives. Those clients might have very different needs. If the client doesn't work with the master data itself, but simply owns data that somehow associated with that, this client only cares about master data records that are removed. So if the user deletes their account, the client app has to learn about it in order to remove the data associated with this user. If the user updates their data, this client app doesn't care. It's different if the client has cached the master data records. So this client also has to learn about updates to the master data because it needs to invalidate the cache accordingly. Some clients might even maintain a full mirror of the master data. And for them, it's not even sufficient to learn that an update has happened. They need the new updated data in order to update their mirror. Now, in order to cater to those different needs, we are simply using different AMQP messages. We have a message with a routing key that ends in user deleted. I cut off the first part of the routing key here. And this message is sent if a user deletes their account on our platform. The payload is very simple. It's just the user ID because there's not much more to say about the user that just closed their account. And basically, all of our applications are listening to this message. They subscribe to it so if this message arrives, they clean up all the data they have about this user. We have a different message which has a routing key like profile updated. And this message is published if a user updates, for example, their job title. So the consumers of those message can check if they have cached this data and invalidate their cache accordingly. So it's a consumer application that knows best what kind of information is of interest to them. And so they can decide which messages they want to consume. Which basically means they add themselves to the system by creating the queue and binding to the exchange accordingly. Let's have another look at the updated message. Its payload contains the user ID, of course, and it contains the fields that have been updated. Now, if a consumer application has mirrored the master data records about our user, this information is not sufficient because this application wants to update the mirrored data. Still, we do not include the new updated data into the message payload. And this is basically our way to embrace the fact that we are dealing with a distributed system. So there is no guarantee about the order of messages. If a user updates the same field several times in a row, the last message about those updates we receive is not necessarily the message that was published during the most recent update. So if we rely on the data included there, we might end up updating our mirror to data that's already outdated. If you cannot rely on the order of messages, you basically aim for commutativity and that's what we reach by omitting the updated data from the message payload and simply including only the field names. Now, our consumer application still needs the updated data, which is why we complement our messaging set up with a REST API that is always able to provide the most recent data. So it's our leading system. And a consumer that has learned that there has been an update and needs a new updated data has to follow up with the REST request in order to fetch that data. As a side effect, we have very small payloads, which is easier on the broker, especially if you have to deal with a lot of messages. So we include just enough information for the consumer to decide if they have to make this follow up REST call or not. In the example, that's the field names. Another property of distributed system is that there is no such thing as exactly one's delivery. So you might end up with duplicated messages. Look at a producer that is recovering from a connection failure. This producer may or may not be able to figure out if a previously sent message has made it to the broker or not. So in order to ensure at least one's delivery, a well-behaved producer is going to send this message again, which might result in duplicated messages for the consumer. So whenever possible, it's a good idea to use item potent message handling to avoid any issues with that. So we looked at consumers a lot and how they can connect to the system. Let's have a look at producers. Producers can add themselves to the system as well. They connect to the broker and start to drop messages with the same routing key into the exchange. And now messages from both producers are going to end up in the same queue and are going to be consumed by the same consumer. This is a useful setup if you want to track something that is happening all over the place. We used to consume a pretty expensive third-party service and wanted to track our usage. Now, of course, we could have implemented tracking into each and every application we have, but this would leave us with the annoying task to combine all this tracking data. So what we did instead was turn our applications into message producers. And whenever one of our apps connected to the third party, they dropped a message about it. And all those messages were routed to the same queue and to the same message consumer, which was a very simple tracker that just incremented a counter. So we ended up with a centralized tracking for decentralized events. Of course, this is not item potent message handling, but for this use case, we didn't care because we were interested in the order of magnitude and we didn't care about the exact numbers. So I've talked about multiple producers and multiple consumers and that they all can connect to the system. In the master data update example, we had one producer, the master data application, which was sending messages to a number of consumers, all the clients that need to be informed about the master data updates. In the tracking example, it's the other way around. We have multiple publishers sending messages about the event that we want to track and we have one consumer, the tracker. So technically, you can also route messages from multiple producers to multiple consumers. MQP is totally fine with that, but you should still think twice if you want to do that. Publishers and consumers do not know much about each other, but they still have to agree on a payload for the messages. And this message payload is as much of an API as a JSON, a REST API response. And sooner or later, you have to make breaking changes to this message payload. To coordinate a breaking change between multiple producers and multiple consumers can be quite painful experience. Also, if you think about your use case, it very often turns out that your business logic is represented much better by having several different messages with different routing keys, where some of them are routed one to N and others are routed N to 1, instead of having this super message that's going from multiple producers to multiple consumers. The messages we've seen so far are basically notifications. Something has happened in the producer of the message and it informs the rest of the world about it. There's a different type of messages that's very common in MQP and that is a command message. This means the producer is sending a message in order to command the consumer to do something on their behalf. So, it's basically creating a task queue. If a user uploads a new profile image to our platform, we have to process this image, create thumbnails in different sizes, upload them to a large cluster of servers. And this is a very time-consuming thing to do, so we do not want to make our users wait for that. And this is why our user facing frontend doesn't do it. Our user facing frontend takes the image, sends an MQP message about it and immediately returns to the user, telling them that their image is going to be processed. The consumer is the actual image processing service, which is doing all the heavy lifting, while the users can move on and are not blocked for whatever they want to do on our platform. And this is a very common pattern in web development that you have a user facing frontend that's basically doing nothing except for sending the message and some backend service consuming those messages that is doing the real work. Now, this obviously helps you to improve your response times. If your frontend is doing basically nothing, it can respond very fast. But it also helps you to avoid down times because the downtime of your image processing service, which is a message consumer, is no longer a user facing downtime. The users can still upload images, the messages are going to stay in the queue until your image processing service is back. On top of that, you gain some reliability if you compare this setup with a simple background job because you can make use of some persistence features that your message broker is providing. If you want to decouple a bit more, you might decide to set up a message consumer that is exclusively doing image processing, nothing else. So if there is a sudden spike in image uploads, the messages are going to stay in the queue a bit longer. But the only thing that slows down is the processing of other uploaded images while the rest of the platform is totally unaffected. If you still feel the image processing is too slow, you can scale on the fly by adding another consumer instance. Now, we have two identical consumers here consuming messages from the same queue, which means that they're basically doing the image processing in parallel. The nice thing here is that the consumer code is still pretty simple. The consumer code is not aware of being run asynchronous or being run in parallel. All it needs to know is how to process one message at a time, that is how to process one image at a time. So it's very, very simple, making maintenance easy. One thing to note here is that the balancing of the messages is not happening between the queue, so the messages are directly balanced to your consumer instances. And it depends on your application code and your setup if you have around robin balancing or something more fancy here. This approach is not only useful if you have to deal with a very time-consuming task. It's also useful if you have some task that you need to do very, very often. For example, if you need to migrate millions of users. Of course, you can write a script that loads one user after the other, like processes it, stores the data bag, and so on. But this script is going to run for a very long time, so it's going to crash, you have to restart it, make sure it picks up where it crashed, and so on. And you're pretty busy babysitting your migration script. A nicer approach might be to build a very quick script that's more like a migration trigger. It does nothing except for sending an AMQP message for each user to be migrated. This script is going to finish quickly, so you don't need to spend much effort on babysitting it. The messages are going to be routed to a queue from where they are consumed by one or several consumer instances, and they are doing the actual data migration. This can be useful if you want to prepare a switch from some old implementation to a new implementation. You can use it to prefill your new system with the data you need in order to go live. And you can do this without a user-facing downtime. We have recently changed the format of our profile images. They used to be rectangular nowadays, they are square. And in order to prepare that switch, we had to create images in the new format for all our users. And we used AMQP for that. So we built this migration trigger script sending AMQP messages for every user and a consumer that was creating the images in the new format. And while this was running, it was still quite a time-consuming thing to do, our users were still hitting the old implementation. So users still uploaded images to the old system. There was no downtime. But we had modified the old system so that it send it an AMQP message every time a user uploaded a new image. So those users were put into our migration queue again every time they uploaded a new image. Which means that in the end, we could be sure that we had created new images in the new format based on the user's most recent update. While we were migrating, users could still use the old system without any disadvantage. So of course this requires a message handling to be item potent. So we didn't really care if the user had been migrated before or not. We just put them in the queue again and that's it. If you replace an important component of your system with a new implementation, you want to test that very carefully. So a while ago, we have replaced the front end of our user profiles, which is where users store their CVs. In order to test that, we used AMQP again. So the data behind that feature is up to 10 years old. So we have all the edge cases you can ever think of. And sitting down and writing a test plan covering all those edge cases is basically impossible. So what we did instead was modify the old implementation so that every time there was a request, it sent an AMQP message about it and we set up a simple consumer that followed up with a so-called shadow call making the very same request to the new system. The users were served from the old system. They didn't notice anything. But we would receive that kind of live traffic to our new implementation, which allowed us to learn if the performance was good enough, if it could stand all the traffic it was going to receive when going live. And it also helped us to identify a number of bugs where our new implementation was not able to handle some edge cases. So we found some hints in our logging or monitoring system and we could fix all those bugs even before going live. Again, users didn't notice anything because they were still served from the old implementation. Okay. I have some simple example code for you. I have to find my mouse pointer. There it is. This is a very simple message producer. I'm using the bunny gem and I'm creating a connection to the message broker. Within this connection, I create a channel which is basically a way to have several separate kind of connections within one TCP IP connection. So whatever you do happens inside one channel and the channels are totally separated from each other. Next thing I'm going to do is create a queue. I call it the awesome task queue because I cannot be sure if the queue already exists on my message broker or not. If it's already there, it doesn't do harm. If it's not there, I'm going to create it here. Now I set up some payload which is basically a timestamp. And finally, I'm going to publish the message using the so-called default exchange. The default exchange is present in every rabbit MQ instance and it's routing messages by comparing the routing key of the message with the name of the queue. As you can see here, I'm using the name of my freshly created task queue as a routing key in order to ensure this exchange is routing my message there. So we can already see this in action. Okay. So my rabbit MQ instance already has this queue but there is nothing in it. So let's send a message and now we have one message in the queue. So we want to consume this message and I set up a very simple consumer for that. Again, I'm using bunny. I'm creating a connection to the broker, creating a channel. I'm again creating the queue because I cannot be sure that the queue already exists. I don't know if the producer has already been there and sent a message or not. So just in case I ensure to create the queue, put out some debugging output and this is actually where the processing of the message happens. So in this block, I'm basically putting out the payload I have received and sleep for two seconds in order to pretend that I'm doing some real hard time consuming work. I acknowledge that I've received the message to the broker and continue to wait for more messages to come. Yeah. So the worker immediately consumes the message I've sent previously. I have prepared another worker which is basically the same but it's sleeping for five seconds. So it's much slower in doing the important time consuming task. I start that worker as well. So now both of them are waiting for messages and I'm going to send some more. And now you can see that the fast worker is receiving the odd messages while the slow worker is receiving the even messages even though it's much slower in consuming them. This is because by default the messages are balanced around Robin and they're balanced to those consumers independent of their speed or whatever. Now, you might want to change this behavior which can easily be done by changing the prefetching setting of the message consumers. I'm going to restart them. This is a slow one. Now, this is a fast one. And I'm going to send some more messages. And now you can observe a different type of balancing. So whenever a worker has free capacity, it's going to receive the next message, which of course means that the fast worker is receiving way more messages than the slow one as you can see here. And from both examples, you can see that the order of messages in this setup is totally not guaranteed. So you cannot rely on receiving message 19 after message 18. It might be the other way around as well. Okay. When describing the publisher, the producer, I already talked about the default exchange. And of course, there are more options to route your messages in your broker. That's interesting. Let's see if we can fix it. So the slide here looks correct, but it's just not displaying it. There you go. Okay. So the default exchange I have used in the example code is so-called direct exchange. It compares your routing key of your message with the name of the queue and routes accordingly. So this is useful if you want to set up a task queue and route the messages to a number of consumers, maybe in around robin fashion in order to distribute those tasks. The fan out exchange is totally different. It completely ignores the routing key and the queue names. And it broadcasts all the messages to every queue it has ever heard of. So this is useful if you have some global state to announce some global leaderboard in a game or something like that. So you're just broadcasting it out. The topic exchange can do the routing based on multiple attributes. It expects the dot separated words as a routing key, and it can handle wild cards. For example, you can use it to ensure that 10 different consumer applications receive the message about the user being deleted, but only five of them also receive the message about the user being updated. So this is a useful exchange for implementing a published subscribe pattern. There is another exchange called header exchange, which can also route based on multiple attributes. This one is ignoring the routing keys and looking at the headers. So if your attributes are easier to be expressed in terms of headers, this one is for you. And it also allows you to have routing keys that are not strings. Okay. So AMQP allows you to easily glue together applications that are based on totally different technology stacks. It can help you to improve your response times by decoupling your user facing front end from your time consuming back end tasks. It also helps you to avoid some down times because the down time of your back end service is no longer necessarily a user facing down time. You can gain reliability through the broker's persistent features and scale on the fly by adding additional consumer instances to an existing task queue. So it basically helps you to decouple your components and reduce the complexity of your system. And you can do all that using very, very simple code. Your consumer code is not aware of being run asynchronously or being run in parallel. All it has to know is how to handle one message at a time. So that's very simple. I've shown you in the example code how easy it is to get started with AMQP messaging. So maybe you want to try it out yourself and decouple all your things. Thank you very much.