 Hi, welcome to the session RabbitMQ on Kubernetes DeepDive. RabbitMQ is known as a traditional enterprise middleware, and many of you are associated with VMs or bare-metal machines. Maybe you have installed it with a package manager and then configured it with Chef or Puppet. But what you might not know is the best experience to run RabbitMQ is on Kubernetes. And that's the part where our team has been heavily investing in over the last few years. If you have deployed RabbitMQ on Kubernetes with a HelmChart or if you have never heard about the messaging to Poly2 operator or never integrated with the Kube Prometheus deck, then you are missing out a big time. All of this works natively on Kubernetes. And let's deep dive into all of it over the next 30 minutes. RabbitMQ is a cloud-native message and streaming broker. It's an architectural pattern that achieves loose coupling. For example, your producer and consumer applications don't even need to be aware of each other. And you can deploy, upgrade and version those applications independently. And for example, you can also have producers that send messages while your consumers are down. With RabbitMQ, you can also have a very high scalability on your system by having an asynchronous data flow. So for example, your producers might send more messages than your system is able to cope with. And in that case, RabbitMQ can act as a temporary buffer for messages. So it can handle very high throughput because it's written in Erlang and runs on the Beam virtual machine. And the Erlang programming model is based on the actor model. So it's a beautiful language that has been designed and created for messaging systems in mind. With RabbitMQ, you can have complex routing topologies thanks to the MQP spec defining exchanges, queues, and bindings. Examples of simpler routing topologies are work queue patterns or public subscribe patterns. In RabbitMQ, there are very modern queue types, for example, the Quorum queue, which is based on the raft consensus algorithm. It supports MQP 0.9.1 and MQP 1.0 protocols, which are, by the way, two very different protocols. It supports MQTT for IoT use cases and stomp and also a new binary stream protocol. You can deploy RabbitMQ on any Kubernetes using operators and monitor it with Prometheus and Grafana. The agenda for this session is we will first understand how the RabbitMQ cluster operator works. It's a Kubernetes operator that deploys and manages RabbitMQ clusters. And this repo got G8 last year. We will then look at the integration with Prometheus and Grafana. We will see how we can reliably upgrade RabbitMQ clusters and then explore the messaging topology operator. So the messaging topology operator got open sourced this year. We will see what the new RabbitMQ 3.9 features are. RabbitMQ 3.9 got released in July this year. We will give you some suggestions how you can contribute if you would like to and then see what the roadmap is for the next couple of months. So let's start with the RabbitMQ cluster operator. The RabbitMQ cluster operator follows the operator pattern. So like any other Kubernetes operator, it has a controller running in the ObserveDiffAct loop, acting as a client to the Kubernetes API. It also defines a custom resource definition, so in our case, kind RabbitMQ cluster, and it first watches for the RabbitMQ cluster's desired state. It then diffs against the actual state. So it compares against the dependent Kubernetes objects of a RabbitMQ cluster object. So what are those dependent objects? Most importantly, the state fill set, which defines the pod spec where RabbitMQ will run in. Then a service object for routing traffic from the clients to RabbitMQ, to config maps, ones for the enabled plugins of RabbitMQ, and one for the configuration of RabbitMQ, and two secrets, one for the default username and password, and one for the ErlenCookie. So the ErlenCookie is used to authenticate RabbitMQ nodes to each other, and also to authenticate the RabbitMQ CLI to the RabbitMQ server. And there are a couple of other Kubernetes objects that have been created, which are not of too much interest right now. So if there is a diff, the controller acts. So it creates, updates, or deletes these dependent Kubernetes objects, and finally updates the status of the custom resource. So let's see how this looks like in practice. To install the cluster operator, let's check out what the readme says. So we should just copy paste this single command here, and this will apply a YAML file which creates the RabbitMQ system namespace, which creates the RabbitMQ cluster custom resource definition, and the RabbitMQ cluster operator deployment. Now, once this is up and running, we can deploy our first RabbitMQ cluster. So the kubectl apply myRabbitYAML, and if we look into myRabbitYAML, we see that we define kindRabbitMQ cluster, we give it a name, we specify that we want to run three nodes with the latest 3.9 management image, and this is how we can enable additional plugins. So in our case, we are going to enable the RabbitMQ stream plugin because we use it later on in the demo. We now check the pods. We see that all three pods start up in parallel. They discover each other via the Kubernetes peer discovery plugin for RabbitMQ, and after only 33 seconds, they're all ready. So ready in this context means that the MQP port of RabbitMQ is open. If you now want to check what dependent Kubernetes objects got created by the cluster operator, the easiest way to do this is to use the kubectl RabbitMQ plugin. So it's a simple batch script that can be installed via crew. So kubectl crew install RabbitMQ will install the RabbitMQ plugin, in my case it's already installed. If the then alias kr to kubectl RabbitMQ and type kr help, we see a list of available commands for the kubectl RabbitMQ plugin. For example, we can list RabbitMQ clusters and you can also get dependent objects. So let's try it out. If we kr list, we see that we have a single RabbitMQ cluster which we just deployed, name myRabbit, all replicas are ready and reconciliation succeeded. We now kr get myRabbit, we list the dependent Kubernetes objects. And as we said earlier, the state will set will be created by the cluster operator, the three pods, our two config maps, our two secrets and actually two services. So one service for the client applications. So that's the MQP port which is exposed, the management port of RabbitMQ, the stream port because we enabled the stream plugin and the Prometheus port. And the other service is for the state will set. So that's the headless service. So that's cool. It was very easy to deploy a RabbitMQ cluster. We just applied a few lines of YAML. Now the cluster is up and running but there's no load yet on the cluster. So let's create a RabbitMQ client application. For that, we again can leverage the kubectl RabbitMQ plugin. So we run the perf test command and this will deploy a Java perf test client. We run it against the MyRabbit cluster. It's going to create a single quorum queue called MyQuorumQueue. By default, there will be a single producer and a single consumer which will produce and consume with a rate of 1000 messages per second. And each message will only have 10 bytes in size. So we can check the logs of perf test and then we see that, yeah, it sends and receives 1000 messages per second. Now we want to access the management UI of RabbitMQ. We again can leverage the kubectl RabbitMQ plugin. So first we need to get the default username and password. So krsecrets MyRabbit will print it. So I'm going to copy this and if you now say krmanageMyRabbit, the management UI will open. So I just paste in username and password and then we see that we are running RabbitMQ 3.9.5. We have three nodes, server zero, server one and server two. And we also have defined a queue, MyQuorumQueue, which got created by the perf test client. So that's cool. Now we know how the cluster operator works. Let's look into how we can integrate with Prometheus and Grafana. So in the cluster operator repo, there is an observability directory. And this observability directory defines many Grafana dashboards for RabbitMQ. It defines Prometheus rules and also pod monitor and service monitor to tell Prometheus how to scrape RabbitMQ metrics. There's also a quick start script which sets up everything. So it uses the queue Prometheus stack ham chart and it will deploy Prometheus, Grafana and all those alerting rules and config maps which then get read by the Grafana sidecar importing the RabbitMQ dashboards. So I'm going to execute this now. It took around two minutes for the quick start script to succeed and at the end it prints out some command how to access the Prometheus and Grafana UI. So I already opened Prometheus and Grafana and if you now type RabbitMQ, we see the different RabbitMQ metrics we're getting. So for example, RabbitMQ build info will print the three pods we deployed earlier on. We also define a couple of alerting rules for RabbitMQ. So if you click on alerts, then we see that for example, we have an alert for high connection churn which is an anti-pattern in RabbitMQ. So in RabbitMQ, client connections should be long-lived and if there are clients that open and close connections very often, we will receive an alert. There are also other alerts such as low disk watermark predicted. So this one is predictive. So it looks at the trend of the disk space of the last 24 hours. And if that trend were to continue within the next 24 hours, then we're getting an alert. And alerts can be received in different backends via alert manager. To see how an alert looks like in Slack, we can look at our long-running environments channel. So RabbitMQ LRE. And this is for example, an alert for un-routable messages. So there are some number of un-routable messages within the last five minutes in RabbitMQ cluster RabbitMQ in namespace LRE39. We also get a nice summary of what that alert actually means and how we can mitigate it. Now, if you look at the Grafana dashboards and filter by tag RabbitMQ Prometheus, then we see that we have a couple of RabbitMQ Grafana dashboards. So some are for the early distribution. And then we have an alerts dashboard for RabbitMQ, one for the quorum queues, one for streams. So we look into this later on. And the most important one is the RabbitMQ overview dashboard. So we can filter by Kubernetes namespace and RabbitMQ cluster. And in our case, we see that we have three nodes up and running, one queue defined by Perftest. We have thousand incoming and outgoing messages and we see lots of graphs. So if you want to understand what they mean, just hover your mouse over the eye icon and there will be a nice description with links to the rabbitmq.com docs. So this is how you can integrate RabbitMQ with Prometheus and Grafana. Let's check out next how we can reliably upgrade RabbitMQ clusters on Kubernetes. So there were new commands introduced in RabbitMQ 3.8. For example, RabbitMQ upgrade await online quorum plus one. And this waits and blocks for all quorum queues to have an above minimum online quorum. So for example, in a three node cluster, the minimum online quorum is two. And this command gets executed in the container pre-stop hook before the RabbitMQ node is being terminated. And if only two nodes are available, this command will block because it's not an above minimum online quorum. If all three nodes in our RabbitMQ cluster are up and running, this command would not block. A similar command exists for classic mirror queues and also the RabbitMQ upgrade drain command has been introduced. So this will first close all client connections and in our case, for example, perf tests will reconnect to different nodes. Now it will transfer primary replicas of all quorum queues hosted on the target node. We'll see in a second what that means and then mark the node as down for maintenance. So let's do an example. Let's suppose we have again three node RabbitMQ cluster and this time we have three quorum queues. So QA, B and C. The leader of QA happens to be on server one. The leaders of Qs B and C happens to be on server two. Now in a rolling update in a stateful set, first server two is being updated, then server one and then server zero. So first the drain script or the pre-stop lifecycle hook gets run on server two. And this will now migrate away all leaders of that node onto the other nodes. Server two is then being updated so the node becomes even more orange. And then same happens for server one and server zero. And at this point, the stateful set update completed. However, as you can see, there are no leaders on server zero. So the RabbitMQ cluster operator will exec into server zero and run the RabbitMQ queues rebalance all command to rebalance all leaders across all nodes. So that's a feature of the RabbitMQ cluster operator and there are other useful features. For example, you can pause the reconciliation if you want to. So this can be useful, for example, if our team releases a new cluster operator version which changes the pod spec of the RabbitMQ clusters that get deployed. And if you then install that RabbitMQ cluster operator, so you upgrade the operator, this might trigger a restart of all pods because this data set is being updated. If you want to guard against this, you can temporarily pause reconciliation for a specific RabbitMQ cluster. And once you're ready to upgrade the RabbitMQ cluster, you can resume reconciliation. Other useful cluster operator features are that you can enable plugins without restarting this data set. So this works by having the cluster operator exact into the pods and run a RabbitMQ CLI command which enables the RabbitMQ plugins on the fly. You can also increase persistent volumes and the operator will configure TLS for the various protocols. So we understood how we can rely on the upgrade RabbitMQ clusters. Let's next applause the messaging to podge operator. The messaging to podge operator allows to declaratively declare RabbitMQ resources. So if you look into docs examples, then we see that we can declare, for example, bindings, exchanges, queues, users or virtual hosts. So for now, let's focus on queues. So the messaging to podge operator declares a couple of new custom resource definitions. So one per RabbitMQ resource, for example, kind queue. And then there is a separate controller per custom resource definition. So for example, a queue controller. The queue controller will again watch the queues desired state. It then doesn't really diff, but acts idempotently by declaring or deleting a queue in a RabbitMQ cluster. And it does so by talking directly against the RabbitMQ management API. So again, let's see how this looks like in practice. To deploy the messaging to podge operator, let's check out the readme again. So first we need certmanager because when we deploy the messaging to podge operator, an admission web focus installed for which we need certificates. So we just use this one command here. I already installed it, but if you haven't, it takes about 30 seconds to complete. Next, we just copy paste this one command here. It applies a YAML file, which will create the custom resource definitions for RabbitMQ. So we are interested in this one in the queues CRD. It will also create the messaging to podge operator deployment. And this is where all the various controllers will run in. So our queue controller will also run in that deployment. So now that the messaging to podge operator is installed, we can create our first queue. For that, we have a queue YAML. So it's of kind queue. We give it a name. In our case, it's my classic queue. And that's important that you reference the RabbitMQ cluster where that queue should be created in. So in our case, it's my rabbit. If we apply this file, the queue controller will now create that queue. It says created. Let's check whether that's true. So if you look into the management UI again, we see that the my classic queue got created. So this queue got created by the messaging to podge operator versus the micro queue got created by a client application. And use cases of the messaging to podge operator are for example, GitOps. So you can declare RabbitMQ clusters and topologies via Argo CD for example. Another place where the messaging to podge operator is used is in Knative. So in Knative eventing, there are different types of broker and one is RabbitMQ. And that RabbitMQ gets created by the cluster operator and then a couple of exchange and queues are also getting created by the messaging to podge operator. Cool, so now that we understood what the messaging to podge operator is and how we can use it, let's see what new features got released in RabbitMQ 3.9. So RabbitMQ 3.9 was released in July this year and it supports Erlang 24, which brings about 35 to 55% higher throughput to RabbitMQ because the just in time compiler got enabled for Erlang 24. So the Erlang OTP team wrote a blog post where they test the just in time compiler with Erlang 24 using RabbitMQ. JSON logging is now also supported. So for example, a plain log line looks like the first one here where the JSON log line looks like the second log line. So for example, you can highly configure those fields but V stands for the positive and seven is just an integer standing for info. For example, eight could be debug and you can also, for example, configure the timestamp. Now the most important feature in RabbitMQ 3.9 is that Rabbit is joining the forest. So it's finally meeting the orders and if you don't know what I'm talking about, that's okay, check out the book gently down this dream from Mitch Seymour. So he wrote a nice illustration on how streaming works on Apache Kafka. Thanks a lot, Mitch. Streams got introduced in RabbitMQ 3.9. It's a new type of data structure in RabbitMQ, modeling and append on the log. It's persistent and replicated across nodes. It has non-destructive consumer semantics. So this means that previously with queues when a consumer consumes a message, the message is deleted from the queue. That's not the case with streams. Streams work with a MQP0.901 and they have a much higher throughput with a new binary stream protocol. Use cases are large fanouts. So previously, if you want to have a large fanouts, every consumer needs to declare their queue and then bind it to the same exchange. So you might end up with many queues. With streams, as you can see on the graph, any number of consumers can consume from a stream. So consumers can attach by offset or by timestamp. They can also replay messages in the stream and do time traveling and streams provide a very high throughput. And that's because replication uses the sand file system call underneath. So this means that during replication, data doesn't really enter user space and not the early VM, but they copy it from kernel space to kernel space. They support really large logs. Think about gigabytes of data and at least one semantics. There's also message de-duplication for the publishing side. So this works by having RabbitMQ remember the publishing IDs of the publishers and RabbitMQ also remembers the last offset of consumers. So if a consumer restarts, it can query its last offset where it left off and then continue from there. And that's nothing the application itself does, but that's implemented in the stream client libraries. And it also supports flow control. So again, let's check out how this looks like in practice. So we can again leverage the KubeCuttle, RabbitMQ plugin in order to create our first stream. So in this case, we have the stream perf test command and we run it against the MyRabbit cluster. We declare a single stream called MyStream and we limit the length of the stream to be three gigabytes in size. So this will deploy a pod which uses the Java stream client perf test. So if you now check the logs of the stream perf test, we see that we send around half a million messages per second. We can also check out the new Grafana dashboards for streams. So if we search for stream, then we see, and if you filled up our last five minutes, then we see that we have one stream consumer, one stream publisher, and that they publish and receive around half a million messages per second. If we now delete this pod again, and if we use the same command, but this time we specify a sub entry size of 100 and again check the logs, then we see that this time we get a much higher throughput. So we achieve around one million messages per second and this is because sub entry batching is enabled. So in Osiris, which is the log-based streaming subsystem of RabbitMQ, there are chunks and the chunk is the unit of replication and read. And a chunk consists of data entries and the data entry can be either a simple entry or a sub batch entry. And if you have sub batch entries, you have much higher throughput. You can even enable compression for messages. So sub entries increase throughput at potentially cost of increased latency. So now we read more than one million messages per second and remember that we are running against a RabbitMQ cluster where we didn't specify upper limits for CPU and memory. And that means that the cluster operator was setting some defaults for these upper limits. So right now every port has an upper limit of two virtual CPUs and two gigabyte of memory. Now imagine the throughput we can achieve with even more resources. So the team did some performance tests on C2 standard 16 instances. So those use 16 virtual CPUs and 64 gigabyte of RAM, so quite big machines. With one stream, they reached around one million messages per second and with five streams, almost five million messages per second. So there's a nice linear scaling. With sub entry batching, we reach around four million messages per second with a single stream and almost 17 million messages per second with five streams. So how can you contribute? If you would like to create a project which will have the potential to become very popular, one idea would be to write a new RabbitMQ stream client library. Pick any language of your choice. So Java, Go, C sharp and Rust are already under development. Or you could write a connector that connects from Spark, Flink or Storm to RabbitMQ streams via the new stream protocol. So those connectors or receivers already exist for MQP0.9.1, but not yet for streams. You can also choose any issue in the RabbitMQ repos and you can contact us via Slack, mailing list, Twitter or GitHub discussions. The roadmap is that we would like to introduce partitioning and competing consumers for streams, but there are no promises. It all depends on the feedback. And thank you again, Mitch, for this nice illustration. Capri is currently under active development, which is a replicated on disk database with the goal to replace Minizia. So Minizia is the Erlang OTPs distributed database. And Capri also uses the Erlang Raft library underneath. We also would like to support external secrets for Kubernetes. So for example, that you don't have to store the default username and password in HCD in plain text, but that you can start, for example, in Vault. And we are also developing a commercial RabbitMQ plugin and commercial Kubernetes operator wrapping the open source operator for multi-data center or active passive replication. To sum up, RabbitMQ is a cloud native message and streaming broker. It enables you to achieve loose coupling, high scalability and complex routing topologies. Modern queue types have been introduced. The Quorum queue based on the Raft consensus algorithm in 3.8 and streams in 3.9. You can deploy RabbitMQ natively on any Kubernetes of your choice and you can integrate it nicely with Prometheus and Grafana. Thanks for watching.