 So, hello. My name is Malin Olson. I am a member of the Redis open source project. I'm also an engineer at AWS. And today I'm going to be talking a little bit about how to use Redis in cloud-native architectures to build an asynchronous messaging system. Before I talk too much about the agenda, like the motivation for this talk is I know a lot of people use Redis. Just kind of like quick show of hands who here is aware of what Redis is. It's like everyone. Who here thinks Redis is just a cache? So it's really tailored for you guys, which is that a lot of people know, just like they don't know what else Redis can do. And they often think about it's just like an ephemeral data store. You throw data in it and it can go away. So the specific use case I'm going to dive deep into today is the event broker kind of design pattern. So I'll start kind of introducing that, explain how it works in traditional service-oriented architectures. Then I'll kind of give a pseudo-introduction to Redis. It seems like most people know, so I won't go too deep. But kind of talk a little bit about why it's good at what it does. Not talk too much about what it is. And then kind of put Redis in to this architecture we're talking about and see how it actually kind of fits that role. And then I'll show kind of a simple toy application, kind of putting everything together. And I'll close up with some best practices and then we'll hopefully have some time for some Q&A if people have questions. Great. So why do people need message brokers? So in traditional service-oriented architectures, you basically have a lot of different services and they're all talking to each other with tightly coupled APIs. This works pretty well. Some people I know, Amazon is well known for having this very complex service-oriented architecture because they used to have this big monolith, had lots of problems, so they broke everything up. So service-oriented architecture is great. The problem starts to stem when that starts to get very complex, right? When you have one service calling lots of other services, you end up with dependency hell and a very high latency as all these services start chaining together. You also start seeing issues with being able to maintain SLAs as when one of these microservices failed, the whole architecture might stop working. So the idea behind that to solve this is either a message-brokering architecture. So the idea behind here is you do all your synchronous work and then you just record an event that all the other services eventually read and then they take their actions based on that. So this breaks the tight coupling. You still have some coupling since these messages need to be still well formed, so all of the downstream consumer services know what's going on, but it solves kind of those two major problems we had before. So the first is now we get back to having super low latency. Since all we have to do is do our synchronous work and then record the event into the message broker. And then we also have much better API SLAs since we're just now dependent on the synchronous work and all of the asynchronous work kind of can just happen in the background. If a service dies, it doesn't affect the main APIs. So if you were to have an e-commerce website, you want to make sure those orders are going through and if you're doing some of the non-essential work that can always happen later. So when you have a message broker architecture, it becomes very easy to scale and add more services in since you just need to find how these services consume sort of from this main message bus. And for the rest of the talk, I'm going to talk about these message bus in two different forms. The first is what I'm going to call an authoritative log. So an authoritative log, you basically have a single stream of every single message that ever was produced as part of this message bus. And so if you just want to speed up a new service, let's say we have some new business intelligence that needs to know, like, what's going on, it's able to basically go back in time, start from the beginning of the log, consume everything to the head of the log and basically bootstrap as a new service. So this is one type of good use case. The other really common use case I've seen and it's pretty common with Redis and with a lot of our customers is what I call like ephemeral logs. So the best frame for this is that you could lose a message. And so that's, a lot of people get kind of scared of that. They're like, why would I want to lose a message from a log? And the best place to think about this is if late events are the same as the message not showing up at all. If you have some type of, you know, maybe you're delivering some type of package and that you're trying to give like real-time updates to the end user, you want to give a real-time update of where that package is. If you're going to deliver two days late, they don't care anymore. You can just have basically the same as dropped it. There's some other interesting use cases like maybe you're okay with inconsistencies between your services because you have some background reconciliation. Storing data durably is relatively expensive comparing to storing stuff sort of as a best effort. So if you're okay with periodic drift and you have some mechanism reconciled, then that's a good maybe use case for having something more ephemeral. Okay, so those are sort of like the main basics of how you have a message broker. So what type of data storage do we want to use for this? Ideally, we want to support those two different types of cases, the authoritative log case and the ephemeral case. We want to be highly performant and scalable. Those are, you know, cloud native. We want everything to be performant and scalable. And we want to be able to support different types of events. So not something that's too specific to one individual use case and can shard to different individual types of logs. You don't want to be, if you have one specific log that holds all your data, it'll eventually become the bottleneck for your service. And of course, we're at Kubernetes conference. We want to be industry standard and we would love open APIs. So with that, I'm going to suggest Redis as a good way to solve that problem. I know a couple of people said they didn't know what Redis is, but Redis is a fast and simple in-memory-first data store. It's completely open source. As I mentioned, I'm one of the maintainers. We love active, we have a very active community. We're still building a lot with inside the, we're still releasing new versions. So the main trade off the Redis kind of thinks about is that it stores everything in memory first and then optionally will persist stuff to disk. So it doesn't try to expose many trade-offs for end users, right? So it's not one of those applications that will try to make it as easy for the consumer to, the client to use, but it wants to make it like predictably high performance. Redis is also well-known to be very flexible, has lots of data structures, and we'll be talking about one of those specifically today, and then it also has high availability and various forms of durability. So the main framing I want us to use for Redis today is that it's that, it's that basically a, it's an API for shared data. And you can build applications on top of it, right? So if you have one client who writes data, everyone else can read from it. So that's the one frame you take away, like that's what I kind of want for the rest of the talk. I know most people, you know, raise their hands when I mentioned Redis as cash. This is what by and large everyone uses Redis for. The idea being that since Redis is in memory first and not necessarily durable, you're able to do expensive operations, take the result of it, and then stick it in Redis. And then when your application goes and tries to do that same operation again, it's able to first check for Redis. So this is by far the traditional use case for Redis. And this works great without durability, without replication, and fits into a lot of service warranty architectures. The use case I'm trying to push more for Redis generally is this idea of beyond caching, which incorporates a bunch of different aspects. But one specific one I want to talk about is data projection. So this sort of fits into this message book we talked about earlier, where you have a primary database, and then it emits a series of logs based for the updates from that, like for mutations. And the rest is able to consume those updates. And then project some form of the data into Redis. So the authoritative data still comes from Redis, sorry, still comes from the backend database. This case I'm using Postgres everywhere. And if Redis were to die, you'd basically be able to go back to the Postgres database, re-project all the data. And eventing works really great in this case. So one of the things that's nice about Redis is periodic backups. So you can basically take an entire snapshot of the data set, save it to disk, and then restore that snapshot. So if you keep track of where you are in the updates from the message bus, you're able to restart from that point in time. So Redis is already great in message bus systems. And I like to argue that it can also be great for the message bus itself. So what types of functionality does Redis have to build messaging? So there are two, there could theoretically be a third, but there's usually two main use cases that Redis has that people use. The first is what's called PubSub. So publish and subscribe. Clients are able to subscribe to channels inside Redis. And when messages get published to those channels, Redis then broadcasts the messages to all the various consumers of that channel. So you might see here this doesn't work at all for authoritative blogs. It's really just targeting that sort of ephemeral workload. And it's really targeting just making it as simple as possible because there's no data being stored within Redis. It's just immediately broadcasted out. So it's really good for the normal thinking about Redis where you don't really have any type of schema. You just push data in, broadcast the data out, and then kind of forget about it. Fire and fire gets often used to describe PubSub. The second day of structure, which was released in Redis 5, which is like four years old now, is what's called Redis Streams. So Redis Streams took a lot of inspiration from Kafka and is basically a pend-only log of data. So this sort of fits more into our paradigm we just had of an authoritative blog. So you put a bunch of events in Redis and then you're able to just cooperatively consume them through what's called consumer groups or an individual client can basically keep track of where they are and then keep reading from that point in time. This data will need to eventually be trimmed out of Redis. Cool. So how do these two actually look? Most everything in Redis uses commands. It doesn't have any query language like SQL. So in this case we have a command called xad which is responsible for adding data into Streams. So you have a stream name and it has a corresponding unique ID that's generated. You can also ask Redis to generate a stream ID automatically for you when you push the message. So in this case we're pushing a message that says message hello into a stream called myStream and it produces a unique identifier when you push into Redis. And then the consumers are able to pull this data out. And then also as I said earlier there's x read if you're just consuming it yourself or you can cooperatively consume stuff. The demo I'll show in a second is going to show this cooperative consumption of data. And then similarly we have the PubSub system which does something very similar but in a sense kind of in the reverse. So consumers are the ones that start the transaction. They say hey I would like to subscribe to this channel. And then when messages are published into the channel everyone's able to get basically how many messages were broadcasted and everyone that caught the consumers. So with these two data structures we're able to basically stick Redis where it was before. Where the message worker was way back when they were part of the stock. So Streams provide the good authoritative log as well as can be used for the ephemeral logs that we talked about. And PubSub can also then be used as part of the ephemeral use case as well. So we mentioned before that we also needed some type of partitioning of Streams. So Redis has a deployment called Redis cluster mode which is a natively sharded deployment. And it then partitions the data by the CRC16 of either the name which can either be the channel name or the name of the Stream. So that's all well and good. But there is still an elephant in the room that I haven't really talked too much about. It's not Postgres. I do like Postgres. Postgres was known to be the most loved database which stole away Redis' title. But it's actually durability of data. So most people think Redis is not durable. And for the vast majority of people that actually use Redis it's not. It's not. People deploy it without any mechanism of strong durability. So the default configuration of Redis is that. And if you had Redis replicas even if you had the built-in mechanism called AOF which is the append only log. Most people use AOF without a very specific flag which is F-sync every right. Which means that before we acknowledge any right to a client we will actually make sure it's persisted to disk. And even in that case if someone persists to one disk it's not super durable. So even though most people don't use res durably it does have ways to be used durably. You can set up in such a way so you actually have syncing to multiple different disks. And then you have to build some type of mechanism to restore that after node failure. So Redis is not particularly great at being durable. But it can be. So it can be used as its authority of logs. It's still good at that high throughput, high efficiency type workloads. So I will argue that it can be used for that. And in many cases using a durable store for an event bus is better than using an ephemeral store because it makes it simple. You don't have to worry too much about messages not being delivered. But a lot of event bridges inherently assume like some type of idempotency or having to think through you know how to handle at least once or at most once type of messages. So the ephemeral workload still you still have to solve a lot of the same problems. So even though if you're not using durable res can still work pretty well as an event bus in either mode. And there is also multiple managed providers that provide Redis like API that do provide durability. So if you don't want to self manage durability there are there are still options and you're still able to you know do all the local testing because you have the open source project to test against. Cool. So best practice is for Redis at scale. As I mentioned briefly Redis has a shared nothing partitioning scheme when you use it in its cluster configuration. You can typically get between 100,000 150,000 operations per like process per core. In typical non-durable configurations it falls to between like 60 to 100,000 operations per second in more durable configurations. But you can scale out to about a thousand different shards which give somewhere around like 50 to 150 million operations per second. Which is pretty much higher than anyone I've ever seen practically use. We have a couple of customers AWS who kind of get close to those numbers. But practically it there's plenty of scaling room as long as they have a good way to shard the data. Starting with Redis 7 we actually introduced what's called sharded PubSub in the olden days before Redis 7 there was no good sharding for PubSub messages. So in Redis 7 now it's more or less been fixed. That was contributed by AWS. And then also if you want high availability for your data a high availability is not as important for event buses because if you kind of take an average for a while and bring up a new node it will still like you have that built-in slack because you don't need the synchronous operations so it's not as important. But you should understand like when you actually do need it. And if you think about if you're adding replicas you're really optimizing for that case when failures do happen and 99% of the time they aren't happening so you're paying that cost. So just be very deliberate about choosing to run with replication. And then the important thing about Redis is it does not try to solve hot keys or hot channels for you. So if you do end up funneling too much data into one shard that shard will get overwhelmed and you'll have problems. So and Redis as I said before just doesn't try to solve those problems for you. Then as I mentioned before the when you're using an authoritative log eventually you'll need to trim data. So in that case you'll typically take the hot data and keep in Redis. Redis is as I said memory in memory first. All the data is stored in memory so you don't actually want that because that's expensive so you eventually want to tear up the disk. So most people running with this will want to have some type of system to take the data from Redis in memory and then spill it to disk. Ideally magnetic drives SSDs can work but it's usually not as needed. And then time-based events for ephemeral logs can usually just be trimmed based on capacity or time kind of based on what you want. The Redis snapshotting mechanism I talked about a little bit earlier can also work kind of well here. If your data is set up in such a way that the events all sort of can get combacted together and put into the end at like an end state like if you have multiple different values and you're like continuing updating it Redis snapshot is probably a good way to sort of compact all the logs events together so you can kind of stick on disk and then restore that. That's a little bit more specific and it depends a little bit on your use case. So I've been talking a lot about just about Redis but since Redis is an open source project and is one of the most downloaded Redis, sorry Docker containers, it has a lot of support for all the CNCF projects. I'll specifically be talking a bunch about how it integrates with Prometheus, what type of metrics to look at to make sure it doesn't kind of explode because Redis says it's like it's a lot of people think it's kind of like a race car like can go really fast but if you kind of drift a little too much it just spirals on and explodes. So it's very important to make sure you kind of understand those failure modes. Cool. So that I'm going to walk through a short little demo and kind of walk through how you can actually use Redis in this configuration. If it works there we go. Cool. All right and I have internet so this is all working. Cool. Everything's connected. So in this example there'll be three main components we'll be talking about. There's going to be a web app, a queue that's consuming data, and a generator of data that will throw a load in a second. I actually don't want this thing generating consuming load for the whole second so we'll turn that off. Some people might ask, one of my favorite things is just that like I love Kubernetes it's so nice to use. I've been so long as a database engineer that I forgot that some things can actually be nice writing in C all the time. I'm used to things being painful. Cool. So the main part of this is we have a very simple web application. It's built in Flask and so you know we have some configurations at the REST connection. Yadda yadda yadda. If people want I could I can I was planning on posting this at some point in the future just to sort of emphasize this. So there's a bunch of constants up here they're good to know about. All of our events are going to have a very specific schema. You should probably use something more like protobuf or something but I wanted this because it's a little bit more readable to at least follow and once you use protobuf everything kind of gets lost. So this is a very simple web API that basically implements an order functionality. So we do some you know unmartialing of the data to get the arguments out, create an object on top of it and then all like this is the main operation we do we just do an xad. You'll see I do in a batch. Redis is a little bit more performant when you send a bunch of argument commands together. I do this just for some monitoring aspects later. In a newer version of Redis it'll be much easier to see the size of streams but right now there isn't so I have to keep track of that all manually. So not super important but as you can see here the only thing we're doing in this order operation is just putting an item in the stream. In the real world we probably want to be doing other synchronous work making sure we have capacity making sure the user is authenticated but that's all you know business logic we'll figure that out kind of in a second. And what are we doing with this? We're putting in a stream and then we'll have two consumers for this. The first consumer will be showing the last item this user has purchased and the second is we will be showing the overall top sellers for this given operation. Okay so let's quickly make sure this is all working, not that one this one. So let's buy an item make sure this is all working. Okay and so we got the basically the stream ID that was generated as well as the size of the queues that were generated. So I mentioned before all this data is being ingested into Prometheus. I can prove that it's you know actually through Prometheus it's over here everything's up. We have a Redis exporter which is generating metrics from Redis and then we have a custom exporter in that web app that's basically showing latency. So we have one item in the queue it's not being processed it's been unconsumed. We have a log size I was putting some data in here earlier so it's non-zero but it's there and we're seeing latency about four to five milliseconds. A lot of people are used to Redis being sub milliseconds but I'm using Redis in a more durable historic configuration so we're actually running the three separate azs inside AWS which takes a little bit more time. So that all looks great. Oops not there so I'm going to go ahead and also going to start putting some load on here. So that's load. I'm thinking what I want to show is I want to show that this kind of the queues kind of keep growing and then once everything is all scaled up we'll add some consumers of the data. I'll have this refresh a little bit more regularly. Oh yeah one other thing I wanted to show you was that I can prove that this actually hasn't been consuming yet because this history is still empty because we the consumer of the data not consume the data. So while everything is starting to start up I'll quickly show what the consumers look like. Consumer. The consumers are really simple. So we have down here we have a function that we're calling basically when an item actually gets processed. So we're using we're still using rats for everything because why not. We're using a sorted set to keep track of the top selling items and then we're just storing the last purchased item in a hash value. So going back and talking a little bit what we said before we're going to use the X group commands which are the cooperative consumers so we can have multiple different nodes consuming multiple different nodes consuming data. So being very unreddice like you actually have to call an explicit function to do X group create and have to like catch exceptions if you do it multiple times. Typically reddice is very schema-less but not in this case. We're still trying to work on a better way to do that but we'll figure out eventually. So how do we actually consume the data? We have the X group read command so we have every stream so we have a stream name here and then we have the worker name which is the so the group is created on a stream and then you read from a group and so that makes sure that every message within the stream is only delivered once. So lots of fun stuff here there's some just plumbing basically bootstrap a node if it hasn't consumed anything this just means start consuming from the beginning and then we're consuming up to 500 elements and we're weighing one second in case there is no items currently in the stream. The other interesting case we have to cover is when an item gets every item that's read from a group has to be acknowledged and if an item is dropped we need someone to go and claim it. So we have a separate command called X auto claim which will allow you to actually claim commands which aren't owned if they're idle for at least one one second. And this does mean in this configuration we're doing it most once since a node could have read an item gone off you know down whatever then disconnected for more than a second. Kubernetes might have not killed it let it keep running and then it will come back and acknowledge that it's done the work. So in this case everything should more or less be item potent. So let's see do this thing run. Okay so at least we know everything's now running. So let's go back look in here. So we have we can see that the number of elements is steadily growing with the log size keeps growing up and up and up. We've been continually adding items to the log and the unconsumed message kind of ticked up for a second and that's because we started the producers faster than the consumers. And it kind of came back down. It's been hovering around zero not quite zero. And we'll say also this metric is not quite accurate but I was too afraid to fix it right before the demo. So it's it riffs a little bit. And then this pending Q-Size shows the number of outstanding messages. So the one last thing I kind of want to show is if we go and then kill the let's increase the number of load generators because we want to show that if Q starts growing over time you kind of want to start setting alarms on that. I talked a bit about how to make sure you're alarming. You definitely want to make sure you have if you're not consuming messages you need some type of alarm on that. So it depends on you know you should figure out your application to figure out how fast you can consume messages and figure out how long you're willing to tolerate messages not being consumed because once the Q-Sar gets too long it will take a long time for it to actually drain back out and you can have other problems as well that aren't immediately obvious in your system. There's actually been a lot of very notable AWS events, AWS outages that are kind of basically this problem where you have a huge backlog of items and you're not able to effectively process them. There's one interesting paradigm inside AWS where if you're falling too far behind what you should start doing is instead of process first in first out you should do the opposite start processing last in first out. So you're actually doing some successful requests and actually kind of getting latency down otherwise you might just kind of keep getting further and further behind. It's all starting. You should have probably figured out that are serious to actually do this. Yeah these are all still creating and maybe we'll come back to this in a second. And I can quickly go and catch the last of the best practices. So yeah this is the last thing I want to talk about. When building message brokering systems the most important thing is you really need to understand what needs to be done synchronously. In this web app example we were talking about like no user likes to get a successfully purchased and then found out oh no async could fail because we didn't actually have capacity. So make sure you know what needs to be done synchronously, what needs to be communicated to users. Since if something is failing asynchronously you can actually know how to resolve that because you didn't give a synchronous error to users. And make sure you test failure modes. As I said this is a pattern which really helps but it also can be a huge door on the side. Reminds me a lot of caching in that case. A lot of people have failures because their cache falls over and they storm their back in database and sad things happen. So just like I feel like in a lot of places with Redis just making sure you understand your failure modes and you test them regularly. So make sure you understand your failure and you test them regularly. In the example we're just kind of talking about to try making sure like just turn off all your consumers, see what happens when you restart them. Cold starts are nice and easy to do in Kubernetes for the most part. Another important thing is to like really make sure you're avoiding poison pills. Make sure you have like schemas for events and that everyone is like adhering to them. Your messaging system should validate that schema and not accept messages that don't conform to it. Once you have like a poison pill in your system it can like kind of bring everything down. I had a bug earlier when I was building this demo where I actually put a character like a s inside the user ID when I was expecting everything else was expecting it to be purely ints and then nothing was accepting it and it was took a long time to debug. And I would have helped if I actually had been properly validating stuff instead of just shoving in Redis. Monitoring we already talked about this a little bit make sure you end up burst capacity to handle whatever type of load you want to do. Especially in the Redis case when Redis can actually burst into memory make sure you have enough like spare memory capacity. Queues are typically a good use case for Redis because you usually don't consume all that much memory. You kind of like you add items or move items so memory stays pretty constant. But make sure you have enough to actually burst into. The last thing is message partitioning. We talked a little bit about how to name stuff in Redis using like CRC16. It's still important to understand like how you like what's the right dimension to partition your messages on. Most people do it by something like a user ID or a customer ID. But there's other ways as well like you know based like just logically grouping stuff instead of doing something like a hash on top of a customer ID. So that's all I have for you at the moment. I believe I'm supposed to show you a QR code and give you something. So I'll just briefly say like thank you Madeline Olsen. Follow me on Twitter. Here's the QR code if you need it. And yeah I think that's all I have. Does anyone know how to like questions? Do people come and ask questions? Is that a thing? Sure. What's up? I have no idea. Does anyone else have examples or do people just go and mingle? Oh yeah. Microphone's nice. I, it's not on. Or to go to Redis? Okay I'll repeat. If we're already using Kafka as a message broker for the same purpose is there a reason to to to change to Redis or is just a matter of preferences? For the most part I've seen like Redis, so Kafka really only sports the logs more of and it stores everything durably. If you don't need that durably that's the biggest reason I've seen people move away from Kafka because Redis tends to be more efficient and has lower cost to operate but I would say it's a lot of that preference. A lot of people already have Redis in their deployments already and people understand it. It's kind of nice to use one system for multiple things as opposed to the kind of purpose built things everywhere. So I, but yeah, these ultimately more or less a preference thing. Yeah. So thanks for the presentation. So when the pubs up messages are being stored as I'm assuming is also stored in the Redis. So what happens if I have a bus and I have this loss of messages that are being generated and that's causing a huge memory usage. So what would be your recommendation around burst capacity planning and what are the things that we should plan memory disk and things like that? Yeah, especially for memory. As I said, it's kind of really comes down to your capacity planning. At some point you should stop accepting messages if you're running, if you have no memory like spare memory to actually like put them. So typically you want to start like you want to reject them and have the back pressure push it back to the front end. So it likes things stopped adding more data in. So you'll want to for capacity planning, that's I mean, so it's very system dependent. AWS we usually do like 10x kind of what we expect to be the worst case to make sure we have capacity for that. In this case, it's much easier to super over provision. This is it's almost message queues are almost always CPU and throughput bound not memory bound unless you're like really far falling far behind. So try to figure out like how like how much time are you willing to keep accepting rights before you kind of have a hard outage is another thing we do at AWS. Like for we have like when we do like authentication we have like six hours. So we usually say you need at least six hours of reasonable extra capacity before you should start rejecting requests. But obviously that becomes very different if you're actually very constrained on memory or disk. Then you might need to keep less surplus. For disk, yeah, you have the same problem if you're using with AOF. You should figure out how much you're willing to spill before you start actually rejecting the front end apps because most of the time message workers are used to keep availability up and have the latency. Does that help? Yeah, we'll try to get you a microphone. Unless you want to yell and I can repeat it. That's fair. That works. What recommendations do you have to ensure durability for streams in the event of node failures? Can you say that one more time? Yes. What recommendations do you have if you want to have durable streams that tolerate node failures? So for example, like if you're a cloud provider or if you want to upgrade nodes and that requires node restarts. And if you want to maintain a given stream, like have it still remain available, what recommendations do you have for that? Yeah, so typically you'll not just want to do a node restart on a node that doesn't have any backup of the data that will lose the data. I think AOF is usually the best thing for that because you can take a synchronous flush basically of the data and restart with it. Some of the managed providers like do that all for you. So I mean, I basically just don't restart the node, have some backup of the data and restart the node. Have some backup of the data. I think we're out of time. So thank you all for coming. If you have more questions, I'll hang out and thanks a lot.