 How's everybody doing? Good? Yeah, as I am right before lunch, so I won't, I'll try to keep this somewhat short so like nobody passes out from like blood sugar or anything. Today we're gonna be talking about consistent distributed elixir. And I'm really, really excited about this because I've been working on some of this stuff for a few months now. So it's nice to finally be able to talk about it and get to share a little bit about the pain and the journey. But before I get into that, I have a problem. And this is like a very substantial problem. It's like deeply affecting my life. About two years ago now, a good friend of mine, Lance Howerson, got up on stage, I believe it was an elixir conf and said some very famous words, maybe not infamous words. Phoenix is not your application. Who's heard this? Anybody heard this? Yeah, a lot of people. And since then, it's been used by all kinds of different people. Chris McCord talks about this. We had this web directory thing and then we moved it and then people were mad so we moved it again and then people were still mad. And I think this is a really, really cool goal that Phoenix is not our application, it's just a UI or just a window into our application. That's awesome, except now I don't know what the hell my application is anymore. How am I supposed to know what I'm building? It was Phoenix, it just made sense, everything was so clear and now I don't know. So I went and looked on the most popular like hex downloads, it's poison. And so I started wondering maybe poison is my application and I've just been looking everywhere for it. So if anybody knows what it is, please tell me because personally, I feel incredibly rudderless not knowing what my application is anymore. Elixir is awesome, am I right? Yeah, I mean we're all here, this is obvious, right? I shouldn't even have to say this. When I say Elixir is awesome too, it's like I don't just mean the nice syntax over that Erlang nonsense. And I don't mean the beam, I don't mean the runtime, I mean all of the people involved in the Elixir community. It's such a fun, fun, amazing time to be part of this. And if you're new, welcome. It's, you're joining at a phenomenal point in time. Because none of the best practices and all that stuff have really been figured out yet. It's still early days. We're still able to kind of push against these preconceived notions that we have and try new things. And I think the greatest failure that we could have right now is to just assume that we figured it all out. These are our tool set. This is what we're gonna stick with. And not try new things and push the boundaries and see what works. And that's, I hope that everybody can continue to do that and continue to push the boundaries of how we work. And I think Elixir in general and the beam runtime is uniquely sort of situated to help us do that. I know for me, when I started doing Elixir, it was a total sea change in the way that I thought about programming. And it wasn't because of pattern matching or immutable data or any of those things. It was because of the humble process. The idea that in Erlang, the unit of computation is a process. It's a thing that you can start inside user land and it's how you do everything. If you wanna hold on to state, you spawn a process. If you want concurrency, you spawn a process. And that just totally blew my mind. It just completely, completely altered the way I thought about programming. One of the things that I always took away from that is Elixir just started making so many things that had been so hard before, so easy. They were so close at hand. Now as I've gotten more into Elixir, I've realized that occasionally Elixir and Erlang make some things that should be very easy, very hard, but that's a different talk. Maybe tonight, when we go out, I can have a beer and then I will tell you all about those things. But that's for later. So I wanna try to peel apart what makes Elixir and what makes Erlang so special. And I wanna do that by presenting some hypothetical problems that we might encounter in our daily life and then look at how we can leverage all these great tools to solve them. So, our first problem. We want to limit access to an external resource. Say it's a file on disk or it's an external service, maybe they're right limiting us. For whatever reason, we need to limit access to that resource to one reader-writer at any given time. In a different way of presenting the same problem, we need a lock or a mutex around some resource somewhere. This is a pretty common problem. And luckily, we're on the beam and the solution is obvious, right? Let's just use processes. Let's just throw the state into a process and then we'll be able to create our lock. And the way it's gonna work is really straightforward. We'll take just a new process. We'll give it some initial state. We'll say that it's unlocked to begin with. And then, when clients wanna interact with it, they send it a message. So for instance, they might send a lock message. It'll go ahead and lock that process and we respond back with the okay. We let the client know that they have that lock and they can do whatever it is that they need to do. When they're done with it, they send a very simple unlock message to the process. It unlocks it by switching to a new state and sends a message back. Have I lost anybody yet? We're all on the same page, right? Cool. This is like no judgment. It's cool. So the nice thing about this pattern is that this works just as well with multiple clients. So if I have another client that wants to interact with this thing, well, the first client can send a message. You can go ahead and move to a lock state and we can respond back to that client. And in that interim, if the second client wants to talk to that lock, it'll send a message and that message will fail. And that's what we expect to happen because somebody else has already got this lock. So we'll let that client know. And that client can either spin lock or whatever it needs to do to wait until the lock is released. Until finally, the first client can send a message back. We'll go ahead and unlock it. We'll transition back and then everything is good. The second client can go ahead and get the lock when it needs to. The way the code looks for this is really straightforward. We can just start with the gen server. We'll initialize our state. This is all pretty standard stuff. And then we'll just handle some calls. So for instance, yeah, we'll say that we get a lock call. If we're in an unlock state, go ahead and transition to a lock state with that client. And if we get a lock, or if we get a lock message from any other client, once we're already in a lock state, go ahead and air it. And the unlocking methodology works pretty much the same way. And I realize that that's incredibly hard to see. But if we go and if we're, if we get an unlock message, oops. If we get it, oh God, okay, hang on. Wow. All right, cool. Damn it, Cook. Come on. Almost. Let's just hop down to where we were here. There we go. Right, so we have our process. I've never actually seen keynote crash before. That's awesome. First time for everything. Okay. So here's the thing is we have a very simple gen server that holds onto that lock. And what I love about that example is it's so close at hand. It's so easy to understand how that state machine transitions and how, and we can all rationalize about that. And I'm not here to tell you that that's even like the most efficient way to solve that problem. It's probably not. And I'm also not here to tell you that mutexes are some like unsolved like computer science problem and every other language. And elixir is the only thing that can do this. But what I do think is awesome is just, it's just how easy it is to understand like what would be a non-trivial problem in other languages. We just create it for ourselves. In general, in other languages, in other runtimes that I've worked in, you don't get the ability to do this that often. And you often have to default to putting your state somewhere else. For instance, in a database or in Redis or nowadays at CD or God forbid zookeeper. And then, you know, that allows you to like maintain that state somewhere. This is great. This is a thing that is awesome about Erlang and elixir. And it's awesome right up until you encounter the next problem that most people encounter in production, which is when you need to run multiple nodes. So this is a thing that we need to do in order to have fault-tolerant systems, sort of by default. It's like, you have to do this. So we'll go ahead and see what that looks like. So we'll take our lock, we'll package it up inside of a node, and then we'll go ahead and just add a new node. And now the problem with this is gonna become pretty obvious because if multiple clients want to interact with these things, it's entirely possible that any client can send a message to one node, transition that into a lock state, and then the other client can simultaneously send a message to the other node. And now we've invalidated our original design goal, which is we need to lock access to this external resource. In other words, this is really bad. This is not what we want. And so the thing that I see people most often do when they're presented with this kind of problem, not people that I work with now, obviously, but other people, I would never say that about my coworkers who are in this room, some of them, is let's just use a global process. We'll just take those two processes and we'll just make them one and then we'll be able to always have a single lock. Easy mode, right? So to do that is pretty nice. There's already a thing built into Erlang called the global registry and we'll just take advantage of that, seems reasonable. So our original genserver.start call would look something like this and to make this global, it's pretty easy. You just say it's global. Done. And what's nice about this is we could go ahead and say like, if one doesn't exist yet, just go ahead and start it. It's guaranteed to be global, so it's fine. So back to our example here, we'll just go ahead and remove this and we'll just mark this as global and we'll say, sweet. And now if a client sends a message to the top node there, it can be looked up and forwarded on to our lock. That seems to work really well. And we can send that all the way back out to the client. And this is awesome. Of course, the next obvious thing you might ask yourself is well what happens if that node goes away, right? What do we do if this whole thing just burns to the ground? So let's find out, let's burn it to the ground and try this again. So we'll send a message to the first node. We'll look and we'll see that hey, there's not a lock here. There's no lock process in the cluster. So let's just start one. No big deal. And that way we'll be in a good state. We can go ahead and get access to our resources again. So we'll start a new lock process. We'll transition into a lock state since it matches back to the client. All those kinds of good things. And now we're in good shape again, right? My tone of voice should indicate that we're not in good shape and we're gonna find out why. Because the problem is, what if the node's not really down? What if instead of actually being down, all that's happened is that the network has gone down? I don't know, maybe AWS, like LS Therong file again, like we don't know, but there's some sort of what we would call a partition between these two networks, right? These two beams are no longer able to talk to each other over a network. Well, when that happens, the way the global works is it'll happily take your process out of the global registry just because those two nodes can't talk to each other anymore. So we'll see exactly what happens here is that node is still available by certain clients. It still has the process running, but the top node can't talk to it. So when it gets a message, it'll look and it'll say, well, guess it must be down. So I need to go ahead and start a new one. It'll start a new lock process, spawns it and it's like, sweet, all these things are good. Go ahead and transition to a lock state, send a message back to the client. Everything's awesome. Of course, the problem here is incredibly obvious, which is that that lock down there still exists. We have the same problem again. And if another client comes along and wants to send in a message, it will happily do that and happily transition to a lock state and send that back to the client. And of course, as we all know at this point, this is bad. This is not what we want. This has violated the actual design goal that we originally set out to achieve. And even more nefarious, and even just more deeply confusing problem to solve is, well, what happens when these partitions heal? What happens when the nodes can talk to each other? So for instance, this thing, somebody figures out what's going on, the load balancers come up, whatever it is, and these things heal. Well now, what do we do with these two pieces of state? We have two processes. They're both registered globally. What do we wanna, how do we resolve this problem? And a different way to phrase this might be, who should win? Which pieces of these states should win? And do we take the last one, the whoever got the lock last, and that opens a whole can of worms about like, how do you know what's last? Because it turns out time is all just a thing we made up and doesn't really actually work on computers all that well. Did you guys know NTP drifts like a lot, like a whole lot? It's really hard. You've entered into this whole world of like very, very complicated problems that have like potentially giant ramifications on your business. In other words, this is really bad. These are very hard problems. And this I think is why most people tend to default to let's just use the database because this is hard shit. Like these are hard, hard problems to actually solve. And it's so much easier if you just say, well, let's just take all this state and let's just put it in something like Redis and then we'll just be beholden to whatever, Redis is gonna give us and then we'll just do it that way. And I wanna be clear, I'm not actually saying that that's bad. I actually think in general, most people just building applications, building web apps and that kind of stuff, you are really well served by doing that. Postgres is a great database. Put data in Postgres and you can use all their hard work, all the time that's been spent into making that solution really robust. That's not, I'm not interested in telling you that you're all doing it wrong or that databases are bad or we should all avoid them. But what I am interested in is having options. I want there to be a way to consistently manage state in Elixir and that's the kind of future that I wanna push towards. For no other reason than I think it makes us better as a community, having that as an option. And maybe that's wrong, maybe we should all just be using Postgres and Redis but I'm interested in other and alternative solutions. So I've said the word consistent and unfortunately you can't have a talk about distributed systems at all. I think it's actually potentially even legally binding without bringing up the most misunderstood, the most overused buzzword in all of distributed systems, talks, blog posts, everything and that is, anybody guess it, the cap theorem. So if you don't wanna hear about the cap theorem, I'm sorry. We're gonna talk about it a little bit because it is useful to kind of frame the conversation here. So if you're not familiar with the cap theorem, what it basically says is when a network is partitioned, you can either be available or consistent and we're gonna define what available and consistent means. Before we go any further, I wanna make it really clear, I'm gonna make it clear again. The cap theorem only applies on that first sentence there, on that first line there, when a network is partitioned and that's gonna become really, really important. But let's define what available and consistent are. So available, available means that every request receives a response without guarantee that it contains the most recent write. To put it a different way, given a partition, given you have two nodes, those nodes can no longer talk to each other, they will still service requests from a client but they give you no guarantee that it is the correct data anymore. It might be old, it might be out of date, whatever the case may be, you don't get that guarantee, you just can't know that. But you are servicing requests. Consistent, on the other hand, means that every read receives the most recent write or it errors. So if we're partitioned, at some point, we have to say we have to return with an error, we can no longer accept reads and writes. And we typically talk about these systems as being either AP or CP. So AP meaning available during partitions and CP meaning consistent during partitions. And again, I really wanna emphasize, it's only during partitions that any of this stuff applies. And also I should be very clear and say, in most real world systems, this is not actually a binary. Like most real world systems fall on a spectrum in between these two things where you tolerate certain amounts of inconsistency, availability, et cetera, et cetera. But we're gonna use these binaries because they're useful as a way of discussing systems in general. So let's look first at what available systems might look like. So we'll come up with a new problem. And I think this problem will help demonstrate how we might architect things. So let's say that we need to keep track of counts, just a generic count of something. Maybe it's add views, maybe it's click rates, maybe it's whatever the case may be. The way we can do that is really simple. We'll take two nodes and we'll have a process in each that initializes to plus zero. And now it's important to note, we're not initializing it to zero, we're initializing it to plus zero. Plus zero is going to be a function, right? It's gonna be a way to keep track of the actual count of these things. And what we'll do is instead of just storing the actual counts, we'll store additions to them. So we're gonna actually store addition operations on each one. And that's what that's gonna allow us to do and it will become clear in a second is it's going to allow us to roll up and provide a materialized view of the actual count. So what can happen here is that a client might say something along the lines of like, okay, add plus two to this. We just sends the message to the node, the node stores that addition and then replicates that addition to the other connected node. And it'll also be stored. At this point, if the second client does a read operation, we go ahead and send that over. And then what we do internally is we take all of the additions that we have and add them all together and we return a real value. So in this case, it's going to be two. And we send that back to the client and everything's cool. Now, when there's a partition between these two things and they can no longer speak, let's see what happens. So we'll send a message to the first node with let's say plus three, send that on over. It'll get stored and then the node will attempt to replicate it to all other connected nodes in the cluster. When that happens, it'll hit the partition, that message will be lost forever. And during this period of time, it's entirely possible that that second client issues a second read request to the first node and of course, what they're going to get back is two. Now, the data here's stale. It's wrong for all intents and purposes, but it's okay, it's servicing requests and that's what's important here. They can also issue writes. So for instance, if we add five to that node, it'll get stored off. And now our data across these two nodes is well and truly inconsistent. I mean, that's no other way to put it. It is inconsistent here at this point in time. Read request going to either one will get out of date data. But the nice thing is about building systems this way is that when this partition heals, all we have to do is we have to ship the data back and forth. I say all we have to do, this is actually kind of a hard problem, but we ship data back and forth. And when we do that, any additions that we're missing will get replicated to each node and now we'll be in consistency again. And you'll often hear this referred to as eventual consistency because we eventually get caught back up to the correct values at a later date. But I really wanna emphasize here, sometimes you're gonna be wrong with a new architective system this way, but that's okay. A lot of real systems are built with this trade-off in mind. The canonical example for this tends to be something like Amazon's shopping cart, where it's always up, but it's maybe not always right. Maybe you deleted something but didn't actually get deleted from your cart. Everybody ever experienced that one? That's by design. They're up, but they might be wrong. Another maybe closer to home example of this is Phoenix Presence. Phoenix Presence is based on CRDTs and Gossip and the way it works is exactly like this. It shares data and for a period of time that data could just be wrong, but that's a perfectly, perfectly reasonable trade-off for Phoenix Presence to make because it lowers operational costs, it allows nodes to come and go, it makes deployment easier, all these kinds of things. So it makes sense to start to look to those sorts of things. And I would say too, if you're thinking about using Phoenix Presence for a problem, if you can make your problem fit that shape, it's a great tool, use it, go for it. There are, in my opinion though, and I would contend a lot of us have these kinds of things where some problems need stricter consistency models. And I think for many of us who are coming from using database-backed stateless applications, we're already using stricter consistency models in general. And it's much harder to bolt your problem onto an AP solution. So other examples of this might be distributed locking or databases or distributed scheduling and coordination, configuration, stuff, transactions. These are things that come up in real-world systems where we need stricter consistency guarantees. So let's look at what it might take to build a more consistent system. So as we said, partitions in a consistent system can eventually, well, I guess let's put it this way. Let's look and see what happens when we have partitions in a consistent system. So right now we have three nodes. They're all talking and collaborating and replicating data very nicely. And one of them gets partitioned off from the other two. When that happens, that node has to stop accepting reason rights. It's not down in the sense that it's not offline to any client connected to it, but it has to know and has to start returning errors for its reason rights. And eventually, given enough node failures, your whole system eventually has to stop accepting reason rights depending on the amount of nodes you have and the way that the partitions happen. The way it ensures this is by a sort of category of algorithms known as consensus. And consensus sounds exactly like what it is. It's a bunch of computers all arriving at an agreed upon value. That's not the actual progenitor of all consensus algorithms. There are things that predate it, but the paper that tends to get the most credit for a consensus is Paxos. And this was written by Leslie Lamport in the late 90s. And it was paper called the part-time parliament. And it was written by this guy right here. That's not Stevens-Bilberg, although it looks a lot like it. And Leslie Lamport's contributions to computer science are just mind-blowing when you look at it all. Beyond distributed computing and those sorts of things, anybody ever heard of law tech? Anybody know what the law in law tech means? That's that guy right there. And he created this whole thing called Paxos. And he wrote this amazing paper that was couched in this kind of interesting metaphor-type story thing. And he was like, on the island of Paxos, we've discovered this way that they had consensus across these different islands. And he often would show up to class dressed as Indiana Jones to explain how it worked. And I'm not sure if the costume helped or not, but in any case, no one understood how it worked. Like literally no one got it. And so a few years later, he wrote a follow-up, this time called Paxos Made Simple, as a way to try to explain and recouch the Paxos algorithm. And that was okay, except still, like no one could understand it. And so just many years later, there was this paper where it lovingly titled in search of an understandable consensus algorithm, the extended version. And this gave birth to what we know as Raft. And this is the algorithm behind SCD, console, a whole host of other databases and sort of infrastructure tooling. And for the past five months, I've been building it in Elixir. And I'm very, very, very relieved that it's, we're at this point where I can actually talk about it now, because it kind of works. And that's very astounding to me. We're gonna dive into what the Raft algorithm looks like, but before we do, I wanna show example of how we might use this library. And so we'll go back to our original problem, which is we need to limit access to an external resource. So we'll just take the code that we had before, our GenServer here, and we'll just do a little fix up on it and we'll turn this into a Raft state machine. So you can literally just take GenServer, we'll say use Raft state machine instead. And the init, we don't actually have to say okay anymore, we can just return our initial state. Those handle calls cease to be handle calls and instead become handle writes. And they only take a message in the existing state of the state machine. And in this case, we don't have to reply, we just return a to tuple. And we return the, excuse me, we return what we wanna return the first element and the new state and the second element. And we can give it this similar treatment to all of our other callbacks. And I'll just breeze through this and so I think that's probably incredibly hard to read anyway. When you actually wanna start this thing, you just have to say start peer. You have to give it a name and then you finally set the configuration. And then once you've done that, you can issue writes to one of the nodes. And you can issue all kinds of writes. This still maintains the same semantics that we have with our original lock. And we can continue on and now we have a truly distributed replicated lock that provides consistency across all of our nodes. And I was gonna have a video of that, but screw that. Let's just do it live. And maybe keynote won't crash again. And we'll just go for it here. So we'll fire this up. Sweet, okay. It's all still working. I'm sorry, like this still makes me so giddy. Like, I'm not kidding, it was like five months. So what we have here is we have right there. And then the three panes on the far right, the top ones are leader and the bottom two are followers. We're gonna describe what that means in just a second. But what's important to note is that that top node is the node that is taking all of our reads and writes and doing the replication to the other nodes. And so down here on our client, I'll make that a little easier to see. And we're gonna issue some commands from here. So what we'll do is we'll say, we'll give it the name of our node, or a peer, I guess I should say, at the node. And we'll ask for the leader and we get that node back, sweet. And now we can issue some writes. So we'll just pipe this into Raft.write. And we're gonna send it a lock message. And we're gonna say that we're client one, just by saying C1, C1. Sweet. And if we do that, we get our okay. You can see that it logged it and it wrote a new command to the log and it's already done its replication. And now if we try to write from a different client, let's say, we get an error. And if I take down this node, you can see, yep, the election's already happened down here. And now we can ask one of our other nodes for who the leader is. So we'll ask S2. And we'll attempt to do our write again. And if everything works, we should get back okay and error because somebody else already has the lock. And sweet, we do. Excellent. That makes me feel so good every time that works. And we can also go ahead and do an unlock command so we can like keep this thing moving. Let me clear this so it's easier to see. And now I'll go ahead and unlock this. And it's all working again. And if we bring this node back, if we bring that first node back up, we'll all be good. So yeah, so that's it. Sweet. Oh gosh. All right. Thank you. Oh man, if you wanna talk later about what it's been like to work on this stuff, please, I will tell you all kinds of stories. But so how does this thing work in general? So we start off with a cluster of nodes. They're all just standard nodes. And what happens is that amongst themselves, they elect a leader. They elect someone who's going to accept all of the reads and writes. And when they do that, the other nodes fall into what's called a follower state. And their jobs are only to accept writes and write them to a durable log on the disk. So a client comes along and it wants to send a message. So it'll send it to the leader. The leader's job is, like I said, to replicate this message. It'll send it to both of the followers. They'll also write their message to a durable storage. And then they send it back to the leader. And only once they've done that entire operation can we say that this log message is committed. And once it's committed, we can then apply it to the state machine that we define as users. And once we've done that, when that's when we respond back to the client, we say, hey, yep, that worked out. Intermittently throughout this entire process, the leader sends out heartbeat messages, which both keep followers logs up today and also assert their sort of leadership over the other followers. So it sends these out. It sends them out like a 50 millisecond, kind of configurable timeframe. And then they get those back to the leader and everything still works out. In the case of a partition happening between all these nodes, the leader will attempt to send heartbeats, but it won't obviously be able to reach the other nodes because the other nodes are partitioned away from it. Given a random timeout that each of the follower nodes has on it, eventually one of them will timeout. And what it will do is it will start a new election cycle. It does that by first transitioning to a candidate state. It votes for itself, and then it sends out requests for more votes from all the other nodes that it can connect to. And if it gets a quote unquote quorum, otherwise known as a majority of nodes that vote for it, it becomes the new leader. It kind of lets all the other nodes, it lets all the other nodes know that it is the new leader and it starts accepting messages. In the event that the client sends a message to the old leader, it will attempt to replicate, but because it can't, these messages will just fail and it will eventually have to timeout and say, okay, well we're not the leader anymore because we can't communicate with anybody and it follows back to a follower state, sends an error message back to the client, and then everything kind of moves on. Meanwhile, the next, the client can kind of say, okay, well who are the other nodes in the cluster? Let's send a message to them and said, figure out who that leader is, it sends it over to there. That leader will then, of course, replicate. It'll replicate to a majority. Once it has a majority, it can commit it and send that back out to the client. And then, of course, once this partition heals, the old leader, the original leader from the very beginning of the example, it just falls back in line and the new leader catches it up on any log messages that it's missed. So it brings its log back up to date and rectifies any errors in the log. So I keep talking about logs. The log is really useful because it linearizes all of our writes. It's replicated across all the nodes and as long as all the nodes have the same logs, then we're all consistent. And under the hood, we actually use RocksDB to persist this. That's one of the reasons we ended up building, or ended up building this myself is because I wanted to use a proper database engine to actually store all these things. So we use RocksDB for this stuff. RocksDB is really great because it allows us to do snapshotting really well. It handles anti-entropy, anti-... Like, it handles things like corruption, which other databases that may or may not be built in to Erlang don't handle as well. No shade thrown. And it just takes a lot of, like does a lot of heavy lifting for us. This is all tested using a property test, which you've not done much property testing. That's a whole different talk into itself. And I'm here to tell you property testing is incredibly humbling. It finds all kinds of fun bugs. And if you're not property testing stuff and you want to add a lot of resilience and robustness to your application, you know, property testing is the way to go. And we're also currently writing the Jepsen Suite for this. So we'll actually be able to prove linearizability once this is all done. Well, prove it. Probably what we'll do is we'll prove that it has bugs and then we'll fix those bugs and then eventually we'll be able to prove linearizability. So what can we do now? So we did this and what good is it? Well, the thing that I'm really excited about, I have a whole list of things that I kind of want to work on and they tend to be more infrastructure things. So for instance, I really want to start working on a KV store to be able to have a consistent KV store that can be distributed across nodes. I'm also playing around with some ideas for service discovery, distributed lock management stuff, configuration management, some transactions. And we're just sort of playing with all these ideas and I hope that you all can also take these things and you have your own ideas and how to like take this stuff and start building cool new solutions with it. All of this is a part of a new organization that I'm helping start with some other folks called Tonic where we're trying to build a really, really robust distributed systems toolkit for people doing elixir work, leaning more towards doing somewhat consistent type solutions but we also have some good libraries out there for doing event sourcing and CQRS. So if that's interesting to you and you want to talk about that, please come up and talk to me. And we're just sort of trying to provide a good toolkit for everybody to just use as a jumping off point. If you want to check out any of this stuff, the links are all right here. I'll leave this slide up so if you want to take pictures or whatever you can go check that stuff out after this. Obviously like if you just want to come talk to me, please do because I'm happy to share the pain and also just like talk about where we might go from here. There's tons and tons and tons of stuff to work on still. The big things on our to-do list are just like more testing, probably better operations documentation and playing around with some ideas of using LMDB as the storage adapter. LMDB has some really nice properties that we might be able to leverage for this kind of stuff. And like I said, I hope that with this kind of toolkit we'll be able to move forward and for the problems that need it, we'll be able to build systems that manage state safely inside of Elixir and just enjoy that nice Elixir robustness that we've all come to really appreciate. So with that, I'll just say thank you and I hope you have a great rest of the conference.