 Hi, everyone. Are you OK? Nope. Tired? I will try to be short. So my name is Pavel. There is James. We are from Last Mile Tech team. So after we received the route from the routing team, it's actually time to start deliveries. And Alex gave a good introduction in how we generally do this. But in simple words, we have an app. Every driver has our own app. He loads the routes. Then he goes to the cross-dock in his van, finds the orders that he needs to deliver, loads them, and delivers to each customer. But of course, life is not ideal. And the weather as well. So we have all sorts of problems on the ground, such as heavy rain, traffic jams. Or sometimes it's not easy to get to deliver locations, especially in the morning. And given that, we have pretty high on-time delivery KPI. We try to deliver pretty much every order on time. We need to have real-time dashboards to see what actually happens on the ground. And act quickly before customers get unsatisfied. We need to have event-driven decisions to make them real-time, send SMS, do some kind of adjustments to what's happening. And we need to have a data warehouse with all the events happening on the ground with drivers and with deliveries so that the routing team and everyone else will be able to get their insights. So the idea is to get all the events, basically everything that we can from the device, from each device, starting from location, GPS, accuracy of GPS, battery, device information, transport mode. On the vehicles, we have devices that produce engine on and off in the events, GPS also. So we want to get all the events into one system. And so we, you know, everyone knows what is CRUD, right? It's one of the most popular ways to build web services. It's actually very good because it allows us to start services very fast and deliver them to production quickly. But the problem with them is, sorry, yeah. So the problem with these services is, I'm very sorry, I have this new technology that I'm trying. And it's not very, the problem is that it allows you to do updates, right? So when you do updates, you actually delete the data. And it doesn't help you to store everything that you have. So if you want to have a change log, then this change log should be written manually, right, in a separate code of the database. And you need to maintain it for every event. If you want to do an event-driven system, you need to generate events every time manually. So we decided to try ACCA for this project. ACCA is a toolkit for building reactive applications. Everyone knows what is reactive. Basically, it allows you to build responsive applications which answer fast, which are elastic, means that they can scale horizontally easily, which are resilient, which means that they are not failing on simple errors in the code, and which are message-driven by design. So ACCA implements ACTA model. And ACTA model is a different approach. If object-oriented model says that everything is an object, ACTA model says that everything is an ACTA, which means that the driver or order or a cart in such system will be an ACTA. ACTA actors are very lightweight. So it's only 300 bytes of heap in Scala and Java implementation. They communicate via messages. And this is the only way to communicate between actors. And they don't share the state, which means that if actor wants to know something about other actor, he needs to send a command and wait for the response. So you've probably heard that WhatsApp uses ACTA model. It was first implemented in Erlang, a WhatsApp user lang, and in WhatsApp, every user is an actor, and they have more than 300 million users right now. So why we decided to use to try ACTA? Because first it's in Scala, and we use Scala and Java in languages in Redmart. Mostly on the single machine, it allows to send over 50 million messages in a second between actors. So it's very performance. It's distributed by design. It allows you to do clustering out of the box. It's fault tolerant. You can easily handle all the exceptions and not fail the whole system. And it allows you to implement a web sourcing, which I will be talking next. So this is Scala. I hope everyone can understand. Let's try to see what happens here. This is a driver actor. The simplest driver actor, and what we try to do is to have a real time view of the driver of his location. Location, it might be a GPS coordinate. So it has a receive method, and we receive update location command with the location, and we just update the state of the driver. And also we have a get state command, and it returns the current state. So I think it's quite simple. And this is pretty much everything you need to write to get a real time view of the driver. But the problem with that approach is that actually if you kill that actor or restart the system or just do a deployment of new version, you will lose all the data. And it's not what we want. We want to keep everything. So here comes persistence, acro-persistence. It's a wrapper, and it supports pretty much every data store that you can name, Mongo, Cassandra, various SQL servers, even Kafka. So here is the driver actor for particular driver ID. We need to define the persistence ID that will be used to store all the data related to this driver in the data store. Now we have a receive command method. But instead of just directly changing the state of the driver, which is somewhere around these dots skipped on this slide, we first persist, we generate an event from the location, then we persist it, and only after that we change the actual state of the driver. This is a call to the method update state. So what is important here is when the actor is killed or the system is restarted, we have a method receive recover, which what happens is we create an actor, we load everything from the journal. Kafka does that for us, and it feeds the stream of events recorded in the data store in the sequence that they were generated, and it sends it to this method. So the problem here is that it may be slow, but the great thing is that we have stored all the events, we have stored everything, and we get the latest state of the actor. But we can fix it easily with snapshots. So for example, if we configure a snapshot to take a snapshot every 10 events, we don't need to replay everything. So for us, if we replay all the events from the journal for each driver, it will be thousands of events. And if, for example, we just redeployed our application with new version, it may take several minutes to just restore the state of less drivers. It doesn't make any sense. So we, of course, use snapshots. So the only change that you need to do is if you put your own rule here, if you need to take a snapshot, you just take a snapshot. And then in receive recover, you create another case, another matching to the snapshot, and you just use it. And it's absolutely seamless. So I mentioned about event-driven system, event-driven logic. How can we do that? Let's say we have, again, our driver actor, and we want to do to execute some logic. So for example, we want to send this message to a web socket and display it on real-time UI, which we show to our transport coordinators. Or we want to do some other processing if required. For example, we want to see, we want to generate some, another different state of the driver and notify someone if something wrong happens. So we create this listener. We subscribe to driver-driver-id events, or all driver events if we want. And here in the driver actor, we just, after we have persisted an event, we just publish this event to the so-called mediator here. It's also out of the box. Then we have this real-time state of the driver. And how about querying all the drivers? So if we stop right here, and we'll only use what we have, what I've just described, you will have to query all the actors, all the drivers, in parallel or in sequence. And it will give you the real-time state of each driver. But it's not very efficient. Maybe it's even slow. So you may have heard about an approach called CQRS. It's pretty old already. So it stays for command query responsibility segregation in separation. So command side, we just discussed. Again, we get the command via HTTP interface or WebSocket or whatever. We verify that this command is valid. We generate an event from this command. We store it in the journal. And then we change the state. This is a command side. But then we need a query side where we can actually see the driver state in the database. So as you see here in the journal, we don't have the driver state. We just store the differences, the events. And on the query side, we want to see this driver state and display it in some typical CRUD interface where, sorry, just view interface with progenation, for example, with all the drivers and their locations. So the most difficult challenge here is how to actually generate this established disconnection between journal and query side. Again, ACA helps us. The idea is just to stream events directly from the journal. What happens is we read events from the journal as they appear, one by one, real time, without any delay. Then we give the driver view, whatever it is, from the data store. We transform it using the new event from the journal. Then we insert results back to the database or update it in that query side. And then we store the offset of the journal so that if we restart this process, we will continue from the last successfully read journal entry. This is the code. This is pretty much everything you need to write. We use persistent query framework from ACA, connect to Cassandra Journal. We run the start processing method, get all driver updated events from the saved offset. We buffer them by 100 entries or 10 seconds. And then we just run process event function, which does all the transformations or writes directly to the preferred data store. And then we just save the offset of this process. And that's all. So now the question is how to scale everything. I mentioned that ACA allows you to do scaling out of the box. So here we have ACA cluster. It's also provided out of the box. It has one coordinator. And on each node, you run a shard region for each type of the entity that you want to track. So for example, actor 1, 2, 2, 6, these are driver actors. And we run each driver shard region on each of the nodes. And because actor 2 can send a message to other actors on other nodes, of course, we don't want to model the communication between drivers. But let's say we have another entity, like driver actor may want to send a message to another actor. Because driver actor doesn't know where this another actor sits in which node. He needs to communicate via shard regions. So he sends a message to shard region saying that I want to send a message to the actor with this ID. Please route this message. And through coordinator, a local shard region knows to which node it should be routed. And it happens seamlessly. So this message is delivered to another shard region. And then it goes to a target actor. So that's the message that I wanted to deliver you today. If you have any questions about this technology or other questions, I will be happy to answer.