 A topic today, the Scala plus ACA, different kind of ACA stuff, how it's used in real-world project and to share the experience and some knowledge. So a bit about myself, I'm from Ukraine, this is a country nearby Europe. So my experience is 18 years from now, 12 years I spend on .NET stack. Now I've shifted for the last three years, I'm working for Redmat, now I'm tech lead work for full human team and it's GDM and Scala and this is my point of interest, this kind of system design and all of this stuff around. And of course functional programming and Scala as a semi-functional language. So let's start. This is our agenda, I will explain a little bit about the system itself, so it's Redmat capacity system, in particular full human center capacity system, I will explain it later. We will immediately take a look at the end architecture, so to bring the end design and then all of this downstream is the, I will explain piece by piece, all of the architecture. And at the end we will take a final look with having in mind all of this explanation already. Again this is a practical implementation, so of course there is a theory and there is a practical decisions which are sometimes not following exactly the theory, yeah because it's real world. So let's start, first of all of course is, let's say business need. So the goal is all of the capacity system in Redmat, describe that but let me first introduce what's actually how it works from customer perspective. So Redmat is online groceries, so we have app, we have website, you can order groceries. To do so you first pick your cart, you fill up your cart with the goods which will be delivered from the warehouse, then you suppose to select the delivery slot, which is two hours when you want it to be in front of your door. So to respond to this query and to show that actually from customer, from customer side via other services, we are not going to explain that, it goes to capacity service and capacity service actually have to check whether fulfillment center can fulfill your order, so people in the warehouse can pick your goods, put to your thoughts, put them to the thoughts, then hand it over to transport system and then transport system delivers it to you. So the purpose of the capacity system is to give the response whether your order can be fulfilled from both transport and fulfillment center to be fulfilled and deliver it at the selected time slot to your door. So this is the idea of the capacity, the overall capacity system. Today we are going to consider a part of it, which is in red, which is fulfillment center capacity system. So it's actually physical warehouse where all the goods is there and it's a response to the request whether your order can be fulfilled from capacity, from fulfillment center perspective. We are not touching the transport capacity, we are not touching the topic how they matched together in the capacity service. It can be another talk and we have one guy from the capacity team, maybe somewhere he will be like next, next time, yeah, presenting all the systems, yeah. So today we focus on that. In short, what's actually the system do is stated there. So actually it has one, okay, this is simplified amount of request which is service, it's more of them. But for the purpose of today talk, I cut many, many, many things which are not necessary to explain. So in general in simplified version, we have to get requests to this service from the different other services which is capacity availability from this service, it's get request obviously and we have some management UI which shows what's actually happens down there. So and we have post request and delete request from same service. Whenever you make a reservation actually, so you pick your time slot, this service responds, okay, this time slot available. Then you pick this time slot, okay, I want to be delivered here. So to this service, post request goes, okay, we reserve this capacity for you. So this is what post request reservation means. It's actually makes the reservation. Then if you change your mind or we have other circumstances when we want to cancel this reservation, we have delete request for this, okay, delete this reservation. Free it up, oops. That's fine, yeah, that's fine, I will update the, okay. And we have same kind of request from management UI. The heaviest one is capacity and the second one is when we want to update its request for managers when they actually set how much capacity do we have available for customers. So this is what management things does. So a few requirements and observations. First thing, it's customer facing real time service. So no downtime allowed and the response must be very fast. So this service supposed to respond within 50 or 100 milliseconds, no more. Next observations. So all the requests for available capacity, which is get, it happens several times more often than the reservation or reservation deletion. This is first observation. Second observation is that available capacity get request and get for management UI is pretty heavy in size and it takes time to build it, relative take time. So of course we are talking about 100 milliseconds, but still it affects. And a special available capacity available for customer, it happens all the time. Whenever customers search for delivery slots, they all the time bomb the service to read this information. Yeah, so this is like a high, very high level explanation of requirements and business needs and what actually capacity staff does. And in particular what we are going to discuss today is the architecture of fulfillment center capacity part, which is capacity service. Of course it's a microservice architecture, so all of this is microservices by definition. Everybody knows the definition of microservice, right? So as I promised, we immediately take a first look to the whole architecture design. I understand that maybe many things is not clear, I will get through it very quick. So we have of course in front of this service, we have load balancer, we have aggregate nodes and we have query nodes, so it's implementation for CQRS. And load balancer immediately split the queries, post and delete queries, which makes modification, they send it to these aggregates nodes, they are in red. And all the get request goes to query nodes, which are green. All of the nodes tied together with Acca cluster, so it's cluster. And all the requests HTTP, HTTP routine, all of this stuff is served by Acca HTTP, so you can see it everywhere. It's an event sourcing system, so the so-called command actor sends all the events, stores and we use Cassandra as a backend for that. And they are replayed with Acca stream to the query nodes. So whenever this guy shuts down or this proxy guys, I will explain why this is a proxy and what this means that it's not a proxy, it's actual actor, I will explain it later. It's a cluster singleton. So whenever crash happens and it needs to be up again, so it reads the same events, replace the events and become alive again. So this is very quick. Now we are going through each and every piece of that, I will explain piece by piece. And then we will take a look at the same picture at the very end. So first thing, let's forget for a moment about the whole architecture and take a look at the actual business class, which reflects the business logic. So the class, we call it schedule and I will use this term throughout the presentation. So this is just a case class. It's not an actor or nothing like that. It's just a case class, which keeps all of this structure as it stated, it's schedule. So whenever we want to reserve to make a reservation, it marks in this schedule, in certain point, I won't explain how it's organized, but it's three-level structure, pretty complicated. It's big enough. It keeps seven days in memory, seven days of current available capacity to be booked inside. So this class is pretty big, relatively big. And it's just a case class to keep all of the fields or all of this data. And it's extended with traits, which the approach calls mixing, with all the behavior. So you can see many traits, each of them implement its own behavior. And there are key points here. So first of all, this schedule, yeah, as I said, it's represent the whole schedule. It's one instance of the class that does everything. Operation modifier as I say here is a pure functions. So by definition, meaning that each time you call it, it has no side effect. Which returns, and this is the second key point, the new instance of the whole schedule. So it's implemented in pure functional manner. So we are safe to, like, when we want to make a reservation, let's say this is one of the methods from a schedule reservation trait. The schedule reservation trait has only one public method and other methods are private, so it's pretty clear. So when we do reserve, it returns a copy of the schedule, at least semantic copy. We don't know what's happening inside. There is some black magic maybe, but we don't know. Symmetrically it's a copy. It's an important point for the rest. Again, delete reservation, it's also, when it finds the reservation, it deletes it in the structure and it returns the copy of the schedule. So this is important points about the implementation of this class. Of course, the pros of that is that it's covered with tests without any actors, without any, all of this burden on top of it. We can just purely test, let's say, unit test this class. We can easily extract it and put to other architecture, so everything is here. So this is all about the business logic on this high level. Of course it's lots of details, but we're not going to cover that. This is key points. Next, from the below, from this business logic class schedule, we are going a bit higher, one level higher, is the actor which covers this as a state. So for just a reminder, what is actor mode in general, and key thing is like state is hidden inside the actor, nobody has the direct access to it. Only actor itself and only the messages which comes to actor goes only via mailbox, so there is no concurrency, there is no race condition inside. So this is the idea, general, very general idea of actor model. In particular, this is ACA actor, for those who doesn't know, very, very brief. So whenever we want to create actor with an ACA actor, so we just extend actor which is provided by ACA, we have, let's say, state here, this is var, meaning that it can be changed. And we have this receive method which receives the messages which can be any type, in this case, in this example is strings, and all the actors live within actor systems. So actor system manages this actor, there are lots of things you can do with them, but this is just a very, very simple example. This is this instantiation of actor ref, so this guy is just a reference to which you can send the messages. This is an example of how you can send the message. So I send this message using tel pattern, which is fire and forget, oops, I send it to this actor. So actually this function got called, and this is what happened, I changed the state of the actor. Whenever I can send this message as many times from different places, it's guaranteed because of the mailbox down there, it's guaranteed that there will be no race condition accessing to the state. So this receive method will be always called one by one. This is the key thing. This is another pattern of calling it, it called ask. So this guy returns the future, and you can get the response from the actor. This is the implementation of that. So we change state, and we reply with something else, with anything else, back to the caller in this case. So that's how we, so when this future completes, it will return you this yo string. So yeah, this is very brief about how ACCA actors works. Of course it's much more complicated, but it's just a brief introduction for those maybe who don't work with it every day. What else? Yeah, one thing here is when you response with something, or when you send to the actor, there is a serialization happens. So sometimes I will get back to this later. So there is not that straightforward if we are talking about distributed systems when actor resides in some other place in the network. But yeah, we'll tell about it later. So this is actually how Scariel actor is implemented in this case. So we have Scariel as I described before, it's our business logic class, it's a state. We have this simple receive method. Of course the production code is much more complicated, but it's just an idea. So we can send, let's say reserve message, then we modify the state. We can send delete reservation method. We call the delete reservation and we, as I said, the schedule whenever we call the method, any method it returns its copy. So I just substitute the state with a new copy which is created from that. So it's very, very simple. And the key thing here is there is no race condition, as by definition provided by actor, so it's direct implementation. So we are safe to do this change of state, no race condition. Okay, moving on, more interesting thing, even sourcing. So this slide is almost from the documentation or from ACA persistence. ACA persistence is obviously the implementation for even sourcing. So whenever we want our actor to be a persistent actor, we just extend the provided persistent actor where we have to specify the persistence ID which will be used on the underlying database, state is the same, this function is for convenience, and these two is the key to explain how persistent works. So let's first take a look at receive command. So it's actually the same receiver which receive events. So whenever we receive a command message, so here we must clearly separate commands, messages and events. So whenever we want to change state, it's obviously a command, for example reserve. So when we receive that, first thing we do is, in this case in this example we persisted, then in case of successful persistent, we do update the state, so we modify the state, then we optionally can write the snapshot. Snapshot is just a snapshot of our current state, it's just a full serialization and save the current state. From even sourcing theory, snapshot is not there, it's just an optimization thing which comes from real life. So then let's assume our actor shuts down, then it's up again, it reads the events, not the commands but events, these guys. This function got called, it's out of the box from persistent actor, and it just replace these events from the underlying database, one by one in the same sequence as it was stored in receive command. So you can see that it's the same update state called, and if snapshot happened, we just assign this snapshot which is deserialized from the database, we just restore it to the state. Typically what happens, so when we modify our actor with commands, we store the events in the database, then after 100 events we store snapshot, then we continue to write events, then actually let's say it shuts down, then it gets up, so it goes to the last snapshot, not the old events, but on the last snapshot, replace some amount of events, not all of them from the beginning of the lifetime, it will be strange. Okay, so this is an overview of how ACAP assistant works, very, very high level. This is how it's roughly saying implementing in a particular case. There is one tricky thing here. So again, it extends persistent actor, and we receive reserve command, so we want to make a reservation to the underlying schedule which represents our business logic. So what we can, we actually do. Everybody says the following, theory of event sourcing says the following. First step you do, you check, you validate your command or event, you validate it first, then you apply, then you persist, or when you persist and then you apply. So validation step is the key thing, because if you persist the event which you cannot apply, then during the reply, it will never get up again. This is, yeah, it will be recovering forever. It's crashed, try to replay, crashed, try to replay, crashed, and in the infinite. So this is a trick, how can we tackle that? So first thing I do, I under try monad, I call this reserve. If it succeed, and I got a new schedule, so try success and failure is a standard thing, and I got new schedule, at that point I'm already sure that it will be applied. This is like application and validation at the same time, because from practical world we know there is no validation that can guarantee us 100% of application of that. So here I'm absolutely sure that it works. So then I produce the event, please take a note that we have reserve command, it's important, reserve command, but the event name is reserved, because we further will replay successful events. We are not replaying commands. Then I do persistence of this event, and this function is called then and only then the persistence successful. As soon as persistence successful, I change the state. I have everything here. I apply, so I validate, I validate the application. Then it succeeds, then I persist, as soon as it succeeds, I change the state. Everything, everything clear. Of course, in case of failure, nothing happens. Neither persistence nor changing the state. So that's fine. So this is like normal flow. Then when actor crashed or we just restarted normally, receive, recover. This is a function as I from the persistent actor, it's overwrite, and this one, this one is not overwrite, this overwrite, so receive, recover, replace the same events I stored here. So what it does, it just applies the event, knowing in advance that everything will be successful. I don't check anything here. Of course, this code looks like duplicates, right? So I applied it here, I applied it here, but this code is just for simplicity. The factual code is actually refactored properly. So it should be one function where we apply all of this stuff. So it's just for understanding. Another special event happens here. So these guys, it's your events and recovery completed, provided by a persistent actor. It's for you to make something as soon as recovery state completed. It's a separate state. The key thing here that unless recover completed, no other messages will be delivered via normal flow. This is the key thing and one of the key things for persistent actor persistence, because you know, right? So actor starts to getting up, normal events comes, and at the same moment it's recovering. So they must not interleave each other. Yeah, mailbox guarantees us no race condition, but we have to recover first and then receive other events. So actor persistence does that for us. We need to bother about that. A couple of implementation details of event sourcing. So persistent storage we picked is Cassandra. It's obviously Cassandra cluster. In our case, it's four nodes if I'm not wrong. And serializer is another thing. So when we store the events to the storage under the hood, the serializer is called. If there is no serializer provider, the persistent calls Java serializer, which is like heavy and slow for high performance system. So typically a better serializer is taken. So we took Creo plus Creo is a Java thing, so we need to pick something for Scala. I picked Romix extension, but we were also considered a Twitter chill as an extension. So, sorry, why did it appear? Yeah, so alternative serializer can be Creo plus Twitter chill, which is recommended now. Those days I picked Romix as it was the best. Or if you consider that Google Protobuf is also fine. It's also recommended. But I don't know how to work with Scala. I haven't tried. Yeah, this is a few implementation details here. Moving on, one of the important points when we are talking about event sourcing in general, in this case in particular, is the visioning, is the pain point for event sourcing. So when you store your especially snapshots, which is a serialization of your huge state in this case, and then you change something in there. So when it tries to replay it, obviously with a new version, the old versions will fail. So there are a couple of strategies which are recommended to apply. In most cases, when you change dust, it feels like it's nicely implemented at Google Protobuf. You can extend your serializer where it can tackle it easily, relatively easily. But in this case, we went to the simplest approach. We just named everything we store. We named it as V1. It's also a valid approach. So it's the simplest one. So whenever we want to change something, we have to create V2 and provide the adapter. This is a pattern for event sourcing. When you provide the adapter, when you see the old version, then you now have a new version. So adapter must be applied from previous version to current one. Theory says that your service must be infinite, backward compatible with all of the events, versions. Of course, it's bullshit in the real world, so nobody does that. So they think how can we cover that and we did that using snapshots. Again, there is no snapshot in the official theory of that, but of course it's real world, so we have it. So obvious way is we play with snapshots, we can get rid of the old versions and have only two latest ones. Yeah, so this is what we use. Next point, aka HTTP. So let's remember that for a while. We'll get back to this later when we grab everything together to the whole picture. Now let's focus on another point, which is aka HTTP. So as you can see in the picture, all the incoming and ongoing HTTP requests and responses are served with aka HTTP. So this is an interesting thing actually. Aka HTTP is, you can implement with it all the interaction, all the HTTP stack interaction actually. It has pretty sophisticated and interesting DSL for that. The key thing though is based on aka actor and aka streams inside. So you can naturally use these two things. But the key thing here is that it's not a framework. It's something, let's say, a bit lower level, but still sophisticated enough and has enough power of DSL and all of this stuff to tackle these things. So they stated this as here, I copied it from the documentation and I like it as general toolkit. So you can use it whenever you want. It doesn't restrict you with anything, but it's not that low level that you are supposed to go down to some very, very technical stuff. You can do that, but in general, you don't need. If you want to find a web framework, for example, in a post to that, like play framework, if you need it, then yes, you can take the play framework and use play framework here. But for this project, it's definitely too much. We don't need this play framework with all of the unnecessary thing. I prefer to keep it as small as possible, so we use this thing. This is the code example how it works actually. Of course, it's much smaller than an original one. So we define routes with DSL. So in this case, we have path, which is reservation. It goes right after the domain name or IP address, right? So it's slash reservation. Then we state that it's post. And what actually happens when, here we state what actually happens when post request slash reservation comes, post request. So this guy immediately deserializes the request, the payload is a post, right? So it's payload there. So it deserializes this to capacity reservation type, which is DTO type for accepting the messages. And now you work with these deserialized objects. This is case clause, actually. You can provide this serializer and deserializer. You want this serializer and deserializer, because this guy definitely is JSON. So it's deserialized from JSON to this one. And in very convenient manner, you continue to work with the instantiated object. So here, what I'm doing, I create the message, which is command message reserve. I send it with ask button to the schedule actor. I request the result. And when it's completed, when it's complete with success, I return. The complete is from from HTTP as well. I complete it with created, which is 201 HTTP code. And I provide the response. Response is my case clause again. And it's get automatically serialized to JSON and send back to the caller. So this is how reservation implemented in this HTTP manner. So next, this is routes. You can put them whenever you want in your code following your rules. Then you, in the main service, and at starting point, you actually bind HTTP. You bind it to your host, which you want to listen to, and you to the port. I should have put exact numbers here to be clear. It's actually constants from app config file. So it's exact values, so not a magic. And I put there the API routes variable. I just provided there. So that's it. And this is future. So as soon as it's bind successfully, future return success. So I just log that. Okay, it's binding successfully. From now on, it's listen to the port for the incoming request. That's it. So this is all about HTTP. It's as simple as that. It's just a toolkit. No need to use everything if you don't need anything else, like play framework or something else. Very lightweight. And the code is pretty clear. I like it. This slide is just where we are now. What we discussed, where we are now. So this is the enlarged, we have five boxes on the diagram. This is one of it, which is enlarged. So we discussed HTTP. It receives request. It converts the HTTP request to actor message, as we saw in previous slide, light message. Then we send it to command actor. Command, our schedule actor converts it to the call to underline state. Modify the state if needed. Then it returns with ask button. It returns the result here. Returns the result. And result gets sent back via again, aka HTTP as stated here over there. It returns back to the caller. During the change state, as I explained before, we persist the event, which corresponds to this command message. We persisted to using aka persistence to the Cassandra cluster. If this is actor already. If the actor died or restarted normally, all the events will be replayed out of the box by persistent actor. And the state will be recovered. And normal processing will be recovered. So this is where we are now. This box I showed here, this box, is this one. A bit more details. OK, moving on, aka cluster. So cluster highlighted in green and tightened together all of those guys. I will explain it later, but in each node of the aka cluster you can have a role. I have two roles here. I will use it later. I have aggregate role, which I stated the name. We have a modifiable state here. And I have query role, which doesn't accept the modification commands. It's just for querying. So I will get back to this later. Just remember that. So cluster, aka cluster thing is an extension. And it's set up like that in the configuration file. Key thing here is the seed nodes. So whenever actor system starts up, you use this construction, which is extension on top of the system. It reads this configuration. So whenever node starts, it reads this configuration and tries to pin other seed nodes to join the cluster, to form the cluster. So that's it to set up the cluster. I will explain a bit later what we are doing with cluster. So the thing we need from cluster, we need to know what is the size. We need to aware of how many other nodes we have, how many nodes of particular role we have. I will explain it further. So next is special extension of cluster is singleton. So what's actually happening? This singleton is not a generic solution. So cluster can be used more or less a generic thing. So this thing is used particular for this business task. So a singleton extension is about a group of nodes has only one instance, as stated in from the name as singleton. It has only one physical instance of an actor. So a schedule is placed only physically, only one node. The other two nodes, in this case, we have three of them, other two nodes are serving as a proxy to this one. And as soon as it's clustered, they know about each other. So this extension automatically. So when the first node starts, it already knows that it will be the singleton, actually. When second node starts, this one, it joins the cluster using seed nodes because this one is seed node. It joins the cluster, but it's already know that the previous one already started. So it knows that it's a proxy. So whenever call happens to this actor, call means command message. Whenever it happens, it just sends this via wire. It sends this to the factual actors, actually. And this node as well. So this singleton actually is like recommendation to either not use it at all or to use for some configuration stuff or use it as rarely as possible. So it's not like a pattern or something recommended practice. It's for this particular business case. So we cannot split this state. The reasoning for that is we cannot split this state any further. We must keep it together because this actor actually represents our house. And we have certain resources to serve the customer. And we cannot split these resources to split the actor over the other nodes. It must be together as one instance. In this case, we use it purely for failover. So whenever this guy crashed, this recognizes that because it's in cluster. It's in a few seconds and knows that somebody crashed. After, I don't remember, two seconds or something, it takes over the responsibility and becomes the actual actor. This is the purpose of singleton. Again, we use it for failover to fast recovery and to maintain the almost close to zero downtime. When this actor goes down, not because of the crash, but in normal way, the handing over the responsibility happens very fast. This couple of seconds delay happens only when this node crashed. We use AWS. And it's actually not a really case when it's crashed. So I mean, not really, it's once per several months, for example. So it's a pretty valid case. We have two nodes here because I will explain it later the split brain problem. So key takeaway from here, so for this purpose, for this task, we need a single actor. And this, we use singleton for failover, for quick recovery in this case, in case of crash. Yeah. Actually, this is the code how singleton set up from this is a bit messy. But from this service perspective, from the code perspective, when we set it up, this guy, this service itself, it doesn't know whether this actor proxy or not after setting it up. So after cluster is up, from this RK HTTP, when the message is sent to the actor ref, this link doesn't know whether it's proxy or not. It happens under the hood. By the way, this sending, yeah, it's a bit overhead for the traffic, from traffic perspective. Yes? So question for my understanding. So for all reservation operation, you use only one actor instance. Physical instance, only one. Reservation and deletion. And all a few others. All the reservations happen on one single actor. Yes. So actually, this actor represents a warehouse. The direct response to the warehouse. It can be split, actually, inside the warehouse. It can be split to chambers. We have a few chambers. So it's actually possible to split even this. But it means that there will be three or several sets of singletones per chamber. But yeah, direct response, yes. At the moment, we have one physical actor to serve reservation and reservation deletion. Does it mean the full concurrent request gets deleted to the same instance? Are they queue up? Yeah, they queue up, because it's actor. This sending, it happens via actor model anyway. So when reservation request comes from here, here, and here at the same time, it's guaranteed that they will apply it here somehow one by one. We don't know the exact sequence, of course. But it's guaranteed that there is no concurrency, because it's still covered under actor. This is key thing here. Can we have the bottleneck? Can you just follow that? Yes, it's a bottleneck. But we cannot do anything with that. We cannot afford having them at the same time. But you said you split them by chambers. What does it mean by some departments? Yes, inside the very house, we have frozen, which is minus 20, right? We have fresh, which is minus 5. We have tea or something, which is normal temperature. We have others, chambers. So the very house, it's not a bottleneck, actually. There is not much load. I have sort of a better answer. So I've done load tests recently on these sol systems. And basically, current implementation takes like five times our peak load. So it probably doesn't make sense to further complicate the setup, since it's already covered well beyond what we need. Yeah. How many chambers do you get them? Five. Five? Five or six. So every time somebody puts something in a spark, or doesn't make a schedule, they actually fire commands to one warehouse specifically. Yes. How many events do you have specifically on one warehouse? Because I can imagine that we're easily talking to millions of events here, right? No. I mean, for reservation? For reservation, no. There's sort of a risk. It's not like every time you pick one item into your cart, there is an event or a command sent here or something. Yeah, every time you want to actually, yeah, to ship. Check out. No, when you make a check out, when you do check out, a reservation comes, right? That's it. Do you see the snapshots in the actual replay? Snapshots happens once. I'm not sure how it goes now. Once per 100 something. Reservations. But you do use them. Yeah, of course. What happens when the persist fails? Do you lose the event? Persist may fail during next event, right? Or what do you mean by fail? So the event comes, and you say that in the actor, you first have to persist it before you update the state? You need to persist the state, right? Yeah, I applied, but not change the state. So I have a copy. Then I persist. If persist and successful, then I take this copy and substitute the state. But if persist fails, then that means you lose the event. And that never happens. Yes, never happens. Yeah, you lose the event. It's crashed. Everybody gets up in the night. If there's a bug in persist, you will lose all your events. Yeah, we have backup policy for that. And this is a Cassandra cluster. So it provides some features. Right now, as your consistency you use a warehouse. Yeah, at the moment, yes, that's enough, more than enough. Why did you choose a warehouse versus an individual item? Like, is there a requirement that your warehouse is consistent? Because it's warehouse capacity. It's warehouse capacity, yeah, right. So you have a certain amount of people who can serve it. It's physical people. They can serve your order, your order, your order, your order. So that's why it's had to be concise. You cannot split in this level. Again, we can split it with the chambers. But we did this performance stuff that we don't need it at the moment. Yeah, theoretically, you can split it further, but then sort of making sure it's consistent within itself requires a lot of interact or communications. And then it's like super complex. Currently, it's not needed. We will consider that we are actually, we'll probably expand soon. Yeah, but significantly, we'll expand soon. We can split by days. So you don't need to. No. We cannot, yeah, good question, by the way. We cannot even split it with days because the picking happens overnight. And it's spread over the night. And the so-called picking waves, when people work, they overlap and overnight. So everything tied together. You can split by picking waves. No. I think we're discussing Redmond, the team model not. We can go further, but direct response, no, you cannot. Sorry, one question over there. So you've got the cluster. Let's say your singleton went down. Yes. So then it takes a few seconds to switch over to the other one, right? Depends on the situation. If it's crashed, it takes two and a half seconds. If it's normal shutdown during the update, it takes immediately. Yeah, because it sends the message, I'm shutting down. So the next one, just take over immediately. If it's a crash, let's say it takes about two and a half seconds. Yes. What happens to the request in the meantime? Is that a downtime seen by the clients? During those two and a half seconds? This parent capacity service will retry to do that. It gets the error. Yeah, it keeps retrying. So retrying mechanism, of course, implemented on a higher level. In practice, requests that hit the node that crashes, they are lost because they are assigned to the next node in the crash. Everything else is rerouted to other nodes, and then they just wait for a couple of seconds to get this. And for messaging, obviously, you are using ACCA everywhere. For messaging? For within the service, yes. Between the service boundaries, it just should be for professors. Yeah. Yeah, so this one. Key problem in clustering thing is split-brain problem. Split-brain means that within these clusters, we don't have any supervisor or some central thing which knows about everybody else, all the nodes. So all the nodes, they know only about each other. This is the key problem. So there is no supervisor. This is the key thing of distributed systems. So whenever we lost connectivity, let's say, network partitioning happened, and part of the pieces of the cluster doesn't see each other. So the question is what the rest of the nodes supposed to do. So if we are in these nodes, the only thing we see that we understand that we don't have other nodes, that's it. We don't know what happened. Whether they will get back or we lost them or whatever. Same happens here. So from these nodes, we don't see these nodes. That's it. There is no supervisor. So there are actually no magic here. There are a few resolution strategies which can be applied for those who are not aware of this problem or haven't read for a lot of time about that. So this is a standard one. So static quorum, this first one. If we know the cluster size, then we can actually realize whether we are still in quorum. So going further, we implemented the static quorum actually because we know that we must have at least three nodes. At least, sorry, two nodes. So static quorum size is two nodes. So as soon as we appeared here and there are still two of us, we are OK. As soon as we in this part, and there is only one of us left, we shut down ourselves. Because we understand that we are not in quorum and we must shut down ourselves and stop serving. So this is what we took because we know exactly that we have three nodes. Because one node and two is just the proxies. And split-brain only, this is the only problem for a single-ton part. No, no, no. It's a standard split-brain. For read-nodes, for nodes that are serving, get read-nodes, you don't care. I will explain it later. So you do care for split-brain or read-nodes? Yeah. For read-nodes, a split-brain resolver works. So why do you need resolver in the first place for read-nodes? Read-nodes? Yeah. I think they are not counted. No, read-nodes are also implemented that. But they're based on the role. It's role-based. But why it's implemented for read-nodes, like at all? Because they must shut down themselves as soon as they are not in group with those nodes. Why does it matter for read-nodes? It still reads. But it's not about split-brain. It's about if event stream stops. But it's different problem. So split-brain is not relevant for split-brain nodes. Yeah, it is relevant. We have that implemented on our read-nodes as well. Because our projections might be side-effective. So say that you want to have a sequence as well on your streets. So that say that you write to the database and you read these events. That's only one node that only does the same task. So you're not writing two times the same thing to the database and overwriting stuff, et cetera, et cetera. So you want only one to be present in your whole cluster that doesn't want a specific task. Does that do writing? Yeah, it makes sense. Yeah, but do get nodes. They don't do any writing. But the read side here is Akastreams, right? So what that read side does is it actually tells your Cassandra database or whatever your event log and then writes it to another database or does something else. No, no, it doesn't write anything. So can you go back? Yeah, I got you. I'm just thinking. Yeah, it looks like it doesn't need it. No writings, no state changes. Yeah, that's right. Did you see Akastreams done? No, it's provided. Akastreams reads from cluster and applies the events to this one to recreate the state to serve the reads. So actually, what Grigory says is why should we shut down these queries as soon as it reads from the valid Cassandra? So whenever somebody writes there, yeah, it's actually, maybe it's not needed. I don't know. I cannot imagine anything right now why. But anyway, it's implemented. It is shut down. Because it was easier to implement rather than not implement. Yeah, I think not easy. It's safer, I would say. Because it's safer to shut down it. It's safer. If I don't see. Common sense doesn't matter for read notes. And I want to validate that my common sense is a still sense. I would prefer to keep the cluster concise anyway, even though if this part of cluster lost the main guys, I would prefer to be shut down. Yeah, but if you're not sure. You can't read it, you can't read it. The thing is it's not. Yeah, I agree with you. If you can't write it, you still can't read it. So if the rate split you cannot write, but you can still serve current states. So basically, at least part of your solution will be working. You can show that current state. Here it's not shown. But write notes can also serve reads if I'm not mistaking. Yeah, but if you have pure read notes, you don't have to shut down them. And then you can still some capabilities of your solution. OK, technically, maybe you're right. But semantically, I would prefer to keep it safe. If the request from the customer hits the read note that has completely absolute data, I will show something totally different to what's actual state. And then it will go to command, handling, and blah, blah, blah. And it will eventually see that this state is not valid and rejected. But this is just basically putting more load on the system and having worse customer experience. So it's a great problem. OK, yeah. OK, again, to sum up, technically, maybe it's valid. But you're right. I technically. But first of all, from current implementation, it still gets shut down. Then, yeah. This is real world example. We care about customer experience. Yes, we shut down read notes that they cannot read. Yes. Nice. So actually, now that read notes, they shut down themselves based on the amount of aggregate notes. Anyway, and aggregate notes does the same. OK, secure s. So this picture is from Martin Fowler's book and Martin Fowler's site. Martin Fowler works for ThoughtWorks. Thanks, him. So yeah, for those who don't remember exactly or something like that, yeah, this is as clear as that. So typically, we have model, not typically, but often we have model which serves both reading rights. So we have same model. We apply changes to it. We read from it, so on and so forth. It's no secure s. When we want to consider secure s, we split the models. And we have common model, which serves all the changes. And then we have query model, which serves all the reads. Theoretically, query model and common model can be different. Moreover, common model can be one. We can have one common model and lots of different query models, which serves different purposes, which is reflected in this site, tightened to the event sourcing. So this is like typically in many applications happens like that. So post, delete, all the modifications goes to the common site. Then it stores, gets stored to the event store. Then it's replayed, theoretically or practically in many applications, to the different queries and then got queried from other services. So again, secure s is not a pattern. It's an approach just to distinguish the commands and query. So we already touched it a bit in our implementation. So this is actually the code. It's a lot of code, but how actors actually implemented. Not the simplistic one from the beginning to get you into the context. But this is much closer to the real thing. So first of all, it's heavily used traits. And let's say we have this scheduled recoverable actor trait, which extends only persistent actor. Then we have write only actor, which and read only actor. Both of them extends the same recoverable actor, meaning that both parts, both are recoverable. And underneath, they use the same schedule business logic class. So theory says that you're supposed to have or you can have different models serving commands and queries. In this case, again, this is a practical example. And frankly saying, I didn't have much time to make this distinguishments. So I picked the same model. And I use the same for both types. This piece of code is actually from Akka Streams. So this piece of code for read only actor. Read only actor very small. Everything you can see is here. So when recovery completed from underneath recoverable actor, it actually constructs with its Akka Streams code. It constructs this stream from our Cassandra read journal, which replace these messages from Cassandra to our schedule, to our instance. That's it. So after these codes executed, and that's executed after actor-read actor recovery completed. So we have this model, which is recovered. Then we bind this to Cassandra. And all the events, which comes from command node and stores to Cassandra, it immediately replays to the read nodes. Just replay. That's it. As simple as that. One way. And this is like classes for command and query actor. They actually extends write only and read only actor and have only a few boilerplate things to start it up. And yeah, that's actually how it's actually implemented. This is what I'm talking about. Schedule. So command actor and query actor. As simple as that. Another thing for CQRS is routing on load balancer. So it's routes automatically posting the GET requests to appropriate nodes. It knows it advance, which one serves what. Next thing, yeah, this is what I already said. So schedule class is used inside both actors so in different manner. And the key thing here, the main purpose of that, as I stated in the very, very beginning, that the query request, which is capacity availability and for management UI, but mostly for capacity availability for end customer, they are pretty heavy. They query all of this big structure and form this big JSON and send it back. So we need to scale out actually the query part. This is the key thing for this particular task. So that's why this separation happened. And we can, actually there are a few nodes only, but it can be infinitely scale out and serve these heavy requests. Yeah, that's almost it for CQRS. I wanted to point one thing. Yeah, there is one thing. I don't know why I supposed to have a slide for that. It's interesting thing, but somewhere it's lost. Okay, sorry. Yeah, there is one thing in actors and that was the Excel. So whenever we want to mix in different traits, which are actors actually, this receive command thing, it can be mixed together. So when you code that and you have trade with one actor, you have another trade, which is another actor, extends actor, both of them can receive certain commands or certain messages. When you mix them together into concrete class, like here. So you can mix these receives together with one line of code. And this result mixing, the result function will accept both messages from both actors. So this is one trick which is applied here. But unfortunately, I haven't included the code here. Very simple, but it's possible just to share that. Last thing, almost last thing is again from real world, how to update that. So there are few strategies for when you want to update all of those. We picked so-called rolling update. The difference from normal rolling update means that you update one by one. As soon as new one gets up, one node gets up. So I stated in the picture. So let's have you have three nodes in this example. You update one. As we stated, this node backward compatible with the previous one, so you can serve. You see that OK, it works. Then you update the next one. OK, it works. Then you update the last one. So the only modification here is that we update query node first, then we update singleton proxies. And only at the very last, as a very last step, we update singleton instance. This minimize. So we update green, then we update proxy, and then we update this one. Why? Because when we shut down this one in normal manner, it sends the message, OK, I'm shutting down, please take over. So this guy takes over its responsibility. And it's already version 2. So we minimize the downtime of that. Yeah, so this is like about how to update that. Another slide I accidentally deleted right before is the slide about testing that. So during each time we build this, we use Travis, so it's built on these virtual machines, to test that and test, in particular, let's say, this split brain resolver for both cases, let's say. Not the crash and normal shutdown. So I implemented this as a, and within same virtual machine, I implemented this as a three virtual, three GVMs or five if GVMs starts up. Each of them is actually an instance, but within one machine. So it's different GVMs on one machine. And then I try to crash one, see what happens and check how it works. And then on normal shutdown, in code, it looks like keel or keel minus nine. So test everything how it's supposed to be on the real environment. Of course, it's not the real one. Docker maybe could be better, but we don't use Docker at the moment, so GVMs is as close as possible. It's about integration testing and testing of the split brain and some other cases which are close to real world. Yeah. So final look, as I promised, we covered each and every point here and just a final look at it. So down below is list of technologies which we use here and the key outcome from that is okay, it works. We tested it on a higher load when we have much higher and it survives, so you can relatively safely use it. And yeah. So that's it. Questions? The split brain resolver, are you using the light bands from the actual resolver or something else? No, I implemented it myself. I took it from somewhere from the GitHub, GitHub who implemented somebody else. I actually tried to ping light band, but they didn't reply and I had no time. So I took it from the internet and there was an error bug there, so I changed it and made it work, put it to the tests and it works. So yeah, they have paid version, you're right. And you can configure, you can just put it to the configuration and put it to SVT and configuration, that's it, it will work. Yeah, I know about that. I read, I could do that, but I couldn't reach them. And the implementation is simple, actually. Another question is the accostreams part that publishes advance to the read instance, right? No, just the diagram, in the diagram, do you have a multiple instance for the publishing part because it's just a line, the accostream is supposed to be something running in some nodes? It's just bind to Accoclust, to Cassandra cluster and pull in the events, periodically calling the Cassandra, whether they have changes, that's it. So it's periodic pull? Yeah, it's periodic pull. And the stream runs on the Cassandra nodes? Our stream right runs on these nodes. Oh, runs on these nodes? Yes, yeah. And this thing provided out of the box, we didn't write it. This thing, we didn't write it, so... I don't know who provided it. Yeah, this one. Yeah, it's Accapacitance part, correct. Accapacitance provides a few things out of the box to make that. If I'm not wrong, it's provided from Mongo for Cassandra and for something else. Postgres, right, thank you. Yeah, so I just pick it from there and I wrote this piece of code, it's on the node. And it starts up the stream and whenever it does pull in, I read it deliberately before the presentation to ask the question, how does it fetch it from the Cassandra? Yes? How was your experience while working for this project? Outside ARCA means for other Scala libraries, like HTTP encoder decoder, calling Cassandra dv queries for those libraries, how was your experience? Documentation is missing. Documentation is amazing. For ARCA staff, all of the ARCA staff, ARCA HTTP, ARCA actors, single tones is a bit tricky, but everything else inside ARCA is fine. And outside ARCA for other Scala ecosystem? Scala... HTTP JSON support is spray, which is basically former ARCA HTTP, whatever, JSON. Cassandra, it's a plugin by ARCA persistence, so there are no direct queries to Cassandra. And that provides your function? There are no database communication in this service, like, at all. Everything? Nothing. Yeah, yeah. The only thing we... The only thing was implemented is the serializers. So serializers is Creo, but I need... It's a pain. So I needed to tweak it a bit. Because sequences... The root cause of that is type erasure, which is the pain point of the whole Java world. So I came from .NET world. So .NET is here, GVM in this case is here, so I face it all the time. So I needed to tweak it because it erasures the type. So this was the tricky part of this. Initially, another problem was handed over during the normal shutdown. The handing over from here to here, this also was a problem when I started to... But recently was fixed. By the way, the same serializer is used to pass the messages from this proxy via this arrow to here. It's the same serializer which stores messages to Cassandra. So inter cluster communications also use the same serializer. So this is a tricky thing, then you have to be careful. And pick the serializer carefully and correctly to be as fast as possible if you have heavy loaded system. So I saw your version in the event serializers. Do you version your commands as well? Yes. Commands. I don't need to version commands. You serialize them, right? Commands, no. I serialize events. Come and come, right? I translate it to event and I store event. So I serialize only events. If you want to update, you also need to serialize your commands. The command serializes events and you're playing events and therefore events need to be versioned. But commands is like intermediary. It's not like... I got your question. The communication of commands is also serialized. Yes, you're right. You're right. It means that I have versions for commands as well. You're right. Because otherwise it will fail. It seems cluster is here just for fun. Seriously. You can implement all this. You can remove our cluster and it will work. How? The only thing that... You cannot build singleton on top of no cluster. I can. Singleton works on cluster, if I'm not wrong. But you get the... We want the cluster. The cluster, the fact that you can easily pop back you have to implement it yourself which could be... What? What? Here the fact that... I feel like because you use CAPCAP you can then this proportion down event and recover properly. You can implement it yourself. People have... I feel that's more... Half a year. Basically you mean like to maintain amount of nodes served? So just like... Just the fact that one takes over. Is that building? I think way lower... But basically you maintain number of nodes. So the only reason to have cluster here is you want to maintain certain number of nodes. Yes. In this case. One serves it right? Yeah. This amount, so when it's clustered these nodes know about each other and then it knows that I'm a proxy in this cluster. You already have a standard cluster with event stream so to implement single tone you don't need cluster. It's redundant. It's enough to have event stream to implement single tone in your case. There's application states which is that schedule. How will you... If it's present on two different nodes if they're not clustered... If it's present on two different nodes you have a chance of freeze conditions between the two nodes. No, no, you have event stream. You cannot... If it's one event stream you cannot have... So the business concern was that we shouldn't... If we just have one unit of capacity left and we have two concurrent requests that try to book that only one must win. If we were to... But you have an event stream which will align them like one by one. No. Because if you have two actors with two different states they can persist the same event at the same time. This can be easily overcome if you identify each right event with nodes like when it starts. Generate the week. That's a optimistic concurrency, right? And then you're basically manually running up your own implementation of single writers. So basically if you wouldn't last node to write then you're not allowed to write. So for example you cannot write a fake elect event with a node ID and therefore actual writes only allowed for elected nodes. So if last event you received it doesn't have your mark then you're not allowed to write. And this is basically your own implementation of cluster security. But in this case adding to the event just node ID replace like entire functionality of cluster here. How does it help preventing access to single... Okay guys. You can talk about it. We can find lots of implementation for this task up to the business task. So we can... The most important thing in this... Okay. Then let's draw. Then let's come and draw all of them. Yeah. So any other questions? Yeah. Can you move over from the dot network into the Java world? Is there any reason why? For fun. I moved from Ukraine. Which is roughly 8,726 kilometers. Roughly. To here. It's also for fun. I guess that you could have stayed in dotnet and came to that shop. I'm considering to moving back to dotnet. Yeah. I'm just... I have 12 years experience in dotnet and zero experience in Java. So I just wanted to expand my knowledge. So fun means expanding my knowledge. So I downgraded a bit. I used to be project lead and even director of engineering or something like that. Then I downgraded to the senior software engineer just to investigate how this GVM works. Especially, of course, Scala. So I don't know pretty much Java. I don't like it. So this functionally... Right before that, I had the chance to implement a small project on F-Sharp. Yeah, it's better. And dotnet organized better. So I would put like these languages, right? So Java is the lowest part. So let's split it in the middle. GVM is right. Dotnet is on the left. So Java is on bottom. Then C-Sharp, right? Then Scala, then F-Sharp. And here's Haskell. So I would put it like that. Yeah, but the response to your question, I just want to explore this word. Without practical hands-on, it's impossible. I can read a lot, but it's impossible. Now I know. And I don't like Scala at all. It's the best choice we have now. In GVM. On GVM, yeah. But I don't like it. Those guys, from Bangalore, created something like Ita or I don't remember what was it. It's basically Haskell port on GVM. Oh, no, I haven't read it. It's not ready yet. No, not at all. Sort of a new something thing. It's a husband and wife team. I haven't heard about that. Let's wait. I'm waiting actually for Scala 3 or whatever they call it. Doty, yeah. This is like something I expect and let's see. So I give it a hope. Other questions? Hopefully it's not like Python 3. Yeah, hope so. Yeah, could you go back to the slide where you introduced actors? You said the actor's lightweight and at the same time have a thread. Is it like one? This one? Yeah. This is not exactly correct. It's the best picture I've found. It's actually not. Yeah, I got your point. This one? No, no, no, no, no. This is actually a bit ugly. So I didn't stop this slide for too long to not give you a chance to read it carefully, but you unfortunately read it. Yeah, this doesn't mean that it's physically a separated thread there. It's an actor's system. It has its schedule inside. And it's assigned the threads within this algorithm to process these messages and to assign threads to one actor or another. So it doesn't mean that it always keeps a thread on it. No, no way. So I think your statement here is like actor's system use threads to be non-blocking. Yeah, non-blocking. Yeah, yeah. But they're more than one thread. More precise, that's correct. More precise is it's non-blocking. It's not the thread. It's non-blocking. Okay, and you mentioned there are two ways of communicating with actors. One is like firing for gas, right? Yes. And another one is... What is the synchronous operation of async? This one, the last one is asynchronous because the return in thing is future. So it will become synchronous as soon as you put something like await.result and put this future in there. Then it will block. So it will use the thread, right? Yes, anyway. It will complete the whole... Yeah. Exactly. You can write as a next step. You can write response future, complete. Then, let's say, I don't know, success or whatever. And then log, I complete it. It will be executing on asynchronous manner whenever future completed, but it's not blocking. No. This one is not stopped here. I got your question. This guy doesn't stop here. It does not block. No, no, no. It does not block. Both of them are not locking. Both of them. And what about the receiver creation? If I need to make a call to another actor, is it possible? Yes. You can... Here is this sender. It's a special thing. So I send this back to the sender. But actually, inside the actor, inside the actor, you have so-called context, and you can write something like context, get actor by name, and get the actor F for the other actor, somewhere you don't know where is that. And then you send in the same manner, you send message to this actor. It's also not blocking. And even if you call like that, it will return your future, but it's not blocking anyway. And therefore, you can send a message there. But you must be careful here. There are some patterns. There are some approaches to not make it like spaghetti calls everywhere. Actors actually also must be used very carefully. Sometimes some projects tend to use actor too heavily. Actor is a simple thing. The best case of usage is having a state per actor. That's it. To wrap one state. Many projects fail to use actor as a function. Actor is not a function. It's something which covers some state. Yes. You can send anything. Unfortunately, you don't have control over it. There's sort of experimental extension to actors, that types actors, but it's so experimental that it doesn't work. Practically. In a file actors call Mailbox and they have that. Yeah, I think so. Yeah, this is the pain point. So I can send to my actor, which receives only reserve for something. I can send this sheet as well. So typically the end of the receive is like that. K is on the score. Log in for receive that non-message. Or log error or something like that. No, you don't have to. If it's a tell, you don't have to. Don't have. No, if you want high-paying ideas, you can have other choices like future or money. Actually, what will happen if you do my actor ask and send a message that can be handled by actor? Then your wildcard case... This one? Yeah. But it will never reply to sender. So this future will never... Complete. Yeah, it will never complete. Most probably yes. Yes. Yeah, so how it works actually. Split rate resolver works like that. There are three of us. One, two, three. So we see together. We see each other. Then something happens. Oops, I don't see anybody. What should I do? How many of us? I'm only one. Quorum is two. Okay. That's it. But if they are also split... If they are also split, they both kill themselves. All the cluster dies. If there is no network... Split brain is a network connectivity issue. If there is no network connectivity, it only protects from partitioning into two parts. No, it only protects. If a part of your cluster is out of network, it's a small part. If you have a huge cluster like 90 nodes, and then you have split them into three, then it's... Yeah, I mean in this example... Yeah, in this particular case, there are three aggregate nodes. So as soon as some part is including query nodes, as soon as it sees that aggregate nodes is only one, so let's say we have one aggregate and two query. They will shut down themselves. All three. That's it. That's as simple as that. But it's this particular task. There are other strategies. How can we implement that? But we must decide and think carefully about that. Because split brain problem is one of the pain points. It's any cluster. Because if they start to serve events, both of them, and start to write down to event source from both parts, they mix together, and you will see, you will spend the whole night, next night of your life, trying to figure it out how to fix that. So, yeah. Yeah. Okay. So, last announcement I must do. Red Mud is hiring. Please, if somebody interested, please follow the link. Also, we have these pens and all of this stuff from Red Mud. You can take it for free. It's a present. That's it. Thank you. Thank you.