 So that's roughly the amount of material I have. So I decided to do it in a different way, and basically pendulum swing back, and now I have a 30 second version of presentation. It goes like this. So if you need a very high-performance, high throughput, low latency, very available service, you can use AKA sharding to ensure that your right side is very consistent. Then you can use AKA distributed data to provide sort of a secure as implementation for each side that is very available, keeps all the data in memory, and responds within milliseconds. That's it. So now we're gonna dig deeper. So first we need to set up the backstage. So basically, we cannot talk about systems without touching on the concern of what they're doing, and especially how consistent they are. So in modern software engineering, there is a concept of eventual consistency, which is the go-to solution for low latency, high throughput systems. Unfortunately, in our case, we couldn't afford that for one particular reason. So basically all the systems that I'm gonna talk today are behind this screen. This is our Red Martz screen that shows you available slots and allow you to reserve one. So the problem here is that we really want to keep our promises to the customer. So there are no way that we can show a slot as available, try to reserve it. And then we say it's reserved, but then it turns out we're over capacity. You know, our service degrades and we ask the customer to, sorry, we couldn't accommodate your reservation. Which means that our downstream systems need to provide very strong consistency guarantee. Not like in strong consistency. We'll briefly talk about different consistency models, but pretty high one. So I wanted to give you some leave low test demo. So I have two services primed for demonstration. So this one is like the service that is on top of everything. It orchestrates all the downstream communication between multiple services, which we'll talk later. So hopefully nothing will break, okay? Okay, so something broke. Nothing happens, fine, I've prepared. So these are low tests, which cannot be loaded. Okay, fine, I really prepared these ones. Seriously, so this shows the low test for this service that communicates to like six or seven other services via HTTP endpoints. So as you can imagine, I think this one was under, yeah, something like 30 concurrent threads per second. Generating load. So let's try another one. This one worked, okay, fine. Well, okay, so let's consider the presentation effect. So these are our contents. So basically what I've- You show us some numbers. Yeah, some numbers. Maybe you care to explain why you did this? Just to get your interest. You could have shown the seven. Okay, fine, so let me explain what's here, right? So the first one that I tried to show first, and I showed you on the spreadsheet, is capacity service, which is basically an entry point into our capacity services family. It communicates to FC capacity, which Alexander talked a month ago. It talks to transport capacity service, which provides a different view into our fulfillment model. And a couple of other services that are not shown, like address restrictions, location intelligence, resolving post codes to addresses, resolving addresses to zones, and so on. So practically there are like four services on the synchronous path, on the sequential path for every operation that I wanted to show. The second one, which didn't work at all, was transport capacity service, which is basically our downstream. So what I tried to show is that, even though we have very strong consistency guarantee, which basically means that we cannot afford booking more capacity than we actually have at all costs. We have like 30 requests per second at capacity service, and this is just one T2 micro instance. And something like 250 here, where we have a cluster of three T2 micro instances. Sorry, was there some remark? No, okay, thanks. And basically this is a technology map of what we use where, how it contributes, and so on. Yeah, so ACA capacity service built with, well, basically all the services built with ACA HTTP used as a HTTP interface. Capacity service uses ACA streams to, well, sort of ACA streams is implementation of reactive streams pattern, or approach, or technique, or whatever you call it. And so all the orchestration happens within the ACA streams graph. Then FC capacity, Oleg talked about that last presentation, but basically we're using ACA cluster there, and cluster singleton to manage like one huge, well, basically think of it as a spreadsheet that controls how much peaking capacity we have in the warehouse, basically. And we use in transport capacity, we have ACA streams, we have cluster sharding, we have persistence, all that stuff that I'm gonna talk about today. So in transport capacity, it's more, like the domain model is more granular so that we can afford using cluster sharding. And I think we'll start with ACA streams first, yes? Can you move to previous slide? I'll try to, no, not like this. Previous slide. Yeah, sure. Back, back, back. This one. So lock free means there are no locks in our particular code. There are some, I'm not so sure about actual ACA implementation. There might be some locks around actor mailbox, right? But well, I'm not considering them here. Serialized means there is a consistency model called serializable, right? So if I'm not, well, I'm not a big expert in this consistency model. To me, something that consistent is consistent, right? But consistency models, serialized means that you can establish a certain sequence of all the operations coming from all the nodes in your distributed system. In this particular case, it means that all the access, including reads and writes to one particular state that's encapsulated in the singleton is 100% serialized. So you cannot get stale reads. You cannot get dirty reads. You cannot lose rights unless something catastrophic happens and so on. So what does that actually transport about what transport is? So this is fulfilled like logistics. We have a fleet of vans to perform deliveries and transport capacity controls how much delivers we can make every day in every geographical zone. Capacity services, how much you can store? Sorry, what? The capacity services about how much you can store or? So capacity service is just an orchestrator. So it just basically reads, sends read requests to transport NFC and few other services then reconciles the data in end form. So we need to have capacity in NFC. We need to have capacity in transport. We'd need to not have this address blocked for this certain time period that you are selecting and so on. Can you say again how many services on those are there in the NFC capacity and how many are in the transport? Sure. So we have two environments. One is staging environment, one is production, right? So what I've shown the results of load test is our alpha environment. There we have, well, alpha, sorry, staging. There we have, I think, three T2 micro instances for FC and same three T2 micro instances for transport. And for capacity service, we just have one T2 micro. In production, it's bigger. I think we have two T2 small for capacity service and five T2 small for FC and three T2 small for transport. Okay. The fact that you're running a single term means that you are just making use of it. Well, we'll touch on this, yes, of course. So with ACA, you can just declare what's zero, right? Is that how you do it? So, you know, just to recap what actors in ACA do, like very basic ACA functionality. An actor is an entity that receives and processes messages. And ACA's guarantee is that at any times, it processes one message without any concurrency. So, serialized messages. Serialized messages, yes. Like all the messages that you send to one particular actor, somehow serialized by the actor mailbox, which is handled by ACA. And then inside the actor, they run in sort of a single threaded compartment. Okay, does it answer your question? To some extent, okay. So, let's move on. Yeah, so, returning back to my question in the beginning. So, if I just talk through all of it, it will take probably two hours. So, I'll just, you know, go through, you know, overview of the technologies, what the features are, how they contribute to performance, availability, latency, and so on. And then I have a few slides with code examples. I'll just bypass them, and if there are any questions that could be illustrated with those code, I'll just return to them, okay? So, ACA Streams is basically an implementation of reactive streams approach technique initiative, right? So, they run the top of actors, work well with all kinds of actors as well as futures. Model sort of a single way computation. There are extensions like bidirectional flows, but they're slightly harder than the core model. Used as I've said in Capacity service, and also we use streams to set sort of stream data into Ratshift for analytical processing from transport capacity. Yes? Are ACA Streams still kind of only run on one or music kind of? It cannot, so I think Lightband, Lightband, right? They are working to enable multi-GVM, well, multi-GVM, multi-node streams. Before 2.5, they didn't have that capability, or maybe 2.5.7 or something. Recently, they added some sort of remote source or remote sync that allows you to sort of throw the payloads over the network. We haven't used that, so I cannot provide any more details so far. But yeah, in general, stream runs in one GVM basically. So, advantages. This is basically a concurrent execution with the same concurrency guarantees that actors provide, so no locking, no thread management, no that sort of thing. One particular very useful feature of reactive streams in general and ACA Streams by implementation is that back pressure prevents overloading slow components. So, your failures, if you're just really overloading, if your rate of request is really higher than what your slowest component can process, sort of failures start to happen earlier before you actually hit that component. So, they happen in less busy components and because of that, it recovers very much faster than other ways. And my other partner in our team has a hardware engineering background and he really likes that. It looks like a senior link or MATLAB. So, you're just basically connecting your components, you know, describing inputs and outputs and then it sort of runs. So, yeah, this advantage is logging is harder. You just need to throw, you know, specific logging stages all over the place. Complex graphs, like graphs really goes complex very quickly. I think I'll just need to illustrate here. So, there are two APIs. One is sort of functional combinators, map, map, map, flat map, filter, map async, that sort of thing. So, these ones are sort of, they can only build you a sort of linear stream. For more complex streams where you need branches, where you need, I don't know, loops, flow control, you have to use graphs. And this is just basically a very simple graph that takes an input, sends it to two services, merges the outputs, this size. So, the quantity of, you know, sort of boilerplate code grows very rapidly. Now, imagine if you have like four services that you need to communicate and then you need to merge them in different ways. So, this really gets complex very quickly. As an anecdote, I've did the refactoring. So, in our initial implementation used flows everywhere. Later, I've just refactored so that they're reserved to top level components. So, DPR was like plus thousand lines, minus two thousand. So, need to be careful with that. What is to using graph like description of the? Yes, but in top level components. So, previously we had graph levels, you know, graph flows in a down, like in a downstream services. I just replaced it with future-based computation, put it to, you know, reserved, everything flow related to top level. So, you can use functional, you can use graph. So, whenever you use streams, you actually use graph API, not the functional combinators. If we have linear stream, we use this. So, graph APIs was only for, you know, where required. How do they say that? We prefer functional API over graph API, but. You get the branch in functional API as well. Yes. So, what was the decision like to go for graph API over functional combinators? It wasn't my decision. So, I cannot provide, you know, deep inside, but, you know, in some cases, some stages are not available. So, for example, merges, I don't think you can do. You can do everything. Is there any, you can give those as like a symbol from graph API, and if you have a corresponding message that you can do the same thing. Okay. Well, maybe I've lost something. Okay. So, a few caveats. Because it's a reactive streams, it basically works right to left. So, data transfer happens left to right, but data is only propagated if the downstream component pulls. And pulls are propagated right to left. So, it's very easy to achieve, you know, to end up with a stream that doesn't run at all. Like, here's an example, this one. Looks really, you know, innocent. Queries merges at the end. Broadcasts. So, without this buffer, it doesn't even run because, or else pulls on, you know, primary input first. So, the or else, what it does is that, if you have two inputs, it first tries to obtain a result from the primary. If primary input doesn't produce any value, it tries to pull from secondary. So, on the other side, we have a broadcast. Broadcast emits elements only when it's pulled on the both inputs. So, in this case, without the buffer, what happens that or else block pulls on the top stream and waits. And waits, and waits, and waits, and waits. So, in order to make it run, you need to put buffer here. So, buffer pulls on the broadcast, broadcast, you know, emits elements, everything's moving. But, you know, this is a sort of a caveat. It's very easy to end up with the stream that doesn't run at all. Yeah, majorization is sort of a coastal operation, even though there are sort of a two modes how you can utilize streams. One is create a stream, like a runnable stream for every, I don't know, HTTP request you have. The other one is to have a long-leaving graph that, you know, a long-leaving stream that runs all the time. So, in AK HTTP, the approach is sort of slightly favoring the first one, even though the majorization is said to be coastline. And last one is, you know, that's sort of a one-way road. If you do, if you start doing flows or streams, you cannot go back. Well, you can, but it's expensive. Okay, next one, AK HTTP. So, this is a very simple concept. Play, Scalatra, if someone's still using it, Finnegal, Lagom, all are frameworks. They have certain, they impose certain way of creating your application. So you need to have controllers, controllers need to have services and so on. AK HTTP is just sort of the opposite. It's a lightweight framework that plugs into your application and allows you to, and allows it to accept and respond to HTTP requests. Nothing more, nothing less. So it works well with, it supports WebSock and HTTPS, even though HTTPS support was added pretty recently, so we're not using them. I'm not sure how well it works. Works great with streams and actors, actually implemented by using streams at the background. Definitions are composable, so functional mongers will like it. Maybe not as HTTP for us, but comparable. Opting caching, easily extendable, low overhead. By low overhead, I mean that we haven't observed, we were doing low test and performance investigations. We haven't observed that AK HTTP added more than a millisecond or something compared to what business logic took. Business logic and maybe serialization. So this is where we use them, basically everywhere, except for fulfillment pipeline service, which is a very special beast, but we'll not talk about that later today. So advantages, disadvantages, I believe we can read, I can talk. So adding cross-cutting concerns is really easy. I think, yeah, not this one, this one. So all the cross-cutting concerns here are added by this directive, which is just a line of code. Behind the scenes, it's very complex, but the implementation of these log request result and request response metrics are somewhat sophisticated, but basically at the route level, it's very easy to add them. So this is again, runs on top of AkaStreams, and because of that, it has back pressure, avoids overloading downstream systems, and runs happy. Can be plugged directly into AkaStream, few caveats there, for example, if AkaStream ends with a failure, you don't have, you receive a rejection, which is just a 404 with a text response, not even application JSON. Could do some handling around that, but this is not that intuitive. Caveats, we observed one thing when we updated Aka HTTP to a new minor version, and then didn't update AkaStreams, and because of some implementation details, we had performance degradation, like three or four times. When we upgraded, everything was back to normal, so minor versions, not 100% compatible. And final, it's like very loudly communicated in the documentation itself. It says client must consume entire entity body, because, response body, because if it's not doing that, producing system, things that is being back pressured, and doesn't continue, all the connection gets stuck. Yes? Which one? This one. The third point of advantage is, can we plug directly into AkaStreams? Can I treat a long-lived HTTP connection as a stream? Is that what it means? Can it act like an HTTP client as well? I think so. So basically, in Aka HTTP, there is a directive that produces you a AkaStream source as from the request. So if you have a long-lived request, if you're sort of a DTO, well, I don't know how do you name these things. So if your DTO has ability to support that streaming, this is basically what automatically happens, right? I think so. I haven't tried. This is a very interesting question. So let me know if you try it. I'll let you know if I try it. Okay. So next step is Aka persistence. So this one is basically a persistence plugin for Aka. It has an opinion. An opinion is that you need to have event sourcing. So, that's why. Okay, so there are multiple persistence plugins. I think only two are officially supported. Not sure what does it mean, but it is available for Cassandra and for LevelDB, but LevelDB is only for local testing, that sort of thing. So main feature is that it's event sourcing with all the disadvantages and advantages that it produces. State remains always in memory so that you don't need to go to database to fetch state, update it, put it back, and then we'll then respond. There are multiple storage plugins and there are a few sort of plugins that allow you to change semantics of message passing between actors, which is at least one's delivery. So basically it's used both in transport and C capacity to persist doctor state. We use Cassandra. We're currently building custom CQRS site, read site, just purely using Cassandra directly. We'll see how it goes. So. So to what extent are you using event sourcing? So as I've said, ACA persistence have an opinion that you need to model your application state as event sourcing system. So essentially, since transport and FCE capacity only have state in persistent actors, it's like, I would say it's 100% event sourcing. How are you from the transition? We didn't have anything to transition from, so we'll just. You're just conversing on the D2. No. Yeah, so we can talk about that later. I actually have sort of a D2 slide into caveats and probably one of the major problems when it comes to persistence, but that'll probably take a long time. So let's just skip it and reserve it for question session or at the end of the presentation. So because all the state is always in memory, it's very high throughput and low latency. Easy to implement. Well, since it's eventual event sourcing, easy to implement security and corresponding technologies. No state records in the DB, so you cannot just go there and look what's your application state right now. So we need to, especially if you're using binary civilization, you cannot even tell without support of special tools, what are the events there? So big sterilization plugin carefully, it's very hard to change because the idea is that all your events are available from the beginning of time. I think we did, you know, not a very good job in here because we used cryo and it's basically a binary civilization that's very specific to Java, so it's not even Pratabov. I think Akka suggests Pratabov as a first civilization choice. We choose cryo. We suffer a little bit. So before version 085 of Cassandra persistence storage, it was not eagerly initializing all the connections and because of that, we actually had very sophisticated problems, so basically when something, when your actor is being restarted, it needs to read all the history of messages, right? In some cases, restarts happen. Well, I think we'll talk about that in charting, but basically, there is a problem that initializing connection takes four to five seconds and because of that, we're losing customer requests to otherwise pretty healthy system. And not so sure about this, but we just recently observed that if persistence fails, there are no usual supervision kicked in, right? So, cluster and charting. This is the slide about the consistency model. So what we need to look here is like there's a link to this entire thing with clickable map with all the deep explanations about the consistency model. What we need to look here is that sort of right branch talks about single object consistency. Left branch is more of a transactional one. So in here, like color coding didn't fit into this slide, so red means that this system is very consistent, but in case of partition, you basically lose all of it or part of it. Orange means that if you have sort of a sticky reads that always go to the same node, in case of partition, most of your clients stay up. Only those that connect to the field mode, to the field node fail. Blue ones mean that there is total availability at the cost of lowest consistency, which means that even under network partition, node failure, all of the state is available. All the clients can proceed. Okay, we'll skip this one. So singleton, this is basically a concept that allows you, a concept to plug in the technology that allows you to run one single instance of something, of an actor basically, in your entire setup. Yeah, basically the idea is that you might want to control access to some external resource like, I don't know, rate-limited third-party API, database connection, something else. In that case, it might make sense to model it as a single entity in the entire cluster. This has an additional benefit is that cluster singleton is automatically restarted on other nodes if your node fail. So you can use it to model sort of a very, sorry, very robust, persistent entity. Right, so, yes? Sir, can you go back to the beginning? Sure. So I understand CAP, you don't have time, can you just say what E and C? Yes, so this is explained here. So there is an extension to CAP theorem, which also takes into account what happens if there are no partitioning. And basically it says, in case of network partition and blah-blah-blah, you have availability or consistency, else you can choose between latency or consistency. All right, so basically it means that in presence of partition, you can have either availability or consistency. Without it, you can favor consistency or latency. Like elastic sense. To some extent, yes. So in this case, consistency model is linearizable. So all the requests are linear because of the guarantees that ACCA actor in box provides. In terms of PACELC, favors consistency always. Used NFC capacity to model actual domain objects and in transport capacity sort of indirectly because ACCA cluster sharding uses singleton as sort of a clustering sharding coordinator. Advantages provides very strong consistency model. Basically as close as you can get to strong consistency. Might be some edge effects in presence of node crashes or persistence failures, but general if your database, if Cassandra is healthy is connection to Cassandra healthy. This is basically a serialized or what linearized consistency model. This advantage is obviously performance bottleneck, single point of failure, can't scale out, basically utilize a single CPU over your entire cluster. Single CPU? Yes. You mean the virtual CPU? Virtual CPU. Cannot run on two calls? If your like internals of the actor can be parallelized than could be. But basically the idea is that it always processes one message at a time. There are no concurrent access to internals. Why would you want to use a singleton? Can you give a use case? So in this particular case, sorry? Consistency. Consistency, yes. So in this particular case, like our business requirements to have very strong consistency guarantee. Like our sort of if we model, like if you represent like under utilization of capacity and over utilization, right? So our cost function is like this. So over utilizing one unit is more expensive than under wasting like 10 or 20 units of capacity. In that case, we want to prevent over utilization happening because of the technological limits and which completely rules out eventual consistency mode. We need to be, we need to guarantee that every reservation like capacity reservation request sees most up-to-date state of whatever it is talking about, talking to. You're talking about over-poking. Yes. Over-poking the slot. Yes. So this is something that we want to prevent and because of that and because of the detail of, gory details of red mart operation in the warehouse, we basically have like one giant entity representing entire capacity in FC, in warehouse. Because of that, we chose to use singleton. Actually, it wasn't my choice. It was Oleg. So you should have asked him a month ago. So. Potential consistency you still can have like strong consistency, but you can guarantee latency for it. So basically you can regulate your latency for it with like hardware resources available. Let's talk about it offline. I sort of could imagine what you're talking about, but not so sure I understand completely. So, yeah. So overall, really stick to the recommendation given in the ACA docs. Each singleton should not be your go-to choice for design. So, this is about charting. So, the idea here is that your application state could be represented as a collection of reasonably independent entities. And you want to spread those entities across the cluster to achieve higher resource utilization, you know, be able to run more stuff like one of the use cases given in the docs is that if your application state doesn't fit into memory of single machine, you might want to utilize charting so that you chart it and utilize memory of multiple machines, right? So, with ACA persistence, it allows sort of a durable entities to be run somewhere in the cluster and automatically being recovered in case of any problem, node failure, network partition, all that stuff. Consistency model is slightly lower than in singleton case because between the sequential leaderizable, there's a concept of messages you need to be able to establish either total order of messages with the wall clock time or total order of messages with regard to each individual sender, something like this. So, in this case, since there are multiple entities, multiple senders, multiple receivers, it's much harder to establish sort of wall clock total ordering. Why would you want to use something like cluster charting instead of having a distributed system like, why not do separate processes running across machines? So, would you have copies of data on both or just push it down to database? It depends on the requirement, but... So, if you have copy, then you cannot achieve this consistency level. If you push all to the database, you're basically having performance issues because your scalability have a bottleneck of database. So, this is one way of overcoming this limitation. In this particular case, since ACA persistence, basically ever, like most of the operations happening is just writing to Cassandra, like appending only writes. It doesn't take, you know, it has much better performance characteristics. So, it actually ever reads from Cassandra when it needs to recover either singleton or sharded entity, sharded actor. And again, it reads sequentially most of the time. Use and transport capacity service as I've said, so we basically represent each, like we have shifts of vans performing deliveries. Every shift is represented as a single actor, single sharded entity. That's because all the capacity is reserved for, like all the capacity available for a shift is reserved for that shift. We have 10 vans in this shift, we can deliver like 100 orders, right? As opposed to what we have in capacity where we have people and they can pick for different things and all that stuff is shared. So, one drive operation to the ACAP assistance is being successful only when the data is returned to Cassandra? I think there is a concept of write concerns similar to Mongo, so you can, you have control. I'm not 100% sure, like, because we didn't need to tweak it. Like default settings were quite okay for us, so we just look at it, marked yes, default settings, okay, move along. So, I'm not so sure if you have, like, yeah, well, actually, yes, sorry. There is a concept of replication factor in Cassandra, right? So, it says that in order to, for write to be successful, it needs to be acknowledged by this number of nodes. Well, sorry, I'm not so much on that. Your side is, what if you write something to the actor state? Yes. And before it gets into Cassandra, some error would you return that to say write is okay? At least a write to Cassandra is a thing I'm saying. Okay, so I think there are two concerns, right? First, what if Cassandra fails? What if write to Cassandra fails, right? So, ACA persistence requires you to structure, well, strongly suggests you to structure your state updates in a way that they happen in a callback to persist methods call, right? So, it first persists, then it acknowledges that event was persisted in Cassandra. Then you apply your event. So, classical event sourcing approach, right? In that case, even if you fail to process this event, it's already in persistence storage. So, when your actor recovers, it still reads the event and can apply whatever it needed to. So, the second one is what if, like what happens between Cassandra write and receive, like when you initiated write to Cassandra and when you received response, right? So, during that time, all the messages arriving to the actor mailbox is, what's it called? Stashed, I think. So, it doesn't do anything, like at all. So, you'll wait for that write to finish. Yes, there is persist async call, which is said, you can use this if you really want to and you're really sure that nothing will break, will just persist and execute your event change immediately, whatever, right? So, then it's much faster, it can consume, but this is sort of risky. So, classical event sourcing is that you first persist, wait for confirmation, then apply your state changes. Okay, does it answer your question? Okay, cool, thank you. Well, yeah, advantages, still strong consistency model, scales out like this, you just add a node, plug in, make sure that it connects to cluster. Some part of the state is rebalanced, migrated to a new node. All the nodes doing something useful, no waste of resources. Rebalancing, crash, network partition, well, disadvantages, rebalancing, crash network partition causes. Part of the application state to go down temporarily. And if you remember my note about not eager initialization of persistence plugin, when recovery, when the node crashed and some of the entities were rebalanced to a newly created node, right? Starting, like, because of not eager enough initialization of persistence plugin, it was only initiated when the first message arrives. So the first actor takes sort of a latency head of four to five seconds. And because of that, because of our load balancer configuration and latency guarantees, we're losing customer requests. We're responding 503 service unavailable, sorry. Not a big deal, but we had to fix this. Okay, this one is basically like a final topic. So, yes. Do you apply snapshots or you just let it answer? Yeah, we do. Let's talk about that later. So this is distributed data. So it's sort of another plugin on top of cluster. All right, so it provides you a collection of this conflict-free replicated data types, which is another concept of having high-performance data structure in memory that can be distributed, can be accessed by multiple readers and writers, and resolve conflicts without applying locks. So features non-blocking masterless updates, so we don't have to designate particular node to perform all the writes and have master to slave replication and so on. Eventually consistent copying. Control consistency levels, it basically has the write concerns and read concerns similar to, I don't know, Mongo, Redis, sorry, not Redis. So if you write, if you write concern all and read concern all, then you're probably getting very consistent level. If you say write concern majority, read concern one, you're getting data very quickly, but risk dirty reads, not updates, and so on. So consistency model, I'm not so sure about this, I just made it up, but it looks like in our case, because we use read local and write majority, it sounds like monotonic reads and monotonic writes, which basically means that if you read something to a node, like on the node one, you write something, your next read will return you this particular value. On other nodes, it might return you some like previous value or current value, depends on time constraints. Yeah, so in terms of availability, latency, blah, blah, our current setup favors availability and latency. Using transfer capacity, basically, yes, sure. Give an example of when you would need that multi-muster replication of nodes. So we're basically utilizing it in transport capacity to have a snapshot of all the internal actor states, right, which is sort of a read-only one. So when we receive a read request, we immediately serve it from this local replica. Right? Redis first is network code. Second, Redis runs in basically single threaded mode, right? So there is just one instance. Here, we explored a few options we had. We had Redis, we had Hazelcast, we had Elasticsearch, because we already had Akka, Sharding, and so on, we decided to try and distribute the data. Spoilers, not everything is that smoothly running. We had to do a few iterations before we got it right. We had to remove a few things that didn't work quite as expected, but so far, it's working really well. But there's a possibility of re-stale reads, right? What if your node isn't going to be repeat? Yes, so in our particular case, we want to be very consistent for writes. Showing inconsistent data for reads is okay. So we use this opportunity to reduce our latency. So yes, very fast queries if you read local, non-blocking updates from anywhere, you don't have to worry about what happens if two nodes update the same state. Not really suited to high cardinality data. So in docs, they say that 100,000 keys on the top level data structure is probably too much. We actually observed problems with like 70K, but a higher rate of requests. So we actually observed like services were crashing because of out-of-memory issues because like entire memory was consumed by some mailbox in the distributed data thing because we're running like, I don't know, 20 to 25 requests a second and every request basically caused part of this distributed data thing to update. And we spent like a few weeks trying to tune like various tunable parameters for the data. Eventually, we just removed part of it and reduced the state to like something like probably 1.5K or something like this, top level keys. Well, this is a very broad topic, but in general it's sort of, well, let me illustrate it by an example. So what if you want to count something, whatever it is. There is a concept of positive and negative counter, which is one of the CRDTs. So it doesn't store you a single number, right? Because in order to ensure consistency, you need to lock on this number so that different thread doesn't update it in the same time. So what PN counter does, it stores how many times it was increased or decreased. In that case, different writers don't need to lock for a single resource, right? They just say, I'm increasing you, that's fine. Counter increases, so I'm decreasing you, counter decreases. So every sort of right just appends, I don't know, and then a number somewhere. And then the value of the PN counter is just how many times I was increased, how many times I was decreased, right? So this gives you a sort of a monotonic way of resolving conflicts between the different writers. There are more sophisticated, like this is one of the most basic ones. There are more sophisticated CRDTs like maps, lists, and so on, but they're much more complex, so I wouldn't be able to cover it here today. Okay, yeah, I think that's it. So this is just for recap, a copy of the very first slide. We have ACCA HTTP everywhere, streams for sort of integration and really streaming data. Single ton to manage one gigantic, not shardable state in FC capacity and ACCA cluster sharding to represent shardable entity state in transport. Again, so a few references, mostly ACCA docs. I'm really like consistency models. This is really great. So these guys are, I think they're doing investigation into how consistent your consistent store are. Like they have a claim that Mongo is really not consistent at all. Like you should not ever use it at all. It just loses rights, reads, and so on. Yeah, and I think that's it. Questions? It's quite a complex system. Like having all the experience now after you build it, will you build it like the same way next time? I would build capacity service as a monolith. That's it. No scaling. Well, probably I need to rephrase this. I would probably build it as a sharding. Well, I'm talking about like monolith from perspective of multiple microservices. So even though they seem to be a little bit different because of this difference in how. You mean you will have monolithic state? Probably, yeah. But that monolithic state would probably be sharded in one single system. So basically using microservice architecture just added complexity of managing all the remote calls, what happens if it fails, what happens if that fails, and so on. And microservices are interdependent? Well, there are a tree of dependencies, let's say. There are no two-way dependencies. So capacity service depends on fulfillment, FCE, transport, a few other ones. Transport depends on one other service that's not shown here. FCE doesn't depend on anything, it just reads from Google spreadsheet. Works really well, actually. Very convenient operations. What else? Yeah, so that's sort of not a graph of dependencies. It's a tree. Well, tree is a graph, but a subset of graphs. Yes, unidirectional graph. The question was mostly about technology, so you would use like a stream persistence symbol. So if you need to implement the system like in another company, but the similar request, you would suggest like same technology, so you think it's like best way to do it. I'm quite happy with what we have right now. That's probably not the best way, but I don't know. What would you say? If you're saying it's not the best way. So streams in capacity service, my opinion is that we overdone it a little bit. So I just recently reduced the amount of streams in there. So we're basically, everything was either a flow or a stream or something. We just had like a few future-based APIs. But in this case, future-based APIs turned out to be much more simpler. And because we needed to do some pretty non-standard flow control, which was really easy in future and not that easy in streams, the amount of boiler play code was quite high and complexity was quite high. So this is what I'm doing. Like for ACA HTTP, this is as good as other frameworks. The concern here is that in RedMart, most of the stuff is tailored towards play because most of the services are written in play and blah, blah, blah. So monitoring, logging, deployment is tailored to play so we need to tweak and adjust so that our services can utilize all the infrastructure help. Everything else, persistence, there are a few quirks, but otherwise, well, it's event sourcing with all the advantages and disadvantages. I think it's very applicable in this case. Single-tone? No. Would it be this bad? So turn, well, that's hard to tell, right? I know that... But if you had enough trouble that you would look elsewhere? Yes, we basically, you know, having periodically we have a problem with them. So for one latest one is that it was failing rights to Cassandra. And because of that, I'm not sure if it's by design or quirk or a bug, but if right to Cassandra fails in persistent actor, it is stopped by passing all the supervision. And in case of single-tone, it doesn't restart when it receives a message. And it turned in basically just one failure, crashed entire, well, it didn't even crash. It just stopped the service. All the nodes are running. No rebalancing happens. Service cannot serve requests until restarted manually. Sharding, I think I would use it. So it's a very, you know, it has some complexity in it associated with it, but I think the benefit it provides is worth it. D-data, carefully, you know, replica of entire states so that you avoid, you know, sort of broadcasting to all the actors gathering responses, very useful way. But, you know, returning to my remark when we removed most of the state from D-data, we actually kept a list of whatever it is. And this list was updated on every reservation cancellation. So when we removed that part and switched to scatter gather approach, performance improved like probably at least five to 10 times. So, need to be careful with that. So, but if the active resistance and sharding can be recommended, the rest only if you... If you're sure, limited recommended with boundaries. Single ton need to be very sure why you're using it. What do you mean by scatter gather approach? You just basically broadcast to everything and just wait for responses. So scatter is broadcast, gather is get responses, join them together, respond to the caller. How is that alternative to replication? It's not an alternative. So in this particular case, what we tried to do with D-data is to have a list of like, if I want to cancel my reservation, I need to know where it is, like in which particular actor it is remaining, right? So what we were doing, we kept a map of reservation IDs to whatever, like actor Fs, whatever allows us to identify the actor. At that time it was crashing because of out of memory with something like 15 to 30 requests a second. When we removed that D-data, keeping all the book off reservations, we now can sustain like 150 and cancellations still take reasonable time. So it was sort of an attempt to optimize so that we just, for reservation, for cancellation, we look at D-data and then we send one message to one actor. Now we don't do anything, just send to everyone, someone responds, fine. No one responds, we didn't have that. Still works well. Have you said something that's forced data affinity like night in our study? State affinity. Data affinity. Data affinity. Basically, what was your first approach? It sounds like you did a manual data affinity kind of setup for some products like Hazel Castrate 2 and Ignite Depth is for data affinity, so a shared job will reach the node where the data is actually located. Yeah, well, so not exactly, I think. So we didn't kept where that data is. We kept a reference to particular actor which might be somewhere in the cluster. We don't know where. We just need a logical handle to that actor to send the message to. So with this optimization, what we try to achieve is that on every node, we have a replica of sort of addresses where to send these cancellations to. Like, I don't know, a phone book to some extent, right? So because of the cardinality of the data, like, we could have hundreds, well, up to 10,000, actually up to 70,000 records, right? Something was, you know, become broken in the data. So it wasn't about keeping affinity of the data. We just wanted to replicate it everywhere and just replace scattering, you know, broadcasting to all the actors. We was just looking at the one single records in distributed data and sending one message. That's it. So Hazelcast, we could have used that. So, well, practically a fulfilling pipeline service uses Hazelcast, right? But that's sort of a different, I don't know, metaphor. It's just used as cash as a service, right? Which is, that's deployed to other services. Why haven't we, because we had distributed data, so we decided to give it a try? Sure. I'm going to say, Kafka, would you think about the end-of-stuff system? Kafka? Yeah, much of Kafka. Yeah, so we're looking into it. In practically, we don't have it right now because infrastructural, like our DevOps team was not ready to support that. We're gradually moving there. So there are a few integration places where we can start. One obvious one is to stream state changes from transport capacity to Ratshift, right? Alternative would be to use Amazon Kinesis and Firehose, but there are considerations of vendor-specific solutions. Even the Ratshift is already a vendor-specific solution, but anyway. Yeah, so we're looking into adding Kafka somewhere in the picture. Not sure where, except for this one particular. Maybe actually reservations, cancellations. Yes? Sure, on the back. Louder? Yeah, I have a two-part question here. Sure. Let's say, you, I'm not familiar with the cluster name or the cluster name side of it, but I'm just curious about what is your partially key? So how does it get guaranteed that one message, which is actually submitted by a user later, actually goes in the front as well? Okay. The second part of the question is that I see that the Gossip Protocol is being used for authentication. Yes. The Gossip Protocol is eventually consistent. So what it means is that one of the nodes actually, the leader gets a message and if it goes down, then the message is lost. Okay, so let me try to answer the first part. The first part was, yeah. Partition key, right. So like operations in, like we use sharding and partitioning, which is basically partitioning in transport capacity. Red Marit model of operation in transport is that we have shifts that perform deliveries. Each shift have allocated a certain number of vehicles. Right, so all the capacity for that shift is reserved for that shift. And we basically use that shift, whatever handle name as a partitioning key. So the goal here is not to, I sort of foresee your next question. The goal here is not to achieve uniform distribution because we know that shifts that relate to tomorrow and day after are more busy than the ones in the future. Our goal is to have sort of an entity that we can serialize access to, right. On the other hand, shift name is just a string basically. So consistent caching on strings should probably give us reasonable distribution across the nodes. Okay, second one about gossip product, all right. So this one is used for the part of our system that can tolerate eventual consistency. That's it. So here we actually do have some eventual consistency in the system. We can't afford it because it's basically used to speed up reads. We are not concerned about sort of showing slightly outdated view of the grid to customer. We're just concerned about not allowing the customer to place an order if it's outdated, okay. Does it answer the question? The gossip leader is the one which is I think gossip leader. It's not, there's no leader there. Yes. What I mean is that say, who is the one who is actually maintaining the gossip? I mean, there's a leader in this one anyway. So the one who has received the data is the one who is communicating about the node. Yes, of course. Once it has received the data and it's closed down. Yes. So we have a sort of safety mechanism. So basically what we have in distributed data is a replication of internal state of all the actors. So if the node goes down, all the actors are restarted on a different node. And last thing they do after being recovered is to populate their internal state. So it's not automatic, it required a little bit of manual work, not much because it's like, we just send a snapshot of the state to some service and that service just propagates it to distributed data. OK. OK. Any more questions? Sure. I think it's mentioned already, so it's about a lot more. I'm not sure where the answer to all the issue that you mentioned just now is regards to long-run or current implementation. Just given that you're using event sourcing, do shading, and use active assistant, so it seems like a lot of nice fit, so have you considered that? We did consider a lagom. What stopped us is that LightBend actually have a paid product that runs it, which basically means that it's not that trivial to get it to run. We actually have in our team, in other team, we have a guy who actually did lagom and implemented the service and production with lagom, except he teared away everything except resistance and the HTTP. And he had to do some learning curve, had to ask in forums, and so on. So lagom is probably very sophisticated, very performant, bently probably, right? But I don't think we can afford just ramping up on that learning curve. Welcome. OK, let's wrap up. So thank you guys for coming. Hope you enjoyed it. Yeah, cool. Yeah, I have. Thank you, guys.