 How much time do we have? How much do you want? No, how much do you can bear? When people start trading, I will try to trap them. OK, just so, I guess we planned for something around an hour. We were expecting to have, yeah, a sliding dose. I leave room for other people. To make it easier, we can also make a pizza break halfway. Maybe a little nap and then get back around midnight too. OK, let's do that. So this is teamwork. So I'm going to, we are going to show you, to explain to you or to try to explain to you what we are doing and what we did after the match. My title, that took a lot beyond relational database. It's a little bit provocative, I know. But I love relational database. It just matters that we are trying to do something different. So the idea of this book is to introduce you to some technique which is for designing which is called pen sourcing that you might probably know already. How we do it, practically speaking in Haskell, and why we are doing it, and how far we are pushing it. How many of you are familiar with pen sourcing? So, yeah, OK, so not that many people. Actually, we are free in front of you, but there is a false person in the back, which is you. We just recently joined the team. So we are now four full-time developers working on this. So there in there, I will give a quick introduction. I will be talking about the event sourcing model. I won't try to give you too much. I will try to give you the idea behind it. It's pretty obvious, basically. I think it's pretty obvious. But the reasoning behind it and what is the impact of this technique on the architecture and design of the kitchen is quite deep. So that might be interesting. And so this will be the painful part where we are only talking and showing slides. And then there will be an interesting part where Amar and Goyal will show you actual code. So that will be the second part of the talk, where we took the implementation and usage. And there will be a third part where I will talk again. Apologies for that. I will talk about how we plan to evolve our current system to take into account different features and requirements that we are planning to build into the system. So who are we? So we are technical match. This is a quick capture of the websites. This guy is very happy. So you can see that we are selling happiness. Actually, not. We are the leading peer-to-peer platform in Singapore when lending to SMS. And our system was... So we started the operation in 2014. But legally speaking, we started in 2015. We thought implementing the system in 2014. And the backend system is mainly developed in the hospital with the front-end. We won't talk about the front-end because the front-end is developed in the closure-strait zone. So this is the husband of the group. I will try to join the closure-strait zone in the closure-strait zone. Actually, I found a good way for people to pay me a beer. So I'm traveling around the globe. I don't think that's the covariance group. So you can buy me a beer for that. And the cordial button, too. So there is Amar. Although I must go along and join you. As you can guess from my accent, I'm not an English or an English-native speaker. I'm French and I'm actually living in France. And I'm working most of the time in OP and sometimes I come to Singapore to meet people on this list and drink beers. Okay, so down to the meat of the pork. So we all know what the original model is. We have been brought up in that model. We have used it for a long time. We have been taught how to use it at school or at university. The chemical example of the traditional model is the employee-person address database. And what do we love in rational learning? The first thing we love is really great for querying. It's a great model to start querying and generate reporting tables, extracting data, gathering data from the model. And actually, an SQL wrongs for that. It's a really great language for querying stuff. And SQL has evolved over time. It's really powerful language. And if you take into account the expansion that I've built with some databases, it can get really, really powerful. I've got a friend of mine who is working, he's really an expert with Postgres. And he showed me things that are really amazing and what can you do with that? Building a type system in Postgres, for example, doing stuff with geographical data, which is pretty basic by the way. So it's supposed to be very simple to understand. It's everything's stable, and then you can start drawing. Pretty simple. And then you build the full algebra over it. And a third advantage, which might stem from the two others, is that it's pretty ubiquitous. It's everywhere. Everywhere you go, you will have at least one or two databases to get data from. And it's really, it's really everywhere. It's been around for 40 years, for 50 years, so no surprise. The question is, what's wrong with the national model? What, maybe not wrong, but maybe less, less, more painful. Rise and death. I mean, when you're rising to database, okay, if you're in certain one data, yeah, sure, no problem. If you're starting setting, you're setting projections of the data and there are the database maintaining integrity, the differential integrity, you start to think, oh, but how do I maintain that? And start to have neutral primary keys, constraint, whatever. It's starting to get pretty complex. There is the so-called universal mismatch. I've read articles around there about the fact that universal mismatch can actually be a strong match, but I really do think that, well, in my real life, in my life, a lot of time, I've been dealing with data which was not really that much relational. I mean, most of the time, a lot of the time, data is mostly tree-ish, so tree-ish is still okay, but then graph-ish and graph-ish can be pretty painful to model in the database. We can't do it, obviously. I'm pretty sure that a lot of the data in Facebook is stored with a relational model. But there's a reason why there has been a furrishing of non-linear square or non-relational words in the past years. Because after 30 or 40 years of using only exclusively relational model, I come from a Java background, initially, and, well, I've been beaten by hibernates one too many, but I've been beaten by that in the instant, actually. And then another advantage is that I've been quite a bit too aware that every... I told the place I've worked, the database soon became a single point of failure, and it became the most critical asset you had in the company, and something that was, I mean, more important than the Chrome duos. Because when you say, yeah, your data is very important, your data, the data was sitting inside, was residing inside the very complex system, and I look at you, I look at you, Oracle, which is something that, when you want to start using that, deploying that, maintaining that, you can get some criticality. And there is a reason why database-only structures are much more pain than developers, because they are more important, because everything is in their relation. And then the cherry on the topic, that basically what is a relation database? What is the database? It's mutable states, states which can be mutated. So, of course, I've seen that, I've seen an implement that. You can model immutable states with mutable states, and you can start storing data with version... with timestamp or whatever, you can do that, of course, but basically you still don't have a mutable state. And once you update your database, once you update your row, and once you change your row, the data is going forever. Not always, but basically forever. If you are unlucky, end of delete forever. Yeah, exactly. So, in the end, what we ended, is that we ended in a world by the database. There is one single word, which is we have that... I've worked on that kind of system, so I've been envisioned by that. And you have the database server which sits at the center of the universe, and then everybody is talking to it and trying to work with it and trying to adapt. And then all of that need my table there. All of that, by the way, there are many conflicts over, and then we need naming rows for tables, and we need to start to have schema reconciliation meetings and groups and whatever. I'm looking back a little while here, and it's... I mean, well, I didn't want that. When we started working on Total Match, I didn't want to go into that nightmare. And besides, I'm not a database administrator, I'm not an expert, and I didn't want the world of maintenance. Of course, we could have used that. We could have supervised or any other service providers, but in the end, you have to put your hands into the... And there is event sourcing. So what is event sourcing? In a nutshell, event sourcing is... It takes the idea, basically, which is obvious for any way of questioning, to come to terms with that. Well, basically what we are meant to be is a world where we have states and we are transitioning into states. A lot of... A lot of the people in the world are doing event sourcing with more or less usefulness, but the idea behind that, yes, we are transitioning into states. The database is actually something that says, okay, I'm in one state, and then there is something that rules outside, which says, how can I transition from one state to the other? So the database is actually a state, or one state among all states, the idea of event sourcing is to say, well, but why was it transitioned? Actually, we have one state here, at the end, in this state, we can arrive at the states by two rules, and we can say that after all, only this state is important, but actually knowing how we get there might also be very important. So that's basically the idea of event sourcing is to start thinking about... at the end, we always add states. Of course, we have something which we have... we are at some point in time, but the idea of event sourcing is to say, this point in time is a result of all the events that occurred to that last to that last state. And what we are interested in, we can query a state, we can also travel time, and we can also manipulate events that went back to that last state, and the core data, the database that we are manipulating is actually the stream of events leading to this state. So that's the basic idea. And that's what's in these slides. And besides, once you have the traditions, once you know what has been the traditions that have been through, then it can be lost through the state probably, you can say, the data will be lost. For a month. So event sourcing came out, this is a quotation from, guess what, one for it. And it wrote the half scale somewhere in the mid-20s, in 2005. So event sourcing is basically the idea that what we store, as what we persist in our system, are the traditions, are the events that impact and change the state of the system, and not the state. So we still are the state, of course, but the state is in memory all the time. It's reconstructed all the time, and when the system crashes, or when we build the system, or when we want to restart everything, we reconstruct the state, and when we want to change the state, we restore the event itself, not the state. Once you get events, once you have those events, then it gets easier to get something which banks love, auditing. Auditability of your state is something that's very important for a lot of people, especially for financial institutions and banks. So the idea is that, well, you know, you can know precisely what sequence of events led to that state. How come there is $1000 in this plan? Well, you know, we have organizations, and obviously that's obvious for banking, there is actually a very close, it's very, there is a very high proximity to the model, the actual model of the connection ledger, but it's also true for any kind of model you can imagine. While you you are in that state, it's very important. You can implement pretty easily, very easily generated, so you can start traveling, and say, oh, I did this event on my own, but I don't want this, so I can travel back in time. Which makes it very easy to translate this. One thing, I've been working at a company which was doing financial software, and one thing which was really, really hard to do, and which was really critical, was simulating bars, something portfolio evaluation, for example, according to different events, assuming that, for example, we've got a spot, the Euro-USD spot forward changes, what happens to your portfolio? Basically, what you can do with that, you can recall that event, or events, and get back in time. If I won't want that. It's somewhat easier to go with data migrations. Migrations, when you are managing your database, original database, it can be done. There is a software that I implemented in the library that was working on the liquid base, which was a software that I developed. It's always something that's a bit painful. You can do it, but it's not very easy. In our case, when we are using it, anyway, it's always something which is rather painful, but the good news with it is migration is actually something that's really local. It's only local to the format of the event that won't change. If you don't change anything else to your model, the model that you are using in RAM, you don't have to change your events. You just have to change the interpretation of your events on your model, which means that you don't have to migrate your data. When you have to migrate your data, it can still be painful because you have to reconstruct. There are migrations that you can't release, of course, but it's easier. It's more localized. We have to do some work with that. Another thing is that you can deal more easily with potentially conflicting change. In relational database systems, the transaction ledger is actually an event sourcing system. What happens is that the RGBMS recalls the sequence of changes data, and once the transaction is committed, it just compares the ledgers and applies if there is no conflicting change. What that does in sourcing is giving you that pour into your hands and really finding that you now manipulate it directly. An even more important thing is that events drive business. When you are starting to think about designing some business, when you are working with business people, business people will talk, if I do this, then this happens, and if I do that, then this happens, and then you can manipulate it in terms of condition, state, and friendship, but transition events are really important and they represent the domain model very faithful. That leads to some interesting techniques, design techniques, which I will go into the state of, which are part of the domain driven design that we are talking about. Event sourcing is also known to domain driven design. Domain driven design can be done more easily with event sourcing, where you can extract events and start to think really in terms of business events, not only technical events. And there is again more or less recent techniques for event sourcing, which applies this principle, but really for user story mapping in job terms, where you really work with business people to extract events. Okay, so those are the principles. In practice, what happens? What we do in practice, and I will go very quickly through those slides, because we will show you the actual work after that. So, the basic architecture of a service that we implemented goes that way. So, we have some REST API and some GCN coming in, and then we have our functions that hold services. And then we have those services that talk to some event sourcing engine in the between them. Those services can talk to different models. So far, I don't really know what I'm talking about. We have a model for managing class activities, and we have a model for managing investors, and we can add to those. And then the generates events are persistent into an event store. So, the core, the most important thing is that the business model is pure. Pure in the sense that it's immutable. It lives in the pure beyond of Haskell, where everything is pure, and there is no I.O. You cannot fire missiles, you can change the world, you can touch fire, and more importantly, you cannot have concurrency problems. So, when you work in that world, then that's really great. And a basic idea when you're designing and writing a program in Haskell is to try to get to that pure core as fast as possible and leave the rest out, because that's where dragons are. So, the basic idea that each business model is pure, it does what's called a boundary context. And it's responsible for a single cohesive part, where we have a bunch of events and a single model, and all events are applied to model. And another important thing that we will distinguish commands from the elements. A command is something that comes from outside. I want to do this, for example, I want to apply for a loan. And then the business model should know, okay, if you want to apply for a loan, then there are business rules that say you need to meet some condition, and if that's okay, then we can emit an event, we can return an event, we say okay, loan applied or loan emitted order. But the idea that events represent something that has happened in the past, something that is always true. You cannot reject events, you cannot, and events are always valid. If an event is invalid, then this means that everything is, the world is valid, so you better stop your system and get back in time. Commands can be rejected, that's the main difference. So, technically, I still have two lines. What we have, the core of our model is actually two functions. The first version is what we call acts. Act takes a command, a model, and generates an event. And then we have the apply, apply takes an event, a model, and return an ability model according to this example. Very simple. This is Askel 101. So, if you know how to write a function like that, then you know how to write a function like that. And that's one, that's something that, that's the reason, one of the reasons that we have started using that whole, that it's really a good fit for a functional program. It's, and it's really a perfect fit for a Haskell, in my opinion, because this is basically, when you start designing about Haskell, when you start forming Haskell, you start to sync in terms of immutable structures, and how, when you, you never anymore mutate states, you just create a new state, so it becomes naturally, you start to sync in terms of, I mean, that state, how do I go from that state to that, to this state, and then we have a function between, well, events is just a way of really finding the future, that's natural transition towards, towards, and then we have effectful services, so the services are, the services are represent more or less complex interactions with the outside world that can interact with several others. And one important thing that we don't have is the interactions. So if you have an error somewhere, then you have to deal with at the place where it occurs. So you cannot say, I won't run for some technical constraints across ten models. If you want to do that, then you have to implement one service that's responsible for that. So the typical use case, the typical case for that is the classical transfer between two accounts. Transferring money into two accounts, actually can be something that you need, some traditional integrity. But when you're modeling with an event, it's just one event and then you don't have any more traditional integrity. So you, actually by forcing yourself to focus on some part of the domain, then you can leave without enforcing integrity over all the domain. And, okay, that's a little bit more daunting, but not that much yet. So we have to express effectful services, we have a monad structure which is the eco-website monad it's actually a transformer which is the UT and once this monad is made of is that we have a global model the G which lives which is bounded and which is encapsulated inside STM so to access the model from the outside world, you always need to run it into a STM connection that makes sure that that takes care of logs for you. Then you have the local data which is something that can be local to a single traditional service and then you have the underlying monad on which you work which usually is IO but we'll see maybe some example of using it differently using something different IO for example for testing. And then in the end the storage of elements is pretty easy you have a very simple structure where you have an event and an event is something that the most important things in the event here are the event version which is a moderately increasing number which says, every time we change the scheme of a single event this means that we can be constructed with the solutions that we do that we load all the events and we reinterprets the events from all the versions into the new the current version of the system so that makes migrations rather smooth and then you have the type which is a classifier for the event variously data that we can use actually they are interesting but once you get events you get something that we recall for each event in the user for each transformation of the system we know which authenticated user was trigger detection and we also know the request ID the outside trigger that generates that event this means that we can have traceability between the system and we can know that some request generated values even somewhere and we even have something which is the SHA one the version of the code in which the event will come so we store the version statically so we know even if it's more precise than the event version this one is actually a natural code version so this makes it easier sometimes to track that that's the get through version let's start the get through version it's the git channel and that's it so for storage the good news is that we can forgo the use of a complex database server we only have our events are stored in an app and only file in the file system so it's a simple file and we continue to write to the end of the file so it's very if the file system goes bust we should stop every second or not okay enough talking Amar I hope that's it's okay yeah it did oh I don't know okay I will just I would just be nice just to leave you playing marines as a dog already talked about we should have two different things one is more advanced users that are not already taken care of beginner users and what I'm not what I'm going to do is try to show how we can map to model it in a high scale preview so what I try to do is just pull out some simple library from our codebase minimalistic I will say so the most important part of our even so single can you see at the back it's a bit small yeah it's a bit small can you increase a little bit plus a bit better so basically how we import our business model in form of type plus I hope that most of already know type plus it and how we make it type plus as I'm not mentioned there are only two important functions in this business model act and apply actually you could have an apply function but then your event should be like a git that is really a three command rate practice I talk about why that is important so what act does is it takes a business model and a command and runs some business if business will fail apply is again as simple it modifies the business model according to what event and this is a default business model because what we do is just fold over we just create a state by folding over all the events so we just do yeah that's it that's pretty simple actually we can go and see how we actually use this simple class and there are some library functions down side in our actual code or like relatively like actual code I mean sometimes I think so how is it made so this is a so I thought that the demo is a good domain to understand because it's a demo so yeah this is a basic example of that how every business model will look like so as I mentioned there is a command which takes part of the model if they are external actions and add them as simple as a grid then event are historical facts about our business model which cannot be changed actually so you can distinguish like this commands are coming from outside events are generated by us and we know that they are flawless we can apply blindly to our business model our state errors again are just algebraic data type some type so basically this is an event act is just by pattern matching all the commands we validate validate commands the incoming data in the acting and if it is successful then we create an event which is which is relative to that command and it will answer simple and we just we can we can just stop coloring basically that's it do you have any question on this example then we can go to the services how services we arrived late but I was interested by the code before where you mentioned that you needed a code for beginners and a code for adult users so maybe there are persons in this room that don't understand this code is this a case no it's a case I don't understand the code because I'm not the expert the real goal it might be the stuff of my friend this is the stuff but I think this is just something to verify it was not the last code but then it's pretty easy okay don't worry about it yeah, so basically two functions we event by pattern writing on the type okay just for those who really don't understand want to understand I want to be sure that it's very simple actually because what you see here the important thing is what art is implied and what I can apply do is actually just a description of the function code for different parameters so we use something which is pattern matching and we pattern match on different agreements and say okay if I have a demo view so demo view D is a structure which is called demo view and which contains a matter which I call D and I have a demo added with a demo object and I can change the map and the interesting thing that you can see here is that we are reconstructing a full demo view object so we are really, it's not the same object it's a new object it's a different and that's what makes the model mutable once you have a reference in the model you know that it will never change most of the time it sounded before as if you had like one global order, one big scheme that where you record all the events many different models we currently have something like 10? somewhere like 10 models and then you can add one if you like it we continuously add new ones so for example we simply added one for emails to manage emails we are going to add the idea is that each time when we are we think that there is some well it gets pretty fuzzy but the idea is that we don't want our model to get too big we already have two big models we already have two big when you start to add say 15, 20 events it starts to get too big that's the smell that probably you are managing too many things at once so yes we try to speed models does this answer your question? follow up question sounds as if it would be very challenging to have to allow two models for the same kind of events for the same kind of commands but here for every model you have everyone has its own commands and its own events so every single model you need your own events store no, that's not the case we have some mechanism that we have actually we have a global grouping of models it can get pretty technical, I could explain after maybe that could be we have a way to group different models at a high level in the application but the idea is the idea in the end is yes basically it's not really a problem if each model has its own store in the end you just rise store so if you have any models and you have entered files it's not a big deal it's not about maintenance and managing those files but just plain files actually we have both cases we have grouping of models they ask you some new models which we have in stores because they don't relate to this core grouping of models that's true the idea is just to separate things I have a question about as you mentioned if you understand there are a whole bunch of events that you have added that you have to deal with and whenever you receive one of these events you run back they will view objects and then you do something with that what I don't understand but what do you actually apply so act act is actually how we create those demo added, updated and limited events so you can see that act actually takes as input a view and a command and it generates what is output is is actually an event so this is a pretty complicated code but not that complicated but you have to hear the idea that some function evaluates the input and then from that input you can generate different events so act is a way to say I'm pushing a button imagine that to give something more physical you push a button and the machine and you have a button you push a button and it gives you a ticket and this ticket is actually a number that you have to enter into a pip app to change the state of the machine so pushing the button doesn't change the state it just has to do something that can you, can I push that button and then you take you take the answer and you put it back to the machine I'm not sure if the analogy is better but maybe you can read this if I understand what apply does because apply actually do the act so if you apply it takes a model it takes a world it takes an event that should change the world and it returns a new world two questions the first one I'm not sure if it's a sensible question but because I'm not quite sure about how we solve this together but what about interactions between models so that's what I said before the idea of interaction between models that it's handled there is the basic principle that there is no interaction between models they live in isolated island what you can have a service that interacts with two models basic example we have in the platform we have something that says suppose for example user registers that acts on users model that generates that user register and on that generation of user register we send an email so send an email and acting on different models so our services are orchestrators between different models each model is it's a independent entity of your business it shouldn't know about each other okay so you never need for inter-model consistency you never need an event which only makes sense if there is some other event another model as well actually we probably need some form of consistency but it's handled at a level of service that's the idea the idea that no you don't want something that ties two models you don't it's a matter it's a philosophical choice I can't say we don't want to do that because we want our models to be very flexible and we want to make them and we don't want to be we work close to building a big model we don't want to do that because it's more harder to change so the idea that we refrain from doing that so it's kind of it can be a straight jacket at times but the idea that the advantage that you get very self-contained and all the other other okay that was it could you give an example of the models like we are like the models that you have defined so have you identified other definitions of models which might not fit in this so going on with SHO after MRI we show you some two or three concrete examples of what we do with that actual code that we use in production and the way we are using those events for doing some special effects that might be more easy for you to grasp the actual model the model itself doesn't guarantee that I can end up ending because it's a mistake because because I understand I can take a demo model and have a command in years by basically grand alone which basically returns like either through or false somehow but it doesn't enforce that I put that event under the same stage because you're basically taking the state of your data asking about that and then you can get a load I think in the meantime some other event came and said okay deductible account and then you're applying that no because these models are pure and are monoclonal so it's not possible to generate a command to generate okay in the function you see there that's not enforced but in practice that's enforced we know that once we apply a command we generate an event and then we immediately apply okay that's enforced but that's enforced at the level of machinery to do the to make it work out sorry so that's an interesting question because I just realized that as you mentioned do you think about using aggregate IDs anywhere to essentially be able to identify any of your model instances what do you use as the equivalent of aggregate ID from the processing which corresponds to the primary key in the short term we have for Americans if you can show the model number maybe the demo I mean the demo here it's a map it's a map from a primary key to a row it's a table just it's a table but actually it's a table with a single primary key we have models where we have the same data with different indices we have indices so there is no magic the data has to you have to find users so you need user IDs you have to find nodes those principles don't change what changes is normally the way that you store that and how you reconstruct state from persistence so in the end your view is actually it can be pretty much anything most of the time it's a map it can be several maps different things and we need for example to do some drones or we are just managing two pieces of information for a single type of object typically a user we are managing for example authorization tokens and we are managing this profile as a user so we have the same view it's the user view we have two tables it can be maps, it can be sets the nice thing is that most of the time because we are still working with some financial data and some traditional basically it's an accounting system most of the time we are mentally always the master table or at least as something with a primary key to something with attributes but sometimes we have the freedom to have sets trees any kind of general structures that can be very interesting sometimes not all the time but sometimes it can become interesting just one on that side what I've seen though in relation to databases for example in learning a set in learning a set as a table then you have four or so semantics of a set outside the relational model or you have to constrain from uniqueness, constraints so it works, you can do it but it's not to me it doesn't seem very natural because you have to somehow constrain the but can I correct you saying that when you rehydrate or read back from your example that the whole map or that whole table gets reconstructed or do you individually put it to the aggregate objects the equivalent that I can think of is of reconstructing all of the user objects in memory in the memory contained data storage or data structure as opposed to individually retrieving a person from the database having only one object for the one individual person so what we do is actually we reconstruct everything we apply we fold the whole stream of events I think it's a classical functional programming practice to use fold which is a reduced transformation of the and so we apply a fold we fold a line on the stream of events to an empty model and and then we reconstruct the model the thing is that usually this builds the whole model in memory what can happen is that if you have if your model is chosen if you have the right stricter you can take advantage of laziness and then you sometimes you might not reconstruct all of your model or you might not reconstruct all the parts of your model all the time so if you want a granularity which is higher than what you if you want a smaller granularity than I mean the big chunk or big table then you can split your model of different entities and then the fact that when you apply an event it's actually a function which is stored and which is applied lazily will magically or at least somewhat magically do the work for you and you will only execute and load the code for you that once but there is a the thing is that because the loading the reading on the file the sequence flow you won't you still have to load the raw data in the file to load it but you don't you are not forced to reconstruct everything do you use a crack point thing or do you always it's part of the plan but we don't the volume of data we have doesn't really okay so the the individual self-contained domain model what we do is we just contain a state in memory of multiple domain model I can show you how state looks like state is simple collection of different there is only one view for now but they can be multiple and what we do is in service so basically this is divided into two parts one is how you compute your business how you compute your business operation versus how you store events and update in memory state right so computing the business model is taken care by the business model part and storing and updating the state is taken care by this is our state so updating the state is taken care by services so I will show you this type for service how it roughly looks like for us simple transformer this one so we have a simple transformer okay this is quite long thing but what it does is we just have in memory state which is in t-var that's why we cannot have inconsistencies because we can only have one operation right operation running at single point of time and obviously we have middle errors using this is not our real model that one is how I am not show the web straighten this is a simplified supposed to be simplified okay so our service algebra looks like this and our services are simply the programs written in this monad so this is our services so creating demo is nothing but I will show what apply command apply command takes care of both updating the state updating the state atomically plus storing the way but these two things are not atomic I mean updating the state in itself is atomic two threads cannot update the state at one time that's what we mentioned that's what's releasing the operation you can have because technically you can act in a way so if you have two acts in a row with the same model they might generate different events so the role of apply command is to ensure that acts can apply and also also I can see that there are some cred with real without real operation and there are real operations which are nothing but reading from the web of S which is like because of htm it's working you can multiple reader at same time two threads doesn't need to be weighted neither of them is writing yeah so basically this is the real operation you can take an example of odd demo what we do is just read the key where obviously and just filter whatever whatever competition or whatever filter you want to apply to both state you apply it here the most interesting part here is apply command apply command operates in a service sorry not this one apply command operates in a service what it does is basically only two things as I said it actually applies updates the state using the generated event and also it stores event to the position the catch here is these two operations are not working you can see there can be chances where apply command is succeeded but store is paid and then you don't have any anything to hold back that's why I mentioned when I demonstrated the business model you might need hit like commits where you have an apply function I remember some case suggesting vector patches that we might try but not currently our state can be actually store can be apart from that I can show the if we even if command fails to apply we just throw it in real life in the actual position curve we don't want the consequences to what we do that we serialize we have a queue to serialize actually command application acting application and store so if any of this payoffs then this three operation app this is where we are hitting on as I thought a limit it's built in future the fact that we have all of that in memory so we can act and we can manage it and we're working but we cannot do that so how do you ensure that you are actually actually updating the model and storing the data because if you don't store the data then that's just like we are not updating the model that's why we serialize different commands it's because for our reads operations are not in that mechanism so they can be run and we can still ensure that we don't have that same thing and then you can have several atomic packages right now the one thing that we don't want to occur is I'm sorry actually there is another mechanism which we push go a bit too far but you can ask at some time two elements that are reaching once the event is reaching that's done automatically and can be done comprehensively so that we can we definitely can do that so this is a state application which you can see it sits in it so this whole world is coming so also this code is very simple one business model at a time that's it sir we are already out of time I'll just show some wiring of the how we wire it but this is really one for you so leave it there go ahead so we wanted to show you some actual use of events that are somewhat specific to the fact that we are using events and that's what's going on with the actual use so usually how the events are created with the ultimate goal of updating the state is that going through the business model the user needs to request this those issues on business logic and it gets verified as an event and event gets stored and that affects the state but sometimes we might want to do some data patching which we haven't written the business logic for in the code so something we can do is to inject events directly if we know because there's a state we want to get to and the state we have right now and instead of going through the business model business logic to arrive at that state which is intended which we haven't got the feature implemented for we can just inject the event directly so for example when we want to change the investor type from an individual we don't have a specific update investor a change investor type command but we so happen to have the update investor and this is pretty brute force way to update investor basically it says the investor is now this way previously so it allows many changes to the investor so we get the investor up here this is just to convert the investor ID from the email I want you to have the idea we get the investor and when we get the investor we do the changes of the investor with the life and with the new investor we update the investor at that investor ID with the new investor so this is essentially the same way you will update the state which is to issue an event instead of going through the commands of doing all the validation you do it the unsafe way I suppose which is what you will do with the data patch the nice thing is the fact that we are in so it's just this is not what we do normally it's just sometimes the fact that we have the application we have built the command line client so sometimes somebody makes an error which is not a technical error it's a practical error we need to change the data we need to change something that we didn't make it change that's one one disadvantage of our system that if you have not something that says you can change that you have not the exact command then you screw you can't change it period and you can do it easily but on the other end if you have something which is more general but not open not open to everybody you can use as a kind of a generic event or a generic command you can actually use it in a control way to patch some human errors and still get a guarantee that no event is lost and you still get the fact that it just may mean that you get data based in production nobody changes data in production using SPL except sometimes you do and sometimes you do and guess what when you do that usually surprises or you've got somebody who made a big error and you have to do it right now so you have to do it and you do it and the problem with the database is that this change gets lost it has been changed and that's it the fact that what we are doing here we do the same, we do something ugly and patched data in production but we have the advantage that it's not even more control way we know that we have changed the data and it's part of the history of the database of the events so this makes the system really stable even if we leave some backdoors to be able to patch data in production which we always do we can do it in a control way so we have a bunch of those scripts which are used from time to time to change to really do some time change here and there the advantage of doing it with you don't have to with with tables you do the edits directly in a flat way you basically have to calculate exactly the patch you want to make and then issue a patch but with a script to compute the new event to inject you can actually use all the pure functions you already have a goal of certain invariance and which makes sense transforms the data in a sensible way you just compose it differently in the script so even though the way you compose it is not currently how it is in the business model this here scales the transaction you doing it by hand you would have to I don't know, lose a calculator and then inject the data but in a raw fashion but here you can just get what the transaction is currently in the state apply the scale transaction pure function and then you get the result and then inject the transaction in that way so basically what we are doing is that we can use SQL in Askeladd except that it's Askeladd we have a bunch of SQL scripts but we are doing it in Askeladd we are managing our data in Askeladd maybe you can show the last one which is anonymization which can be so something that we do for example is that for testing purpose or local usage we have an anonymization of the database we protect a portion of the database and we run an anonymization of that database and we scramble names, numbers, emails a lot of different personal data and what we get is that the nice thing is that you can see here in action the fact that to get actually to change the database we take a stream of events of stored events we apply some transformation functions and we get a new stream and this stream can be stored as another database which can directly be used so that's something which is interesting for example for anonymization for example if you have a query and when you want to trim down the database you can back in time takes the database and for example it's very easy for example to take a snapshot a snapshot of the transaction at some dates because we roll back the data with the database and say we issue the query for example to say balance of all the counts and that's it and we can do it all the time you have the question I don't really question I think you should stress more that because you know of the user for every event in retrospect you can see who changed what so basically you have more accountability for all your data in your system I mentioned it in the beginning but at that point where you said we can patch up I didn't mention it actually I do that to connect to the system you are identified it's not about your we are going through the API of the system except it's an administration API but you are identified to if I'm connected to this credential I know that I'm changing from data from where or I change from data from where I didn't mention that how does it work to execute a script like that if your model is in memory in some pros the model is always in memory when we do something in that case no, in addition no but for example when when Gorian was doing some some action inflation we reconstruct the model in memory but then why you do that your website is down I don't know why because you end up with a new model no it just enters new events no I did practically what we do we connect the website to the application we retrieve all the events so far we reconstruct the part of the model that is interesting for us at that time it can be all the model or it can be some part of the model we hack on it generate a new event obviously this is a less control because we don't go through the locking verification it's not something that you would do for a novel usage it never goes through some it never does it goes at runtime if you inject an event which can potentially be conflicting then that's potentially right but how does your main application then get the event oh there is an API for that there is an API for injecting actually there is an API for injecting events and there is an API for engine components that's a question I'm still thinking yeah so if you are storing everything in memory isn't that a bit of a big limitation because I can imagine like I want some kind of quote on quote table you know a table equivalent in your data store with a billion rows and the key is going to be an inch 64 so that's eight bytes and then the value is going to be at least another eight bytes so that's 16 gigs of RAM just to store that one thing and that's a pretty small table I can imagine if your values are email addresses then it would be quite a bit bigger so yeah that's differently that's this kind of stuff is not really meant probably not meant for when you got it a big data whatever the things but currently we are using DeteroOcean a provider the biggest machine that we can have on DeteroOcean is 96 bytes of RAM and what we are doing now is that we are splitting the nodes this means that we can actually distribute the nodes so the limitation the consensual limitation we have is that a single business model cannot be larger than 86 bytes assuming that we are using only DeteroOcean if we go to if we use EMI from a method I guess that we can provide something like I don't remember the exact years but we can go to two and real few bytes of course if you have the usage at runtime of that much data then probably at some point you are going to probably own your own machines or you're in the test lab or whatever I'm not I'm not sure about what I say but I say that for the foreseeable future for us for the next I'm pretty sure that for the next 2-3 years that's okay that's not I don't mean that for you it might not be okay I don't really understand the problem can you explain you've got an event log which can be big and then you've got model which is the state I'm not saying it's a model that's big like say Facebook Facebook's an excuse they've got a billion users and even if you're not very big I can still imagine all kinds of tables don't have a billion rows that's when we get to this level anyway when you get to the level of messy amount of data at runtime you have to use sharrings you have to use splitting your wall and even conventional use you have to use some techniques to do that you have to sharpen the data but not if you're going on disks because disks are way way bigger than radon we couldn't have a terabyte on this and then we can have a really small machine so it seems like you're kind of moving your things that would be limited by disk size now you're limited by ram size you can't build can't you start check pointing at that point yeah it's a different problem it's a different problem what you're mentioning Doug what Doug mentioned you can have very good models not very big persistent stores even for us we can have very large stores but when the store is loaded maybe we just have a million of events for the same tiny bit of model and then the model itself is very small there are two things which I guess you wouldn't really cover you were talking about event sourcing the one is that the truck is not finished the other thing of course is that not all event sourcing models actually hydrate the entire model into memory for example some activist event sourcing systems like aggregate id with a particular instance of a domain object because associate aggregate id with the row in your table and that row in your table the equivalent of row in your seat or your national table will change over time and each event update over time will be attended to the event log and then when you reconstruct just the instance of that domain update with that aggregate id only the events applied to that aggregate id are replayed from the event log so that your final aggregate state that you end up with only for the small domain object in memory is basically the realized state with the whole ordered row still in the back so in that kind of system and post scenario you don't have to reload everything into it that's basically shouting it's really shouting your model when you have a row say user profiles user profile at some point you are forced to show on your data if you want to manipulate it in some way and to have it accessible in some way and this always happens around the whole time even if you are I mean if you are if you are one of the one of the big players I mean even not that big player but if you have thousands or hundreds of thousands of requests of different fuses in the end the whole the whole part of data is getting close to get all your data you need to access profile of fuses all the time so you have to get it somewhere in memory so that's why you get maybe some aggregate or whatever manage some caching you have plenty of techniques for doing that what we do is it's basically caching we are basically caching or if we encounter that condition that well one model is 2B then maybe yes we are going to split the model it can be split logically that different models and different events are just scaled down or it can be split across a lot of you be shouting that's something that we are honestly if I had 240 hours a day but we don't have the problem right now but I know how to solve it so when we will have it we will hire 10 developers and build the new data store but right now I know how to solve it but we don't solve it just for the scale what is the memory that your largest model uses the largest model I don't know I don't have the current memory footprint application itself it's 3 4 it's very small we are a startup we are not an online we are not a B2C or B2C startup so we don't have 100 jobs request per second we have a small user base we want to we are more focused on integrity security and getting things right with the model and computation and being able to scale very well so we didn't we haven't tackled the performance I'm confused about you mean you are generating a new model they don't want to change to just a single row so you are generating a new model but the idea is that if the structure of my model is well formed I just have to change the spine in the model because if I change the outer object let's talk about ponters those are ponters so I have a chain of ponters in my structure if I change something somewhere in the leaf obviously I have to change only the chain of ponters leading to my name so yes it's costly but once again it's costly if you do it a million times per second it's something that we can afford to do right now and by the way haska is garbage collected so that's something that those old references are garbage collected over time so it's just a matter of it's not efficient and it's much less efficient than doing an in-place editing or in-place mutation of the variable except that sometimes under the room don't go but it works it's very simple conceptually very simple and then we let haska take care of the gory details of the runtime garbage collection reference counting spread management whatever but we do something very simple and really I mean I've been working in the data world and I have stopped working in data when I wanted to know what those minus xx flags were and how many of them there were and what it was doing and how I can treat the size of my new generation garbage collector to be I say okay I don't have that problem now we do something simple and this is very efficient for us this is not a simple product but I'm pretty sure that it can scale quite well and once we reach some limits then we can start applying some classical techniques of distribution to distributing to different machines shouting and we can push without a farther and just like you said actually we can diminish the granularity of each instance of each size of the model and get a single non-stream there but I don't know so I think we have exploited the one element I would just talk about future works so we have pretty lengthy discussion about this slide which doesn't probably mean anything and I'm thinking which is quite a tribute to the film which doesn't mean anything or it means everything so what I wanted to explain to express in that slide is that we are trying to push that model a bit far and trying to really understand and trying to go we won't we are thinking that this way of thinking is really fruitful for us and we make us evolve into something different and rather than adding a modelist okay it won't be a modelist it would be a bunch of services but the idea is the same so we think that we are just at the beginning of the world of evolution of our evolution as a team to go towards something that will be much more manageable actually I'm much more obsessed with the speed of deployment and the speed of delivery of features than in the actual performance because that's what I'm asking you so guess what, yes you're right once you have an insertion optimized for writing then maybe you can say wow SQL was cool we love to write in SQL and we think that writing queries in SQL is much better than SQL but sometimes a lot of users might not different users might have different points of view yes with something that we are thinking about is check out these are very relevant streams and then reconstruct different kinds of models and for example having a real model that could be a relation of database that could be populated so that's why I'm talking about getting better at CQRNs so really using our even store and our even stream to construct really long database and really new make models more resilient so we are thinking about starting to get replication formulas to be sure that we have some full tolerance and be able to maintain strong consistency and see it for schools so I'm exploring several initiatives of rafting and asking to see what we can do with that and then yes I have too much spare time so and then we have something which is really funny and I would love to do that but I'm pretty sure that it won't be done this year or even the next year but that could be really fun I mean it could be really fun from technical perspective and really changing from business perspective is actually what we have is a transaction law it's a ledger and this ledger is pretty technical right now but it contains actual business data and it says that somebody did something at some time and maybe sometimes it says that somebody did something with somebody else at some time so we start to have something that could be contractual so the idea would be to there is a project called Juno which is very interesting we see implement smart contracts using a ledger with cryptographically signed events and the idea is that if you have it's just like a blockchain if you do a transaction between two persons then you sign the transaction with your private key and then you sign the event inside and you sign the event with the hash of the previous event so you get an unforeseeable log and then once we get there then you can imagine that it gets possible to prove that up to the limits of breaking RSA or the usual limits of cryptography it gets possible to prove that somebody did something and not only to prove by you could have done whatever you wanted in the back no, that's sign so there are already a lot of questions I wonder if you have more questions I guess there are other truths maybe we can leave room for the other truths and maybe I don't know if you have freezing questions otherwise we can talk a bit after one question did you provide this draft implementation of the business model as a package for other studios or is it totally internal? currently it's totally internal because it is ugly honestly I'm not this is something I wrote at the beginning of Capital Match and we all learn that it could be really improved and one way to improve it would be to open source it so we are further to use some open source projects here and there which are very small scale and we have started to contribute here and there but we are pretty shy and we have that impossible syndrome complex we are just bored so that could be something interesting on the other hand the core what I've shown you is really it's just adaptive but it's very close to what we have in production so the core is very simple I don't know if it's more bad than the actual library so I don't know if it's really bad what would be interesting would be the store once we get something that's more interesting but they are already even stored out there not a couple of them thank you very much for your attention and apologies for the time