 Alright, hello everyone and welcome to this session how to build serverless event driven microservices systems with Kalex. So in this session, really what I'm going to be showing you is the how to build like the title says the event driven parts are really going to be focused a lot on event driven design. And I think what's going to be interesting is some of the kind of guided design driven approach to this, this process where, because of some of the ways things work in an event driven system, and also with Kalex, and just the realities of doing distributed systems, things like that. There's, I think there'll be some surprising outcomes of what it actually takes to do an event driven design like this. I know it's for me it's been an amazing, about a year now, doing this with Kalex, and finally getting into something where I can stick my teeth into building event driven systems I've been talking about and then doing it to some degree for for some years now. I've been talking about event driven conferences and blogs and webinars and things like that. But now, because of Kalex, it's gotten so much easier to do it. It's quite amazing. So let's get going. Like I said, we're going to dive pretty deeply into event driven design in this session. So it would help if you have some familiarity with Kalex I'm not going to go into a lot of the details that I often do in many of the other videos and content that I produced around Kalex in this video. So, if there's something that I'm not covering here. There's plenty of other resources to find I've done a bunch of videos other people have done videos on Kalex, and you should be able to find. And of course, as a documentation so you should be able to find the some of the basics but here this is a bit more advanced and we're going to go into the design. Now, I've never feared if you're not that familiar with Kalex yet. This is really about kind of the fundamentals of designing event driven systems. And, and that process I'm not going to get into code or things like that this is really about the conceptual process. So, I'm going to be walking you through the design of this demo application that I've been using for a while now. And I think it's a really good representation I wanted to dive deep into the design of this for a long time, and I've kind of covered it to some degree in other videos that I've done, but this time I'm going to go pretty deep into the flow here. And it was really interesting for me to, to walk through the process and also kind of share with you my experience as I was working through the design of this and I was, and actually what was happening I was unlearning things that I've known for years about building say stateless crud like crud types of systems that a bit building that those types of systems for decades and agree to get into an event driven type of a process and design. It really kind of, I had to kind of let go of some things that I just took as almost kind of intuitive ways of doing things and kind of, you know, rewire my brain to do things in a more event driven way. Hopefully you'll see that it's great fun, your head will hurt I think the first time you do it. But there's some patterns that you'll see here that not only apply to this application but you'll see the same recurring patterns in other event driven applications this is certainly what I'm saying and the other things that I've been doing along these lines and different applications but the processes. There's this kind of nice pattern that's emerging that like I reuse over and over. And all the designs that I've been doing. So, to start out. When I first started working with Calix and I was learning it myself, which wasn't that long ago because Calix is new. I was really doing demos on a shopping cart, you know the classic shopping cart demo. It's interesting. And, you know, so that I was building a service so that's the circle here in this diagram represents a you can think of it they're called entities and Calix. You can also think of it as like a small microservice. It's a self contained service that has an API, you interact with it and it. It persists data to a database that can retrieve data from the database those types of things so it is, you know, you think of it as a, as a tight focused loosely coupled microservice that this really what what this is. So building up the shopping cart, you know, adding the ability to add items to a cart to remove items from the cart to change items in a cart, and to check out the card. And I was kind of playing with the, the programming you know that at the code level, what I was doing, how I would write the code and things like that and I was you know so I was learning. But then I was getting kind of bored with it because like okay, you know shopping cart, my big deal but I started thinking well, it would really be cool to do something more interesting like, what about, okay we got a shopping cart, the shopping cart's been checked out. So what about implementing the process of allocating stock to the shopping cart. So that's where I started to build up. So the first thing I did, I started to add some of that logic to the shopping cart service that I had built. But it started to feel like it was getting heavy weight, you know the design was starting to smell, you know that feeling when you, you're doing something and it's like, I'm doing too much here. I really should be kind of breaking this thing up. So what I did was I introduced a second service that I called order. So the idea was that shopping cart was focused on building shopping carts and once a shopping cart was checked out, then it would emit an event, which would be picked up by the order service and the order service would create an order. And so the big difference here is that, you know, with a typical online shopping system shopping carts could get built and then abandoned and no orders are getting created so you know that data is interesting to analyze but they're not real orders. So they have a kind of a clear delineation between just the code and the services responsible for handing shopping cart and the code that was responsible handling orders. So that checkout event is a significant event. When a checkout event occurs, the shopping cart basically freezes itself. It's no longer possible. The shopping cart service won't allow any more changes to the, to the cart contents, because it's been submitted as an order so when an order gets the shopping cart now the order goes okay, now we've got an order that we need to fulfill. So in the order service I started to add a little bit of like the life cycle into an order service. And what I mean by that here's the, the data structures, the data structure is simple here but here's the data structure of a shopping cart. And you can see the main thing in the shopping cart on the left this is just some Jason is just it's got a card ID a customer ID. And the timestamp for it's checked out. And there's a deleted flag, but sorry about that. And then there's a list of items in the shopping cart so you can see there's a skew one with a quantity of two and a skew two with a quantity of three. And the order data, you know, this is the, say the object that the order is, or the state that the order is, is building. It has a looks very much like the shopping carpet has got some more fields meant mostly a bunch of timestamps that kind of show life cycle like when was it ordered. When was it shipped when was it delivered. Was it returned was it canceled those types of things. And you can see in the order items, which is just a list or an array here and Jason, there's also a new field called shipped as well as the ship timestamp. So, so I had the order, and then I started thinking right well what do we do, how do I write the code. How do I do a doing an event driven type of approach to allocate stock to an order so I started thinking about it and start really implementing it because I was started I was kind of at the early stages of the design. Okay, what's the stock order what does it do and how does the, what's the kind of the eventing interaction between order and stock order. Right away I started running into some interesting things. So here I'm taking things down to a little bit more level of detail so the circles again represent the microservices which are called entities in Calix. So this is a order is an event source entity stock order is an event source entity. And what happens is there's a, there's a, just these a's there's one in the center here and there's one right on the edge on the left. Those are called actions and actions are stateless functions. And these v's are views, these are just courierable views so this is event sourcing and CQS command query responsibility segregation, where the commands are handled by the service entity, and the queries are handled by the views. And in between those is that each entity has an event term. So what happens is that the action is a function that received that checkout message that that a checkout event from the shopping cart off to the left side here. The action then sends a command to create a new order entity. The order entity is created. And when it's created it just simply emits an event like order, order created event, which gets written into the orders of venture, and then that triggers that event is picked up by another action, which the arrow in the center is this action then is responsible for taking that order and getting stock. And this is where things started to get interesting because here in order could have multiple stock items you know skews to that we need stock for. And we need to, and we need to do this. Transactionally, you know we want to allocate stock to the order but we don't want you know we want to make sure that this happens precisely and doesn't corrupt the data or anything like that. So, you know they say that this action function sends a command to the stock order and the stock order does something to allocate stock, which emits, maybe one or more events to the event room. But the challenge is that the problem is it's like we have quantities, like the order has a quantity you know that skew one wants a quantity of two skew two wants a quantity of three. And the challenge here is that when we're doing this, we're trying to, you know, say, consume some stock like I need two of this you know so like say that skew one has, you know, maybe 100 an inventory. We want to reduce that inventory from 198 because we need two items. So we're doing some kind of a decrement on the inventory quantity, but this is not a durable, totally reliable operation. And the reason being that what can happen is that the event of say, decrementing the stock that will trigger the decrementing the stock and getting written to the order the stock order event journal. That could happen but before it acknowledges that back to the action the actions waiting for all this to happen so that it knows that the message was consumed. Say something fails I turned everything right here something fails so the action goes out that last operation failed. What the actions will do is, it's just going to do a retry, whatever the failure was when the failure is the problem is resolved and everything's back online. The action is going to retry that operation. And this is where the problem starts because we retry the operation, we need to make sure that we don't decrement some more stock. Yes, because we, we did decrement some stock, but the action didn't know about it. And now we're going to, because it's got an automatic retry, we're going to decrement some more stock. So we're running into here are the first two of four things that I want to cover here that really kind of guide you and are the realities of doing event driven types of systems. The first one being at least once delivery. What I was talking about here is that this action is, it is fed by Kalex events, and the action then perform some operation, and only one that action is successful in performing that operation that tells tells Kalex right, it's time to get more events. If when there's a failure of this process here and the action doesn't acknowledge or it says there was a failure that tells Kalex okay we still haven't processed this message. So it's going to retry it. So that's what it at least once delivery is that what it means is that every single message will get delivered, but in some cases, some messages will get delivered more than once. And this gets into the next point that I've highlighted here that that the consumer of these messages need to consider item potency, they need to be item potent so that, like in our case, when we do get that same. Hey, we need to allocate some stock message twice that we don't keep allocating allocating stock over and over and over again, until the whole operation is successfully completed. And this really drives things because now it's like okay, I can't do it this way I need to come up with some other way of designing the system to handle this stock allocation to an order. All right, this is getting complicated so I introduced a third service, a third entity called shipping order and in shipping order. The idea was that order is just kind of interested in the full lifecycle of handling orders, but the actual process of allocating stock to an order I was starting to delegate out I wanted to have something that was just focused on allocating stock. The same problem exists with shipping order that I had with order between shipping order to stock order or order to stock order, so I needed to do something else. The solution that I finally came up with and this took some thinking, but, but now that I've see the pattern, like this is one of these patterns that I see a lot is I needed to take things down to another level of granularity. What I mean by that is that what happens here. These arrows represent flows of events between these services and these backwards and forwards flows between some of these. And I'll walk through this in more detail in a moment, but what's happening is the shipping order gets created. And here's an order and here's a shipping order, the shipping order takes the order but you can see that there's an order items list in the shipping order hopefully you can see that it's big enough. And then within each order item, there's a sub list of what I called order skew items, which is the this entity this new entity that I created. So you can see that for an a given order item for whatever the quantity is there's a an associated number of individual uniquely identified each have their own ID order skew items. So the quantity is to so this has to the quantity for the second skew items skew item skew ID to the quantity is three so there's three order skew items. So what's happening is that the shipping order gets created it emits an events is a I got new shipping order, the action gets that there's another action that receives that event from shipping order, and this action is written to take that event and decompose that one big monolithic shipping order down into five individual in this example, order skew items. So five individual order skew items get created. So, what what's happening here is that the the processes the design process is, it was being kind of driven by the realities of at least once delivery and item potency. And I don't want to consume stock incorrectly, and it shows that it has been consumed, but some stocks not actually needed for a given order because we we've double booked it baby basically. So I want to walk through some of these, these four things in a little just a little bit of detail so it. At least once delivery is what I was talking about that this is an example that's very common in other approaches that's called at most once delivery, and at most once delivery is kind of like this you have a function, say, some kind of operation that it does two steps one is that it creates an event say in an event journal, it could be updating a database or you know we're doing some some operation, but then it has another operation where it's second operation where say it's putting something into a topic. This example into a conflict topic. And this is this is a two step operation, and actually ran across somebody, this isn't Klex code, this is from some other, some somebody else. And I got a, I did a screen capture of this snippet of code where they have these two examples and I got red arrows and underlines to kind of show what I'm talking about but you can see in this code. There's a state operation, and then there's a publish event operation, and they're there two separate steps. So what's going on here is that there's this gully, this, this gap between the first operation which persists data to a database, and the second operation which persists something to a message bus. There's a non transactional gap between the two. I grabbed this picture this guy jumping across this little gap. It's got, if it's kind of hard to see but it looks like he's got two very expensive cameras and each hand. And I don't know if he's made it or not. To me it doesn't look like he's going to make it. In any case, the, the idea is maybe he makes it because he's looked pretty tall and he's got long legs, but I don't know if I'd make it. And I think other people would make, but the point is that this gap here is kind of like this valley of doom or this period of vulnerability in this code. And really what's happening is what's being put into production and I think this happens a lot is leaky code that most of the time like well into the high 90, 90%, 99% range. Everything works fine. But every once in a while there's a failure that happens at the very, very worst time, somewhere in between this operation and the second operation. And now you have data corruption that the code obviously wants both of these operations to happen, but it's not an atomic operation. You're kind of going into production with these teeny little leaks that will drive you crazy. I've been in this kind of situation, and it's not a fun place to be at all. So the solution to this to get at least once delivery is that the function, the producer simply produces whatever they're producing. And then there's some other process running on the side or in the background that is consuming the new data as quickly as possible and passing it on to the consumer. So the, the, the, and the this read side processes I'm calling it is usually this is a very Kafka like type of operation, the Kafka is famous for is just reading by offset. And if it fails, when it's restart it's just going to pick up where it left off which means it could be that some messages that were sent will get sent again, which is okay. As long as you've designed you know that's going to happen, and you design the system for it and you can deal with it. Potency also comes in here, because if there's a state changing operation on the consumer side that state changing operation has to be written, where if it when it gets the same message twice, the result is the same as the first time the second time the third time and the end time. It's an important operation. And what I've been doing in the test code that I write for these calyx services this I'm getting in the habit of always writing item potency test setup the service. I'll get an entity into a certain state, and I'll send it a command to make a change of states and I'll make sure that the state change the way I want it. And then I'll send that same command again I'll make sure that the state still looks like what it's supposed to be. So I'm designing specifically and implementing for item potency. This is multi transaction was well this is really important because for for a long time. We've lived in a world where we had this wonderful comfortable database world where we have these lovely atomic transactions, but when you start to build distributed systems in which a lot of us are now with other systems and things like that. You've left the multi transaction world if you have multiple services they're triggering each other to perform perform state changes outside the bounds of a transaction. You need to consider these things if you don't, you're going to have those leaks, those little buckets picking up those little drops of water, every drop is a unhappy customer is unhappy business is unhappy manager is unhappy developer. The fourth thing the fourth thing is that the system is eventually consistent in that the function rights and operation the data is in the journal, it's there to be used. Some time later, this read side process is going to pick up that data and send it to the consumer but it doesn't happen as a single atomic operation. It happens is multiple independent operations, which is, which is the eventual consistency. And this design. This is where things get interesting and I'm going to walk through some of the higher level parts of it and then we're going to focus on the, the eventing between the order screw item in the stocks group. So like I said you know there's a client clients are loading up their own shopping carts that you know I have a shopping cart you have a shopping cart somebody else has a shopping cart each one of us has our own our own instance of a shopping cart. The data of each of our shopping carts is, you know what our shopping carts look like. And when somebody presses the, the, the buy button that triggers an event, you know, like a checkout event. And then checkout event admitted is emitted by the shopping cart, and there's an action that picks it up and sends a command to a downstream service in this case order to say hey order and we need to create a new order. It's created. It emits an event and here I just want to show where the same event could be picked up by multiple consumers. So in this case there's an order item. I'm not really going to go into detail on this because this is just used for like an example of doing queries and manipulating data, but the shipping order picks up the event. When the shipping order picks up the event like I said before, it emits an event and says hey I got this new shipping order it's blown up internally. The order items into the order skew items sub lists that whole object it's passed to the action there's an and that action take kind of decomposes that's the single shipping order into the multiple order skew items. Again, the example to two order items. One quantity of two one quantity of three that means that that action is going to send five send out five commands to five different order skew items to be created. And this starts a kind of a whole cascade of a lot of concurrent operations that are happening all at the same time. When you order skew item, order skew items are created they emit events which then start a process of trying to hunt down available stock for that order skew item. And this is where I want to dive into more detail of how the interaction occurs in an event driven flow between the order skew item and the stocks. If we grok this, then I think this is one of the most important things it's just kind of how how this whole flow works so I'm just going to walk through it step by step. So, we've got the order we've got the shipping order. Remember, again the shipping order there was there's multiple order skew items. We've taken each one of these order skew items that were built into the into the shipping order, but now we created individual entities to each of these order skew items. So, what happens is the action that pick that sends those commands to order skew item, create a new order skew item that produces an event. That event gets picked up. There's some views that pick up the events but that's not the main thing here, the event also gets picked up by another action. The actions are stateless functions are kind of the glue that ties everything together, because actions are the things that can do things like perform a query so in this case that's exactly what this action does. It got a message from the event or Kalex passed a message and invoked the function to say hey I've got this new message for you. And the function gets that message. And the first thing it does is it performs a query against this view, and this view is set up for doing a query against available stock skew items for given stock ID. The result comes back, and there's two possible outcomes either there is available stock for the skew item, or it there's no available stock. If there's no available stock, the action is just going to in turn send a command back to the order skew item and say hey put yourself into a backorder state right now there's no stock available, and I'll get back to this one in a minute. However, if there is stock available, what the action will do, it was a, it will grab one of those available stocks skew items, and it will build a command to send to that stock skew item service. And the service request is the order skew item effectively wants to join with a stock screen. We want to identify here's a specific order skew item. We want a physical unit of stock. So the command comes into the stocks skew item. And there's two possible outcomes here, either the stocks right is still available, or it's not. So it's the request is either accepted or rejected. Now, the reason this can happen again, this is the eventual consistency that between the time we do the query. The command into the stocks, you have there's no locking going on here, and somebody else could have grabbed that stocks you item that this command is going for, and it's no longer available. So, the idea is that the stocks you item is the single point of processing of handling this no locks are required because it's a single writer type of a principle here. The stocks you item takes that command, and if it's still available, it will accept the joint request if it's not available, and we'll reject the joint request. So neither case, it produces an event that gets picked up by a third action, which is taking the event from the stocks you item and sending a command to the order skew item. If the joint was accepted, then the order skew item is updated, you know it's status change to reflect yep, we, we've got a stock skew item for this order square. If it was rejected, what that's going to trigger is that specific order skew item is going to just simply admit an event that's going to trigger the whole process all over again to try and find an available stock skew item. So this is the way the system works. The, the really interesting thing here is that we've got two independent very loosely couple microservices here, order skew item in stocks. They're both making independent non transactional state changes, but because of the design of the sequence of events that you know I designed as a design, you know the designer of the system. I belong to a mutually agreeable set of state changes. Your first, the stock skew item, if it accepts a joint request, it changes its state to say yep. I know exactly what order skew item, I belong to it updates itself with that information. And that stocks right in the midst of an event, which goes back to the order skew item in the order skew item gets that that command is as I've got a available stock skew item, I'm going to change my state to reflect the stack, the state the status that I've been allocated stock. So it, the nice thing about this is that each one of these services are really simple. There's barely any business logic in it. The kind of the, the fun of this is the actual design of the sequence of events that goes on between these two things, which honestly, to be honest with you, it took me some time to work out. Just because it was this, this was one of my first exercises to go in this much detail of doing it but like I said, now that I've gone through this, I've seen the same pattern in other things that I'm doing, and this kind of sequence of events where you come with up with mutually agreeable state changes is getting to be second nature to me now it's it's quite easy. Also the brilliance of this is that this is a dead solid type of implementation, anything can break here. And when it recovers itself feels it just picks up or left off so anything to break, and it just keeps going and of course it scales and all those other and because of the way because I'm doing it kind of a lower level of granularity I'm not doing it at an order level of granularity. This is all happening like at the items the individual stock items that need to be allocated. It's a high degree of concurrency that's happening here so in a real system processing lots of orders. This would be simple database operations were just inserting events into journals those events are being picked up, sent to actions actions make decisions to queries that that type of thing. And then the individual services basically perform very simple, non locking non blocking types of operations, and the system just kind of screens screens along as it's doing this. It takes as well. It's all taking into consideration. At least once delivery means every single message will get delivered. It's built to be solid in that the each of these operations are designed, and I was kind of driven to the design to make the mind impotent. So it can handle when multiple, the same message is delivered more than once. No problem. And, and it's fast and durable. One other thing I want to cover here is that when new stock is added new stocks you item entities are created. And when that happens, those that they just trigger events and say I've got a new stock skew item that event gets written to the journal which gets picked up by this action on the left. So what it does is it does a query against the view that is a view is there to look up back ordered order skew items by skew ID. So if the query finds some some back ordered skew items order skew items, it's going to then send a command to that order skew item and say hey, I've got some stock to join with you so now the flow is reversed. I've been talking about from order skew item to stocks Graham and that flow but the kind of the the same flow, but now in the reverse direction stocks hunting down order skew items instead of orders grams hunting down stock. But, you know, there's a lot of cemetery here in the way this this works. So that's the the behavior here. So the stock skew item, you know, does all this work. It emits it those events. You know we've seen this at a lower level of detail here in the past few minutes. The order skew item when it changes its state. It emits an event that goes back to the shipping order and the shipping orders, updating its state to reflect individual order skew items that have been allocated stock. And what's happening here you can see like this is on the left these are a couple of the order skew items and you can see now is an order skew ID underlying the first underline and then there's a stock skew ID. The second underlying so you can see that these both of these order skew items have been allocated stock. So this order skew items emitted an event with this information which goes back to the shipping order to the shipping order hunts down just digs into the order items list and then the order skew item sub list and hunts down that order skew item and it changes its state in there as well so now that the shipping order sees every change as different units of stock are allocated to the shipping order. And then it can monitor the progress and so for example when all of the items within a given order skew items within a within given order item have been allocated stock. It will emit an event that says I skew one is we have all the stock we need for skew one. The same thing would happen for skew to which means that the shipping orders emitting events which are updating and changing the state of the order item that the order item has ship dates in it as well. So the whole shipping process flows back upstream from stock skew item to order skew item to shipping order back to order to show that the state of the order that we found all the stock that we need that we need for an order is something still not something back over that type of thing. And you can see here. You know all the, the, the ship dates are time stamps are are updated and the, the, the associated order has that shipped you to see it was kind of at the default 1970 time stamp and now it said a, you know, a current time date that we found stock for, for all this order. So that's the flow in the system. Another thing I'm going to point out though is that the shipping order this this concept of lists of like a parent, you know, you can think of the shipping order is kind of a parent to the, the order skew item children. It's kind of a one to many relationship. And this, this pattern of having the parent, have a detailed list of its children is also kind of a recurring theme because this provides item potency as well. When a say stock allocated event is sent to a shipping order for the same order skew item ID, more than once, those are item potent operations if I was just like changing a counter that would be impossible to make it as a or not impossible to be more difficult to make it as an item potent operation. So part of the reasons for the sub list is also to handle item potency as data flows back upstream from the order skew items to the stuff that to the shipping order and to the, to the order itself. So, in this adventure of in design, these things are really important, you have to think as you're going through the design you have to think about at least once delivery, your, your services are going to get the same message more than once. How do you build an item potency that will drive your design that I just every time I've gone through design. It's remote is kind of an intellectual exercise how do I design this, how do I unlearn what I've known before and learn something new and redesign my flow into something that takes the realities of at least once delivery which is great because every single message will get versus the alternative which many people use, which scares the heck out of me is that they're using at most once delivery, or some kind of sloppy retry without an impotency that just I think is a nightmare. Here, you just kind of go, it's kind of like go with the flow, you're going to work through the design process, you're going to run into something, and you're going to say, oh man, this is not an item potent operation. And my, I think the voice in your head should be, you need to make it item potent. It's like Luke. You know, go with the force. Yeah, you know that it's the same thing. Luke, go for it. Impotency figured out you redesign the flow so that you're dealing with the reality of least once delivery and your operations are right important, because that will make it so that you things are multi transactional and also you have to consider things like eventual consistency. I showed you a little bit of that as well that the you have to consider that so a lot of the event flows are kind of type flows, because of this, these realities of building a venture in types of systems. And that's it. A little bit hopefully a little bit quicker session here. The QR code is for the, excuse me for the GitHub repo of this demo project that it's a Java version of the demo project. And, you know, the one I use to test and play with and the logins fun to watch when you enter orders and, you know, things like that so feel free to grab that and, and play with it and, you know, look at the code the codes, you for the most part, the actual business logic the code is pretty straightforward the fun part is the design piece of it. So I don't know if we have any questions but thank you very much for your time, and I'll hang around for some questions but otherwise we're