 All right, let's get started. I think I'm up against the guy from GM. So yeah, he's a hard act to follow there. But anyways, welcome to this talk on the end of the Data Ice Age. Again, my name is Hugh McKee. I'm a developer advocate at LightBend. And I've been working on some things lately that have kind of made me call into question the way we do things. And in particular, in the way we do computing, in relationship to the underlying infrastructure or hardware that we're using, in particular, with the compute and persistence. So I'm a big fan of Blender. Anybody ever hear of Blender? It's Blender.org. It's a really cool, very, very cool tool for doing 3D. And I mean, I'm a dead novice at it, but people do movie quality animations with it and everything. But I like using it for stuff like this. So what I tried to do here was have kind of a picture where the circuit board kind of part here is represents a compute space, and these glowing towers represents data, compute and persistence. And why is the way we write our applications so much influenced by this hardware? And what I've been using is something that is at a higher level of abstraction. And this is what really started to call into question the ways we've been computing. Jonas and his keynote talked about this, building systems with higher levels of abstraction. So I want to give you a closer look at this. But it really caught me thinking is I could be looking at the end of what I call the data ice age. So I'm going to, hopefully by the end of the talk, understand what I mean by data on ice, basically. So I want to start with, for back end applications, one of the most common patterns is the stateless pattern. And it's a tried and true pattern we've been actually using well before we called it stateless. We've been doing this for decades. It's a well-oiled machine. We all know if you've been doing back end systems for any amount of time, this is an extremely common pattern that works very, very well. So I just want to walk through it a little bit from this perspective of what I'm trying to make in this talk. You've got these two layers. You've got the stateless app code layer at the top. And you've got your database layer at the bottom. And the nice thing about stateless apps is that you can scale them out horizontally at the compute space. You can add more instances of the application to give you more compute power. Databases would scale in whatever way database can scale, which is the challenge sometimes for databases. But the way things work is that a stateless app gets some kind of request to perform some kind of an operation. So what it does is it retrieves state information from the database and pulls it into memory so that it can compute on it. So I'm just saying here we've got a copy of some state and I put a little v6 on it for optimistic locking. So it's got a version six of this data on it. So that's fine. So another request comes in to another one of the instances of the stateless app to retrieve some other data, some other state. And this is version three. So this other state's pulled up. And so the stateless app on the left can start computing on that data. But at the same time or around the same time, while the app on the left, the center one gets a request for the same data. So it pulls up a copy of that data into memory and now we have two instances of the data going on here. So say the first instance of this version three data makes some kind of a change and it updates the data and it, you know, this again, this accrued representation optimistic locking but it changes the version. So this is no problem because if the center instance of the app tries to change that data it's going to get rejected because, you know, that's how the locking works. But the situation here is the actual behavior of this application is that the compute layer is basically kind of borrowing the data, getting data from the database, pulling it into memory. And as soon as it pulls it pulled it into memory we've got to use these like optimistic or pessimistic locking tricks to coordinate data changes because we have kind of two layers of the data going on here. This is just the way we do things. So there is these two layers but this is where it got me thinking when I started to do things a little bit differently is that I started to think, well, you know, if I really, I think if we really had our choice we wouldn't have this two layers of complexity. What we really want is the data to be in memory all the time. Now I'm still not saying that we write this code to do it but as programmers it would be nice if there was some kind of abstraction that just gave us the data that was always kind of in this hot state. By hot state I mean the data is in memory ready to be computed on versus cold state in the database. And this is where the data on ice comes in. Data in the database is frozen and it can't change in the database. It has to be pulled out of the database pulled into a hot state, computed on and then those changes have to be pushed down back to the database. So why do we keep doing these? It's like we're in the third decade of the 21st century and I'm old, I've been programming. I won't tell you guys how long I've been programming but it's way longer than you think. And this pattern has been around for decades, many, many decades. Like I say it works but it's like why aren't we challenging this more? And this is what's starting to happen that we are starting to challenge this notion of the kind of the status quo of how we do computing and it's with these new abstraction layers. And this is why I really like this. I saw this diagram, it was a tutorial on YouTube that this, I mean if you do Blender there's thousands and thousands of tutorials on Blender on YouTube. And I saw this guy that did a tutorial kind of with this picture and I thought this was perfect because all these little spheres and things wired together is kind of what I had in my head is a visualization of how data looks like when it's in this hot state abstraction layer. So that's why I kind of use this picture. So I wanna show you an example of, and I'm gonna show you some code as well but I wanna show an example of how this works in kind of a higher abstraction level way. So I'm gonna start with the concept of an entity. An entity you can think of as an object. It's like say the contents of a shopping cart or the state of an IOT device or whatever data it happens to be but there's this entity that lives in memory, okay? This entity has an API in it. So it's, you can define the, you're exposing how the outside world can interact with this entity like create it, change it, remove it, whatever kinds of operations this particular entity does. And then within the entity, it's composed of not only its code, the business logic but it's also composed of its state. So every instance of state like every single shopping cart would have an instance of an entity. Every single IOT device would have an instance of an entity. So if you have millions of IOT devices you have millions of these instances of these entities where each instance has within it its own unique state. So commands come in through the API to the entity. It executes the business logic that performs some kind of state change operations and those state change operations are then given back to the abstraction layer in some kind of abstract say state store for example. So let me show you what this looks like in some code. Let's see. This is Java and what this is is the code that implements an entity and you can see that there's these methods that take in commands. You can see the command all already over here on the right and this is the definition of the API for this entity. Now what's happening here though is this entity is running within the abstraction layer. So when a command comes in, notice the other parameter that's being passed to like this create order item method. What's being passed in is the state. So where the state comes from is not our concern as developers. We don't have a connection to a database. We don't implement transactions, those types of things. What we simply do is we are given a command. They say some external client or whatever sent in a command. The serialization, deserialization is all being handled by the abstraction layer. The command comes into my code that I wrote and what's passed with it is the state. Now again, where the state's stored, is it hot, is it cold? That's not what I don't worry about that as a developer. Remember Jonas had that picture with all the knobs? There's a lot of knobs that have been removed here. All the controls, it's like no database, all those types of things. The implementation is, so the command comes in and what happens is that I wrote this kind of handle method. That's just my own style of how I did this. But then I wrote the command. I did this. But then I wrote my own update state method. But then you can see there's return effects.updateState and then reply. So what's happening here is this is where the, my business logic is running in my update state method. I'm passing in the command in the state. The business logic, in this case, I'll show you this real simple. It's just building an object. Taking the command and building an object is just doing a create. But the thing I wanted to point out here is that the abstraction layer passed me the state and the command. I run my business logic and then I pass the altered state back to the abstraction layer. And that's the effects.updateState is where I'm passing the altered state back to the abstraction layer. That's it. So there's a lot of complexity that's removed here. And this extends into things like event sourcing. So that was the first one I showed you was kind of like a crud-like type of you bring in the state, alter the state, save the state. In this case, you have an entity who gets a command, gets a state, but what's, it's emitting is events, not emitting the complete altered state. So it's a, you're implementing event sourcing. And it's the same kind of approach here. And I'll show you an event sourcing example. It's almost the same as the, what's called a value entity or a crud-like entity. So you can see here, this is the APIs again where there's like an add item, change item, remove item. You know, we're just loading up things in a shopping cart. Again, this is my own style. Some people, a lot of people don't seem to like it, but I'm just kind of experimenting, but I have a reject. I take the state and command. That's just validation. I'm just doing a validation and it returns an optional. If the, if the command and the state are okay, it returns it, you know, it doesn't return anything. So it'll do or else get. And then I go into this handle method. And you can see in the handle method, I'm doing a return effects emit event. So instead of updating state, I'm just emitting an event. So this is it. The state was passed in. Now this, in this case, the state's more interesting because to recover the state with an event source and with event sourcing, the state has to be rehydrated. You have to replay from a snapshot and all the events after the snapshot to get back to state. If you look at some other event sourcing systems that in some cases that's code that you have to write or you and I have to write. We have to write the code to recover the snapshot. Then we write the code that recovers all the events from the last snapshot, run those events to update the state. A lot of border plate. In this case, it's all being done by the abstraction layer. The state's passed to me. And this is the current state of my shopping cart in this particular example. And so in this case, I'm just doing emitting an event that's just an event for adding an item to a shopping cart. So this is really cool because now it's making, I think for developers that are new to event sourcing, it's like a really low friction way to get developers to do event sourcing, which like what Mary was talking about earlier as well, we're both kind of talking about is like to build robust systems today, you're looking at asynchronous messaging where events are being passed around asynchronously between components, loosely coupled components. And this event sourcing is kind of the fundamental piece of it at the code level to do these kinds of things. And it keeps going. So there's in this abstraction layer that I'm showing you, there's three, they call them components. The one component is called an entity and there's entities of different types. There's an event source entity, there's a key value entity, and there's also a, it's called a replicated entity. So, but they're basically entities and you can think of them as like objects that have the business logic in the state within them. The next component is views. And this is CQRS, Command Query Responsibility Segregation. So you can see that in what this abstraction has given me is my entity is emitting events that are getting written into some kind of event journal, and then you can create views which are plugged in to consume those events from the event journal to update the views based on the new event data that's being passed in. It's an eventually consistent type of approach which is classic event sourcing and CQRS, but it's quite easy to do as well. I'll show you another example for doing that. I'll show you, there's a lot of views in this app. And this apps up in GitHub, I'll give you the QR code that you can use to get it if you're interested. But I'm just gonna show you a view of how it's first defined. And here what this is using is that services like entities, like views, are initially defined using Google Protobus. Now we're just getting ready to release for Java using Spring annotations to do this, but for right now it's Google Protobus. But you can see the nice thing about Google Protobus is you define services. And this is part of just the Google Protobus definition. I'm defining a service which happens to be a view. And in that service I'm defining the query that this view supports. Also in this service what I'm defining are the methods that are handling the transformation of events coming in for the event journal to transform the data to be put into rows into the view. So you can see here in the Protobus I'm saying what's the source of my entities? What's the shopping cart entity? And I just call this shopping cart. And what's the destination of the data? Well it's going out to this view table called customer cards by customer by date. Simple as that. What happens here is that this Protobus is used to generate like a template of code depending on what language you're using to get you started to do the implementation. But the main thing I wanted to show you was that it's quite easy to implement the second component type which is these views. So that's the CQS example. Now I'm gonna be introducing the third component coming up but you can see that I've got an entity here who is receiving commands from some HTTP client or a GRPC client. But the entity doesn't care who's sending it commands so a new component could be, the third component is called an action. And an action is a stateful serverless function. So there's all kinds of different things you can do with actions. One kind of thing that you can do with actions is it can be set up to consume messages coming in from a message bus. So right now it's Kafka or Google PubSub but it's just as easy to plug in other things like Mary and I've been talking about champing and getting Pulsar integrated in with this abstraction layer. But what's happening is the action is receiving messages as we're arriving in the message bus. They're performing some kind of transformation to create a command to send to some kind of a downstream entity. And this is where things get really exciting because this is how you start to build out event-driven types of systems using the abstraction layer where say I've got this shopping cart entity and its job is just to build shopping carts. And all it's doing is it's submitting events that are going into an event journal and I'm wiring up things to consume those events. So I've got a couple of views that are consuming events from that event journal. But I've also got an action that's consuming those events from an event journal. And this action is wired in to send commands to what I call an order entity. So what's happening here is when a shopping cart's checked out, the action's watching for checkout events. And when it sees a checkout event, it goes, aha, I need to create a command to create an order. So now you just strung together, loosely coupled, a couple of microservices basically. This shopping cart microservice, tight, small, loosely coupled microservice and this order microservice. Loosely coupled, they don't know about each other. Shopping cart could be running along, order could be down, and shopping cart doesn't care. It can keep creating orders. And when order comes back up, it'll pick up where it left off and start consuming the events to try and catch up with the shopping cart. So this is where the fun starts. And again, there's that picture with all the little spheres and everything that I have, I think it's next, this. This is why I like this picture because I think of all these little spheres as entities. Instances of entities, right? And they're all wired together. They're emitting events and they're that are getting fed into other entities and you're building systems that are wired together that way. So here's a full working example of what I call an order entity, our order processing demo in Java built using this tool. The circles again represent entities, but you can think of them as small, tight microservices. The arrows represent journals of events and the arrowheads I think of as the actions. So you can see that the shopping cart I showed you in the prior slide, there's an arrow from the shopping cart to order. And that arrowhead is that action that's taking the events from shopping cart out of its event journal and firing off commands to go to order. Well, the same thing happens when an order gets created, it fires off events that are being picked up as commands to other entities. So this system was a lot of fun to implement the design and implement. So what's happening here, like I said, shopping cart emits a checkout event, order picks that up as a command and the order gets created. When an order gets created, it emits an event that actually triggers to other entities to get created, a shipping order. So the difference between an order and a shipping order is that order is kind of interested in the full life cycle of an order, whereas shipping order is interested in getting stock for the order. That's the difference here. And then when the shipping order gets created, it creates a bunch of order skew items and when order skew items get created, it starts looking for stock. So there's kind of this event conversation that's going on back and forth, event driven that's going on back and forth between order skew item and stock skew item to allocate stock to an order. Non-transactionally, kind of a mutually agreeable series of state changes that are occurring between these two entities until the system comes to a consistent state where say stock is consumed and it's consumed by particular orders. So I want to show you this back and forth between the order skew item and the stock skew item in a little bit more detail. So I've kind of wired it up here with actions. So the arrow on the far right, I'm sorry, the left, I'm kind of pointing at where the action is right now. So that action is getting events from the shopping order entity. And what's happening is that order skew items are getting created. Now order skew items are, if you have an order that says that you have two order items in it, order skew one quantity of two, skew two quantity of three, there'll be five order skew items created, one for each physical unit of stock that needs to be allocated to the order. And the reason for this is it's driven by the realities of at least once delivery, item potency and things like that. I wish I had more time to get into it. If you're interested, grab me out in the hall and I'd love to talk to you about it. But in any case, where order skew item gets created, it emits an event and action picks up that event, sends a command, does a query against the view on the bottom right. And the view says, give me any available stock for that skew item. If there is no stock available, that action is gonna send a command back to the order skew item and say, hey, put yourself into a back order state. If there is available stock from the query, what the action will do is send a command to that stock skew item and say, hey, I'd like to join or I want to allocate you as a unit of stock to this specific order. The order skew, the stock skew item is the one that makes the final decision. Am I still available or not? Because the view could be behind. It could be, you know, it's eventual consistency. So the view says the stock was available, but just a moment ago, some other order grabbed it. Something like that, that can happen. So that has to be taken into consideration. The code to do this isn't really that complicated. In any case, the stock skew item's gonna admit an event to either say, yep, I've joined or no, I'm no longer available. That event gets picked up by the action back over on the left, which sends a command back to the order skew item to updated state. So like I said, what's happening here is stock is being allocated to the order without any transactions. And because all these messages have to be item potent. And the situation is like a counter, like a quantity, is notoriously difficult to do as an item potent type of an operation. Again, I wish I had more time to get into it. Let me jump ahead. So it just, when the stock gets allocated, the stock's order skew item admits events that are picked up by the shipping order. The shipping order sees when all the stock's getting allocated. When it gets stock allocated, it admits an events that goes back to the order to say, hey, order, we got stock for you. It's ready to be delivered, that type of thing. So it's a working application. This one I'm gonna have to jump ahead because we're running out of time. This is another sample application. It's in GitHub as well. The slides where I got the QR code for the slides so you can get that, but it's the same kind of thing. What this is doing is taking large volumes of transactions that are being aggregated into merchant payments. Really interesting application. I wish I had more time to talk to you about it, but I'm gonna have to jump ahead for the sake of time here. So what this means, what this abstraction layer is providing here, and this is kind of a new breed of abstraction layer. I'm kind of calling it a frictionless abstraction because it's not only serverless, it's database-less, it's broker-less, it's Kubernetes-less, it's ops-less, it's services-less. What I mean by services is that, and Jonas was talking about drowning and complexity, that dealing with all these things can take a huge amount of time. And to be honest, I could have stayed for KubeCon to go to this after today. And it's like, I don't want to because I don't want to deal with things like Kubernetes anymore. It's too low-level. Yeah, I spent so much time with Aka Cluster, I would write code for the Aka Cluster, but then I would spend a huge amount of time trying to get the Aka Cluster running on Kubernetes with this particular Postgres service on whatever cloud vendor I was on, all these different things. So it was like the proportion of my time that spent actually writing features versus doing everything else, it was like 80-20. 80% of my time spent on stuff, and 20% is on real features that matter to the customer. And this is what's going away with these abstraction layers. So this is why I think we're kind of looking at things like the end of what I call the data ice age where your data from our concept, the data is always hot. But it's hard because I talk to people, the reaction is, well, I need the control of all these things, and I'd love this quote from Arthur C. Clarke, any sufficiently advanced technology is indistinguishable from magic. And it's like, we always run into this. When something new comes along, and I've been guilty, but there's so many times that I know now that when my instinct says, oh, I don't want to look at that as too magic or something, I know my instincts is 180 degrees off. And we're in the past, I would ignore something and I would regret it, something new, some new technology. Now I'm fighting that urge to ignore things and trying to be much more open-minded to remove the magic. The magic, it's only magic to me because I don't understand it. It's only magic to you because you don't understand it. But once you understand it, then the magic goes away. So I've been talking about it, no big surprise. I've got a Kalex t-shirt on, Kalex here. Jonas was talking about Kalex, but it's ruined me for other things. It's a really interesting programming model. It's really changed the way I look at things. It's the extreme simplification of how you do build systems. The other thing though that's been really interesting is that it's kind of freed me up to really kind of dive into thinking about event-driven types of systems and having the freedom to spend most of my intellectual capacity on that and not on how do I do Kubernetes? How do I do this YAML file? How do I integrate with this database service? On and on and on and on. It's very liberating. So the GitHub repo for the order processing is the QR code on the left and the slides for this presentation and the QR code for the right. I think we're out of time. I really appreciate you being here instead of avoiding the GM one. Do you have a question? Two minutes, any questions? Okay, well, I'll be around. If you guys have any questions about this or Kalex or whatever, I'd love to talk to you. Thank you very much.