 Are we live? All right. Good morning, everyone. My name is Doug Mutart. I'm the chief architect for the Big Data Infrastructure and Engineering Organization at General Motors. You probably figured this out by now, but GM is headquartered right here in Detroit, Michigan at the Rensen, just a block over. I work with 155,000 other people that have the same passion that I do for design, engineering, technology. And we have this shared vision of zero crashes, zero emissions, and zero congestion. My contribution to that is really working on our connected electric and autonomous vehicle programs to really give them the enabling technology to see this vision come to life. It's actually, there's some significance of having the conference here today. I was sitting in this building 22 years ago, almost to the day when Scott McNeely said something that really would set the stage for a lot of what I'm going to talk about today. Back in those days, we talked a lot about genie technology, Corba, Java Spaces. Today, we're going to talk about actor systems and reactive streams. The thing that still amazes me is that we're still talking about Java on wheels, I mean, essentially. So I just thought that it was a really interesting tidbit that at Convergence 2000, we were having this discussion about Java on wheels. Everywhere I work at GM now, I'm finding applications for reactive systems that are responsive, resilient, elastic, message-driven. Every team I talk to is using event-driven architectures. We're using high-speed vehicle telemetry to build customer, dealer, partner experiences. We're using industrial IoT in the assembly and manufacturing plants to do device-level analytics. Look at how do we prevent machinery from breaking down. We're building digital twins to configure, command, control the vehicles themselves. They've got tons of different electronics in there that we can manage through a digital twin. We're building fast data pipelines for that vehicle to be able to interact with its environment, something like the smart grid. And we're even using it to really entertain motorsports enthusiasts, really build that brand loyalty around our products. And we do that in motorsports as well. So just to give you a little bit of my definition of what an actor is. An actor, to me, it's this fundamental primitive of concurrent computing. Actors receive messages. They react to those messages. They may update their internal state. They might log those messages to some sort of event store. They'll probably spawn other actors. They might send additional messages downstream to other actors. With that sort of primitive in mind, you can see where the actor's purpose is to process messages. And so if you're building an event-driven architecture that needs to be low-latency, fault-tolerant, and done at scale, an actor system is a really good starting point for what you're building. They're also, working with actors is great for developers as well. I know when I used to do development, I didn't want to have to worry about all the air logic, thread safety, half the errors I got when I went to compile my code was around not casting things to the right type. So those are all their huge advantages of working with the actor model. And as an architect, when you're working across business domains, an actor system provides a really great bounded context that you can use to reason about with other people, talk to other people. And so it's this natural way to talk about what could be very complex systems. An automobile is really just a huge complex mechatronic system. We've got actuators, sensors. We've got MCUs, microcontroller units all over the vehicle. They're all passing signals to one another over different communication buses. We have a low-speed bus, a high-speed bus. And it's really helpful if you start modeling the software systems you're building around that domain. It becomes this natural way to create a digital twin of the thing that you have in the real world. It makes for a very intuitive API. If you want to do a door unlock, talk to the door actor. If you want to start the ignition, talk to the propulsion actor. So we've got this kind of natural fit of an actor model to the thing we're building, this mechatronic system in the real world. It's also super handy to have this as a digital twin in the back office that really provides that shadow state of what might be running in the embedded system. So you can have a copy, and if you get really ambitious, you can have a copy of the same code that you're running on the vehicle, running in the back office. And that enables all kinds of opportunities to do emulation, simulation. You can practice doing a firmware update. Colin mentioned some of those use cases you see. So it's really just, to me, the biggest advantage of working with the actor model in automotive is this notion of a domain-driven design that everyone can latch onto, engineers, software developers, business folks. Jonas actually mentioned it, too. You're climbing that abstraction ladder, right? The better your abstraction of your problem domain, the easier it is to work with these problems. So this notion, though, of an actor that manages the behavior and state for itself is really powerful. I don't know if any of you are familiar. If you work with the Aka Toolkit, there's a sample application that everybody looks at called the Dining Hackers application. And it's really just an example of a finite state machine built with actors that kind of message one another and change their state. And so if you start looking at something a little bit more complex, like a mechatronic system, like a vehicle, you can build a state model as well. And it can be arbitrarily complex. What happens when your charging is interrupted for your EV? What happens when you've got super crews going, you're level-to-autonomous going down the highway, and somehow that disengages? So you can build these state machines, basically, that handle all these different events. And you really start to build, you're starting to define almost the day in the life of a drive or a driver, if you want to think of it that way. But what happens really when, instead of just one drive, you start thinking about a trip? Instead of just one trip, maybe a journey. Instead of one journey, tens, hundreds, thousands of journeys, you really start to build this day in the life of a driver that you start to understand that's way more important than just the software that you wrote. Now what happens if you start infusing that state model that you've built with intelligence from other sources? It turns out there's some prior art on this. The military intelligence analysis community had this notion of something called activity-based intelligence. And so what they're looking at is these diverse data sources, data streams, if you want to think of them that way, coming in. And they're trying to look for patterns, or more importantly, they're trying to look for changes to patterns. And they use that to really create decision advantage for their war fighters. And we said, well, hey, wonder if we did the same thing to create value for our customers. So we started to look at all of these data streams coming in as a way to look at patterns of life. You've got geo-intelligence. We've got the latitude, longitude, speed, direction of the vehicles. We can draw polygons to paint various geofences around points of interest. We've got human intelligence. We know driver preferences. We know passenger preferences, their settings. We've got human intelligence. I just mentioned that. Signal intelligence measurement stuff. We know weather, road conditions. Our LiDAR can even pick up, is there a pothole in this section of the roadway? We use open source intelligence. You can do social media mining. You can look at government data sets. Or we've got our own fleet of vehicles doing millions of drives every day. We can use that data that we capture. And then we've got the signal intelligence, the actual vehicle telemetry coming in from the hundreds of sensors that we have on the car. And all these sources really can be, you can use reactive streams to kind of broadcast them, aggregate and filter these events coming in. You can train AI models. You can survey AI models to make predictions and inferences about this data. And then you kind of merge the streams back together for action. So this is sort of that notion of using an actor as sort of an autonomous agent who's looking at these patterns, making decisions, and then taking action. So it's very powerful when you start to think about using an actor system in this context to enable activity-based intelligence. So one of the things that I think I've heard a couple times at the conference today already is what's the developer experience like when you're building these kinds of systems, right? So it's really important. I mean, you have to pick, you've got that whole menu of choices to make, right? We use the AkaToolkit quite extensively. But there's so many different features and capabilities. We've taken a product management approach where we start to build what we call a product out of that toolkit that we can then deliver to our data science community, our developer community. Some folks want an SDK. So we'll take specific features, capabilities, and we'll give them an SDK that makes it consistent, conformant to what we want. And they can use that SDK to build these actor systems and streams. But some folks, maybe an engineer that's doing some analysis of one of our electrical control units, he just wants the data, right? He's like, hey, just give me the raw protobuf coming off the vehicle bus, rapid in a cloud event so I know what the metadata looks like. So for them, we just give them a simplified API. And we've gotten pretty creative with the tools that are already available, right? We can use something like similar to Spring Initializer where we can just basically generate an actor system that they can use. They can use configuration files to say, hey, I want this Kafka topic, this Pulsar topic to be streamed to this Azure Event Hubs topic. And I want to transform it from Avro to JSON. And if we don't have that, maybe they can build their own plugin that does that transformation, right? So that's this notion of really just tailoring the whole developer experience around the preferred programming paradigm of the people that are using these systems. Then you've got your runtime model, right? So we've talked a little bit about this in other talks today. We use the Akatool kit, which runs on the JVM. That's very versatile. We're able to apply that on embedded systems. We can do it with virtualized systems. Works really well in containerized environments. We often run it in Kubernetes. These all actually really help because they provide these additional layers of elasticity and resiliency that you want in a system like this. I would say one of our challenges and one of the shortcomings right now is there hasn't been enough work done on running JVMs in safety-critical environments, Colin mentioned, safety-critical systems a little bit, microcontainers, micro-VMs like Firecracker and Cata. There's just not enough work going on really on what the next generation of this runtime model really needs to look like. But once you've got a runtime model to work with, and this works perfectly well, you've got to figure out, well, how are you deploying this? And so we're seeing already just a wide array of different places that we can deploy these actor systems and these reactive streams. We run it in our own private cloud that we've built, usually on Kubernetes, sometimes in virtual machines. We run it in Azure Cloud. For some of our use cases, I'm going to talk a little bit more about motorsports and NASCAR. You may want to run it on hyper-converged infrastructure. You might want to run it in the pit box at Trackside. You might want to run it on sort of AT&T's compute in a 5G network node. So you've really got this whole sort of different spectrum of deployment models to work with. Some of the things, though, that are solvable really are, you've got to really start understanding like geo-location-based routing and DNS. What you build needs to work really well with the service discovery models that are in place. You need to be able to locate sort of the actor system that you're trying to talk to. And I would say one of the challenges, Stell, is this control plane. And Colin talked about that a little bit, too. Maybe WebAssembly is part of the answer, right? But you want to have the same pipeline, basically, that's able to deploy these actor systems into these different deployment models. And then when you get into these different deployment models, things become distributed very quickly. You've got things running embedded. You've got things running on the edge. You've got actor systems and streams running in the cloud. You really need to start working out, hey, where can I back pressure? Where can I not back pressure? Where can I buffer? If you're doing 10 million messages per second, you're not going to be able to buffer for very long. So every hop, sort of as you go through these different network topologies, some of the hops are fast, very low latency, sub milliseconds even, in some cases. And some are very slow. Maybe going from cloud to data center is 25 millisecond latency, right? So you've really got to start working this out for yourself and figure out, OK, where can I buffer? Where can I back pressure? And where can I put brokers, right? So there was a presentation earlier today on NATS. That was certainly something you could consider. We use Pulsar and Kafka for the brokers between, say, data center and cloud. You might use Azure Event Hubs if you're running in Azure Cloud. But you really do need to figure out, OK, what's the role of a broker? And in some cases, we've been forced to kind of go broker less, right? As the message volumes increase, you start to see a lot of IO problems even with brokers like Pulsar. And so you need to start thinking about, hey, maybe we switched to a protocol like GRPC and we go broker less. So once you start to distribute your actor systems like this and distribute your streams, it's a whole different ballgame. So speaking of ballgames, so having done a lot of this in the connected vehicle space, the electric vehicle space, got a phone call probably a little over a year ago saying, hey, Muteart, can you and the team come down to Charlotte, North Carolina, and do moneyball for motorsports, right? And motorsports is a big money business. A NASCAR Cup Series sponsor might pay tens of millions of dollars to have their logo to sponsor a car for a 38 race season. Companies like GM, it's really important because it builds brand loyalty, right? If we win the NASCAR Cup Series, if it's a Chevrolet that wins, that means a lot to our enthusiasts. So we were asked to kind of go and say, hey, how can we apply this notion of actor systems, reactive streams, activity-based intelligence to help our Chevy racing teams become more competitive? And it really requires that millisecond mindset, right? I mean, we're talking a processing stage of one of these streams might be tens of microseconds, right? And so we found that this is super helpful because we have, again, we've got such a diversity of different kinds of data coming in. We've got people all around the track taking photographs. We've got spotters. We've got live video of every pit stop. All that needs to be processed in real time, right? We need to do text analytics on all of the different radio communications between the driver and the pit crew and the spotters and what the NASCAR race officials are talking about on the radio. The race strategy, we have to use AI to really augment the decisions made by the driver and the crew chief in real time, right? We have to make decisions like, do we pit or do we not pit now? Do we take two tires? Do we take four tires? Do we top off our fuel? Do we make any adjustments to the chassis? And we're looking at when a car gets hit, what's the size of the damage on the quarter panel? And we have to use photogrammetry to do that. So we found that using reactive streams, we could actually just process an immense amount of this data in real time, serve up AI models, make decisions, and really just generate that competitive advantage. And you can understand some of the challenges here. You've got 40 cars racing around a track at almost 200 miles an hour. You're trying to do a haversine distance calculation between all 40 cars to see how close they are to one another. You're trying to make decisions in real time based on the traffic density, how much dirty air is there. Because drag and all that really matters. I mean, just an example would be a 1% reduction in drag, like at a track like Daytona, could be 10 positions at the end of the race for you. So it's really a really cool place to start to apply some of this reactive stream, reactive system architecture to a very interesting problem domain. And obviously, the digital twin also comes into play. You've only got 40 cars. So you can pretty much have a ghost car, a digital twin of every vehicle running for the entire race. And you can go back three years and replay these races as many times as you want to do training and simulation. So it's a really powerful application of this. So Jonas had a quote very similar to this earlier in the opening keynote. This one kind of is a little older than that. But boss Kettering was really the wizard of General Motors. He's the one that invented the electric starter and 175 other different things in the automotive business. But again, the concept was, and this is really what I'm getting at with today's talk, we really need to start rethinking about the primitives that we're using to construct these systems. And what I mean by that is, first we have to kind of figure out what's this pattern that's reoccurring over and over and over again. And every example that I've given today really is, hey, you're processing an unbounded stream of events. It's nice to have functional operators, merge, broadcast, concatenate. They have to interact with this actor system, this digital twin that you've built that really encapsulates all the behavior of the system. You may want to write all those events out to a log, a journal of some sort, so that you can replay them later, so that you can recover from failures faster. And then you typically put the results of those decisions out in some sort of messaging system. It could be a grid or service mesh of some sort. It might be a message broker like Pulsar or Kafka. But this is sort of the pattern that kind of comes up over and over again. And it's very easy to insert AI and machine learning into this sort of virtuous cycle. I purposely do the arrow that way to make it look a little bit like a smile. Because every time I see a new problem, I immediately kind of think about this pattern and how we can apply it. But the real shift in thinking is to stop thinking about all these different web service frameworks and different logging frameworks and figuring out service registries and containers and container orchestration. At this point, data is an afterthought. It becomes an accident of the architecture. The last thing on the list is sort of your data. And what we found is we're really starting to take this data-centric view of the world, where everything's an actor, an event, a stream, a function. We may or may not log those events somewhere to an event store. But those really become the primitives that you start working with. And that's been a real game changer for us. But where it goes from here, we've seen a little server less definitely. You get that with the cloud. It's a little harder to do in your own data center. But that's kind of evolved into this notion of database lists, where you may have an event store where you're putting stuff, but you as a programmer don't worry about it. You let the infrastructure take care of that. We're starting to think really heavily about how do we go broker-less. A broker is a bottleneck at the end of the day. It's going to be I.O. that might become a bottleneck. But where we really want to get to is data scientists, developers, they're scarce resources. If you can build a developer experience and maybe even get beyond the developer experience to a point where it's developer-less is what I called it here, then you're really talking something different. And I guess the way I see that, if I was a futurist, is that you've got these actors who are coming up with their own new behaviors on the fly. They're always learning. They're always adapting. It's almost like a neural network of actors. And this is really letting machines do the work. And that's really where ultimately we have to go. A lot of the systems I showed you, it takes teams of architects and developers and testers to get those systems right. I'd really like to get to a point where these primitives, these compute primitives that we have, are able to do that for themselves and just take advantage of the infrastructure that we give them. So that's really basically the big takeaway for me is switching your mindset, pivot from this code-centric view to the world to this data-centric set of primitives. And hopefully we'll get to that point where it's just kind of doing its own thing. Thanks. Yeah, I kind of like that. And there's some, like I said, there is some prior art on some of that, right? Like Netflix has tried to integrate AI into reactive streams. There's libraries out there like Deep Java library that lets you take TensorFlow models or really anything from a Model Zoo, have a plug-in sort of AI. And it's not a big leap to just say, OK, I've got an actor that serves a model and I think the harder thing, and I've been actually talking to some of our R&D people about is, how do you do the learning that way too, right? Because if it can be self-taught and you can serve the model, that's even better, right? But the actor model is a great fit for AI really, awesome. But it's been great to see, though, all the speakers today kind of hitting some of the same points, right? Maybe from different angles. But I think we're all seeing sort of the same trends starting to emerge and some of the same challenges that need to still need to be overcome.