 Hi everybody, welcome back to SuperCloud Five. My name is Dave Vellante. We're here inside the Emerald Lounge, which is Mongo's hang space. We're here all week. We're doing theCUBE today. We're also upstairs. John Furrier is upstairs on the third floor. Outside the press room. We got Palo Alto Studio live with Savannah and Lisa. We got Rob Strecce and Rebecca at Barcelona pumping in content from HPE Discover. So SuperCloud is multi-continent. I'm really excited to have the folks from Toyota here. Kevin O'Dell, director of engineering at Toyota Connected North America, and Isaac Broyles, who's the managing engineer at Toyota Connected North America. Guys, thanks for coming on. Appreciate your time here. Absolutely, thanks for having us. All right, Kevin, let's start with you. So you gave a talk yesterday. We're going to get into that a little bit. But why don't you start by explaining sort of your role a little bit about Connected North America. What is that? Yeah, absolutely. So Toyota Connected North America was established in 2016 and was really an innovation and software development hub for Toyota, really with the mission to drive the global, you know, conducted intelligent vehicle type of opportunity for Toyota. My role specifically is within Toyota Connected, an area we call Drive Link. We're really responsible for safety services, so Safety Connect. These are subscription services available for Toyota and Lexus customers. So like the SOS button in the vehicle, the automatic crash notification where if you get in an accident, one of our safety agents will call into the vehicle. We can track your car if it's stolen. We can hook you up with roadside assistance. All the standard safety features that we want to be able to provide our customers. And your role, you're director of engineering. So what does that entail? You just sort of... Yeah, yeah, yeah. So I lead the area. So reporting to me, I've got products, I've got engineering, and I also have what we call our call center, our call operations folks. So it's really all of us, all three areas, holistically. My role there is to make sure that we're, you know, heading down the right path. Everybody knows where we're going from a product perspective, prioritization, all that good stuff. But obviously I come from a very technical engineering background. So I get to get in the weeds sometimes with these guys, make sure we're making the right architectural direction or decisions. And that's really where I... And Isaac, I presume you're in the weeds. Yeah, yeah, more in the weeds. So I started at Toyota Connected in 2017 as an engineer and since moved into management under Kevin currently, primarily working on the drive link platform and safety connect though. I'm in there with the application engineers day to day kind of helping build the platform. Okay, so I own a Toyota, a truck, a Tacoma. I also own another car, it's a Mercedes, sorry, but and I got in a little accident last year. It was like the car is crap in the snow and I'm living in New England. So I spun around on the highway, I wasn't speeding and I hit a like a patch of ice and started spinning around. And I kind of ran into the guardrail and then I was like, what do I do? And all of a sudden this car started talking to me and I'm like, I'm going to get hit. And like, so what happens? Like that experience, I mean, there's like an infinite number of scenarios that you guys have to deal with. Absolutely. So if you go, you know, the working backwards theme, if you go back to the beginning when you guys first started doing this service, it was really innovative at the time. How did you think through those different scenarios and how has it evolved over time? Yeah, it's really, it's a partnership with, you know, actually developing the physical hardware and the ECUs on the vehicles, but then also knowing what the software services, connected services are going to be on the other end of it, right? You know, and these services have been around for a while. We actually built it from the ground up, starting in 2016, 2017. We did have a third party handling that in Toyota and Lexus vehicles, but then we took it on and rebuilt it ourselves, tried to make it a little bit more efficient, more cost effective that really allowed us to bring a cheaper, from a cost perspective, but a very solid solution to the customers. And actually recently, we migrated all of our services over to AWS and actually it has enabled us to provide not just a short trial period, like a year or three years, which is what we do for Toyota and Lexus vehicles, but actually we offer it now for 10 years for our customer. So you as a Toyota customer don't have to make the decision, do I want the safety services? I'm a great driver, I'm never going to need it. We're taking that decision out of your hands and saying, you know what, we're going to give it to you for 10 years with the purchase of your vehicle moving forward. And that really came to light with our recent kind of cost reduction stuff and moving to AWS. But to get to your point about the, you know, someone calling into your vehicle, that is us. The vehicle is smart enough to say, hey, there was an impact or the airbag deployed and there's a communication module in that vehicle that will call out to our platform. Our platform is 99.99% uptime, very, very reliable. That then links into safety agent and within three to five seconds, safety agent will call into your vehicle and say, hey, we noticed you got an accident, are you okay? How can we help? And a click of a button, they can have safety services respond to you and help you out. Amazing, and this service is called Drive Link, is that correct? The service is actually, our product is called Safety Connect. The area that we are within Toyota Connected, we brand of that as Drive Link. Okay, great. Okay, what's under the covers Isaac, pick us underneath, paint a picture of how you guys make all this happen in real time. Sure, yeah, I mean, you know, so when you get into an accident, a couple things happen at once. One is we're going to have the voice call to you as you experience, right? So that's really key and that's one of the foundational elements that we can always fall back to is that voice connection with the customer. So once we have that, we can also get the data as well though and that's one of the things that really impressed me around the culture at Toyota was the resiliency that we have at that service layer to be able to get the data in many different ways. So they've really thought, we've really tried to think that through as far as whenever somebody does get into an accident, what are all the different failover mechanisms we can have to make sure that either we can connect with that customer via voice and or get the data as well so that we know where that customer is located. If they do have an emergency situation where they're going to need an ambulance or a fire truck, we can dispatch very quickly to those locations. And I love that you guys have made this a standard now because sometimes the consumer, and there's a safety issue. Sometimes the consumer's like, ah, I got to, do I have it? Do I take it or not? So forcing them to make that decision because it shouldn't be an economic decision is ultimately what you sort of decided. And you know, whatever, charge me more for the car and I'm not even going to notice and then build the service in. That's a really smart thing to do. Absolutely. What are some of the other trends that you see in the industry broadly that are specifically driving your strategy? I mean, obviously EVs and batteries is a big trend. I don't know if that's affecting you guys, but what are some of the changes that you're seeing that affect your division? Yeah, I mean, I would say the EVs is something that we are taking a hard look at is safety the same with EVs. On the surface, yeah it is, but there's other things that we can dig in a little bit deeper and see if there's different services that we need to provide our customers. I would say with that, vehicles are just getting smarter. More and more data is produced from the vehicles, more and more data we can get and we can take some of that decision-making out of people's hands, our agents' hands and actually do some stuff. I'll say using some AI type of logic to make some of these decisions. Vehicles in the future, right now we might know how fast you're going, where you're located, that type of stuff. In the future, we're going to know how many people are in your vehicle. What type of person is in your vehicle? Do you have three kids in the vehicle? Do you have three adults in your vehicle? Did the vehicle roll over? Where do the impacts come from the vehicle? And if we know that, and our systems know that, we can tell law enforcement or first responders, hey, this is the scenario you're going to. We have communications with the first responders and one of the things they tell us is the more you can tell us when we want to respond, the better it is. There's about what they call a golden hour and a golden hour is where they can really save lives. And if it goes past an hour, that's when things start going downhill. So if they know that, hey, we need to send out a chopper, a life chopper to that accident or four ambulances because it's a big enough accident, that's valuable, valuable, valuable information for them to have. Future cars are going to be able to get that to us. That's huge, right? Because you think about, oh my, I have a flat tire, come help. But we're talking about saving lives, that's big. Isaac, so what's the data underpinning here? I mean, I know obviously you use a Mongo. What do you do with it? What's your database strategy? What's that all look like? Right, I think the biggest thing for us is really resiliency at that data layer. So we want, you know, when we use Mongo, we're not doing a ton of crazy stuff with it honestly. For us, it's really that uptime. So that's what Mongo is able to give us is uptime at also a very large scale. So whenever we started building the platform, we knew that we were going to have millions of vehicles on it. We currently have nine million plus vehicles. We've serviced five million calls. We needed to know that the platform could scale for us and that was just something that we knew we could pick Mongo as that and it would kind of be with us for that journey. So what was that journey like? You migrated to AWS. Tell us about that. Why the decision to migrate? Take us through that. Yeah, yeah. I did talk on it yesterday. So it's very close to my heart. We are a very large AWS customer in other areas of Toyota connected in Toyota. So it made sense for us to move and be closer there. There's fungibility and knowledge and people. Also you get some of the enterprise discounts and support. It just made sense for us to be on a platform that was closer with everybody else. You know, the migration, it was, you know, it took over a year for us from like planning it out to actually executing on it. But when we actually executed it on it, it was a six step execution that started with moving one thing at a time. And Mongo was actually step two in that process. But one of the strategies with the team was, or one of the pillars for the team was, we don't want any downtime for our customers. Like doing a migration is an excuse to say, okay, we're going to take it down for 15 minutes or an hour. No downtime at all. We did not have any downtime at all. The Mongo piece to that was actually probably one of our easiest migration parts of there. We clicked the button. We saw the database start building on AWS. We waited a little bit while they kind of populated. We saw, we could start tearing down the ones that we had in Azure. And then all of a sudden, we're live in AWS. So comparing that to all the different moves that we had to do were firing up EKS and all the other services on the AWS side. And then moving some DNS, and that gets a little bit crazy. The Mongo piece was actually pretty easy. But it took, it ended up taking about four months from the start. We made our first production update into AWS to actually say, hey, we're good. We're fully on AWS and we can tear down the other cloud. So step one was provisioning the core infrastructure, I presume? Yeah, well actually step one for us was because we were very much microservices, event-based architecture, the backbone of our services was actually a cloud application from Azure. So it was an Azure service bus. We couldn't take that with us AWS. So we had to replace it with something. So we actually tore into that code, replace that with actually a SaaS provider. So it wasn't the AWS MQ, it was cloud AMQP. So that was a pretty big step for us. That involved not just our platform team, our DevOps team, that was much more developer change to pull out that core infrastructure or the core messaging and point it to a new one. So that was actually step one for us. Once we got that done and they kind of let it bake for two weeks, we were like, all right, we feel good. Then it was step two of, all right, now let's move Mongo over there. Then we fired up our services over on AWS side. Then we actually pointed the vehicle data over to the AWS side, prayed a little bit, things were working really well. And then some of the smaller services like Redis, then we moved that over to AWS and that's when we were complete. Interesting, I was watching Microsoft Ignite last week and Satya Nadella was making a big deal about how many data centers they have. We have bigger footprint, more footprint than anybody else. And you're here, I don't know if you saw the keynote today, but Adam Salipsky's response to that was his common AWS responses. Yeah, but we have three data centers in each availability zone and they're far enough apart that if there's a fire or a flood or something like that. And so how do you, so I presume given that you want 99.9, probably several nines of uptime that's critical for you guys. It's important how you architect those availability zones. So how do you guys think about that? Maybe you could take us through that a little bit. Yeah, yeah, we're definitely, because we do want 49 uptime, we are definitely multi-region and multi-AZ within each region. So if we ever do experience a regional failure, we have a capability to point DNS to either region and be fully deployed in any region. And we're usually active-active. So it really gives us that uptime that we need to support the safety for- So multiple availability zones and each availability zone's got multiple data centers those are asynchronous links or are they synchronous links? They're synchronous links. So synchronous links is under 100 kilometers, right? So, okay, so you've got that. But you're active-active is across AZs or was it within an AZ? How does that work? Yeah, yeah, so basically we just run our services in multiple AZs so that gives us that capability and we are basically able to round-robin traffic and then all of our back-end services are actually based on a messaging framework. So they can plug messages off either queue that they need. And the system's smart enough to know from a latency standpoint where the request should be serviced, right? Right. Yeah, we do partner, obviously, with our mobile carrier, AT&T right now. They've got an MPLS built for us that has routing within it that knows what's the closest link for where that vehicle's at? Is it on the east coast, is it on the west coast? So it's going to go to the data center that is actually closer to it. Okay, what about the AI question? As a consumer, I'm pushing a button or saying help or you're actually anticipating the problem. So how are you thinking about AI, Gen AI? Where does it fit in your value chain? Yeah, it's an interesting question and you knew it was going to come up. I had to make a slide in it in my presentation just because every slide I had to have everything. You know, for us we have pretty established services. So on the back-end there is some things that we could do from a Gen AI perspective, maybe from a back office. You know, bringing in documents and automatically approving things that we have to do today. You know, police reports, things like that. We really focus, we've got 300 safety connect agents and those are the folks that are going to call into your vehicle if you're in an accident or if you hit the SOS button, they're on the other end of the line for you. They get in some pretty tough situations. So there's some opportunity there for Gen AI to potentially assist our safety connect agents and maybe even anticipate where the customer's going to go with whatever the situation the customer's in. So we are definitely looking at that to see if that's someplace where we can pop in some Gen AI. Yeah, and you're capturing all this data. I mean, right, from the vehicle, you persist most of the data or just some of the data? Do you only persist like serious incidents or do you pretty much persist everything and then filter it out? Yeah, we basically get very, our data is very event driven that we deal with on drive links. So we get the data when an incident happens. So when you get into an accident, when you press the SOS button, we don't deal a ton with streaming data. There are other organizations within Toyota Connective that do that, but our team specifically is a lot more event driven. So we really just have the data that's relevant to that incident. Most of it you throw away and if the deer runs in front of me and I hit it, then you're going to capture that data and send that back to the cloud. Awesome, guys, thanks so much for sharing your story. Congratulations on all the success and your talk yesterday and appreciate your time. Yeah, thank you very much. Thank you so much. All right, keep it right there. More from SuperCloud five here. We're at the Mongo hang space inside the Emerald Lounge. John Furrier and team and I'll be up there tomorrow all week wall to wall coverage live out of our Palo Alto studio coming in from Barcelona as well. The AI wars continue. We'll be right back.