 Welcome back everyone to Cisco's coverage of theCUBE here in Las Vegas, Francisco Live 2023. I'm John Furrier with Dave Vellante. We're breaking down all the action, talking to all the smartest people, customers, executives, all the people making things happen in this next generation internet and this next guest. I'm super excited because our family all has Audi cars and SUVs. Dr. Henning Lozer, head of Audi Production Lab with Audi, great to see you. Dr. Henning, thanks for coming on. Thanks for having me. Yeah, good to see you. So I got to get out of the way. I have Audi's, love Audi's. So I'm a little bit biased disclosure on this interview. Dave, you know that. I do. You're always pushing the Audi on me. I kind of wish I went that way. Audi's a very tech-friendly brand because they're at the KubeCon open source event in Amsterdam. You guys do a lot of tech. A lot of things going on in technology. Your roles are unique. You're bringing automation to the factory floor by virtualizing, hyper-converged, moving from old to new with Cisco, low latency, high performance robots. Sounds like a movie. Tell your story. It almost does. So we don't only love tech and our cars, we also are tech-driven on the way how we produce cars. And in production it's always about becoming more efficient, building better cars in a better way. And so this is the entire smart factory story a lot of companies talk about. And we did too. We want to go down the smart factory route. At some point we figured out, hey, we need to change something. What do we need to change and why? Talking, being from production, talking to automation people, what you usually do if you want a new feature, you buy a box that provides that feature, you put it into your automation cell and then you forget about it because it will run for the next, I don't know, decades. That's the way of thinking, classical thinking in automation. And now you talk about smart factory, you talk about algorithms that get updated, having to deploy a new firmware if you think of all the IT security issues and so on. And we were at the point where we've discovered, hey, if we continue down this old route, adding more sensors and all of them smart with all that compute, it will kill us maintenance-wise. And so we started to look outside the box, hey, how do you do that somewhere else? And that's where the PLAP comes into play. We're the group of nerds that get to play around with technology and figure out hands-on, is this helpful for us in production or not? And we talked to the IT folks, the data center folks, and asked them, hey, if you have this update problem, how would you do that? And they looked at us like, what are you talking about? We've been doing this for ages, like what have you been doing for ages? And they- We talked about a new machine on the floor. And so it was, they introduced us to the concept of hyper-converged infrastructure, CI, CD pipelines and all of that. And like, okay, this functionality, that's exactly what I want. But I wanted to work in the automation constraints. And what are those? And that's basically latency. So we have brownfield factories with all our equipment installed. We don't want to throw it away, put all new equipment in just to change the protocol. So we need to figure out a way how do you tunnel a field pass protocol that's key for automation through a layer three network to a hyper-converge infrastructure that then does the compute and get the data back down to the shop floor so that it does what it's supposed to do. And that was quite a challenge. We talked to Cisco for the networking puzzle. Hey, this is our vision. This is what we want to do. Take the old friend of automation, a single ethernet cable that you plug into the robot and you plug into the PLC and then you know if the electricity is there, if the connectivity is there, you're fine. Nothing can happen. That's rock solid. And take out this cable now and have that data packet, not only travel this one cable but travel it through a layer three network to your hyper-converged infrastructure and make that as fault tolerant, as available as the single cable. So with a smaller lapse time as possible? With the requirement, what we are using is the Profinet safe protocol. You have eight milliseconds of latency requirement, robots talking to the PLC and get the answer back. That's eight milliseconds. PLC is the controller. PLC is the old school controller that controls the machines. So a PLC, a programmable logic controller, that's the kind of the brain and automation cell. It tells the robot, no, you need to do this and then the robot kicks up. It's a robot program, variant ABC, whatever the PLC tells it to do. Now, we want to take this brain which is right now is a piece of hardware and actually it's a piece of hardware that's very intricate. Why? If you have a safety PLC, it's the entire PLC has to be certified for safety. Yeah, all kinds of regulations. You name it. And so we want to take this hardware and then, but what we want to get rid of that hardware, we still want to keep the functionality. So we need a virtual PLC that is certified that runs on a hyper-converged infrastructure and for that to work the network has to provide what we need. So your problem statement is this is a classic OT, IT project. Get rid of the old operating technology. I mean, we've seen this on the factories too. Windows 95s machine, old school. Yeah, but what we're doing solves that very same issue. We've got over 17,000 IPCs in our factory, 17,000 IPCs. And now you need to do an operating system update on all those 17,000 IPCs. Oh, great, next challenge, you only have 20 minutes time within the regular shift to do that operating system update. Big question, how do you do it? A lot of people on radios, go, press the button now. And so it's antiquated, it's old, old school. That's the old way of doing it. So now the solution is what? VMware and Cisco, right? Virtualizing that hyper-converged to the Cisco network. Right, so what we're doing is, I mean, the entire thing is really big. If you think this through, it goes to every single automation piece that we have in our factory. We want to conquer the world, but we want to take it step by step. So we started out with our application that does the quality control. We virtualized that application. We have this running since November of last year in our factory where we produce the e-tron GT and the Audi R8. Right now we are transforming our assembly line. Still an application that does not have these strict latency requirements transform the assembly line. We have the first two texts running already and by the end of the year, we want to have the entire assembly line transformed for the worker guidance systems and also for our screwing control systems. Once we have done that, we have gotten rid of about 80% of our IPCs that we have on the shop floor. That's great. We really consolidated it. Now, after that, the next step will be the PLCs, which we want to have a safety PLC or do the purchasing decision for safety PLC in the middle of next year. And so we have this time until then to show ineffectory environment in operation that this is actually doable and this can be done and then roll that out for our next car project. When I hear PLC, my mind immediately goes to Stuxnet and so I get concerned about security. So what is, the OT guys are always paranoid, rightly so. How are you handling that exposure? It's, what we're doing with this project is actually also benefiting IT security as well. Why? Because we are, once we've done the organ this step, we can do microsegmentation in the network, using all those tools that are available in a layer three network for threat detection, for remediation and stuff like that, that can be done with the Cisco tools that we have for a fabric that we are going to set up or that we're setting up right now, which can actually do the threat detection. Automation works on the layer two network, that's flat. So if there's anywhere, anything plugged into this network, somehow it can get to every single piece of hardware that's in this network. So yes, we have firewalls in place, trying to keep the area small where this can happen, but still, this is the OT network setup, it's not setup for IT security. So by going this step, we get all these new IT security tools that we know of the data center, that we know from our office infrastructure, that we know out of the internet infrastructures and can use those same tools in the IT world. And that is actually then, well, the threat vector goes up, but this helps us doing it. What does the data flow look like in this smart factory vision? A lot of the data in the factory, of course, there's analog, it's temperature, whatever. How does that work? Where do you put the data? How much has persisted? We want, in the first step, we want change anything from what, how it is today. Our factories, how we have been setting them up is, since a couple of decades, or as long as it is available, we transfer analog signals into an ethernet signal by IO boxes, very close to where the signal arrives. And so we are on an IP-based network anyhow. And that enables us to send the data to the PLC, the PLC reacts to it, sends data back. Now, the only thing that happens is that this data gets sent over a little longer distance than it was before. That's the first change. What we can do now is, all of a sudden, we have all of this data that used to be in this automation cell, a firewall, zoning it off from the rest of the production network, and only the data that you explicitly wanted left that cell and was sent somewhere. And in production, you only send data that is meaningful. So if you then want to start a data analytics project, you stand there and say like, okay, I want this and that, and that data as well. And you say, well, but that data doesn't get sent. So you have to go down to the automation cell, reconfigure something so that the data is sent. With our project, all that data is already sent into this hyper-converged infrastructure, and it's processed there and sent back. Nothing changed at the first glance. But if you now want to do data analytics, you want to collect more data, it's merely changing software and changing software and rolling it out over all of your automation cells with a CICD pipeline that you know from a hyper-converged infrastructure. So instead of giving some drives to some maintenance guy and tell them, hey, yeah, just walk, you've got these 1000 automation cells, you walk into everyone and update it and reconfigure it. No thumb drive. No, I don't want those around PLC's. Which takes forever. You get to do this. And so it's, the whole thing helps us with IT security. We're going to set this up as IEC 62443, compatible the network and the entire thing. And it also enables us in the future to do much more and gain much more from the data that we have out of our processes. Am I correct though? That much of the data is ephemeral, it just disappears. You don't need to store it. Is that changing? How much of the data, do you build a data warehouse? I mean, is that changed your thinking? No, we still only store the data that we think is meaningful. Small part of it. Small part of it. What we do do, and we already developed some algorithms that do it, do the inferencing right there in the automation cell, throw out all the zeros and only keep the meaningful data. So, but if you want to do that as well, your inferencing algorithms that look and what is meaningful in this data, they do change. And again, you're back to the question, how do you roll out a change algorithm in an automation cell? The old way of doing it is, walk down there, program it down there, and what we are able to do now, it's hey, it's a couple mouse clicks. And that's where we want to get. That's exactly a great use case of OT, IT and the security angle, it's huge. I guess my question to put you on the spot, Dr. Henning is, since you're such a nerd, you got your lab there, by the way, we like that by the way, you got to be thinking about AI. As you start to succeed with this project, you can almost imagine the AI aspect coming in. You got a lot of data. Inferencing. We just had a great customer, Thomas Scheiba, product manager of networking. There's a lot of use cases of networking. There's also some unlocking of value from data that might not have been there. Just how do you think about that as a tech leader and a tech scientist, tech innovator? What's on your mind? How are you thinking about? How are you framing how to attack AI in a good way to make it valuable for Audi and your customers? I, there are two parts to that answer. First of all, I usually try not to talk about AI. I try to talk about algorithms. Algorithms that help us enhance our production, that help us with insight into our production and that help us steer our production processes. If it is an AI algorithm I need in order to do that, I'll use an AI algorithm. Most of the time, it's probably just a very simple algorithm if this, then that. Which we haven't been able to deploy in the past. So, I strongly believe that software and algorithms will help us enhance our production outcome. Will help us enhance our efficiency. Will help us understand what's happening in our processes and then react to that. And then depending on if you say react to that, you can also start training an algorithm what's the right reaction to do when. So, what we've already done is we've developed an AI algorithm, for example, that detects the quality of each and every spot well that we do. State of the art in the industry is you do statistic sampling, quality control of all of your spot wells, which leads down to every single spot location on the car is measured by ultrasonic measurements once a week. Well, I'm sorry, once a day. Now all of a sudden we have an algorithm that tells us this one's good, this one's good, this one's good. I'm not sure about this one. And then you can change the way you look for quality control because then you can specifically say, hey, besides the statistic sampling, go look at that spot, figure out what's wrong there. That retrains the algorithm. So, AI will help us in our quality. And that's what we want to bring our customers. And there's a human in that loop, obviously. There is a human in that loop that does checking, hey, what this algorithm did, was it okay or not? So, this is the way we think about this AI in production. I cannot foresee that we'll have unchecked AI doing decisions, that won't happen. It will always be kind of like assistant system. Humans plus AI, as we've been saying, is better than AI by itself. We know this from even chess, right? I want to go back to the constraints. You mentioned latency, but given that you're doing so much in the factory, our cost and power also factors in your thinking, or will they become more important going forward? Well, for the sustainability issues, yes, we are transforming our factories to CO2 neutral. We already have our, we have five factories owned by Audi, Brussels, Gure is already CO2 neutral. Mexico, I think almost as well, same as for English at the Neckarzone, we're almost there. And our pledge is to be CO2 neutral, I think by 2025. And so we look into energy consumption of every single bit. And as you know from IT, a hardware consolidation project, always, always saves you energy. It always saves you, how do you say it, material. You'll need much less copper, whatever, silicon for the servers than you need for all those IPCs that you're replacing. So yes, that's also an issue that we're glad to take with this project, but it wasn't the one that was pushing this. What was pushing us was finding a way to have the smart factory and do algorithmic updates, do software updates and so on without having to employ billions of maintenance people that walk to every single device and reprogram it, do firmware updates, whatever. It's possible to scale in that model. It's a more efficient, higher quality, everyone wins. It's about scaling. Only if we do this, we come from a proof of concept in one automation cell and are able to scale it over the entire body and white shop and actually over the factories of the VW Group. Dr. Hennig-Lozer, thank you so much for coming on theCUBE and sharing your awesome story. Love the smart factory. Love the IOTs, industrial IOT in there. So much going on. You're the head of production lab at Audi. You're famous, great to meet you in person. And I'm looking forward to driving my next Audi. All right. We get free Audi's for everyone, brought to you by the Audi, from theCUBE. John and Dave driving Audi, brought to you by... No, not yet. We're working on that, right? Okay, all right, working on that. Keep coverage here. We'll be right back after this short break. I'm John with Dave Vellante. And let's break it down all the actions. Just go live day two. We'll be right back.