 So, after half a year, Volkswagen committed the tweaks to their mission readings. Those two boys, Daniel Lange and Felix Domke here on my left, will share some insights with us. Daniel will not only focus on the ECUs, which is the acronym for the Electronic Control Unit, and I think we're seeing one over here already, whereby Felix will show us some tricks to extract and tweak the firmware. On both sides, we will see how many people have been involved in the entire process, and we would get an idea of what everything is involved in there. So, you applause, and I'm going to take over the pitch here. Good luck. All right. Hello? Okay. Hey, so, I'm Felix Domke. Do we see the video output yet? No. Anyway, I'm Felix Domke. I'm here on my own because I was personally interested in how Volkswagen is cheating on their emission control, and maybe we'll get video at some point. I'm basically, I want to stress that I was self-funded. I did this with my own money because I was personally interested in this, so I did not do this on behalf of anyone else. I'm going to start the keynote again and see whether that one works. I'm sure you figured this out. Oh, it worked before. Yes, that's because some, one of us too wanted to use a Mac. I wanted to use Keynote. I don't care which operating system. This one works. Anyway, so I will now hand off to Daniel, which will give the first part of this talk, and after that I will give the second part of this talk on. Okay, thanks, Felix. My name is Daniel. I used to work for a big Bavarian auto manufacturer, which is not Audi. For 14 years, I've been running the IT strategy, I've been doing IT architecture, and most relevant to this talk, I've been responsible for the process chain electronics and electric. I've done the rollout of connected drive in China, so I kind of have quite deep insight into how the automotive industry works, and I'd like to share a bit with you. I'm an engineer by training, I guess many of you are, and I kind of want to share how the engineers think inside such a big corporation like Volkswagen and kind of what pressures, what boundary conditions they are working on. I have my own company now, which makes my life a bit easier than Felix's, as you see in the legal disclaimer. These are folks from the UK. They're called Brandelism, I hope you noticed the McDonald's M at the end. Those are folks who started a few years ago to reclaim the public space. They were just annoyed by all of that advertising, and when the Paris negotiations for the eco-treatment came, they just kind of felt a big invitation to use the opportunity that Volkswagen has created for all of us and make advertising in their style, but perhaps not in the message they would usually have conveyed. I'm a strategist, so what is the thing like that defines how the automotive industry works today? We're in a saturated market, right? In the developed countries, so everywhere in Europe, in the North Americas, everybody has a car that wants one. Some have too. So when you want to sell another car, you're basically talking about replacing an existing car with another one. The only growth you have is in the brick states. Brick is Brazil, Russia, India, and China, and here especially in China. You have a big overcapacity. There's just too many automotive manufacturers, and there's too many plants they have, so all of them basically struggle to get the loads on the plants, to produce enough cars, and to have those cars sold at some point in time. Because the queuing in between production and sales is actually the big parking spaces you see in Bremerhaven or so, where there's 10,000 in some countries, even 100,000 cars basically stored in between production and sales. 10 years ago, 15 years ago, that didn't exist. The cars were basically sold off the factory, but people have been moving away very, very slowly from cars they have as a saturated market. That's just not that easy to sell a car anymore. That is because of social shifts, right? When I was young, there was like, you know, the dukes of hazard. There was generally this car that basically is the star of the show. There was Night Rider, and nobody watched it for David Hasselhoff, not even the girls. They watched it for Kit. So when I was young, I wanted to own a car. I wanted to have Kit, possibly, and when I grew old enough, I found out I can get a car that looks like Kit, but all the fun stuff is not in there, so I turned to computers. The next thing is urbanization, mega cities. We're living in very condensed spaces in those cities. If you talk about a place like Beijing, where there's like 21 million people in an area that is one city where there's nothing big in between, there's no river, there's no forest. It's just like one city. If you go to Tokyo Yokohama, you can drive on the motorway for nearly three hours when you enter the city before you leave the city, and you're driving on the motorway. You're driving on an elevated road for which you pay toll, so you actually can drive, but it's three hours before you leave the city again. And in these cities, owning a car and operating a car is about the worst thing you can do. You just don't want to do that. The average speed of a car in Beijing these days is 12 kilometers an hour. So, you know, if you're a good runner, you can beat that. And incidentally, this is exactly the speed that a horse carriage makes. So we have managed to basically undo all of the innovation of the last 200 years. It's just the interior is a bit more comfortable, but the actual getting from A to B is the same as with a horse carriage these days. Okay. And then there's technology shifts. The problem is there are big things, big visions that everybody follows, like electric mobility. So electric mobility means you buy a car that's one and a half times the price of your standard car. You lug around 300 kilograms of batteries for no apparent reason to do so, and you now need to install something in your garage, which you most probably don't have. Look at mega cities to be able to recharge the car because it only goes 100 miles. So it's currently not a very compelling thing to sell to the end customer. There's self-driving cars, which is kind of a great big vision, but I would really call that a vision. A vision is something that's not being implemented in my lifetime. And then there's downsizing. Downsizing means everybody wanted to have the biggest engine. Everybody wanted to have the biggest car, let's say 10 years ago. You wanted to have that six cylinder that was giving you status. But now the automotive industry has an overall cap on how much emissions the average new car fleet may have. And they can only reach that if they manage to sell smaller engines to you because for everybody who buys a really big engine that will never ever make that emission cap, they need somebody whom they've sold a small car to, preferably an electric car because they even have statistical advantages in this to make them a bit more attractive, to set that off, right? So very literally the poor guy with the small car needs to exist for the rich guy that drives the 8-cylinder and doesn't give a shit. All right, now strategy-wise, there's only two things that an automotive car company is driven in. And that's really everything there is. There is reach a target, Rosy. Rosy is return on capital employed. That is just two numbers. You're EBIT, which is your earnings before income tax. And the amount of money you have in the company, the employed capital, which you've got from people that lend it to you or from your stakeholders, from your investors. And that is what the company has measured again. Every automotive company basically runs like this. Just this one figure. It's a percentage, like 30%. 30% means on the money you have, you basically made a 30% return during that year. The downside of measuring in Rosy is that every time you use that euro or that dollar, it counts again because the money works for you. So that means what you're looking at is a company that gradually moves from a very industrial type of application to something that tries to move faster, that tries to be quick and regain money more fast. And then there's outperform the competition. You have to understand the situation that there's a good dozen companies and everybody has the same strategic position. We will outperform the competition. So statistically, you know that half of them are going to fail because that won't happen, right? Somebody has to be the lower half. But the only thing I've seen in about five or six companies where I know the strategy in detail is the sequence. Is the first or is the letter the more important one? And sometimes that depends on markets, right? There's this new emerging market and you want to outperform the competition. You want to grow more. And then there's this legged market somewhere in the European Union where you just look at the money, you know, how much money are we making on this? But that's all. That's how an engineer is basically steered. That is the strategy. And that means when you break that down through the levels of hierarchy, what is counting is like how much money do you need to make this? How much money are you going to make on this? Those two divided will be basically contributing to the Rosie. And do you deliver anything that can help us outperform the competition? You notice that there's a lack which is, you know, what does the customer want or, you know, what is good for associates or something like that. Just in case you hadn't noticed before. Okay. I'd like to do a bit of a quiz with you before you all fall asleep after lunch. 11 million. 11 million in the context of the exhaust emissions scandal. What is it? What is that number? Correct. Cars affected. So 11 million is actually the Volkswagen cars which need to be recalled worldwide to get, you know, this little filter thing fixed and their software updated to meet the emission targets which they had been produced against. 1,500. Number of engineers? No, not correct. Number of engineers would be above 10,000 for a car in Volkswagen Group. Sorry. Cost for fixing it per car? No. Maximum 600 we're going to see later? No. Well, that was too difficult then and that was a bit intentional. That's the amount of hard disks they collected from the associates. Now the thing is we've had in the press that there is maximum 30 managers which are responsible for this emission scandal within Volkswagen. But then they collect 1,500 hard disks and USB sticks from 380 associates and that number is a month old because they haven't reported new numbers. So something is mismatching there, right? Something is mismatching there. So the first number we have for how many associates are actually somehow affected by this is 380 because you come to work somewhere in Wolfsburg I think, right? And then there's this nice chap coming up and telling you actually we took the hard disk off your PC, you're going to get a new one from IT. We guess tomorrow they're a bit behind with, you know, 6.7 billion. Just shout. Fine. No, that will be less, much less. Yes, you're getting close. It's the money they put back they set aside to actually pay for the recall and the legal fees. Now if you divide that by 11 million you get about 600 euros per car. So it's not that kind of much money per car. In Europe the plan is basically that you go to the dealer and get a software update. In the States people already got $1,000 in cash and in coupons as a goodwill measure. So something I learned from Martin Haase here going to CCC Congress all the time is that we need to read text really well, right? So the upper one is the original in German. The lower one is my English translation. The English translation is as accurate as possible. So it's not good English. Please excuse that. It is so you get the gist in case you can only read the English. So that is Mr. Putsch. He's the president of the Volkswagen Supervisory Board. He's the poor guy that now has to sort it all out. He used to be the CFO. We're going to see why that is important a bit later. And he has made this analysis. It was individual misbehavior. So it's not an organizational problem. It's weaknesses in particular processes. And it's the attitude in particular sub-partitions. You can take part in this. It's impossible to translate in English. It's actually impossible in German, but the legal team came up with that. So the attitude in particular sub-partitions of the company to tolerate rule violations. Now, if we go through this very quickly, it's not a rule violation. You violated the fucking law. The other thing is, if you have particular processes, you have particular associates and you have particular sub-partitions of the company that tells you something, right? It just tells you something. This was probably two days' work of somebody in the legal team. And I guess you notice, right? I guess you notice. Legal team is probably these people. Johnson Day is a big American lawyer company. And they've asked them to help with sorting out this. Now, the funny thing is, there's public prosecutors all over the planet interested in Volkswagen. But Volkswagen thinks it's not really clever to have those people come in and find all the info. It's better to have Jones Day, their own kind of board in legal team, ask the associates first. Now, the problem is, whenever the, let's say, German prosecutors wake up and go in there and say, like, we would like to see what has happened. So please hand over the material. Please hand over the hard disks. They will get a very, very nice reception. Be greeted with coffee and show in a room where all of the hard disks and everything is stored. We collected it for you. I have no idea whether they're going to show everything to them. I have no idea whether there may be some material lost in between. We've heard from Anna earlier, you know, in Germany it seems to be that things hit the shredder and hard disks get lost and everything. So if it works like that in the government, I have no idea how it works in companies. But if I was on the prosecutors, I'd probably see that I kind of speed up a little because otherwise you'll get all pre-prepared material. And because Jones Day can't do all of that, you have to interview all of those people and you have to look through the hard disks, they ask Deloitte to come in and help them. Now, Deloitte are a very good company. They have very, very good forensic teams, so that's a very good choice. But the important thing here is out of the four big consulting companies that do finance analysis and stuff, those are the only Americans. The others are headquartered somewhere else. So what it tells you here, American legal team, American auditors, that's where Volkswagen looks. Volkswagen is actually afraid of America. They are not that afraid of Europe or some other country in some other continent. Now let's talk text a bit again. We have no findings on the involvement of the supervisory board or the board of management presented. Now again, no findings, okay, presented, right? It's not we don't have any findings or there is nothing, because we have no findings presented. And the other thing is involvement, that's an odd term. In German, involvierung, that's not even German, right? If you look it up, involvierung, nobody of you talks of involvierung when you talk to your family or when you do something in work. The trick here is the supervisory board has a reason for existing, supervision. The board of management has a reason for existing and that is decision. They are the deciding body. None of them are ever involved, right? When you work on something in a big hierarchical company, there is no involvement of your board member. There's no involvement of your supervisory board member. So per definition, they cannot have an involvement, right? If he wanted to be straight, he would have said I as a former board of management director and now as the head of the supervisory board, I guarantee there was no involvement of my or my colleagues in this. And if there was, I will pay back my salary. I will go to jail. I will whatever, right? Sacrifice a goat. But that would have been kind of straight communication. But this is not straight communication. This is bullshit. Okay, quiz time, 10. You remember this guy here told us there's no involvement in anything fishy, right? It's all those small engineers, all those bad, bad people down there, but they are going to hunt them down, right? So there's no involvement with anything fishy here. So in that context, what is 10? 10 board members close. They have a little more. Levels of powers. Quite good. I think eight or so, but you're quite close. No, it's actually the amount of planes that Volkswagen owns. All of them are jet planes because if you're a board member, you have to fly in style. And because there's nothing ever fishy at Volkswagen, it is run by Lion Air Services out of the Braunschweig Airport. And obviously, Lion Air Services is registered in Georgetown on the Cayman Islands. Nothing fishy ever in that company. Okay, let's get back to topic. I have about another 10 minutes before I want to get Felix the chance to show you what he has done on the ECUs. So I need to kind of get you a bit up to speed about how all of this context here works. And this here is called the NEDC. So it's the new European driving cycle. This is what your car is tested against for emissions. It works like that. You can condition the vehicle a day before, which means you really drive it hard on the Autobahn. So, you know, the exhaust is really free and everything. And then you do these cycles here, where you basically accelerate the vehicle, slow down, accelerate the vehicle, slow down, accelerate the vehicle, slow a bit down, slow a bit more down. And then you cycle again. And the last cycle five is an optional one, depending on what you measure that is actually going to the Autobahn. And you're going up to a top speed of 120 kilometers for a very short period of time. The people that have detected the tweaked emissions in the Volkswagen Jetta and Passard they looked at, they have called this is a very light usage cycle. And they called it unrealistic, right? Because basically nobody drives the car like this, so it's a very artificial thing. And that is the problem for the engineer, right? The engineer looks at this and says, yeah, it's a standard, it's something we do to measure against. But nobody drives like this, it's not realistic, right? So if we fake the data in this, we're not actually faking something our customer uses because no customer drives like this, right? It's very artificial. And there's a very good report by ICCT, which is mind the gap, which is like what you hear in London when you go in the tube. And what they mean is the gap between what gets out when you measure emissions like this and what gets out when you actually drive the car. And that gap is widening year by year by year. Because engineers get better and better at optimizing for this cycle, the cars on the street, they do get better as well, but less, right? That's why the gap widens. And trickery on those tests is very common. I'm sorry you can't probably read that in the stream and probably can't read that when you're back down there, but that's an original slide I had to take from transport environment, from that report which I just named. And what it says there is like what tricks people are doing to actually drive down the emissions. For example, they blow up the tires by three bars more than you could actually use them on the road. Now when you do, the bottom of the tire looks like this, right? So that means you only have a very, very small portion of the tire that still touches the ground so your resistance gets reduced. They put diesel into the oil because diesel is lighter than the oil which you're using inside a vehicle, so friction gets reduced. They take off the side mirror on the passenger side because that is not legally required to be existing, so it's resistance, so get away with it. They tape close all of the openings of the vehicle because obviously when the wind goes over it, it goes much smoother once you have everything taped. Now in all of these things are either okay or they're a kind of borderline gray area, and they do this. This is how actually emissions are tested. So this is why an engineer when he looks at this says, yeah, it's an optimization problem, right? They want me to get a low number, and I have pretty clever ideas which involve diesel and sticky tape and everything to reduce the number. The results are this. That's from a 2012 report from, no, 2013 report, sorry, from ADAC, the German MOT company. And what you see is the lighter blue ones are actually the emissions which the car produces in this cycle. The darker blue ones are the ones which they produce when you just go on the motorway and drive them. And you see that there is a discrepancy which is, you know, 10 times, 20 times, 30 times, what is the measured data? So what you need to understand is that even in the past nobody ever thought, nobody in the industry ever thought that the data which was measured had any real connection with reality, right? The only connection was you knew that what you're measuring within the duty cycle, NEDC, is definitely less than what you would ever see in any real time use. But that's it, that's it. This is kind of, that's no secret, right? It's something that has been out there for years. Now the folks at Deutsche Umwelthilfe, which are actually people that helped find out what Volkswagen did, they wanted to see did others do it as well. And because I wanted to kind of give you as much information as possible, we're going to look at this product here now, which is not a Volkswagen, as you may see. And when you measure this car, it actually looks like this. So that means when the car is thinking it is running an NEDC because it is conditioned to do so, it is the right temperature, it is the right setup, it actually delivers the blue box. And if you run it because you just run it and you don't do the conditioning, it delivers the gray bars. Now there's many things you can say about how they measured this, because obviously this is not kind of science to the best level of accuracy. But you do see a pattern here, and you do see the pattern of about the 30, 35 fold emissions. And that is what you always see, because this is what an engine like the one in this car, 1.6 liter diesel engine, if I remember correctly, actually does when it's just operated normally. Right? And the lower ones are the ones which you get when the engineers did all the good tweaking. Now, why has all of this, oh, sorry, so this is just one test, right? And you see that this test basically, when the vehicle is cold, you basically get fresh air with a nice rose smell out of the exhaust. And when the vehicle is operated normally, you basically get what you expect, you get the combustion products out of burning diesel. Now, why is all of this now a problem? This is now a problem because of the American legal system. The American legal system is very, very different from what people in the European Union are used to. In America, there are two things which are a bit strange, perhaps, to somebody who's accustomed with, say, a German legal system. The first thing is there's juries. So there's common people that actually decide about what's right or what's wrong. And that means what they award as compensation to people that have had disadvantage are often astronomical figures. Now, these figures are sometimes reduced again by the judges, but it's not uncommon that if something hurt you or you got into an accident or something like that, you're awarded million dollar sums. In Germany, when somebody shoots your eye out, you may be getting 100,000 euros. So there's a huge discrepancy there. And the other thing is in America, there are punitive damages. Punitive damages means you did something wrong, you did it on purpose, and you are punished for it. In Europe, a company basically is you did something wrong, so now you have to compensate the disadvantage somebody else had. So to a certain extent, a company that doesn't try to trick actually kind of loses an opportunity because if they are not detected to be tricking, they have just saved money. There's no punitive element. You will go to jail for this, at least in this context of environmental regulation. Now, in case you couldn't read that, that's actually a sign I took in California. You go into a store and it tells you that basically everything you see there and touch there is giving you cancer and your unborn children will be damaged. This is what it says there, like belt shoes, jewelry, handbags, all products with metal, and everything causes cancer, birth defects, and other reproductive damages. So this is America, right? Their view of protecting the consumer is completely different from Europe. And this is why Volkswagen goes and says, we will show good faith. We will give you American Volkswagen owner a thousand dollars because we just want to make sure that, you know, at least now we care. It's important that you care because the jury will say, yeah, well, at least they awarded a thousand or maybe a little too little, but, you know, at least they did something. The jury would say that a professional judge in Germany would say, like, why? So this is why as a European customer, you actually go to the dealership and if that guy is really nice, you may be getting a coffee while you wait the hour that he flashes your car, right? So that's the only thing you're currently supposedly getting in Europe. Okay, now the problem is what they did hurts because if you do the statistics, very nice people have published a publication here, a real scientific publication, where they did their maths and they say, like, 59 people may be dying earlier in the United States because of the additional emissions in the environment which they took in and which may damage their body. The social cost of treating those people because they may be developing cancer, they may be going to hospital and so on is about 450 million euros. Now that's statistics, right? Lies, damn lies and statistics. Mark Twain is often quoted with that. But the problem is that is a real cost. It is a real damage. If you do violate emission laws, it is something that is damaging people's health. It may be something that is difficult to prove statistically, but it is something which you kind of don't only do to kind of save money here or there. It is something which you do to actually hurt people. Okay, I need to speed up a bit. So I'll very sorry skip this. That's the next quiz. 15.9 million is actually the salary of this guy here. That's a lady from BMW. I just wanted to put that out there. She says, like, it shouldn't be called diesel gate. It's Volkswagen gate. We never did anything wrong at BMW. And the SZ actually, yeah, they follow, right? In November, it was Abga scandal. In December, it's Volkswagen Abga scandal. The only problem is that even in 2000, BMW was caught cheating on the motorbike. So this is 15 years ago. 15 years ago, BMW actually put the same code which we are now seeing in Volkswagen into their ECUs for the F650 motorcycle. And we were seeing in here the same 34, this case, fold increase in between real use and test bench use. Now, honestly, they've been caught. They've been caught early and they fixed it. So in 2001, they actually brought a new version and apparently that didn't have this cheat code anymore. But here we see a pattern again. Too little time for development. Too little money willing to be spent on this. So engineers try to trick. When you get caught and you get caught early, nobody probably of you remembered this here. It's fine. It kind of fades away into history. If you're Volkswagen, you have 11 million cars out of there, you have a big problem. Okay, I'll skip this one. It's really nice. You'll see it in the slides, but I have to go to this year to give Felix enough time. So how does component development work? There's a huge set of legal frameworks. It's a very structured top-down process. You get requirements from the people that represent the market in the company. You get requirements from the CFO, from the finance director. And these are broken down into documents which are more than a thousand pages long, and there's every single detail that could exist in this ECU written out. There's a piece of paper for everything it does. Everything. There's not a bit in this thing which is not pushed down into a very hard set of requirements. This is then put into a tool, often, you know, rational doors by IBM or something like this. And then every time something changes, this is documented. There's a complete paper trail, right? So that means unless there will be a cover-up, unless we are not given all the information as a public, there's no way Volkswagen cannot find out who did exactly what at what point in time which level of management was involved. Because every step of the development goes through a queue gate, a quality gate. There's managers sitting there and they're approving everything it does, every progress that has been made, and they're getting reports at least bi-weekly on the progress. And these reports go up the ladder, right? They are copied to the next levels of management. So this is a fully transparent process and this is a fully top-down driven process. It is completely impossible that you have an engineer that sits there and sits like, well, I want to cheat and does the code. There's no motivation for him to do either, right? He doesn't get any money for it. It's, you know, it would only kind of be risking his career. So he won't do. And this is why we have paper trails and this is why engineers have written down, I'm doing this because my manager told me to do this. Or this is why you have Bosch sending a letter in 2007 to Volkswagen and says, we delivered you this code you requested. We are your supplier, we do. But if you use it in production, it will be illegal. And they did. So this is how actually this exhaust system works and this is a little bit important to understand what Felix is now doing and showing you how the ECU that manages it all works. So to the left would be the engine. To the right is the exhaust, the end of the exhaust where the rain has come out. And the first thing is you have diesel-oxid catalytic and it basically takes out the interesting stuff here is CO, so carbon oxide, and PM the particle mass to 98 percent, 50 percent. The hydrocarbonides, before that, they just kind of, you know, don't go through the rest of the process anymore. Then you have a filter that basically traps all of the diesel particles, the stuff that causes cancer in your lungs, but you have to burn them out at some point in time about every 700 kilometers when there have been enough collected. So it's a bit of a trick, right? The trick is you collect them so they don't kind of exit the exhaust, but at some point in time you have to burn them again so they do exit the exhaust. Now the positive thing here is they get larger and the larger they are, the less risk they, at least as much as we know, cause as a health hazard. So this is kind of the DPF here. And then at the end, this is the really interesting thing. This is what kind of most of the scandal now focuses on. There's a selective catalytic reduction and what this thing does is it does reduce a particle mass. It does reduce a particle, so that's nice. But the interesting thing is NOx. It goes against this to about 90%, right? So this is what it is made for. It basically injects urea into the airflow and helps to reduce the NOx content by creating byproducts which are mostly water that comes out of the end of the exhaust. And this is a system, this is a very complex technical system that has to be managed and this is managed by an NCU. This ECU which they selected to do this and everybody does is the engine ECU because to the left of the diagram before was this big engine, you didn't see it and fell off the diagram, but that's actually the fan blowing into the system. So this is what you want to manage to actually control what happens there. Now this thing is quite a sophisticated processor. It's about the most complex device outside multimedia and entertainment which you find in the car and it is a very proprietary thing because it contains a physical model of engines. So there have been hundreds, if not thousands of engineers sitting there and modeling how an engine works, really physically modeling it. And the things that an OEM, so an original equipment manufacturer, a car maker can actually tweak are variables. They can say, okay, my engine has this and this size, my combustion cycle looks like this and that, but the code itself is opaque to the OEM. It is a proprietary product which you can buy from Continental or Bosch or so. And there's about 20,000 variables which you can tune and this thing is simulated and tested to death because it is hugely important. Because you have this machine here that has like whatever, 100, 200 horsepower and if you steer it wrong it will blow up and it will blow up really hard. So this is why this thing is about the best tested piece of software you will ever find. Which also again means there's everything documented, everything is written down, everything is seen by everybody who is working with these, whether it's in development, whether it's in integration, whether it's in the plans that flash these things and so on. So there's nothing secret here in this, right? The functions which are there are actually there to be seen, well seen if they are named apparently and that is something that Felix will talk about. Thank you. Hey, okay, so I will do the second part of this talk. I'm Felix by the way. So my motivation at this was a little bit different. I'm curious and we can find a lot of source material for this whole scandal. We can find a lot of information in the press, a lot of information in the Volkswagen press releases. However, it should be easier because all the cars are there, the 11 million cars are out there that have the cheat code in them. And we are hackers and we know code and the truth is in the code. So my approach was rather, well, let's take a car, let's take it apart, let's take the firmware out of it, let's throw it in a disassembler, maybe get some measurements and then look at what the car is actually doing instead of relying on all of this second-hand, third-hand information. So what do we need for this approach? So first of all, yeah, we need a car that's affected. You need to drive that car somehow and driving a car on an open road can be dangerous if you have to follow a particular driving cycle or something. So there's a dyno which you can put the car on and then you can just drive without the car physically moving. The wheels aren't moving but the car isn't moving. And this is what other people have done and they've taken very interesting measurements out of this. However, we as hackers, we can go one step further. We can take a look at the ECU itself. And not only that, we can also ask other people who worked with these things and may be able to get more information about them. I will talk about this in a minute. So first of all, this is my car. Luckily, that car was affected by the recall. So I was very happy when I got the letter telling me I have to go to the shop in January and get a firmware update because firmware updates are exciting, right? I love updating things, so updating a car seems great. Yeah, it sucked that my car was putting out more emissions than it should have, but otherwise, it gave me the chance to actually look at the car. I mean, I could have rented a car or something, but that makes it much easier. I also went on a dyno with my car. On a dyno, there are no speed limits or no people to run over when you just have to keep a constant speed or something. So it makes things much easier. Yeah, okay. I talked about ripping apart my car and just assembling it. I didn't really want to do that. So what I did instead was what I always do. I go to eBay and I bought an extra ECU. Here it is. Can you... Maybe we can show it on... You can go here after the talk and take a look at this. So this is the ECU. This here is the main CPU. That also includes the flash. On the other side, there are the power drivers that drive the actual stuff in the car. And then there's other like watchdog circuits and so on. Okay, thank you. Okay, yeah. So it's actually... The ECU was built by Bosch. It's an EDC17C46. That's the name of the hardware. And again, it can easily be obtained on eBay and you can put it on your desk. You apply 12 volt to it and then it boots. I mean, it will complain about a lot of sensors being missing and so on, but you can see it executing code. And it doesn't have this very same firmware as my car, but it's very close. And the flash chip is unfortunately in the same package as the main CPU, which is an Infineon tricor chip, which is apparently only used in automotive equipment, or at least I'm only aware of that being used there. And I was able to dump the flash by taking the hardware and exploiting a bug in the hardware that I haven't found documented anywhere, but it was not that complicated. And then I had the firmware done. They had a two megabit binary and I throw it in a disassembler. And what we see is interesting because the code is written very different from other code that we know. So usual code has a lot of flow control and usually more or less resembles spaghetti code. This was the exact opposite. It's more like someone took electrical schematics and put them into code. So there's a set of input signals, there's a set of processing on it, and there's a set of output signals. And that gets updated every 10 milliseconds or once per rotation depending on the process. So it's a very interesting way of writing software and building this. And also it's very data-driven, so a large part of the firmware is not code, but it's data. And all of the computations, they don't use constants at all. They always refer to something from the data section. And as Daniel said, that Bosch writes this code, the code is not directly visible to Volkswagen, but they have visibility into this data and they know what the data does and they have tools to change the data. So Volkswagen and other companies can customize this. Really, they cannot just customize it. They can change the whole behavior of this ECU by changing just the data, not the code. The ECU really is a small embedded machine in your car that takes care of the engine. It's an engine, electronic control unit, or there are multiple names for it. The most important thing that it does is that it takes sensor input, for example, the throttle, and then it applies control to the system. For example, it calculates the amount of fuel to inject, the amount of air to inject to make the motor running at the speed you want it to run. These days it's much more complicated and one important thing the ECU does these days is emission control. And so this is why we would expect to find the cheat code, the code that cheats that Volkswagen used to cheat in the whole thing. We would expect it to find it in the ECU. Now, taking a look at two megabyte firmware binaries that doesn't have any visible strings in it, it's kind of painful if you just do setting code analysis. What I did was to do real-time logging. You can actually read data from your ECU by plugging into this OBD2 port which is next to your steering wheel, and while the engine is running, you can read out certain data. Usually you can read out boring data like RPM and speed and some things that the vendor wants you to see, but there's also a mode that's a little bit hidden, but you can get pretty easily into it. Where you can read by address, where you can just read the whole memory. Well, not everything, some security data is logged out for, but the data we are interested in is we can read that memory. Now, we still need to understand where the interesting stuff is, so we can disassemble a firmware and that's all fine. We can also get a little help from something called A2L files. So the chip tuners use them extensively when they change the mapping so they want to optimize an engine for a different goal, for example, for more power instead of a long lifetime or something. They change things in the ECU firmware. And they do reverse engineer a lot, but they also got these files and I'm not sure how they got them, but they are out there and if you use the right Google terms, you will find them. They are specific to each firmware. I wasn't able to find one for my actual firmware, but I was able to find one firmware that's close to mine. And if you look into this file, what you see is you see the symbol names, it's basically a fancy map file. You see the symbol names, you see a mostly German description of that symbol, of that signal. You see the real use unit and you see the address in memory that we can read it. So with the help of these files, we can read almost any internal state in the ECU. We still have to make sense out of that, but at least we know where the data is and what to look for. It's surprisingly how complex an ECU is. For example, this thing, well, what does it display? Everybody would say, well, it's a function of RPM, right? It shows you how fast the engine is running. Well, it's not quite the case and if we look careful, we see that this code is post-processing the RPM signal. It's 12 kilobyte of densely written code that has a lot of internal state that tries to make the RPM value to convert it to something that the customer wants to see. For example, you want your idle speed to be stuck at 780. You don't want to oscillate, but in reality it does and this code takes away all of that and makes it flat 780. So you realize probably at this point that there is a lot of cheating that could go on here without most people noticing. You don't really believe the speedometer in your car displays your actual speed, right? It displays something, but it's related to speed. But let's get back to the topic. So selective catalytic reduction is the process of, well, if you don't have it, you get a lot of NOx of nitrogen oxides at the end of the exhaust. That's bad. You don't want that. So there's one way of getting rid of this is to add more catalyst. And the SCR catalyst, I simplified this a lot. You can find a lot more information about this. SCR is a process that reduces the NOx using something called DEF or add blue is a term for it. It's some fluid that you put in there. It's a urea water solution and the add blue at a high temperature converts to ammonia and then it reacts with the NOx to nitrogen and water, which is great because that's not any way harmful to us. However, there's a problem here because the dosage of the add blue needs to be correct and it's very hard to do. If we dose too little of that, well, we get the conversion is not perfect and we will still get a lot of NOx at the output, which is better than not doing anything it's not perfect, but it's not more harmful than before. However, if you put in too much of the add blue, what you get at the output is actually ammonia and you really don't want that. So the primary goal of emission control is if you have the SCR system is to eliminate as much as possible of the NOx and minimize the amount of ammonia that still comes out of the exhaust pipe and only as NH3. Calculating the right dosage works with a model again. So they modeled everything that happens in the exhaust processing. They have a model of the catalyst. They have a model of the internal state. They do have a number of sensors and outputs from the other models that tell them a lot of values and the model uses this with a lot of internal storage, internal state and the model then calculates the amount of add blue to those to convert as much NOx as possible without leaking any ammonia. And the way things usually work on an ECU is that there's one system that controls things and there's another system that monitors things. And it's independent from the main system and it tries to be as independent as possible. It's still running on the same hardware, but it's not sharing a lot of code. And there's an efficiency monitoring scheme that if the conversion is not good enough anymore it will flag this as an OBD2 error and you will see your check engine light going on and then you go to the shop and the shop will diagnose your car and will fix this. For example if your catalyst is broken. And based on the test results we would have expected this efficiency monitoring to actually flag the inefficiencies. But it didn't. So it turns out the main model doesn't always work. There are some operating conditions where the main model is not sufficient. It has certain bounds where it works and outside of these conditions for example if the engine is too hot or if the exhaust mass flow is too large or something like that the model doesn't produce meaningful results and it may overdose the air blue and we don't want that. So there's an alternative model which is much much simpler with only a few sensor inputs and doesn't rely on as many variables to be perfect and we'll still calculate an air blue dosage. However the main goal of this alternative model is to make the exhaust processing work in all situations without ever overdosing the NH3. And they're calculating both of these models and then they are selecting one of the model and the output of the selection controls the air blue dosage the pump that injects the air blue into the exhaust and there's code that controls which of the model to use. And there's also a statistic module that counts how often each mode is selected. And again all of these model selection depends on the data. It's code that does the selection but it depends a lot of data that are parameters to all of this. So let's take a look at the selection criteria for this alternative model and a lot of these parameters are dummy variables, right? Things that can never happen, for example the atmospheric pressure can't be negative so that can never happen. Or the air temperature I hope it's never larger than that or smaller than .1 Kelvin, right? Or how the one thing stack out and that was a check if the engine condition is larger than a negative temperature which does not exist, I mean the temperature is always positive so the last one is always true. So the model that would be selected would always be the alternative model and that sounded weird and I was looking at the film and maybe I understood it incorrectly or maybe I looked at the wrong place when looking at these parameters or something but if we look at the intermediate results there's a bit at a certain location that tells us which model was selected and that bit is indeed always set. That is weird. It sounds fishy. So let's take a look at the statistics. The car counts whenever what model you're in. So 20% of the case my car does not do dozing at all. So I drove some time and then looked at the values and that's the 20% where it doesn't do anything, it's mostly the warm-up cycle but every time it does something it's actually the alternative model which we know does underdose NH3 because it doesn't want to leak ammonia and yeah that makes sense but my car uses much less than expected of the AdBlue so the expected value is roughly 2.5 liters or something per thousand kilometers of the AdBlue in my case it only used .6 liters per thousand kilometers which is great for me because I don't have to refill this tank very often in fact I never had to do it the shop always does it when I'm there but this is fishy and let's take a look at this and what we also see is that sometimes the regular model is active so there must be something more if we look at the selection criteria we will find that there's an additional term there that I haven't found before and there's an additional condition that has to be true in order to go to the alternative model that underdoses if we look at the particular conditions and we find a lot of stuff that's related to diagnostics things that you can do in the shop so that's definitely not happening on the street but one of the criteria was weird because it looks if the engine and fuel temperature is larger than 50 degrees Celsius it looks at the atmospheric pressure and if it's lower than 750 meter that must be satisfied and if all of these conditions are satisfied it will move back to the main model that does the proper exhaust processing and one thing was really weird there were seven curves not all of them used that define an upper and a lower bound on the distance driven after a certain amount of time so this is how it looks in this assembly and not sure if you can read this but the comments are from this a2l file and they call it acoustic function I'm not sure if this has anything to do with acoustic I tried to find all the usages and there was nothing related to sound or anything, I think it's just a name for it and now if we go and take a look at these upper and lower bounds and so this is the amount of so these are three curves that are defined each of them has an upper and a lower bound and it's basically the distance that you need to have driven after a certain amount of time and if you ever fall out of one of these curves we're switching back to the alternative model that underdoses NH3 and causes the inefficiencies so this is weird and I didn't really know what this is so let's get back to something completely different which is the NEDC so we've seen this slide before the NEDC mandates you how to drive and one thing is also interesting it mandates you that I mean you want to test a cold start and what's better for a cold start than heating the car to 20°C and keep it that warm until you start that's the cold start that's the cold start as defined in the law 20°C so this is time, sorry, this is speed over time so to get distance over time we need to integrate this to sum up this and we get this graph and if we overlay what we found in the firmware we get this what we can see here is that if you drive the driving cycle correctly you will exactly be in the bounds of one of these curves and you can do this on the street you can do this everywhere as long as you satisfy the distance over time and your car is warm enough to detect this in some way and while driving you can drive this on the street but it's really dangerous because you have to follow a given speed pattern so I did this on a dyno I put my laptop in there I logged the data in real time and then displayed basically this is what it looked like in the middle you see a bar and you have to drive and keep this middle bar in the middle which means you are well within this upper and lower bound and not try to escape it and you have to do one of the other green boxes tell you that the car is still detecting this as being in this cycle and then what I did at the end I just I stayed in the cycle for a while and I logged all the data and at the end I just hit a constant speed which would eventually get me out of the conditions and this is the lock that I made the first graph you see the vehicle speed you see how I try to follow the NEDC more or less successfully and the second graph you see the distance over time you see that I stay within the bounds enforced by the firmware and you can also see in the third graph this is the actual signal at the at blue pump that it actually doses quite a lot of at blue and it calculates the amount of at blue to dose based on the model output which you see in graph 5 and 6 and by the way graph 4 is the actual NOX emitted by the engine based on their model so the the mission model then calculates the amount of dosing to happen and as we see as long as we stay within the limits enforced that match the NEDC everything is good and a lot of at blue is dosed and then at the end I drove too fast and you can see in the second graph that I crossed the upper bar blue line goes over the red line and you can see that the car immediate detects this that I'm no longer in the driving cycle and the interesting part you see here is the effect on the at blue dosing which is here it immediately stops doing the dosing and you can see in the model below there was the model still calculates that at blue should be dosed but after they have the max the model and switch to the alternative model the alternative model just outputs zeros it doesn't dose anything so this is this shows that when we're following the cycle everything is fine enough you rear is dosed and then once we leave the cycle no the there's a severe reduction in dosing and it's all based on detecting the striving cycles two more slides two more slides two more slides here we go so I have to be clear on the limitations here so I mean all of this was looking at disassembled code and so on so I could have done something wrong here so take this with a grain of salt we couldn't do NOX measurements on the dyno unfortunately and I have to stress we looked at one particular car that uses SCR processing not all the affected cars are doing that so there are some other mechanisms in the other cars and I looked at a car for the German market so at least the curves have to be different for the other markets so let's renumerate the results and this is my last slide most of the time on a regular car a non-standard treatment mode is active that is not as efficient as the real mode that is implemented we can show the code that is responsible for this this is this negative temperature limit that they look at which doesn't make any sense and always selects the alternative mode and we can see in the LOX the state selection bit is that count that the alternative model is active we can see that there is an air blue under-dosing in this state which causes the inefficient NOX conversions I mean that's what we've seen before when people put the car in the dyno we know that the efficiency checks are only enabled in the main mode and the car does exceed the limit so this shows how the alternative model is selected that does too little air blue and causes the inefficient conversion we can see that if we follow the driving cycle the minimum temperature and the distance of a time we will see that it switches to the main model that should have been active all the time and we can show the code that is responsible for that the driving cycle detection that uses the upper bound and the lower bound and we can extract the exact limits and overlay it to the NEDC data and we see that there is a match on a dyno we can see how it switches the SCR state and we can show the effect on the DEF dosing on the air blue dosing as we've seen in the slide before as soon as we switch out of the driving cycle into the street mode the dosing will get close to zero and I mean once you're back in the main model all the efficiency checks are enabled to detect bad urea or something so the efficiency checks are there but they are not active because the car is forced to run in the alternative model and these results are all in line with the Volkswagen press releases and these are basically just the details as extracted from the firmware to show you the background ok thank you wow thank you very much Daniel Felix so I'm really sorry but we have to clear the stage there is not going to be time for QA session do that down there a few people just come down they're going to grab you and ask questions unfortunately we can't do that I have to close it in exactly 4 seconds because we have to go off the stream thank you very much Felix thank you very much Daniel thank you