 Present Felix Domke with the software defined emissions a hackers review of diesel gate Hey, everyone. Thank you for coming here. I saw there are a lot of interesting talks At the same time in the other room. So thank you for coming here and listening to me about software defined emissions A hackers hackers review of diesel gate is the subtitle I'm Felix Domke. I'm I usually do embedded software mainly security Definitely not on cars and definitely not on things that have combustion thingies So I only got dragged into car software last year when my own Volkswagen car was accused of cheating And I wanted to know what exactly was going on I Had the talk last year about some of the details of the Volkswagen Charon defeat device For the details you can take a look at that talk This time I want to look more at the process of Finding or analyzing car software. I want to look at whether this process scales to more cars The first step When having a piece of software that does not always do what people think it does is well obtain a phone We image obtain a binary image of the phone where and in the case of my car I knew it was a Bosch edit EDC 17 Which is a Bosch ECU that a lot of cars use including my Volkswagen car So I didn't know anything about ECUs dumping software and so on so I asked Google Hey, what do I need to do to dump an EDC 17 and Google had a lot of answers for this But usually those were people that wanted to sell me some device Those were chip tuners that they have their own they built their own devices where you can plug in the ECU and Then it extracts the image usually by exploiting some bucks in the software But I didn't really want to buy something and it takes a lot a lot of time until I get it in my hands I wanted to start so I Was looking to do this on my own What these sites usually tell you without paying is how you wire up your any given ECU for their device so they tell you where to connect 12 volt where to Connect the can bus which is the serial communication bus that the ECU uses with their to communicate with the rest of the the car devices usually it's pretty easy so When analyzing the ECU it makes a lot of sense to reproduce the scenario on your desk and not in your car So in order to make an ECU boot all you need is ground 12 volt There's usually an ignition pin that you also have to supply 12 volt to it And then it boots so on my desk it looked something like this and And then once we have the setup we can boot the ECU we can use Python to talk to the ECU which is great and then we couldn't use socket can which is the Linux can support That's really great and We can even use micro Python if we want to have a smaller device that we can put in a car So we can talk with the ECU talking with the ECU in modern cars. There's a protocol called UDS Basically, I simplified it slightly you can ask the the ECU. Hey, I want to read memory by address you give it an address and You ask it to read four bytes in this case of that address and then it returns it to you So I thought hey, maybe I can use this to dump the software The in my case the device responds with a security access denied So I looked into what I need to do and you actually have to Do a security access command you send a command that's called a request seed you get back basically a 32-bit random number And then what you have to do is to have to process the seed sure a super secret function and then return it in a response call The question is how do we know this super secret function? There are multiple methods. We can look at the ECU software itself if the algorithm is in there to verify it We can reverse diagnostic software that uses this mechanism So for example the Volkswagen software that they use for car shops or maybe someone else already Reversed this and put it in their own tools Which may be easier to get it from in terms of third-party Diagnostic software and in the case of my Bosch ECU the super secret function was this and I basically had to add This number to it The mechanism is called pin code. It's I mean, it's not super secret Anyway, um, what once I know this Yeah, thank you. I mean it Once you do this you send back the result and hey, then you can read You can send the read command again and hey, you're getting back data. So this is great Right, we can read memory at runtime of the ECU and we can even do this while the car is operating However, it turns out that for the for the Bosch ECU you can only dump specific regions You can dump most of memory some memory areas memory are excluded But most of the interesting stuff you can read But you can't read any code. You cannot read anything in flash What we are hackers, of course So we find a way the the CPU used in these ECU's is a infinite tricor CPU and it's used at least This particular one was used in these these you I cared about and the security model for this Chip is that you can always enter specific bootloader mode and execute your own code So you can strap a few lines. They that the chip units tell you that right? They tell you with high and low which pins you have to connect to ground and three point three volt And then it enters this bootloader mode. You can upload some piece of code However, you can't read the flash because the flash is locked it when you start in bootloader mode The flash is not readable until you write a specific password to register Yeah, that was not so great So I looked into what else I could do the the data sheet is very specific on how to operate this chip For example, there's this one Flash supply pin so even through the flash is in the same package as the rest of the CPU it has a dedicated supply pin and It tells you which parameters not to exceed to ensure correct operation, but I really don't want the correct operation, which is in this case Preventing me from dumping the flash, right? So what can we do? Um, we can violate the requirements the requirement is three point three volts Let's see what happens outside of that range and turns out well Down to a certain voltage level roughly one dot six volt everything just works as normal. Oh, that doesn't help us And below that voltage the device hangs in the bootloader. So that doesn't help us either the interesting parts happened when you are at the very specific voltage level and this is a little bit Unscientific because it's really just a voltage level. I tried and then Most of the time the device comes up and the flesh is protected and then the remaining times the device comes up and hangs in the bootloader But one in ten times something interesting happened the device came up and the flesh was not protected So I could dump it out So yeah having the image now in my hands I could start in actually reversing the defeat device and What I found was I don't want to duplicate a lot of what I talked about in the last talk I found a function called acoustic function or acoustic function in German It's a function that senses vehicle speed the duration of the engine operation and some other things and then controls Emission-related functionality or in short you can say that this is the test cycle detection that enables the defeat device and I verified it to exist on my Charon device by driving through the test cycle and logging data And I during the last year I verified that it's actually the same defeat device more or less That exists on a lot of other folks one cars all this folks parking defeat devices that we talked about for the Euro 5 cars They they use more or less the same acoustic function Basically to remind you that there are a few curves stored in the software that look like this. This is the This is the the The NEDC. This is the test cycle you have to drive a car through they Exactly define how fast you have to drive for a given time and seconds So it's speed over time if we draw this as distance over time. It looks like this. So this is the distance you got well, you're not really moving the car because you are doing this in a lap on a Pneumometer, but um what the car thinks it has moved to and if we overlay this with the curves We found a software. There's a perfect match. So this is the way how they describe the test cycle So this was for the for my Charon So I looked into what well what do the other cars do especially what do the cars in North America do because they are not using the NEDC and I found something interesting or some someone sent an interesting document to me that Was this it was an emissions service action it it basically describes how there was a Recall for some vehicles that required a software update in the shop so this is basically the document that informs the car shop what they have to do and It had something very interesting it by the way, this was in December 2014 So this was way before the whole diesel gate was public But this was already while the EPA was already talking with folks one already demanding explanations all that Investigation was already proceeding folks bank knew about this that people Figured out about the defeat device and it had something very interesting And that's that in addition the vehicles engine management software had been improved to assure the vehicles tailpipe emissions are optimized Operating efficiently that sounds really fishy to me So I was curious what exactly did they change in the software update and luckily, they tell you the old and the new software versions and you can then go and look them up on a firmware DVD that you can download on the Fox bar inside and it turned out yeah It's an ECU software similar to the the Bosch ECU software looked at before so there's an acoustic function again there and The curve stored there. Yeah, they match the US test cycles right this is one of them There are many more test cycles in the US So there's another curve that matches this and this is a core curve stored in the software And this is the corresponding test cycle and there are a lot of them Yeah But I noticed something really interesting and some of the curves They were much wider open than the other ones. So for example this one There's a really a non-zero probability that if you just have your morning commute true I don't know some streets or something that you accidentally match this driving cycle Every time you start driving in the morning So the car would every time you drive this think it is in test cycle mode and would operate with the Optimized in the optimized emission mode and apparently this caused problems and What I saw what Volkswagen added in the software That was part of this recall. What's this function? So this is from it from a disassembly In in Zutacill. This is this so they started looking at the steering wheel angle And if they figured out that you move the steering wheel angle then they ignore ignored the curves The the more open curves. So the idea is yeah, if you move the steering wheel, you're definitely not in a test cycle So at that point we do not try to operate in this optimized Emission optimized mode and it's a little bit of speculation But it matches up pretty well with all the facts that I read is that Because those cars operated in the test cycle mode too often and that eventually caused the the particulate filters to clock and Their solution for Volkswagen and again This was while they were already investigated by the EPA was to add this steering wheel angle detection For more details. I work with this with the NDR and they produce the feature on that So that there are some more details. So this is Volkswagen, but right, but there are more cars and if we look at this This is a meta study based on Something that the the Ministry of Transport that they tested a lot of diesel cars and what they found was this This is actually a representation by the ICCT So the orange line is the emission limit, right? And the bars have an upper and the lower end and the lower end is How much emissions the cars have this is just for nitrogen oxides for NOx emissions What they had in the lab when you're driving the test cycle So and you can see all of these cars managed to stay under the orange line So they get their certification But when driving them on a real street, they produce the emissions corresponding to the upper end of that bar Which is for some cars significantly higher. It's off by a factor of Ten and more right that when you're driving the the car on a street And this is interesting because the cars they can meet the emission goals the questions Why don't they always meet the emission goals? Why do they? operate so differently in the test cycle than on the street and I Try to give you a partial answer and let's look at how a car can optimize emissions But the first thing they do and so this is a very simplified diesel engine So fresh air goes in fuel goes out and there's an exhaust pipe, right? and a lot of Nitrogen oxides a lot of NOx goes out as well, and we don't want that so we added an EGR valve which is basically a valve that causes a Fraction of the outgoing air to recirculate again through the engine and burn again and What this causes is that the flame temperature goes down and if we look at the relationship It's very simplified here, but with a lower flame temperature you get fewer NOx Concentration so you you improve emissions by lowering the flame temperature However, at the same time you're increasing the suit level or the particulate matter and Yeah, so there is this trade-off if you do too much of EGR too much too much of the exhaust gas recirculation You're getting too much suit and the other hand you're getting if you do it too few you get too much NOx so you can argue that the green area isn't really great because there's no point where both of them are great and here we see the result of a Clocked EGR valve. So if there's too much sooth it will claw a claw EGR so as the conclusion is a least cost solution It doesn't really work at higher loads. It It works at low loads and it does not require exhaust high exhaust temperatures, which is great, but excessive use of that Yeah clocks particulate filters affects the combustion the drivability goes down and there are trade-offs with this It's also not very useful for higher engine loads. For example, when you're accelerating you have to disable EGR or at high speeds So a better method that was added on top of this is called selective catalytic reduction I am so Basically the idea is you have an SCR catalyst in your exhaust pipe to do more catalyst there, but let's talk about NOx and In there this happens We can simplify this somehow and say if you put ammonia Into the catalyst the NOx is converted to nitrogen and water and I don't even water is great, right? It's harmless. It's already part of of there The only issue is ammonia is this and this is not something you want the driver to refill in your car So instead this solution is we can create ammonia in the car from using from something that's less dangerous And we have the reaction there We can simplify this again and say we take urea Harnstof of do it and heat and we create ammonia and Urea or urea solution is this it's called at blue or DEF diesel exhaust fluid. It's not dangerous You can buy it you can transport it. It's relatively cheap so the idea is um we have this reaction requires ammonia in the catalyst and We put at blue into it or urea and using the heat that we have in the exhaust pipe We create the ammonia that we need to reduce the the nitrogen oxides back to nitrogen and water There's a Great property of this that some of the ammonia that's produced in the catalyst stays there and is until it's used up So there's some storage there So the the requirement for creating ammonia is heat and if you don't have heat For example, because you just started up your engine if there's still ammonia from the last use usage in your catalyst You can still use that and use that up and by the time you have used it up Maybe the heat is enough to supply more at at blue and then fill up that storage the downside is you need a pump to dose the at blue and You need lots of software to control this process and you need a heater because the at blue freezes at some point And it's an expensive solution. It adds roughly five hundred dollar to a car Which can be significant amount of money for a small car and it requires a large at blue tank For for long service intervals, so you don't have to refill it every Few thousand kilometers or something the the great thing about SCR is that it's efficient at higher loads There's a third method called LNT. I'll lean and oxtrap. It's cheaper than SCR for smaller engine It doesn't require anything. However, the bad thing is it requires frequent regeneration and which decreases fuel efficiency So it's kind of a stop-gap solution and it's not efficient for continuous high engine load For example, if you're driving on the German Autobahn at full speed then LNT is not going to help you much For the sake of this talk, let's keep in mind that EGR is exhaust gas recirculation That's the thing that operates within the engine And then we have the SCR the selective catalytic reduction that uses at blue and is after the engine We also saw that all these technologies have significant trade-offs For for an OX compliance So we can kind of see the motivation for a defeat device here because it would be the solution to all of these trade-offs You you get no downsides during regular driving, right? Because nobody can measure your emissions and while Maintaining conformance because during a test cycle you have perfect emissions. So that kind of explains why there are defeat devices Okay, let's get back to the bigger picture and see What other cars do? So this is an Opel car. It's a Safira car. It's a euro 6 car It's a pretty modern car. It has an SCR catalyst in theory. It should have a really great It should have really great low emissions, especially at higher speeds because that's where SCR is good at But quite surprisingly it doesn't if we look again at this report We can see that this Zafira exceeds the limit by up to 12x compared to the euro 6 Limit and this is especially interesting because there's this Opel advertisement where they at verges their diesel technology applying to the insignia and the Zafira and They say a lot of diesel fun without regrets the new diesel generation of Opel achieves best emission values and gasoline levels Yeah, after they got sued for this they had to change it slightly and they had to add this to the sentence Okay, so during this testing they had this 12x emission limits or For example, one particular test was to drive the test cycle in the same way But at a different temperature at 10 degrees Celsius and the car exceeded the values by a factor of six even through the car Would be operated in the very same ways was just that the ambient air temperature was 10 degrees Celsius instead of 25 so They asked Opel why this was the case during their investigation and Opel responded saying that yeah the e-gear and the SCR injection They work to the full extent in the temperature range of 20 to 30 degrees Celsius It's what they call normal use Okay, so our question was Is it really just the temperature window? So we got a car and investigate it The ECU in that car is a General Motors ECU that's developed in-house Opel is a GM daughter It uses an automotive power PC. Yeah power PC that Uses Somehow obscure the variable length Instruction extension to power PC. So How do we start again? We need a firmware image, right? So Let's ask the internet. How do I dump this ECU? Luckily someone in some chip tuning forum already uploaded their stock ECU, which means there what they dumped from their ECU using some chip tuning tool and it's not the same ECU But it's very similar and I hope they shared some code so I can analyze the software maybe find a way to dump it Yeah, so the dump was made with this Tool and the tool did not let me export the binary and the tool is free However to use any of the features in that software You have to buy their expensive hardware that then connects the car which I didn't want So this is this is how their software looks like I loaded the image I found on the internet and I couldn't save it or anything without having the The device attached. However, I can just use a memory debugger and just dump it from the address space And that gave me the first one way image to start with I put in a disassembler And I found the UDS function and yet implements read memory by address. So that's good Most of the the RAM was readable without a security challenge. That is good So I didn't even need a security challenge to read RAM However, the the flash it is readable, but only with the security challenge. So let's take a look at the security challenge Maybe it's as simple as the Bosch one So their way of doing this is they store a 16-bit input and output value in the firmware and it's different for every device and they don't store the algorithm to compute the output from the input instead they just store the pair and Well, it's just 60 mid-right. So that's proof force it. Yeah, the issue is you can only try every 15 seconds. So it's kind of lame The question is how do the GM tools the original factory tools get access to that luckily chip tuners already reverse that and then obfuscated it into their own tools, but that can be de-obfuscated and eventually it's just a little bit amount of bit shifting and so that was easy to fix and Also the the GM repair manuals tell you how to wire up the ECU they tell you where to put 12 volts the can bus again and ground and the ignition pin and With all of that in place I can do the security challenge. I can now Read all of flash memory and read the four megabytes of power PC code which mostly consists of mathematical functions There are no strings or anything. It's really hard to find what a function does what There are thousands of variables. It's really hard to find what they mean, right? So I need to know some entry points some known data values and then that I could refer to One thing I could find our real-world constants. For example, there's the density of diesel fuel stored which Allows me to understand that this is something related to fuel an amount of fuel Or more useful are the OBD to call so There are some standardized things you can ask an ECU that includes engine RPM vehicle speed and things like that And I would find that table in the firmware and then I had a first start of things like RPM speed and so on that was a good start It's not much more than what you can see here. So there's a lot of stuff not included in these The next thing I did was I drove the car for a few weeks and I let a device attached I would constantly lock all memory using the read by address thing and Every few minutes I would get one memory dump. Basically. It's a few hundred kilobytes of RAM and then I put this into my disassembly and that allowed me to understand more of what the individual variables do there and I found some interesting things so the first thing or one of the first things I found was basically something that resembles this so there was something that look at the ambient temperature and This basically checks for range, right? And this was for controlling the SCR systems and it's interesting to know that the NEDC requires the temperature between 20 and 30 degrees Celsius and this is right centered around this when they check from 17.5 to 33 degrees Celsius But this was of course nothing new. I found something similar however Another temperature check and as you can see it's written in a different way it effectively keeps the same thing But it's a separate piece of code and this time it was for the EGR system. So we have these two exhaust treatment or Optimization mechanisms they are EGR and SCR and they don't share code. They have their own temperature window so we found the temperature window which was known to exist the question was is there more and One thing we found was this it's basically reading the vehicle speed and comparing to a fixed number and it turned out it's Something like this. So it would would check the vehicle speed and if it's above 145 kilometers per hour it would set a flag and an under 140 it would clear Keep in mind that the NEDC maximum speed is 120 kilometers per hour. So during a test cycle that that would never happen So let's see if this what we found in software if this this really translates to something the car does in the real world And it's getting slightly technical here. I apologize, but we need to lock some Variables and a useful value to to know is How much NOx is there after the engine and after the SCR catalyst and luckily there are two NOx sensors in the car? One before and one after the catalyst and they give you basically the NOx concentration and PPM So we locked that and we also locked the signal of how much at blue is dosed into the system And we locked the the catalyst temperature and one thing to keep in mind is that there's also this amount of Ammonia that's stored in the catalyst. We don't have this as a value, but just keep this in mind and this is How we've driven the car the blue line is the vehicle speed You can see that it goes from zero to 150 kilometers power and the critical point here is the 145 kilometers per hour that we found in the firmware the The green one is the the catalyst temperature which we see between Ambient temp ambient level and then up to 380 degree the critical point here is 200 degree Celsius where this Eurya to ammonia process starts to work We locked something that is the SCR strategy So it turns out there are multiple ways how the ECU computes how much at blue to dose and I call them strategy so zero means off no at loose dose one means the regular way that keeps into account the storage mechanism and then two is a special reduced way and then also we locked the actual dosing value and then we also had the sensor data from The between the engine the catalyst and between the catalyst and the exhaust and So the first thing that happens or Actually, nothing happens until the point where we reach 200 degrees of Celsius at the catalyst You can so until that point that as I said the required temperature is not Does not allow at blue dosing and then it starts dosing quite a few amount of at blue But then when we cross the 145 kilometer power the SCR strategy Changes and no further at blue is dosed until Basically this point and this point is exactly 120 seconds after we go lower than 140 kilometers per hour So this matches what we found in the software, right? This was what we found in the software So we can see that this was actually true. This is a real behavior of the ECU and To look at the effect of this you have to Check the basically the difference between the blue and the green line in the lower diagram between Basically the amount of NOx that is removed by the SCR catalyst And you can see during the regular operation quite a lot of NOx is removed the the blue line goes up because we're driving faster and faster and the green line goes down almost to zero and This this works for quite a while and it even keeps working a while Until the the catalyst runs out of ammonia and then it would need more at blue to operate But because we are in the reduced mode. It does not put any more at blue into the system So the SCR basically stops working and the the emission levels reach the engine emission So no further that the SCR system does not work in this red area And here we see this again. So here are the sequences of active dosing here We see where the catalyst temperatures too low for dosing We see the regular operation. We see what Where it's still working because there's still ammonia stored and then until we run out of ammonia and no refill happens until exactly 120 seconds after Going below 140 kilometers an hour So our conclusion after this is that the SCR is programmed to stop working at 145 kilometers per hour the efficiency goes to zero Although offered a hand-waving explanation one of the press release why this is necessary They argued with some physical details and we presented these physical details for some experts professors that work with combustion the engines for a long time and Most of them disagree with these explanations. I mean, yeah, but more importantly other cars including my Volkswagen Charon device, which is a Euro 5 car. So one generation Older and it's known to be have a defeat device and it performs significantly better than this car Okay, we continue to look we found something Here that is a that looks like this it takes them that there's a barometric pressure sensor that Senses the pressure of the air and compares it with a value and if we look at how pressure is related to hate we see that The what they check with is 91.5 kilopascal and which corresponds to 850 meter and Apparently Europe's highest test centers at 800 meter, which may be a coincidence or not But above that point they reduce the SCR system as well Now the interesting thing is yeah barometric pressure is something very important to know for an ECU There's a good reason to have the sensor for all of the combustion process You need to know how much air there is so for EGR. It makes a lot of sense to have this But for SCR, which is the system after the engine no combustion is happening There isn't the dust we are not aware of any effect that the outside air pressure has on the SCR system and also other SCI equipped cars don't have this mechanism, so Yeah, it for us. It does not make physically. It doesn't seem to be physically required So far we looked at SCR. Let's look at EGR What we saw was when we drove the car during the test cycle So we put it in a lab and drive the test cycle we consistently saw much higher EGR values much higher And then compared to driving on a street compared to all kind of scenarios that we drove on the on the street So higher EGR value here means that the EGR valve that I showed you earlier is more open more exhaust gas recirculates to the engine it causes lower NOX emissions before the SCR catalyst and We really were curious. Why did the car behave so differently when running on a street? Then running in a test cycle and we already took into account temperature. So the temperature was not the issue anymore and Thankfully the car when it computes the reason for reducing each year it stores a reason in some variable that we can look and It looks like this there is a number of things that can happen that causes the ECU to switch to some low EGR mode and a few of them make sense for example if something is broken if fall flex or sad or if the I Don't know the coolant temperatures out of range it makes sense to just keep the device running at all cost but When we and when none of these reasons apply the the value stored is two and two basically means that the full EGR operation is Used so it's basically the NOX optimized mode With the fused emissions and then we looked at some real-world driving You can see this in the background this the vehicle speed is in the background and we saw that and The red graph shows you the reason to go to this Limited EGR mode and when we saw is that most of the time the reason is 13 and Only a few times it's two which means that it's not limited and Looking into this more details. We could see it sometimes drops back to two to the unlimited mode to the optimized Emission optimized mode, but any acceleration or almost any acceleration switches back to 13 and then it stays there for a long time and 13 if we look it up is that what I call load limit And then interestingly if we run it through the NEDC we never saw 13 So the engine stays in mode to all the same all the time and 16 just means that the engine is off But we never see 13. So this explains why the EGR values were so different in a test cycle So let's look into this load limit function that we found. It's basically defined by Curves by five curves for every gear. There's a curve or four Bucket of gears It's basically that they look up RPM They get a value for that curve and if you exceed that value They switch to the reduced EGR mode What they compare the threshold with is the the amount of fuel injected per cylinder per revolution But you can also say this is torque just with a constant factor. So And then once you are outside of one of these curves it switches to the Non-optimized mode where it emits a lot more Emissions and then you have to go back into the green area to switch back to the optimized mode So let's see what this means in practice. So here we have a car, right and the traffic light is red so the car stops and then The traffic light goes green and the car accelerates and accelerates and accelerates It gets faster and faster and Then it's at the highest speed here and drives for a while and this is a typical city cycle, right? This is there too. Yeah, that's how you drive in a city and then the next traffic light turns red and the car Yeah, brakes and stops in front of the traffic light Let's take a look at this again with one more variable and the RPM So we can see that when the car starts moving The RPM goes up and Then at some point there's a drop in RPM and this is because it's a manual transmission and the driver switched to the next gear Now it switched to again the next gear and just cause it the RPM to drop but the the speed to remain almost constant and it drives for a long time in the same gear and then The traffic light goes red the driver presses the clutch the engine goes back to idle state There's no connection anymore to the wheels between the engines and the wheels and the car gets slower Okay, one more variable. It's the last one. I promise. Um, it is torque The the engine power in kilowatt or something is not just a function of RPM It's a function of RPM and torque. So RPM and torque together are very useful to characterize engine behavior and a very good way to do this is to have a graph where we put RPM on the one axis and torque we put on the the other axis and then we draw this in two dimensions And so we we get this basically. This is the operating points. We go through when driving the cycle we saw Um So the green dot here indicates where we are and So we restart the car the car accelerates. Sorry the car idles for a while So the green dot stays there and it idles at around 800 rpm almost no torque because there's nothing to move And then the driver accelerates and the torque goes up the RPM goes up slow more slowly and then at some point the driver presses the clutch Which disconnects the engine that the torque goes down the RPM adjusts to the speed of the next gear and Then the driver releases the clutch and now the engine again has to move the car So the torque goes up until reaching the the highest RPM value and then the drive against which is to the next gear So the whole thing repeats and then while the car is driving the majority of this the cycle The engine spends in this one operating point where currently at 1,800 rpm or something and 80 Newton meter or so torque and then at some point the driver presses the clutch The engine goes back to idle and and stays there basically. So this is how you read this diagram and Now what we found in the firmware was that overlaid basically on this representation We We see a mask or a limit if we go over this curve Those are the same same curves that I showed you earlier just late on top of this if we go over this If we go over this curve then we switch to the To the worse emission mode right we switch to the mode where the EGR Value is limited so we can see in our driving that this happens Basically at this point right that the point where the driver accelerates above a certain point that causes it to go over the load limit and the engine basically Switches or significantly reduces EGR and and that's fine because EGR doesn't work when you need a lot of engine power So it makes sense at that that point and what we would think is that it switches back once we leave this Load envelope once we go below the limit again once we are inside the limit We would expect the ECU to switch back to the full EGR operation but what we see instead is that this does not happen and The reason is that you don't have to go under the the maximum the load limit You have to go into this green area You have to go back to idling at a very low rpm to switch back to the full EGR mode and this Only happens at the very end when the driving side is almost done when the driver presses the clutch and let the engine idle so Especially this this long sequence where the drive the the car was driving at the same speed We were technically in The within the load limit where we're not exceeding the load limit But because we previously exceeded the load limit and it doesn't matter for how long you exceeded it And we did not go to the green area before we were still in this Low EGR high emission mode even though we're still within the load limit imposed by the software So let's take a look at how often this actually happens in real-world data So here's us driving through a city and we can see we constantly exceed these load limits And this is driving on the Autobahn and yeah, we constantly exceed those but They look interesting. They look as if they had been designed according to something, right? They they have this specific form and it's not just Yeah, I don't know And it turns out if you do something really strange you can stay within these limits So we tried that and we managed to stay within the limit by doing something and we it was reproducible We could do this off a lot of times and we would always stay in this limit and the answer is If you're right drive the test cycle, you're staying in this limit So yeah, these curves basically defined They closely correlate to the limits that you need to pass the NEDC Okay, to be clear it is fully acceptable that the EGR rate is reduced when for a higher engine load It's natural you have to do this for example when you accelerate the EGR rate will Decrease up to zero probably when you do it when you're running at high speeds all of that is great So this this method of having a load limit well You can argue if really having the load limit exactly where the NEDC is makes sense But having a load limit is okay, right? However, what we think is not okay is that if you only exceeded the limit once you would stay in this High emissions mode for potentially a long time until you get back to low-speed idle the next time And we think that is the problem We so so far this was all based on what we saw in the software So let's see if this translates to something that happens in reality. So to repro this we Let a car on drive at constantly or we let it idle then we accelerated to 2000 rpm We let it drive there for a while and then we quickly exceeded the load limit by going to 3000 and then going back and then after doing that we would again stay at 2000 rpm so it looks like this and we would naturally expect the engine To operate in the same way on the left and on the right side because really the engine is doing the same thing there It's the same torque level. It's the same rpm. Everything is the same. So we would expect the same emissions, right? And it turns out it isn't and this is a slightly convoluted diagram. So if you look at the green and red Bars in the middle that you can see what happens before and after Exceeding the limit for just once and in the middle you can see the eGR position the eGR valve position and you can see that We get pretty high values between Six well, maybe 65 percent or something Before exceeding the load limit once and after we exceeded it once even through the engine again is operating in the same exact Operating point. We see much lower eGR valve positions around 50% or something and if we look at the bottom We see what the the engine NOx emissions and we see that they are significantly higher on the right side And they are on the left side. So this for me This does not sound like this is truly optimized for emissions because the upper the engine is doing the same thing The in both cases the emissions should be low so going back to this quote That it works the the eGR and SCR injection work and to the full extent in the temperature range of 20 to 30 degrees Celsius Okay, but what about the eGR load limit and what about the the barometric pressure limit for SCR? And what about the SCR speed limit that would not be to the full extent, right? And the Opel answer is really interesting. Of course, they denied doing a test cycle detection they say they don't do that and what they said is Yeah, so when asked whether they lied to the KBA when saying that it works to the full extent They said well the statement fully was really related to the NEDC test schedule, right? Which yeah, um, okay It went on and the further the so the Opel CEO had to say this He said the recent accusations based on the findings of hacker Mr. Felix Domka. Hey, that's me Are misleading over simplifications and misinterpretations of the Complicated inter-relationships of a modern emission control system of a diesel engine Emission control devices are highly sophisticated integrated systems, which cannot be broken down into isolated parameters Especially not by a hacker, right? Yeah, what's kind of funny There was another funny thing. I'm sorry. I only have a German quote and I didn't want to translate it But when when Opel basically they repeatedly say they don't have a cycle detection Right, and they say it's not a cycle detection because If you use the car on the street in the same way as you would do them during the test cycle The car would behave in the same way. So it's not it, right? and Okay But what is with Volkswagen, right? They have the same thing if you drive the NEDC on the streets The car will go to test mode. They have the same thing. It's so Okay, I don't see how this does not represent a cycle detection Okay, that's what a lot of bad things to say about Opel, but on the bright side They also said that they will even do all of that was incorrect what we found, right? They said we will further improve the efficiency of emissions after treatment or as your diesel engines and so on in terms of Or as far as the loss of physics allow and this includes a voluntary service action And this basically means a software update for your car For the cars that are already on the road and a start in June So that it's great. They actually improving something questions in which year because this statement is from May 2016 and It's not out yet but Opel actually provided a new software already in July and I think they already worked on this for quite a while and In July 16 the German KBA the Kraftfahrt Bundesamt the Federal Motor Transport Authority They they are pretty nice actually and that they do know about what they do They are a bit limited by the the resources they have and by the manpower they have but they know about cars and they Know how to do these investigations I mean they're a little bit bound But what they should do and what they should not do but they asked me to review a new ECU software That was given to them by Opel for the Safira in question and it is an insignia Which had a similar ECU and I looked at that software I dumped the firmware and I looked at basically all the code sequences that I looked at before and I was positively surprised because they Removed that they addressed each of all concerns of them Within the physical limitations, of course, so they improved the temperature we know and everything so there was a significant Improvement they were able to improve the software and they let the DUH which is the German environmental aid that They used a PEMS system PMS is a portable emission measurement system It's something you put on the exhaust pipe on your car and then you can measure the exhaust during a real-world driving and Opel you gave them a car with the new ECU software otherwise the car was identical to the old software and The results are this right so on the left side You see the old software that has all these things that we criticized and on the right side You see the same car with a new ECU software and it's significantly better It's only slightly above that the limit right, but it's much better than before and to put this in in relation Before they were on the list pretty bad. So this is sorted by worse to best So they were in the well upper half at least and now they are almost one of the best cars Just by switching the ECU software And I mean that this is great news right they actually improved Their cars and let's just hope they get this out to the cars soon Let's just hope it doesn't have side effects and something, but I'm sure Opel knows how to test for these So going back to these yeah, we worked on the Opel thing I think the Opel case once they once they actually Upgrade the cars and once the cars really show these great values that the the preliminary software showed I I think we can close the Opel case, but there's a lot of other cars still to look at and really I mean The effort to do this does not scale to so many cars So we need to do something more fundamentally to improve the situation What I found out and is that digital control systems They are black boxes the manufacturers have designed them to be black boxes They they even boast to you that they are 7,000 parameter in there and no hacker can understand this and it's a very sophisticated problem They are designed to be black box and this is not just true for Opel This is true for all car manufacturers. Nobody wants anyone to look into their ECUs and And people seem to be okay with that like they they think oh, this is so complicated There are so many German engineers working on this problem. They must have found a great solution So we are trusting these black boxes We are not able to review the black boxes that we put into our cars and we have to trust the manufacturer to do the right thing and currently The investigation to to do this without assistance from the manufacturer. It does not scale so Yeah, we can do it, but I mean the manufacturers can put more security on their ECUs Yeah, it's probably can be broken, but it takes a lot more time. So it simply does not scale sufficiently The issue is black boxes are really powerful right black boxes can hurt people with for some excessive missions They can kill people if we think think about autonomous cars that do mistakes So what we do need I think is more transparency a System that can kill people Needs to be reviewable by the people. I think this is a very important thing So and yeah, so to have a system that can kill people to have it reviewable by the people We need to do things for some we need we want access to source code for reviews It doesn't necessarily mean we want open source But we don't ask all the car manufacturers to open source all the software that's not what I'm talking about what we need is Think about how Microsoft is sharing source code of windows with universities or other countries We need experts to look at the source code And we want control software that is reviewable by design that has a lot of documentation that has Good comments that is human readable code. I don't want to see a disassembly I want to see the source the med lab or whatever they are using to define the functionality source and read that and I want to understand why did they choose that That curve of that map in this way. What was the design criteria that that needs to be reviewed and We need transparency for control software decisions, which means that The if a car operates in a certain way If I'm driving that car, I want to choose that I can lock what the car is doing For example by putting in I don't know a USB stick or something if it's my car And then the car will lock all the data to that That is in the end we that in the end allows me to reconstruct any decision that the software does I think this is required to have the necessary transparency That allow that allows us to unblock box these devices All right Thank you very much Okay, I actually finished five minutes early. I didn't think this would happen. So I'm so surprised I'm surprised you are on time. You have five minutes left. Wow. What do I do with five minutes? Around the stage. Yeah, we can maybe people have some questions. Are you think so? Well, let's ask the internet Well the internet ready. Yes first question What do you think is the responsibility of Bosch as a supplier for having the software and hardware used for this? Yeah, that's a good. So the question was what's the responsibility for Bosch who built the software right for folks bang It's a good question and I have to be careful in what I answer right My personal opinion and let's take this aside from folks bang and Bosch is that if you build software that you know Is used to be illegally it should it must be your responsibility to not do that and I'm not sure if this is something that Is legally enforceable, but it should be something that's enforceable Ethically or for all of us programmers right that we don't build software that is designed to break the law We quickly hop over to microphone one, please. Hi there. Thank you for a wonderful talk I was just wondering if you're aware of Some cases of Volkswagen cars in Australia, which were suffering from sudden and rapid power loss This was happening about five years ago, and there was a case where a Volkswagen suffered rapid power loss on a motorway The driver was Mrs. Melissa Ryan and she was rendered by a truck and killed so when you say that These things can cause death. Were you are you aware that any sort of Volkswagen software has been leading to Power loss in the vehicles and affecting performance on the road Now, I don't know whether Australian Australian driving conditions are different to European driving conditions And how that might affect that have you done any tests that might indicate that could be happening in normal driving Yeah, so the question was whether I'm aware of I think an Australian incident right where There there was can I yeah, there were many Reported cases one of them was fatal, but there were many reported cases of that office sudden power loss. Is that right sudden and rapid? Power loss in the engine. Yeah of the engine. I'm not aware of these incidents. I what I do know and is that The the personal safety is the number one design criteria for ECU's that does not mean that they are perfect Of course that that could mean that there are bugs that there could be malfunctions I don't know about this, but at least it's the first design principle To provide the safety for the people driving the car, which I think is a good thing, right? It's not the profit or anything or at least we can hope so. I'm not aware of this particular incident and So I can't really say anything more about this It would be great if I are you aware of any additional details that were found in the investigation So please send them to me Volkswagen was claiming that some this was a gearbox problem on automatic cars But then it started happening on manual cars as well So that excuse went out the window these problems that most of them are very complex So they probably involve more than just the engine ECU So that they're very but it's a good example of where we need to Understand exactly what is happening and where we may not want to rely on Volkswagen or any other manufacturer alone to Assist in yeah in figuring out what happens, right? We need more transparency there so that we can have definitely neutral Accident investigations. This was a long question and the really detailed answer. Sorry. I will be short Felix, that's your applause