 Okay, I think it's time to start so good morning everybody and Thanks for stopping by so We have a session here just to make sure you're in the right session where the session is security connected car So if we look at what's happened with the car over the last few decades in terms of technology you see In the 90s you had electronics then telematics more intelligence around Your fuel economy and how Soon you have to refuel and you have the infotainment system, which is basically the display you have between The front seats where you can have a map and you can have all sound typically like music So that brought a lot of intelligence in it obviously and then towards the end of the 2000 around 2010 It became standard that cars were connected to 3d networks and the internet Right now there's a lot of focus on assisted driving developing these type of technologies so safety features in case you're close to the edge of the road or You're in too close to a car in front of you or Things that could could be dangerous And then going forward, I'm sure you've also seen a lot of work on autonomous cars it's especially from various Silicon Valley companies like Apple and Google and also Yuber is doing a lot of work there and obviously also OEMs or the car manufacturers are working hard on Autonomous driving so obviously there's quite a bit of work until that is safe. So but when that hits the road, it will be a tremendous amount of Software that will be in the car in order to Make this work So the points I guess not surprising but this is more of a proof that there is more and more software that ends up in the car and We'll have a look at some of the implications During the session. I just first a few words about me. So my name is Einstein Stenberg right now I live in the US and they typically call me Einstein and That's that's fine So it's a good good way to kick things off and I've worked in software in security and Management software for about seven years have a background from computer science and cryptography Right now I work on a Project called Mender So we're developing an over-the-air updater for Linux. So it's open source and there are actually two license And yeah, we have a booth upstairs also if you want to find out more and there's my email So oops, it's a bit high volume. Okay So yeah, if you have any feedback on the presentations I've been giving this a couple of times already and improved it because of feedback So it's something you don't like or you like. Yeah, please let me know so So we can make it better So for the rest of the session, this is how it's structured. So We'll first look at so obviously you see There's more and more software and then why is this happening? What kind of opportunities are the car manufacturers? Also known as the OEMs looking for what are they doing it? We'll look at the more on the security side next We'll look at one specific attack where a car got hacked It's a famous story, but Not a lot of people know the details of what happened So we'll try to cover at least some of these details and Then we'll look at more in the patching problem and how we can do that Because that's closely tied to the security here. Okay makes sense So one of the main drivers for Having more software in the in the car is is obviously that the car manufacturers see that they can make more money That way that's usually how it goes So for example, Tesla is considered one of the most innovative car companies And they have this add-on feature that You can buy for a couple of thousand dollars Yeah, so so this specific sample is about the autopilot so that's that's assisted driving as we mentioned in the beginning and They can do that by by delivering software After the car car is sold Yeah, and there's a couple of reports there also showing that Predicts a lot of new Revenue there's an estimate of 27 billion From Navigant here But the other side of it is also the cost saving you can achieve so if you look at the IVI which Yeah, so like in technology you have the same name for several things. So IVI is the infotainment system So the maps music system you have in your car. And this is typically how The stack looks so you have hardware board support operating system. Maybe a OTA update or some middleware application and then the HMI on the top and The key thing to take away from here in terms of cost saving is that you can see the The cost of maintaining the lower layers are Very high. So about 60% of the cost come from these lower layers Where the differentiation is actually on the top layers which might be yeah around 30% of the cost so The reason is that when you have so what the end user or the customer sees is the apps and the interaction with the system They don't see the drivers underneath as long as they work than that will Satisfy them So that's why you are today seeing More support on on having open source in this lower layers How many of you have heard about a GL or Couple of people So there's a That's one example. There are several initiatives. So a GL is a Automotive grade Linux. So it's basically a Linux distribution that focuses on on automotive and IVI systems in particular So they're trying to build and bring open source into this lower layers of the of the car Car IVI system So which makes a lot of sense to use open source there because you don't have any Differentiation here or at least very little and the cost is Quite high if you're going to build all this yourself Yeah, so a bit on OTA updates Obviously, I work a bit in this but The reason I or the main reason I think this is important is that when you see that all this complexity comes to the car in terms of more and more software then obviously you need more frequent improvements as well because I Don't know how many of you develops software primarily Okay, majority. Okay, so you You know that you don't release just version 1.0 and there's no bugs Typically doesn't happen. I've also developed a bit of software and at least my code is not like that. So typically there is a Connection between them now like the number of lines of code and the number of bugs there is some kind of ratio You can you can work out there depending on the project So and also in the connected cars today you could See that about one three one third of their recalls can be Fixed by just software updates instead of just driving the car back to the Manufacturer which is obviously quite expensive and in terms of security fixes It's also very important that the end customer does it, but maybe they don't And yeah, so this can result in quite a bit of savings. There are some estimates on that 35 billion and then this example will have a look at the Fiat Chrysler hack Happen last summer So it was a couple of Security researchers that managed to hack it. How many of you heard about that one the Fiat Chrysler? Okay, about one third So I'm not trying to pick on Fiat Chrysler. There are problems everywhere, but it's just as an example of How things can happen and why it happens so we'll have a look at that so this Is the end result of what happened to them? So there were two security researcher Charlie Miller and Chris Valasek. I hope I pronounce it correctly, but They presented that black hat USA in 2015 And they showed how you could gain full control of the vehicle so it was a bit interesting because this The person that you barely can see inside the car he He was actually a journalist They wanted to make a story about this because they they told him okay, so we managed to hack this car and Okay, so I have to run a story on this and The way it worked was that he had to sit in the car while they hacked it So he was kind of screaming while this this was filmed and he was driving down highway or Autobahn and they could take over the car and just like steer it and Apply the brakes, so it seemed like quite a scary x experience So when this happened, they also didn't have a way to fix it remotely, so they had to recall 1.4 million cars So yeah, this this happens today, so Admittedly, it's it wasn't that easy hack So these guys I think they spent two years to figure out how to do this. They had to go through several systems But it's definitely possible And it's probably possible in a lot more cars than just this one So if you look at what happened in particular, so if you look at that the infotainment system or head unit There is a Wi-Fi hotspot in I guess several New cars now and the reason is that they sell that as a service on top of Yeah, you purchase the car and then you can subscribe to the service you get Wi-Fi in your car for about $40 or 40 euros a month The reason so the value proposition there to the customers is that so you have a lot of annoying kids I don't have kids myself, but I've seen seen seen that where they can Give the kids some Wi-Fi put them in the backseat and give them a iPad or whatever entertainment unit they want So that's why they have this this Wi-Fi in the car so so that you can Yeah, use your devices there And it's password protected. So so it seems Seems pretty safe, but What happened was that the password was very easy to guess because it was based on the provisioning time Which was January 1st, 2013 000 plus minus one minute So if you had about 60 or 100 guesses you could pretty much guess the password in any of these Wi-Fi's And then there was a software vulnerability in the In the multimedia or infotainment system so once you got in through the Wi-Fi you could actually get into that Yeah display unit that you can see here So when they did that they could control that That multimedia system so they could control the music and It did also have a GPS so you can track the car that way Obviously at this stage it wasn't that interesting because you had to be very close to the car to To be in the Wi-Fi range so Probably at this point, it's not like a very serious attack Mostly annoying you can walk past cars and change their audio output But what they did next is that they reached the cellular network, so the sprint cellular network so now instead of using the Wi-Fi they use status the entry point and they could control the same things the Radio and the GPS coordinates, but now they could do it in the entire US basically, so now It became a lot more interesting to have the GPS coordinates because you could actually track a car that way so The next thing they started to look at was the what's below that? display or infotainment system so at the bottom here again, well, I don't think you can see it, but There's the engine and Unbreak systems and also like locks of the doors and they're like very critical Safety critical components that are connected via something called a controller area network So that's basically this lines that you can see is the so-called canvas So it connects about 70 electronic control units and then You need a way to see what's going on here because if there's something wrong with your car your oil your transmission Something is unsafe then it needs to be reported up to the user so that he knows about this and it can be put to Yeah to Maintenance So there needs to be like this flow information going up from the canvas, but It was designed in such a way that This is read only through a v8 50 chip as it's called Which would sit close to the? Infotainment system So this seems fine because Obviously the dangerous thing is If you could have something from the top right If or tell the canvas or put on the canvas that the brakes should be applied and so on but so now You know that that's exactly what happened But the reason was that they updated That chip with the malicious firmware update so So now they could both read and write to the canvas that was controlled by the The v8 50 ship that that this was just a read only And That had unit was able to update that firmware So the main takeaway here security wise is that it wasn't checked for authenticity The firmware so you could install anything And that made made this possible So this is basically how it all Was put together so you had a cellular breach at the top Which reached the head unit Or the infotainment system through a vulnerability and then they could put Malicious firmware update on that v8 50 chip which would allow them to both to read and write to that canvas And now you have full control of the car so again Trying to see what we can learn from this so obviously the Wi-Fi password was quite easy to guess and Easy to guess password is not a new problem. I would say but And then you have this Service that you'd access in a head unit that was vulnerable not updated so out of date software or Vulnerability staying in a system for a long time also not a new problem. I would say Then you have the firmware update didn't have the authenticity checks And then finally when all these bad things happened then the only way to fix it was to try the car physically back so it all lined up pretty Nicely or Shouldn't say that I guess but yeah No, so they don't have to be in the car because they used So the cellular network is a 3g network that's connected to the the infotainment system so they used either the Cellular network or in or the Wi-Fi that the car had Yeah That's one way to fix it don't have a network connectivity I I'm understanding that we are the future of a run-up on the baddest thing you could do out of the club, and that's what's very scary. My understanding of the CAN bus, obviously, in transmitting you need to receive light if you need to detect collisions. If you're receiving only, you don't actually need to hook up the received light between the CAN transceiver and the CPU itself. If that V850 chip was always on a board that should only receive a never-transmitter, why did they actually populate the one-and-a-half or at least particular distances that I could stop it physically from transmitting on the CAN bus? Yeah, that's a good question. I don't know the answer to it. You use Google Run. You only need to acknowledge the stuff. Officially, you just need to acknowledge the transmissions. But to have the CAN bus work, only one person needs to acknowledge. In practice, everyone does. If you have multiple things on the bus, you can't get away without having one person acknowledge. But today's system is always useful and I don't think we do. And then we do it together. All right. So, also to summarize the discussion a bit here, I think, is that the more complexity you have, the larger is the attack surface. And there's one metric, like you mentioned also, that the more software you have, the more bugs you will have. And there is a metric that there's between one to 25 bugs for every thousand lines of code. So, this is a very wide range, obviously, depending on, I guess, who develops it and what kind of system it is and what kind of QA processes you have and regulations, regulation requirements. But the point is that it's not possible to have zero vulnerabilities in software and we shouldn't assume that. We need to assume that there are vulnerabilities and handle it from there. Of course, you should try to reduce the number, but they will still be there. So, one thing we could do is to rely on well maintained software and keep it up to date. And there's, I'm not sure if you've been involved in these discussions, but there's a lot of discussions about open source versus proprietary, what is the most secure and so on. But I don't think those discussions are very fruitful. There are other things that are more important than whether or not it's open source and that's more about the maintenance of the software itself. So, also if you build less software in-house and use more well maintained software then you will probably be in a better spot. So, of course, there's quite a few examples on vulnerabilities that like in the open SSL hard bleed, which is quite famous by now, people thought it was very well maintained. So, they thought like open SSL, everybody uses it, half of the internet uses it. So, it has to be well maintained, but it turned out it was like two maintainers doing a part-time job on that project. So, obviously it's much better funded now and there were some big companies that put some money into fund this project. But it turned out it wasn't that well maintained after all. And then there are some security principles that apply pretty widely also in the car case. So, you have the principle of least privilege. So, don't give components or processes more access than they need to. Well, in some cases it could be a matter of what user a process runs as, for example. And then you have separation of privilege. So, maybe you can split up software into two parts. So, one is able to access the network and the other maybe the file system and maybe they don't need access to all at once. So, it's more about compartmentalizing the system so that if one part is breached then you don't necessarily have full control of the entire system. And there's a lot of ways to do that also with virtualization and hypervisors and so on. And Kirchhoff's principle that's related to cryptography so where you should only assume that the keys that you use for encrypting or signing things are secret but the algorithms are well known by the attacker. So, the bottom line is don't invent your own crypto algorithms and try to be secure that way. But rather use well known algorithms and keep the keys very secret. So, moving on a bit to the security patching problem. So, this is a graph of this is the probability that vulnerability has an exploit and this is the number of days after the vulnerability was published. So, basically as you would expect the longer or the more amount of days to pass the higher is the probability that somebody exploited the vulnerability or made an exploit available. So, some numbers you can see is like after five to ten days there's less than 10% probability it's exploited. After 60 days it's 90%. The problem I guess you can see as well is that 110 days is the average remediation time or average time that the systems get patched. So, there's quite a long time here in general that people have to exploit the systems because the exploits are available and the systems are vulnerable for two months at least in average. So, this is a very wide survey from you have the source down there also you can look at it if you want to. But I think also embedded in connected cars as we've seen in the example as well it looks much worse because they are maybe never remediated. So, you could ask why this happens and one of the typical things about security is that you don't fix it because you can't see it so it's not a feature and then when it happens it's too late unfortunately so that might be one reason. It could be expensive or risky if you want to update manually for example or to integrate the updater might be quite a bit of work depending on how you do these updates. And always with deploying software there's a risk of downtime or breaking the production environment which is like the big scare when you're doing things like this. Maybe there's politics reason and if you have systems you can also think about how frequently you patch them or if you have a way to do it. You have any other reasons why you think it doesn't why security patching is not happening that frequently in your or other environments if you rather speak about other environments. Yeah, exactly. Yeah, it didn't mean that's all right. Yeah, it could be a security hole as well. Yeah, that's true. Thank you for a little bit. So we need to the bank one. Same with your smart phone and same with your public. Yeah, sometimes you don't want to upgrade because you are busy doing something and you have to drop down and that's great. The user support, the source support, they are always setting the functionality when you update. Yeah, so they have an incentive. Yeah, that makes sense. Yeah, and as you might, well, you probably know this is a harder problem in embedded like I said and that's one of the reasons it looks a bit worse there I think as was mentioned that expensive to have physical access like in a car case you have to drive it in for repair maybe. The power was also mentioned and this can also happen during the patching itself which needs to be handled as well. So if you're in the middle of patching the system and then you lose power and next time you do what happens. And then you have the network connectivity. So I was also mentioned here that if the updater doesn't handle that in a secure way it could be a vulnerability itself. So, yeah, so that's around the security of the network and then of course it's also can be very unrivaled and reliable. So in cars it's typically 3G connections I believe still today. And this, if you go into a tunnel you lose connectivity and then when you're out of the tunnel you probably don't want to restart from the beginning. You don't want to start from where you left off. So there's quite a few extra challenges that need to be handled. So diving a bit further this is typically how the patching is done for an embedded client. You could have more details here as well but the flow is typically that you try to detect updates somehow that can be initiated by the user also potentially or by the, when you're in for repair you could initiate it through some physical means that you're nearby and you would download it into the system, do integrity checks, authenticate it so signatures are typically the way and this was not done by the Jeep or the Jeep Cherokee. So they didn't actually verify the authenticity of the firmware update so that has it placed there as well. You might want to decrypt and encrypt the update also depending on what's inside of it if there are security fixes. Some people prefer to do it and extract and install it afterwards and then as the last step, like I mentioned, what if the update failed for some reason, maybe it lost power or you figured out that the system didn't boot afterwards you need to handle that somehow so you don't have to drive out and fix it yourself. So just a few words also on the different types of updates. I know this has been covered a bit in other sessions as well but so you can do full image updates and then you have package-based updates or tar balls or compressed files basically and then you also have Docker container updates. So how you do this also has trade-offs with respect to download size and installation time. A rollback is easier on the full image updates because you can have dual partitions and consistency is also a bit easier there but the big thing obviously is that the packages are smaller than the images and so on. So there's always these trade-offs but what I've found is that typically people prefer to have that rollback capability and they will rather take the performance trade-off in order to have that. So you avoid these bricks as much as possible. So what can go wrong? I'm sure you're familiar with that question. So these are some things that you can do when you're doing software updates to reduce the risk of it. So integrity checking, that was one part of it that checks some. It's pretty easy to implement as well. A rollback should be there. So as just a catch roll, so you don't really know all the reasons why it might fail or how it might fail but you just know, maybe you know the main reasons but there's always something new that will show up and rollback can be sort of a catch-all for all these cases to handle at least most of the unknowns as well. But then you have to think about how you do the update type as well in order to design the system to support it. So face rollout, I think, more common way to name that is campaign management. So basically what you do here is that, so this is used by all big infrastructures from a completely different industry. I can give one example which is Facebook. So the way they develop a lot of new software every day, obviously, and you might not see it. I'm not on Facebook personally, but you have all these feeds that are going on and the way they deploy their software is that they choose one region or one group of people based on some attributes. For example, they can choose everybody in New Zealand. We're going to deploy this to everybody in New Zealand first and then they will monitor the metrics, what does the network look like before and after and then after some time they will move on maybe to Australia and New Zealand and then expand like that. So you can do that geographically or you can do it in any other way maybe based on age or other attributes. And this is a general thing that you can do no matter what kind of software you're rolling out basically. I think it's campaign management is the correct word there. Okay. So just to wrap things up, some of the points for securing the software defined car use open source software where there's no differentiation, there's no reason to do anything else and especially if it's well maintained which I hope Open SSL is right now but you should check that as well. Have a way to deploy over the air updates and then also these security principles that we talked about we saw that in the hack that happened in the Fiat Chrysler that they were not followed that well either but they're often overlooked but if you do think about it while you design the system you can still avoid a lot of these problems. I believe that was it. So thank you for coming. And we had some good questions but if there are any other questions or comments please feel free. One comment. I think to update single device is quite interesting. I think the major problem within actual cars is the network and constraints of the single ECUs. So if you have a network consisting of 50 ECUs and they are constrained together because normally the protocol is quite hard and then you need a rollback but for all of them it's one single ECU update. And then it's more likely one of the major problems. So you have a distributed rollback basically. So I guess that sounds like an interesting problem. So we use full images. You'll read the fast today. Yeah, it's basically because it's more robust and we start out with making it as safe as possible and then we'll see if we'll add other types of updates afterwards. We just figured we'll start with this and maybe expand from there. But you can stop by and have a deeper look if you want. Any other questions or comments, feedback or ready for more coffee? Okay, thank you.