 So it's my first time talking at DEFGRON, so just bear with me. My talk is a bit about all the supply chain and all the areas around the car. So I saw lots of talks, talking about the vehicle itself and and like how to hack into Canberra and how to hack into through the different areas. But my expertise and where I came from is from outside the ecosystem, from the areas beside the car. So I will talk about this in my talk. So a bit about me. So this is me. I work for CYMOTIVE. It's a company partially owned by Volkswagen. What I do in my company is we do lots of stuff. We do lots of end-to-end security testing. And what I mean is like really end-to-end, like going from the vehicle itself and hacking into a vehicle but also hacking into different IT environments and trying to go through these IT environments towards the vehicle in different ways and that will show you part of the ways that I do. But then we also go into the components themselves and we mainly focus on research in order to build security products for the different companies. So I have here different concepts I built. Partially I built them, partially the company did. And it's just, this is what I do in my day job. More about me. From the age of 15 I was playing with robotics. I was a winning contest around, if you know the Trinity firefighting contest or the first day at NASA. There was a NASA competition. So I was playing around with robotics from very, from when I was very young. I participated in bug bounties inside car hacking. We won most bugs awarded last year in a private car hacking event. And most important of all, last year I was on vacation. I took my four-year-old daughter and my wife and just went on one year vacation in Southeast Asia. So that was very fun. And I came back. So it's really good. A bit of a disclaimer about what I can say and I cannot say. So because I work with all these different companies I have lots of NDAs and lots of stuff. Nothing in my presentation exposes any of my clients. And I want to really go into technical areas so I didn't put any name dropping of any client or anything. And this is very important for me to, so you will understand. Like I have lots of examples I cannot show you. So I'm trying to put the examples that I can show you over inside my presentation. So if we start a bit, we look at different connectivity evolutions. The vehicle industry is moving from a very unconnected area to a connected area and different fields which are connected in different types of areas. For instance the keys. In the past we had this very old key that didn't have any digital protection. We very fast went into getting keys that are digital that have a mobilizer and have RFID and different types of protection. They are not good. You can ask Leonard if he's here. He hacked most of them. But then they're getting better and better. But what I think is we need to remember we have lots of suppliers working with us. So each key, this has the production key, the physical key itself is building but also the digital like Siemens and Lear and Bosch and Conti are creating all these issues that you need to connect with the keys. And we are getting even more into the connectivity with the iPhone or Android. So we are trying to move the keys towards the mobile phones and this gives us more connectivity issues. We want to now have a trust with Apple and the trust with Google and the trust with different mobile providers like Verizon and we have all of this connectivity going on in the life of the key. If we look at charging, it's the same happening where we have the old bottles. This when I was in Thailand I saw lots of bottles and lots of gasoline bottles. But then we go to pumps and we also have a plug in the charge and different capabilities of the charging models. And now today we are having NFC and different charging providers that are not part of the OEM. They're a different third party supplier that you pay to them through Apple Pay or through credit cards. And then you have all this connectivity together. Again in diagnostics I go fast. We have the old diagnostics area. This is old engine diagnostics. We went into OBD. OBD diagnostics, the garage and the different service providers want to connect to your car. But then they have this device and this device is also connected to the internet. So when it wants to download data, new data, so how can I diagnose my car? So I connected through the garage's support. So the garages have support and it is a new, the right to fix my own car so I can download part of the software. So I'm getting connected over here. And now in the future cars are getting into remote diagnostics. So I have diagnostics over IP. So when I want to diagnose my car I just need to have to approve a third party company or the OEM company in order to get the access to my car that it already has 4G access and is able to diagnose my car automatically. So this is we are getting more and more connected. And we are going to go over to a way that we have different types of players inside. So we have insurance companies and we have content providers that have web apps and different applications, media. We have the repair shops. We have fleet companies that have like different fleet management solutions. And of course we have the driver and the owner that he wants to connect his phone into the car. But then we also have the V2X and V2V and we have cars talking with cars and the Bluetooth and the LeMatics. And we have this whole mess of connectivity happening in a very short time span. This is really interesting for me. And if I'm looking I did a very 2D simplified overview of what I see as a connectivity area inside the digital key. So we have this key and we have the phone. Like let's say we have the phone. But then the phone has access to a TSM enrollment from the mobile chip. The OEM doesn't own the phone and cannot enroll by itself the key. So it will have across the mobile chip OEMs. So the mobile chip OEMs are connected to the OEM cloud. The OEM cloud needs to be monitored and needs to be accessed by the OEM IT. So there's connections over there through VPN, through different other areas. The OEM IT has connections to the production plants. And we have third parties that most of the OEM companies are now using third party. I say third party but sometimes it's just like second tier car companies like Bosch or like Lear or like whatever company it is. But we have also fleet management services and different startups starting to aggregate data. And we have all of this mesh of access in order inside our companies. Furthermore we have this production plants and different stuff like also 3D. So we have different continents. We have different continents have different technology and it's very hard to secure this ecosystem. It came very fast and it's very hard to secure it. So a bit about technology as I see we have lots of MQ2T and HTTP connections to the from the color to the OEM cloud. We have just in time connections we have SSH connections. We have REST APIs through the third parties to the OEM chips. OEM through the mobile OEMs. But then we have HTTP connections on WebSock and in different propriety connections. So this is a hackers paradise you can start playing with all of the different stuff. So one of my thesis and one of my thoughts that I'm looking at is that this is a normal cloud like that talks with all these vehicles. And now let's say I own the cloud and if I can own the cloud I can send over there updates to the different vehicles. And if I can send over there updates to the different vehicles I can send and create all the vehicles to go into the evil cloud. This is like my personal doomsday scenario. I hope that we will never get to that because this is a total recall of all vehicles or I don't know what else we can do. But if we start to go further so I want to show you how to attack this area and how I I play around with stuff. So first of all I will want to start searching for clues and this is a regular icon like looking at different embedded applications, the internet and the different resources NISTF and the internet also is a good clue to getting information about the automotive area. So at first what I want to do I want to attack the cloud but in order to attack the cloud and attack everything I want to go to the area that is least probable that nobody will start hacking from it. And this is low level. If I get off the chips, I get the passwords and I get different URL and back and endpoints from the chips after I read them from JTAG and different capabilities I can have secret keys and access to the cloud services. So I would just like get like Amazon URLs, S buckets, SV buckets, different communication layers that the chip is talking with the OEM or with the third party or with the supplier or with whoever I'm targeting. This is a very basic because nobody has access to it or only you guys have access to it then usually they don't protect these endpoints. And this is or even if they protect sometimes the secret keys and our inside over here inside the chip and also HSMs you can break the HSM and say different areas and it's fun to look at it. Other places is like APKs. So I was starting I started downloading different APKs and different applications inside like I saw a Bosch interact driver or like E-Flitz mobile or different applications that are not connected to the OEM but are connected in some way and then I started like looking for different back end URLs I can start hacking into them. So what I did eventually I created like there's an open source library over here Libsearch that what it does it downloads all of the APKs that I want and it extracts all the metadata all the resources into it and puts them into Elasticsearch so I can search easily like resources the searches over here you can see the like URLs and resources but I also found passwords and secret files and like Twitter APIs and stuff like that inside actually what is interesting is the WeChat. The WeChat secret actually has API payment capabilities and like you have yes not only a messaging app it has much more stuff over there. So I won't show you everything over here because some are confidential and I cannot talk about them but I found a lot of stuff over here but more if I'm looking for different end points. So I use DNS dumpster but I also just sometimes run my own list of subdomain searching and I found so much domains of the different companies so I didn't try to hack them but you can just each one of them you can start recon starting understanding which languages programming languages are over here and then understanding what do you want to attack but not only the end points I would also go to the suppliers themselves but this is a sometimes you get redirected to different sub suppliers through these domains but using Shodan is also a possibility and Shodan I looked for like Autosar Autosar is a very it's I don't know how to call it it's like a method or compliance for developing stuff for developing stuff for ECUs so the whole industry uses it. If there's someone for Autosar here actually it was pretty nice because I found some parcels of them so I can access I didn't use them but but then you can find also continental and different other like I saw some scatter devices some some devices that was interesting and different IPs and this is another way to get in but what's most interesting is that you can just ask for all of the subdomains you don't need to go start scanning because NASTF has all of the right to repair components software components that I want to in order for me to diagnose my car so if I want to diagnose my car and I want to have access to different I can like Bendex commercial vehicle systems I can access over there or Aston Martin and it has different softwares I can download and with these softwares I I can find again back end connectivities because these softwares usually work with a back end and then if I try to hack into the back end I can get more information about all these different areas so I was looking inside the different I saw software updates as a diagnostic software you can buy them you can download them at different forums like different hacker forums or diagnostic forums there's lots of car forums of like hobby car makers that you can download and illegal tech illegal diagnostic software and in this the diagnostic software you can find lots of back end end points and lots of keys and secrets into the different areas so but what eventually I did I built a big dictionary of lots of words and lots of stuff that are from these different areas and I started searching them on the different open source areas so I don't know how much of you know github but in github you can when you delete stuff you don't really delete them so I like to look and what I like to do usually is I go to github and I look in the comments I deleted secret keys and the company target I want to look at so I find lots of the comments that they deleted and you can see the change you can see the change itself over here that the change request that they deleted the six keys but they still have them and they didn't remove them and they didn't duplicate the keys and it's just over there so you can start looking at different stuff and connecting to the back end services and it's pretty fun so I found a telephone a telephone code telephone code is a way to create your own infrastructure inside this telephone I found a secret in the sub-ID of one of the clients I was looking for I was targeting and this was one of the suppliers I found over here from this area so it wasn't one of the OEMs it was a supplier of these OEMs so this code gave me eventually access to about 100 servers around inside the supplier and at first it was that this was a test telephone but this same telephone gave me access also to production environments so I now had production environment access I had full route access to different 100 environments and this is like a good place that I like to be so I'm now inside one of the third parties one of the suppliers and this was one of the diagnostic services inside this specific example but I want to go further I want to go and hack into the OEM cloud now because the trust between the third party and the OEM cloud is very they trust each other and this is a technology going back a decade ago two decades ago so eventually I just like found the ftp open I scan I start sending the OEM cloud from inside and found the ftp open ftp open and next thing I know I have a shell access and one of the one of the servers inside the OEM cloud itself so inside the OEM cloud I now have access and I'm trying to think out where should I go and this is a real question because I can go start lateral expansion inside the OEM cloud and go start looking at different stuff but what I learned from the automotive industry is that you have the monitoring servers like they like to monitor everything so what you want to do or what I usually like to do is go after the monitoring servers the development servers the different areas and this is why I am focusing on the monitoring server so from here what I found out that the server that I hacked into had like it was a non-critical server it wasn't connected to any production areas it wasn't connected to any interesting stuff but the monitoring server did log in into my server every midnight it logged in into my server it ran some code and then got the result and fetched it back into its monitoring capabilities so they can show the IT what what's happening and this was very interesting because now the question is how does it log in into my server and can I use it some way so I replaced my ssh server and I did a s trace and I found out that every midnight it connects to all of the servers and I have the password of the of the monitoring server that is connecting to me and is connecting to all other servers beside me so this is pretty nice this is a it took me one day actually it was a very fun interesting area this particular area the password when I tried it the next day was was wrong and why it was wrong because they had a policy on the passwords to change the password every day so they put a date as the password like the company name and then the date and then every day it was just changed the date so I knew exactly what was the password the next day and I got in so I have access to lots of different areas over in the IT area and they have different access to the OEM and they're also in the production plant a bit and this is like I don't I the monitoring server did when I connected and I saw it did have different like very specific areas it also wanted to monitor production and this is interesting because you don't want to have the IT area and the old cloud area connected to the production but the monitoring server does want to monitor everything so if you have one monitoring server that monitors everything that you will behave and you will be able to jump into production a bit so now I'm trying to look what I want to do so I'm inside the room and I want to like look I'm looking all around I'm looking inside the different areas and I'm trying to find what is my targets what what do I want to do now without I'm inside this automotive area cloud so most companies use jump servers this is a very secure way to access to give suppliers access to different things they want so when I want to access when I want to give a supply access to some web server of mine or other web server over here you see lots of IE so it's good I like IE and when I give them to say the access they have access to a terminal server but this is not I don't give them access only to our internet explorer this is Citrix so the first thing I do usually I try to break out of the internet explorer I try to break out of different stuff usually it's pretty easy you just go to file open go to find a command and you open it nowadays it's a bit more harder you need to find different old power shell tricks or different ftp nobody knows about ftp if you go open an ftp console you press enter the exclamation mark runs code so you can just like use that there's different methods of getting into and when you when you get access to the Citrix server you can find everything all the suppliers are connected over there all the different companies go through this jump servers and this like when you own this you own everything but what I found out looking inside my network is I found like a server connecting with lots of lots of different printers like I had 100 printers connected to the server and I asked myself why why should I have a server connected to so many printers and after investigating and understanding the reason why inside the factory in order to know which part you want to install into the vehicle you need to print out a barcode or a QR code and they scan it so this is the printers that you print out the QR codes and then you scan it and then you know what part you want to take and you put it into the into the other part like if it's a door you put it into the chassis so what I did I did a poc I did a poc over here I printed a different QR code or this was in another country I printed a Pikachu and I showed them how I can how can I stop the printer what's interesting over here so if I disable one printer I basically disable the production line the production line will stop working if one place and one printer stops working the whole line will stop this is how production line works it's a line so this is a pretty interesting target I also have lots of other targets like web based targets that I can find but I'm looking for robotics and I like robotics robotics the whole shop floor is a full of robotics and usually because there's a rule you do never stop production never do something in production so you never upgrade production also you never do changes and for instance I had some some robots I wanted to hack so I looked up inside the web different forums I just saw use this administrator password this this is a user administrator to this password and the fun thing about this is that they cannot change it because this is hard coded into the device and you will to give to send a supplier to change this password it's almost impossible and to change stuff in production it's also almost impossible so for this reason this is the password of the and it cannot change maybe so in the future they will change maybe there will be different stuff and different robots have different passwords and different back doors like support back doors but even if you cannot find usually what I do I find out who installed the robot itself and which supplier installed it and try username and password as the supplier name it works 50 percent of the time the other 50 percent you need to add a one two three four so this is a this is an area another area I like is development so if I'm inside the office IT area I want to go to development and development I find different areas about connectivity projects like I was looking around the the whole day and found like different areas and then private keys and this is like a live private key of some kind of update apple before I told you the key is like a we're talking about key so this is like a some kind to update the apple certificate in order to have access to it but then if I continue and I saw like some IoT hub connections going over there and with together with this key I found out I got access to lots of different uh this was over MQTT so I subscribed to this MQTT endpoint and I got lots of different devices coming to me so now I'm monitoring all the devices and I'm just like seeing and writing seeing all the live vehicles that are on the system right now and this is not even from production this is from development development areas have weaker passwords have developers they don't like security so much so it's pretty interesting to see so I found a connectivity organization a bit I found the stuff on desktops as I'm just like playing around the network now I own the network I have like different stuff I have even memorable words and favorite school subjects and the IoT admin and different like stuff over here but and the main thing like what I'm looking at is that we are getting connected all of the different companies if we are looking at Bosch or over here I put FCA but all of the different companies over here have connectivity areas and they are becoming connected and they are relying on these suppliers and they're relying on different other fell party vendors to help them build their own ecosystem nobody does it by themselves they do it by or Microsoft or Amazon or a Continental or Bosch or Lear or whoever or the other fell the smaller smaller fell party vendors are looking are helping them connect everything and we have all these connections the insurance companies and the vehicles and everything is becoming connected and because of this this you need to look at a more general approach and to understand how to secure this stuff how how do we secure this ecosystem that we got into over here so a bit about my conclusions over here it's important to understand like the whole ecosystem to look at this stuff not usually when a client comes to me and wants me to secure stuff he wants me to look over just in one place I want you to secure this ECU I want you to secure this backend endpoint I tell him I want to decide my own I want to start research and understand exactly on my own out of scope out of your scope how to hack into your system because usually the ways to hack into the areas are out of scope and this is I have a lot of problems with bug bounty I'm always out of scope it's it's a problem it's but also like how everything is connected and doing security architecture and putting security into areas before SOP it's very important going from the beginning and talking together with all the suppliers and this is like we just need to work together and I know this is a place that has lots of MDAs and lots of competition and lots of areas that the suppliers the suppliers know more than the OEMs sometimes because they see all the OEMs and the OEM sees only itself so sometimes you need to look at the different suppliers you need to talk with them you need to communicate and to be open in security so I don't know how much time we have but if we have questions I cannot hear nothing yeah it does a microphone or something or just like I can answer this this is actually I'm my focus I came from mobile and pen testing applications and networks and they don't have any knowledge and embedded at all so I came to the company and like I tried to understand how to do it so I have from robotics a bit of embedded knowledge but then I have knowledge like reverse engineering is something that I just don't have and for me now at age of 35 to learn the reverse engineering it's with a kid and everything it's almost impossible so you need to have different capabilities in the team so I have I have a pet reverse engineer always beside me so I can always ask him and I found this interesting area can you help I find found this binary can you help me and this way when you have you need to like I like to know a lot and I like to look from above but then those places I cannot go and you have to have like people so in this industry you have to know OT you know you need to know a bit of like scatter and when you go into scatter stuff this is a different perspective but you also need to know embedded security so for once sometimes we see our embedded board and the supplier just removed some of the chips so we need to guess which chips to put back on and to guess which chips to put back on it's like I don't have this capability so I go to my hardware guys and to do reverse engineering I go to my reverse engineering also for new stuff like I'm in this world that I came even from my one year vacation I came back everything moved everything changed I have lots of like different SSRF abilities and different sub domain takeovers and stuff that I I didn't know when I was when I was studying this so you always need to be on your edge and learn stuff but also have this team because you will never be able to know everything yeah so so this is like an example from a four-day assessment only so we were like crazy going to the targets going in low level like a shallow level not not going deep into stuff just seeing that we found going on and so you don't have time and it's a very important thing to understand as OEM or so what kind of assessment you want if you want one week assessment but so you will find like lots of shallow stuff but you will get a feeling of what's interesting for you and then you will need to deep dive into different areas so let's say the bar code you want to make sure that if somebody hacked into the system he can't do nothing or you can't have a fraudulent bar code that can hack into the system by some kind of putting dash or one so yeah I next time I'll find it I'll try yeah yeah of course yeah yeah I cannot talk about it so yeah any any more questions yeah yes this is I'm working with the OEMs directly so they're reported this is why I didn't put anything that is I didn't put any names any clues any way I have so much more I want to tell you and this is like a big problem in this industry I cannot it's a problem I don't know how to solve it if someone can help me or I don't know how to make the research better it's I have vulnerabilities from a different bug bounties I've done in the past for the OEMs and I'm already waiting eight months for stuff to for them to acknowledge it and fix it and I'm sure that even there are much more there are two year plans and stuff and it's basic like if I go to Google and and give the same thing they will fix it in two days they just have this whole process and it's very hard for them to fix currently because it's not them the OEMs don't do the stuff they the suppliers do it so if I if I give the OEM a problem and then LG is it needs to fix it they need to convince LG and then there's a change request and after a change request you need to convince them why it's security and is such a long process at the end it takes you so much time just to fix stuff no so no so I don't know there are capabilities like you can access the servers sometimes the keys and it depends on each company I might or I don't want to go into the companies that I know so like I don't know how to answer this in a politically way yeah any other questions comments thoughts of experiences maybe what you thought yeah yeah so tesla has a bug bounty program and fca has one and I think GM one has one and who else trying to think you guys don't have one yet um I don't know maybe soon there will be better but most of the bug bounties are non-disclosure and this is too bad this is like one problem it's you have this bug bounty but it's a full non-disclosure bug bounty so even after they fix you cannot say nothing about it you cannot yeah I think so it's also you mentioned China for instance China has different servers usually usually there's different requirements so and when we have like vehicles in China so they go to different back ends and they behave differently so this is a one very funny looking at security you don't know every country is different like China is very different but also Europe is different in the United States so you have different chips over and even different software different hardware different back ends everything is like different and but everything is still connected somewhere in different areas so okay so thank you everybody for coming