 Hi, everyone. This is Sameer from Intrapid Control Systems. Welcome to this talk about fuzzing can and can be automotive ECUs and networks. Fuzzing, when applied to automotive systems, can make the systems safer and more secure. I hope this presentation helps today. Share some concepts and see what all is on offer for fuzzing automotive ECUs and networks. So this will be my agenda today. I'm going to talk about these four items here today. First, I will briefly discuss what fuzzing is and why fuzzing is already becoming a part of standard verification and testing for OEMs and ECU manufacturers. In the second part of the presentation, we will discuss some of the techniques and methods for can and can of defuzzing. And this should be really interesting for automotive people who are interested in automotive world. Third, I will show how some specific products from Intrapid Control Systems, which I have used, can help you with your can and can of defuzzing needs. These are the tools I have primarily used for developing fuzzing capabilities for OEMs and peer suppliers. So this information will be also pretty useful for people who are interested in these areas. Last, I will discuss the quickest way to get onboarding on can and can of defuzzing for automotive ECU software development lifecycle. This has been a question asked to me in various times in the past that, hey, how can I start? What is fuzzing? What do we want to do? So I'm going to discuss some of those things as the last part of my presentation. Fuzzing concepts and definitions. So fuzzing as defined by OS for open web application the security project is essentially this. That's the definition defined by OS. So basically, idea is to send unusual, malformed, unexpected and error-induced data to the target DUT. Specific to the DUT functionality and performance requirement, a closed loop control can then monitor the DUTs for any anomaly or any problems in the DUT. So fuzzing has been widely used to find security issues and bugs in IT software from actually many, many years. So the concept is not really new. It's application to automotive is interesting. So that's why we kind of like to add for, to define what automotive ECU fuzzing is. So it's a highly recommended last mile test to reveal any possible vulnerabilities in the network with ECUs for CAN and CAN-FD or there is a design step or, you know, for CAN-CAN-FD design strategies or implementation and it performed as part of product design, development and validation cycles. So that's where industry is going in our experiences. This does not necessarily mean that test should only be done at the end of the design or development and validation cycles. Fuzzing essentially is now being considered as part of software development life cycle. And so it is even, you know, a better strategy to include this testing as part of milestone releases of software during the entire cycle. And that is how the importance of, you know, fuzzing is there for an automotive. It's actually well understood fact, you know, we all of us know that it's a fact of cost of fixing a bug exponentially increases if discovered later in the development cycle. In fact, this additional testing may not be a large impact on the cost. You know, people are trying to assess, hey, you know, if we do CAN fuzzing, you know, CAN-FD fuzzing, you know, and you know, test, you know, perform this kind of testing, is it going to be very expensive for me? So that's not entirely true. The impact on cost is actually lowest if you're existing tool chains support such testing. And, you know, some of the companies like ours, like Intrapid, we have recognized this pretty early in the game. And this is alongside working alongside with customers. So today, there are readily available tools for CAN and CAN-FD advanced level fuzzing. This means for existing users of such tools, the adaptation is actually easy. And for the new customers, it is something they can try to look into for, you know, for advanced level of fuzzing on CAN and CAN-FD. So let's look at, you know, some of the frequently encountered terms. And I'm going to just, you know, talk about, you know, as a concept here, these terms actually are very common in world of information technology, very common. You know, this is nothing new, you know, in most of the audience here, I'm sure is very familiar with these terms. But the concepts are important for people who are coming from automotive world, you know, for the things going on in the industry right now. So let's look at these four, you know, conceptual terms, you know, to learn what they mean. Vulnerability. So vulnerability is a weakness in a system. So it could be, for example, a result from a design deficiency or an implementation bug. Vulnerability then actually could be exploited to obtain unauthorized access to the system and then affect the system. One of the examples I can do is, for example, an unsecured diagnostic code or a weakness in a gateway module which could lead to unsecured access to a subnetwork ECU. That wouldn't be good, right? So you don't want your, you know, down the stream ECUs to be accessed through gateway unauthorized. So exploit, exploit refers to an action that takes the advantage, you know, of a vulnerability and exploit may reveal a behavior in the system which was not a design or implementation intent. In penetration testing, we want to expose maximum possible number of vulnerabilities and exploit them. You know, that's the purpose of, you know, penetration testing on Canon Kennedy. You want to exploit all the vulnerabilities before an adversary does it. Attack Vector. Attack Vectors, basically these are mechanisms. These are, these refer to the mechanism and the targets or the paths an attacker can use to exploit a vulnerability. So for instance, you know, an exploit may use an unsecured message vulnerability. This means the message on the bus on the can bus is not secured. And so thus it makes it, it makes it vulnerable. And it could use attack vectors such as, you know, network management frames or there is time dependency of normal mode transactions between the nodes for correct operations. So those are the things an adversary can use and confuse the network or even cause problems in the safe operation of the system in the car. Attack Surface is a set of attack vectors. You know, essentially a group of attack vectors, you know, identified and, you know, they are loosely related to achieve unexpected behavior in a DUT or an ECU or on a network. So Attack Surface is essentially a group of attack vectors. You know, which are kind of related with each other for a specific, you know, identifying an unexpected behavior in the DUT. Let's look at fuzzing concepts. You know, what are the objectives and what are definitions, you know, some basic things before we get into, you know, deep dive into more into automotive things. So fuzzing in information technology has been prevalent for many, many years. It's nothing new, you know, it's been done there many, many times. But with modern cars in the complex edas in autonomous systems, they run millions of lines of safety critical code. Fuzzing is actually now becoming extremely relevant for this automotive manufacturers. And we can feel it being part of this, you know, community working with OEMs and tier suppliers. With fuzzing, essentially in the, so the whole my talk here is going to, you know, kind of relate what it means for automotive world to do a penetration testing using CAN fuzzing. So with fuzzing we are challenging the software design and implementation through injecting most unexpected, you know, semi-malformed frames. It's important to note this concept in the terms for automotive. We do not want the frames to be rejected at electrical level, otherwise it's meaningless. The fuzzing will be meaningless if the electrical system rejects those frames. We do want this, we do want to have enough validity in CAN and CAN every frames for them to reach to the transfer layer. And ultimately, you know, to the application layer and they should pass, they should go past the object layer. So it should reach the application. So we want enough validity around that, you know. So otherwise the fuzzing is not much relevant to CAN. It must be noted that most of such frames are still protocol-compliant. You know, when we are sending a CAN frame for fuzzing it's gonna be protocol-compliant. And so this is what we call as protocol-aware fuzzing. And in fact, this is most relevant to CAN technology because CAN, inherently in itself, is very robust at controller and electrical level. And it has excellent fault confinement. So CAN itself is very well protected. So, you know, so what should be our key objectives when it comes to automotive fuzzing, essentially? First, you know, early identification of security gaps and software bugs, you know. It's paramount to reduce the number of field updates because of software issues and bugs. We all know that because, you know, they are not cheap. They are very expensive, you know. And so for example, to have an OTA or a FOTA campaign or their update campaign for a bug fix is really least desirable. You know, nobody wants to, you know, do an OTA campaign to fix a bug, you know, that's not something, you know, OTA campaign should be focused on. OTA campaigns are really to provide new features to the customers. There is tremendous need of testing these OTA updates before they're deployed in the cars. And that's when the CAN and CAN fuzzing is very important. Thus, the recommendation, essentially, is to add the fuzzing first testing as part of SDLC or software development and release cycle. And automation tools are the most, you know, most important in this case. Vehicle spy software is a good software to integrate the automation testing in automated integrated testing for running the fuzzer tests at every release cycle. And that would be very prudent to do this to identify security gaps or software bugs in the target ECUs or nodes and networks. Second fuzzing can be extremely useful tool to try to provide robustness to the system level validation. Are the system level fail safe mechanisms good enough against an adversary getting partial or full control of critical messaging between subsystem nodes? That's why a system level first testing is also very, very important to be done. Third idea is essentially again to look for vulnerabilities on CAN and CAN FD node and networks. And before somebody else finds it, essentially, hack into your own network, find out all the exploits, use all the exploits before somebody else does it. And so yeah, that's one of the important objectives in automotive world for using CAN and CAN FD fuzzing. Lastly, it's great test to just test the network and node performance against extreme, I would say unusual traffic. In itself, it brings the robustness. So adding robustness to this in-vehicle network performance is extremely useful technique. And that's why penetration testing using CAN fuzzing is getting more grounds. Let's look at some of the techniques and methods of how you can do CAN, CAN FD fuzzing. So I'm gonna talk about a few of them today. And some of these come from our experience working with a lot of our customers and doing some research on my own. So I'm kind of sharing some of the things which is being practiced in the industry right now. So let's become aware of some of the key mechanisms when we are fuzzing ECUs or CAN and CAN FD networks. CAN and CAN FD networks, as I said, are inherently highly fault tolerant buses. Also note that traditional automotive networks have been embedded device networks which have many electromechanical sensing and actuation systems under their control. Thus, the identification of attack vectors and careful identification of first scenarios is probably where the most focus needs to be. We have to be very carefully identifying all the attack vectors because these ECUs control the vehicle operation, the automotive operation. And it's important to identify all those possibilities. For example, we all know about denial of service situation in IT world, but how do you define or can we envision a denial of service situation in the context of ECU cluster? Or more importantly, how do we design fuzzing so that we find all the vulnerabilities before an adversary finds on an ECU cluster? So some of the methods and techniques which we could look into are here. So for example, it's important that we thoroughly study to identify all possible attack surfaces. Some of the examples I can give is focus on payload data for it. For example, the payload, the content itself of the frame, the payload which is gonna be consumed by different nodes. Frame priorities, those are based on the arbitration IDs on CAN. So can we see a situation where a design was dependent on a priority and an adversary attack changes the priority of that frame? Critical factors like dependency of internal ECU algorithms on, for example, a periodicity of frame with needed data, such as these are the things we need to look into and identify and study. If is my algorithm totally dependent on a fixed frequency CAN data and can the adversary attack that and affect the algorithm and make a wrong decision in the car? These are good pointers to direct your first test and to cover all the aspects. Some of the mechanisms which would be useful include monitoring the nodes for their abnormal outputs and such as through diagnostic queries. You can query the diagnostics or you can stimulate diodes, for example, and then observe the response on the network while you're conducting the first or the first tests are being run. In ECU world, getting a feedback from the system is the key to make sure the fuzzing actually use the relevant results out. Classical pervasive definitions in fuzzing like generational techniques. There are generational techniques, there are a mutational fuzzing, there is randomization. So for example, in generational fuzzer, new frames are actually created by the first tool. That's the concept in generational fuzzer, where you generate the new frames from the first tool. This means an intelligent fuzzer can establish a closer relation between generated first frame to the normally present frames on the network. And that's an important concept. To deceive a system, the frame should appear to be very normal but causing abnormal effects on the bus. So thus making such injected frames, they have to move over value from the duty perspective. This kind of protocol aware fuzzing is actually most relevant for CAN or can of denotes. I've been talking about that in this presentation. Mutational fuzzing also needs to be protocol aware when it comes to applying to automotive networks. That is not that much relevant when it comes to, for example, IT networks. For automotive networks, even the mutational fuzzing technique needs to be protocol aware. So how do you do that? You introduce minute variations in the data through automated mechanism in a tool, in automation. And you inject those minute variation of the data on the bus. And so essentially to see whether the relevant nodes and ECUs understand and can take right actions when the data is not the real data but looks like real data. With randomizer, for example, it's a unique black box technique where you can realize the first frame can really create a totally agnostic situation to the arbitration IDs or DLCs. And in the randomization, you can actually on the fly switch between CAN and CAN of the frames to see how the reaction is from the network. And there are tools like Waker Spy and there are hardware tools from us, Neo VI family hardware, which allow, they're very helpful in realizing randomizer for fuzzing. Let's look at some more techniques here. As I mentioned before, it's important to understand if the duty response is valid or is it invalid or unexpected or abnormal. So in automotive world, rest bus awareness is the key. How is the rest bus behaving? How are the other nodes behaving when I am, for example, doing a mutational further? So it's very important to get the feedback from the network. Another important aspect is what is called as, what essentially it's a new phrase, which is quite recent is perpetual fuzzing. So this again comes from the automotive concept. In perpetual fuzzing, this means the first testing is a part of your development life cycle. And these fuzzing configurations can be continuously improvised by adding more and more unusual scenarios. One aspect of CAN and CAN of the fuzzing, it's conceptually that these tests actually do not directly depend on the duty. I mean, duty specifications and requirements do not affect the CAN first test. This means first test could be independently created as a suit to apply to various different duties. Some parts of such first test mechanisms as feedback measurement may only need to really change depending on what ECU we are talking to. For example, if it's an ABS ECU, I could measure its performance differently than if it's an airbag ECU. I'm going to talk more about this, more about vehicle spy software which has capability to run first test, first test execution engine and the measurement engine in parallel. That's the key for vehicles. That's a key which I've seen in vehicle spy software. I should be able to run my further execution engine in parallel to my measurement engine that gives very good control on the tests. And this remains, I think, a critical feature for successful capturing of the failures. Lastly, a thorough consideration is needed if duty is a gateway device. And you'll see a lot of gateways these days. New network architecture has OTA updates. So there is generally a gateway module, a telematics module and a gateway module. And these are very special characteristics. We have had experience and we have tremendous amount of experience working with gateway modules for very complex scenarios and testings. And these gateway modules are special characteristics like FOTA or firmware or the air updates. Or if it could be like a telematics ECU or a V2X ECUs. In these scenarios, fuzzing could be a lot more complex. And it may require to run your first scenarios across multiple of clusters or networks at the same time because gateway module connects to maybe let's say eight subnetworks. And it's important to be able to monitor all of them while you are attacking the gateway module for your first testing. So in such cases, the response monitoring also requires pretty advanced setups because you could have a downstream ECU on a six scan network misbehaving because we did the first test or we did the first attack on the gateway module. So these are all the things happening in the industry. And I'm happy to share with this community here. My concluding observation on this particular part is fuzzing for automotive is more design oriented. To be effective, it has to be design oriented. There has to be design idea behind how we want to fuzz. Simply standing random frames without like a closed loop control and without the ECU behavior, aware monitors may not necessarily yield desired outcomes. We need to minutely observe what ECUs are doing what network is doing when we perform this first testing. But the technique itself is doubtless in the industry now is required for the robustness of the vehicles and the performance of the vehicles in the field. Finding a single vulnerability in a post-production environment when vehicles driven in the road, it's very expensive to update, it's very expensive and it could be safety issue for people. But here I want to spend a lot of time working on interpret vehicles by tool. And I want to share my experience in using this tool for can fuzzing. This mostly is used, I've been using this tool with automotive OEMs and peer suppliers to see how they can fuzz their networks and ECUs. So interpret has created a set of tools in the framework. The framework is pretty involved framework where you could set up a bench for a feedback loop mechanism from the ECUs, from the DOTs. And you can perform extensive fuzz frame injection and execute the first tests on these networks. So some of the advantages which I have seen from using VehiclesPy and NeoVI family of hardware is it VehiclesPy has a UI-driven fuzzing engine. This means even associating a complex test script with the first test doesn't require like a whole team of Python programmers. That is the benefit I have seen our customers using and myself using it for my own fuzzing research. You don't need like tons of script creators or programmers who work on creating this thousands of scenarios which you can easily create in automation. And in the perspective of automotive, all of these things must sit well with the current verification and validation teams. They need to be onboarded immediately with this. You can develop a whole team who just does the can fuzzing. So VehiclesPy allows those kind of capabilities in the tool itself. And Intrepid, we as a company have been doing automotive networks for I think almost 26 years now. And all the OEMs have been our customers. We've been working with them for years. So we have applied all that extensive experience working with ECUs and networks and building high-end, high-speed FPGA-based tools. And our tooling for fuzzer is also designed totally around automotive ECUs and networks. One of the biggest factor for automotive ECUs fuzzing is diagnostics. Diagnostics is also an integral part of an ECU and also an integral part of VehiclesPy software. So while you're fuzzing, you can do your diagnostics on the diagnostic channel and query the ECU. At the same time, you can also essentially fuzz the diagnostic channel itself. And that's the beauty of some of the things which we have done with our customers. So the native support for a transport layer is already there, native support for OBD or worldwide harmonized OBD is already there. Some of these tools are already supported in VehiclesPy and we were able to leverage all of that for our fuzzer. With inherent capability in these tools for multiple channel fuzzing is very important. If DUT is a gateway node, I do want to monitor all the subnets at the same time, in the same context, when the same tests are running. At the same time, the tooling also need to be able to simulate some of the rest bus behavior because some of the ECUs depend their operation on the rest bus being perfectly fine else it's gonna shut down its transceiver. If it shut down its transceiver, we are not gonna be able to fuzz anymore. So we have to kind of keep the ECU awake to a certain point where it accepts our first frames developed and then you can attack the ECU. So rest bus simulation is also extremely important when you're fuzzing. So remember that, if you have a network of ECUs, make sure you have also the capability to make sure the ECUs are awake. And you simulate the rest bus behavior to make them active enough to accept our frames. And at the same time, there is also support for measurement of physical quantities or physical signals and IOs. For example, I want to see if the ECU went through a running reset. I can measure the current on the battery line. I can measure how much is the voltage drop across certain point across certain connections on the ECU. So all those capabilities are also important when you're fuzzing on automotive networks. Some of the other features which I have used in the tool in Waco's pie is I was able to generate thousands of test cases from the database files. And this process itself, if done manually, could really take months and months of efforts by team. But Waco's pie allows to create automatic signal attack from the DBC file, and you can create various different generational fuzzing test cases. You can generate any kind of range you want. And for mutational fuzzing, you can make a small variation in the payload as an input to the DUT, try to override the real data. And then you do this minor mutations, minor variations and see what how it goes. And Waco's pie has a lot of inherent capabilities I found and I've been using to support extensive test design around mutating known inputs. And that's a very powerful use in Waco's pie. The few other types of fuzzer techniques like randomization and some other techniques are very unique to Waco's pie. If you end up using it, feel free to use those. You can give your own seat for randomizer. You can have a Waco's pie generated seat. There are many various other things possible in this software. One other thing I found is support for database. And I did mention a little bit on ARXML. Supporting ARXML, supporting DBCs for automotives are extremely important because that's the real way of being actually be able to affect the performance of the device or the ECUs. Again, the classical question I've been always, I had to add this slide just to make sure how you actually do all these things. Seeing some pictures helps understanding what can be done if you're thinking to do print direction testing on CAN, what are the actual things I'm gonna do? So how and where one could start with CAN and can have defuzzing for production vehicles or ECUs or your bench networks. So you could start with some of these products. They're available in the market. So VehiclesPie Enterprise Plus, for example, is the software which has full-fledged support for CAN-fuzzing. You could, you know, you could new year fire too. Again, the device here has eight CAN networks so you can simultaneously fuzz devices of eight CAN networks. So this is the one I typically use. This is one of my favorite tools. And so it's widely, VehiclesPie has been widely used in automotive software development for more than, you know, 30 years. So our focus remains on the customers and, you know, all the capabilities you see in VehiclesPie's because they've been working with our, you know, customers for a pretty long period of time now. About Neovia Fire II hardware, this has a multi-channel CAN, CAN-FB and in fact, it also has DOI support. It also has, which I also use very frequently is it has inbuilt data logger. What that means is if I am fuzzing in the vehicle itself, I can record the entire traffic in an SD card inside the device. And if I happen to cause an issue in the car, shut down the car or put it in, you know, for example, lymph node or what have you or cause some multiple, you know, DTCs, then you can record everything in the device itself. So you have the least latencies from recording on Windows or, you know, the PC or other mechanisms. So Fire II remains very useful device for in-vehicle level fuzzing and also actually bench level fuzzing. This year, Intrapid also launched our latest, you know, platform agnostic, very advanced tool that we call it, it's really next generation tool. And it's like advanced automation and testing for in-vehicle networks. It's called Vehicle Spy X. You'll hear a lot about that in this event. And so I was also asked a question, hey, you know, can vehicle spy be used as can first tool? You know, and I'm gonna answer that in, you know, the next slide. So I summed up a lot of information I've selected. I tested it with, you know, working with the customers, you know, internal bench level setups and things like that. And here is, you know, my compiled information. And I think, you know, this is available for you guys to use it. And I compared the current capability support in three different situations, you know, I didn't do a lot of open source tooling myself, but this is the feedback I get from, you know, from the engineers or using open source toolings and methods to do fuzzing on automotive. And so I'm gonna go through some of these things with you guys here. And so the support for like dedicated can fuzzing support. So my observation is it's reasonably, it's low in vehicles by X currently. Open source tooling is really, you know, limited and it's exceptionally high in vehicles by three enterprise edition. It's really accepting the high because there is a set of tools which are can first tools. For scripting, I've seen Python support. We have very powerful support for Python in this vehicles by X and, you know, of course open source tooling use Python extensively and they use other languages and other programming environments. In vehicles by three, it's a UI based scripting engine called function blocks. And you can also do CC plus plus programming inside vehicles by scripting. As far as maturity, I've seen very high maturity when it comes to vehicles by enterprise again for can and canopy. And for fuzzing, you know, like how much automation is there? How much support native support is there? So for these by X, it's under development. You know, this is what I was told by the team. And automatic generation of first scenarios is right now not supported in vehicles by X, but it is fully supported in vehicles by three enterprise edition. So you could generate thousands of test scenarios, possibly to a single click. And again, I was told, you know, that in vehicles by X, the team is gonna use the same technology in vehicles by X in upcoming releases. Now ARXML DBC support, very powerful support for, you know, in vehicles by X and vehicles by three enterprise edition. I haven't seen really very native support in, you know, there is open source project to read a DBC file or ARXML, but being able to put like, you know, zeros across entire payload for all the received messages in a single click, you know, that kind of support lacks in open source tooling. Programming, flexibility and development, very high, I would say in vehicles by X because you have full control on each layer independently through programming environment, through API calls, through, you know, different layered, you know, call through using Python. So very, very much, very powerful, very programmatic in this, in vehicles by X. I have seen the same thing in open source tooling because you have really a programming environment. There is nothing, you know, automatically done that you have to develop everything. In, in vehicles by three, there is limitation of not being able to use Python, but personally, I'm a C programmers, I'm happy with, you know, vehicles by three. Automotive native protocol support like XCP, UDS or Autosauer, et cetera, they are fully supported in again, vehicles by three, and there is a reasonably good support in vehicles by X. But again, like, you know, looking for a UDS based support in an open source environment is kind of difficult and complex to create in itself. I have not seen much support on that front. Report generation, we have very good support in fact, in vehicles by X, there is a good support in vehicles by three. This is again, you know, I would say this is a area where we are focused and like I seen both of these tools, you know, next releases bring a lot of, you know, test reporting. Like potential for a tool to be part of the development and design scenarios. Vehicles by X is defined, you know, designed around it, you know, you could actually run your first test in your development pipeline through using vehicles by X. In all those capabilities will run in the pipeline while you are actually developing your ECUs. This means you make a change in the ECU software and your first test could be run in the pipeline on vSpy X and test out, you know, all your first scenarios. So this is very, there's an acceptable support for this to be able to do that. And I've seen very, there is almost negligible support for that in vehicles by three. This you still have to do right through a text API or something in your pipeline. So I would say the support is low here. Cost wise, both the tools have a cost, but I think, you know, as I say, you know, bang for buck, that's highest for vSpy X because in future, you reduce the amount of time your validation team is gonna write first test cases or run the test cases and look into the results. All of those things we are gonna eliminate through vSpy X for future. So it's a good investment now, but of course, you know, that's why I listed all the things here, you know, based on what you are, what your scenarios you are gonna do. And based on that, you could choose one of these mechanisms to do your can buzzing. One thing I should talk about is the OS support. Vehicles by three only supports windows. So if you're running your, for example, virtually see which you want to fuzz, you do want to go for vehicles by X in that case because, you know, we support vehicles by X or run some Linux Mac or Windows at the same time. I mean, all three OSs are supported. Hopefully this is, you know, this may help knowing what capabilities are available now in different tooling and how, you know, industry is doing their fuzzing and what are pros and cons. I hope this presentation was useful to learn more things for you guys. And well, that's it from me now for now. I think this last slide of my presentation and thank you very much for your time to listen to this. If you have questions, you know, just ask me if you have fuzzing experience you have worked on, you want to share with me, you know, if you found a issue in the ECU, let me know. I'm very excited to know all about that, you know. And just share your experiences with me. My email ID is here. And if you have more questions on some of these products I've worked very frequently, then just send me a message. Thank you for your time.