 Hello everyone, I'm Vaishnavachat from Texas Instruments and today I would like to talk to you about emulating devices in the Linux kernel using the Graber subsystem. I work with the Texas Instruments Linux development team and mainly work on upstream Linux and UBOOT development for TI processor devices. During my free time, I also try to maintain the TI platforms in the SeperRTOS and I got introduced to the Graber subsystem in Linux as part of a Google Summer of Code project while I was a student and it was related to adding add-on boat enumeration support in Linux using the Graber subsystem. So this is the overview of the talk. The presentation is based on a proof-of-concept to emulate devices in Linux using the Graber subsystem and using it for use cases like automated testing for device drivers. So first, why would someone want to emulate devices in the Linux kernel? By emulating, I mean reproducing the functionality of an actual device in software and and making the devices appear in as in the as per the driver model and device driver model in the Linux. So there are multiple use cases. Most important one of them is automated testing for device drivers, libraries and user space applications. As you know, Linux kernel has a huge collection of device drivers and automated testing for them is a challenge. And if you want to test all of these device drivers on actual hardware, you will need a huge test form with all of the supported devices with the device drivers in the Linux kernel. So testing on an emulator platform makes much sense. So we will be discussing about how we can emulate devices and what are all the existing methods to emulate devices that exist. And one other use case is let's say you have a behavioral model of your device and actual device or the silicon is not available at and then you want to develop device drivers for those and you can have a behavioral model and make it appear as a device in the next kernel and do the driver development. Another use case is let's say you have some subsystem level bugs that are issues that are not dependent on the particular hardware, then it will be much easier to reproduce these issues through some record and payback. So you don't need to have access to the particular devices in the field to record and payback and recreate those bugs. So what are the current ways you can emulate devices in Linux? So you will all be knowing about QEMU, which you can use to emulate machines or systems and it allows you to, sorry, it allows you to even emulate complex systems consisting of resources with multiple peripherals like network cards and all. And there is something called UMock Dev, which allows you to emulate devices in user space. It is mostly used for user space library testing. It allows you to create arbitrary C-suffercentries, then fake block and care devices and you can test your user space application or library API using the UMock Dev. Then there is something called USB IP, which is a project to export USB interface over the IP or an IP network and you can extend this to emulate USB devices in your system. And then there is something called GBCM that is part of already the GraBus subsystem, which emulates an arbitrary set of project error modules. But the focus of the GBCM was to actually test GraBus, but it does not enumerate any device or perform testing for a device driver. So what's GraBus? If you are not aware of GraBus, GraBus is an application layer protocol which is developed as part of Google's project error modular smartphone. So the idea of this project error modular smartphone was that you will have a base module which will have a minimal set of features like you'll have the processors and all and then you'll have multiple modules like camera modules, display modules, speakers and all and you will be able to customize your phone as per your requirements. And the underlying subsystem or the software architecture that helps with the enumeration and working on these modules is GraBus and GraBus has its strong roots in the Linux kernel. So the fundamental idea of GraBus is keeping the intelligence in within the host. So if you look at this, you'll have these base modules or the base smartphone that has the processor and all the intelligence is within the host. All these modules are the, let's say you take a camera, you plug in that the camera by itself does not have any intelligence. The application processor on the base module sends transactions over the GraBus protocol to these modules. Then the module performs operation like let's say taking the picture, then sends back the data to the application processor all through the GraBus protocol. And the GraBus is unique because you can think of GraBus like any remote procedure called like protocol where you send RPC calls to a remote module. In this case, it's an external module, which sits on the Unipro bus and you perform the operations on the remote node and you return back the results to the application processor. But GraBus is unique because it has strong roots within multiple Linux kernel subsystem and you can create virtual devices in the Linux kernel subsystem. So let's say you plug in a camera to your projector or motor modes phone, you will have a virtual device or virtual camera that node that appears on your host processor and you will be using the device as if it was actually on the host, but not as an external entity. And when you perform some operations on these virtual devices or the controllers, those operations are converted to GraBus operations and sent to the modules. And the module performs the necessary actions. So in short, GraBus allows the peripherals on a remote device to appear as if they were on a host. So with the projector, the actual transport layer used was Unipro, which is a high speed protocol, maybe standard protocol, but GraBus by design does not necessarily take any particular transport, it can work on any arbitrary transport. So one of the popular transport use is TCPIP because you can then convert TCPIP over anything and it can even work on, it has been experimented on UART as well. So you can see that this snapshot, I have an I2C controller on a remote microcontroller that's appearing on my host over GraBus and if I perform operations on this virtual I2C device, the operations will actually be performed on the remote device. And all these transactions you will be doing as per the standard, it will appear as a standard Linux I2C controller spy. And you can see that GraBus supports multiple subsystems and even complex subsystems like camera, display and all. So in the typical emulation mechanisms that we discussed earlier that it is difficult to emulate a complex module, let's say you have a camera, you have a camera control interface over the I2C, then you have the camera data interface. So emulating such complex systems is difficult with the existing platforms, but GraBus by design has a concept called bundle where you bundle multiple controllers together as an entity. So a camera will be considered as a bundle of I2C than the CSI interface or similar ones. So how does GraBus, what's the fundamental GraBus, how does a GraBus operation looks like? So GraBus packetizes all these transactions on these virtual controllers and sends our to the remote device. So the GraBus operation looks like this, it will have a small header, which you can see it has the size of the actual message. Then the ID, it's used to track the all the messages. So when responses come back, it's matched based on the ID, then the type is what tells you what operation it is to perform. So let's say you have your host processors, which runs the GraBus host in its kernel and you have a remote microconductor which has the remote node firmware of GraBus. Let's say you want to toggle a GPI over GraBus, what will happen is you want to send something called a GPI or set value request. You can even trigger it through the user space, but what happens under the hood is something like a message like this will be sent from the host processor to the remote node. And the remote node will have the basic firmware to perform this operation. So it will decode this message to get the GPIO line offset and value and perform the GPI operation and then return back the response. So unfortunately, the modular smartphones are not being actively developed for now. And one of the important use cases that is emerging is being GraBus being used in Internet of Things applications. So in this case, from comparing to our earlier modular smartphone, instead of the base smartphone, we have something like a Linux, a single board computer running Linux. And instead of the individual modules, you'll have some microcontrollers. And in the actual smartphone concept, it was physically attached to the smartphone, but we know GraBus does not necessarily any particular transport. So we can have the transport as TCP IP over a wireless network as well. So your smartphone with modules attached is now for an IoT application becomes a single board computer gateway and you have multiple nodes spread across your deployment system. So what will happen is you will have peripherals on your remote microcontroll node that will appear on your Linux SBC. So I have a recorded video here. So what I'm doing is in the host. This is the host running the host is a big world, big play SBC. And I'm starting a gateway application, which allows you to send the GraBus messages from the camera to the remote node. The remote microcontroller is the device you see on the top. It works over sub-regards, low power RF. So we send this package. So at the earlier, we saw the GraBus messages. We send those messages over TCP IP. Then the remote node will have something like a very generic firmware. It won't have anything. It will have the firmware to, let's say, take an input. It can perform, let's say, you send an I2C transfer. It can decode the operation, perform the transfer and send back the results. So in this case, this remote microcontroller has some sensor devices, which has device drivers on the Linux kernel. So how this concept becomes powerful is that your remote microcontroller only has some generic firmware that can translate this I2C or GPIO operations on that particular platform. But let's say you take something like an I2C sensor. You can have a device driver on the Linux kernel. And we already discussed that the I2C bus of the remote controller appears as a virtual bus on the host. So let's say I have a light sensor on this microcontroller. I can probe a device driver on the host to interface with this sensor. So the advantage is that I don't need to do any low-level firmware development on the remote microcontroller side. I'm keeping all the intelligence within the host, which was what already was being done with the Projectara smartphone project. So Graves changed all these operations on the host to the Graves operation messages and the remote node performs this operation. So from a kernel Graves perspective, there need not be an actual device. So let's say you send a GPIO get value request. Let's say some entity responds back with the proper Graves response. Then the Linux kernel Graves subsystem will think that there is an actual Graves device there. So this need not be an actual device, but we can emulate as a software entity within the host device itself. And you can do it in multiple ways. In the previous example, I had some light sensor that has been fetched over Graves from the host. So I can record those transactions, what happened during a device driver probe for that light sensor. Then what happens when I fetch the Illuminance input reading. And then I can play back these operations and then create a fake device. And but to the kernel or the host system, it won't appear like a fake device, but it's the transactions are actual transaction value transaction happening over Graves. Then you can do other ways as well. You can implement handlers in user space. This will be very useful if you want to write unit units for your device drivers. Then there is also another concept called native POSIX target in the SeperRTOS, which allows you to run a SeperRTOS application as a general Linux application on your host. So there has been already efforts being done to port the Seper remote device firmware to the SeperRTOS. So it was basically focused to run on microcontrollers, but you can run the same exact same application on your Linux host using the SeperRTOS native POSIX target. So before we go into how we can emulate or create fake or Graves controllers on the user space, we can have a look at what are all the important operations. So we'll be discussing about creating a GPIO controller and also an I2C controller and attaching devices on it. So on a GPIO controller, the important operations are listed here. So things like getValueSetValue gets the on an actual device, it will fetch the details from the controller registers or write to the controller registers based on the API calls. So on a virtual or the emulated GPIO controller, what we can do is we can just so I just chose Python to create these handlers. So these handlers will just instead of emulate the remote device, it will fetch Graves messages that's coming from the kernel. It will do some operations, maybe it will be just storing the context maybe or maybe sending the responses back. So in this case of a GPIO controller, what I'm doing is I'm just storing the context of the IO states. Let's say if I receive initially I'll set all the IO states to default value if I receive a GPIO set request or get request. Accordingly I'll change the GPIO values or fetch the existing values. And similarly, you saw the previous slide where all the API was described. We implement all the handlers for all the Graves GPIO operations. So this is a virtual GPIO controller that got created. And you can use standard Linux GPIO utilities to interact with this GPIO controller. And then you can even perform some simple unit testing on the simulator GPIO chip. So it's a very basic test. You are just setting the GPIO states using the GPIO API. And going and checking the controller if the actual GPIO set has taken place. So in this case, you are actually testing the entire from the Python bindings for libGPIOd to the libGPIOd API to the Linux kernel GPIO subsystem. So you can test the entire subsystem. So you can see the virtual GPIO chip created. So the first one is the one that was actually being present on my laptop. And the second one is the Graves GPIO chip created. So you can create with an arbitrary set of parameters you can play around with NGPIO or anything and emulate systems according to your use case. So this is a very simple test, just checking whether the GPIO set has worked properly. Similarly, we consider the case of an I2C controller as well. In the case of the GPIO controller, it was very simple because you can just set, get, or maybe trigger IRQs on those operations on GPIO controller. But in that case, your emulating system is limited within that GPIO controller itself. But in case of an I2C controller, you will have devices, you will have the emulated I2C controller and you will have devices on those emulated I2C controller. So if you consider this case, this can be generalized to almost most of the embedded buses like SPI or UART as well. So you can see there in I2C, there is only a minimal number of Graves operations. The main one is just transfer. All the I2C relevant transfer operations are contained within just that operation type. So in this case, you can just implement some handlers that will fetch, receive the messages. So if you look at this protocol functionality, I am advertising that my virtual I2C controller has supports all these transactions. And on transfer, if you look at an actual I2C controller, we need to put the transfers on the I2C bus and some devices need to acknowledge and respond back to those transfers. So if you look at the symbol on I have emulated, you can see if I run something like an I2C, there is no device to respond to. But to create an actual useful system, you will need to have devices on this virtual I2C controller. So then you will need to translate all the incoming Graves and the I2C messages and put it on something like an emulated I2C bus to which you can attach the devices. So it will be similar to how that physical connectivity looks like. You will have your I2C controller, then you will have your I2C bus and on those you will have the devices connected. Similar to that, we create something like that in software. So you can see, so I have a virtual I2C bus called I2C10 on my device and using this new device API, I am probing the device driver for something called an OPT307 light sensor. In the previous case where we had one demo with the Beagleboard and the remote microcontroller, it was the exact same sensor that was present on the remote microcontroller. But this time we are emulating that sensor on a virtual bus that's emulated on the host itself. And during the device driver probe, there happened some operations like the reading the device ID registers and all. And the probe being successful means we have emulated the device and then we can perform the readings of the sensor values using standard utilities. So similar to the GPIO case, I have handlers and then I have the device, the light sensor device being emulated. So you can see that's the application where that Graebus transaction happens. It's doing a hex number for all the Graebus operations. And you can see that the Graebus device was probed and you can see there will be a I2C device, a controller that was created, a new controller that was created. And that name is Graebus I2C adapter. So for creating all these virtual devices on the Linux side, you don't need to do anything. Everything is already present in the Linux kernel. Just to respond back with the proper responses, you will need to have some handlers in the user space. So you can see that the light sensor appears as a IIO device. Then I can use that you can have the standards, the surface entries for the IAR device. Then I can use tools like IIO info to read the light sensor value. So in the model for the light sensor, I have put the illuminance value as a random value so it will return a random illuminance value. So you can do testing like there are different integration time values possible. You can do different testing on this device driver by emulating the register values on the user space side. Yeah, and I am almost done with my slides. So there are already existing mechanisms to emulate devices in the Linux kernel, but Graebus by itself, it was designed to interface complex module with the modular smartphone and it supports describing devices that are complex like camera or display which needs multiple subsystem interaction or multiple entities to work properly. As I discussed before, let's say you take a camera sensor, you will need your I2C interface, then you will need your CSI2 data interface. Or maybe if you take some simple example like an I2C sensor which has GPIO IRQs, which is something like a ready signal, it will trigger when the conversion is ready. And let's say there is a GPIO enabled. So to emulate this device, you need to have an I2C client device and also a GPIO consumer device to emulate this complete unit of the I2C sensor with interrupts and enable GPIO. So Graebus was all designed to describe devices like this and if you can send the proper responses to the Linux kernel Graebus subsystem, you can have virtual devices like this created in the Linux. And also in this case, I kind of wrote those handlers from scratch, but you can reuse the effort that was done for the Graebus software port and using the native POSIX target. And you don't need to write those I2C handlers and all because the remote set firm were already implementing that. And since this is just an application layer protocol, you can have that same application run on your host Linux system as well. I'm done with my slide. I'm open for questions. I have a question. So actually, what kind of devices we cannot emulate using? Okay, so if you look at the what Graebus, usually it's the controller side that Graebus packetizes those transactions. So you cannot let's say you want to emulate a spy controller. You cannot do that because the spy controller, this Graebus spy controller is a virtual spy controller. So you cannot do devices like that. You can do emulate devices like only the client devices.