 Okay, let's start. My name is Sakari Ailus. I work for Intel in Finland. I'm happy to see so many people here, so welcome everyone. I'm going to talk to you about the cameras in embedded systems. And in particular I'm concentrating to the firmware interface, device 3 and ACPI aspect of the system. Typical if you have an embedded system with a camera, the system contains an imaging processor, a camera sensor, and the camera sensor or a camera module may have a voice coil lens. The system in the diagram looks like that. The ISP limit single processor is part of an SOC. And in the camera module you have the sensor here. And the lens voice coil and the lens itself sensor. And the lens voice coil devices are typically I2C devices. You could have other buses, but most of them are I2C. Both of these devices, because embedded systems often, they are battery powered. There are resources that are used for powering the map. There are regulators in this case. There is Vienna, VDG that are used for various purposes in the camera module. And then we have the clock signal here in this case. In the example it comes from the ISP, which is often the case. I don't know exactly what is the reason, but the hardware engineers tend to attach the clock output to the ISP, the mid-single processor. And then connected again to the camera module, we have Reset GPIO and I2C bus, which is used as a control bus for both the lens voice coil and the sensor. And finally the CSI2 bus. Some of these resources may be shared between the sensor and the voice coil lens. But it depends on the module construction, what it actually is. So there tend to be different kinds of solutions. So there is no one particular kind of implementation that would fit for all the use cases, that they need to be flexible. Going forward to continue the background of this, the raw sensors. They have very little processing logic in the sensor itself. Typicaly there is analog and digital gain, but not much more than that. There are two test images. This is how white looks like, so that each pixel has a single color. And if you want that to look like a real image, you have to interpolate. There is a test image with very little processing applied and with very bad quality algorithms to convert it. And it is also upside down, as you see. And it's about the imaginal processors. They convert the images that you saw above to something that looked like this. There is an image from the US Northwest Pacific Coast and a redberry. I don't know still what it is. And a display of Linux and Zimbian-based devices in a Microsoft office. And white. That's how white looks like after it has been processed using ISP. So if you look at the previous slide, there's quite a big difference between the two. As long as everything is all right in the ISP, that's how it looks like. It could look different if it's not. Then, a couple of words about the software interfaces that we use in Linux for the user space. We have a bit of a Linux API for capturing images. It supports video capture cards, USB webcams, and of course cameras in embedded systems. The API is the same, but the devices are different. The capabilities of the devices are very different. The media controller is a control interface for complex devices where you have an internal pipeline that the single video for Linux device node is not enough for. Provides you also pipeline discovery and configuration and device discovery. So if you have a complex device, you probably want to start by opening the media controller device to discover what kind of devices you have, what kind of hardware pipeline you have. This is an example of a media graph. This is from my actual mobile phone, Nokia N9. It's a pretty old device, but I'm using it as an example, but because it's relatively simple and the code is in the mainline kernel. There we have a sensor represented by these three sub-devices, and the ISP represented by the rest of the green sub-devices, and the yellow boxes that are the video nodes that represent the data interface. The sub-device interface is used for control, and the Viferal 2 interface is used for data. The ISP is driven by a single driver, as well as the sensor, and these are internally separate devices in the system. To proceed to the next slide. Each device is probed separately. The ISP is probably connected to PCI, or AMBA, or some other fast bus, and then in the same bus, or a different bus, you would have the I2C controller, and the sensor, and the lens voice call, so that they are internally separate devices that don't have really information on the other device at probe time. So how does the media device, how is the media device initialised? The call sequence is to first initialise the media device, then the rest of the component devices can be bound to that media device. There's Viferal 2 device register. This is just mostly internal to Viferal 2. You don't need to know much about it. It's mostly in drivers. And media device register, media register device is registering Viferal 2 device node. And then in the fourth step we seem to have Viferal 2 device register on sensor sub-device, and that is registering the sub-device, sub-device or sub-devices for the sensor. And then that is proceeded by the same work basically on the ISP side to register these devices in the media device. And finally the sub-device device nodes are registered and the media device register registers the media device node. So all is well at this point except that this part is run in the media device drivers probe function, and this part is done in the sensor drivers probe function. And it needs to happen in this order. And that doesn't obviously work. It actually did sort of in the old days when we had platform data, but the platform data has had a bit of bad reputation in the recent five or so years. So how does this work without the platform data? And we get to the next topic of the presentation which is the Viferal 2 async framework. This is a framework that facilitates registering the sub-devices, basically gets around that limitation which we saw in the media device registration sequence that is historically only done in the probe. The Viferal 2 async works by letting the drivers to register callback functions for completing the registration after the probe. So when all the devices have registered and then the Viferal 2 async framework has done each job then the device should be ready to be used in the user space. This is a sequence diagram. Fully you can see this. The text is unfortunately pretty small. The first, the device base calls the ISP drivers probe function. This is split in two slides. The first slide is about the ISP. The probe function runs here. And then it proceeds to the ISP device 3 node to parse the local endpoint nodes. Device 3 graph endpoint nodes. And based on that information it can obtain the remote endpoints. And obtain the reference to the sensor's device 3 node. And this information, it constructs the Viferal 2 async object for each async sub-device that it can found as endpoint nodes in the device 3. And then after it has enumerated all the endpoint nodes and obtained this information, it can register a notifyer with the Viferal 2 async framework. And then the Viferal 2 async framework checks the async sub-device list and sees whether there are any sub-devices. And because we haven't yet run the sensor drivers probe function which would have registered the sensor drivers sub-device with the async framework, we don't have any sub-devices yet. So we return with an empty result and the ISP async notifier is added to the async notifier list. And that is the end of the ISP drivers probe function from the Viferal 2 async point of view. So the registration will proceed later on. And now let's see the sensor drivers probe. Again, the probe function is called the sensor driver. Then we go to parse endpoint properties. The sensor driver will get to know some internal information. It needs to configure the CSI 2 bus, for instance. And then the sensor driver notifies the registers, the sub-device with the async framework. And the async framework then proceeds to match the newly registered information with existing information which contains now the ISP drivers async notifier here. And it finds a match and then calls the notifier's bound callback to the ISP driver. This tells the ISP driver that the sub-device that corresponds to the sensor DT node that it was aware of previously is now available. And it is registered to the media device. And once all the sub-devices, all async sub-devices have been registered like this, the notifier callback, the complete callback of the notifier is called on the ISP driver and the registration can finish. This may happen well after the probe, depending on when the drivers are getting loaded. Basically this solves the problem. And it has been there for some time, quite some time now, a few years perhaps. But there are not too many drivers perhaps, at least not the ISP drivers that would use this yet. Proceeding to the next topic, device 3. I believe many of you know quite a lot about this topic, but for the benefit of those who don't, I'll give you a summary. System hardware description in a human readable format. Pretty nice. And it originates from Spark and Open Firmware, but primarily is being used by embedded systems with Linux nowadays, especially by ARM, but I think also by PowerPC Spark and also X86. Not many X86 implementations, but there are some. It gives you a tree structure of nodes, and the nodes can contain other nodes and they can contain properties that are key value pairs. And before the kernel makes use of it, the device 3 compiler is used to compile it into binary form. The device 3 specification is maintained by device3.org. It's syntax and some semantics, but especially it does not define any individual bindings for devices that would actually tell the software how to use a certain kind of device with device3. So Linux device 3 binding documentation is part of the Linux kernel source itself. I'm not a free PSD developer and don't know much, but based on the brief research I made on the topic, it seems that the free PSD developers appear to be using the same bindings as the Linux, at least to some extent, which is good because then you could use other operating systems with the same device3 binary. Device3 crafts, behandled properties. In the device3 can be used to refer other nodes in the tree. That comes from the device3. So the device3 supports that. A port concept is an extension to that in Linux. It describes an external interface in an IP block or in a device, such as an ISP. So ISPs, this port, would be describing, for instance, a CSI to receiver that you can attach a sensor to, for instance. And an endpoint describes an end of a connection in that port. Endpoints have their own properties, for instance, the bus configuration that is needed to configure the bus that is used by the other device, referred to by the behandle. This is an example of that. I like it because it's from actual device3 source file and it's quite short. This is the sensor part of it. So here we have an... This refers to an I2C controller, and under that we have a node that describes the I2C device. And we have a compatible string for it. A couple of things that are not very relevant now, but then we have the port node. That port node describes the... Corresponds to the CSI to transmitter in the sensor. And we have one endpoint in it. So there is one connection in that port. And in that endpoint node, we have a remote endpoint property. And the behandle reference in that property. So this describes the link to the CSI to receiver elsewhere. Then the corresponding ISP part. We have ports node here because the ISP has several receivers. So this is just one of them. So the port receiver number two is here. And we have a single endpoint again in that port that we call CSI to AEP. So this is a name that refers on the previous slide in here. So in that port node, we also have an endpoint node. And in that endpoint node there is a similar remote endpoint. So there is a link from both ways. I think there has been some discussion one time ago that it would be enough to have only a single remote endpoint. So the same kernel would generate the backlink, but doesn't seem to have happened. This works well. I guess that's why it hasn't been changed. About the graph API, it allows the drivers and the frameworks to parse the port and endpoint nodes under the device node. Without knowing all the possible details related to that node. For instance, you can enumerate the nodes and obtain the remote endpoint based on the P-handle value. And then we get to next topic, which is the ACPI. ACPI is the Advanced Configuration and Power Interface. That's the abbreviation. It is very similar to the device tree in many aspects, but there are a couple of differences. I think the origins of both ACPI and device tree are around in the same time, mid 1990s or such. One of the differences with ACPI and the device tree is that ACPI is operating system independent, whereas the device tree binaries, at least historically, have been very operating system dependent. And the origins of the ACPI are in X86 and PEC. And that's where they primarily are being used nowadays. But increasingly they're also being used in embedded systems. The differences between regular pieces and embedded systems is mostly that in embedded systems there is a lot more variation between the hardware and you usually cannot probe. All of the devices in the system are I-squares devices that you simply cannot probe, so you have to know about them. And the information in this case would be provided by ACPI. One of the other differences between ACPI and device tree is that ACPI also provides power management in the device tree-based systems. Mostly I believe the drivers are responsible for implementing that. But in ACPI it comes from the ACPI. And that is done through ACPI methods. So those are actually runnable code and then there is a virtual machine in the kernel that runs them. The specification is developed by the UEFI forum and looking at the history. It seems that there is one release per year. The ACPI specification itself contains or is supposed to contain all the information that is needed to create ACPI tables to describe the hardware. What do you do if you get a random device that doesn't fit to any of the classes of the devices that are supported in the specification? Do you try to get it to the next ACPI specification? There is a solution to that as well, that problem as well, and it's called device-specific data. The device-specific data object type was added to the ACPI 5.1. I think this is about three or four years ago. There are two extensions to the device-specific data currently. One of them is property extension that keeps you key value pairs and then there is a hierarchical data extension that keeps you tree structure. That again can contain other hierarchical nodes or the nodes can contain properties. Together, this looks like very similar functionality that the device tree provides currently, but with ACPI. Then the next obvious question. If you can use this kind of constructs on ACPI, where do you document them? If ACPI is still supposed to be operating system independent, it would feel probably a bit awkward to ask developers of all the other operating system to go to the Linux kernel source code and look for the device tree documentation that they refer to and try to get patches in if they need something new. There is DSD property registry, which is intended to be a lightweight approach for registering DSD properties. But the scope of this is at least as far as I understand quite a lot more limited than the scope of the device tree bindings that we have in a Linux kernel at the moment. I think historically there probably has been no need to use too many properties in the ACPI because if you use it on a PC, ACPI itself already provides what you need and there is little use for properties or hierarchical data extensions. But this is one of the use cases where they would be needed in order to provide information to the software about the camera-related devices in ACPI systems. Then something worth mentioning also is the firmware node property API. It provides access to the properties independently of the underlying firmware implementation. There is device tree support and ACPI support. So you can access the properties independently of what you have underneath and the drivers don't really need to know that. And that makes the use of the DSD property extension on ACPI. And then this is up to now what we have and is basically in the mainline kernel so it's all nice and working. But then looking at what we would need in the future is what we are going to see next. FirmnodeCraft API. This is currently at the level of an RFC batchet and it is being currently debated on the Linux ACPI mailing list. It provides very similar functionality to 10.0 of Graph API being able to enumerate the ports and endpoints on the device tree-based systems but being firmware independent so you could use this on ACPI as well. The device tree implementation of this Graph API directly uses the device tree functions obviously and the ACPI implementations makes both use of the DSD hierarchical data extension and the property extension. And looking at the Vifral 2 what's also on an RFC level is the Vifral 2 firmware node API. So this is the Vifral 2 support for embedded systems based on ACPI basically. And it depends on being able to use Vifral 2 and embedded systems with cameras on ACPI. You need both the firmware node Graph API and the Vifral 2 firmware node API. So the two are dependent. And it basically gives you the same functionality exactly the same functionality on the Vifral 2 OF API but in a firmware independent form. And the Vifral 2 firmware node and the Vifral 2 OF are fully interoperable. So for instance if you have an ISP driver that uses Vifral 2 OF and the sensor driver that uses Vifral 2 firmware node the two can work together without any workaround. And flash devices we can describe flash devices in device tree currently. That's not a problem. But the kernel has no knowledge of which sensor this flash device is related to because typically you put the flash device next to the sensor. And if you have multiple cameras in the system which one of them has a flash. If you have multiple flashes is there one flash per camera or both of the flash lets related to a single camera usually so. But which one is not known. So probably a property should be standardised to tell the information to the software. And about camera module I have been talking about camera sensors and camera modules briefly at least within the past half an hour. What the software sees only is the camera module currently. But what we really have Did I say camera module? I meant camera sensor. The software sees really the camera sensor. All the sensor part which is a tiny piece in the module itself. There is a lot of other stuff in the module as well. The reason why only the sensor has been visible to software is that there has been little need for the kernel space software to be aware of it. It's very important in the user space though. If you have a system which is controlling the low level camera image processing algorithms in the user space which is practically always the case if you have if you are using a raw camera sensor and an ISP. The information typically that would be needed here is which sensor and the lens are related so that's a very similar problem which sensor and which flash are related. What kind of lens is there in the module? What is the voice coil spring constant? The voice coil lens drivers are controlled so that the controller lets you control the current which you are applying on the voice coil and the more current you apply the more the lens is moved off from the default position. The default position is usually needed so the more current you apply the closer you get in focus. Then is there an infrared filter and if there is a filter what kind of filter it is? Typically there is a filter, sometimes no. What's the aperture size? All of these parameters may affect on how you want your user space algorithms to behave. And another aspect in this power on and power off sequences. If you look at this, we have the lens voice coil here and the sensor here and some of the resources that they need to be for power at on are dedicated either for the sensor or for the lens voice coil but some of them can be shared and if there are shared resources they need to be taken into account when the device is being powered on. So actually you can't define a power on sequence or you cannot implement a power on sequence in general case, in software for sensor or for lens voice coil you have to implement it for the camera module because otherwise it could be that for instance in this case if you first power on the lens voice coil that all goes fine but then if you later on want to power on the sensor you maybe have lifted the recent GPIO and the clock needs to be enabled before doing that and then there is probably a delay between the two as well so you already failed power on, powering on the lens so what is needed is basically a power on and power off sequence for camera modules so this should be visible in the device 3 as well this is what I had now it's time for questions so you are talking about user space so Canon drivers are cool but what about user space? do we have something useful there? I'm a kernel programmer so what do you use for testing? you got the images you have something right? I mostly use couple of testing test programs but they are mostly used and usable for capturing images algorithms are a whole different sort of problem domain thanks any other questions? in the front I remember there was a specific SOC camera support in video for Linux why it isn't enough today I know this support has to be faced out eventually are you referring to the SOC camera framework? if you are by Gennadile Havetski I think the author Gennadile Havetski I think or at least he is the maintainer of this stuff sorry I didn't quite understand could you repeat there was specific support for SOC cameras in the video for Linux 2 why it is not enough today why it is going to be faced out? okay I believe you are referring to the SOC camera framework that was a framework I did well long time ago in Viferal 2 to support very simple devices where you have single sensor usually not raw bayer sensor why you be sensor with the small ISP inside that produces already usable images as such without control algorithms in the host user space and for simple receivers parallel or CSI2 that simply receive the image data from the camera and write that out to the memory that's the problem domain for the SOC camera framework and the interface is it used did not really match too well with the Viferal 2 what the Viferal 2 in general uses so you basically when you are implementing a sensor driver or a bridge driver or ISP driver you needed to choose whether you are going to support the Viferal 2 in general or a SOC camera that's not a very good starting point if you basically should have a class of ISP drivers and a class of sensor drivers two classes of both and then you need to match somehow so that's the history of the SOC camera framework so you mentioned that the DSD is device specific with a lightweight way of registering well we'll call them bindings as reference to DT bindings but a set of properties for the device you also mentioned that you have a way to represent a graph interconnecting nodes that is being standardised for ACPI but I assume that you are not going through the ACPI standardisation process so that's going to be device specific and express in the DSD, right? well, those are very good questions thank you how about it? the current prototype implementation makes use of the DSD property and hierarchical data extensions which by themselves are part of the ACPI standard but the standard doesn't define how do you use them so that leaves the question somewhat open and the DSD property registry I believe it's intended to be registry of properties or how properties are being used by different devices but what is the exact scope and what would be the best way to get this supported in more general for instance by other operating system perhaps also Windows there are good questions but my point is that if you define those properties that allow you to express the graph as properties in the DSD you have no guarantee that a given sensor vendor will use that no, there is no guarantee it is up to the so that means that you could have classes of ISPs and sensors that would not be interoperable it is up to who will actually write the firmware I have one more slide if I can find it here this is as far as I understand this is roughly the logistics of how it works I don't claim to be an expert on either but this is my understanding so on ACPI DSLC vendor perhaps BIOS vendor provide the BIOS to the system vendor and then it eventually gets from to the BIOS in flash memory either with motherboard or support websites or something but comparing to the device tree there is no Linux kernel device resource on the right side of this picture we hope that the picture at the left changes and someday we would like the DTS to be distributed by the hardware vendor in the bootloader, not in the kernel I understand that's been the hope but I think that this deserves a very important purpose of providing the right kind of information to the hardware vendors hopefully the hardware vendors will be looking to this I believe and the binding documentation can be found there as well but what you do there there's not actually a unified opinion on that some people want to distribute the device tree with the kernel, some with the hardware and you'll find people with both use cases and that'll continue for a long time I'm sure if there are no other questions thank you for listening