 Okay. Can you hear me? Yes. Okay. Welcome. Thanks everybody for being here. This is a talk on how to write modern video camera drivers for Linux. And I would like to use an example as a well-known framework which is going through a removal or deprecation process to show how the system we work on has changed since the time set framework has been implemented. And hopefully I would like to give some suggestions and examples if you have drivers depending on set framework on how to make them or remove those dependencies and have your driver working for next kernel releases. So these are a few of my contacts. My name is Jacobo. This is my email address. This is my IRC contact. I'm an embedded Linux and free software developer and I work as a consultant and I've been lucky enough to work with the excellent renaissance mainline kernel team in the last two years, which gave me the opportunity to contribute to Linux in a kind of a regular way. I would like to thank renaissance, of course, for sponsoring me and supporting in giving this talk and these activities. And that's the talk outline. So we're going to look at what's happening at SSC camera, which is the framework that I've been talking about, and what has changed since the day that SSC camera has been implemented. The main difference is that how system boots, because we have moved from a world where system boots are using more files to system through two firmware supported boot process. And that changes the way we discover and probe and creates devices. Power management has changed as well. That depends on how the image capture devices are showing up in user space. And finally, I would like to give a practical example of a driver that was developed for SSC camera and that's been made a more vivid for Linux driver in recent kernel releases. I would like also to introduce a bit of glossary because words are sometimes confusing. And to capture images, we need, of course, an image sensor that produces those images. And the receiving port, which is usually installed on the SSC. A sensor driver controls an image sensor and the bridge or receiver driver controls the receiving port on the SSC. On modern system, we still have an image sensor that produces images and an image receiver, but we also have several components on the SSC that takes care of image transformation and manipulation. Image sensor and drivers for those kind of components are generally called video sub-device drivers. Let's start by discussing what's happening to SSC camera and start by saying that SSC camera was great in my opinion. Can I ask how many people you have ever worked with SSC cameras? Okay, just a few, but... So if you work with that framework, you know, it was great because it provides you nice abstraction away for the crude V4L2 API, which might be kind of scary if that's the first video driver you have to write. It's kind of scary to deal with all the complexity of the V4L2 API, all the IUCT else you have to take care of, buffer allocation. And SSC camera obstructed all those things away in a nice way, and that's why it was adopted in a lot of drivers in Mainline. And I don't have statistics for that, but my feeling is that in BSPs and in downstream kernels, it was kind of everywhere. All the BSPs kernel I've been working with, if they have a camera driver, it was based on SSC camera. And it was so adopted because it has good points, like, as I've said, it provided nice abstraction away for V4L2. It's the same framework for writing bridges and sensor drivers. So you learn one framework, you write two drivers, that was nice, and also provides an easy way to link bridge driver to sensor drivers because the two of them had to be linked in order for the bridge driver to call separation on the sensor one. Of course there is a bad side, since we are removing that, there is a bad side, of course. And SSC camera was developed in a time where system booted through both files, and the support for OF or device tree, and now the ACPI, which is gaining traction in better systems as well, is limited. It isn't there, I know, but it's limited and we're going to see why. It uses a set of deprecated operation, which is a fixable thing, but while the V4L2 API evolved, the SSC camera framework using those APIs has not been evolving the same way. So this is fixable, but it's not been done so far. And more than everything else, the media controller and subdev API that has been introduced, like, five, six years ago, are actual gain changes because they change the way that how image capture devices show up in user space, so SSC camera haven't really kept up with that. And what's happening to SSC camera? SSC camera, it's deprecated for a long time, so you are suggesting not to use that for right drivers, but it's going to be removed, finally, because it's a long time that talks about that. Possibly in the next kernel release. The SSC camera bridge driver has been removed, has been important to be a video-filling driver last year, so there are no more platforms that depend on that framework. Although there are some sensor drivers, in order of tens, ten of them, probably, that have not been imported yet, and they're going to be possibly removed. There are discussions this day, moving them to staging or removing them completely. It's possible they're going to be removed completely. And that's the file organization. We know that platform bridge drivers are usually in drivers media platform and SSC camera drivers are drivers media platform and SSC camera, and here we have no more dependencies. While for drivers media I square C, where sensor drivers are, we have some of them which will be removed. Currently, Mayline, we have kind of a confusing situation because we have two drivers from the same device, which is kind of confusing, but from next release, this is going away. So there are drivers here that needs to be ported here, and that's work to do. If somebody would like to contribute to that, it's a nice thing to do. What has changed, then, since the time when SSC camera was implemented? As we said, the device discovery and linking mechanism has changed. Nowadays, we do that using notifiers and async matching. Power management has changed as well due to the way how the device are exposed in user space. And we now have standard frameworks for clock and regulators. And so SSC camera doesn't use deprecated frameworks for that. So every time it's possible, we should use the standard frameworks for dealing with those two things. Let's start talking about device probing and have a look at how the device probe was performed in the legacy way. So we have five components here, board files, bridge driver, SSC camera, sensor driver, and of course the video for Linux 2 framework. The board file is nothing but the plain C file that register devices and drivers one after the other, all devices in the system. And at a certain point, it will add the platform driver for the bridge driver. So that causes the bridge driver to probe. At the end of the probe section of the probe function, the bridge driver will probably register itself to the SSC camera framework. That causes the SSC camera to do only initialization operation and at a certain point, it will start registering I2C devices. How does it do that? It does that using a V4L2 function, which is this one, V4L2 I2C new subdev board, which creates a new I2C device that causes the sensor driver to probe. And how are those two identified? Well, the board file knows the I2C bus number and the I2C address of the device and passes it down in the call chain until here, where those two information are used to identify the sensor driver. So in the old world, we have that devices are identified by I2C addresses and more important than everything, the device probing is sequential. So we have the bridge driver probing before the sensor driver. And that guarantees that every time a sensor driver probes, it has a bridge driver to connect to. In the new world, we have moved to a firmware-based boot process. So nowadays, devices are creating parts in the firmware description of the system and the devices are not identified anymore by I2C addresses by the firmware node references. And again, more important than everything, there is no guarantee anymore on the probing order of the drivers. So this is a DTS and in the DTS, we have a description of the video input port here and of an I2C bus. On the I2C bus, there is a sensor identified by an address and the system builds, Linux builds and start parsing the DTS until it finds the video input port nodes. That causes the bridge driver to probe and at a certain point, it starts parsing the I2C bus, which creates the sensor driver, which probes again and can safely connect to the bridge driver. But we can also have the other way around. So the I2C bus is registered before the video input port. This is probed first and the sensor driver probes completed its probe operation but finds no one there to register to. And that might be a problem. It's actually a problem because now device probing is totally asynchronous. We have no guarantees which is the probing order. And again, we need to identify devices by the firmware node references. How to do that? Well, V4L2 framework to the rescue here because it has two components that are designed for helping you, help drivers doing that exactly, which are V4L2 async and V4L2 FW node. How do they work and how drivers use them? Well, we have a bridge driver again, DTS and the two framework components. And in DTS, we have a description on the input port and the output ports of the bridge driver, which has two ports connected to two remote endpoints, which are possibly sensor of sub devices driver. The bridge driver probes and uses V4L2 FW node framework to parse the DTS and collect references to the remote endpoints. Those two are collected in the form of V4L2 async subdevice, which is an abstraction provided by this part of the framework. And those two devices are collected by the bridge driver. What does the bridge driver do with that? The bridge driver stores them in what is called a notifier. It's actually V4L2 async notifier, which is provided by this part of the framework. A notifier is nothing but a collection of firmware node references the bridge driver or generic driver is waiting for. V4L2 async maintains a list of all notifier registers in the system. Now we have three, four in total, which is kind of a likely in the system, but it's possible, totally possible. And the bridge driver does nothing but register its notifier with devices is waiting for to V4L2 async. V4L2 async maintains as well a list of waiting devices. These are devices or sub devices that are not yet probed and no one is waiting for them. So they put in the waiting list. At a certain point in time, we have that the sensor driver probes eventually and it uses V4L2 FW node to parse its local endpoint and create a V4L2 async subdevice representation of itself. It will then register that to V4L2 async, which adds them to the list of waiting devices. But the two of them get smashed, so there is someone waiting for this sensor. When the two of them get smashed, that causes the V4L2 async to call a callback on the bridge driver that bounds the subdevice to the driver. So in this way, the bridge will have the V4L2 async and handle a reference to the sensor driver. The second, okay, we are waiting for two sensor drivers and the second sensor eventually will prob in future does the same things. Usually V4L2 FW node create a V4L2 async subdevice representation of itself and register that to V4L2 async. That causes the same device to be matched and the subdevice bound and the bound callback to be called on V4L2. So in this way, the bridge has reference to both the sensor driver is waiting for. There is another thing I have not shown here, which is there's not only the bound callback, there is this thing called complete callback. It's usually called when all the subdevice notifier, all the asynchronous subdevice the notifier is waiting for have been registered, the complete callback is usually called year. The complete callback usually creates all the user space representations, V2 device node and V2 subdevice node connected to the wall capturing infrastructure. There is this question going on nowadays, if it's a good things, if you have eight, let's say you're waiting for eight cameras and one of them is not probing, do you want your system to be working or not? So should complete be called only when all these subdevices have probed or sometimes it's a good thing to have a working system even if one of your subdevices or camera phase. There will be discussion about that in the V4L2 meeting on two days from now and let's see what's happened. Of course, that's what we show so far is the situation where the bridge driver probes first, but we wanted to solve a problem which is the asynchronous probing problems. So the sensor may probe first. So the sensor probes uses V4L2FW node to register its async subdevice and that gets headed to the waiting list. Nobody is waiting for them because there are no notifiers waiting for this subdevice, but in a certain point in the future, the bridge driver probes and will register a notifier waiting for this device. The two of them get matched and the two of them get connected. So we effectively solve the problem of async probing sequences using those two framework. And the ones of you that knows that the camera knows that the camera can do that actually does that it uses those two framework. And so why it what has changed since then? What has changed since the time where SFC camera has been implemented is that now subdevices can have notifiers as well. This has been introduced one year ago by Nicholas, Sakari and Laurent, which are the main author of the V4L2Async and V4L2FW node frameworks to support the Renaissance RCR CSI in infrastructure, which has subdevices that are connected to sensors. So we moved from a situation where we have a receiver which has a notifier and connects to a subdevice to a situation where a subdevice can have a sub notifier. And eventually that sub notifier will be connected to other subdevices. This can create a chain of arbitrary complexity. It's usually just one of two levels, but there's nothing preventing you from making more complicated things here. And right now I think a couple of drivers and main lines are usually that IMX, well, RCAR for sure, but also IMX is now using subdevice notifier. And it's expected that more devices will use this abstraction as well. Power management. As we said, power management has changed as well due to the way that video devices are now represented in user space. And that depends on the way on the introduction of media controller and subdev API. So media controller, the old device, the old world, non-media controller keep devices, they work with a single device node abstraction. So the one that we were all are used to, the dev video zero abstraction. So for a wall capturing infrastructure, you just have a single device node in user space. And that causes all operation to be sequential. They goes through a single device node and gets directed to the subdevice. We now live in a world where media controller is everywhere and it's going to be everywhere hopefully in the next year. And video device node are not the only abstraction we have in user space because we have also video subdevice node. And that causes all operation not to be sequential anymore, but instead they can be perform on subdevices and video devices at the same time. So let's see an example of that. This is a legacy system where we have the simplest possible capture infrastructure. So we have a sensor that is connected to an I2C bus and it's connected to a receiver port where it transmits pixels. In kernel space they will be managed by a receiver driver, a sensor driver, that's the framework part which is coming to the kernel frameworks. And the user space, we will have just a single device node abstraction. So all this infrastructure is represented by a single device node. We have, of course, a video for L2 compliant application and which interfaces with all that with the V4L2 APIs. Usually at the first thing we operation we have to do if you want to use the video device, it's to call an open on this video device node. And usually at set power at open time the bridge driver just powers up the sensor. So in that way, every other operation, the sensor driver, it's ready to send pixel to the receiver. So the V4L2 compliant application start calling different IOCTLs, set formats, get formats, allocate buffers, whatever, and a certain point. And all the operation will be translated by the receiver driver to the sensor driver using a V4L2 subdev call operation. At a certain point, we will receive a stream on. So the application won't actually the sensor to stream pixels. And this causes a lot of settings to be sent on the ice. And pixel we start flowing in this direction. This is what this is what a modern device might look like a very simplified actually modern device might look like. So we still have a sensor, we still have a receiver port, but we also have a lot of components on the SoC that takes care of image transformation and manipulation. That can be resizing conversion between one format and another formatting system memory depend totally depends on your platform. And of course, the drivers that handles that are much more different from the legacy one, and they might have different components, they you may have one ASP driver, handling all three of them, one receiver driver, that depends on your platform. But the important thing is that this is how it would look like in user space. So we still have the video device node zero, which the application uses to start streaming and call a certain set of operation. But we also have all these device nodes here, which are subdevice nodes, where the application can call subdev operation as well. So this is what may happen, we may have video, we have subdevices I use it, yes, call on video subdevice node at any time. And there is no relationship between one and the other. So we might have those two call at different times. And that causes all of your operation to be now asynchronous, because there is not a shared notation of power settings anymore along all this pipeline. So the only thing, the only suggestion that I have for if you are to implement a driver in this kind of situation is always cash your settings every time, because you never know the power state your driver is working in. You may receive a set format and your sensor might not be powered at all, because we don't have the single entry point we add when working with no media controller systems. That calls for maintaining a driver wise power state notation. Every time you should know if your device is powered or not, and even better if you want to make that ref counted, it's even better because if you receive two set power that should not happen. But who knows what user space is doing. You should receive two power off to to actually power off the device. Also, you should cast cash or your settings and apply them and the time where is known the sensor or the sub devices to be powered. And that's usually stream on time, because when you receive a stream on, you should start sending pixel. And at that time, the sensor should be powered on. Also, this is general suggestion for not just for video devices, but try to use runtime PM runtime PM makes provides even abstraction that it's more similar to the sequential flow of operation that we've seen before. So it is a very easy development and ref counting of power states. Of course, it's not always possible, but it's welcome. clocks, GPUs and regulators as said, we have frameworks for that right now, and they should be used whenever possible. And relating to power management routine in the legacy world, we have the board file that provided power management routine to the sensor. So this is how SOC camera used to do that. There is an SOC camera link with a power callback and the board file just filled that pointer with a routine defined in the board file. So when the driver needed to power up on power off the sensor, it call this things here and the board file as all the references to regulators, resets, whatever you need to power on and off the sensor. Of course, we don't have board files anymore. We have DTS or ACPI. And how would you do that? Well, you should use the GPIO clock and regulator frameworks. Every time they interface with the DTS, you collect references from firmware, which is usually called DevM, or not DevM depending if you want to use DevM clock or regulators get and use the name using DTS. And the driver itself should not rely anymore on the board files, turning on and off the single components, but is the driver itself that should enable or disable the regulators or the reset line as power time? Practical. So this is an example of a Vita driver that was developed using the SoC camera. And in recent, I think, two releases ago has been imported to be a plain Vita for links to drivers. So I would like to go through all the path, not all of them, but some patches that and show you what are the steps that has been performed to do that in order. And if you have any driver depending on the SoC camera and you want to port them and possibly submit them for inclusion, that's a kind of guidelines for doing that. So the first thing we did was just copy the sensor driver as it was from SoC camera to the Vita for Linux 2 sensor driver directory. That was a choice that allowed us to see the differences without having any modification in the first comment. Then the first comment actually removed all the dependencies from SoC camera from the driver. And that's exactly what we have been talking about handling clock and GPIOs in the drivers and not relying on the board file for doing that. Register the async sub-device because SoC camera was doing that for you. And now you should do that explicitly in your driver. Remove SoC camera specific deprecated operation. Vita for Linux 2, this operation are deprecated. SoC camera still depends on that and still wants them. So they had to be removed and re-implemented in the proper set format and get format operation. And then there are a few changes which are specific to this driver, which are specific to this driver, so I'll just hear for reference. Of course, the build system has to be adjusted, but that's trivial. And then after a plain Vita for Linux 2 driver has been made out of the SoC camera, depending on the fun begin, because people actually start using that. And that's exciting because patches are start coming. So people actually start using that. And specifically patches have been adding components, parts of to that driver that made the modern Vita for Linux driver out of what it was an old one. So the first thing we saw was having media controller support to this driver. That means that the driver now has a sub-device, as a sub-device, as a sub-device in user space. So you now need to handle nested set power calls. That's exactly the thing that cache your setting and keep a power state in your driver. Also, you should not access registers when the sensor is powered down. Now you can receive a get control from user space while your sensor is powered down. And you should pay attention not accessing the sqrc busway the sensor is powered off. There was support for frame interval handling. This is kind of a request right now if you want to submit a Vita for Linux driver, frame handling is something that is kind of mandatory doing that at least for a few frame rates. That's another thing that calls for shared state notation. So if your driver is streaming, you should return ebz or another rareflex if you want to set format or change the frame interval. And the last change, it's the creation of the sub-device node that goes along with the support for the media controller operations. So I've been probably too fast. So we have 10 minutes for questions. I hope you have some at least because otherwise I've been really too fast. I had a hundred slides. I was worried that the time was not enough, but actually I've been probably talking too fast. If you have any question or anything you want to talk about or any discussion about how things could evolve not just for SOC camera base driver, but sensor driver in general, there are two microphones here. So please go ahead. So I'm just wondering when you're writing this new code, you're adapting it, what sort of techniques you use for debugging and working what's going wrong because it strikes me this is a more complex setup of everything being asynchronous. And when it's not working, you might not have much idea as to which bit isn't actually hooking up. So you get hungry. That's the first thing. I mean, you get disappointed when things doesn't work. That's the first debugging tool you use usually. And well, that depends totally on the system that you use, but things are now asynchronous. So having a notion again of what is the power state that's always useful. And talking about streaming start and stop that's and now handle it through the media controller frameworks. So you have a pipeline notation, a pipeline, it's where is that? That's it's basically a pipeline. All the components here are put in a pipeline. And when you start streaming the media controller frameworks goes one on the other and calls start stream on all of them. So having the bug in adding the enabling the bug in the media controller frameworks help you understanding what's going on at each step of the capture process. But in the end you should just know what's happening in your sensor driver. If you have problem streaming depends on what problem you have. You don't receive images. You receive bad images. You are missing a set format call that depends on the driver usually. And what the bug in tools your system provide because compared to other parts of the system it's hard to debug those kind of things in JTEC. Because there is an sqrc bus in the middle so you should the last resort is printing out all the messages you are sending on the sqrc bus, printing all of them, see what happened, go and go with the data sheet and do the comparison. So there are different degrees of complexity you may want to handle. I don't know if you have a specific use case for that. Well, maybe it's not. We have time. Five minutes afterwards so that's okay. Okay. Thank you. Anyone else? Okay. So thank you. More time for coffee.