 Hello everyone and welcome to listen my talk on the subject of camera's device 3.0 API, a device driver perspective. We are going to have an overview of the firmware interfaces in the area first and then see how this works, how these are being parsed and eventually what this multitude of interfaces means for drivers. First, there are basically two kinds of cameras, two main classes of cameras found in today's mobile phones and laptops, USB webcams, those are external devices and traditionally in found in laptops and then there are what we call complex cameras, the typically consist of a camera module and the CSI2 or VPCI2 or parallel bus receiver to transfer the data out of the camera to the SOC in the system and an image signal processor where the data is being processed and virtually all mobile phone cameras are like this but these kind of cameras have found also their way to laptops which places a bit different requirements to them as well. USB webcams, if you have connected one to your laptop you may have found that or if there is one already there you may have found that it works with a UBC driver and that driver provides a VIFRAL to interface applications can readily use that so that's basically all the software integration you need for USB cameras so that's simple enough but then when you have complex cameras this all this hardware that basically in some form exists inside these USB cameras is now exposed to the operating system. You will find here in the SOC couple of devices that are needed to control and supply power to the camera module which is found here and then there is the ISP there almost always nowadays it is part of the SOC and there are a number of wires, voltages, reset signals, I2C for control, CSI2 bus for data transfer and obviously a clock. The camera module typically does not have a clock signal generator itself and the camera modules themselves contain a camera sensor and then there is the lens voice coil and the lens itself and the lens voice coil is used to move the lens there and then you also may have a flash driver and a flash LED and this is an old photo of a camera module and this image here is an example of an image taken on such a camera on the mobile phone there was not very much light but still it is not a bad image. Don't try to use regular USB webcam in low light conditions, the images will be horrible I suppose there are better ones too but they are rare and then a couple of words about processing the data from such a camera first usually you receive the image to the system memory and then the image signal processor reads it from there and processes it and writes it back to the system memory. Some devices don't have this first memory loop at all but they instead read the data from the camera sensor directly and then in the user space we have all camera control including camera control algorithms so there is a feedback loop which uses the statistics produced by the image signal processor and then come up with the new configuration for the processing parameters in the ISP as well as the camera sensors exposure and gain typically and this has materialized in form of lib camera which is very nice because there wasn't any kind of standardized solutions especially in open source solution in this area. If you look at the view of how these devices typically would look like in a bus so you have or in a system firmware interface you would have a CSI2 receiver here and there is an I2C controller and the I2C controller has it as these child devices the image sensor as well as the lens VCM. This is the situation without any support for complex cameras and for software this is not going to be very useful because these devices for instance they are obviously connected together they are related but this doesn't yet provide that information so you have basically a CSI2 bus here between the CSI2 receiver in the SOC and the image sensor and the image sensor is obviously associated with the lens VCM device as well. If you have a single camera in the system this might not be an actual problem but typically if you have cameras you nowadays have many of them so this information is mandatory and there is also configuration for the CSI2 bus itself the information that there is the CSI2 bus is not alone enough you need to know the number of lanes how the lanes are connected what kind of pi there there is pi type because nowadays there are multiple and if you have a parallel bus you also need similar configuration for the parallel bus then I'm going to discuss the firmware interfaces in this area a little device tree that is basically a system hardware description that typically mainly used in embedded devices originates from spark and open firmware and nowadays used in a large number of different architectures basically provides three structure of nodes and properties and source is compiled to a device tree binary before it is being used but the source code is human readable and editable and usually is edited that way and on camera support in device tree so that's quite old it first appeared in 2012 it has been extended since but basically it has been around since 2012 there is quite quite good documentation for it and a lot of examples in form of device tree source in kernel as well and then on acpi is the advanced configuration and power interface it's the firmware type that is virtually used in every pc nowadays and this operating system independent in principle device trees as well but sometimes bindings might not be acpi as device tree provides this device discovery and enumeration but importantly the important difference is that it also implements power management whereas device tree it is done in software and in acpi you can also have a AML code run in an acpi virtual machine so there is some room for hardware control as well at least for instance sometimes used for implementation of that power management and further on acpi there is dsd it is an abbreviation of device specific data and that basically gives support for device tree like nodes and properties through these dsd objects it is very similar to device tree nodes and properties but there is one difference that is birth mentioning i think and that is that there are no p handles if you need to refer other other nodes in the tree you practically need to use string references to these nodes and whereas on device tree if you have if you have spelled your p handle name wrong the compiler will complain about it but the iasl the compiler for acpi does not do that so there is run but there so there is only runtime validation i put also the url where this is documented just like if someone someone is interested in that and then the first implementation of supporting camera in firmware system firmware for acpi was added in 2017 so relatively recently compared to the device tree support it is also documented as part of the kernel source this implementation is linux specific but it otherwise uses device tree definitions so it's very very similar and this implementation is used by many x86 chromebooks and secondly there is also a different variant of acpi support but that is ipu3 specific it is actually used on the same hardware as the previous definition but on computers that have been shifted windows it's a bit unfortunate situation but nevertheless support for this has been added in 2019 daniels kalli has done great work on that the information available from the firmware in this case is a bit incomplete so the some of the information is kept in drivers as well which gives you a bit uneasy feeling is this going to be right am i going to select the right set of information here right regulator configuration for instance and this differs from how things are supposed to be made in acpi also in the sense that it leaves the power management for drivers as well so basically if you need if you have a skylake or kb lake system that has a raw camera and it has been shipped with windows then this is the the firmware format that is found there and then a further acpi object type crs current resource settings if i remember correctly and that is used to describe various things such as serial buses including csi2 bus so it basically tells how the csi2 bus is connected but crucially it is missing some features of the bus that need to be known to the software so so you will still need a dsd so both of the previous formats have been either operating system or hardware specific but soon we are also going to have a new definition which is hardware independent and operating system independent it's called mipi disco for imaging which is a short of discovery and configuration so people sometimes wonder the name this is a part of a series of mipi specifications defining acpi firmware interfaces for mipi hardware standards and it relies on both crs and dsd acpi objects and it is expected to be released later this year so it is not yet out and then let's let's look how this information gets parsed in firmware so this is the problem that driver developer faces mostly in this kind of situations so first you look what kind of firmware there is and then you parse it parse the information from that firmware but we have three different definitions for acpi and one for of and there are a lot of drivers for device tree it's simple it already works with the existing kernel infrastructure but with acpi that that has not always been the case if you have a Linux specification or definition of acpi firmware that you are parsing it relies partially on binary data structures that can be a bit annoying to parse and there is little or no similarity with device tree in this case and so the parser determines the type of the acpi firmware in question based on the dsd data nodes that that that exists as well as acpi objects found so if it sees the non-standard objects that are specific to that Windows definition it will assume that it is using that and on beautiful Linux framework, beautiful Linux firmware node framework it is basically parsing almost all the information that is relevant for complex cameras from firmware and it has its origins in the referral to of framework so basically it has been extended and the of api has been replaced by the firmware node api so the device tree backend of the of the firmware node api still accesses the information from the of framework and basically this also means that it assumes that all the information it parses needs to be in the device tree format or at least in the memory structure the referral to firmware node endpoint dialogue parse is the main function that is used for parsing this information and this is called by both camera sensor drivers as well as the the csi2 receiver drivers and in the end you will need to call referral to firmware node endpoint 3 because this also allocates resources and sometimes driver developers forget that a couple of words on the on how this information is parsed from the linux specific definitions so the firmware node api backend is being used and for this purpose there was also graph support added for api so it has very similar graph support than device tree has nowadays as well there are a couple of additional checks in drivers acp properties that are just there for the camera definitions but apart from that it's just like parsing the device tree then a new concept software nodes they were added to the kernel in order to provide drivers with information that typically comes from firmware but doesn't always come from the firmware at least not directly and they're mostly added by other drivers but maybe also added by kernel framework saleswear and so there is also a software node backend for the firmware node api and on the ip3 specification there is really no similarity in device tree definitions there are a couple of places where the parsing code is found basically that parsing code instantiates regulators clocks and additional additional software nodes in order to construct the device tree like data structure and on mipi discover imaging basically the parser would need to parse the CRS binary data structure as well as to look at the dsd nodes and properties in order to parse this and then recreate a similar device tree compliant data structure where this information is placed so as a summary here's how it looks like so we have device tree there's nothing special it's straightforward as well as the link specific acpi definition but then you will need additional steps if you have one of the other two so I would say that even though this link specific definition is somewhat well it has been around for quite some time but it is the easiest to to edit so if you for instance have a development board and you want to add support for a camera there then I would go for this because it's so simple whereas firmware developer should be using the the disco specification because that is the most generic it will work with linux and hopefully windows developers will start also using it because so there will be a single definition that both are happy with here's a bit simplified driver's probe function example so first we get the firmware node of that of the device and then there is a struct referral to firmware node endpoint which is used to contain the biased information from firmware you first need to specify the bus type in this case it is the mpd5 there are bus types available and then you call a function to to obtain an endpoint node from your device and here the number here is zero but that is the port and that reflects the csi2 port so if you have a single single csi port in your device then the number is always zero but if you have multiple then it may be something else but it's up to the device what to what to put there and when you have the endpoint then you can call referral to firmware node endpoints alloc bars to depart the information from from there and it will be it will be placed to this to this struct and then usually at this point you can already release the endpoint this is also somewhat something that many drivers forget and then here you have the configuration stored in the endpoint and eventually discussed previously and it needs to be released and if you look at the referral to firmware node endpoint struct there is the base field that basically contains contains the endpoint id as well as the port id there is also the the firmware node of the endpoint and then the bus type is is maintained here if if the parser here happens to be parsing and happens to see an endpoint that doesn't correspond to this bus type it will it will throw an error so you no longer need to check this but it is there in order to maintain that information in this struct as well because there are three structs under the bus field each for the different bus types there is parallel and then there is csi one that's pretty rare nowadays probably well probably all the hardware is at least 10 years old or something and then you have csi two this is the most common these days and at the end we have also link frequencies most of the firmware definitions provide you a list of allowed link frequencies for the hardware so that it is not only for the central driver to decide these are the link frequencies that are known to be functional on the hardware so that they don't cause a harmful interference with other devices there and on the csi two bus configuration there are a couple of flags but most importantly perhaps data lanes and clock lanes this is both selecting the lanes and configuring their order depending on the device typically for central drivers your clock lane is zero and the data lanes are from one onwards so most devices are like this and then have lane polarities in case the polarity of the lane has been changed may sometimes make circuit board design easier and on the future work so basically things are pretty good nowadays everything is almost almost working there are a couple of things that could be done in the future to improve improve the or to allow making the user space camera stack more generic so that you would not need to keep information on the local storage on the hardware itself so that you would get that all information from the firmware interfaces and here we need to think of the camera module because the camera module currently is is not part of the the firmware interfaces because it's just a collection of pieces of hardware but it tends why by itself there is nothing you can you can control there you can't change it anyhow and so so that has that has been the reason why why it hasn't hasn't been there so far but the camera module still determines many properties of the of the of how for instance the ISP configuration is affected so so for instance it affects how the the lens shading table needs to be configured you typically for instance have a big netting effect so so that the captured image from the sensor if you have if you don't have lens shading correction it will be darker in the edges for two reasons typically there is less light that gets there and also the light that gets there may not be used as efficiently and more of that light is lost on the edges as well and then there are things like what are the ranges of current that you can apply to the to the lens to keep it moving between between the desirable distance from the camera sensor so that you are not for instance hitting the hardware stoppers because that produces a clicking sound and it is of no use for for for imaging purposes but yeah or actually mentioned that this yeah lens shading table and other parameters already so so we have discussed this topic in the media summit on monday and what was proposed was to but the name of the camera module to the to the firmware and probably this information is best still itself kept outside the the firmware itself for for easier updating and then some of this information is like the like the vcm currently meets it is it is only suggestive so if you happen to have for instance another batch of devices it might be a little different than than than it was previously and then on power management on acpi and device 3 based systems acpi in acpi it works through two ways you have psx methods as well as power resources pr something and in device 3 the device driver itself is responsible for power management of the device and also acquiring the resources that are needed for it and additionally runtime pm supports may or may not be enabled at compile time depending on the config pm option so this might as such not be a major issue but it tends to cause a number of a number of different combinations of configuration for a sensor driver developer to to to to to take into account in the driver and it it's very hard to get that right if you're not looking at driver that that that does it's right to begin with and it is it there are not so many of them many drivers are actually either supporting device 3 or acpi but increasingly there are drivers that need to support both but this this problem is made worse by powering on i2c devices if they are when they are pro from acpi based systems so when you're entering drivers pro function under device 3 base system the device is off but if you're on acpi based system it's powered on so so this is causing additional complications that's driver drivers and this problem is mostly visible in camera sensors but it is in principle touching all the i2c drivers that need to be working on device 3 and acpi systems there is also the use case is not perhaps very interesting no there are privacy leads found in laptops and typically typically when driver programs the device that the driver is is controlling is powered on but in the case of camera sensor this camera sensor is also wired to the privacy light so when you power on the computer the privacy lead will blink because the camera sensor was powered on so this this is obviously for non-technical users a very concerning matter so so so all that is being done is that the driver pro function checks that the sensor is there but to address this problem there is a property defined for acpi with this which is linux specific though but still if that property exists and a couple of other flags are in place the the sqrc framework does not power on the the sensor on and this makes makes it quite quite difficult yet more difficult for sensor driver developers to get it right but maybe similar functionality will be added to the device 3 as well so to reduce the combinations it would help if the acpi framework did not power the device on while it's actually done by the i2c framework if it's an acpi device and yeah that's that's that's about the future work there is an example still on the on the device device pro function so there is the acpi dev state d0 which is basically telling whether the device has been powered on on acpi systems it will return through on on device 3 based systems so then you're going to check that full power yes it's if it's true i used an integer to keep keep this a bit shorter then you call power on device now this is a bit confusing because we just learned that the device was powered on um but this actually depends on firmware interface so that oops firmware interface that if you're an acpi system you get that uh you get that device powered on there but with device 3 based system you need to do it here and so if everything went fine then you proceed to identify the device usually check that it's in place and call pm runtime set active which is telling the runtime pm framework that that the device has been powered on and then enable runtime pm and finally call pm runtime idle so that the device that the the device can be powered off or suspended this doesn't seem very complicated as such but there are many many ways how how driver developers can can can get this wrong so hopefully we can we can simplify this and will also benefit other users of or other developers of drivers who need to have the devices working both on acpi device 3 that that was my presentation oh thank you for listening are there any questions yes yes it it doesn't mean this 1970s dancing well um well well if you if you get new new new firmware uh from somewhere that contains all the necessary definitions in in in the mp tisco for imaging format then then yes it solves your problem but i i think it is solution of or or or will be will be found in future systems rather than the current firmware being replaced with that you you might still see that a bit but but i my wish is that at some point that will end i i think that ip u 6 may still have something like that yes i i would i i would say that they haven't really had an alternative well or if you if you're vendor you you sometimes tend to solve certain things as as easy way for you as possible and this is what the binary data structures are for but but at some point of time once they have switched to disco for imaging i wouldn't expect to see any further instances of these binary data structures non-standard binary data structures anymore acpi before disco for imaging there has been no acpi no standard acpi way to to describe these devices yes well you could do that for sensor drivers but many sensors also include e-proms that are controlled by this 8024 driver that is used in a lot of places so so you can't just remove that from from this driver you could do it for sensor drivers sure but not from the 8024 driver i would i would power is on unless unless there is well an indication from firmware that it should not be because because if you have a problem it's still better to find it that driver probe time than than then later any more questions all the questions have come from this direction anyone else oh thank you