 Okay, hello, hello. Is it working? Sorry, it took some time to set it up, but here we are. So my name is Jakapo, I'm going to talk about complex cameras and now we are trying to make it less complex. So my name is Jakapo, I'm an embedded Linux software engineer. I mostly work on integrating camera systems and multimedia devices. Here are a few of my contacts. Here you're welcome to ping me. And why we're given this talk, why we're doing this work in first place? That's because the increasing complexity we are used to in embedded systems regarding the camera acquisition process, it's coming into canonical computing space like laptops and tablets. And Linux systems have a problem with that. Are those Linux systems ready? No, they're not. And so we have a problem and we're trying to solve that. It's pretty not necessary to state either this here in this room, but digital imaging is a complex task. It's not complex that it's complicated, but it's computational expensive. It's computational expensive during the acquisition phase because you go through all the debiring and interpolation and pixel transformations. It's complex when you have to do image quality tuning like auto balancing, white balancing, out exposure, whatever. And we usually work with very big buffers like not that big resolution in the quite common format. It's four megabytes of data and we are transporting this back and forth different devices. If you look at the architecture of the system like it is today, like it presents itself today, regarding the image capture part. Let me... It doesn't go away. I'm sorry. Regarding the image acquisition part, we usually have a few acquisition ports. Today is most of CSI too. Backed by a big, fat ISP. The ISP interfaces itself with a system memory for doing DMA transfer of images. It interfaces with other system components like DRM. It interfaces with a CPU. It has very fast interconnection because it's a system peripheral and externally we have a sensor of course to provide the image data. What's an ISP? An ISP, it's an image signal processor, which is nothing but the set of digital processing blocks that perform transformation of images. It's composed by configurable box, so you can create a pipeline of transformation that does different kind of transformations. What it usually does, it's interfacing with memory, so DMA transfers, reskiling, resizing, cropping, all kind of transformations. It can convert formats from one format to another, pixel encoding, make it easy to present image to application or whatever. It can do a lot of advanced image processing features like 3A and reprocessing, tuning, et cetera. It's very efficient because it's a system peripheral so it runs a very high frequency. It has a high speed interconnection with CPUs and the design is very pattern specific. For a system on chip that maybe are targeted to the automotive markets, you would like to have a lot of DMA channels because you're maybe transferring a lot of different images from different sources. If you have an SoC which is designed from the mobile application space, you maybe would like to have different outputs with different pixel encodings to present images like still capture or view finder, easy to be application. That's what a system looked like 10 years ago and the thing was kind of different from what we have today because we have maybe one acquisition port, a few acquisition port that usually parallel, backed by maybe a scaler, a pixel converter but nothing complex and we just have interfacing to the system memory. Of course, the sensor outside dealt with most of the complexity at the time that was called the smart sensor, that's why, because they were doing most of the things that nowadays run on the ISP. Vidafro Linux has been designed, the Vidafro Linux API in a time where system look more like this. So we have a single device, no abstraction, user space that controls the whole acquisition process. That fits very well if your acquisition process is basically just moving frames to and back and forth from the system memory. At the IOCTLs, which are called the media controllers, that allows you to link the different processing blocks and create different pipelines and through the v4l2 subdevs API, that allows you to control what format and what transformation each block performs. This is very nice and powerful, used to capture images through the device and through the Vidafro Linux 2 API like it used to do. And this is very nice and powerful, but it has a problem. The problem is that the system usually doesn't boot in a usable state. Before you can do anything, you have to configure the pipeline, set up links, set up the formats, and to do that, all the applications that want to interface with that need to have a very good knowledge of the underlying hardware, so this is very platform-specific. Also to perform a capturing application is not just like we used to do with a start stream on the VIDA device node, but you have to set up precisely all the pipeline components because otherwise you got back an E-pipe and error and you cannot stream. So for embedded system, this is okay because it's acceptable because you usually bait your own Eurotaphase, you can embed script there, you can do whatever you want, it's very system-specific. But if you think about something like that running on a laptop, that doesn't work for real because distribution are not ready for that. They aim to be generic, so that doesn't work very well. And until today, that was not a problem because most cameras or laptops and tablets were USB cameras. USB cameras are much sensor, they just connect through a USB. They are very platform-agnostic because USB is universal. And it fits very well in the VIDA for the single device node abstraction. You have a single entry point for the wall control, image capture process, and you control everything that goes on through that single entry point. We have a library, V4L, or V4L2, which controls, which work with most of the sensor that we have today, USB sensor, and does form a conversion and makes easy to write application for VIDA for Linux. But something's happening and we start receiving bug rapors. That's very recent one. It was on the Linux media mailing list one week ago, I guess, and basically says, I have a laptop and I cannot see the webcam. I like the concept of dual boot. That's a complex camera. It's an IPU3, it's Intel ISP. And you need a driver, of course, but you also need user space to do all the configuration and tuning, so that's not going to be supported. What's happening is that we start to seeing Intel, Kaby Lake and Skylake platforms which embed a complex ISP and producers have started making laptops and tablets using that one. Driver is okay, isn't staging right now, it has been submitted to Linux media, so good work with that, but we're still missing the user space part because this device simply doesn't work on Linux. That's how, just to give an idea how an ISP works, that's a simplified model of what happens on the IPU3. So we have two different blocks, one that is in charge of the IMGU, IMAGING unit, which does all sorts of complicated transformations and you've got three different outputs, one which is scaled, one which is full-size and a set of metadata, which is used for tuning a very complex 3A and tuning machinery, which is fed to another input, which controls how the next images are captured. So this is complex and this is a complex camera, so in order to have something like that to work, you have to bake some script that's sucked up all the pipeline, all the formats in its single piece of the capturing pipeline and that's very complicated. So the discussion starts like, I think was July, there was a meeting in Tokyo for the Vitoferlinus community that derived meeting notes and the issue is that we have device which are not working. Mauro gave a very nice presentation at ELC about complex cameras and why they are complex and how they work so far and the output from that was that a project has started which is called Lib Camera, which is a joint effort between the Vitoferlinus community and part of some industry partners. When Lib Camera is not just a library, it seems to be a complete full stack in user space to deal with camera, with complex camera or Linux systems. Lib Camera aims to abstract away all the complexity to setting up and controlling the media controller and the Vitoferlinus sub-devices and it seems to be compatible. First of all with Linux because we want to solve that problem for Linux system but also with very widespread system like Android and Chrome OS because a lot of devices that run with IPv3 and other ISPs are Chromebooks. In 25 minutes I won't have time to go into technical details for that but Laurent gave a very nice talk that details the architecture of Lib Camera and the internals at ELC, it's available on YouTube or the ELC website so if you want to know more details about the technical specificities you should prefer to this talk. But I would like just to give an idea of the architecture and that's a stack of application that capture images and starting from the top we have, we might have frameworks like Gistreamer we might have application that uses Lib V4L like they are doing today we will hopefully have native Lib Camera application and maybe we have Android on top of that. There will be an adaption layer and there will be Lib Camera on the camera object concept. So if you have a work with media controller you know that nowadays we have to specify which media device you want to work on which video device node you want to operate on Lib Camera aims, the pipeline ender will receive requests for formats and capture profiles from the camera and the camera manager and we'll apply that to the underlying hardware. As we've seen the 3A in tuning it's kind of important on those kind of platforms so we design an RPC system to isolate processes that run 3A algorithm because those kind of processes are usually provided by the vendors. So we don't want to tie in any binary and we want to give space for open source 3A implementation to compete with the vendor ones so we are designing an RPC interface to isolate that process and separate the other space and interface with Lib Camera separately. Of course we got set of outbursts to interface with actual devices and that's what we have in the application, that's a very brief introduction and I would like to concentrate since we don't have much time on how we are actually doing that because Lib Camera is a new project, it started a few months ago but we already have something that is usable and you're welcome to interact with. So we have a Git repository which is hosted at Linux TV where all the beautiful Linux development happens and we are doing development on the mailing list. I know you love it or you rate it but we decided that it was best to do that to guarantee that we keep following a development process as much distributed as possible. We are using Mason and Ninja which I don't know how many people know that I didn't know about those two projects before starting with Lib Camera but if you use outer tools, those two things are amazing because it makes very easy to distribute an application and track dependencies and also helps with testing a lot. We do Enforce a Coding style which is something that most projects do and we do that because we want to have the code that which level of quality does it as close as possible as the kernel one so it's partly to satisfy our OCD and have the code that looks nice but it's also because we are using that we are developing in C++ and C++ is a language that makes it very easy to mess up so since a few years there is a lot of attention on concept of ownership since Rust came out and doing that in C++ requires a lot of enforcing and coding style and coding rules and in order to do that since we are doing C++ and we don't have support from the language itself we love peak reviews so if you are used to the level of reviews of peakness of reviews on the Linux kernel mailing list that's what we aim to have so you're welcome to bike shed on the variable declaration order names of the variable and stuff like that but also we invite you to take part in that and help to keep the code quality very high and we would like to have we also try to be serious at documentation and testing we found out that writing proper documentation it's more difficult and takes more time than writing code possibly and we are using the oxygen for that which is great we generate the website from the source code as well as the documentation and we enforce testing because we would like to have all major components that gets in to be associated with the test and we aim to have much more platform as possible that support those kind of testing where are we right now? the project is very young very very young just two months old and we have support for UBC cameras which are the traditional USB cameras and we have a homeless fully working support for Intel IPv3 which is the first and most urgent targets that we have we can list and detect outplug and out unplug of cameras and we can capture frames actually which is kind of a nice thing considering that until a few months ago those devices were not working at all in Linux where do we want to go? libcamera won't ideally should support OSP that has a media controller compliant kernel support so there are a lot of OSP that are possible targets for next development there is Rockchip OSP there are other platform which has a good support in kernel space and we aim to support all of that of course in mobile space there is a big competitor for all Linux system which is sort of Linux system which is Android and Android has a very complex camera stack if you are used to work with Android cameras you know that camera OSP v3 supports per frame request so each frame is associated with a set of capture parameters which is something that requires development both in the library and both in the Vdefer Linux API which today is not totally ready for doing that also we would like to get in touch and work as much as possible with vendors because we would like to have something like when you supply a driver for Vdefer Linux and for media controller that should be associated with the support for libcamera in order to have a complete user space and kernel space support for the camera itself one of the reasons which is C++ for that is because we want to interface with vendors and we know that most people in there know C maybe they are kernel developers so it's easy for them to drag people in using a language which is more familiar so I would also like to this are a few references we have a website we have a mailing list where we share patches we will have a patchwork possibly and we have an RSC channel like most projects do and you're welcome to give a hint but because the project is very young and we want to know as much as possible for people that is working with cameras before going to question which I hope you have some I would also like to give a brief I would say that's not a demo because it's very, very simple but this device is a device with as an IPv3, it's a Chromebook as an IPv3 device and was not supported by Linux system until a few months ago and right now I think we can do we have a small utility which is called COM that allows you to right? wonderful so it was working before oh I'm not rude my bad there you go we have two cameras wait wait that's not all and we can also catch you from cameras as you can see you identify cameras by name the name is now created by parsing the not the ACPI table but just parsing the sensor name you can identify camera by calling that by name you don't care about which media devices they are connected to and we can capture frames hopefully and thanks to with a bit of conversion there is an image there trust me it's very dark because we don't have all the machinery to do the tuning in place but at least we can capture frames which is something that until a few times ago was not possible so I think it's a good achievement so I hope you have questions for me I hope you have questions for us in general there are other people which is involved in the project here so feel free is there any overlap between what the camera does and what G streamer does so should I repeat the question is there any overlap between what G streamer does and Lib camera does not really because G streamer is one layer up to the camera nowadays G streamer is what is called the V4L2S or SRC element which interfaces with the V2 device we hope to that we will have a Lib camera element where you can just interface and ask it could you see all in future also being picked up by the embedded community or are they just happy before they're different so are we focusing just on laptops and tablets or is that something that should be used also in embedded systems or embedded systems are fine the way they are I don't think that embedded systems are fine the way they are because today everyone is making his own solution so you have a script using media CTL but it's very difficult to kick track of the resolution along the pipeline or people maybe write their own application using the interfacing with the device drivers interface but I don't think that the system are fine the way they are so we hope to have all the ISPs and platforms that have a kernel support which is mainline to use Lib camera as well so we I think that they are on the same level you know this is more pressing because you buy a laptop and the camera is not working on Linux so that's very bad on the embedded system they are already used to work with that yes I should repeat the question it's a very long one by the way so the problem with 3a and statistic it's usually that it's very system specific and vendors are not are not keen on giving away documentation on those things but 3a and tuning is very important time's up so I'll be very quick okay thank you so what is the situation with IP3 is Intel behaving well in this case and is Intel providing documentation I would not say that it's it's providing documentation they're doing in a way that they are doing development on mainline so we have the driving staging and it comes with documentation on the parameter and everything as far as I know there are no private IOCTLs which are vendor specific so that's a good thing so yeah Intel is doing good in this case and we want more vendors to do that because Lib camera works only on mainline and standard beautiful Linux interface okay thank you