 Welcome. So after my bonus talk earlier today about CEC, now I can talk a little bit about video flinux, roughly what happened in the past year. I don't have a lot of time, so I am sticking to two main pieces that in my biased opinion has been the most interesting and most important, and that's hardware codec support and testing. Let me first start. There are two types of hardware codecs. We call them stateful and stateless. Stateful codecs have been supported for quite some time now, and basically you just feed them and say A264 bitstream, and they do everything, and outcome the decoded frames. So they keep track, the hardware will keep track of the reference frames, they will keep track of the state, and they are really simple to implement because all the complexity is in the hardware. Problem with that is that takes a lot of, that moves the complexity towards the hardware, and of course hardware designers don't like that, so they've been pushing back, and what they basically did was more like accelerators. So you give the frame to decode, you give the reference frame associated with it, or frames, whatever. You basically give the whole state together with what you want to decode, and the hardware no longer just takes that, processes it, returns it, and it no longer remembers any state. Those have not been supported until very recently. But first stateful codecs, as I said, we support quite a very big range of them already. What was missing was a precise specification how to use the Video Phoenix API for that, particularly in corner cases like what should happen when you do a seek in the middle of a stream, or you have a midstream resolution change, and Google, specifically the Chromium and Chrome OS team, they have been working on writing all those details down. Incredibly useful, it is not yet merged. We are going through hopefully at least one more version, I think. But the good chance that we'll get into the 5.1 kernel, we said 5.2 by now, 5.1, where are we? In order to test these things, so testing with actual hardware codecs is difficult because you need the hardware. It would be nice if you had a hardware emulator. We have that for various other devices, types of devices as well. I come to that in the next section. But we also have a Vi codec. It's a virtual codec driver. It implements in software a codec, not any of the standard ones due to IP issues. It's a very simple codec that actually was used decades ago by NASA in some of the space probes. It was a student who did a project on that, who made the codec itself and I turned it into a driver. But it is not compliant to the specification that we're working on. And Outreachy is an organization that provides money for people who want to get involved into kernel programming for projects of three, three months to work on particular issues. Not just kernel, it can be any open source project. I'm co-mentoring together with Helen Coyker from Collabora, Daphna Hirschfeldt, who is working on making the virtual codec compliant to the spec. It's a moving target since the spec is not fully specified yet. So that makes it a bit interesting. And I've started work, so this is slightly out of date. By a few days I started to work on adding compliance tests as well for stateless codecs, stateful codecs. I already, while doing that, I already saw some issues with the specification that are, there are certain exceptions and they're already difficult to program, so it will probably change a little bit. But this is great to have this combination of documentation, compliance tests, and basically an emulator for codec to figure out these issues before you make them available in MadeLine. The other big one is stateless codecs. So these, there was no hard for support in the kernel for these devices. It was, however, used already for three, four years by Chrome OS because they had an early version of what is now called request API that I made many years ago and they've been using that on Chromebooks. So one thing that they're also doing is writing documentation on how to use it. That's a bit earlier, not stateful codec, the documentation is further advanced than this one. Daphna is also working on turning the vi codec driver into a stateless codec. So we can test all this. She just started on that this month or last week actually. That needs to be compliance tested as well. But the big thing here is request API. It was merged in 4.20 together with the first stateless decoder driver. It's in staging. The request API is in MadeLine, but the driver is in staging because there are still a few things that are being developed. Work is actually being done already by various people to adapt it to the Rockchip codecs. And I don't know what the state is, but someone started on work for the Tegra decoder. That's also both stateless codecs. There's also work on the way for H264 and HEVC. And some people are already looking at VP8 and VP9 or at least thinking about it. So there's a lot of interest in these because the hardware that uses these codecs, they have been blocked for being MadeLine for quite some time. You couldn't get the drivers in because the request API was missing. What the request API basically does is that it creates the name request actually comes, I believe, from Android Camera Hall. You make an opaque object. You put a buffer into it and all sorts of formatting data configuration. And you give it to the hardware. So you have basically shorthand for per frame configuration. The hardware processes it and at some point the request is completed. You can dequeue the buffer and get any updated state information out of it. This is needed not just for stateless codecs, although they are the first one that start to use it, but also by complex video pipelines. The co-post talk earlier today about Lip Camera, it mentioned that they want to use it for Android as well and then you need this. This is basically per frame configuration, which is a requirement from Android. The current request API framework is, it probably gives between 80 and 85% of what is necessary for that type of application. So more work will be done in order to support it. It's not quite there yet, but a very large chunk is finished. So how do we use it for stateless codec? We have two video nodes or two device nodes, the video nodes that was always there, but now you also need to use a media controller node. Media controller, very quickly it basically shows the internal topology of your hardware. It is primarily meant for complex video pipelines, but we also use it for stateless codecs. The reason we use this instead of just the video node is exactly because the request API will also be used for the complex video, complex camera pipelines. So this keeps it consistent. You have the same method for both. If you have a stateless codec, that means that user space is responsible for parsing the bitstream, no longer the hardware. So it parses out the headers, it figures out the states of what the preferred to be decoded. I'm primarily talking about decoders. Encoders is very similar, just the order direction, but make it easy. I'll just talk about decoders here. So what you need to do is, first of all, you need to create request objects. And usually you make one for every buffer that you allocate it, but it's really up to you. Depends a little bit on the codec that you're using and other factors. So you use the media file descriptor of the media device node. You allocate a request and you get the file descriptor back. It's just an opaque handle to the request object. You always refer to request objects using that. So it's a decoder, right? So you have a buffer with a to be decoded frame. And you decoders are implemented as a memory to memory device. So you have things that you put into the hardware and you get buffers back. So you actually have two buffer cues. One is called output, one is capture. This is very confusing. The point of view is the software. So an output cue means that you give memory to the hardware. The capture cue means that the hardware fills in the buffer and gives it back. So for the decoder, the compressed buffers go as an output cue and decompressed buffers come in in a capture cue. For the output buffer, so it's a codec. So you always start with the output. You say it's a video output buffer. You have a request file descriptor. So that means that you don't queue it directly to the driver. You queue it in a request. So it doesn't go to the driver yet. It's it's stored in the request. You have to set a flag to indicate that as well. And there's a timestamp that you can set. Now for memory to memory devices, timestamps are not used by the hardware. They're copied from the output buffer into the capture buffer. That's all that is done. Timestamps, basically you're completely free to stick in whatever data you want. And in the case of codecs, they are used to basically name buffers. It's used as a tag. So the problem is you have reference frames for a decoder. The decoder, you have iframes. They are decoded all by themselves. And then you have P and B frames. And they are basically the difference between a reference frame, an earlier iframe, and what it should be. So they are a lot smaller because of that. That means that a stateless decoder needs to basically get two buffers. There's more complicated with ACE264 and more complicated codecs. You can have multiple references. Let's keep it simple. You have the buffer that contains the P frame, the difference, and you have another buffer somewhere that has already been decoded that contains the iframe, the reference frame. The decoder needs both. And the tag is what you use to later after this frame has been decoded. The decoded frame that you get back has that tag as well. It's copied from the output to the capture once it's decoded. So it gives you a unique identifier in the future of this decoded buffer. I'll get back to that in the next slide. But the last bit is that you queued the buffer into the request. So that's just the buffer is now in the request. The next step is to contain, to add a configuration, the state to that request. For MPEG-2, there is a slice params control. There's basically a fancy structure with data that you can set in your application and it can be stored in the request and read out by the driver. There are actually two controls related to MPEG-2, but no room on the slides to complicate it. So I just addressed that one. We're using the control API. Again, you need to specify where do I do that. Oh, yeah, here. This is the request file descriptor. This is a parameter that tells you that the control shouldn't be applied directly, but should be stored in a request. Here you set it. And the critical bit is actually, there are a lot more parameters, but I'm trying to concentrate on the important part. The backwards reference timestamp, so that is the one, the field that refers to the reference frame that this buffer is the difference of, that you have to refer to. And that's been the earlier timestamp of an already decoded buffer. So you reuse the timestamps in order to refer to it. So this is the output part. Then you queue up capture buffers because the result has to go somewhere. Capture buffers are not parts of a request. They're just independent buffer queue. You put buffers in, the decoded results gets put in there and you get them back. And final one is that you queue the request. So now the request object is queued to the buffer. The buffer has all the information. The driver, sorry, is queued to the driver. The driver has the buffer, has a configuration to state, and it can start to decode. You can wait using select for an event from the request object. And when an event arrives, that means that the request has been completed. All the data, the buffer is in a validated state. The parameters, configuration is validated and updated and you can read it out. And finally you need to dequeue the buffers. The output buffer, that's not what you, that's now, can be recycled. And the capture buffer containing the decoded frame. So when you dequeue this buffer, the timestamp in that buffer was a copy of the timestamp that you set in the output buffer. And this timestamp can always be used to refer to that buffer. If you need to do that in a, in a following p-frame you can refer to that capture buffer through the timestamp. I concentrate on this because it's relatively complicated and easy to mess up. Not to mention that it took a very long discussion on the main list to come to a way that works and that everybody agreed with. Testing. So video is complicated. Just like in the DRM system, you had several talks about continuous integration and testing. The same thing is true for video input and for hardware codecs. We have already quite a few visual drivers. VIFIT, that's the oldest one. It emulates a webcam, HDMI receiver, SVDO, even a TV tuner. Also radio, software defined receiver. Even CEC is being emulated there. VIM2M, that emulates the simple memory-to-memory processing device. There are some upcoming patches being made by Mauro that improve it quite a bit. It basically emulates a scalar or a pixel format converter. VIMC emulates a complex camera pipeline. There too there are improvements planned. And then there's the latest one, that's the VI codec, the virtual codec driver to emulate the codec and encoder, encoder and decoder, both stateful and stateless. We had recently several, let's say, embarrassing regressions when we made some patches, pushed some patches to the kernel. That could have been easily called if we had some regression tests. We have a compliance test that is continuously improved. And what we basically wanted to do is to create a media script that would just run the compliance test for all these virtual quality drivers. Again, because of the complexity, it's difficult to test your driver. And these compliance tests, they test all these corner cases. Make sure that all the fields that you are supposed to return in structures that are actually properly filled in and you didn't forget anything. And it's actually more strict than the specification itself, because it checks what should be done. And sometimes the specification is more relaxed due to legacy drivers. But this is really meant for new drivers. So new drivers should really do the best quality that you want, that you can do. It's also very useful, as I mentioned, for when you're developing documentation and making a new API. You also write a compliance test, because it's everything, anything is not quite right. You will very quickly figure it out. Our fairly new is that there is a fuzzing checker from Google and several drivers have been added. These virtual drivers have been added there. In the beginning we got a whole pile of book reports out of it, but most 80 percent around, about 80 percent have been fixed. For the remaining 15 percent we have patches that need some more attention. And occasionally there's, I think there's one or two new ones that appear. I haven't looked at that yet. So this has been very useful. I'll do the resources then. I have a very quick demo of the test scripts. API documentation, there is a version that I maintain that contains the Google specification for codecs. Since it's not mainline yet, I build it myself. For easier reading, easier to read than the patches themselves. Git repository, for the Cedrus staging driver, there is a test utility to test it from bootlin. P4Utils, that contains compliance test. Oh, mailing list, the RSE. We're always looking for volunteers to improve the virtual drivers. Since they're virtual you don't need special hardware, so you can run it on any PC or virtual machine or whatever. It's quite an extensive list of projects from just a few days to a few weeks. If you're interested, let me know. So I have a little script. You have to run it as root. This tests all the media drivers. So first is it unloads all the existing media modules and then loads the virtual drivers. Hopefully I can avoid that in the future. The main reason is that the video numbering, video zero, always has to be the same device at this moment. That's why I need to clean it. And then it's running with VFRAC compliance. It's testing all the very extreme modes. It's testing it with blocking mode, with unblocked mode select, testing DMA buff as well, single planar, multi-planar. It's doing about 1700, so it's pretty quick actually. It doesn't take too long. So there's 1700 plus tests. And this is not yet including VI codec. It is currently, there are no failures at the moment. This is the latest kernel master branch from us. There are a bunch of warnings. I know what they are. Someone is working on at least a few of them. So this should really be zero as well. It will be, I promise. And I've done this earlier. So this is the full test. This includes CEC. You see that up there? CEC tests. But CEC tests, as I mentioned in my talk earlier today, CEC bus is 40 bytes per second. It's extremely slow. And if it emulates CEC, including the speed, or lack of it, so doing the CEC tests take a fairly large, I think this whole test probably takes about five to ten minutes. I haven't timed it. But you can see that this is close to 2000 tests. I hope that this will either be included in kernel CI and or I'm doing a daily build of our latest master, making sure that it compiles for a large set of architectures. I would really like to, at the end, do a run on the virtual machine to test whether everything is okay. But I need some time to figure out how to do that. So with that, questions. That's one year of eofilinux development, at least part of it in 20 minutes. No questions. Everything clear? Ah, look, thank God. I'm using the time-stamping for referencing the reference, but how do you manage the memory for the reference when you're free then? It's right. So memory, usually you allocate memory beforehand. No, let me rephrase it. You have a request buffer step that the buffers are created. The actual memory for a buffer can come from outside, but internal structures for the buffers are created. They basically, those buffers, they mapped once they acute. It checks if it knows about the buffer already, the buffer memory. If not, it will map it. If that's it stays mapped the memory until you explicitly free all the buffers. So internally, you always know that given a time-stamp, it belongs to that memory and you also are guaranteed that that memory stays. The only exception would be if you completely free all the buffers, but then you stop streaming anyway. Or if, and that would be the case for, for example, a DMA buff, the same buffer, internal buffer structure gets a new DMA buff file script. And in that case, you, you actually release the internal information. You disconnect it. So we, we keep track of that internally. Yes, Vivid and VIM2M and Vycodec is being worked on. No more questions? Okay, thank you.