 Nazywam się Marek Szuprowski i pracuję na Samsunga R&D Instytut w Bolandie. To prezentacja będzie o naszym propozyciem nowego filmu procesującym api na linux DRM subsystem. To jest agenda mojej prezentacji, więc na początku powiem kilka słów o linux DRM subsystemu. Teraz zrobimy zainteresowanie do kms api, która jest częścią linux DRM subsystemu. I potem przedstawię co jest Xenos DRM IPP subsystem i dlaczego to jest ważne dla nas. I co jest ta prezentacja. Więc co jest nowy api, co jest celem i jak to można użyć. I ostatnia część tej prezentacji będzie kilka kodów. I potem będzie krótki sumar z naszym propozyciem i oczywiście z tej prezentacji. Pierwsze z tego powiem, że nie ma nic o którym się z tym mówić, bo ten akronym jest... ...at least, że są dwa inne subsystemy DRM subsystemów, które mogą być zawodniane w grafach. Więc moja prezentacja jest o dyrektorze randującym. i to jest subsystem dla grafiki rozkawionych rozkawionych drzewoń. Teraz, w nocy, mogą handlesować wszystkie napewne GPU na topi produkcji, jak Intel, AMD, Nvidia. Natomiast to też ma zamiast drzewoń dla bardzo prostych grafów, Mamy grน many embedded SOCs. Main goal for this subsystem is to let user space access the graphics hardware and manage it. DRM subsystem also provides a way to manage graphic buffers known as GEM objects. więc to jest mnóstwo grafików. Następny ważny udział w tym DRM subsystemie jest kernel mode setting. To jest część aplikacji aplikacji. To było kilka lat temu. Powiem na kolejnym slajdzie, dlaczego to jest ważne. Ten DRM subsystem też ma respektywą część w user space, który jest w nim implementowany w LIB DRM library. Więc ta aplikacja user space nie musi osiągnąć IO CTL. I to może używać niektórych funkcji libraryjnych, które są o wiele bardziej przyjemne. Więc co jest KMS? Kernel mode setting. To jest, w moim opiniach, bardzo ważny udział w DRM. Ponieważ to pozwala za pierwszy raz aplikacji. To jest nieraz generyka. To jest abstrakcja hardware, który zainteresuje różne entety, jak CRTC, konektorów i kodów. Powiem Wam trochę więcej na to, natomiast to był pierwszy raz, że nieraz generyka hardware independent w IO CTL i to, co było wytrzymane. Więc aplikacja może całkowicie konfigurować dysplikę pipeline. By konfigurować dysplikę pipeline, to znaczy, że rozwiązanie piksel formatu zainteresuje dysplikę bufferu do zainteresowania. I także ta cała pipeline jest konfigurowana w sposób, w jaki aplikacja chce. KMS też wprowadzi emulację Legacy Frame Buffer Device API. To jest już wcześniejszy API, używany przez dysplikę w Linuxie. I dlaczego to jest ważne? Dlatego teraz, że FBDef API jest jeszcze używany przez kernel konsol. Jeśli chcecie mieć konsol na grafiki, na dysplikę grafiki, to jest jeszcze używany, implementowany na topa Legacy FBDef API. KMS API, razem z tak zwanymi, tzw. dump frame buffers alos, aby skupić kompletnie genericzne aplikacje, które nie spodziewają na żadnych hardwareach, i mogą być używane na więcej niż jednej grafiki. To znaczy, grafiki z różnych wędrów, tej samej aplikacji. Są już kilka nami, które oznacza, co to znaczy i jak możemy ich zrozumieć. Pierwszy nami, który oznacza, był GM Object. To jest teraz, ten akronyjm, który jest graficzny egzekucjonalizator, natomiast w DRM, teraz jest to, który używa nas jako węgłego węgłego. Gdyby się odnaleźli, więc przez GM Object to znaczy węgłego węgłego. Kolejny, który jest węgłym węgłym, to jest to, co jest w tym, bo w DRM, to nie jest więcej niż GM Object i jakieś metadatele, jak format, który wie, i pixel format. To wszystko. To ma jakieś idee, które jest używane dla różnych proporcji. A potem mamy aplikację. To było odnaleźło recentnie. Więc przez aplikację zrozumieliśmy jakąś część Hardware, która jest umieszczona na skanie video pixel data z framebuffer. A potem mamy entity, które nazywa się CRTC. To jest pewnie najwyższa, ponieważ nazwa historiczna jest Kato Dry Tube Controller. I w nowadays Hardware to jest część graficznego subsystemu, który ma to zrobić mięsko i blendować wszystkie plany. Pokażę egzemplę z piłką, więc to będzie mniejsze. A potem mamy encoder. To jest część Hardware, która odnaleźła ssignal z outputu CRTC. I ostatni obiekt jest konektor, który po prostu odnaleźł ssignal od encoder do jakiegoś eksternalnego warstwa, np. display panelu lub HDMI outputu. I tutaj mamy egzemplę z piłką graficzną. Trzy graficzne buffery na górę. Mamy GM obiekt na graficzny graficzny. A potem mamy filmik. I ostatni obiekt jest GM obiekt na górę. Z każdego z tych GM obiektów, jest konektor. Jest dwie plany, które skanują pixel data z tych plenów. Wszystkie z tego obiektu odnaleźł ssignal z outputu od encoder. I ostatni obiekt mamy konektorze, który odnaleźł ssignal od encoder. Tym samym obiektem odnaleźł ssignal z outputu. To jest tylko przykład. Są obiektów nie używających w konfiguracji dysplejów. I dlaczego mówię o tym? Ponieważ obiektów są dm obiektów. Dm obiektów są bardzo ważne w KMS API. To jest to, że obiektów odnaleźł ssignal od encoder. To jest to, że obiektów odnaleźł ssignal od encoder. To jest obiektów od encoder. To jest obiektów i dm obiektów. Tym samym obiektem odnaleźł ssignal z obiektów. To jest obiektów, obiektów, obiektów. To jest obiektów z obiektami. USA obiektów, To nie jest listy tutaj, ale jest też ważne, że każda propieta ma jego własną nazwę. Z tym, co może być sprawdzone przez nazwę, która jest ustawiona na prostym stronie. I wartość propieta jest ustawiona na 64 bitów. The KMS API provides IOCTLs to enumerate all objects available in the system. And to get properties of any of those objects. And also application can acquire each property for the value, but also for the name of the property and the type. So another major step in DRM development was the introduction of atomic KMS API. Why the name atomic is used? Because the main idea was to let user space to configure or change the configuration of the display pipeline in a single IOCTL. There was various problems with the first version of the API, because there were many use cases when user space wanted to change more than one parameter of the display pipeline. So the simplest use case would be to update more than one plane at the same time. Usually you want to update the buffer on the page flip. So on the next scan out all the video data will be scanned from the new buffers. In the original KMS API you had to issue IOCTL for any of those updates. In the new atomic API you can change all of them in a single IOCTL. This atomic KMS API also solved the problem of all the intermediate steps in the pipeline configuration, because some of such steps might be hard to define or hard to implement. The atomic KMS API has been enabled in Linux 4.2, and till now most of the drivers has been already converted to atomic API. How does it work? The main idea of the atomic approach was to use DRM objects and their properties. So if we can define the state of the whole pipeline by the state of all objects and their properties, we can just provide the list of objects and their properties in this single IOCTL. So this is the main idea of atomic KMS API. Let's now switch a bit and I will tell you a bit more about Exynos image processing API. What is image processing? Basically this API gave application access to various hardware modules that implements memory-to-memory operations like image scaling, cropping, color space conversion, rotation and flip around vertical or horizontal axis. This has been initially implemented as Exynos DRM custom extension. Exynos DRM is a DRM driver for the display hardware that is available in various Samsung SOCs. I'm from Samsung, this is important for me. The name might be a bit confusing, especially by the part post-processing, because it's just about processing. However, I wasn't involved in designing and implementing the first version of this API. So this is a very simple example of how it looks like. We have two image buffers, the source and the destination. Each buffer has some properties like feed-hand height. We have some operation area, which is defined by the top left corner and the feed-hand height. We can do the operation like copying pixels from the source to destination, possibly with some kind of color space conversion in between or rotation. All this is done by the hardware, so it is much more efficient in terms of speed and power than doing it by the CPU. However, there are some problems with this Exynos custom extension. This API has been introduced in the mainline about three years ago. The user space API is heavily based on the internal way of operation of the hardware modules that were available at that time. What does it mean? From the perspective of API, it means that the whole API is built on the idea of state machine of the hardware module, even if we want to do a single operation of copying image data. The API is somehow hard to understand and prone to errors. Initially, there were additional modes of operations defined, which were never fully implemented. However, they are part of most of the API calls. That's why it's very hard to understand it from the beginning. Probably the most important thing about this API is that there is no client of it in the mainline. I mean there is no user space application in upstream that makes use of this API. To było to poruszenie, że decydowaliśmy wytrzymać to i po tym, żeby to zrobić, żeby to zrobić, żeby to zrobić. Wytrzymajmy jakiś IPI, który będzie generyjny. To jest bardzo łatwe, aby oznaczać. let's take a look at the single image processing operation again. Single operation requires to have source buffer, destination buffer and optional transformations that has to be done to the image data while they are copying from source to destination. Of course we also need some operation area nie chcielibyśmy operować na całym bufferu. To wszystko. Te parametry są wystarczające, aby wyobrazić imię procesowania, kompletnie. Więc co były sądy nowego API? Chcielibyśmy fokusować memoria-to-memoria operacji i zapłacić utwór od wszystkich wyobrażonych rzeczy, jak imię skaling, kroping, kolor-space-conversion, rotacja i flip. Chcielibyśmy zapłacić ten API generyczny, więc aplikacja powinna opłacić do akcji akcji hardware, ponieważ nie wszystkie modely hardware wytrzymały te operacje. Chcielibyśmy opłacić każdego hardware specyficznego rzeczy, które nie są naprawdę częścią API. A więc pracowały na to, decydujemy odpoczywać konwencję DRM i KMS. Chcielibyśmy używać i opłacić te rzeczy jako częścią nowego API. Kolejny to, że chcielibyśmy opłacić aplikacje hardware indepentyczne, więc chcielibyśmy opłacić te API generyczne. I jeśli możliwe, żeby być gotowe dla przyszłych ekspansji. Nikt nie zna, co nowe rzeczy będzie dostępne w modelach hardware na przyszłą generację SOCs. I jeśli możliwe, to może też być przyzwaną przez ten API. Teraz kilka słów o imię przed nami. Buffers w DRM są głównie GEM-objekty, natomiast GEM-objekty są tylko w size i ma niespecifujący format. Więc w tym nowym API propozalem chcielibyśmy mieć jakieś property do imię więc pójdąc do używania dm frame-bufferobjekty. Co nas dało przez wszystkie metodaty które może być ważne dla procesowania imię. To znaczy, że mamy GEM-objekty co znaczy memory które kontains video data. Mamy start of set, the size of the buffer, pixel format, width height, stride and the frame-buffer object also provides access to multi-buffers format like the video formats with separate luma and chroma planes. So this we got for free if we decided to use the DRM frame-buffer frame-buffer object. And our proposal has been submitted to mailing list in August 2016 so two months ago and the main idea was to introduce new DRM objects which has been named frame-buffer processors although I has been already notified that this name is somehow confusing so maybe the next iteration of this proposal will use a bit different name but we wanted to make all operation on all image operations to be defined by a set of properties and this API was heavily inspired by the DRM atomic KMS API so this API consists of three new IOCTLs we have an IOCTL to enumerate all the introduced new objects which are called frame-buffer processors we have an IOCTL to query properties of the given DRM process given frame-buffer processor object and the last IOCTL perform an operation on a given frame-buffer processor so together with two standard DRM IOCTLs for managing DRM objects and their properties the application can write can implement hardware independent image processing let's scan quickly do all those free IOCTLs the main convention of the DRM IOCTLs was the fact at least at the KMS part was to let application to call it two times first time to get the total number of the objects that we want to query and the second time when application provides a buffer to store information about all objects so in the second call DRM subsystem fills the object IDs to the array provided by user space the second IOCTL was to get to give application the capabilities of the given frame-buffer processor object and the capabilities are stored as a flag and there is also a list of supported pixel formats because not all modules can operate on image data so and in the DRM subsystem there is already a set of defines for various pixel formats and the capabilities are rather self-explanatory so they let application to know whether the given frame-buffer processor let it to crop or to convert from one pixel format to other and like I already said the source and destination formats are defined by standard for CC values and the last IOCTL was to perform an operation there are some flags very similar to the flags already available in the Atomic KMS API so first of all we can decide whether we want to have an event after performing after the hardware finishes performing the operation so we can set that flag we can call this IOCTL only to check if the hardware will do this operation with all the defined properties only to check if the values are correct and the mode of operation is supported by the given hardware block and we can decide whether the call or not and how do we define the operation we have to provide a set of properties and values so we provide to arrays the first array is an array of property IDs and the second array is an array of use of that properties user space can query and get the list of those IDs by performing standard DRM get properties IOCTL on the introduced frame buffer object I will show this later in the code example and what properties has been defined for the DRM frame buffer processor object those are the names of the properties so we have source frame buffer ID which is the type of these properties object ID and it can be set only to frame buffer IDs we have source x, y width and height i to follow the convention of the KMS API those values are stored as fixed fixed point integers so for some hardware blocks which can process in subpixel precision we can provide such information and the same set of similar set of properties for the destination buffer and destination operation area and optional rotation property that defines what rotation or end flip has to be performed to image data of course application won't call the IOCTLs directly so we have corresponding we introduced corresponding calls to live DRM so we have two functions to get an array of frame buffer processors and then to free it which so the IOCTL for getting the list of frame buffer objects is hidden behind this function we also have and we have a function to get capabilities about given frame buffer objects and the last function is to perform the operation so it matches the third IOCTL to let application build the set of properties that is needed to perform frame buffer processing operation we introduced this kind of request additional request structure so application can allocate this request structure free it and add a property to this structure so this is the third function in the middle so by adding it we added a property ID and the desired value of that property application can also get cursor and set on those properties and then after if there is a need for call and perform the operation many times and change only some of those properties when we have a cursor we can only update a single property inside these requirements how does it look the code we have probably very simple function for doing all the steps needed to perform image processing so we have this process fb functions the first argument is the file descriptor of the DRM DRM subsystem and then we have the rotation property and a set of source and destination parameters so the application first gets the frame buffer processor resources from the DRM call checks if there are any frame buffer process objects so it takes the first available one freeze this structure then gets the capabilities of the given frame buffer processor and check if the capability for rotation is set and another thing to mention that not all error checking is done here in this code makes it really simple and short and probably easy to understand and then we allocate the requirements at structure we get a handle to the properties of the selected frame buffer processor and then we add each property for the for performing image processing operation so we first look for ID of the source frame buffer ID property and assuming that it has been found we add it to the requirement set with the value of the frame buffer ID that came as an argument for our function and then we do the same for the source x value, source y value width height and so on and once all the properties are added to the requirements set we can free the properties the handle to properties and we can simply issue the function we can call the function to perform the operation so we provide this requirements set as the third argument this zero means that we don't set any flag this would mean that the processing will be blocking so the call will end once the hardware finish operation or we will get error and the last argument is used when we want to have an event after processing this is simply user data that will be attached to the event if it would come once the operation is done we can free the allocated requirements structure and what should be noticed if we want to perform simple calls to this commit function we can use the same requirements structure we can only change some properties on them like we change the source and destination buffers if needed and so on so this is some kind of very simple code example how this new API can be used and this is the helper function that was used in this code here the goal is to get the ID of the given property when we when we look it when we look it by the name so we simply get all get parameters of the given property and check if the name matches the requested one so what was the conclusion of this API proposal the purchase has not been merged to mainline yet so the whole presentation was about our proposal which is not yet available in the mainline and we got no positive feedback from DRM maintainers the main from their perspective the main disadvantage of this proposal is that we introduced new generic IOCTLs that has to be supported forever and right now they think that there is no need no need for such thing in the mainline DRM subsystem so we would like to continue work on this API change however it will probably be limited only to the Xenos DRM subsystem and that's all thank you for listening and are there any questions you probably won't like that one but why don't you use video for linux yes video for linux provides functionality however the main reason that we don't want to use it is that changing any parameters of the operation to be performed like for example changing the image size or mainly the image size require setting up the whole operation from the beginning so there is a need for a bunch of IOCTLs to configure everything and even to perform a single image operation there is a need for issuing about at least a few IOCTLs that's correct at the moment it's indeed we have a performance issue there but there are extensions to the API that are being developed and should hopefully land round at the end of January that will allow you to have a single IOCTL to reconfigure the whole pipeline and issue a processing command yes that might solve the problem the other reason was that this is already a part of Xenos API so we just wanted to improve it working on improving it we found that it might be interesting for someone else to make it also to use it and maybe for some other embedded systems might be interested in this kind of API so it was worth to try at least even if this try ended up in reduction Thanks Beyond the short code example that you showed, what is the typical user space usage of such a new API like is it going to be used at the XOR driver level in Wayland I don't know in Gstreamer in the toolkit level directly in the application which part of the user space stack would benefit from this API and how it would be used target for some middleware liars so definitely Wayland or XOR implementation and I think that Gstreamer also might might make use of this API however there might be some problems to let it work together with XOR or Wayland implementation because DRM API is very focused on single application that manages whole display however there are some extensions to let other application also use it So the main problem I see with this API is it seems to support only memory to memory devices How would you support an IPP that is able to upload directly to the LCC Well if the operation ends up directly in the hardware I mean in the display pipeline it can be modeled somehow as a part of the display pipeline for example if because those blocks for image processing can for example get data from the image buffer and output it converted directly to the CRTC however we don't want to make applications aware of the fact and this kind of usage means that we should rather implement it as another plane object, another standard plane and let application to use the generic KMS objects instead even if behind them there will be additional blocks used in the DRM driver itself so such kinds of usage should be hidden by the driver behind the standard DRM KMS API Ok, if there are no more questions thank you for listening