 Hello, so welcome to my ELC talk about how we ported we've made an Android TV BSP using manual Linux. So first of all I'm Neil Armstrong, I'm a Bay Libre SS employee. I work in Villain-Vloubet near Nice in France. So I am embedded Linux engineer. I do regular upstream porting of SoC. The last four years I was porting and pushing code to support the I'm Losing SoC upstream. And last year we contracted to design and create an Android TV BSP based on AOSP for a new AMG SoC using exclusively manual Linux without any vendor BSP code. So the timeline is what short story of Android and manual nukes. Then short speak about hardware abstraction libraries and Android, which is a foundation of Android. How GPU is integrated with Android. Let's note about the modern Android boot flows of issues and conclusion. So the project was to build an upstream AOSP BSP for new AMG SoC. The target was the TV profile because now AOSP integrates the TV profile. No more only the phone profile. So you can actually build Android TV for AOSP for devices. The target was 419. At the time it was already a long-term support. So we needed to back port a lot of patches, all the new patches. It was for the new SoC for AMG, not yet supported. And the team was not experienced on the recent Android release. Most of us only did early Android 1.6 ports. So a long time ago. So Android and manual nukes. So it's a long and complex story with manual nukes because at the beginning Linux was not really made for multimedia ARM devices. So hopefully recently Google outlined an effort, multiple efforts, to make sure they follow more the upstream manual nukes than before. Because until now you had vendor trees, which were really, really different from mainline. I will say for example Qualcomm who had a multiple million line codes on top of an LTS kernel. So they started with project Treble a few years ago, which first of all tried to have a stable kernel ABI, a pseudo stable kernel ABI on an LTS kernel. So the vendor would only use these kernel ABIs to provide their drivers their modules. So this is a target to have a generic system image. So we would have a single kernel image for all devices and the hardware support will only use a list of kernel ABI functions for example. And the target is to have a single kernel for all devices. For example a single ARM64 kernel for all devices and each device will have a set of DevS3 and DTBO, DT overlays and custom model using the Treble vendor interface. So IOSP, the last release can run using pure vignette kernel because with the correct kernel config it can run. We still miss one or two stuff in the kernel, but you can check out kernel 5.7 now and you will be able to run Android. So we still use Android derivative branch because they have the specific kernel config needed, so they drop what is in it and they really keep what is needed to run Android. They still have some patches and fixes for Android, notably for MTD because now they have a super partition with MTD emulation to have a variable layout. So the layout is no more fixed in the putover and the build system use the ML. So as I did a lot of work on the cardabit system because before every vendor built their kernel and pushed only the binaries. So now you have a centralized system with a usage of C-Lung by default to build the kernel using the Android specific tool. So no vendor kernel at all. The kernel was mainline based. So we did the work on two new SoC, so the S905X2 and recently the S905X3. So these are the new generation SoC from ML, notably they use the new Mali byfrost GPUs, so the G31 which is able to render 4k for example, unlike the previous ML SoCs. So the strategy was to push supporter stream first and backport on the 4.19 after. So we used the classic, usual upstream process. So we submit on the main list and we fix the patches until it goes in the middle of the street and when it's merged by the installables we backport it. Hopefully a few years ago the Chrome OS kernel team added some rules so you need to add some prefix on the commit messages saying where comes up the commits. So when it comes directly from the mainline Linux tree you prepend upstream for example and you need to add using jerrypick you will have the commit added to the commit message. When you need to backport it with some modification so it applies and builds, you need to use backport prefix and for example this is still allowed to take from list because some patches, some very complex patches like for example when you need to submit a pick-up system or hardware decoder you can directly take the patches from the list and it's allowed for example to revert it, to revert them and take the upstream version later. So you can see the process we used so like I said before we worked in our integration tree post to the main list and at each kernel release we did the backport to the yukawa this is the name of the project yukawa tree. So the final the strategy is because when they did a rebase on 5.4 for example Git automatically removed all the backport and upstream patches and only kept the from list for example patches so it was really simple to forward port to to rebase on the on the new LTS. So for example we didn't do the rebase Google did it because it was simple to do actually so it's a whole goal of the of the upstream first effort so why upstream first because it's easy maintenance you don't have to rebase and rework your patches at each LTS release because Android only use LTS so you have one year gap of kernel work so you can miss some some framework or work some new API some removal of functions so with the upstream first strategy everything is maintained by the maintainers by you as well because you need to maintain your own subsystem but when you have three or three wide fixes it's done automatically. Like I said a very fast rebase in a few minutes you can use the new LTS for example and you can only focus on the non-ofstream from these patches for example and the most important it ensures the code quality because you need to meet a good code quality to be merged it depends on the maintainer but most maintainer demands high code quality so you need to meet it before going upstream and the codes are quite heavy because it's slow instead of hacking something and pushing on the tree you need to go in a long process often one or two months before having your patch merged there is more work because you need to ensure it's a good quality and you cannot use forbidden APIs you need to fit in the kernel so for example if you want really complex features that are not yet available in the kernel for example complex crypto insecure stuff it will be long to go in the kernel so you need to upstream it but you need to take from the list and it can take a few months before having the support upstream which is the most complex for example all the multimedia code the multimedia features it's the is the worst because because for example a simple example is hdmi hdcp which only had a recent support from intel but we still need to have a lot of support to use the synopsis ip which is used by amlogic so for example this kind of feature it's really is much longer to develop upstream first than downstream downstream first so like i said it's not a really issue because if you can still if you don't use the main the main call it the mainline android tree for example like like us we use a separate tree you can still backport the from list patches on the tree and you can you can even have android specific patches in a limited fashion so another big part of android are the hardware abstraction layers which is a foundation of android because android it's hardware implementation of the framework api for example so at the time google only focused on the frameworks and let funders develop their own hls why because when android was developed at the time the mainline nukes was really poor in terms of multimedia there is no dynamic of extract at all only frame buffer which is very limited you couldn't you couldn't change mode you could only change mode on the pc's on arm those very limited you have hacks to change mode for example you you couldn't have multi multi connectors for example when there was one shared pixel generator you had no sensor framework at all you had almost no runtime proper management only the pc part so everything was really x86 oriented so nearly nothing was for the arm except the uart for example there was no clock clock from work there was no it's really really so they relied on almost on the vendor trees hopefully for example ti had a very very advanced arm support with a clock framework and so on so they managed to support some android device very early so like here you see the android stack where you have the services and the frameworks using the hl so typically on the phone you have the camera hl the audio hl the graphics hl and plenty of other hls like wi-fi bluetooth etc that uses drivers and the kernel camera drivers all the different audio stack display drivers so typically on modern devices you use asa for example but some vendors for example has their own audio stack for example on their trees so and the camera was a long issue because until we have a complete like a lip camera solution you rely on the vendor drivers which is a really complex situation so vendors of roads and all their own hl for display in jpu etc they had their own implementation of a pm power management logging for example so it was it took a long time until iosp could do in vandia because it took time for kernel developer to to propose a new alternative and push them to accept for example to find an alternative to for white clocks it was a very long multiple years discussion and then recently google used for example the alternative for white clocks the drm framework took a very very long time to mature it was an intel lead project and only recently all the arm socs started using drm framework and for use the drm hardware composer for example and the drm hardware composer is still very young in lack of a lot of features unlike all the vendor implementation so how do we what is our hl usage so hopefully the use the main hl for android tv it's a display so hopefully the drm hardware composer hl exists and works very well because it was if we needed to use old fashion or the custom made hardware composer hl it would have been a lot of work so hopefully we use the standard drm hardware composer it has some issues but it works very well we use the standard butoof wi-fi hl without any modification but we need some some custom stuff for them for for gpu the main one is grahlock which is highly tied to jay through the jpu library and driver hdmi cc google has a nice framework for that and we developed a small very very tiny generic cc hl for the linux kernel upstream cc framework all the lighting of android is still uh custom made they don't use at all the lead framework on the kernel because the lead framework is very very limited so until we have an advanced lead framework in the kernel we still have some user space only lights and work toggling gpio and writing ad high square c packets so the biggest part of the project was gpu integration because without the gpu integration we don't have any graphics or it's so slow it's unusable so this is a representation of the graphic stack of android more the android the the the dependency of the graphic module on android so you have the android framework with all the graphic frameworks like as you which use different hls and modules to achieve the rendering on the screen so for example when you when you want to render something on the screen it will need to allocate a buffer through the grahlock module which is vendor specific jpu specific and soc specific uh that will uh this this uh buffer which will go through the open jl implementation vulcan all the the graphic accelerator library that will go to the hardware to to render and then we come back to the drm order composer that will take the buffer and display it on the drm so it's a very complex path because you have a closed source library for jpu for example when when you use mali you have some vendor hl like grahlock it is which is specific which is specific to the vendor mali library for example you have upstream drivers like ion in the carnell which has a new api since 4.12 you have the very complex drm display that is handled by generic hl like drm order composer so you need to mix all that to have an actual display output and to finish the other big stuff it's to decode a video using uh vfl2 memory to memory decoder and it goes through a generic codec 2 so i i did set in the vendor hl for now because it's not yet generic because it's not yet published completely it's it's half published in the isp3 so it's not usable in the isp so the relationship between the gpu library the grahlock is the other composer is uh because they google decided to make their own open gel api unlike uh the lenux egl api the the key foundation of the api is they they have they hook a private handle t structure to each gpu buffer uh this is used because uh it can go through the gpu rendering and come back to the order composer and you can you can still track the buffer and add whatever you want to the structure to say to add the specific handles for example and even the open gel library can use this this private handle t to to do the rendering so in our case it uses the properties of the buffer so the size the stride and for example the file descriptor uh of the demo buffer for example so for the mali implementation uh unlike uh the power vr uh so ARM provides a vendor grahlock module like every gpu provider but uh they change the private handle t structure at each grahlock so it's really strange so you need to have the exact same version for the grahlock module and the open gel library which is built by your vendor and your vendor can also modify this private handle t so you need to use the vendor derived derived grahlock module so for us um mlogic took the the mali ddk for example from for the for the bifrost gpu they added some properties in the private handle t structure and so we need to use their their modified grahlock otherwise it won't work because the size of the structure one match and they added for example they added some properties in the top of the structure so we can you can't even use a subset of the of the structure for example so this makes uh all the the mali uh uh support complex uh because we need to use uh all the grahlock module for example we cannot use a uh a recent one or we will we will need to port all the mlogic code they added in the grahlock module on top of it it's kind of a lot of work and the the counter side is we also need a vendor implementation in the deraille emberter composer because we cannot have a generic mali deraille emberter composer because we need to to to handle and interpret this private handle t and handle t structure um which is not the case for power there for example so uh so we need we also need to go upstream first for this as well because uh we need to have a specific support for this structure in other composer so hopefully uh deraille emberter composer is handled by android guys mostly and uh it was easy to to support our asoci platform in deraille emberter composer but it's it's like upstream you need to wait until it's supported on the free desktop tree then you need to ask a rethink on the isp3 so you can have multiple weeks between uh your your support and discussion on the kit lab of free desktop and actual support in the isp3 uh we have another issue right it's not a big issue because nowadays the new soc can render in 4k for example but i'm logic a rock cheap and murder for years only render the android ui in a 1080p uh to reduce uh bandwifes because uh the video decoder needs this data bandwifes so you need you need to leave it to the decoder and keep so it the problem is that the hardware composer api uh doesn't doesn't distinguish the display physical mode like 4k 60 yuv 4 4 4 uh 60 fps and the ui layer so is the android framework we try to render at maximum of the of the of the display mode for example you can't say okay uh we will go we set the mode to 4k for example and uh keep the ui divided by two for example so it's not possible so the vendor frameworks they lie to android simply uh for for the for them the rock cheap deraille murder compo harder composer and the music editor composer they have a separate handling of the physical display mode so they have uh the list of the mod and ad dec and so on and they have a fake 1080p mode they give to 200 so 100 only has a single mode for example and they didn't know it doesn't know they have the tv for example support plenty of mud so they need to have a vendor api to switch mode for example they cannot use the android api to switch mode which is an issue was in our case we don't have this we we support the completers all the mode up to up to 4k 60 but for example in 4k 60 the ui can be maybe sluggish because it's much more complex to render and 4k for example and for them when you when you use apps apps don't need 4k to render for example so you cannot reduce it so one a big chunk of the project was to implement the new android boot flow because until now the boot flow was a visible vendor specific everyone implemented a different boot flow for example the dt it was sometime added as added at the end of the design image but with arm64 is not possible but still some vendors tweaked the system to still have this dt appended you could have a separate dt and and not so and even like how you passed the the vendor for the vendor the vendor partition was passed in dt for example and not the other partitions the other partition was using the ui d names for example and other times what do you think the nand empty the names so this was not really there is not a single boot flow on how to boot and find the partitions in android so they introduced a new way for android 10 with an update of the boot image format where you use the dt in the second payload and you can pass a dtbo this was the first evolution so you you mount all the system using a ui d and the dt is still in vendor partition is still dt so it was introduced in android 9 and the mandatory for android 10 is still supported for for android 10 because they introduced a new way using the dynamic partition because one issue was about the partition size the partition size was defined in the bootloader because it was fastboot it was the task of fastboot fastboot to partition the the disk the nand whatever if you need to change the partition you need to change something in the bootloader which is can be really complex so now is as i introduced introduced a a super partition which is a big partition you still have the the vendor the recovery the boot partition like before because all these are very specific to the to the soc and all the rest of the disk is is considered at has a super partition where you can flash the the the partition is used as a mtd with a dm part where you can say you can emulate mtd devices with offsets in the in the big partition so it's much simpler because you you only flash a huge partition with everything from android and you can change the partitioning in the android bsp and you don't need to change the bootloader when you want to grow for example the the system partition and reduce other partitions uh so it's mandatory for android android 10 if you don't use the previous one so it's hard to support all the the the bootflows because uh uh these three the three bootflows are not compatible so the simplest support is the android 10 one with a system as root and a super partition and hopefully uh we use the uboot for amlogic so we didn't have to port elkai uh little kernel uh for the soc which would have been a lot of work because uboot was already supported so it was a simple task for us so hopefully google ti and our vendors uh push support for fastboot for example for avb for all the new uh requirements to boot android but you still need to have a very complex boot config in android in uboot which uh with the different which is different between vendors for example house is very different to ti because we use the the generic uh uboot uh boot uh boot steps if we add a custom boot steps with all the steps so for example fastboot recovery and main one and ti did a complete boot boot of uh themselves power issue okay so the reference board uh we managed to support the board in uboot which is great uh but uh we still now need a few patch on top to complete the android boot flow which is not good um of integration issues so uh audio audio is complex because uh there's no there's still no generic audio uh hl uh every vendor still needs their own uh hall which is not good even if they have a very uh good now they have a very good uh framework for audio so ever so you ever you are welcome and you can have a lot of engineers to implement a good one or you need to take the good old tiny hall from siri sojik and do a custom implementation so wi-fi it works uh but it's the mess so they use the the dubia supplicant fork with uh uh weird communication with it but hopefully it seems uh trying to implement their own support for wi-fi like they did on bluetooth bluetooth now has a clean and uh nice uh support implementation support so one big issue is the display mode uh the graphic the the amlogic associties are um uh tv associties so they are designed to support uh old tv modes but since they are supported by stream we also support the dmt modes which are the the classical monitor modes so uh and we also support interlaced mode for example because it's uh it's a digital tv platform but android was designed only for phones from the beginning so it doesn't know what is an interlaced mode it doesn't know you can have multiple multiple uh aliased mode which are still slightly different but still you can have multiple and they are different you can have for example multiple 1080p modes you can have the the hdmi official mode but you can also have the 1080p native mode which are slightly different for example in terms of audio audio won't go through the same you won't you won't you won't be able to have the same hdmi properties like hdr for example if you use the the dmt 1080p for example so all that is complex and it's still very basic so we still we need to merge modes we need to reorder because when you you submit all the modes to the android hardware composer modules it will shuffle it and sort it in a way uh where you will use a weird mode instead of the native mode of the monitor for example so there is a discussion on the the the remote on composer where i did a ugly hack to filter and order differently the mode to give to android until we have a proper way to say to android okay i have this 4k mode which is different to this 4k mode with the same fps but these modes are different we have no way to to tell to that side that android and we have other issue which is ongoing right now which is not listed on this document is the hardware decoder android did google did quite work going forward on their hardware decoding framework so now they have implemented something called codec 2 which is their new framework for video decoding and they took the chrome v4 l2 memory to memory stateful and stateless hardware decoding code to support the modern hardware decoding as they are implemented in upstream but still the code is not yet public for example which is not very good and it doesn't follow exactly the new memory v4 l2 memory to memory rules so it's still ongoing and it's not clear when we will be able to have accelerated video decoding on this platform because it's kind of the main feature of the soc it's lame to don't have hardware decoding on the digital tv platform so we have a support upstream in staging because all these hardware decoding stuff is still moving so we are looking to have it supported in android tv so the conclusion you can find the url of the bsp so the codename chosen by google is yukawa you will see a lot of files because we thought when looking on the vendor bsp we thought oh why do they need a hundred of files to support android and we started with the few files and finally we need also need a hundred of hundred of files which is kind of issue because it makes support android support very complex since we most of it is not documented you need to look in the code now for sure android is much more melanics friendly so he doesn't don't need a pile of axe to boot you can use the melanics classic melanics so it's great because you can do the api that we had be adopted namely dm for graphics which is really quite so we can we can have a multiple connectors you can it's much more beautiful it's much more simpler so it's still a long run before we have support for multiple boards and we still need patches on top and for the jpu part hopefully panfrost will solve the jpu nightmare because it's really complex to integrate vendor gpu for now and still for hardware composer we still need some vendor adaptation because we need to decide how we handle all the vendor specific features