 Okay, our next speaker is Robert Foss. Who's going to talk about running Android on the mainline stack? Hi, so I'm Robert Foss and I'm not named after open source software my mom's also named Foss she does not do software in any way Unfortunately for me. I work for Calabra and we're totally hiring in case you're looking for something open source related to do Just come talk to me or look at the website And if you want to get a hold of me, I'm on Twitter. So, you know, just shoot me a tweet Let's get into it Today I'm going to talk about running Android on the mainline graphic stack, which Sounds maybe easier than it actually is. It's a little bit complicated, but let's get into it so we're going to talk about the Android history with respect to open source and also How it's actually done how you run Android on the mainline graphic stack And what the current status is as well as maybe the bigger picture What's happening where? Android is going where open source is going with respect to Android Yeah, so if we start with the history This is sort of what it looks like. This is the number of lines of diff against the Linux mainline kernel. So this is for Qualcomm. They're just, you know, a vendor There's a bunch of them, but chances are that you're using a Qualcomm SOC now most of us are so Most of these vendors have a diff of around one million lines at the very least and as you can see Qualcomm is between like 1.5 and like 3.5 million lines and This may be a slight like downward trend. Maybe it's too early to tell. I guess we'll see when we have the sources for 4.14 of the kernel Anyway, it's kind of interesting to visualize what's going on So this all sort of started or this talk is starting where Android forked the Linux kernel and That was in three ten days and they did it for Yeah, for good and bad reasons, I guess But they really needed a better graphic stack So the current one that was in the kernel the ABI was just not good enough. It didn't suit their use cases. So There wasn't enough support for the low power use case, which obviously is important on a cell phone you don't want to burn through your battery in an hour and Also, they didn't support Atomic operations and atomic operations are When you change some settings, for example, let's say you change the resolution of your monitor and maybe the bit depth at the same time you want to change both and With atomic operations, you can sort of do both at once and either both will fail or both will succeed But it prevents you from ending up in a really weird like unknown state where some things have been applied Maybe some haven't So atomic operations was definitely definitely needed for for their use case And as far as the low power use case goes There wasn't much support for like exotic hardware like dealing with the low power display use case. So display Managing hardware of different kinds had pretty poor support So they really needed something better So they introduced the Android atomic display framework or adf as you'll see it called No one really explains what it is. It's just adf and the the adf was written by an Android kernel engineer Greg Hackman and It was sort of written from scratch. So it wasn't built on top of the current or then current Graphics ABI is provided by the kernel. So he just did his own thing Which made it possible for him to get this working quickly But it also ended up being a little bit problematic so Adf ended up being not in terribly Extensible or generic. So it suited their use case of mobile hardware Well and did everything they needed but for the generic maybe desktop use case. It wasn't enough It also didn't really support like all kinds of different hardware So there was it was kind of limited in how it was designed For example, it didn't support atomic operations for for all parts of the subsystem so in the kernel, there's a bunch of different like logical components like There's the connectors which are you know connectors like the DRM port or the the VGA ports or the HDMI ports or the display ports There's also planes planes that are like buffers with some properties attached to them So it's basically what you see like here's a plane of pixels And there are like CRTCs, which are CRT controllers there It's maybe perhaps a bit a bit of a misnomer nowadays CRTs monitors aren't really around but CRT or monitor controllers still are and in the kernel they're referred to us the CRTCs so anyway all of these components have properties and They need to be changed in an atomic way in order to reduce or to avoid flickering and avoid getting into an inconsistent state So that's why you want atomic Operations, but just having them for planes is not really good enough like sure it solves some problems, but it doesn't solve all of the problems Also Also, it wasn't built on top of the current KMS API Which was problematic because that meant that every driver that existed for Linux didn't support this like none of it did So what happened was that all of the proprietary vendors went away and Got busy coding and they all implemented ADF. So for example Yeah, let's just go with arm and Molly like they have an ADF driver everyone has an ADF driver if you're shipping a cell phone and That's entirely separate from from what's being implemented for the open-source mainline kernel, which is unfortunate So this wasn't really upstream of all while it was an improvement in some ways It was not really good enough for being accepted upstream so instead a bunch of people Daniel better specifically worked a lot on on implementing the KMS atomic API, which is Pretty much the the same ID, but a bit more generic So it's the atomic API Built on top of the previous KMS drivers. So there's a pretty smooth like upgrade path you have your KMS drivers that you've always had for like the Intel drivers or whatever and then you start adding support for the atomic features and You don't have to throw all of your driver away Which is pretty nice because it turns out riding a driver takes a lot of effort and time and money So people aren't too keen on doing it And it supports all of the ADF use cases and it does it in a more like generic way And it's being generic through adding Properties to these different modules like the planes and the CRTCs and The connectors so there there are these properties. They're basically key value mappings like you'll have a string Go with like name and then you attach a value to it and that's basically it's very simple so you can inject whatever property you want and user space can read out these properties and Understand what's going on with the hardware and Now like no as in currently it is displacing ADF from the vendor drivers It's still very much like a work in progress and their drivers are still very much not open source for the most part I'm still talking about The mobile vendors here, which they're notoriously not so great at this I would say But internally they're switching from using ADF to to using Atomic KMS, which is very nice because it's the step towards open source It's not perhaps very tangible yet, but it's a step in the right direction so Let's have a look at what the Android graphics stack actually looks like It's a bunch of stuff On top of which we have the apps and this is the stuff you actually care about like like really This is what it's all about And below these apps and there's like a bunch of apps There's not just apps that you think of as apps, but there are done a bunch of different applications doing But yeah stuff like running the the status bar for example updating your clock Below that we have surface flinger and surface flinger sort of takes all of these Surfaces that apps draw so let's let's speak about The parlance here so there are surface surfaces there are planes and there are layers and these are all the same thing So for various first for some reason every vendor has chosen their own word and Now we have a bit of a confusing word salad going on, but the surfaces our planes our layers So surface flinger basically deals with all of the planes and layers and surfaces that Applications draw to and then combine them into something do something useful with it And then sense it onwards so for example if we look at this Android desktop It's just an ordinary Android desktop in portrait mode. No landscape mode and on top here got the status bar and As you can see it's like mostly transparent. It's pretty skinny. It doesn't require like a full buffer like a full resolution like 1920 by 1080 or whatever resolution buffer to to represent it And then there's the navigation bar, which also is pretty small like it doesn't require a full buffer to represent it and they sort of overlap here and as As you can see there's no problem there and then there's the background Which is rendered obviously for this for the most for this most in the back And it does require a full buffer and in order to handle this and like render it properly you have to pay attention to Well where your surfaces are supposed to go and they have to be organized somehow and that's basically what surface flinger does so it takes all this this mess of surfaces that applications create and then Gives it to the hardware and And that's basically what it does so it takes the surfaces Communicates them over a protocol called Hwc the hardware composer API and talks to the hardware composer So what's the hardware composer? It just receives layers from from surface flinger through the Hwc API and then it optimizes them We'll get into that and then it outputs it to the display hardware And that's basically it. It's it sounds very straightforward. You take the layers and then give them to the hardware But it gets kind of messy in that hardware is not like ideal it's it's not ideal hardware, so if you have a a piece of silicon that supports Outputting stuff to your display it probably doesn't support Infinite amounts of layers because that will require actually having infinite amounts of memory Which hardware vendors are very much not into so there's sort of a sweet spot around four layers. That's what Android Google and Android Tells the hardware vendors to implement. It's not too much memory, but it'll cover most use cases and In return Android will try to not have more surfaces than four because it gets messy So what happens when you have more than four Surfaces if we look at the or think about the the background image or the Android desktop image We were looking at we listed three surfaces But the icons weren't listed. So that's another one and what if you get like an error message or something something pops up? That's five. That's more than the hardware can support and this is where we get Into optimizing layers for display. So now you have more layers than you actually can output And you'll have to do something about it. So you'll have to smash some of the layers together So that you'll actually reach them the magic number of four or less and The way you do that is sort of important You can choose which layers to combine intelligently. So maybe you'll choose to Combine the very the smallest layers you can find just combine them and do as little work as possible outside of the display hardware and just Output the remaining combined layers to the the hardware So this sounds really messy and it sort of is it's definitely complexity and there's a good reason for it so oh There is a good reason for it and the reason is power savings So the display hardware is far far more efficient at at doing this hardware composing work than your CPU is and also than your than your GPU is and furthermore, it's not just about power efficiency, but Removing work from the GPU and moving it to the hardware Display hardware means that your GPU can do more work. So you'll be actually faster, which is nice We all want that power efficiency and speed have your cake and eat it, too So if we look into this little box that implements all this junk, it's the It's the driver. It's the use of space part of the driver and it implements the hardware composter And it also implements all the good stuff that you're used to like open GL or Vulkan and implements memory allocators all kinds of little parts and Below it, of course, we have the kernel has to be somewhere so Part of the driver also most likely lives in the kernel or definitely lives in the kernel But we'll get we'll get into that So So now that we have like now that we got like an atomic KMS the atomic KMS API for talking to to the graphic subsystem of the kernel we're full of the happy fuzzy feelings and What Google actually did? Was to ship a device using KMS. So they took their own framework Android atomic display framework and just threw it out and Built the Google Pixel C which is an interesting device because it uses the KMS API But it also runs a close source Nvidia driver because there's no sufficiently good Open source Nvidia driver for their part so They actually needed something to implement this HWC part Or HWC API because there there's nothing in the open source stack that Implements it like Mesa doesn't implement it the kernel does definitely not implement it. So something has to implement it and What they came up with or rather Sean Paul at Google came up with was DRM hardware composer and It's an it's a hardware composer built on top of the normal Everyday Linux desktop stack and Let's have a look at that So this is the same stack we were looking at before but the open source stack is is a little bit different So if you want to replace this proprietary blob with with something we'll have to figure out what something actually is It has to implement HWC to which is the API currently used and It does that using DRM hardware composer, but there's also more to it We need more parts the the driver Does have a few moving parts in it and The the biggest parts are its Mesa Mesa implements OpenGL and APIs like that, and there's also DRM that's the kernel DRM subsystem So that's where your kernel drivers live for your graphics card And in order to talk to DRM or the DRM subsystem in a convenient fashion There's lib DRM, and it sort of abstracts away some of the pain of talking to the kernel When you interface with the kernel you'll talk I octals and I octals are no fun to program and So lib DRM abstracts away a bunch of that and just gives you nice functions nice Functions that give you the warm fuzzy feelings, and it also abstracts away some some other boilerplating stuff Which you're not particularly interested in But there's then there's more there's even more then there's a growlok It's sort of It's one of the I don't know sort of pain points in the stack There is currently as far as I know at least four open source implementations of growlok And they all basically do the same thing, but they're slightly different so There's the DRM growlok for example, and that's the old open source implementation There's GBM growlok, which is the new cool one that we all use and love and then there's what chromium OS ships Which is mini GBM, which is probably even nicer And then on top of that Intel has their own like fork of mini GBM called just Intel mini GBM I guess Which is a support for even more features and they need features to enable their hardware So that's very understandable, but now we have four four implementations of basically the same functionality So it's a slight mess And we're working on that we're gonna see if we can shrink it down to one again. We'll see So yeah, anyway, that's what the entire stack. It looks like I've omitted basically nothing. This is what it looks like So if None of you have an idea of what the DRM Harvard Composer does and where it lives. Let's talk about some some recent developments, I guess Sometime I'm not sure when it was introduced in an Android, but a Bunch of years ago Android introduced a feature called fences or buffer fences and it's a very nice feature it allows you to have a buffer and then sort of associate Events to it so you can wait on the buffer to become available which means that you can do stuff like Just communicate buffers between processes in a secure and reasonable manner you can get you'll always get the expected results and You can also synchronize buffers Which means that you can have a part like for example the the webcam it creates a buffer like it takes a picture creates a buffer and then touches like a fence to it and With that fence it can then send the buffer before it's even done writing it out to the GPU and The GPU can go. Okay. This fence isn't ready yet ping me when it's ready and When it's actually ready and all of the pixels have been written out to the buffer the GPU will be poked and then it can use the Buffer immediately for like as a texture and do GPU stuff with it So it allows distinct pieces of hardware or processes to communicate in a nice way and it Really reduces complexity for hwc Or for hardware in the hardware composers previously you needed a bunch of threads to do like to like Paul buffers Have a look at them and then some some u-texts and semaphores, and it was like a slight nightmare But with fan says it's a lot simplified. You can just wait for the fence and do and it's done. It's done and The user space implement implementations sort of just shrink and the complexity really shrinks. So it's very nice in that respect and Mainline received support for fan says in 2016 thanks to Gustavo over there and As a part of developing support for Francis the orm hardware composer also received support for for Francis and hardware composer too So it's sort of a I shicken in the egg problem where if you want to upstream a feature to the kernel You have to implement it somewhere can't just upstream on tested code. It has to be implemented somewhere So the orm hardware composer was chosen because it's a relatively small implementation of Client to these a bi's so The orm hardware composer implemented support for Francis and hwc2 and then the kernel got explicit fencing implemented or upstreamed Yeah and the current status for a The orm hardware composer is that we were just moved like a few months ago Google Allowed us to to move from Chromium OS like the project was created as a part of the Chromium OS project A lot of good interesting stuff comes from there But it was also hosted within that project which made it kind of hard to contribute And as an open source project you kind of want to lower the barriers of entry to zero And we moved to free desktop org where most of the You the Linux desktop graphics projects actually live like Mesa for example lived here. I'm all this good stuff lives at FD. Oh And I'd like to just thank Google for it. They didn't have to do that and It's certainly inconvenient for them Now they have an external dependency instead of an internal and it just makes their workflow More cumbersome. So it it wasn't nothing for them to do. So I appreciate it Sean Paul who created DRM hardware composer pushed for this Pune Kumar and Marissa Wall also helped out. So thanks guys. It's very much appreciated And furthermore We're creating a GitLab instance for really for all FD. Oh projects, but hopefully a DRM hardware composer will be one of the first ones To to move and Also further lower the bar of contributing just have an interface. That's What people are used to seeing I guess Make it simple as simple as possible We'll see exactly how how it turns out if we'll keep mailing list development or not, but We probably will we'll see how it goes Anyway, so we've covered what it is and and why and how we got here. Let's have a look at where we are and There's a few platforms that have been tested and I said I say tested but that we brought up There's the IMX 6 Which is a very common embedded platform It uses the the vivante GPUs the 3000 series or some of them use the 3000 series And last last year like for most of last year, we were working on on bringing this up and thanks to the The vivante etnavib graphics driver. This is finally possible Like a lot of really good work has gone into reverse engineering the vivante GPU and it's like in a really good shape now like More so than than most other graphics drivers actually so that's very nice Lucas stack christian germiner Vladimir van der laan Were amongst those that helped out doing this or actually made it happen And I I just used their work. So I'm very thankful There's also the Dragon board 410c. It's based on the Qualcomm 410 So it's an old cell phone so see that they sort of made into An embedded system because I guess that was the the simplest way of doing it And it runs an adrino GPU. So adrino GPUs also have Rather good driver support in the freedrino project and the freedrino project was Created by rob Clark and he's sort of been pushing that far that ahead at Like an alarming rate. He's incredibly productive and so bringing That android graphics stack up on this device wasn't too hard The the gpu support was already there And that brings us to what's being worked on currently it's the high key 960 it runs the I think it's the high silicon 960 so see Which has an arm GPU and there isn't really a viable open source driver for the The well basically any arm gpu But specifically not For the g71 which is their latest architecture that it's called bifrost and there there's a There is a project for creating a driver, but it's still in the very early days. I really hope to see it Push ahead because it would be nice to have a Truly open source graphic stack running on this but for now we're stuck with proprietary drivers And currently Lenaro is working at bringing Android up on this device John Stoltz is doing a lot of really good work And it shouldn't be too long before this one is in the the verified Column So hopefully they'll work out Yeah so That brings us to the last part the big picture what's actually going on here and there's There are some conclusions we can draw So new features seem to get introduced into android and some of the features are like like really good like Solid improvements from the current status quo And those features eventually get moved into the kernel So that that's true for the fences we looked at And that's also true for the atomic display framework. Like it was not moved but Re-engineered like it's it was a good idea. So we did it again properly And now the kernel is way better because of it so When you think of android It's obviously like a mixed bag. There's lots of closed source stuff in there But it does push us forward like if nothing else through Just doing their own thing that maybe is actually better But this isn't necessarily true for all parts of the kernel If we look at the diff again, here's the the qualcomm versus mainline diff for version 4 9 and There's lots of stuff like mostly drivers that Never really will be moved into mainline And that's for various reasons like there's maybe a lack of incentive on qualcomm's part Or maybe the driver is only available for this one so see one time So if they take the time to upstream it it'll be sort of Not so useful going down the line But if you look at this graph About two-thirds Our drivers and there's some other stuff in there, too Much of it's multimedia related so Graphics or or video Not all though Yeah And the diff seems to be like fairly constant like if we go back to the the graph we had a look at initially It's not a clear trend here like If we look at at the 318 bump here, that's The the diff from 314 to 318 is mostly drivers again like Drivers can get added drivers get removed there seems to be a lot of churn for their uh, associates Which is an interesting data point but Uh Maybe it isn't the most optimistic or or it doesn't cause a lot of optimism not that a point like there's churn. I'll probably Never be fully like mainline the stuff But it's interesting nonetheless to in terms of painting a bigger picture um and also We've seen That open sourcing graphics drivers has been really like Pushing the development speed forward in terms of like You creating a new device and you already have 95 percent of of the hardware supported And maybe you have some quirks, but if you just can get started and see something like render on the screen That's a pretty good place to start you can fix what whatever is remaining and uh, it It isn't too much work Whereas if you have a proprietary driver and it doesn't work like that's that's unfortunate That's uh, there's not much to do about it except for like calling the vendor and hoping that you're Like big enough fish for for them to care, which may or may not be the case It also lowers development costs for vendors. We've been seeing this recently with uh with qualcomm and the adrino drivers They're actually contributing to the free drino driver because it is good enough that it is valuable to them So they're pushing support for their latest or upcoming gpus into the free drino driver So that it can be used, uh, which sort of means something And it's also interesting Um, also we've been seeing like a nice bumping in driver quality And if something is broken, it's easily fixed. So that's in very much in our interest Uh, and uh, and lastly all of this sort of sums up to pushing open source forward Uh, at least in my mind, I see like compelling evidence for features being moved from the proprietary domain Into the open source domain and then even being main uh, mainline, which is what we all want like that That's what gives us the the warm and fuzzies Yeah That's that's it for me. Uh, if you have any questions, just go ahead and ask Any questions from the audience? So do you think the reason that um, some vendors are beginning to pick up and uh, use the existing stacks is because There's a lot less for them to develop I think uh, the reason is that they're seeing the man like people use this stuff and they wanted to work Well, like out of the box, there's there's value there to them. So That's why they would be inclined to to contribute. I think so if you create something that's valuable enough They will be interested and they will contribute if if they have the resources If you look at a vendor like vivante, they're like pretty tiny like it's a very small company Maybe they don't actually have the resources to push for something open source themselves But on the other hand, we're pretty far along in terms of support for that platform. So that's good Hi, you mentioned that uh hardware composure Also does some squashing For the layers, right? You mentioned that the hardware composer squashes layers. Um, yeah Does the drm hardware composer do that also? Yeah, but maybe poorly so When you have too many layers, uh drm hardware composer will Fire up the gpu and uh through open gl smash some of the layers together It won't be particularly clever about this. It'll just grab some layers But maybe what you really want to do is grab the smallest layers because you're doing expensive computation And you might want to minimize it if you can do it more efficiently somewhere else Yeah What's the status of uh vulcan support and where does it Where does it stand in the stack you shown? Uh, so vulcan support as for the status. I'm the wrong guy to ask But it's implemented in mesa and uh also in the kernel drivers But you mesa would be what you would be talking to so mesa implements lots of these apis It's basically like a sort of a catch-all project for many of these. Uh, thanks Any more questions? Okay Thanks very much