 OK, so I think we're all settled now. I'll start my presentation. Hi, welcome to the session on Android Common Kernels. My name is Amit. I work for Lennaro. In this session, I'll talk about Android Common Kernels and it's out of three Petset. We'll see how the Petset has evolved over the years, started from 2009 now. It's been almost nine years. Then we'll take a look at the Android Petset in Android 4.14 Tree. I'll not be covering what is in Android 4.4, but I'll compare the Petset, as you'll see in the next slides. Then I'll talk about the upstream story of those patches if people have tried upstreaming those patches and whatever success they've got. Next up, I'll talk about Android Mainline Tracking Tree. It is something new, which has started a couple of months back. I'll talk about why do we do this. And I will wrap up the session with the Android Petset Delta, how much we are carrying, and a call to action if something which, as a community, we can do about that. OK, so I'll start with Android Common Kernels. So Android Common Kernels is, OK, Android Common Kernels is respective LTS version plus a handful of upstream, a handful of out of three patches. So I have listed down these patches based on why they got rejected upstream when they were pushed for RFC. So broadly, you can classify them as the features which are specifically tailored for Android. For example, Qtag ID and interactive governor, which is a good example. It is no longer there, but it was mostly Android specific. Then there are a few features which got rejected upstream because they were not implemented the way upstream wanted them to be. For example, paranoid networking and few USB gadgets. And Android developers use this Android Common Kernels as a testbed of the patches which they are experimenting with, which they are trying to get useful, trying to make them more useful, and like a testbed before pushing them upstream. OK, so this one is quite interesting. There are a few features which got implemented at that time because there were no mainline equivalent of them. But for example, this PPP patches, when these patches got implemented, there were no equivalent functionality in upstream kernel. But at that time, they implemented these patches and they kept using it. They did not switch to upstream implementation until recently. Then there are a few features which Android Common Kernel is carrying because it is something which is more useful from community point of view. The patches which they have got from their OEM partners, for example, SD card FS. Now Android Common Kernels, it acts as a channel for downstream OEM partners to get LTS fixes, security fixes, and new Android features. So this is the Android Common Kernel tree URL. You will go through the URL. You will find all the release branches which are currently active. I'll briefly talk about the kernel branch hierarchy here as well. If you see that this Android Common Kernel is forked from LTS and then out of, I mean, for example, here you can see the example of 4.4. It doesn't make any sense here because 4.4 is mostly on Android O. I'm not sure if it is on any of the Android entries. It is mostly on Android O. So you'll see that Android 4.3, the Common Kernel, it was forked from LTS tree. Then you see Android 4.4-n, which is desert-specific release, and those are desert-specific release. Then Android again forked those particular releases for their OEM partners, which are not really interested in taking the LTS fixes as such. And you will see these release-specific trees, they get patched with Android security fixes. And those Android security bulletin-specific fixes. So the good practice here is to start, to pull from the upstream trees, for example, to pull from Android Common, not from Android Release Trees as such. You can get a more detailed information at this URL where they talk about Android Common Kernel and what are the best practices you can follow. Now so this slide is a last-minute addition to this talk. If you have ever worked on Android Common Kernels, so you will, I mean, let's talk about two or three years back. At that time, these trees were more like a patch-set dump. They never used to compile and forget about booting them on a real device or any of the, even on QMU. But now if you see that they are, I mean, they are continuously being tested, build tested on Kernel CI. I've posted the link here. And we test these common kernels on, as a Linux Kernel Functional Testing Project. So Tom gave this talk yesterday, so if you're interested in learning more about the Linux Kernel Functional Testing, you can check out the slides, and hopefully videos will be available soon. So as part of Kernel CI, as I talked about, they do this build tests and basic boot tests. The next Kernel Functional Testing, we run VTS, ETS tests, and upstream LTP and KSELF tests. But those, we run mostly on open embedded builds, not on ASP builds. Now the Android Patch Set Evolution. So this is the talk which John Sturz gave in 2011, ELC. So this is what he said, OK, so these are the patch sets which we are carrying in Android Common. So all the usual suspects there, Ashman, Binder, Wakelocks, if you're familiar with Android Common kernel. Now fast forward to 2016. So this is the talk I gave two years back. So you will see that all these red lines, all these patch sets are dropped. So we no longer use them in Android Kernel. And all the green lines are the patches which actually got upstream. And yellow was Ashman moved to staging tree. But with 4.4, we saw that, OK, so we don't have this delta, but we have this additional atomic display framework. We have verified boot. We have SD card affairs. We have interactive governor. And energy of a scheduler was new at that time. So now that we have seen how the patches have evolved, I'll talk about Android 4.14. I mean, from now on, all the patches I'll be talking about, all the status I'll be talking about, are related to Android 4.14 Common Kernel. So this is the GitDiff delta. So majority of the code is SD card affairs. Then we have got net filter changes, energy of a scheduler, USB gadgets, input devices, FIQD burger, and other things. So I've put another slide for comparison. So this is the slide which I just talked about a few minutes back. So this is the 4.4 GitDiff status. And this is where we are today. So you'll see that apart from the networking, because at that time, networking used to be the bigger set. It used to have this UID-based routing and PPP patches, which I talked about, and other things. USB gadgets had MTB PTP. Now, all the red lines is that all these frameworks or all these patches, we are not getting anymore. So UID-based routing, it got upstreamed. PPP is something which we have moved to a upstream interface. So we are not getting those patches in 4.14. MTB PTP patches got dropped because we have a user space dimmer now, which takes care of MTB PTP functionality. Atomic display framework got dropped in favor of DRM KMS support, hardware composer support. Infective governor again, we are no longer using it because we have moved to energy of a scheduler now. So if you see the shift here, so earlier the networking used to be the bigger set. The big chunk of code sitting outside of mail line tree. Now it is SD card FS. Mostly because we have dropped so many things from networking. Same thing with USB gadgets. It's just dropped from third place to now fourth or fifth place. The rest is pretty much the same because nothing much has happened on those patch set yet. So to give a broader view of out of three patch set, I just dig it more. So you'll find some useful information here. So I'll try to touch most of these patches and most of these features and talk about the upstream status if we have tried anything, if someone is actually working on these things or not. So we'll see. So we'll start with SD card FS. SD card FS is a fuse replacement. For a long time, we used user space fuse dimmin, which takes care of this emulated storage in ASP. So Samsung came up with this SD card FS. So they say that, OK, it is working much better than using a user space dimmin. Let's have an internal implementation of emulated storage, which can emulate all these permissions and case sensitivity. So it was tried and tested because Samsung was shipping this driver in their devices for a long time anyway. So upstream, sorry, ASP just pulled out these patches and they pushed into 4.4. I mean, it started with 4.4, I think. I'm not sure about 3.18. I might be wrong there. So Daniel Rosenberg gave this talk about upstreaming SD card FS in plumbers last year. So if you are interested in getting us know what is happening from upstream point of view, you can check out his talk and some very useful information there. So a couple of months back, I think, ASP dropped fuse support at all. So first it was you can either use fuse or you can use SD card FS. So now they have removed the fuse support, and we are using SD card FS only. So what happened because of that is that a few people in NROV run mainline kernels on development devices, high-key DB410C, and we run into this breakage. So it's not something which we cannot live with, but it is something which is clearly broken, and we have to do something about it. Maybe get the fuse support back or speed up the streaming process, because if it stays like this, then it's going to be difficult to run mainline kernels on supported devices. I mean, if your device supports that. So last is, Huawei's SD card FS implementation. So if you track NROID, get it, you'll see that with 4.14, I mean, Huawei has their own SD card FS implementation. And if you go by the commit messages and the replies which you see on that message, they claim that it is much faster than using the SD card FS which they have currently. So I just put these two lines just like that, if you're interested in this. Right, so the next is F2FS. There is no out of tree delta here. I just put it because there are a few chunks of code which is there, but it is all upstream, so nothing to really worry about it. Squash FS is, yes, there are a few patches which the claim is an optimization, and if you should do this, it's a, I won't say recent changes, but yeah, it's fairly new. It's fairly new, and I have not seen any upstream activity on these patches. Right, so the next big chunk of code was networking. If you have any questions, you can just interrupt and ask. So networking. So there are a few net filter module specific changes, for example, Quota 2 and QTag ID, where they track the usage, where they track the, which mostly use for socket accounting, right, based on a UID, that this particular user, this particular process, used this much of data. So this is why this QTag ID and Quota 2 is mostly used for. The good news on this particular module or on this particular patch set is that they are moving to EBF now. So they are saying that, OK, for now, you can use QTag ID matching module, but going forward, I mean, they are very positive that by the time OEMs will move to 4.14, we won't be using QTag ID. So we will be using this EBF implementation to track the socket accounting and everything. And there is a very small change in idle timer notifications, where, right, so current driver supports sending new events that, OK, I'm in idle state, or I just came out of idle state. So instead of that, user space expects net link messages. So there were just few lines of change of code in this one. So upstream, when it got upstream for submission, the maintainers did not have much objections to that. They said, OK, if you are sending new events, if you want to send another set of messages to notify user space, they were OK with that. But no one followed on that particular patch set. And when I talked to one of the Android developers, they say that even they are not really worried about this patch, and they say that let's move this particular functionality to user space instead. So hopefully, next Android release won't be having this particular patch. And then there are few patches which implement this paranoid networking. Paranoid networking is restricting the use of network to a particular user or group ID. So if a process, if a user is a part of this particular group, then only it can access network. It's a very small chunk of code, just a couple of header files with user IDs which are mapped corresponding to the user space Android IDs. Now, that is a problem, because upstream developers do not want hard-coded user IDs. So it got rejected because of that. And there have been a few suggestions that maybe you can use namespaces to implement this much better. But we were told that it requires a fair bit of userland changes. And it is unlikely to happen just to save a few lines of out-of-tree code. But even user space, if you follow the changes, you will see that now you can restrict or you can allow particular services or particular processes if they have upstream compatible capabilities. For example, CapNet draw and CapNet admin. So upstream capability support is slowly growing in at the user space. So hopefully, the checks which we do at the kernel side, that if this particular ID is there, if a user is of this particular group ID, then do this or don't do this, will. And I mean, I am hopeful that we might be able to get rid of this feature quite soon. Not soon, but eventually maybe. Energy-aware scheduler, I'm pretty sure that most of us here are aware of this. It's a hot topic. So it's, again, a big chunk of code which is sitting outside of mainline tree. But ARM and linear developers are actively working on it. And you can track the discussions on mailing lists. You can track the discussions on ARM GitHub, who have provided the links here if you want to see the current patch set. So there were a few patches. I already talked about MTP and PTP. Those patches are no longer in 4.14. You will find it in 4.4. But at least pixel devices are not using those drivers. They are all moved to user space. But few devices, for example, all the 3.18-based devices, I think they are still using those MTP-PTP devices, but not config.fs-based gadgets. They're still on the composite gadget thing, the Android gadget, which was years old now. So the existing set of USB gadgets which are there are what they provide UEvent notifications to user space. So config.fs has a hook that if a USB is plugged in or plugged out, you will just send a UEvent, that, OK, I'm disconnected. I'm connected. So for these gadgets, I mean, developers have implemented a custom class. This custom class, again, is something which is, sorry, it is a legacy implementation. I'm not sure if you're using config.fs-based gadgets, you need that. But this is still there. Now, this patch got rejected upstream because upstream wanted to track the UEvent from UDC core. So instead of putting this particular UEvent notifications from config.fs-layer, they wanted to have it to be sent from UDC core instead. So no one followed up on that because there is a small hack, which we do at the kernel space side, which should be moved to user space before we actually move to using this particular code from UDC core. So that was our device state changes. Then we have accessory drivers. I'm not sure. I mean, I have never seen any Android accessory in use if anyone is even using that. But we still have those drivers, accessory drivers, audio source drivers. And if you go through the documentation on Android official pages, you will see that this accessory mode audio is not widely adopted and currently not recommended for new designs. So I think it's on its way out. But the packages, but these drivers are still there. Then there are two additional drivers, this dual roll USB ccfs interface. Provide a user space interface if you want to switch from host to device or vice versa mode. OTG wake lock is to grab a wake lock if you are connected to a power source, if you have a V bus interface. Common use case is that when you plug in your USB charger, then it just grabs the wake lock and your screen will lit up. So OTG wake lock driver is used for that. I'm not aware of any upstream submissions for these two patches. Android verified boot is the Android's verified boot implementation. So the DM Android variety is completely out of tree. It's not submitted upstream. Mostly because it has some few AB hooks, seamless update hooks, so it will be difficult to see how it will go upstream. Then there are a few changes in forward error correction code. It's not a that OK. So let me take this opportunity to tell you that most of these patches which we carry, they're mostly, I mean, if you will see that trend, that OK, so this particular patch is only to provide stats to user space, right? So I think, yeah, for example, we saw that USB get this thing, right? It was just you and notifications, sending something to upstream. Same thing is here with forward error correction code. It provides a set of CSS attributes to track how many errors we have corrected or other particular stats. So these are the few out of tree changes in DM subsystem. So input devices. So if you have key card and key reset drivers, which when you press certain combination of buttons on your phone or an Android device, then it does something, right? So these input drivers take care of that. Chunk of it got upstreamed in SysRQ driver, but still a lot of code is there which sits outside of this. Then there is this generic GPIO input support which supports different GPIO based input devices. I've not seen any, but yeah, they still have these drivers and support in Android Common currently. Then we have FIQ debugger patches, which is mostly used for kernel debugging for ARM. So the plan is to use FIQ interface for debugging. FIQ is an NMI-like feature which you see on X86 now. So if you're not able to grab or use FIQ, you can fall back to using normal IRQ to get the CPU back traces. Daniel Thompson from Ninaro worked on it for some time. Few of the functionalities got integrated with upstream KDB, but still, I mean, a majority of FIQ debugger code is still out of tree. So few items which people are still working on are KDB extensions, FIQ debugger for AMV8 devices, AMV8-like devices, because it says we are using FIQ debugger, but recently, people figured out that we are not actually using FIQ. We are still using a normal IRQ. So these are the major subsets which I've covered so far, right? Networking, SD card fs, your input devices, FIQ debugger, and other things. So these are small, small patches. The small, small fixes, if you remember that big chart which I showed you. So there are a few patches in memory subsystem which tag memories so that it's easier for user space applications to tag their memory usage. So this one is sitting outside of, I mean, all of these patches are sitting outside of main line tree. Then there are a few C group changes when you move a task from one particular, when you start putting applications based on their usage. So Android has these four classes. It's a top app, it's a background app, or it's a system app, or something like that. So they use C groups, C group changes to do these transitions. And recently, there are a few, recently, the upstream, OK, not upstream, the AOSP supports moving the tasks if you have capsis, nice capability. So it is similar to what we saw in case of Android networking that, if you support a particular upstream capability, you can do that. So this is a positive change. Then there are a few goldfish changes. Goldfish is the Android emulator. So recently in the last few days itself, I see a lot of goldfish changes. I have not covered those changes. So maybe if you look at these changes now, you'll see that, OK, these particular changes are nothing compared to what got included just last week. So this is that. There are a few art specific changes. For example, appended DTB support. And you build, I mean, sorry, when you can just append your DTB at the end of your kernel image, and your bootloader will just support it and boot from it. There are a few patches which dump memory around a few registers in case of failures or in case of a panic. So this support is something which no one tried to upstream yet, as far as I can tell. There are a few changes in tracing subsystem to trace GPU usage, CPU frequency usage, and other memory specific changes. OK, so there are a few changes in MMC subsystem to support embedded SDIO support and other SDIO specific fixes. Additionally, it relies on SD card detection and all those small, small fixes. OK, so this one is new. Sorry. So this memory state, driver is new. It tracks the time spent in different DDR frequencies, memory frequencies. So this got added recently. Well, by recently, I mean last one year or so, there is one security patch which prohibits any user to just don't probe for any kind of events. It got shot. It was pushed upstream for RFC, but no one was interested in that. So it got rejected. Then there are a few small networking fixes, like providing knobs to control the TCP window size or initial congestion window. And there is a small hack using which you can actually keep your Wi-Fi, Bluetooth, or connectivity modules up, even when you are in suspense state. Then there is one in it, in a time-office change, so nothing interesting there. There are a few NFC changes which looked interesting. I tried to push them upstream. But at that time, meltdown and inspector were kind of hot property, so I will try to see if I can resend a new version of this. Or this wake-up reason patch. So what it does is it logs the wake-up source and the time it spends in suspense state. So this got traction, a lot of traction in between. And as I said, so there are a few patches which exist only to give or only to relay certain information to the user space. So you have this wake-up reason source and UID status. UID status does the same thing. It provides stats to the user space. So our next topic is Android mainline tracking. We recently started working on it. So if you track Android common kernels, Android common kernels are mostly LTS releases. So we base Android patches on top of LTS releases. But Android mainline tracking, what it does is it tracks upstream implementation. It tracks upstream kernel. So we keep rebasing Android patches on top of Linux master and we tag it as experimental Android 4.15 or whatever is the next version. So why do we need this? Because it helps us catch mainline regressions much earlier in the development cycle. So instead of just fixing all these breakages, if it is something which is broken and upstream, we can just report it. So this particular upstream patch broke this particular Android functionality or we can report certain ABI. We have reported ABI breakages before while following this particular tree. Again, if you are trying to push something upstream, this particular tree can be used to rebase your changes and work on top of it. Sometimes these trees can be used as a reference or experimental preview tree. For example, let's say we are on 4.13 and you know that 4.14 is going to be the next LTS. But the implementation which you have is Android 4.9. So in that case, if you have a tree like this, where you know that the next kernel version is most likely to be the next LTS version, so you start preparing your SoC BSP based on 4.13 and stick closer to next LTS version as much as possible. So this is why we have Android mainline tracking tree. So this is Android 4.10. So this we did with this Git diff stat is in comparison with 4.15. No new changes here. If you will see that pretty much the same set of changes which we saw with 4.14. So Android patched Delta. So this is the trend with every, sorry. So these are the number of lines added per Android common kernel version. You see that there is a downward trend, but it is not as good as it looks like. Because by that time, the Android 4.9 will go into production as a production kernel. You will see that the functionality, I mean, the developers will keep adding more functionalities based on the newer Android releases or Android features they are trying to get into, let's say, P. So as I said, the trend is there, but we always end up rebasing about 38, 40K lines of code on next LTS releases. And though a lot of changes we dropped in between 4.9, 4.4, and 4.9, and then 4.9, and 4.14. But on the other hand, Android developers keep adding new features based on the next release they're working on. So it's like a never-ending cycle here. So if you are aware of any upstream alternatives which you talked about, these are the patches which sit outside of tree. And if you think that there is a upstream functionality which can replace a particular out-of-tree patch set, then please do suggest or get in touch. Because that's how we got rid of a lot of things, right? For example, PPP, we already had PPP OLTP, I think, is the upstream equivalent of OPNS and what Android already had, but we convinced them that, OK, we already have upstream implementation. So just move away from what you are getting for last so many years now. So that was my presentation. Do you have any questions? So it's a combination of both. So the question is who is actually working on upstreaming these out-of-tree patches, right? So it's a combination of both, right? Google developers, I mean, John Sturz did a lot of streaming and red patches of streaming. And we keep on picking a certain set of patches and we feel that, OK, this is something which can go upstream easily. It's like a low-hanging fruit. So we just take it and try to push it upstream. So we got success there. And while going through this presentation and preparing this presentation, I figured out, OK, these are the patches which looks like are reasonable enough to send them. So we keep doing that. Yes, please? OK, so the question is that when a particular functionality which is implemented in, yes, they have been moved to, yeah. I don't think anyone is formally tracking it. Maybe USB developers are tracking it. But from our perspective, which we looked from outside, right, that, OK, I mean, I don't think there is any, is there any particular process which anyone follows? So we found out about stuff like this when we actually ask them, OK, for example, with 4.14 coming up, when we were revising for 4.14, we asked them, OK, this particular functionality, right? No one is using this particular interface anymore, right? So do we still need it internal? Or they say, OK, we have already replaced it with something else. But how do you do that? Yes, please? There are a few examples, right? For example, OK, so we did not fix it, but we just got rid of it. For example, mtpptp, we tried to push it twice, thrice. We said, OK, so many people, right? Billions of devices are using that particular driver. I at least put it in staging. But there was clearly objections. No, we are not doing it. And they said, no, move to user space, move to user space. And they did. And then we got rid of drivers. And similarly, I mean, John might be having much better examples on this. Another example I can think of is UI Debates Trouting, right? Lorenzo spent a lot of time on it. And eventually, he got it upstreamed. It's a big chunk of code which moved from 4.4. Yes, please? So we've been rebasing from one particular LTS release to the next one. So we used to do this Android Mainline Tracking Tree I talked about, right? So we used to do this activity within Linaro before we started doing it for Google. So we already had, no, instead of doing it from 4.9 to 4.14, we were gradually doing it for 4.10, 11, 12. So I mean, RC ones are always hard. It's like one week, I would say, to keep things in check. So but with Android Mainline Tracking Tree, where you eventually gradually keep rebasing your patches for every RC or release things, it gets easier. So just the basic feature sets, right? So I mean, for example, I use DB410C, right? So 4.14 boots with your Wi-Fi, Bluetooth, with DRM hardware composer and everything. So it makes our life easier. We don't have to do anything. And we know that 4.14 boots, right? So when we rebase and write patches on top of it, right? We just make sure that that particular functionality which we tested with vanilla 4.14 is not broken anymore, right? And then we enable QTAG ID to see if data usage tracking is happening or not. And we enable UID status, right, which provide all these status services. So we do it, I mean, I do it manually. I mean, since I know that these particular patches do all these things, so at least these things should be working. Oh yes, I mean, now we do, now we track all these and write Mainline Tracking Tree as in LKFT as well, which tests out all these, and VTS tests, CDS tests, and no extended set of LTP, KSELF tests, and everything. OK, thanks for coming, everyone.