 All righty, so good afternoon. My name's John Stoltz, and I work for Lenaro. Basically, this is going to be my talk, Android Kernel Web Streaming and Overview and Status. So just an outline of what I'm going to talk about. So we're going to basically go through all the components of the Android Kernel Patch Set and do an overview of it. We're going to talk about the status of what's made it upstream and what's still out of tree. And then finally, going to talk a little bit about why upstreaming is important. So first, the overview of the Android Kernel Patch Set. Basically, the Android Kernel Patch Set is the modifications that the Google Android developers made to the Linux kernel in order to basically support the Android platform features that they needed. Now, for each, basically, Nexus device, there's a huge set of patches that changes from device to device. And these are all in the, basically on AOSP, basically called the Device Git Trees. And I'm not really going to cover those. I'm going to instead cover the less device-specific patch set that's in the Common.Git Tree on AOSP. And basically, this is this functionality that basically all of the device trees are built upon. So I'm going to use this little Butcher's diagram to kind of help us go through all the different components as we do the overview. So first of all, the core platform components. These are basically the components that are absolutely necessary for the Android platform to function. Without them, basically, the Android environment will not run, or you won't be able to interact with it. These functions are Binder. Binder, I guess there was an earlier talk on that by itself. And I think most everyone was there because I couldn't even make it in through the door. But Binder is basically the IPC layer that's pretty core on Android. Basically, since all applications run as different users, basically they want to initiate any communication between applications they need to use Binder. It's basically the framework upon which larger Android concepts like Intense and Content Providers are built upon. We also have Ashmem. Ashmem is the anonymous shared memory system. Basically, it provides to applications file descriptors that allow them to map in basically anonymous memory. Those file descriptors can then be passed through Binder to other applications for them to share that memory. It's very similar to, basically, TempFS file descriptors. But it kind of unlinked TempFS files. The thing that's nice about it is it avoids the problem of having a TempFS mount that applications could write to and basically take up memory since that's a RAM-based file system. And so, basically, it kind of cleans up after itself when the last application that has a handle on that file descriptor closes. Another cool feature with Ashmem is that it provides an ability for applications to basically unpin chunks of that anonymous memory. And this allows the kernel to then basically reclaim it if it needs. And this allows for basically a kernel-triggered application cache purging. Yeah, so it's pretty neat. Next up is Logger. Logger is basically an in-kernel memory log buffer. It's very similar to Dmessage, except for it allows for applications to write to it. In order to avoid one noisy application flooding the entire log, they kind of break it up into chunks. And so you have the system logs separated from the application logs. Finally here, the monotonic event timestamps. Basically, this provides, I guess, so with Android, you have a lot of gestures. And those gestures are basically a series of input events over time. And so you can kind of imagine that the problems that could happen if you set the time of day, you would end up having basically a gesture possibly end before it began. And so that's very confusing. So in order to resolve this, basically, they changed all of the input event timestamps to use clock monotonic, which is a great solution for them, but basically breaks the kernel ABI. So that's a little problematic. Next up, we've got the power and performance improvements. These are basically items that are not absolutely critical for applications to run, but basically are critical for devices. You're not going to want to have an Android device without them. These are the strong shoulders in back of the bug droid there. We've got probably one of the best well-known Android changes is the wake lock infrastructure. This infrastructure basically serializes against races that can happen between wake up events and suspend. When this was first submitted to the Linux kernel mailing list, it created a big controversy, mostly because these races can occur really anywhere throughout the kernel stack all the way from the interrupt layer through basically the driver core through the actual interfaces and into user land. And so it requires this chaining of these lock structures throughout that layer. And it was just not very pretty at the time. I guess connected with the wake locks is also alarm dev, which basically allows applications to trigger events in the future that will wake up the system if it's suspended. So this is useful for things like if you have maybe an email client that polls every 15 minutes checking email, it's able to wake up the system and check email similar for calendars. They can trigger alarms at certain events. We also have the low memory killer. Now because with Android applications, they usually aren't explicitly quit by the user. We need something to kind of go around and garbage collect old applications. Low memory killer basically monitors memory on the kernel and basically when it starts to get a little tight, we'll kill applications in basically LRU order. Now it's a little redundant because in the kernel we already have the out of memory killer, but basically that only triggers when the kernel is absolutely out of pages. So basically once it's flushed the page cache and there's no other pages that can be found. So you can see some pretty negative performance impacts when you hit that out of memory point. And so the low memory killer is a little, I guess preemptive and it acts before we run into those performance problems. And then we have the interactive CPU frequency governor. It's basically kind of part of the code that manages the speed of the processor. Basically it's different than the other CPU frequency governors in the kernel in that while it tries to keep the CPU in a low power state as much as it can, anytime that the user interacts with the device it maxes out the CPU immediately. That way you get good interactive response. So next up are the platform debugging features. These are the hands and feet of the bug droid because you have to kind of feel out problems and then stomp the bugs. So there we've got the Android gadget driver. Basically this is a gadget driver that allows the device to basically multiplex a number of different interfaces over a single USB connection. It can also be configured dynamically at runtime so it can do things like ADB as well as file transfer protocols like MTP or PTP, tethering protocols like ACM and et cetera. We have the FIQ debugger which is basically a low level kernel debugger that uses ARM's Fast and Arp mode. I guess the kind of most interesting part on this one is that if you have the right cable you can actually plug into the headphone jack on some devices and you can get a serial port that gives you access to the FIQ debugger so it's very useful if you're debugging a problematic device. We have the RAM console and persistent RAM. This is basically infrastructure there. The kernel puts aside at a specific physical location in memory, basically the console messages and records them. This is helpful because if the system crashes and it's rebooted you're able to basically capture that data and be able to store the log messages. This is useful because on ARM devices frequently they don't reset their memory when they're rebooted. Then we have the key reset driver. This is basically control alt delete for phones. Basically allows you to set up a key combination that when pressed for long enough will trigger a file system sync and basically reset. The ETM and ETB improvements. This is the embedded trace module and embedded trace buffer. On some ARM SOCs basically it allows for coordinated logging between possibly different OSs that are running on the same SOC. So things like if you have a cellular modem that's running some micro OSs and Linux, they're basically both able to do logging into a hardware buffer and be serialized properly. And then finally the goldfish emulator support. This is basically changes to the Linux kernel to support the goldfish emulator which is the emulator you see when you write your first Android application and start up it's very, very slow. So then we've got the networking changes for the little bug droids antennas there. We've got the paranoid networking framework. Now this is kind of a, I don't know, for Android it's kind of an elegant hack. Basically since Android applications all run as separate users, they have a special group that some of those applications can be a member of and then in the kernel basically it checks to see if the current task is a member of the networking group or not and it allows it access to networking. It's a very small patch, it's kind of elegant but it's not something that could ever really go upstream. We also have the net filter changes which are changes that basically allow for improved accounting of what applications are using data. Then we have the Bluetooth improvements which are basically improvements to air handling and specific Bluetooth modes. And then the BCMHD driver which is basically a Wi-Fi driver that's very common on devices. So this is a vendor kind of OEM drop that lands in the Android kernel periodically. Next up is the graphics infrastructure. For those we've got Ion and Sync. Ion is basically a graphics memory manager. It provides access to different pools of memory that can satisfy various constraints of the hardware. So for example, if you have a camera that's basically recording images and then you want to use the GPU to put some nostalgic filter upon them and then send it off to the compositor to get rendered to the screen. We want to avoid copying memory through that path and so we want to use the same memory or the same buffer basically that entire way. One of the problems is that there may be specific constraints that the hardware requires or various pieces of hardware require. So you might have something like the camera driver that requires that all of the buffers that it accesses are physically contiguous. And so basically Ion manages different pools of memories like contiguous memory or carve out or kernel memory that can be used to satisfy those constraints. After that we have the Sync driver. This is basically a way to serialize different or coordinate different drivers so they can serialize around events. Sounds a lot like a mutex and basically allows the system to be able to expose things like hardware mutexes with some GPU support. This allows you to do things like time drawing with screen refreshes and this is part of the project butter effort. And we kind of have a catch all for miscellaneous items. There's the Android battery driver. This is sort of a meta driver. It's a kind of regular power supply driver but it also has enablements for wake locks as well as uses alarm timers to wake up and check thermal constraints when charging. There's input drivers so specific synaptic drivers for instance and other tweaks like improving or adding wake lock support through the input layer. The switch class and the time GPIO. These are for things like physical switches and vibration buzzers and phones. And then we have the MMC tweaks which again are mostly our handling fixes and wake lock enablement and then further even smaller changes all over. So this is the little vestigial tail on the bug droid for deprecated items. So these are basically things that you may have heard of that are no longer in the current OSP commentary. There's PMEM. This is basically kind of a precursor to ION so ION's gone on and replaced this. Early suspend is also gone and it's been basically user land's been reworked so it's unnecessary now. And so I think with Jellybean 4.2 it's not needed. APANIC, I believe this had a lot of duplication with the RAM console work and that's why it was removed but I'm not totally sure on that one. And then YAFS2 file system. This is basically a file system for raw NAND support and while it's still actually in the tree not many devices I believe use this so I wouldn't be surprised if it disappeared at some point. So now that sort of is the entire overview portion of the stock. So now I'm going to go through the upstream status. So we're gonna lead with the good news. Basically these are items that are already upstream properly in the kernel. So we've got Wakelocks which I think is a, given the firestorm that that created that's basically a huge accomplishment and a lot of credit goes to Raphael Wasaki who's the power management maintainer and Arvay at Google who wrote Wakelocks initially basically collaborating together and managing to basically create what are wake-up sources in the kernel and then extend that so that it was sufficient to provide the semantics that Wakelocks needed. It even includes a Wakelock compatible user land API so applications didn't need to change. For monotonic event time stamps, basically the change that the Android guide was very simple but wasn't appropriate for upstreaming because it broke the kernel AVI and so we basically extended the input layer with an I octal that allows applications to select which time stamp they want to use or which clock event or which clock source they want to use for their event stamps and with the Jelly Bean 4.2 they're now using this so this is quite nice. The RAM console and persistent memory this is more recently merged via the Pstore and Ramoops infrastructure which was basically very similar infrastructure that was there already upstream for enterprise servers and then the switch class has finally gone upstream as the Xcon driver. Now there's also the staging directory upstream and this is basically an area in the kernel where drivers that don't meet kind of the expected quality guidelines are able to be merged upstream and sort of improved iteratively until they can be basically merged properly. Up there we have binder, we have Ashman, we have logger, low memory killer, the alarm dev, the Android gadget driver although it's been renamed as the CCG driver. Also I just got told today that apparently there's some portions that are missing so it actually doesn't have ADV support and then the time GPIO. Now while all of these will eventually need to continually be reworked and merged properly upstream, the nice thing is that because these are in the staging directory that today you can basically boot an Android environment on a vanilla Linux kernel. So for items that are currently in development we've got the FIQ debugger, currently I guess Anton Veronsov has spent some time integrating that with KDB. We've got a good number of that, good chunk of that's already upstream. Further changes are queued for 3.9 I believe and then there's still some portions that need some maintainer review but hopefully that will be all upstream fairly soon. The key reset driver, Matthew Poillet has been working on that as well. He's got some of the functionality queued for 3.9 but it still needs some more work, things like adding timeouts and being able to sync the file system before resetting and it's being integrated with the SysRQ driver. The low memory killer, this is another bit of work that Anton Veronsov has been working on. Basically he's trying to create basically an interface that the kernel will provide to userland that will give notifications if we start seeing memory pressure and this allows basically pushing the killing decision out to userland so that userland could maybe take other actions like notifying the applications to shut down and save state. That's still kind of in discussions. It's one of those things that keeps on seeming like it's almost ready to go upstream but then somebody suggests one more change. Ashmem's unpinning functionality. This is something that I'm actually working on. Again, this is the functionality that allows basically applications to have kernel triggered cache purging. And I'm trying to make it more generic and pushing it as what I call volatile ranges. Unfortunately, I keep on having to push it further and further into the VM layer so it's been a little slow going so that's still something that's being worked on. And then also the Goldfish platform support, Alan Cox I believe has been submitting this for staging and I'm not sure if it's queued for three nine but it might be I believe in the near future that should be headed upstream to staging. So there's still also quite a bit of work to be done. We've got basically the graphics changes, both Ion and sync. Neither of those have been submitted upstream or sent to staging. There's some work in the community that's trying to basically enable similar functionality but I'm not sure if those efforts, things like the DMA buff fences, yes. I'm not sure if that will be sufficient functionality to replace what Android needs. So hopefully we'll be working on this more in the near future. The interactive CPU frequency governor, this was actually submitted upstream but was nacked by the scheduler maintainers. And the reason for this is because basically they want to kind of merge the CPU frequency governor logic at some point sort of closer with the scheduler itself. It's kind of a pain to have these two separate systems when deciding where processes should run and then a separate system deciding kind of what the speed of those processors should be. The net filter changes and paranoid networking. So the net filter changes were submitted upstream and got a fair amount of feedback. There weren't any major objections, basically just a lot of cleanups. So that's an area that basically just needs some work to clean up and get those patches merged. The paranoid networking, now that's not something that is really appropriate for upstreaming since it is such a very Android specific kind of hack. That said, it's been suggested that the network name spaces would be sufficient in order to provide the same functionality. Unfortunately it would require a fair amount of heavy lifting to change the Android user land to make use of that functionality. So I'm not sure if that's likely to happen in the near term. The alarm dev, basically while it's still in staging, much of its functionality has been properly merged upstream. It's basically just the raw dev alarm driver that basically the interface that needs to be preserved at the moment. This interface is very similar to the existing timer FD interfaces. So basically folks have suggested that we kind of merge these at some point. The difficult decision there is basically if we do merge them older applications that depend on the alarm dev interface might break. So trying to figure out what the right approach is there is something to be decided. The ETM and ETB improvements, those both, or I guess those have been submitted upstream. One of the problems there though is that the maintainer of that code has actually changed jobs and is no longer really maintaining it. And so it's one of those things where there's not a lot of documentation around it and not too many folks who are well versed enough to be able to review it in detail. It does need a little bit of cleanups. A lot of documentation as well would be helpful. So that's still some work to be done. We also need a non-staging path for binder and logger. I know for logger, some folks at Sony were working on it, but I haven't heard anything recently, so I suspect that's kind of stalled out. And binder as well, it's kind of a monster of a problem. But apparently Greg Hartman has recently announced that he's gonna be working on a new in kernel IPC layer, trying to merge some portions of basically the debus functionality into the kernel. And I believe the hope is to make it so that it would be something that binder could be implemented on as well. We'll see how that goes. Also again, all the small little miscellaneous items throughout need minor cleanups and pushed upstream. So this slide's a little old, so hopefully I'm not missing anybody. But I went through all of the upstream commits and basically pulled out anything that was related to the Android patches and pulled out the patch authors and have them all listed here. I've highlighted in green the Google Android developers, mostly to credit them for the substantial original work that they've basically contributed here. But also to show the contrast that there's quite a few folks in the community that's also helping, albeit in smaller ways. So thanks to everybody on the list and hopefully we'll see this list continue to grow. So folks wanted me to have some sort of color coded status to show kind of where we were. And so this is kind of a rough swing at that. And so we have green for things that are upstream, yellow for things that are in staging, orange for things that people are working on and red for things that are out of tree. Now again, this is not a very scientific but it does show that a good chunk of the work is actually up in the mainline kernel in one way or another. And it also shows that there's a fair amount of work still to be done, specifically in networking graphics and power management. So for a little more data-driven analysis, I went through basically the entire out-of-tree Android 3.4 branch and that basically contains as of last week about 568 patches. And it's 127,000 lines of code or it's lines of change, which is substantial. So I decided to break that down kind of by directory and the really surprising part, I know this is a little bit of a eye chart here, is that the huge 60% chunk there is the Broadcom driver. So that's just the Wi-Fi driver. And it's a vendor drop and so it's not really, it's not something that's being pushed upstream anyway. And the second biggest item there is the F2 file system. And that again isn't super interesting for upstreaming, but I mean again here it's something like 70% combined of the lines of change. So to get maybe a little more useful view of things, I dropped the Broadcom driver and the F2 file system out and that brings it down to 35,000 lines of change, which is definitely more manageable. And we can see kind of the big chunks here are things like the Android Gadget driver is the biggest followed by Netfilter. We've got a fairly big chunk of change in arch arm, which is mostly the FIQ debugger. We've got now after that there's the Ion driver. Header changes is next and that's sort of the, that comes from all the changes basically. And then driver's base is also sync and sort of on and on and on it gets smaller. The other thing to note is that there's a huge long tail. Basically anything that was smaller than 1%, I just grouped into other and there's just tons of them. So it sort of keeps going there. So this chart unfortunately isn't as cool as the amount of work it took to make it. But I'm still pretty proud of it. Basically I went through all of the patches in the 3, 4 tree that are not upstream. And charted them by their author date time and grouped them by their patch subject topic. And so this is kind of neat because it lets you see kind of where the Android developers have spent their time over the last few years. There's one change that's kind of a stray change way out here that's from like 2005 before Android even started that they basically had pulled in in the arm directory. But you kind of notice through the middle here it's a little more sparse. And part of the reason for that is they do periodic rebases so every once in a while they'll squish a number of commits into a single change. But also the fact that a lot of the older Android functionality is already upstream. And so there's less stuff that's out of tree through these years. I think the most interesting part is over here on the right where there's a lot of changes. You can basically see where they've been spending the majority of their time this last year. And so we've got a fair amount of changes in the arm tree, a lot of changes in the CPU frequency governor, especially recently. There's kind of a whole bunch of changes in the net but that's the Broadcom driver again. They just do periodic syncs with the OEM driver. There's a fair amount of changes in the sync driver. We've got power, which is the Android battery driver. GPU, which is Ion. And then net filter down there is another item. So I don't know, I thought this way. If I could do this again, I'd try to make some way to have the actual commits show the size of the patch set but I think that would just be unreadable. So I guess the point of that slide is that despite all the work that's gone into getting stuff upstream, Google isn't standing still. They're still generating changes. Again, we have a fair amount of recent development around the Broadcom driver, Ion, sync, the interactive CPU frequency governor and then less in size but more recent changes have been also in the battery driver and net filter. And to some extent explains in that earlier status chart why we had a lot of red on the bug droid around graphics, power and networking. And so it basically shows those are the areas that we need to focus on to catch up. So finally, why bother with all this? What does all the sub streaming actually do to help anyone? So in the media or kind of web comments or whatever you'll see a lot of people kind of grumbling oh, Google forked the Linux kernel and it's terrible gasp. And yeah, they forked the code but that's the way Linux kernel development works. It's one of those things that people create these little areas and they make experiments and they see how it goes and then the maintainers can see how those experiments go and if the code continues to be worked on then clearly it's something that people care about and it's more likely to go upstream eventually. We've had a lot of cases of this of major work being done out of tree. For example, there's precedence of like the preemptor tree RT patch set which has some really core kernel maintainers focused on it and it's been out of tree for the last seven years. A lot of it has gone upstream but they keep on writing more and so it just kind of always stays out of tree. There's also other cases like the four gig, four gig split kernel. This is something that REL shipped in the REL three days. And basically it was useful for 32-bit PAE kernels when you had a ton of memory but when X8664 became more common it became unnecessary and so it was something that just was done for a while and then stayed out of tree. In fact most file systems are done out of tree for quite a while. It's now kind of funny because EXT3 is considered kind of this old stodgy stable thing but for a while it was too risky to merge upstream and so Red Hat had to carry it out of tree for quite a while. And so it's one of those things where the GPL it allows for these forks to eventually be remended and or at least if it was a bad decision we can learn from kind of the mistakes that were made. So code forks, they aren't really an issue but community forks can be more problematic. I guess in the upstream Linux community there's already kind of an existing issue of there's somewhat of a rift between X86 and ARM developers and mostly that's due to kind of the historical aspect of a lot of the core maintainers being somewhat X86 focused. And this is something Linares actually been working pretty well to address. But when Android came on the scene and became such a huge force for ARM vendors basically this caused a lot of ARM developers to kind of go off and focus on getting the Android kernel working. So much so that basically if you know there's a lot of Android platforms that basically are shipping or non-android platforms that are shipping with the Android kernel. We've got the WebOS and Firefox OS and apparently even the Ubuntu phone is going to ship with the Android kernel. And this is all because the developers who are focusing on this makes it even harder basically for developers who are writing hardware enablement for their ARM boards to be able to push stuff upstream because they're going to end up with funny little Android dependencies. And so basically by pushing the code upstream it's important because it increases collaboration. Basically folks are able to interact with the upstream maintainers. They're able to basically work with, ARM folks are able to collaborate more with X86. They're also ARM folks are able to collaborate with other ARM folks where otherwise they might be just keeping the stuff in private trees somewhere. It allows them to be able to find common solutions and basically work together on things. Kind of the takeaway that I'd like for this is that if you're working on the kernel it doesn't really matter whether or not you're doing Android development or big iron development. I think you really should consider yourself a member of the upstream kernel community. Maybe not a majorly influential member but a member nonetheless. The Linux kernel is kind of a shared commons that we are all invested in and I think we should kind of consider it as our community. And yeah, much like family, there's gonna be folks who will frustrate you and who will kind of drive you crazy at times and make life very difficult. But at the same time the upstream kernel community can also be a really great source for help and feedback that would otherwise cost a lot of money to be able to get. So basically join in, there's plenty of Android upstreaming still left to do but even outside of Android, if you're working on the kernel, please submit patches up to the mailing list and kind of read LKML and join in in the discussion and also check out Lenaro's website here for other cool stuff that's going on. All right, thank you.