 So here we have a very recent kernel on the Android device. Hi, so what are you doing around here? So this demo is about updating kernels on production devices. Probably if you have an Android phone, you've seen that kernel is at least two or three years old. And for some reason vendors just don't update their kernel. They stay on 4.4.13 or whatever forever after making the initial cut. We're trying to fix that situation by showing it can be done. This is a phone that is off the shelf and usually comes with 4.4.23 or so. We updated it to 212 and it only took a day and a night. And so this is what you talk about here. If you're using an Android phone, you're probably using an old kernel. It doesn't have to be that way. So how's it going to be better to have a new kernel? There's many reasons why kernels are updated. We don't release kernels for nothing. They have bug fixes, sometimes they have security fixes. New features? New features, not within the LTS line. In this demo we've only stayed within the LTS line. So going from 4.4.23 to 4.4.12 or so. So that wouldn't give you any new features but it would give you a lot of bug fixes including a couple of security fixes. And of course it would be interesting to go all the way to 4.16. That's the newest, right? Yes. So what would you need to do to get to 4.16? Essentially just port all the patches for the device to the newer kernel versions. Tools like Git Merge and Git Rebase make it quite easy. But the problem with some phones is that they have millions of patches needed just to make them work and that stuff adds up. And when you haven't been updating your kernel for four years and then all of a sudden you need to merge four years of external work into your set of millions of patches. That's obviously going to take a lot of time. That's a pixel right here, right? That's a pixel to XR. One of the few phones that actually have done the right thing. It was originally released with an early 4.4. And now with the P preview that's running on it. It has already been updated to 4.4.78 or something. But definitely something newer, not quite the latest. But 4.4.88, which isn't all that bad. So this is an example of a phone that's trying to do the right thing. So they did it without your help? That was already shipped like this? Yes. But this is 4.4. And the latest is 4.16. What's the main difference between 4.4 and 4.16? There's a lot of differences. There are new features and performance improvements. Of course also some bug fixes, but a lot of bug fixes also made it into the later 4.4 kernels. Is it possible that there's lots of new ARM optimizations? There's definitely some. In the newer kernels? Yes. So it would be great if every new phone had the newest kernel. When is that going to happen? It's probably still going to take a while before the vendors wake up to this situation. It's actually starting to get better. Some chipset makers are starting to put patches to make a specific device work into the upstream kernel. And then they get all the upgrades for free, because the community does the rebases for them. Once the device is supported in an upstream kernel, it will be supported for years. Then obviously you can just pick an upstream kernel and maybe apply a small set of patches that you still need. And that's much less work than rebasing millions of patches for this phone. I was just doing a video with Amit. What's he doing with the newest kernels on the Android? He's doing a lot of kernel updating and kernel testing. And essentially making sure those newer kernels work on the PSPs we care about. And making sure that the kernels actually work and don't bring about any recreations. Does this have anything to do with the trouble? Not directly, but in some ways it's all connected. Travel is partially about unifying devices on a newer kernel version. It's mostly about keeping the drivers separate so everything can use common user space. But maybe at some point we get to a point where devices can even share kernel modules. Alright, the work that you're doing is being noticed by the industry. Are they going to be like, okay, let's ship new kernels? I hope so. So far a couple of vendors have shown some interest. We haven't gotten any new members from it so far and we haven't received any. Oh great, we must do this immediately. But I think they're starting to look into it. So for example, when you updated that one, the XZ1, right? You did this update, you said you call on? About a day and a night. So what does that work consist of? Essentially checking out the kernel source that I've released that works with the phone and adding the regular kernel repository as a Git remote. And then using Git merge and Git rebase to make sure that the changes made in the upstream kernel and the changes made for the phone specifically get merged into one tree. So they get any test, everything? Check what's buggy or what works and fix it? Yeah, of course. And with that day and a half, we had a kernel that boots and seems to work. But obviously that didn't have any additional testing beyond, okay, I can still make a phone call, I can still play a video. And then you can run tests. It's like CTLs and VTS or LKFT to make sure you don't bring about any regressions. So is that the Qualcomm and MediaTek's job to look into this? Or is it the device makers at the end? All of them. Essentially the way things are done most of the time is Linus makes a release and then the chipset maker takes one release for folks that are at support for their particular chipset. And then the device maker takes what the chipset maker has been building and possibly adds a couple of patches on top of that. So in the end probably the chipset maker should pull those patches and then the device maker still needs to pull them from the chipset maker. But obviously a device maker could also bypass the chipset maker and just do it by themselves. Do you have a solution for all this? One solution would be to not start for forking old kernel versions in the first place and just work on top of mainline kernels, upstream patches to make a device work immediately. And then pretty much the entire middlemen are eliminated and you can just go for 4.17 or so as soon as it appears. Obviously you want to do some verification but... But how can you make a device that doesn't need a fork? Look at your computer. You can update to a new kernel without having to have any patches specifically for that computer. Essentially on some of the boards that we are supporting it's already the same situation. Highkey can boot mainline kernel. The Macchiato bin can boot a mainline kernel. Probably some of the other boards that are about to come out can boot a mainline kernel. And they can because they upstream their stuff. Yes. Otherwise you can boot a mainline. Right. Either upstream or make sure someone else upstreams it for you which is what some of those guys are also doing. They get some outsourcing companies or what's it called? Some guys like free electrons or something like that? Yes, for example. They use their linear membership or they just put out some interesting stuff that is not totally obfuscated and find someone in the community to volunteer to take it and put it in your kernels. That stuff is essentially happening as soon as you have an interesting device. People will care about it and want to see it supported. So I'd be very surprised if you put a reasonably interesting device out there and couldn't find anyone willing to help out for free. And then there's somebody at Linaro. I heard that's working hard on backporting all the security stuff to all kernels. Yes. Is the team to do that? Yes. At the moment that is needed because like I said, there's millions of patches just needed to make this phone work and updating that from 4.4 anything to 4.16 or so would just take a lot of work and that's why it makes sense to for now keep maintaining the old dot release as well. And you backport it to all the old kernels? All the old LTS get the backports? For important stuff like that, yes. Few important things only, okay? Yes. But much easier to just get new kernels. I mean it would be better. Right. Ideally every phone should come out with the current kernel and keep updated. There's probably some certification issues but those can be resolved. Shouldn't Google just say, okay from now on we want everybody to please consider getting new kernels? That would be really nice and I think they're starting to do that but it's going to take a while. So.