 Let's get right into this. Just the nature of this talk, this is a very, very quick overview of embedded topics. It's the springboard for further research, so I don't go into anything in super depth during this talk. Basically, I'm just going to glance across the surface of embedded Linux and hopefully give you some pointers to some stuff that's been going on. I know your brain is already full from this conference, and my slides, actually, they're not available after the event. They're available right now. If you go to elinux.org, and on the top of the page is a link to the presentations page, and my PDF is already uploaded, so if you want to follow along, you may have to, because I'm going to go faster than you can probably read. It's going to be a little bit like this. So this is the general outline. I'm going to go through kernel versions, talk about technology areas, yadda, yadda. So this is kernel releases that we've had in the last little bit. Just the cadence here is really consistent. My prediction for Linux 4.8 was off by seven days. I actually predicted a longer cycle, 77 days, which is dumb of me. So the way I've addressed that is to just widen my prediction. It's going to be either December 4th or 11th. And the interesting thing there is Greg Crow Hartman has already announced what the next long-term stable kernel is going to be, which is pretty interesting. That's a little bit different. Some of the things just really quickly that have been in recent kernel versions over this last year, light NVM, I'm going to talk about that a little bit more. Perf can build and load EBPF files. That's the Berkeley packet filter files. Anyway, I'll talk about that later, too. ARM 64K can have 16K pages. We've got a Broadcom GPU for the Raspberry Pi, some work there. Dev Freak cooling, various pulse width modulation drivers. Big news in 4.5 was ARM multi-platform. This is a major patch. And it really hits an important milestone. We can now build a single ARM kernel image that works on multiple platforms. And I'm going to actually come back. My opinion, this is the opposite of embedded. I'm going to talk about why that's, it's a good thing, but it has some drawbacks. But interesting, not much else specific to embedded. In the 4.6 system, we saw some GBIO subsystem rework, some new tools for looking at device trees and helping you debug issues there, and some improved page poisoning. I told you I was going to go fast. Linux 4.7, we've got the Schedule Frequency Governor, kind of a little bit different in the way that we schedule tasks based on a little bit more knowledge. That's kind of a part of an overall effort to improve, we're kind of inching our way towards power aware scheduling in the scheduler, and this is kind of part of that. The VFS directory, VFS later can now iterate through directors and parallel. It sounds like an enterprise-y type of thing, but it's, in general, it's good for performance for file systems all over the place. And again, BPF programs are showing up again, ability to attach those to trace points so that when you hit a trace point, you can actually run arbitrary code that you've compiled from user space, so that actually adds some flexibility there. And then histogram triggers for ftrace. And then the Android sync file feature, and I'm going to talk about that one a little bit more in detail. So if you're running on Android, there's this feature that I think has been in staging for a while, but basically it allows you to speed up how the buffers, well, it speeds up the throughput of the system by allowing the buffers to be used in a more parallel fashion. And this avoids a lot of waiting, you can look it up online about that. So Linux kernel 4.8, and we're getting kind of closer to, there's a new kernel documentation system, pseudo random number generator, a new timer will implementation. This is a big surprise. I didn't think anybody was paying attention to the old timer will, and I can't remember who did these patches. I think Thomas was involved, right? But it's amazing, you can take this really old data structure that people have been using for years, and you can remove some kind of bad aspects of it. There used to be this cascade operation, and by looking at kind of the modern way that timers are used in the system, they're able to improve that. And as with almost everything in my talk, I've got a link to a LWN.net article, I shamelessly just point their direction for almost everything because they do these really good write-ups of the material. So 4.9, I'm going to make a prediction, we're already in the merge window, and the LWN.net articles were already written, so I can do this. I think a big issue in this one is virtually map kernel stacks. I did a lot of work for Sony on kernel stacks, trying to make them smaller, and this actually is a really neat feature. If you kind of know about how the stacks have been historically, they've kind of had this kind of kluge architecture to them, where a portion, you know, the thread info structure is at the bottom of the stack or the top of the stack or wherever it was, and then you had the task struct was somewhere else. Well, there's been kind of a big cleanup made, and not only does it provide this feature that you can now catch stack overflows inside the kernel, which is a huge deal. That provides a level of robustness we haven't had in kernel stacks ever, but it cleaned up the kernel code, and in the process they made the kernel, the process creation actually faster. And unfortunately that's on x86 for now, but I think we'll see very quickly other platforms using that new system. So my observations, if you look at the last year, a lot of those things that I listed off, they're kind of neat general features, but we're not seeing a lot of embedded specific, what I'd call general features, non-drivers. Of course we're seeing a ton of driver work, which is really good, but we haven't seen a lot of stuff on boot up time. I don't know the last time I saw a feature specific to boot up time going to the kernel. Same thing with system size, and I'm going to talk about those in more detail. Embedded file systems, there's a little bit of work going on. Embedded security, and by embedded file systems I'm talking about the old standbys, UBIFS and J2FS, and even Flash2FS, right, have not seen a whole lot of work in the last little bit. So there is work progressing on power management, real time stuff, the timer wheel, and I'll talk more about that as well, solid state short GPU drivers, and there's lots of processor support. But this is a list, I give this talk very, very often, and I kind of have a list of things to keep your eye on that I think are kind of interesting. And this is the list that I've had for the last little while. And the status is actually, it's kind of mixed. So the kernel tonification is really kind of stalled, okay? There's not a lot of new patches going in there. The RT preempt is doing pretty well. I talked to Thomas, and he said in the hallway that there were only about 10K lines left, but I looked at the patch. It seems like it's a bigger patch than that. I don't know, am I missing something? They just had the real time summit. I didn't go to the meetings, but I know something's off here. Persistent memory, there's a lot of interesting stuff going on with persistent memory, so I'd call that in progress. And then me and a bunch of other people have been beaten our head against the wall trying to get this SOC mainlining stuff going. It's a tough problem. There's a lot of the SOCs in the mobile space have a lot of lines. You heard Greg say it's in the millions. And so there's a lot of really thorny issues trying to attack a patch mountain that big, so progress is being made, but it's painfully slow. So let me go through each of those technology areas really quick and just talk about kind of where we're at. And I'm going to breeze through this super, super fast. I'm already kind of off my schedule. But in boot up time, you know, I have XIP, asynchronous probing was discussed at the last kernel summit, but didn't really go anywhere. Reduction in probe deferral, there were some on-demand probing patches that were nacked. There were no talks at either ELC or ELC Europe this year on boot time, which is the first time, this is the first year that we haven't had something like that. There's some really good previous talks. I recommend these two to look at. I'll go on record as saying that Andrew Murray's talk just totally changed my perception of the problem space and is a really excellent talk to go look at. And then, of course, there's a ton of problems in user space. Hello, Android. Device trees, device trees are coming along. Device trees are maturing. We've got the overlay stuff going in. The device tree validation, there was some early work on that, but it got stalled. And there's been talk about a new specification. Hopefully at Plumbers, they'll kind of get some more juice in their tank and go off and attack some of these. In terms of graphics, you have a new graphics API that is an alternative directory 3D or OpenGL. For those of you kind of paying attention, there was a QT license change that has a lot of people in the industry worried. In terms of our GPU support, there's been some good progress on a couple of the SoC GPUs, particularly Fridrino and Etnavive have really made some good progress with the free drivers there. File systems, the only thing I could find on UBIFS was the great guys at Free Electrons are doing some stuff with MTD and multi-level cell flash. But we're seeing increasingly in the embedded market the rise of Blackbox block-based storage. And so a lot of the features that we're seeing come out in the kernel are really addressed that. One of the things is this new LightVM, which is a framework for holding SSD parameters so you can move the flash translation layer from the Blackbox hardware up into the software where you have visibility on it. And I know I'm going super fast. Networking, lots of stuff is happening in networking. Linux Foundation, real-time project. I think we all know that that's great. Zenami has a new model in their 3Auto that was just released this year. And there have been some good security. Casey's is still doing a lot of great stuff with security. Their new hardening project. Again, the slides are online. I know I'm blowing past them super fast. System size. Linux Tiny is dead. It's really sad. I don't know what the last set of patches we got in was the single-user patches to get rid of users and groups, saved about 25K. Other stuff has kind of been in the wings but has not made it in. The XIP patches from Intel actually were a nice thing. And Nicholas Petrie, bless his heart. That guy is total stud. Getting a lot of really interesting work that we have to do with GC sections. And Vitaly Wool's also working on size things. Testing, I don't know if you noticed. So I'm the chair of the program committee and I'm interested in testing. I don't know if you noticed how many testing slots there were at this conference this year but it's kind of self-serving, I know, but what can I do? So there's KSelfTest, which is making a lot of really great progress. There's gonna be a lot of sessions at Plumber's coming up about continuing to enhance this to make more automated tests, more continuous integration capabilities for the kernel. Fuego is my own project. I'll talk more about that later. Kernel CI, I would argue, is the most successful public distributed build and test system for Linux in the world as if there's some other scope that would be appropriate. But no, it really is. I mean, these guys, it's got 2 million builds and boots. They actually have a track record of bugs fixed. They have distributed in multiple labs. It's really awesome. If you don't know much about it, go check out the presentations. In terms of tool chain, Chem is continuing to do interesting stuff with Clang so that we have alternatives in the embedded space. Tracing, lots of stuff is happening. Oh, that, not mainline yet. I think it is mainline, right? Didn't the histograms, did that make it in? Well, I don't know. Okay, we don't have time. We're just gonna move along. Okay, so the next LTS kernel version is 4.9 is pre-announced and there was a worry that vendors might rush to push stuff in but that doesn't look like it happened. So that's actually a good pattern. So there's some non-Linux stuff going on, of course. Google announced Magenta and so I have yet to kind of see what that's about. It's based on little kernel, it's got a BSD license. Zephyr, of course, you've heard a lot of stuff at this conference about Zephyr. One of the things you may not realize, I think it's a really big deal that Lenaro has announced support for Zephyr. And so I think that really gets Zephyr to kind of a critical mass that's really valuable. In the CE workgroup, we're doing these projects, shared embedded distribution, device mainlining, LTSI and the Fuego project and we continue to manage the E-Linux wiki and so I'm gonna actually just blow right past all those. Okay, this is where I get most of my information from. If you are not subscribed to LWN.net, I say this every single conference, go get a paid subscription because they're an invaluable resource to the community and to Lenux and thus to the entire world and so you really should pay them some money. And then I get stuff off of Colonel Newby's but this is what I wanted to get to and that's why I'm going so fast is I wanna kind of take a step back. So that's the nuts and bolts, the really quick stuff, what's going on in the Colonel and the software. But I wanna talk about some of the trends that I've noticed for the last couple of years and especially talk about generalization versus specialization and fragmentation. So first, before I go into a little bit of a rant, I wanna say, we're doing great, okay? You know, I made a joke a couple of years ago, it's like it's kind of embarrassing when your goal was to conquer the world and then you accomplish it, it's like, well, what next? Lenux is in over 1.5 billion devices and that's a very conservative estimate. I wouldn't be surprised if it was closer to two or 2.5 billion devices, it's hard to count these things. But I worry, I worry that Lenux is not gonna be on all those leaf nodes. It's gonna be in the gateway, everybody knows all your IoT gateways are gonna be Lenux. But most disturbingly, I'm really disappointed that Lenux is not gonna be running on my cereal box. I was really hoping for this. Of course, you're not gonna be, this is a goofy picture because no one's gonna put Lenux on there and show the boot screen, right? What you're gonna see Lenux running on the cereal box, if you were taking your kids down the aisle at the grocery store and there was Minecraft running on the cereal box, do you think you would be able to not purchase that box? I don't think so. So, what does it take to get there? I kinda wanna see what our status is. My contention is that it takes a $1.10 cost of goods to achieve that, to get Lenux on a cereal box. And so, that's my totally made up numbers for what you have to do. You have to do a 40 cent CPU that includes the flash and the memory, a 50 cent display, figure it out, hardware guys, we just need some breakthroughs in material sciences. Input devices, 15 cent battery, and we can do it. We could have Minecraft on our cereal boxes. Never mind the disposal aspects of that, that's another problem. But where are we? Are we getting there? I worry that Lenux is not gonna be able to run in a 40 cent CPU before lots of other things that could potentially run Minecraft or Pong or whatever it is. So, what I've seen is a decrease in the number of talks in a few key topic areas. And you can kind of sense from my earlier portions of my talk what those are. There's a slowdown in kernel submissions and kind of a parent slowdown in progress. And we see a lot of RTOS fragmentation. These are the traditional embedded Lenux topics. When we first started this conference, these are what we talked about. We're still talking about most of them today. Some problems you have with you always. And one of the reasons is that it has to do with just the nature of the business. We can get fast boot times with Lenux, but you don't see the patches submitted upstream. And part of the problem is that those boot time reductions that people make, they're specializations of Lenux. And so they're not accepted upstream. System size is the exact same thing. There are patches to make the Lenux kernel smaller, but they don't get submitted and they don't get accepted. And it's because of this issue of generalization versus specialization. So upstream wants generalization. They want the most general solution possible because that's how you get the network effect of people collaborating on the software. Okay, and reducing boot time really is not about adding something more. For performance, you might add something like a caching layer. But for boot time, you really just want to yank as much stuff out as possible. And again, based on Andrew Murray's presentation, it's really you have subtractive versus additive engineering. Most of the engineering you do is additive, but when you're doing some of these things in the embedded space, you're doing subtractive engineering and you're making tucks look really weird. So this is Lenux. Lenux is this really complicated Lego set that has tons and tons of features, okay? But if you try to rip it apart and make a little toy car out of it, you end up with Franken-Lenux, okay? It turns out that you can't pull the pieces apart cleanly, okay? And so it gets really, really difficult. RTOS, oh wait, let's see. That's not the slide I was expecting, okay. Oh, upstream there's resistance to specialization. Read the, okay. So RTOSes give you that smaller Lego kit. It's got smaller pieces. You can make the nice, tiny car. And of course, the in-house custom OS. If you want to go and spend all the money and time, you can make the perfect little plastic car that fits your needs exactly, but it takes you too much time and set up an effort, right? It's much easier. You can, if you go to the store and buy the Lego kit, you can get your thing built in 20 minutes, whereas you have to set up an injection molding facility to do the car. So there are different approaches. So this is something that happened to me just this year. There was a group in Sony. Sony is a far-flung organization. Lots of groups doing different things, and Sony corporate doesn't mandate what OS different projects use. There's a project that came to me and said, well, we just had to use NutX. We had to end up doing something non-linux in the embedded space, because it was just too hard. And so we worked on all this stuff. And what that mean, well, so, and we have a fragmentation in the RTOS space, and I really worry with all this fragmentation that we're not gonna be sharing as much. The whole thing about open source is that we can share our solutions with each other. And I'm really disappointed, because I think it's gonna be really, really difficult for Sony to share, for example, our elf-loading customizations that gave us like a 40% boot time improvement because of the way the linking happened. Anyway. So, in my, I gave a keynote in 2014, and I talked about the paradox of embedded Linux, and it really comes down to this. In open source, we want to generalize the software because that's how we get a bigger number of participants engaged in that software. That's how we get those network effects. But in embedded, we really wanna deeply specialize. We wanna squeeze every ounce of performance out of there. We don't wanna have to buy that extra RAM chip, okay? We really want to make it as efficient in cost performance as power light as possible. But when you do that specialization, you lose all that community effect. And so the issue, I think, is a really big kind of, take a step back issue. How do you balance that tension? I don't have the answer. But I do see a problem. We have way too many embedded distributions. We need to find ways to, even if we're not building the same things, we need to find ways to share our management of our packages. We need to share our test capabilities with each other. That's something that's shocking to me. We know how to test software, but every company actually kind of builds up their own test infrastructure themselves. Now, there are test projects, and there's lots of fragmentation in that space. Wego being another example. But we need more sharing. We need to be able to share the security fixes, send more stuff upstream. And we need to share not just the test packages, but the test experience. We need to make it so that if someone in the world figures out what LTP is supposed to produce as a result on a megalbone, that no one else in the world has to go do that again, I can guarantee you that that's been done thousands of times, and that's just a waste of effort. It goes completely against the open source philosophy. So that's what I'm worried about. I do have some recommendations. I'm interested in testing because I think there's a capability there to make up for some of the loss of community effect when you specialize software. I think we really need to work hard at reducing fragmentation. Okay, this next bullet point is very controversial. Do a license to your code as GPL and BSD. I don't even have time to explain why that's a good idea. It's a great one though, so just do it. And then look at other projects though. Don't keep your blinders on in your own projects. Look around at other projects. Find the commonalities. Find some ways to share ideas at a minimum and code if you can. Keep working on upstreaming that and making the world a better place. Let's keep working together to make Linux the best OS possible. It's a lot of fun. And it's something that we can enjoy as we move forward. And that's it. So.