 There's stuff going into the kernel every release in terms of drivers and SOC support. I'm not going to talk much about that. But I'll also cover a little bit of development stats. These are the kernel versions that we've had over the last year, starting with 5.14 in August of last year. You can see the cadence, the duration of each release is about nine or 10 weeks. You can practically set your watch by it. The kernel is always released on a Sunday evening and we're three weeks in to the 5.19 release candidate cycle. So we expect 5.19 at the end of July. So this is boring. This is exactly what Lena said in Esquino yesterday that we like the Linux process to be boring. Just a couple of things in each of these kernel versions, I'll just point out. Something called MemFD secret was a new system call that was added. I'll talk about that. Some new tracers have to do with real-time analysis. A fair amount of Qualcomm and MediaTek driver code went into this release, clocks, pin controls and sound drivers. And there's a new driver called the simple DRM driver for direct rendering interface for simple frame buffer devices, which is good for embedded. So the MemFD secret system call, this just creates a region of memory that even the kernel can't access very easily. The purpose of it is to create a region of memory that you can't unintentionally overwrite and that's even difficult for this kernel to access. This is a type of thing that you'd use to store cryptographic keys or something. Of course, it's possible for the kernel to access everything with enough effort, but this kind of hides the keys a little bit better. In terms of 5.15, really, really big news is we finally got sleeping spin locks and I've got a whole slide about this I'm going to talk about later for the real-time preemption patch. The core scheduler has support for asymmetric systems. So it turns out that there's chips that have, particularly there's some ARM chips that have cores that can run 64-bit code and some cores that can run 32-bit code on the same core. So the scheduler has to be kind of aware of that so it knows which process is to run on which cores. And then if you happen to be doing something like a NAS server, you may be interested to know that there's a kernel version of the SMB server that's actually in kernel. It's not a full replacement for Samba, but pretty interesting. Something called printkindexing. So there's a bunch of tools that are used for parsing logs from the Linux kernel. And what happens is as developers change the printk messages, those parsing tools break. They usually write regular expressions to say, oh, if I see this event from the kernel message stream, that's something I need to report. But if the kernel messages change, then the log parsing breaks. But unknown, right? It's like silent. So now you can generate a list of all of the possible kernel messages for each release and see if the messages change so you can update your tools. So that's pretty good. There's a new data access monitor tool and I'll talk about that. And the kernel, OK, so I know in embedded, a lot of us are using old kernels that were not at top of tree, the kernel we got from our vendor or whatever. When you switch to 5.15 or above, you're going to notice that your builds break probably because the kernel now uses the dash W error flag, which means that any kernel warnings are turned into an error that will break the build. And that's intentional to kind of encourage people, nudge them to fix all those warning messages so that we can get clean builds. The data access monitoring system I thought was kind of interesting. So this provides tools to record data access and show visualizations. And there's a couple of different visualizations that are available. So it's a diagnostics tool, like a lot of the other tracers we've got in the system. But is it actually more? And in 5.16, we see that, yes, it's a sneaky way to get some new functionality in. So let me talk about these other ones. So the enhanced read-only file system continued to get some new features, multi-device support. So there's a lot of work been going on with something called IOUring, which is the new asynchronous API for file IO. This is for high-performance file IO. If you're not using this in your product, this is something maybe you want to look at. The thing in this release was they added support for SE Linux and SMAC security controls. And then data monitoring, operation schemes was added. So now the data monitoring, you've got a diagnostics tool that can monitor usage of memory, but it now has a way to actually do something with that in terms of hinting the kernel to reclaim memory. So this could help with low-monitor situations. So if you have a product that is really low on memory, this is something you might want to look at in terms of handling low-memory conditions. In 5.17, a couple of changes. The kernel can now decompress kernel modules within itself. It doesn't have to rely on user space to do that. And it turns out that that's important when you're doing a type of secure boot where you're looking at the hashes and having the kernel analyze the hashes of the kernel modules. So the load pin security module, for example, uses that. The real-time Linux analysis tools were added. And they used those tracers that we mentioned before. And then there were some changes to the flags field. So if you're using Fuse, I'm not sure if that's the right way to pronounce that, but a user space file system, there were some changes that might cause incompatibility. So I need to watch for that. And then 5.18, this is interesting. The support for some older architectures, ARMv4 and ARMv5 MMULA systems was removed. So it's not always true that the kernel just keeps getting bigger and bigger. Sometimes stuff gets removed. So there is still support for MMULA systems, but about ARMv7. There's lots of RISC stuff was added in this release. The tracing system now supports user events. So if you need to trace user space applications, you can use kernel tracing for that. And then the kernel compiles against the C11 language standard. I'm going to talk about that later. And then the final thing is that in 5.19, we're still in the merge window. This is not a released kernel yet, but we've got a lot more SoC support and device drivers. The big news in this one I think from an embedded standpoint is that the ARM multi-platform work is almost complete. So if you're not familiar, ARM, all the ARM kernels used to build separately and were different from each other. And now you can build a single kernel that you can feed at a device tree. It's called the generic kernel image that you can feed at a device tree and it can adapt and boot. Just like on Intel, you never had to build like an Intel for each individual board. Same thing on ARM. You no longer have to build a kernel for each individual board. You can build a generic kernel. So that has been work going on for about eight years. And it's really nice to see all that consolidated. There's initial support for a new architecture by Loomson, which is a Chinese chip manufacturer. So new architectures are going in. And then there's a new hardware timestamp engine, which I'll talk about that again later also. So in terms of developer stats, in case you've not noticed, if you've been looking at my URLs, I really owe LWN.net a debt of gratitude. And I see Jake is here and I'll buy him a drink or something after the session. But we get all this data from LWN.net. If you're not a subscriber, please go subscribe. They do excellent work. And if you want to stay on top of stuff in the Linux community, go do this. Pay them a little bit of money. So there's about 15,000 changes in this release, which is pretty average. You can see the areas where some of this is happening, device tree updates, folio patches, refactoring of the block and FS. Kristoff Hellwig is always in the top five because he's always refactoring something. If you see by lines of code, then you see that the GPU, the AMD driver, almost 400,000 lines of code in one release. So that's pretty heroic there. But you see, you can get into the top stats by removing stuff, too. And then in terms of commit log entries, the stats for this are just a little bit, numbers are different because I use my own tools for this, but it's about 13,000 to 16,000 somewhere in their range of changes every release. So that's every 10 weeks. That is pretty amazing when you think about it. Sometimes you've got to take a step back. Oh, the other thing, I want to back up and just point out one thing real quick. So Linus said that he thought there were about 1,000 developers on Linux every release. It's actually closer to 2,000. It's pretty consistently around 2,000 developers. And if you have never contributed to the Linux kernel, note that about 2 to 300 every release is a new developer. So if you've never contributed, don't be shy about saying, well, the Linux kernel, it's done by all these experts. No, there's new developers every single release. So go ahead and jump in and contribute your stuff. Just a real quick, I didn't finish the slide on this one, but about 40% of all bug reports are now coming from automated test robots. So the reported bylines, we're getting a lot of reports from test robots. OK, so technology areas. These are the areas I'm going to go over real fast. And again, I apologize. These slides will be made available after the talk. And actually, I have a detailed version of these slides that is the full hour and a half that I gave just a couple of weeks ago for Japan that I'll also include a reference to. But so in the area of audio. So PipeWire is kind of the new shizzle. You ought to be looking at that for your new products. Pulse Audio is not taking it lying down, though. They've just had a new release, Pulse Audio 16. I told you I was going to go fast. In the core kernel, we have these things I already talked about, MMFD secret, printkindexing. So the kernel compiles against C11. So what is that about? It turns out that they found out that because of our adherence to the C11 spec, where variables need to be declared inside certain kernel macros, according to that spec, it was leaking information that you could detect with speculative execution attacks. And so they made the decision to go ahead and move to a newer version of the spec that allows you to declare variables right in the for loop. And so this unscopes the variable better. And so that was actually the justification for this move. And I think it's a good move. I mean, we can come out of the 90s or the 80s, actually. It was the C89 was the last spec we were adherent to. The other big news in the core of the kernel is this Rust. This was talked a lot about by Linus and Dirk. So there's this effort going on to support Rust code in the kernel. The patches are still considered experimental. And the support for these patches was recently moved to a recent version of Rust. There's some grousing about how fast that changes. It's about 35,000 lines of code. It's out of tree. And there's ongoing debate. There's a debate raging this week on one of the kernel mailing lists about whether we should let it in or what are the ramifications. But most kernel developers are on a wait and see. And we're willing to see what happens with it. In terms of file systems and IO, so like I said, IOU ring, which is the new asynchronous IO that continues to mature. And there's a bunch of new stuff happening, zero copy networking. There's some patches flying around for that. In terms of enhanced read-only file system and F2FS, flash friendly file system, those continue to mature with better compression support, better X attribute. And then there's those changes to Fuse that you need to be watching. In terms of graphics, a couple of small things and a couple of big things. So simple DRM, driver was merged. The frame preferred FVdev subsystem was kind of orphaned, but it got a new maintainer. So that's really good for us in the embedded space. We have these old devices with 2D things. And it wasn't being supported really well by DRM. There was some conflict. The DRM maintainers didn't like some of the things that the new maintainer did, resurrecting some code that they thought should die. But that friction was all eased over. And hopefully, I'm hoping the new maintainer does a good job. So one of the things though, Mali GPUs. Mali is a GPU architecture that ARM has been supporting. That now has a fully conformant open GL driver, fully open source. And we have a lot of color board, I think, for that. Also really big news, NVIDIA has announced that they are going to open source a lot of their code. So they've actually released some of the code already. It's not upstream, but this represents a big shift for NVIDIA. And I happen to have an NVIDIA card on my desktop, and it's a pain. It's a hassle to deal with the proprietary drivers. So sometime in the future, maybe I'll be able to use an open source driver that's all upstream. That would be awesome. Not sure the effect of this on the existing open source project, which is the Nuvo project, but this is definitely something to watch and kind of big news. Networking, there's just kind of a steady stream of weird networking stuff going on. Not a lot of new protocols. And some of the esoteric protocols that get in are not really relevant to embedded. But there is this new thing with there's an internal function that allows driver authors to instrument, to provide the reason that they're dropping a packet. And so that will help with some diagnostics. OK, so real time. So this is really, really big news. Well, the small news is there's some new tracers, real time Linux analysis tools. The big news is RT preempt, right? So sleeping spin locks went in in 5.15. And then patches have been going in continuously through 5.19. So the sleeping spin locks, if you're unfamiliar with that, or this is what allows a process to switch while a lock is being held, and this is what allows you to get low latencies guarantees through the kernel. And this is a core, core feature of preempt RT. Possibly, I think this is probably the core feature of preempt RT patch set for Linux real time. So you have to turn on a config option to get this. But this has been extensively tested and verified. It's been 17 years in the making. It's taken 17 years to get this feature into the kernel. So a natural question is what's left. So if you go look at the actual patch set, it's not big. It's about 1,300 lines of code in only about 50 patches. And that may sound like a lot, but compared to where this was before, it's unbelievable. And they've upstreamed about 1,700 lines of code just since February. So I mean, if you were to extrapolate that, the whole thing would be upstream by August. But of course, there'll be details. Like the print K code, they just got up. Apparently had to be taken back out because there were some bugs or something. So if you want to hear more about that, there is a Birds of a Feather session with Steve Rosedet on Friday, so make sure you go to that. In terms of security, we basically have a whole bunch of kernel hardening. There's something called control flow integrity to make sure that you're jumping to the place you thought you were jumping to. And then mem copy bounds checking. And there's always a stream of specter mitigations. Specter as an exploit is the gift that keeps on giving. And so one thing I thought was interesting is there's actually been a removal of some specter mitigation code that was found to not be very effective and kind of bad for performance. I'm running behind, but I'm going to try to squeeze these out quick. So Zero Day is doing an awesome job of helping reporting bugs at the time of patch submission. There's Kernel CI and SISBOT that are finding a bunch of bugs, CKI and LKFT. I just want to point out, if you haven't seen this other one, Compass CI, I think that's a new automated testing framework to watch. It's by Huawei. It's by the same developer who did Zero Day. So he went from Intel over to Huawei. So this is one, I think, to pay attention to, because if you look at it theoretically, he really knows what he's doing in terms of testing. There's a whole bunch of suites that have new releases and new tests, new coverage. Don't need to talk about those. GCC versions, GCC 12.1 was released in May, so point release for them. And the new focus seems to be one of the things that comes out is a lot of really good stuff to handle uninitialized variables. There's some runtime stuff and also some static analysis stuff that's in this version. So that's pretty good. So more catching of bugs. LLVM also had a point release in May. People are using this for whole distributions, not just for the kernel. And if you're interested in that, using it either with Yachto or some other way, there's a good talk by Bernard Rosencrantz around that. OK. Tracing. Steve's not here, so I can just skip over this quick. Oh, the only thing here, the hardware timestamp engine. This is a pretty neat system for actually causing the hardware to grab a timestamp based on a hardware event. So without any software interrupt required, that is really good for low overhead tracing of some events. OK, so I'm sorry how fast that was. I know it was a quick run-through. The slides will be online. The main reason for that type of quick run-through is just so you have in your mind a search term that you can go look up and go find it on LWM.net and read the full article. So in terms of industry news, I want to talk about just a couple of things. So we have the Open Source Security Foundation, and in particularly the Alpha Omega Project and a couple of miscellaneous things. So if you haven't heard of the Open Source Security Foundation, you've got to be living under a rock or you missed the keynotes yesterday. But this is a big, comprehensive project to analyze security throughout the Open Source ecosystem. And they're going to do a whole bunch of things, vulnerability disclosures, security tooling, best practices. They are well-funded. They have over $70 million of funding. And they're well-connected. So the people from the Open Source Security Foundation represented the Open Source ecosystem at the White House, at a special cybersecurity summit, and also at some recent congressional hearings. So this is good. We have a well-funded organization that's looking at security. Here's some of the activities they're going to be doing. They're going to be doing scorecards and reviews of existing code bases, a metrics dashboard, package feeds, and package analysis. And then doing a best practices badging program so that you can see if you're dependent on certain components, you can see if they have a certain level of security badging. They have some standards for vulnerability schema and something called Salsa, the supply chain levels for software artifacts. So they're doing the SBOT software build materials work. And then also guides and training. So they'll be a vulnerability guide and help if you want to work on security. One of the big projects they recently announced was called the Alpha Omega Project. And this is an effort to systematically search for vulnerabilities in open source code. The alpha part of it is they're going to identify the top 100, I think, or maybe 1,000, but the top critical code that is in widest use. And then actually go fund developers, go in and find problems and fix problems. But the problem with open source, or I don't know if it's a problem, is the benefit is there's this huge long tail. There's over 10,000 packages or libraries that need to be analyzed, and a lot of them have very few maintainers. There's like one maintainer or two maintainers. And the project doesn't have enough resources, even as well funded it is, to go after all of those. So they're going to do things like provide training and provide automated tools to help people analyze their own. And hopefully as a community, we can raise the level of security of all of the open source that we use. So that, I think, is a really good project. In terms of miscellaneous, I have to note, this one was interesting, that Intel has acquired Linutronix. So Linutronix is the company that Thomas Gleichsner. I don't know if you know who that is, but he's the guy behind preempt RT. So Intel has acquired them. They're now, I guess, Intel employees. But I think that represents a good sign that Intel is going to keep funding Thomas to work on security stuff. And then I've been hearing a lot more seeing stuff about Oniro. I had never heard of this before. Maybe it's been around, and I'm just out of the loop. But it's a project by the Eclipse Foundation. It's an IoT operating system. So it sounds like a user space stack, or if you don't have user space, just a stack. It's just driven IoT OS. I don't know that much about it. I saw a bunch of, I'm on the program committee for ELC Europe, and there were a bunch of talks submitted on this. So it has blueprints to build end user-ready products, according to information I've seen. And it's Yachto-based, builds the entire system at one time. So this is something maybe to watch and look into. I'll see more at ELC Europe. I got to talk some about, I guess I'm doing OK on time. So I may have rushed too fast. I got to talk a little bit about Mars Ingenuity Helicopter. So for years, I've been asking, have we got Linux on Mars yet? And it turns out we finally did. So we got the Ingenuity Helicopter. It's a little helicopter. And in there is some custom off-the-shelf hardware that is running Linux on Mars. It's just amazing. So it landed last February, well, not this last one, but February 2021, and performed its first five flights. And it was supposed to be a technology demonstration. But it did really well. And it is still flying. It has performed 29 flights so far. Just a couple of recent news. They had an opportunity to go back and look at the back shell as the rover and the helicopter as they come down the descent. There's a part of it called the back shell. It has the parachutes that flies off and crashes separately from the rover itself. And so they were close by, so they went over and took some pictures. The first time ever, they've been able to get detailed pictures of what some of the crash site looks like. It's not just a super cool picture to see a picture from this on another planet. But it actually has scientific value because they can analyze the debris field and see what the effect was of the landing, how fast it landed, and how the debris spread. So they can use that information for future. In terms of updates, it's the little helicopter that could. It's, bless its little heart, it's trying to survive the Martian winter. We are entering Martian winter. And it's getting, there's colder weather and less sunlight. And there's dust on the solar panels. And so it has been having problems maintaining enough charge. So the helicopter right now, it's supposed to charge enough to heat the CPU core, keep its core cold overnight, or warm overnight. Now it doesn't have enough power to do that. So it has to let the actual, the computer shut off. So Linux shuts off overnight. And the core temperature gets down to about negative 80. These are off-the-shelf parts. In fact, some of the parts for the helicopter, they ordered from Sparkfun. And they're not rated for negative 80 Celsius. So anyway, so because of this shutdown, they had problems with the clock synchronization. And they missed one of their communication windows. They were very worried. Also, I think because of the cold temperatures, the inclinometer is now broken. But NASA figured out how to way to compensate for that by using other sensors. And of course, NASA being NASA, they had a patch ready. I don't think they have yet to submit it. But they're going to do a live update. They're going to update in the field on Mars. That's pretty cool. And just so NASA knew this was going to happen, or they figured it would happen. And so they had prepared and tested a patch months ahead of time. So the patch that they're sending is actually several months old. But that just shows you how cool NASA guys are. Guys gals. So it's keeping up with the rover as best it can. And it actually goes in scouts locations for the rover. Totally cool. I think it's one of the most interesting uses of embedded Linux in the solar system. If you want to get more technical details, Tim Canham is the JPL lead on this stuff. And he gave a great keynote last year's embedded Linux conference. And so you can get a lot of the technical details about what version of Linux and diagrams of their CPU utilization while they're in flight and stuff like that. So scorecards. So I have to move through my scorecard stuff. I took too long on the helical. I knew I would. So are we done yet? Well, it depends. Where are we? So I'm going to, oh, let me back up. I'm going to talk about three different scorecards, technology, development, and markets. So the original focus areas that we decided in 2003, how are we doing on those? Well, if you look at the contributions to the Linux kernel, we don't see very many on system size or boot time or really power management. Real time, I just said, was done. So we're done, right? Well, so congratulations, everyone. We did it. It's like, well, OK, let's back up a step there. So system size. Yes, we're done. We don't see a lot of system size patches going into the kernel. That's because we kind of gave up. The lower limit is about 16 meg. Well, I won't say we gave up. Moore's Law made it so that we could care less. Well, and that sounds bad if you say could care less. But alas, Linux is never going to run on 1 cent processors. And believe me, OK, there are such things as 1 cent processors, and they're not capable of running Linux. So those 10 trillion IoT nodes that we were hoping to run on, we're not going to be running on those. It'll be something smaller like our FreeRTOS or Zephyr. But we're in a pretty good space. So boot time, we're done. We don't see any boot time things. Well, cold boot time reduction. We don't see any new patches for that. Most products these days are just kind of cheating, right? They're going into suspend-resume, or they're going into a low power state. Like my phone, you think your phone is off, it's not off. You think your TV is off, it's not off. OK, so cold boot, again, we kind of gave up. But we've worked around the problems. Power management, is that done? This one is harder to call. So the stuff we need to do power management, ooh, three minutes. Ooh, I'm off on my timing. Oh, I better. OK, I'm going to have to really hustle. So we have the stuff we need to upstream, but it requires SoC and board support. So you really rely on your architecture or your SoC vendor to write the drivers that utilize those features upstream. In terms of real time, done? Yes, preempt RT code is almost all upstream. But again, it's going to require ongoing maintenance. And security, of course, is an in-progress thing. So the technology scorecard looks much more like this. I'm going to go through the development scorecard. These are the areas that make it good or easy to develop for Linux. And I'm going to say that actually we're doing really well. We have tons of build systems and distributions. We have good training and consulting. Lots of companies are available to help you with your projects. Our tool chains are really in good shape. And SoC vendors take care of writing a lot of the code for the tool chains. Debugging capabilities are good. I would say test systems are still in progress. I think there's still a lot of work, a lot of gaps in testing that we can fill with automated testing, particularly in the area of hardware testing. And then hardware support I think also is still in progress. So we've come a really long ways. SoC vendors now very customarily provide support right in drivers. But the issue is that oftentimes it's not upstream. And so it's very, and that requires that as product makers, if you're making an embedded Linux product, you're having to do a lot of extra work and carry a lot of extra technical debt because of that. But things are improving. And so I'm going to give our overall development scorecard mostly good in a couple areas in progress. And then the market scorecard. And I know this is not a comprehensive list of the markets, but just really quickly. Drones, we're doing great. Walmart is about to roll out drone delivery in 34 states. So get ready to be attacked from the skies. Amazon has also announced trials in California. Most of those drones are running Linux. Robots, robot operating system is really kind of taking over the world. It's already a majority in academia. And it's also projected to be 55% in commercial robots. Cars, AGL just released a new release with instrument cluster and infotainment. And it's used in a lot of telematics and self-driving. And then in space systems, surprisingly, I thought this would be the last frontier. But Starlink is running Linux, Space Rockets, the Mars helicopter, there's a lot of CubeSats running Linux. I think as we get more commodity hardware up into space, which will happen as launch costs come down, we'll see Linux even more in space. Routers, anyone can make a router. You're running Linux. Mobile phones, we have 70% market share with Android. And for consumer electronics, we're just killing it. I think it's about 100% of TVs, about 100% of DVRs, most cameras, a lot of audio. So we are doing really well in these verticals that I chose for this session. So conclusions. Overalls, we're doing really well. Linux is widely deployed and embedded. I would say we're at world domination. We're not going to be in the low-end devices. That's kind of disappointing. But you know, one-meg RAM system that's running on just harvested energy, that's going to be tough. And I don't think we'll get there. We're not going to see Linux on a serial box. That's a callback for any of you who are here 10 years ago. Core kernel systems, though, are in place to handle everything we need to do and embedded. New hardware keeps being made, so of course we've got to keep writing new drivers. Core code can always be improved. But the bottom line is we're not done yet, but that's OK. We've got job security for the foreseeable future. Hopefully you can use Linux in whatever embedded product that you want to build. And I want to encourage you to go forth and use embedded Linux, and I hope that, especially here at the conference, you find out the bits of information that will help you make a better product. And help you also become an active and productive member of our ecosystem and our community. And so with that, I will say thank you for your time. And I don't know, do we have time for questions, or are we? One question, I guess. I know it's like after you've drunk from a fire hose, it's like, anybody? OK. Oh. Oh, I will get the slides up. There's supposed to be, well, keynotes, they give you an exception, and I'm calling this a keynote for that reason. But I'll get these up by the end of the day. So they should be on the site. The question was, when will the slides be up? And they'll be up soon. OK, I think that's it. Thanks.