 Good morning, everyone. That is quite a crowd. Wow. I'm happy to see all of you here. I would have loved to see all of you here two years ago when we all wanted to be here. But hey, it's good to be back. My name is Dirk Holndl. I am the Chief Open Source Officer at the Kodano Foundation. And you are? And I'm Linus and some of you may have seen the whole Dirk and Linus show before because I don't really do public speaking anymore. We do this, Dirk asks questions and I answer because this way I avoid having to prepare a speech or try to figure out what the audience is interested in. I hate to break it to you, but you're publicly speaking right now. So we are back in Europe, which is super excited. We both, of course, were born in Europe and we got to come back. And traveling in Europe is a lot of fun. So standing in Amsterdam in the security line for two and a half hours with my 2,000 best friends, Sieg Kudrowl, that was fantastic. I really enjoyed the welcome back home. So yeah, that's awesome. But we, of course, are not here to talk about our travel tales, but we are here to talk about fun stuff in open source. For example, we're going to talk about Linux 6.0, major release, new major version number. So everything changes, big announcement, right? Right, no. Intentionally, 6.0 is going to be a fairly boring release. And I've had this policy for the longest time now that version numbers have no meaning. There's no big things coming because the major number changes. The numbers are just numbers and the major number changes when the minor numbers get big enough that it starts to become more of a bother to keep things straight in my head. Are you running out of hands and toes? Yeah, that's what I say. That 20 is about the range that I can count to. It's really more of, we used to go up to 50s. And when you do those kinds of numbers, trying to remember what happened, like making a difference between 45 and 46, you really can't remember anymore. So for me, the version numbers are a mnemonic that has no meaning. And to make that clear, 6.0 is hopefully the standard boring kind of release where we just do incremental improvements. And then next version, 6.1, we may have some more excitement. I was going to say, I mean, currently it's super low key, super low drama. RC5 was, as you just said the word, boring. Boring is good. Don't get me wrong. When you do development, you do not want a lot of excitement, not at the same time at least. Yeah, especially you want the right kind of excitement. The excitement of, oh, everything is working. No one is complaining. All the patches are applying. That's the kind of boring that we like. Well, it might be boring right now because so many other core developers are traveling this week. So it may be very quiet partly because of that. But I think overall what I call the boring ratio is going up. We seem to have more boring release cycles. We seem to have fewer outliers where things go crazy, where you start yelling, where we have breakage during the release cycle. I'm not allowed to yell anymore. No, it's true. And at the same time, so I used to say, and I used to say this like almost two decades ago, that kernel development isn't where the excitement is supposed to be. That if you want excitement and doing new and world-changing things, you should be doing all these things in user space like artificial intelligence or whatever. And all those are all about stability and making sure things work. And I used to say that. And the funny thing is we have more developers than ever in the kernel. And despite the fact that I've been doing this for over 30 years, we still end up making major changes to core stuff that has been around forever. And I find that great. I mean, it means that the kernel really is alive and it's not like just coasting and we're just in maintenance mode. But it's kind of strange at the same time. Literally yesterday, one of the talks at the Plummer's conference was what? PrintK, which is getting a major overhaul. And PrintK is literally the first thing that ever made it into the kernel because the first thing you want to do is show things on the screen. So this is not some like new feature that people are improving. This goes back all the way 30 years ago. And we still have people who not only are rewriting it and making major changes, but we had a room full of people where people were like, they weren't screaming at each other because it was actually fairly, it was fairly civilized. But people were excited enough to like voices were raised because people were talking over each other. And that's kind of funny to me that a project that's been around for their years, we still end up having these kinds of things that are so fundamental that people still get excited about from a technical standpoint. But I think it's also noteworthy that the kernel community is a community of people who can get excited about a print function. We have to let that settle in. That is the other side of the picture. The kernel developers are maybe kind of strange. Maybe. But there was some noteworthy use. And actually in your interview with Steven yesterday, you talked about it already. You have switched to an ARM64 laptop. And a lot of people have talked about this. And so it seems like a fair question. What's the experience like, drivers, performance, user experience? How does that work? So I have to say I'm very happy so far. It's very smooth. And I mean, part of it is obviously that the ARM support has been there in the kernel for two decades, whatever, a long time. And I've been literally, whenever I talk to the ARM people, I've been saying, hey, we need to have an ARM laptop that you can do development on, because most of the infrastructure has been on cell phones, obviously, but also a lot of the effort has been on the server side. And very few ARM machines have been developer-facing. And this is something I feel very strongly about, that you do want to have actually end-user, like kernel developers who are end-users, not just support for somebody else using the machine. And we still have some issues. I've used the laptop for two months. And during those two months, we've had, it wasn't me who found them, but we had two different entirely unrelated memory ordering issues that came up. And I think it's partly because people are starting to use ARM, not just as a supports thing for somebody else, but you kernel developers starting to use it themselves and finding problems that way. And I think that's very important. I mean, this is something that you have been very consistent about. I, 20 years ago, in 2003, when I worked at Intel, I brought you into a meeting with Andy Grove, Craig Barrett, Paula Delini, Pat Gelsinger, back then was the CTO. And one of the things that happened in the meeting was that Andy Grove and you got into a shouting match because Andy said the future would be itanium and everyone would be running itanium and you said that's not going to happen because? Because there was no development platform. And for ARM, when I talk to the ARM people, they always said, yes, we want to have a laptop. We just don't have the infrastructure yet. But back then, when you talked to Intel about itanium, they were claiming that they will take over the server world and that the rest of the world will follow eventually. And that's completely wrong. And history proved me right as so often happens. It's applied, yes. But if I look at this, if we look at the transitions that have happened in the Linux space, we had 386 to x86-64, is this really that next transition that we see in other platform that becomes a complete mainstream part of what we all use? Do you think this is the beginning of a shift? I don't know. I mean, I think there's a general path that small stuff grows up. And that happened on the PC side and it is clearly what's happening on the ARM side. I'm using an ARM laptop right now and I'm perfectly happy with it. Does that mean that I think ARM will take over? No. I think the two can coexist and I'm not saying that one has to take over. I do think ARM has clearly a... is in a great position thanks to the licensing terms and how that has allowed ARM to really grow from the embedded world into laptops and servers and has a lot of companies putting effort on that. And that is very important. But at the same time, there's no question that legacy software and inertia on the user side is also hugely important. And it's something that is very easy to maybe sometimes forget when you work with people, like tech people in particular who often are hopefully fairly fleshed with money compared to a lot of the community who like to try out new things and who like new machines and who like to play with things. So when I get a new ARM laptop and it has a few problems, to me that's exciting. To normal people, that's not exciting. So I don't think PCs are going away any time soon and I think it's going to be a big battle on the ARM side to really take over in that sense. And that wasn't what I was implying. I was much more talking about... for so long, it always seemed X86 is the one main platform. And I think that's changing. But one of the things that is so interesting about ARM is that diversity that you just mentioned, that you have on the one-cent, the pure ARM course, you have the things that Qualcomm and Samsung do for the mobile phones and then you have Ampere, the AWS folks with Graviton and the teams at Microsoft and at Google and at VW and so many companies that are all saying, oh, we're building our own chips that are custom for our world. And this really shows the strength of Linux and of an open source infrastructure that these people can build all these different platforms because they know it's very easy for them to then have a kernel that supports it and a software stack that is optimized for it. To be fair, we've had huge problems with this situation, too. Having this wild west of crazy hardware companies that do custom chips and SOCs and servers was a huge issue when the initial ARM work was being done to the point where at one point I told the ARM people that you need to get your act together because merging your code is too painful because when you have a thousand different platforms and they all act very differently, it's just a huge pain for bring up. It's a huge pain for all the infrastructure work and I have to say that that has changed enormously in the last decade or so where we really have had enough of a standardization that our machine these days is no longer a random machine where a vendor did strange things, but you tend to have enough standards around the CPU core that you can also get some strength from Unity, which was always one of the points of PCs where you had a fairly unified base with a unified BIOS that was not great, don't get me wrong, but it meant that you had a certain infrastructure that you really could rely on and that used to be lacking in the ARM space. I think the transition to ARM64 really was the moment when they did a much better job of getting back together. Enforcing some common infrastructure, yes. So one of the things that I... We have talked about this before, the hardware bugs that have been the plague of our industry for the last six, seven years, and I stumbled across this news article just a few days ago that VMware, my former employer, claims that they see a 70% performance drop if in a Linux guest, the red bleed mitigation is enabled. Well, so first off, yes, it made the news last week for some strange reason, but it wasn't actually news. We made the changes back in June, public, I think, and had benchmarks about the great performance impact it had. It makes your CPU work so much harder to mitigate the problems. The good news is it only happens with certain CPUs, of course. So the 70% performance impact is really a worst case for a very specific benchmark. So in reality, very few people are affected, if at all, then nearly to that expense, but the whole hardware security information leak situation has been very painful for us. It has been painful for us in terms of performance impact, especially on some paths that we have been very proud of doing very well, but it has also been very painful from a development standpoint because all these hardware bugs have not just come with the requirement of writing workarounds for security problems in the hardware, but have come with long periods of, we can't talk about this. And this is so anti the whole open source ethos and how we fundamentally work that it's probably been more painful for us in the open source community than it would have been and has been for some of the traditional companies where you already kept all your source code secret and you already had an environment where people were working inside one company and were supposed to keep others out. But in the open source world and in the Linux kernel world, we've been off now for several years in the situation where we can bring in the experts, but we can't bring in the thousands of eyeballs that we're so used to. And it's been horrendous to that point where some people are really burnt out about the whole situation. But the good news is it is getting better. There are fewer of the security bugs in hardware because hardware people are better at thinking about them and figuring them out and also we've gotten somewhat more used to the whole pain of the process. But a big part of all this has always been speculation and speculation is the way hardware can try and guess what it will have to execute next and therefore get ahead of the game and work through its pipelines better. And one of the things that you pointed out is the M2 processor in your Apple laptop speculates a lot more than many other processors. But is there this logic that says if you don't want these security bugs, you need to go back to in-order, which gives you horrible performance? No, no, no. Speculation is fundamental in any modern CPU. If you want any kind of performance at all, you need to speculate. There's no alternative. The hardware has literally decades of doing this where people only worried about the semantic meanings of the speculation being correct in the sense that the end result was correct and did not worry about all the data side leaks that happened due to how cache accesses worked and various buffers inside the CPU. So most of these problems are not for modern architectures, micro-architectures. Most of these problems are literally for things that were designed long before people even started thinking about, well, I say that some people were thinking about this problem decades ago, but before it made enough of a splash that it really forced companies and CPU architectures to think about it very actively. Well, obviously we can talk about security in the 2020s without mentioning Rust, because Rust will sell absolutely all security problems. Good segue. I hope you all know that. Rust is the answer. So Rust, you said yesterday, might be in the next kernel, but it's not in this one? Well, you're actually asking this about five hours too early, because right after this keynote address, I'm going over back to the plumbers and we are having the yearly, well, hopefully again, yearly maintainer summit. And one of the topics will be Rust. And I'm personally hoping that it's one of the things where we will start merging the initial Rust infrastructure in 6.1. But with that said, I was hoping we'd do it already in 6.0 or 5.19. So at this point, it is I'm going to bring it up and if there's no huge outcry against it, I think it will see a 6.1 with some very minimal initial Rust infrastructure. So what you're saying is we all should be back here tomorrow morning at nine and do this again and I'll ask you the question again. I'm sure I'll announce it if that is going to, and you can read it about it online. But I think it's super interesting that there is so much debate that is not so much about the language spec, but about the infrastructure, the compilers, the tools, that most of the concerns seem to be much more about, oh, right now you need to use Clang because the GCC Rust front end isn't in shape to actually compile things the way we want them. So there is obviously, this is still very actively developed space. That's true. At the same time, I think a lot of it is also. People who have, if you've been a kernel developer for a long time, you've been programming in C, maybe assembly the whole time and you feel very comfortable in that mindset. And then you see this whole notion of new drivers coming in in a language that you really don't know, and that puts some people off. I wouldn't say scares, but it makes them very nervous. And I think that's been one of the fundamental driving things for them being searching for problems. And to me personally, I see this as, hey, something exciting and new and interesting, and I look forward to learning the Rust syntax. I've already spent some time on trying to understand it, but honestly it's one of those things that you don't really learn a language until you've used it for many years. So I think it's going to be a fun experience, and I also understand that not everybody else is necessarily of that mindset. So we'll see how it goes. I've always been fairly positive to Rust and I've been one of the proponents of merging it. At the same time, I'm also very pragmatic and who knows in a few years after we merge it, and it turns out to be an abject failure. Maybe this will be one of those, because we look back at this, slightly embarrassed about our naivety in thinking that adding a new language is that easy. And when I say as easy, I will point out it's been going on for a long time. This does not happen overnight. But I guess what you're announcing is that likely the next kernel might not be as boring as this one. Yeah, I do hope. We have a few things really lined up. We already have pending for 6.1. We also have the multi-generational LRU for the core VM. Rust is a potential. We'll see when the print case stuff happens. Right now it's more of a series of patches that people are commenting on. I don't think that will happen for 6.1, but it's being slowly put in the queue. So we have a lot of fairly fundamental things coming up, and some of them will probably not be boring. So as you mentioned, you're going straight from here to the kernel summit, the kernel maintainer summit. And that is often a meeting where the problems that have piled on interpersonal problems or technical problems that have piled on over the last few months, or in this case, three years, get brought out. So do you expect the highlights to be what is Linus going to complain about at the kernel summit? So I actually don't have any complaints. Oh, my God. Usually my complaints are not so much about the technical stuff. I don't worry. To me, I don't lose sleep over technical issues. Technical issues you can always solve. I mean, mistakes happen, bugs happen. Some of them are really embarrassing. I mean, especially the security ones, some of them make you go, how the heck did this ever get through, right? But at the same time, technical issues we can fix by just revert them. If worse comes to worst, we find a problem, we say, okay, we should never have done that. We revert. So the things that stress me out tend to be about process and people. And maybe that's because I'm not a people person, but that's the kind of things where you can't just say, okay, let's just fix this person, right? Whether it's yourself or somebody else. And we've had, over the years, we've had a lot of process issues and we've had personality clashes and we've had issues like that. And that's the thing that tends to worry me. But right now, I don't see that. Maybe I will be told otherwise in an hour or two. So for me, the big question today is probably going to be the rust issue. And then every year, I ask the maintainers who show up whether they have any issues with me. And so far, people just mumble and look away. But maybe this is the year when somebody actually raises their hand. So we'll see. Maybe we'll have the rebellion and somebody else will take over. I have a bunch more questions, but we are out of time. So thanks very much for showing up in the morning. Thank you, Linus, for the conversation. And you have a good rest of your day.