 Good morning, everyone. Wow, it's bright here. After sitting here in the nice dark side, this is quite a change. So my name is Dirk Wontl. I'm VMware's chief open source officer. Been around open source for nearly 30 years, actually over 30 years now that I think about it. And you are... And I'm Linus and many of you may have seen the so-called Dirk and Linus show. I don't do speeches anymore. I hate doing that. So we tend to have this kind of fireside chat thing going where Dirk asks questions that he thinks you are interested in and we'll see how that goes. Great throwing me out of the bus. Thank you there. And we have no artificial fire behind us this time, which I'm very thankful for. So my first question is always the same, only the number changes. So 5.4, RC5, there's still a lot of churn. It's pretty big. Is this worrying? Are we on the right track? Is this a normal release? So 5.4 was actually odd because it was smaller than usual for the first few RCs. And that's usually a good sign. It means that we just didn't have a lot of churn and we didn't have a lot of changes that made people find bugs. And then something happened last week. And last week we're back to normal and slightly above normal. And it doesn't worry me because maybe there was a lull for some reason for a couple of weeks and now we're catching up. But this might be, if this trend continues, it might be one of those releases where we take an extra week to make sure everything is calm and we found all the problems. But there's nothing special and odd going on per se. It might just be a timing issue. But talking about odd things, so in the last kernel release we had this really odd bug where a change to a file system causes boot hangs on some platforms. And we see a lot more of these complex deep system interactions that are incredibly hard to debug. And is Linux as a system becoming so complex that the testability is starting to be worrisome? I don't feel that. I feel that the fact that 5.3 had this very odd bug at the end that we just punted on and we solved it in the 5.4 release cycle. I think that's more of a sign that we have fewer of the obvious bugs. So the bugs that then get noticed are just more subtle because we have better testing infrastructure. We have a lot better tools for both dynamic and static checking of the sources. And our process has been pretty good at figuring out the obvious bugs long before they even hit my kernel. And that obviously then means that the bugs we do hit are often these subtle interactions between two subsystems, often things that only happen on certain platforms and certain hardware platforms and also certain usages. I think that's a good sign in general. It just means that we do the basics fairly well and then occasionally you hit something. It is a complex system. That can be wrong. So there will be these interaction bugs that happen. RT real-time extensions are finally almost merged and we are talking about the end of the 10-year journey of getting this into the ground. It's been more than 10 years. I was being generous. But with that we see increased focus on using Linux in safety-critical systems, whether it's in cars or in avionics or whatnot. Isn't this increased complexity and the subtlety of these bugs something that is worrying in the context of RT, in the context of safety-critical systems, where the more complex and the more non-deterministic a system is, the harder it becomes to actually trust it? Linux has always been fairly... I mean, any operating system by definition does complicated things. And I don't think we've been getting worse. I actually think one of the things that makes me very happy is how people are still looking in poor detail at very core kernel code and doing changes and doing cleanup of code that has been around for decades right now. And when you say safety-critical systems, one of the things that people don't always realize is that these safety-critical systems tend to do one thing and one thing only, the safety-critical thing. And that means that the kind of bugs that we see on desktop systems in random distributions are things that you never would normally even hit in the embedded world in those safety-critical systems because they do one thing that has been tested very, very, very well. And the kind of bugs we tend to find are the random, crazy user bugs where somebody does something that nobody even thought would be remotely saying. And so the bugs we see are not bugs that I think would hit safety-critical systems. So put words in your mouth. You're saying that the size of the community and the long experience with our development process is actually helping us despite this complexity to create better software. Also, if you look at what happens now, the kernel I make is not the kernel that even the distributions use. So the development kernel where we found a bug just before the 5.3 release is only the first step. Then we have Dreg, who's here somewhere, who is the second step, who does the stable kernels. The third step is the distro kernels that take the stable kernels, usually work their own testing on top of that, often have a few patches that they do for their customers, and then way beyond that, you end up having all the people who actually do long-term stable stuff, like in automotive, in industrial automation, things like that. So not only are we getting better on the development kernel side, we now, over the last many, many years, we've had this process in place where you have multiple levels of filtering. By the time it hits a train, not hopefully hitting a train, but by the time the system is installed in these industrial applications, they've gone through years of testing. So this really is a case where the size of this community and the long experience that we've had in how we actually develop and then test what we do and maintain it over time is to the advantage even of these typically rather extreme use cases, like real-time or safety career computing. Yes, and one of the problems we see is that in some of the embedded world, people are still working with kernels that are so old that the base kernel actually is from before we got all the processing in place. Before we made... Yes, exactly. So some of the really long-term systems are using kernels that are very much inferior, and we've actually improved on our stability and on our process and testing a lot in the last decade. So I think one other thing that is interesting for people to understand is what do you spend your time on as the kernel maintainer? Because I don't think a lot of people have insight into your workflow. What do you do? I read email. I read email, I write email. I do no coding at all anymore. Most of the code I write, I actually write inside my mail reader. So somebody sends me a patch or more commonly they send me a pull request or there's a discussion about the next pull request and there's something I react to and say, no, this is fine, but... and I send out pseudo code or I'm so used to editing patches that I sometimes edit patches and send out the patch without having ever compiled it, ever tested it because I literally wrote it in the mail reader and saying, I think this is how it should be done. But this is what I do. I don't have a programmer anymore. I read a lot more email than I write because what my job really is... In the end my job is to say no. Somebody has to be able to say no to people and because other developers know that if they do something bad they will say no. They hopefully in turn are more careful. But in order to be able to say no, I have to know the background because otherwise my no is... I can't do my job. So I spend all my time basically reading email about what people are working on. I think you just de-glorified your job by quite a bit. On that very same vein. So what do you think the kernel developers should spend most of their time on? At the other end of this pipeline? So there's so many different levels of kernel developers. I see one of my primary goals to be very responsive to the sub-mentainers who send me patches or send me their floor requests. I want to feel like I can say yes or no within a day or two. During the merge window the day or two may stretch into a week. But I want to be there all the time and that's as a maintainer. I think that's one of the main things you want to do. You want to be responsive so that the people who are sending you code, either as patches or as floor requests, feel like their work is maybe not appreciated because sometimes it's not, let's be honest. But at least they get the feedback quickly so that they don't sit there wondering what's up. So that's one part from the managerial side is to be there all the time and be responsive. And it really is not very glorious. It's not a, it is an interesting job but you do end up spending most of your time reading email. On the developer side what I hope people are doing is trying to make not just good code but these days we've been very good about having explanations for the code. So commit messages to me are almost as important as the code change itself. Sometimes the code change is so obvious that no message is really required but that is very, very rare. And so one of the things I hope developers are thinking about the people who are actually writing code is not just the code itself but explaining why the code does something and why some change was needed because that then in turn helps the managerial side of the equation where if you can explain your code to me I will trust the code. And it also helps if in three years somebody else is trying to fix a bug and it looks at the change and says what the heck is that? If you have a good commit message that explains why you did this it's usually much easier to actually understand what's going on. A lot of open source in general is about communication. I mean and part of it is the commit message is part of it is just the email going back and forth communicating what you're trying to do or communicating why something doesn't work for you is really important. I think that's a very important message that communication is key to all of this. Honest which gear completely for a moment and talk a little bit about history. So your rather early memoir was titled Just for Fun and if I look back at the early 90s when we started computers had four megabytes of RAM megabytes with an M to those young people in the room. A big hard drive was 100 megabytes and libraries were simple. The tools that you had were fairly simple. I remember when I first started Linux kernel development I printed out the complete source code of the Linux kernel tree. I don't recommend doing this anymore. Do you think for people starting today it is as much fun it's as easy to have fun and feel that you understand the space that you in that it was 25 28 years ago. I don't know. I think in many respects excuse me kernel development has gone much easier because yes it's bigger and more complicated but on the other hand we have much better tools and we have much better documentation. We have a lot more community where people there are lots of people who feel that it's part of their job sometimes the primary part of their job to help new people come into the community. So that all helps. There's also a lot less of the problems that stop you from even jumping into it. I mean early on you might not have the right hardware just because the kernel would only work on a very small subset of the hardware out there and that just isn't a problem anymore. At the same time it's clearly the case that the kernel has gotten huge and hugely more complicated and the kind of immediate low hanging fruit where new people can feel like they have something to work on might be not as easy to find. I think the learning curve, the amount of things that you need to understand before you can do something and that's not just about the kernel, that's about Kubernetes, about Docker, about anything in this space it just feels to me that there is so much that you need to grasp of the environment before you can get really something done. Back then you send a patch to Linus, he yelled at you and life was good. Today it's very different. On the other hand it's still the case that almost nobody got started doing Linux by getting into some core kernel functionality. Almost all kernel developers started doing a small driver, doing something pretty small at the edges of the system and the edges have grown. There's a lot more of those drivers that might have small annoyances there where even if you don't know the system very well in general you can find something very particular that you want to get into. I used to say that nobody should get into kernel programming because there's all these other open source projects that are easier to get into but I think there's still a lot of people who are interested in interacting with hardware and are interested in the kind of problems that a kernel faces and we still have a huge community and we still have a lot of new faces coming every release. Things seem to be working very well but there are places to get into development. I wasn't implying that things aren't working well. It feels to me and that was my next question. If you look at this transition from 1991 tiny community very, very basic in its functionality let's face it. When I started there was no login prompt you just got a root shell. Moving forward I did at some point to you this switch for something that felt like just for fun a hobby project to something that was big business that was mission critical software. Was there an inflection point where you said the nature of this project has changed? No, the nature of the problems have changed and there's always been things that are not fun. When you hit your head against the wall for a week chasing a bug that part is not fun and never has been and that was always there. The kinds of things that are not fun these days tend to be different. These days sometimes you have the CPU bugs. Those are not fun with a capital N. And there's also what is maybe slightly not fun is we have to have a lot more rules in place. It was much more freewheeling back in the days and there was more banter and you could try things out without really worrying about breaking systems as much as you can these days. There is a lot of seriousness and some stuff that isn't fun but the reason I'm still doing it is the fun thing. I may spend most of my time reading email but part of the reason I do that is I'd be really bored if I wasn't. I thought you might pick up on the change from early on you had the production kernel and the development kernel, the odd and even 2.6 we changed to a completely different development model, the one that we still have today with the 10-12 week release cycle. I would have assumed that you would pick that as kind of this transition from we have this experimenting branch that is just for fun and then we get the release kernel but you didn't proceed that way? It turned out that the reason we switched to the modern model where we don't have an experimental kernel and a stable kernel was that old model was so much not fun because the experimental kernel was too experimental. There was too much craziness going on that most regular users would not want to touch it and that meant that from a development standpoint the experimental kernels were not that much fun to develop either because if you, well this may be me I feel like when I write code or when I'm a part of a project I want that project to be meaningful to other people if there are no users what I do is not fun and that's the situation we were in back in the 2.5 days where users didn't dare use our experimental kernels because they were too experimental and changing to a model where we have to be more careful and we only integrate things when they are ready to be integrated actually made a lot of that go away and it made our release cycle these days is so much more predictable and there's much less stress over that side that I actually think it's much more fun. So in Jim's introduction he pointed out 28 years of Linux 15 years of Git and he always forgets mentioning 8 years of subsurface the third most important open source project the third most important open source project I agree, I wholeheartedly agree but isn't it time for a new project? No, no, no, no I am so dumb I'm very happy with how Git ended up but every time Git is mentioned I want to make it very, very obvious that I maintained Git for 6 months and no more and it's been 15 years and the real credit goes to UnioHumano and a lot of other Git developers I'll take credit for the design and the thing that makes me happy about Git is not that it's taken over the world but that we all have self-doubt we all think are we actually any good and one of the self-doubts I had with Linux was it was just a re-implementation of Unix can I do something that isn't just a better version of something else and Git proved that yes, I can Yes, you can and that to me having two projects that made a big splash means that I'm not... three projects means that I'm not like this one hit wonder To those in the audience who don't get the joke so the third project, Sub-Surface is one that I now maintain so I keep trying to talk it up but it's okay so we have only a couple minutes left a few years ago you started using code names for the development kernels and in the new release of RC5 you changed the code name of the current kernel can you talk about that? I've used... how many in the audience know that kernels don't only have version numbers they have names too see most people now so it's not just a few years ago I've done this since way before 1.0 in the main top-level make file there's a name variable that gets set and it's usually a random creature where I live there was this one squirrel that kept running in front of my car for a week and so I named the kernel Suicidal Squirrel and the name is not used anywhere it's literally just a variable that gets set and then never ever used again but it's been this ongoing joke for me for the last 20 plus years and I think Greg names his stable kernels too and if something happens either in the news or in my personal life you can sometimes see it in the name change so the last release on Sunday named change to Kleptomaniac Octopus because we were diving this is what Dirk was aiming for we were diving and playing with an octopus two different octopus and the first one tried to steal my flashlight and the second one tried to steal Dirk's camera and the picture of that if you look at my Twitter handle that was up a moment ago I actually posted a picture of that it's pretty hilarious when the octopus goes for my camera so the name has absolutely no meaning but sometimes you can guess that oh, something happened in Lina's life where some random animal did something silly okay, so we are out of time so the most important parting question as always where should the 30-year celebration for the next be? so where should Angela host those conferences? it's only a couple of years it's only a couple of years Helsinki? no, I'm thinking diving and Hawaii I've done so Tahiti in a couple of years maybe on that happy note thank you everyone