 Good morning, I hope everyone is as awake as I am for some reason my body decided that 4 a.m Was a great time to wake up. I disagreed, but we couldn't settle my name is dirk hondel. I'm Freshly not quite freshly at the ember. I'm the chief open source officer for the ember and I'm Linus and I have Been around Linux foundation for the last seven years, so not so fresh anymore Seven no, it must be Whatever the little nation was founded in 2007 and before that you work for OSDL Anyway, clearly. We are well prepared for this morning Numbers numbers speaking of numbers 4.8 RC 3. Where are we? This looks to be a fairly regular release 4.7 was smaller than traditional well, not traditional Lately and that was I blame it on just being summer in the northern hemisphere, which tends to slow things down a bit the Poor Australians will just have to take a break with the rest of us But but 4.8 looks very normal our Development really has been fairly smooth lately. We have a very regular schedule for releases and And everybody knows how things work by now. I mean, we've only been doing it for the last ten years this way And things are looking on track, but we still have at least four more weeks until the actual 4.8 release Anything exciting in it anything noteworthy? I'm actually right now when it comes to exciting things. I'm looking at 4.9 So I know some of what's coming up for the next merge window So 4.8 to me is already old news because all the exciting stuff was merged in the merge window For 4.9 we have What I've been surprised about is that we were still doing low-level stuff We're still doing For a while there it looked like we're only doing drivers and we're only doing file systems like new file system every like clockwork several times a year and And then something changed and now we've been doing low-level stuff like really low-level People are rewriting assembly code changing how the stack layout is things like that So that's to me exciting to most users not so much, right So of course I I had to start with something else But everyone expects us to talk about 25 years of Linux now the critical question What is the correct date to celebrate? Is it really August 25th when you wrote the email or is it the first release or I? Think August 25th. It's probably the correct one because the first actual release was not public so When I released zero point zero one which was September Seven or fifteen or something like that So two three weeks from now I wasn't quite happy with it, but I told people that I'm going to release So what happened was I I made a release on a public website But then are at that point FTP site But then I didn't actually announce it publicly and I just emailed the people who knew and had shown interest And I've lost all those emails So I don't actually have an exact date except you can go back and find a Copy of the original tar file and look at the date in the make file and that's how I think it's a I forget Which date it is but but the one public announcement was obviously The August 25 one and that's the one that I think most people use and certainly likes what I should use this so I This of course this email has been shown many many times and I'm always amused by your predictions in that email and And the the accuracy that you have shown you said it's not portable and we today support 80 different architecture something I don't think it's 80, but it's a lot of architectures. Yeah More than anybody else You said you would never support anything but 80 hard disks Because it was what I had Well, I mean you had to realize the the background there is that it was a completely personal project And I didn't really I mean I expected other people to be interested from a theoretical standpoint like Okay Students all operating systems might want to look at okay Here's another operating system that we can look at right that was my expectation which meant that the kind of hardware I had was The only hardware that he ran on and it wasn't even the kind of hardware it was literally things like for the first two or three releases You not only had to have an 80 hard disk like I do you had to have the same 80 hard disk as I had because the geometry of the hard disk was Hard-coded in the source code right which I mean you could have a different 80 hard disk But you just had to change the numbers to match how many sectors you had So the first releases were a bit Unpolished So so when I started at 011 you already Supported any MFM drives, but if you had an RLL drive, so the the run length Compressed once they were small inconsistencies. Well, I wouldn't even say any MFM Drivers are still most of what the kernel does and the early drivers were Slightly less complete than what we have today. Yes, but that is why the early code was 10,000 lines of code and now we're at 17 million lines of code plus documentation Yeah, so when when I started I first thing I did was I printed out the sources. I do not recommend you do that today But my very tiny font very time But my favorite prediction is always this won't be big and professional and obviously you were correct This is all pretty amateurish Simple sorry Jim no offense. Yeah, no it took a while though I mean it really did take a while before it turned professional and some of us still struggle with it at times So what are the highlights or the low lights of these 25 years The highlights are really hard to even mention because I mean there's most of it has been really really good But this has got to have been the moment where you said, yeah, this is it. This is awesome aren't there these The moments where I really went this is awesome were actually that I remember like clearly have been When I've been struggling with something That just took forever Like I was struggling on the same thing for weeks and nothing worked And most of those that were actually very very early on the reason nobody's saying does operating systems from scratch is that When nothing works it is really really frustrating But then even the slightest sign of life makes you go wow I Really mastered this machine when you the first character shows up on the screen ever and the system does nothing else Right you're really pumped because you actually get got a character on the screen Wow, right completely useless, but that those Early times were actually some of the most like memorable for me when when nothing really worked and you go from from that Nothing to to something Later on and these days I'm actually most happy about when process works because I don't write code anymore So I don't have any code I can be proud of because I send off emails to random people and say hey This guy reports a bug. Can you you're the expert in this area? Can you fix it? So I don't have code that I can be proud of I'm making things work I can be proud of when when The release process really works and people People get things done and we don't have a lot of issues That's that makes me happy these days, but that it really has been working very well for for a while low lights We've had some Most of them have almost none of them have been about technology I mean we had we've had difficult times When things really didn't make progress Some people who I mean the developers in the audience may remember who if you've been around for long enough Power management was such a bummer for so many years and we really struggled with that And it took years and years to get it to the point where You could just take a random laptop and suspend it and resume it and you would just assume that it works That was very frustrating But on the whole low lights on the technical side have not been a big issue I mean you have frustrating issues that come up and are hard to debug but but most of the low lights have been really the social and and especially again the Development side when the process doesn't work. I mean if you go back 15 years ago, we had huge process issues. We really had During the 2.4 series we We really I'll take blame I screwed up the memory management and and we had this situation where during a few releases We completely switched around which memory manager I think Andrea might be here somewhere he he ended up fixing a lot of things and it was very very very painful and and and that I Still remember as a low light and as a this is not how we're supposed to do it but but for example Switching over to bitkeeper. I mean everybody Has heard about how bitkeeper was a big failure and how that grew ha ha one when I had to go off and Make get to work, but bitkeeper actually was a big savior for me personally because the situation We were in from a process standpoint before bitkeeper was such a disaster And and people may not realize because a lot of people Linux people have come in fairly lately But but 15 years ago We were going from there was a fair amount of commercial interest But it was still fairly early and and our processes at the time Were geared towards the pre-commercial interest Models that we had when we had a lot of of individuals at universities and our kernel was much smaller We had maybe 10 to 50 developers and then all these big commercial companies come in and now we have hundreds of developers and lots of new code Lots of problems that we didn't have to face before and our process was just a mess And it was very painful and that was probably the only time in the history of Linux where I was like This is not working and and I felt like Like in retrospect. I look back and say that might have been the moment where I just gave up Right and but that was 15 years ago and we had problems after that But literally for the last few years the release schedule and everything has been so smooth And and we have thousands of people involved and it's working. I mean don't get me wrong We still argue. I mean there are arguments where we're not all happy people. We don't love each other. I I I suspect a lot of developers really don't like each other but quite often even if there's not a lot of Happy Love feelings. There's I get the feeling that there's a lot of respect for the technical side and things are working very well In ways that things have not always worked I find it interesting that you mentioned that so did you ever really consider walking away from it saying and done? It's never been at that point There's been a couple of times where I Decided that we had Had big enough arguments that I took a week when I just actually it never ended up being a week But I decided that I'm I'm walking away from the computer. I am too angry at the people involved This is actually not lately this most of these go really back 10 15 years ago And and I decide I will be offline for a week because I can't I am just not Able to take this anymore and usually the next day I'm back and things are better But but it's been a couple of times that has happened where where I felt like this is This is not fun. This is not worth my time But but it not not in the last 10 years at least but I mean if I if I look at the 25 years of the next development Overall the the amount of success that that we made as a community the What we've created is impressive and to to look back and say I can't really think of any lowlights that alone is impressive The other well, I mean I can't think of it, but but they have been very rare I mean they have not been it. It's been a great community. I really I mean we're We're actually somewhat famous like the kernel mailing is this famous for not being a great community, but that to To a large degree it's been because people then highlight the bad things going on and if you actually it's People are so nice in general people are I wouldn't say polite very many people are never ever polite But but people are like you can just feel the fact that people want to work together and want to make a better system And that really makes for a great community But as a as a kernel community over the last 25 years There have been challenges at least challenges that were talked about One of the ones that always came up and I remember this in the early days when SMP was introduced was the question of fragmentation. How do we keep one single kernel? What's your thought on the technical debt in the Android kernels by all the different vendors that are so far away from from mainstream? I mean I used to be worried about fragmentation and I used to think that it was inevitable at some point Part of that was obviously the history of Unix where Everybody who was looking at history Unix and comparing Linux to traditional Unix was like saying It's gonna fail because it's gonna fragment because that's what happened before and we've seen this and why even bother, right? and There I think the license made a huge difference Me and the FSF we don't have this loving relationship, but I love the GPL version too I really think the license has been one of the defining factors in the success of Linux because the the whole Enforced you have to give back Has meant that the fragmentation has never been something that has been Viable from a technical standpoint Fragmentation can still happen due to social issues or just My biggest fear was that fragmentation would happen due to different markets where Originally when SGI was pushing Linux into Thousand core machines and RSMP was kind of wobbly even on eight CPUs I mean it was it was good if you had to it was okay If you had four eight was kind of pushing it really obvious and then SGI comes along as it Yeah, we have this 256 CPU machine and next year we want to put it on 1024 and long term more and I say They wanted to talk to me about how to manage this and I told them that yeah You should just go off and do your own thing because we're not there You should do your own big iron version of Linux and and sell it as SGI Linux And it's fine, right and they did to some degree, but they kept moving things back and The standard kernel also kept trying I mean that's where the license comes in that the code is out there And you have all these people who can try to merge it back because anybody can't do that and and we ended up just improving to the point where our Limits from the standard kernel went away and the SGI people were really happy to try to push their code to us so that they'd have less maintenance headaches with all the changes they did and And and that experience just convinced me that even if you have completely disparate machines and and Target markets we really can have a same single image from the source code standpoint and Right now we're seeing you mentioned Android It's not I don't think it's a huge deal the biggest problem with Android is that in the embedded market on the cell phones people are used to You have this odd dichotomy where the hardware is the newest of the new right? there's a new chip coming out and Three weeks later. There's a prototype phone and two months later in some parts of the world There's actually a phone on the shelves with this new hardware by by hardware manufacturer, but they have this Model where software tends to lag by much more than that so the software versions are often like two years old or a year and a half old because that's what they had and and Hardware developers are really excited about new hardware, but they just want to run the old software And then they hack it up to work on new hardware So that has been some of a problem always in the embedded space and now in Android But but even that is actually getting better So you mentioned the the strength of the GPL many new kernels have shown up in the last couple of years Mostly geared towards really small devices the IOT space. So Zephyr by Intel Now fuchsia by Google they're a bunch more and one of the interesting Commonalities that they're all under BSD or MIT, right? Do you think that what a do you think they're interesting be? Do you think that one of them could grow up and become a competitor for Linux or replace Linux? So I I think they're interesting just because I it's where Linux came out of to like small devices Riding tiny kernels that only were very limited and and I think it's it's always an interesting space and it's the only reasonable space where a new operating system can really make a big impact because the complexities of an operating systems tend to be in in all the hardware support and Flexibility of supporting Lots of different use cases. So if you start off you need to start off with something small and then today that means IOT At the same time the reason I think they show up under the MIT License or BSD or Apache is that most of their time these projects are started by companies and The flexibility for them to use something like a BSD license that allows them to do anything They see that as a big Upside and I think that if you actually want to create something bigger and if you want to create a community around it The BSD license is not necessarily a great license. I mean, I it's it's worked fairly well but You are going to have trouble finding outside developers who feel protected by a big company that says hey Here's this BSD license thing and we're not making any promises because the copyright allows us to do anything It allows you to do anything too, but but as a outside developer I Would not get the warm and fuzzies by that because I'm like oh this big company is going to take advantage of me While the GPL says Yes, the company may be maybe big But nobody is ever going to take advantage of your code It will remain free and nobody can take that away from you and and I Think that's a big deal for community management. It wasn't something. I was Planning personally when I started but over the years I've become convinced that the BSD license is great for code You don't care about The for code you want to make like this is I'll use it myself if there's a library routine that I just want to Say hey, this is really useful to anybody and I'm not going to maintain this I'll put it under BSD license But I mean I do want to whenever licenses come up I want to always say that hey this is a personal issue some people love the BSD license Some people love proprietary licenses, and you know what I understand that I'm like if you want to make a program And you want to feed your kids It used to make a lot of sense to say that you want to have a proprietary license and sell Binaries I think it makes sense less sense today, but I really understand the argument So I I don't want to judge. I'm I'm just kind of giving my view on licensing That's why we're here. So do you ever look at the sources of these other kernels? Do you know I you know it's on the BSD I can look I can no no I actually used to and not really I used to try to let's put it that way back in the very early days of Linux I forget what it was and but this is This was some issue I had and I tried to figure it out and I had just Hit my head against the wall for a long time and I said, okay I mean BSD is out there. I can go and look what BSD is do and I don't know how much time I waste that trying to follow somebody else's code I mean for a complicated problem, too. I mean it was obviously complicated enough that I had Had issues with it and I like how do I solve this? So when you have a complicated problem like that the worst thing you can do is look at somebody else's code It doesn't matter how much comments it has. It's not going to clarify things for you It's like you need you need the background to understand source code, right? So I made the mistake once of saying, okay, let's see how anybody else Solved this problem and I found it I Found it not useful for me and maybe it is useful for others, but I just I decided now It's actually easier to just read up Like a paper or a book about this OS design and try to actually understand the problem instead of Looking at somebody else's code and get it that way this may be personal But after that I just After one bad experience. I decided no I'd rather read Literature than read source code I'll read my own and I really want people to read their own source code You don't just write it you read it and try to understand it afterwards, too But but reading another project source code is really really hard So which projects do you look at the sources of besides the kernel? Um honestly The last project I looked at the source code of was Z mailer Which is a old mailer program that is still used by Veejer the kernel mailing list and nobody else and it hasn't been maintained for ten years and or more and So the only way to figure out problems is to look at source code And I did figure out a few problems and we fixed a few issues on the kernel mailing list with email authentication issues we had so I Think it actually is useful to look at other projects code if you want to fix a bug in that other project I mean that is how we get most of the kernel developers is they a Lot of bugs are really easy to fix. They may be one-liners, right? You don't look at a protest code to write your own you look at a protest code because you want to fix that project and you start out small and in 25 years later you find out that you are a big maintainer That's how we all start. Yes, so so let's talk a bit about this big maintainer thing One thing that I find fascinating about the the Linux development process is it's completely unique in its scale in its velocity Do you think this is? Something that other projects or even companies for in-house development can learn from the way Linux does things with the clear maintainer roles with the tree structure with the timed releases and all that We don't actually have clear maintainer roles Really, I mean I doubt anybody here can really even Google and try to find a maintainer ship Like traditional org chart for Linux. Maybe somebody has tried to do this I would guess that maybe you would find it on LW and you would find it from ten years ago John Corbett probably try to make an org chart for Linux and and that's why he's bald now but So the only people who think we have a clear maintainer There's two classes of people who think we have a clear maintainer ship roll one is the outsiders They my guess is that from the outside this all looks very organized, right? It we have releases on Every ten weeks it looks very organized. It's very smooth and then the other group is the the people who have been around forever and they just know who maintains everybody not because There's a documentation not because we have an org chart. It's just because when you've been around forever. You just know Oh, it's a driver Bug bug Greg about it or David handles networking and if it's wireless you go to something else, right? So So we have maintainer ship rolls, but they're not they're actually and I think it's nice that they're not like an org chart Because one of the downsides of org charts. It's all the politics You want to be upright because it makes you look good and and so you have the jockeying for position And when you don't have an org chart and you can't really show off your position in the org chart to anybody Some of the politics goes away. That's how many of here work for large companies and know about company politics Some people at least admit to this But the other thing is we're actually way more flexible when it comes to things and and people have been trying out different models like having Two or three people work as maintainers of one subsystem We now have one subsystem that is playing around with having two maintainers, but 15 people who can actually commit which is Sounds a bit odd and we don't know if it works, but Maybe it will work. We've we've tried a lot of these kinds of experiments with maintainership and And I wouldn't say that most of them work But the ones that a lot of changes that on the surface look completely crazy Do actually work really well after you give it a few iterations and you kind of Make the obvious problems that you didn't think about go away So we have this fairly fluid org chart and it's been it literally has been changing and and and it's it's not even a Hierarchy I would say it's more of a network of people that sometimes it goes across Boundaries just because you can email doesn't really care who you said it to yeah I always love it when you send your patches to some other maintainer Oh, yeah integrate them so that later you can pull if if I write code It depends a bit on the area sometimes I will just commit it myself because it's some trivial fix and I just I'm on a schedule. It's like something stupid that needs to be reverted for a Release then I will commit it myself, but quite often I will like say hey this is my my fix and I'll send it off to the real maintainer of that area and Maybe most of the time they commit it right so I I seldom get rejections, but but If they want to improve it, that's fine, too So we are almost out of time I For very current reasons I have one last question and I've asked this question of Linus a few times when we've had these Conversations and I usually phrase this conversation What happens if Linus steps behind the bus what happens with Linux and of course yesterday as we walk back from dinner He steps behind the boss. I'm like no and so What what happens when when when there is no Linus anymore? You know, I'm fairly careful even when people are I think it would have been a bigger problem 15 years ago, maybe even 10 these days There are so many people who could take up the work the mantle most of the people who could do it wouldn't want to do it I think right they they've seen enough that they say yeah now It's better to let Linus handle it because there is going to be conflict and then I don't need that headache But there are clearly I mean there are people who have been around for almost as long as I have been and There are people who are universally trusted and do a lot of the work right today that could step up So but my traditional answer is hey, I won't care so And on that very helpful note Thank you very much Linus. Thank you everyone All right Let's give a standing ovation to this man 25 years to continue to do the same job That's enough. I made him uncomfortable so But I couldn't resist. Thank you guys. Thank you so much