 Good morning everyone. So I think that conversation was not eight years ago, but something like 2003 Not to date you or anything, but it's been a while and You did this Linux thing I just learned mm-hmm. Okay good. So my name is Dirk Hordel This is my hometown. It's it's very rare to actually wake up in my own bed and then be able to go to a conference I really appreciate that we need to do more conferences in Portland. Who hears from Portland? It's actually fewer people than I thought wow. Okay, never mind My hometown as well these days for the last 13 years I have to say having a conference in Portland in February. I At least it keeps the people inside is what I say But Well, we were in in Tahoe last week and Jim's joke was that there would be mudslides as drinks at the evening reception did not go over well with all the mudslides blocking people from going there, but Okay, so normally when Linus and I go to a conference one of the things that we do Traditionally is we go somewhere really exciting to go scuba diving and then go to a conference This time there wasn't an easy Stop on the way to go scuba diving from our homes to here But I think we're going up to hoodsport in a couple of weeks where the water temperature is going to be a very Bami 42 degrees and it's going to be awesome. I'm very much looking forward to this. Yeah This is the second year in a row that we're at ELC Yeah, I don't know why I avoided ELC for many years and now we're doing it two years in a row How did that happen? I will not take any responsibility So let's start with some actual content For what 4.2 you released it on Sunday for 10 4.10. Did I say something else? Oh, I read it binary Sorry 4.10 you released it You released it on Sunday highlights lowlights anything interesting 4 to 10 was fairly calm 4.9 was our big release because Greg had made the mistake of telling everybody that 4.9 would be a Long-term release which always means that now several companies decide. Oh, we really need to hit this So 4.9 was bigger than usual and I expected 4.10 to be calmer It wasn't it was very average These days it means that we change a lot of things but there were no like Big things that you need to like bring up we had tons of new drivers We had tons of updates to the core I always find it surprising that even now like 25 plus years after we still have Updates to really core like including low-level assembly files for x86 that have been around forever But there's The whole process has been so smooth and normal for the last 10 years that that we seldom have these huge peaks We have a lot of working on all over, but 4.10 did not have anything that Specifically I'd like to shout out for so you said that Greg made a mistake Pre-announcing that 4.9 would be a long-term supportive panel. Why do you think that's a mistake? Isn't it good to give people advanced warning? This is the one you want to aim for so Greg has actually been trying it both ways and I'm not sure mistake is necessarily the wrong Like the right thing to call it I'm actually happier with having releases that are purely time-based and are just regular and then the we pick and Greg has done it that way several times too that he just picks it then based on timing and often after the fact and That just means that you don't have this situation where you have developers who feel pushed into hitting a particular release And I think that's healthier it just means that we We merge code when it's ready and we don't have any other pressures and and and it's being a very successful model for us and pre-announcing LTS releases is Kind of fights that it's not so bad. I mean realistically It was bigger than usual, but we didn't have any huge issues I just like how our process has literally been about Doing steady releases all the time and then you have the stable trees after that And then you pick one of the stable trees to be a long-term support release and and it's been we've been doing this for 10 plus years and it works really well So I want to poke at this a little bit because there are a lot of of groups not just Commercial companies but a lot of groups who actually like the idea of knowing which stable which kernel will be the long-term stable one So and and I remember that it was maybe a couple years ago There was a lot of frustration that the feeling was that the people were tricked to thinking it was a different one And there were some hard feelings So do we have any evidence in for dot nine that things got merged too too early? So more patches after the fact or or more stability issues after the release It's too early to tell yet. I don't think I think four point nine is a fine release And I don't think it actually caused problems. I I think our Regular release schedule is so predictable now that people don't actually worry too much and And there are people who care deeply about the long-term support releases, but at the same time those people also tend to have a Longer-term Look on these releases in the first place So they seldom have a lot of things that they want to get into this particular release So you did there was definitely some pressure and four point nine ended up being the biggest release we've ever done But but at the same time, I don't think we I'm not saying four point nine is bad I just say that I prefer the the model where where we don't have to worry about particular features at all And we just merged what's right. So you announced this crazy idea 12 and a half years ago. Is it that long July? 2004 I actually did my homework So literally half the history of our Colonel it has been this 10 week roughly Revolving the war of a new release and I mean when we started this I think we all in the room told you that you were bad shit crazy. Yeah, I think that was concept That may have been the exact words. Yes. Yes, they were Yeah, we've had it's interesting the Colonel is a big project and we've actually done a lot of things that that Tried a lot of different approaches Over the years and every time we do something it ends up pushing other projects to do the same thing Which I find interesting we used to do the even odd thing which was a huge mistake, but it worked for Several years where where we would have even releases are stable releases and odd releases are the development releases And we do that for like each release would Effectively take we were aiming for one year, but it would effectively take two to three years Which was very painful, but it became Inside the open source groups a lot of other projects ended up Trying the same thing and in fact when we switched over ten years ago to the new model where we do every Releases every ten weeks it took years before people realized that okay 3.11 is not actually a development release And 3.10 is not actually a stable release. They're all stable so we've We've tried a lot of different models obviously also in source control management with good and and the last ten years have been really pleasant for me because The process really has worked and we have not hit any pain real pain points yet and I say that because when you hit pain points in in your release model in your code flow That ends up being really painful. I mean those those have been the times when I have gone. This is not fun anymore, right? and and I like to say our Current release schedule is almost boring, which is exactly what you want You do not want your flow to be where where people start Trying exciting new things because that's really painful. It was interesting We were both in in Tahoe last week and before our but before my presentation The Kubernetes guys were on stage and they described their release model which sounded very familiar They have two weeks where you can commit features to this feature Repository after that gets frozen and then for ten weeks they develop on this I've heard that model before, you know, it really has been a very successful model at least At least as far as I can tell I think so I'm Definitely in charge of the development side So I have very little connection to Companies and productization because after the development kernel we have the stable kernel then we have Long-term releases then we have the companies themselves doing their own Releases on top of the long term so I'm often at least two or three trees away from productization, but from what I can tell even the product side really likes just this regular cadence of Releases and knowing that whenever they do start something they they always have something that is not too old And so it works both from a development standpoint and it does seem to work very well from a Productization standpoint to and now somebody can stand up and scream. No, you're wrong. This is horrible. Oh, that would be awesome anybody So But but you seem to indicate that this is all working well, so are people really still disciplined are people really? Following the rules Merch window and then after the merch window. It's only Fixes now how we can we're following the rules way better than we used to I mean the first five years It really took five years for people to really understand that there were rules at all Open source development is sometimes a bit Fussy people people do their own thing and people have very strong opinions of how things should be done So it's a changing the whole model of development. It took a long time before people really got it We're doing fairly well One of the metrics people use to check how well we're Actually following the the merge window rules is is looking after each release How much of the release was actually? Ready and in the so-called random next tree Before before the merge window open and we're usually 90% So 85 to 95% it depends a bit on release and that's actually a fairly I mean that's Clearly the bulk of it, but then there is that roughly 10% and a lot of that is bugs Getting fixed and that just happens Obviously after things have been merged, but there have been I mean there are some Systems that I just say Guys, I will not pull you can't follow the rules Go away, right come back in two months when the merge window is open again And then do this right because this you you can't send me stuff in the late RC release that I have never seen before and that is not a fix and and it still happens and Sometimes I'm public about these outbursts and sometimes actually most of the time I'm not and as I tell people in private guys. I'm not pulling this shit Because you can't follow these easy simple rules but on the whole I mean things work really well and And one of the advantage of the fairly short cycle is that even when people screw up You don't have I mean when we used to have two to three years of Release cycles if you screwed up and you missed that window You miss two to three years, right? And now if you screw up and you can't follow the rules and I have to shout at you You only lost two months maybe so it it still happens, but but it's It's way better than it used to be you can see it in a lot of other things people have gotten used to get I Used to every release. I used to have to really talk to people and by talking I mean shouting About how their git trees were rebased how they had more Merge commits than they had actual work because they were just merging from me every day during the middle of the merge window Taking random stuff that may or may not work. Don't do that. And I told people over and over and over again And I don't think I need to anymore I think people kind of a understand git meant much better than they used to but be also the whole thing where Where the process has been working so well for so long that that people kind of have learned what works and what does not work So I take this to mean the process is gonna stay around for a while longer I hope so I hope so it really I mean we've had we've had serious Disruptions in the process and they've been very painful and they've been good I mean don't get me wrong. They've been good looking back But at the time we've had so many very painful situations where where people really hated each other's guts and and people walked away And and there was a huge amount of stress Even if our process right now is not perfect. I'm not gonna say it's perfect, but it's it's working What it what having a stable process that people know how it works means that you don't have to worry about Okay What will happen? Will I have to change the way I work? There's a lot of us that Okay, I'm not gonna say we're autistic because we're not but people end up being very Set in their ways like you have you have You may be great at coding, but it's very annoying when Things change around you and you have to jump through hoops and you have to learn new things to do and We do have a lot of people who are I'm borderline OCD and and and really Having a process that works makes everybody so much happier. I Think I'll change topics here So one of the things that I always find interesting especially when we are in embedded events security and the Linux kernel and Specifically the insane size of the attack surface that Linux offers and Of course, I mean Chips are getting getting bigger and bigger. So the size isn't as critical anymore in the does it fit But I think it still is very critical in the attack surface Do you think that Linux really is the right kernel for IOT? I Think it depends on what you're doing. I mean if you're doing a Small sensor thing at the end point Linux may not be the right thing I'm almost sure that if you're doing these small sensors that our battery operated and they're they're doing one thing They're sensing temperature humidity, whatever and and they're just phoning back to a local Box that then gateway that then actually does the real work. I Understand that you maybe you want to use Linux even in the sensor for other reasons But but that's where where Linux is certainly questionable But then once you get the to the gateway or anything that actually talks to the internet Once you hit networking Things get much more complicated and at that point you probably do want a real operating system And I obviously think that Linux is the real operating system to speak. I Had a great conversation with with a an expert in the area of secure and reliable operating systems and he was convinced that we should start from scratch using rust and C++ for the outer layers and create a secure design. I know what a huge C++ fan you are I I Believe I maintain the only open-source project to which Linus has sent C++ changes Commits written in C++. I'm very proud of that, but What do you what are you I don't think so what what do you think of of the idea of creating something in a Domain specific language designed around security to create a more robust Colonel for the edge for the for the network facing devices I think the main specific languages and having something that is Clearly more designed towards security than C is a great thing I don't think those languages tend to be all that great at system programming It's doable in small niches, but and that's not where Linux is going So if you want to do a niche thing Go wild the whole point of Linux is to be general purpose that you can use it across a very wide variety of devices and and use models and And and there is clearly a certain dichotomy between Generic and security you're always going to be more secure by making your device or Service More specific and less capable. There's no question about that and Linux is not interested in more specific and less capable At the same time We've certainly seen more specific and less capable also be a complete disaster from a security standpoint I think in the end even a secure language will not help you when you have people who are rushing to get some random Small consumer device out the door and then you'll just make the security mistakes on a higher levels in your protocols and in your lack of encryption and in your Updating models and and all these details. So I Think you do have a huge advantage in Linux in the fact that you have a lot of different people You know a lot of different areas that do actually care about security. I Don't think security is an absolute and if you care about security and you clearly should in this environment You want to have multiple levels and layers of security and the kernel will never be Perfect, don't get me wrong. The kernel is big. It's complicated In in the embedded world you can limit that by compiling away as much of it as you can But even then you'll never be perfect Do a security shell around it do a security shell around that make sure that you don't accept any random connections And then the embedded people still put their own back doors because they want to debug things. So I'm sure you've seen the same stories as I have right. Yeah, so So security is is never going to be perfect Anybody who tells you that they aim for perfect security stop talking to them. They're morons. Don't don't even bother the people you want to talk to are the people who mitigate security issues and and admit up front that nothing is ever perfect and In this kind of crowd one of the mitigations need to be able to to do upgrades secure updates over the network don't depend on consumers doing them for you because they won't and Don't make your upgrade process an attack surface, which is The other common mistake, but make sure they can happen at every single layer Level and then you'll still end up having security issues with the hardware not being secure And somebody comes up with a really clever scheme to actually attack the hardware itself and at that point You just admit, okay, there is no perfect security next generation will do better Yeah, and you you need to make sure that you're resilient to some extent and have multiple layers of different Let's let's go back a little bit more to Linux today. We have 32 main Platforms in the kernel tree and if you know, no, no, no, we may have 32 architectures I would not call them main platforms because some of them are very very random My next question was gonna be Are they all maintained tested working useful? Realistically, I think all of them are useful to somebody there are people who enjoy tinkering and But yeah, now a lot of them are are really just toys and and are Testament to how portable the kernel is that you can actually create a new architecture in support in not very much code if you have a GCC port to your architecture You can literally do a Linux port to it in like a couple of thousand lines of code. It won't be maybe a completely full Capable system, but it will work But the end result of that is we do of the 30 plus architectures we support four or five are actually Real and meaningful and something you should actually think of using in a in a Production settings. Oh now I'll get you in trouble. So which four or five are those Just from a maintenance angle, I'd say x86 arm power PC and s390 seem to be the ones that get Reliable mean I might have missed something. I mean if I forgot one or two Don't don't think that that your architecture is not worthy But but on the whole those four are the ones that that I call out as being Both actively maintained and and in good shape And by arm, I mean both 32 and 64 bit. I don't Same for x86 the rest tend to be sporadic so in those four architectures are there Interesting innovations are the new things that you're excited about I mean this morning I read about AMD's Ryzen platform that will revolutionize the world which I come about but Are the innovations where you say oh, I can't wait for this to be available in public or is this all? Realistically most architectures are pretty much the same. I mean the Isis differ, but they all do the same things There are a few things coming up I'm very happy about for example power PC seems to be finally getting rid of the hashed page tables that I always hated with a passion and they have a radix tree so That's somebody can we close the door And that's a very welcome thing just because their memory management Will be so much better It's it's still early days for that, but I enjoy seeing those kinds of improvements on x86 We're actually it's funny We used to I mean so the kernel obviously used to be x86 only and on the original paging model for x86 was the two-level page tree that Tables that that the original 386 had and We expanded that from two to three because alpha had had a bigger virtual outer space Then we expanded it from three to four and now we're on the cusp of expanding it from 45 and The previous ones were fairly painful going from 45. It looks like it will be it might even happen 4.11 probably 4.12. It's 20 patches and they're not even that big and I'm impressed by how we can now I mean partly because we've done this several times before we can now just add a new level of paging and It doesn't even look like a big deal anymore So we we do have core functionality coming to core architectures At the same time One of the issues we have is is we still support Processors from 20 years ago, we finally did drop support for the 386 and Because realistically you cannot make a safe Operating system on a 386 because the page faults don't work right on that But but we still support ships that are 20 years old and and that don't have certain features So we still support chips that do not have the don't execute bit Which is coming back to the security issue. That's a security issue if you don't have no execute bit You are going to be open to a lot of attacks that ship modern ships aren't even open to So so some of the issues we always have are about compatibility and supporting chips that are Not as good as what you can get today and what you can get tomorrow So as someone who follows your active social media life I've noticed quite a few posts on Models that you're building whether it's RC cars or little dragons or whatnot Are we seeing a shift from Lina's the software developer to Lina's the hardware tinkerer? no, no, I've done hardware and They always end up not working that well now I grew up doing plastic models and One of the effects of the current roughly 10 week release window is that right now I'm in my really busy time and I sit in front of the computer 12 hours a day in five weeks, I'm expecting and hoping to be in the situation where I I'm mostly just waiting around for bug reports and fixes, right? So I can do most of most of my time. I do end up Checking email, but I need to check email once an hour and most of those hours not a lot of has happened So I started building models while I'm waiting for that to happen. Some people like watching TV that Doesn't do it for me. I think are with these tiny models instead and that that was good for last release And here I hope that you would announce the big project number for the Lina's maker kit No, no, no, okay, then I guess our time is up. Thank you very much Linus. Thank you