 Hello. Good afternoon. That's bright. Wow. There's a lot of people here for a Friday afternoon. This is nice to see. I thought it was going to be the two of us and nine people in front, but this is nice. So, I'm Dirk Hoondl. I'm VMware's chief open source officer in UR. And I'm Linus Torvalds. And I don't like doing public speaking because I get nervous and all stressed out about it. So, we've done this thing now for several years where I'm perfectly okay with Q&As. And that takes the stress out. Yeah. And speaking of taking the stress out, you're stressing two weeks over. 419 merch window. How did that go? Yeah, that wasn't so good. So, as hopefully everybody knows, we've had this release schedule where we have a two week merch window when all the new code goes in. And then we have six or seven weeks of calming down period when we try to find all the bugs in the new code. And that's been going on for a long time. And usually, it's not a big deal. I mean, the two weeks of merch window is my busy period. That's when I sit in front of the computer the whole day, especially the first week. And then usually by mid-second week, it's the stragglers and we're already starting to fix bugs. And it's busy, but it's not too bad except the last two weeks. So, 4.19, we had several issues like a new hardware security issue came out in the middle of the merch window. That just made it much more stressful than it should be. And maybe it's me, although I talked to Greg earlier today and it's definitely Greg too. But it's particularly stressful when you have a busy season anyway. And then there are all these other things going on at the same time. And we had the security issue. We had a stupid bug that has been around for a while. The report came in at just the wrong time too. So some merch windows are better than others and this was not a good one. You and I have talked about this many times where you say the best thing you can say about your job is when it's boring. And so when I talked to you about a week and a half ago and you were completely stressed out and clearly not happy. So right now, what are we doing to go back to boring? Well, I think we're over the hump now. The merch window for the mainline kernel was actually better off than the stable kernels. One of the issues we have is when we've had these hardware security issues and they've kept happening now the last year. They're kept under wraps. So we knew about the issue for the last several months. But because it was secret and we were not allowed to talk about it, we couldn't do our usual open development model. And we do the best we can and people really care deeply about getting a good product out. But when you have to do things in secret and when you can't use all the nice infrastructure for development and for testing that we have for all the usual code, it just is way more painful. And it should be. And then that just means that especially when the information becomes public during what is otherwise a busy period anyway, it's just annoying. And I have only myself to blame because if I'd not delayed 4.18, the timing would actually have worked out way better. But it is what it is. I felt at the time that I really needed to delay another week, the previous release. But so you have alluded to the security issue a couple of times now. So what do you think right now about speculative execution? I still love speculative execution. Don't get me wrong. I used to work for a CPU company. We did it in software back when I worked there. I think a CPU has to do speculative execution. It's somewhat sad that then people didn't always think about or didn't always heed the warnings about what can go wrong when you take a few shortcuts in the name of making it slightly simpler for everybody because you're going to throw away all that work anyway. So why bother to do it right? And that's when the security, every single security problem we've had has been basically of that kind where people knew that, hey, this is speculative work. If something goes wrong, we'll throw all the data away. So we don't need to be as careful as we would otherwise. And I think it was a good lesson for the industry, but it was certainly not a fun lesson for us on the OS side where we had to do a lot of extra work for problems that weren't our problems. It feels somehow unfair. I mean, when we have a security bug that was our own fault, it's like, okay, it was us screwing up. It's fair that we have to do all the work to then fix our own bugs, but it feels slightly less fair when you have to fix somebody else's. But so we had the first round of them in January, then we had another round in May, now we had another round in August. So should we check our calendars and not book anything in early November for the next set of bugs? I don't know what the schedule is, and if I knew, I wouldn't be able to tell you. The good news, I mean, the really good news, and I'm serious about this, is that the bugs have become clearly more and more esoteric. So it impacts fewer and fewer cases, and clearly hardware people at Intel and other places are now so aware of it that I'm hoping we're really getting to the dregs of the hardware security bugs, and going forward we'll have much fewer of them. But your expectation is that this is not going to stay with us, that every three months we get a new hardware bug, this will peter out and we'll go back to the last 20 years before that we had what, two significant hardware bugs that affected the kernel, and now we had seven, eight so far this year. Do you want to go back to the good old days? I hope so. I think we're going to the better days when, A, we got the bugs fixed, and B, people were thinking about them beforehand. We get to do our own bugs again. Good. But so one of the things that has been discussed a lot in the context of why these bugs came to be and how they are driven by a von Neumann machine being, you know, pushing at the edges of what the model allows you to do, and so over dinner last night the conversation came on quantum computing. What are your thoughts on that? Is that where our future will be? Yeah, I'm a huge unbeliever in that whole thing. I don't think it will ever happen, and if I'm wrong, I'm pretty sure that I'll be long dead by the time people can prove me wrong. Yeah, so Google has now what, 57 qubits? That was the claim. Yeah. What's the value? Where's the opportunity? I think the value has probably been more in physics and playing around with things, and hey, it has been known to happen that I've been wrong before. So maybe the whole quantum thing is going to be a thing, but I think if you actually look at where hardware is going today, the much more relevant part is that even traditional computers are not scaling, and people really don't see a lot of realistic paths forward to go on the hardware side. And I actually think that's probably healthy for the industry eventually, and especially for us software people who have gotten kind of complacent knowing that the saying used to be that every two years performance doubles, and that has clearly not been very true lately, and it's not going to be true going forward. And I think that's good. Well, maybe not fun, but it means that it will maybe go back partly to the time where you cared more about performance on a software side, and you had to be more careful, and you can just rely on hardware getting better all the time. So I already see the headlines that are being typed over here. Linus Torvalds, the plateau is reached. No more performance improvement in the hardware. Well, I'm not really saying that, but... It sounded like it. I was trying to help our friends with the press, yeah. It's pretty clear that the whole Morris Law thing is definitely not something you should take for granted anymore. I mean, this very much impacts the hardware people, but I'm saying it also impacts, I think, us software people, and especially our system people, where it means that software itself has to take that into account. I think it has a huge impact, because if you look at the bloat and the growth of more and more complex libraries, and if you look at the architecture that is increasingly commonly... We had this conversation with someone earlier where a Hello World program is 120 megabytes, and it prints Hello World. That's awesome. Obviously, some things got to give, and if the computer stopped doubling in performance, then maybe we will need to start running better software. Yeah, I think that's a somewhat welcome change. But one thing that I was going to poke a little more at, and since you dissed quantum computing, we'll just have to invent something else real quick here on stage, because the question is, what is the next thing after the von Neumann machine, or is that really the end of the development where we are now? So, I mean, I'm a software person, so asking me about hardware is kind of questionable to begin with. Which is why I do it, yeah. I'm actually a huge believer in neural networks. Back way in the days when I was at the university, I was studying artificial intelligence and the traditional kind of artificial intelligence, and always felt that that was snake oil, and that the real model of AI is to actually look at what we know works, right? And I'm really happy to see that this is clearly the direction that the industry has been going lately. And neural networks don't need an OS. Well, the network itself doesn't need an OS. But part of it is also that they aren't really computers in the traditional sense at all, and I actually think that we'll be in the situation that we'll have the Neumann model, the traditional computer model, for when you want to just give the computer commands. And I think we've also known for decades now how operating systems are supposed to look. So that is not going to go away, and that's not really going to change. I mean, the details will change. And then we'll have the neural networks on the side the same way 10 years ago. People were so excited about GPUs, and 10 years from now maybe somebody comes up with a new model, but right now it's clearly AI that a lot of the industry is excited about. And it's interesting, 30 years ago when you and I were both in college, AI was the big new thing, and now AI is the big new thing, and eventually it will maybe even be the big new useful thing. Well, eventually it will be just AI, and at that point we're done. There is that. So, let's switch gears for a moment. I talked about us being in university, we both actually did go to college. But if I look at today, if I look at the education system versus the people that I talk to here at these events, and the people that I see being successful in open source, does it make sense today to get a college degree, a master's, a PhD, or is just being a good developer enough? I wonder where that question is coming from. I actually, I enjoyed my college years immensely, and I think they helped a lot too. At the same time, programming to me, but this may be because of how I got into it, is something you do on the side. You don't learn it at university. You do it while you're at university learning other things. So I don't think it's an either or at all. It's a combination of both. And I don't think, I think you're setting the question up for a false dichotomy that isn't really there. I actually wasn't trying to set you up for anything. This was a truly curious question because two of your daughters are in college. The third one is going. I have a few more years before my daughters get there. But to me, this has always been something that I was interested in. I know so many fantastic developers who don't have a computer science degree, who have not finished high school, or are veterinarians, or biochemical engineers, or business majors, or a great friend of mine has a degree from Juilliard. And it's a fantastic development. So that was my question. How important is that college degree as a computer science or a double E major? So I actually think computer science is not like a science in the traditional sense because you can actually be self-taught. And in many ways, being self-taught is preferable. In a way that you don't want your art products or your structural engineer to be self-taught. There is a big difference there somewhere. So you definitely can drop out of high school still and be a good programmer. And it's not necessarily always a good idea to plan that way. I think programming and software engineering is kind of special and not real engineering in that sense. I would absolutely agree with you that the idea of a self-taught dentist is not one of my favorite thoughts right now. But I want to double down on this a little more because I think this is a really interesting direction to talk about. You said being a software developer is something where you can be self-taught, where experience plays a huge role, where curiosity and basically stubbornness. I know the word is perseverance. Perseverance is really important. And if I look around, if I look at the core kernel maintainers, if I look at some of the better-known developers out there, a lot of them aren't CS majors. And that to me was the interesting part where I said, is this a field where it's maybe different, where the focus on you have to have your eight years of college to have learned all the basics and know all the Latin words or whatever? Is this a field that is different and in such is maybe more open to non-traditional entry points? We had Van Jones talk here about the fact that we have so few African-Americans in this field. And obviously if you look around, it's kind of hard to see, but there aren't a lot of women in the audience. They all look very dark to me because of bright lights. And so to me the question is, could this be an advantage for our field that you can come in from a broad variety of background and experiences and educational paths? I see where you're going and I sadly don't actually believe you. I do think there's a huge advantage to getting a CS degree. I think you may partly be coming from... I know a lot of the current developers who never had a CS degree and part of it was purely historical. When we started, not a lot of universities even had a CS degree. So there were a lot of people who came in with physics degrees or came from math background. And I think that's still true. That's a very good way of getting into software engineering is by having a background in your problem space. But that doesn't mean that you don't want to have a college degree necessarily. It just means that maybe you shouldn't go for CS as your primary. You could go for physics or biochemistry. A lot of biochemistry these days is computational. And you are still going to often want to have at least the four-year degree is my gut feel. That's what I tell my children. Don't drop out. I'm kind of disappointed that you assumed that I was going to set you up to say drop out of school. Because I was actually curious about your thoughts on this subject and thanks for enlightening me. Let's switch gears because this is only going to get worse from here. One thing that is interesting to me as I go over to the exhibit hall and I talk to the community managers for many of the cool open source projects that are over there and Linux and Git and this other project that you do none of them have community managers. How come? I don't know. I never even thought about that. Particularly the kernel is kind of odd. If you look at open source projects in general a lot of open source projects are very small. They're like a handful of developers. A lot of them started inside companies so you actually needed somebody to push the project to not be inside the company. I think that's where a lot of the community management comes in is that you kind of need to be an evangelist and push the project out. The kernel for some reason never had that and yet is one of... We don't have a handful. We have a handful of thousands of developers and we never had a community because we had several different communities. People still... Almost nobody is on the kernel mailing list. A lot of people consider it to be an archival system not a discussion system and most of the development is done on much more targeted mailing lists. So you have one for MM issues, one for specific architectures, things like that and then you CC the kernel mailing list just to see if somebody does a drive-by comment or so that it's there in the archive. The kernel in many ways is not a good project to look at if you want to look at open source in general because it is so different not just in technology but in the whole community model and it always has been. But the same is true for all three of your projects? Well, you say three but let's face it the dialogue software community is so small that we don't need a community manager. Yeah, I don't know about Git and I do want... Whenever Git comes up in any public situation I always want to make it clear that yes I started the project and yes I took it to the point where it was useful but I have not maintained Git for the last 13 years. Junior Hamano has been a great maintainer and he deserves all the credit for Git and apparently he didn't need to have a community manager either and I don't know why that has been true of Linux and Git but I do think that part of it is they did not come out of a company. So then let's turn this around how should a project attract developers? What is your advice for a project that says we want to get more community, more developers, more participation? So my advice is actually slightly negative in the sense that if you see the point of your project to be to grow and you see your job as being to get more people involved you're not doing your job, right? As a maintainer of a project your job is not to find other developers your job is to make sure that the project works as well as it can and your job is to be responsive to the developers you have and if you do a good job at that and you make a good project if you build it they will come, a model of motivational speaking saying you don't go out and look for developers you do so well that developers come and look for you and that's my theory and I think it does match what we did in the kernel and what Git has done You just said something very interesting you said you have to be responsive and this is something that I had a couple of discussions here in the hallway I talked to Julia about this in the context of maintainers saying oh I can't respond to this thing within two weeks and my response to her was well that's a failure of the maintainer if you maintain a subsystem a two week response time is just way too long maybe you need a second maintainer What is your point on that? I see my job as being I mean don't get any wrong I do not answer email usually when you send me a patch if it's a good patch I will apply it and push it out and I generally will do that in a responsive manner but I will not respond to your patch and say good email applied normally but I do think that as a developer and I've been on the other side too not just as a manager but as a developer when you send out patches the last thing you want to do is not know where to send them and not know what happened to them once you sent them I don't so as a manager or maintainer of a project I think the primary goal is to make sure your developers know whether your patches got accepted and if they didn't what should happen to them and not on a two week notice more on a I'm not saying 24 hour notice because nobody's there 24-7 but I try to keep my response I tell people if you don't get a response and you expect one 48 hours then resend that's what I used to just say of course I've also moved up the maintainer chain so now I don't deal with individual patches very much anymore at all and if you send me a patch the real maintainer for that area did a bad job of explaining where you should send the patch so most of the time when it comes to the kernel we now have two or three levels of maintainership and that is definitely a way to solve the whole problem with one single person cannot keep track of even a medium sized project but we can quote you that you think a two day turnaround time is a reasonable expectation yeah no that's what I do I go away occasionally for I mean usually it's diving but so I'm not online all the time but I see my job as being making sure that when my sub maintainer send me a pull request I answer or I usually do the pull within the day and that includes weekends and the merge window is special and everybody knows that because then I get basically a few hundred pull requests oh it's 150 pull requests in two weeks and then I spread them out and I don't try to do them all on Monday when they come in but you talk about the fact that you respond on weekends and that you drill down into the hard problems and one of the things that I was curious about is what causes you to get sucked into a challenge and I'll give you an example where you didn't get sucked into a challenge so on the one hand we have kernel bugs where you get completely monomaniac and just never mind and on the other hand we have that long ongoing conversation about blurry fish but so photography where there is a challenge where you clearly did not get sucked in and did not try to get to perfection so what drives that oh I actually I claim that a part of any good driven strategy is knowing what to ignore right it's one of the reasons why we have many many kernel maintainers is because I was so open to saying this is an area I don't care about if you have any interest at all in this please come in and be a self-maintainer and quite often this was not explicitly said as such but it was I mean I was actively encouraging people to take over areas that I I'm not interested in and nobody has time and nobody has time especially for long periods of time of being there 24 7 for your developers that is not what I'm claiming you as a maintainer should do at all because that's you just burn out in weeks so my solution has always been to know what I care about make sure I do concentrate on that part and find people and give them rope when there are areas that I don't care about so when you're a sub-maintainer for the kernel and you send me a pull request and I trust you I will not look at your code because by definition it's not my job it's your job what happens and what people sometimes react to is when I get a pull request and I have during the merge window and I start complaining about some minute detail in that pull request and people sometimes think it's because I look at everything no it's because that maintainer sent me something that I was thinking that maintainer should not have touched that file why is he touching that file he does not own that code and then I start looking and that's when I start also saying I'm not going to pull you from you because you should not have touched that file and when you did touch that file I found a glaring error from just scanning your patches and at that point I'll go no, not going to happen so there's two sites to this one is give maintainers sub-maintenors the rope to hang themselves you are not supposed to micromanage them we call it empowering by the way oh empowering, is that the business term for give them rope but the other part is you need to also keep track of who owns what and if people start doing odd things you need to react to it and look at it and it's worked very well for the kernel I think obviously the teaching you should take away from this is once you get me to trust you you can do anything as long as you stay in your own area and it's true why the kernel maintenance works because I will I mean when you're a Greg when you're a David Miller you can send me stuff and I will not micromanage you unless you start going crazy and that's the only way to scale when you have a project with thousands of developers involved every single release I love you how you avoided talking about your photography skills but I implied right I implied some things I don't care about it turns out I suck as a photographer and I'm okay with that because I don't care and now I'll try to do the hard thing now I'll try to make sense of all of these questions that I've asked you which clearly surprised you earlier so if I look at the way software is being built today if I look at the insane complexity of many of these frameworks if I look at the fact that as you said there are parts that you cannot possibly care about because you don't have the time you being anybody and how this becomes a more and more indirect more and more layered more and more complex world aren't you afraid that we are dealing with a technology that we no longer understand that we aren't trained to have a top to bottom view you have told me many times that one of the things that attracted you to the kernel was that you love the interaction with the hardware and knowing how things work but then when I take a step back today and they use a typical application whether it's on my phone or on a computer there are so many layers so many components that no one knows how all this works and so what is your thought on that on this on this conclusion of complexity so I mean this is why I actually don't worry about technical issues in the kernel so much I mean I that may be phrasing and misphrasing I worry about them but I'm not worried about them in the long term because what I really worry about tends to be the flow of patches so the workflow to me is actually way more important than the code if you have the right workflow the code will sort itself out if a bug happens you know how to deal with it and this is actually why I think open source I mean I've always been a proponent of open source obviously but I think this is also why open source is so fundamentally the right model of doing development is because when you have complexities like this you cannot manage it in in a closed environment you really can't it's you need to have the people who actually find the problems you need to give them the ability if they have the possibility to get involved and help you fix them because it is a complicated world and when you're doing a complex project and the kernel certainly counts but it's not the only one the only way to deal with complexity is by having that open exchange of ideas and having the code out there so that people can participate so let's try again for a perfect linear sound bite something that fits into a tweet so do you still understand Linux kernel? No, there are certain areas that if you send me a patch you should expect way more reaction there's the one area I've cared about pretty much since the beginning was the virtual file system layer that's the only area I think in the kernel where I am still very active so that I will actually look at all pretty much all of the core patches certain parts of the x86 architecture stuff I will also follow but when you have a project that the scheduler is also something that you tend to I don't worry about the scheduler anymore that I used to but there's 30 plus thousand files in however many 10 million lines of code nobody knows the whole kernel so the one one thing I have is having looked at patches for a long many many years and I know the kind of big picture of pretty much all the areas in the kernel so I can look at a patch and I can often tell whether it's right or wrong even if I don't know that area in detail and I then obviously know who to blame and who to approach when things really go bad and I say hey here's a bug report I know it's yours go off and fix it for me because I can't this is everyone's favorite git command git blame damn it's my code so what worries you about the future then if you think that everything well maybe it's fine that things get more complex are the things that you say oh this this is a problem I still worry a bit about our maintainership I have no reason to worry about our maintainership we've been doing really well and we have new people show up all the time and we have old people who still love doing what they're doing so they're still sticking around and it's not because I hold a gun to their head I promise so we're doing everything right and we haven't really had any issues but at the same time that is the one area where I'm sometimes looking at key people and saying oh that would be really really painful if he got hit by a bus I used to get the question what happens if Flynus is hit by a bus and nobody really cares anymore because there's always a maintainer for whatever subsystem we've got that covered but there's a few subsystems where you have a bus factor that is pretty low and there are a few people who I think are somewhat overworked and that worries me a bit why are you bitching about Greg I expressly didn't want to mention him he misunderstood so the last time we were in Vancouver I'm sure you remember no the last time we were in Vancouver I'm sure you remember we were driving up here we did our dive master training this was 2011 and we had the 20 years for the Linux event come on help me out here so we had 20 years of Linux celebration up here in Vancouver and it was awesome and so I was thinking as the closing question for our conversation how about we cause problems for the Linux foundation make Angela really mad at us and create that perfect way to get off the stage so where would you like to have the 30 years of Linux celebration because that's coming up 2021 she already booked the 2020 conferences so if you want to get ahead of the 2021 scheduling now is your chance people are clearly they know how I work so for I think for 25 years it may have been Angela maybe it was Jim who had the great idea to do it in Helsinki because the birthplace of Linux and I was like no let's do it in Hawaii and that didn't work I don't remember where we actually did it I think it was Germany somewhere Prague not Germany just making sure you know I'm not that confused but I don't think it's so much about the location but somewhere warm and scuba diving would be good happy note thank you so much always a pleasure and look at that