 So this week we decided to take Linus Torvalds to meet another black and white animal. How are we doing this year? There he is. This is Linus Torvalds in Chengdu this week and Linus leaned in to the panda He said, you know what? I Don't think we want the penguin anymore I think we're gonna go with the panda for Linux and look at the reaction from the panda. I can't tell if the panda is scared or happy or Just really likes that carrot. I think it's the carrot But Linus and I got to spend some time meeting pandas Going around China We love this country. We love furry black and white animals and with that I want to welcome to the stage our next speaker Linus Torvalds for a conversation with Derrick Kondel the round of applause for Linus and Derrick Thank you. Hey How is everyone? Oh, it's quite a bright light So my name is Derrick Kondel. I'm VMware's chief open source officer and this is I'm Linus and I'm not very comfortable doing public speaking so to solve that problem We've done this questions and answer session for the last few years and that works better for me Since I don't know the questions beforehand. This tends to be Slightly more interesting for me too. Hopefully for you and it's also a lot less work for you to prepare That's that's the big part of it. Yes So I always start with the same question So I apologize if this is boring for everyone in the audience, but maybe it isn't so yesterday You released 418 RC 2. What's the state of Linux? So I have to say to kind of explain the background we've been doing the same process now for almost 15 years and I don't do development anymore. I Am a manager. I've write code most of the time when I write code It's actually in email when I send people saying hey This is not working. Maybe you can try this but what I care about most and what I have cared about most for the last Couple of decades by now is actually that the process works so that people can write code and and we can Keep on improving the kernel So we have a release schedule that is about eight to ten weeks for every release where the two first weeks are merging new code and I am so happy about the fact that it continues to work So for 4.18 for example, we had roughly 1500 people who actually wrote code and I merged that code from about a hundred maintainers that that maintain different sub areas of the kernel and This process hasn't changed and that's I think the most important part is to allow people around the world to just keep on improving Linux and This release there was no new architectures there it was One of those maintenance releases where we're cleaning up some code We're actually removing more lines than we are adding this release, which is always nice to see when people decide Some piece of hardware for example never made it into the marketplace and we just end up removing the driver and and things are going fairly smoothly and I think With a project like the kernel which is the infrastructure for a lot of Other software projects and a lot of other hardware problem projects to having this kind of smooth fairly predictable Development model really helps everybody So we've had very little change in the process in almost 15 years But still I mean people are changing what we're doing is changing Yep Are there parts in the process that where you think we're doing better than be used to or where you think we need to Pay attention we're kind of degrading. I think we're doing better than we used to simply because people have gotten used to the process We were better following the rules just because by now people take the rules For granted it doesn't mean that we're perfect and I end up shouting in email at people every single release for People who forgot to follow the rules that release We had a few cases Last week where I know why I was somewhat unhappy with with changes that came late and shouldn't have been sent to me During the so-called RC period when when things are supposed to calm down, but on the whole I think my Email outbursts have gone much less common and and that to me is a good sign It means that that our process is actually getting better So so what you're saying is boring is good. Oh boring is great. I I love boring if I'm I'm actually somewhat Disgusted with the whole tech industry being so enamored with the word innovation and and a lot of these other buzzwords about how we're Revolutionizing the world because I think a lot of engineering and when you really get down to Engineering is about getting the details right and doing a good job and just Very little engineering is really about innovation boring plodding work is the real work and and the fancy stuff makes a lot of the news but but I If if I can keep my project boring, but alive and doing well from a development angle I will be very very happy There's a saying Genius is 1% Inspiration 99% perspiration that I'm a huge believer in Yeah, but so on the boring side. We certainly did not have a boring first half of 2018 thanks to spectra and meltdown and and friends What's your take on that? So I don't know I'm assuming everybody in the room is very familiar with the CPU bugs Design problems. We've had for the last half year. It's actually been an issue for the kernel community for about the last year and People were kind of whispering about it before that It's been very frustrating. I Actually one of the reasons I got into kernel development the main reason I got into kernel development is I'm I Wanted to know how hardware works. That is the kind of thing I find personally very exciting is it's just on a very low level how a CPU and how a computer works and it's why I'm doing kernels and I'm actually finding the whole CPU bugs I find them fascinating So it it should be something I'm excited about but especially with the security issues and all the secrecy that in that Ends up involving it's been very frustrating. I know it's been frustrating inside companies because they have to try to contain the whole Security issue so that it doesn't leak ahead of time But it's particularly frustrating in open source where we're so used to Talking openly about all our issues and I'm trying to find the best solution by talking to Literally thousands of people every release and then you have issues like the security problems in the CPUs where we have months and months of very controlled Silence where we cannot talk except to this small group of people who know about it and it has been a Very frustrating experience and I really hope that we're We're not going to see a lot more of this because it's it's just very painful for everybody But do you think that the vendors are learning from from the frustration that you have expressed? Is it getting better? I it is getting better a we have gotten slightly better at the process. We are When when the news first hit we really didn't know how to work at all when we're so used to the open mailing list model of doing development and now we have Six months plus of experience with this so that helped a bit People are still very frustrated. I'm I'm actually hoping that Even more so that the hardware people learned from the experience so that in a few years We can look back and laugh at this, but but it's gonna take a few years But I mean if you look at the the type of bugs that these were they were fundamentally Information disclosure leaks, right? So that that makes me wonder about computers and in the context of privacy that we talked so much about these days It should we be worried about our trust in computers? I Think we Always should have been worried about our trust in computers and I want to make it clear that the Hardware bugs even though they've been very painful for us We I think in the end We on the software side have had a lot of bugs to say it's not like we can say hey Hardware people get your act together We we have our own issues that we need to fix it's just that when we have bugs in software They're so much easier to fix them. We can't just fix them on our own When we have hardware bugs, it's very frustrating for a suffered person because this Underlying piece that we thought was reliable isn't so it's it's a different class of problems It's a different class of pain for us But fundamentally you're saying not much has changed it feels that that our The level of trust or distrust that we should have to Compute-based systems haven't really changed. No, and I I don't think the whole security issues. They will never go away As an industry, I think we're getting better. We're getting better both in software and in hardware, but it's Obviously an area where we need to just remain vigilant I want to switch gears a little bit and talk about Developers as individuals not not so much about the projects But about the people behind the projects and one of the distinctions that I think I'd love to talk about a little bit Is the difference between the typical developer a let's call him a tech lead or a lead developer and then a Maintainer can you talk a little bit about the skill sets and what is different between these roles? So one of the big difference actually, let's talk about what's common Pretty much every single maintainer started out as a developer I mean, that's the only way you can become a maintainer because the most important portion of being a maintainer is Being trusted It's not a you need to be trusted by me So that I trust you enough that I can take code from you But you as a maintainer need to be trusted by all the people who are sending code to you as well and the Trust is needed that you have the technical understanding of the code to make the rate right decisions as a maintainer so we all started out as developers and The thing that then makes some people maintainers once they've got it at the trust is usually communication Especially in open source, but I think it's It's true pretty much everywhere a Lot of what we do is Communicating people always think that hey the big part is is the writing code the writing code part It's often the smallest part. You need to communicate to your maintainer Why you wrote the code and that's often a problem you need to Be there when bugs happen and they will happen and you will be need to be open to communication Both from your maintainer, but also from your developers That's obviously one area where we've had one of the issues we've had in open source is because we're so spread out Sometimes the communication barriers are simply cultural or language barriers Something I'm sure people here in China are very aware of that Sometimes you need people who do nothing but communicate because they end up being translators between the developer and The maintainer or between two different developers, but it turns out we don't have enough maintainers. I A lot of people think Maintainership is the glorious thing where you don't need to do the low-level encoding anymore and you can just take code It turns out Finding people who are technically able to understand the problems and our good communicators and Are willing to be there every day pretty much so that they're responsive to developers is really really hard So if you want to be a kernel maintainer Trust me. There are spots open It will take time to get there because you will need to get that trust and get that network of People who who you know and who you can communicate with but we need maintainers So it's it's a good job market to to look into so you you focused on on Communication and on on being available on being being there every day One angle that I always found interesting in maintainers is the ability to deal with conflict Because very often you will have different developers with different ideas of what is the correct solution or you have Competing commercial interests that push for a certain architecture and alternative. Can you talk about that? so I have to say personally, I'm actually less worried about technical conflict than about Non-technical conflict so technical conflict usually when when you have two competing approaches to solving a problem the good news is If it really is a technical conflict You can usually put it into numbers. You can say this solution is simpler and smaller or this solution is faster and and I can do more with it and you can usually Make a fairly informed decision on a technical conflict and more importantly even if you get it wrong Which happens all the time Even if you get it wrong if it was a technical decision you can say hey I made the wrong decision. I'm perfectly happy saying that as a maintainer saying This is not what we should have done and you undo the technical thing and you go to the other approach instead. This is This is normal development There aren't a huge I mean it's it's a waste of time obviously but but it's not a huge problem the real conflict problems come in when it becomes personal or Particular when it comes becomes political inside companies. I'm sure you're all very aware of this that when there are conflicts that arise from from non-technical issues and My approach to that and I think this is where open source really really shines is That we have been very good and we have been able to try to Keep our conflict resolution to technical issues so when political issues come up I Personally will not care. I will say this is all bullshit. I won't even respond to it. I will not Spend one second worrying about it and I will tell you guys Give me the technical reasons and I think this is something where open source really can work in ways that often Commercials offer cannot because inside a company company politics are all encompassing you cannot avoid the company Politics when you're in open source You can you can say hey there are all these internal company politics But look at all the people outside this company that don't even understand our politics and and I think open source Tends to often successfully avoid politics for that reason So we talked about this this Ark from being a good developer to becoming a good maintainer and what are the additional Skills that you need but let me turn this around So do you think that every great maintainer is also a great developer is so is being a fantastic Developer a requirement for being a great maintainer. I Don't necessarily think so. I think it's hard though. I mean if you're not a good enough me, so I Have a hard time judging. I don't know. That's the maintainers. I work with I don't know how they feel about this but to me personally One of the things I do a lot is I have to make that judgment call on looking at a pro request and saying Do I need to talk take this or do I need to tell my sub maintainer that hey This is not acceptable. This is not how it works and It's not just about making the right decision and or at least trying to make the right decision It's also about how much time you spend making that decision and one of the things that takes a long time For developers to go to maintainers is in my opinion being able to look at code and just by by looking at the code saying, okay, this is the right approach or this is the wrong approach and I almost think you have to be a good developer To be able to make that judgment without spending hours and hours and hours agonizing over the code itself So I always look at this as as to related but slightly different skills One is the skill to look at code and evaluate it and the other skill is the ability to actually write that code So as a maintainer I faced this when people sent me code in Python. I Hate Python Yet, I I think I can read it well enough to to judge it as it isn't that part of what a maintainer does Reading code instead of writing. Absolutely part of what you do is not just reading code But you also read the explanations as I mentioned one of the big issues was not just writing the code But also writing the explanations for why you wrote the code not what the code does You can see that from the code itself But explaining why it does something and maybe why it does it that way So it is the different Skills that definitely one of the things I've talked to people about occasionally is What I look in developers and what I look in especially maintainers is something I call good taste And it is very hard to explain. It's kind of like art That you can't really say what is good taste you can just say some people have it and some people don't and I'm not sure you can learn good taste. I think you probably can but there are certainly people who are able to express the problem in More beautiful way and there are people who are able to make that distinction between This works, but it's really really ugly and it's clearly not the right way to do thing and saying that code not only works, but it is actually the right way and there that's Ability to have good taste is fairly rare and it's probably one of the reasons we We are Desperately looking for maintainers, but this is an interesting question that you raise here So how do you become a better developer? There's obviously there's the book outliers that say 10,000 hours of practice and Practice is certainly useful, but are there things people can do can focus on can learn Can practice that will help them become better developers and maybe even become maintainers and better maintainers I think most of it is just being I mean not necessarily writing code That is a part of it obviously, but a lot of it is about participation Being walled Don't and when it comes to a big project like the kernel But there's a lot of other open source projects. So don't get me wrong This is not just about the kernel, but you don't have to try to be involved in some big portion of the kernel You don't have to go for the central part that Experts have been working on since the very beginning of the project being walled find some small detail Something that you care about find something that people are discussing Maybe people are not discussing the code itself But they're discussing the problems in that area and go and be involved on the mailing lists on There are IRC channels, I don't know what I assume in China We're looking into we chat or something like that and and just get involved look at the code Start making patches and realize it's going to take a while before you get to the point where you're an expert But it's okay to be an expert on a small part and then you tend to expand after that Don't try to be an expert on the whole project. That is not that is not how the world works But I think that's a very important point that you're pointing out It starts with actually getting involved and and it's okay to be focused But then bring the patience bring the patience to say hey my first attempt maybe wasn't good enough Let's do this again. Let's come back and that's learned from it And I think it's this feedback cycle that it's really really important part of it is also It's not just You as a developer learning how something works and learning how to make patches you have to realize It's all the people around you that see you being part of the community So when you're a developer As a maintainer the people who are taking patches from you the first time they see a patch They have to be very very careful because they don't know you they They have to assume that you are not very competent And that means that they have to look at the patch very carefully So it really helps if the patch is small simple and obviously fixes a problem After you sent a maintainer 10 20 50 patches What happens is the maintainer stops looking at the patch as careful The maintainer knows you the maintainer has seen you act the maintainer has seen 20 patches from you that are good the maintainer knows that you're doing good work now the maintainer trusts you and That's how you eventually become a maintainer yourself is because people stop looking at your code entirely and say hey, I trust this person so much That when he sends me something I will read his description of what he sends me and I will integrate it That is how I work with the 100 plus sub maintainers in the kernel is that Most of the time I don't need to look at the code and there is a graduation There are sub maintainers that I never look at the code for and there are sub maintainers that I tend to look much more closely at what they're sending me Some of that is just I've worked with some people so long that I I say They own that subsystem if they make a choice Who am I to argue and other people I've had more problems with so this is a growth cycle at every level even as an Developer when you learn to be a developer Part of it is not just you it's also how people see I think it's interesting this network effect this on the one hand I try to develop I try to become better But then there are all these people around me who are helping me who are giving me feedback who are sending me ideas how to make my code better or how to describe the code better and I want to do this in a very small way here on stage for a moment and You you talked a little bit about this a moment ago The question what makes a good commit is something that I run into on on pretty much every open source project I'm engaged with but even inside of my company when talking about the developers writing the proprietary code What is it that makes a commit good that makes it Something that the maintainer looks at and says okay. Yeah, I like this um To me what really makes the difference is if I can see a Theme to the commit I see that a the explanation is really important I don't know how many of you have been involved in kernel development And how many of you have been involved in some other projects For the kernel we have very very high standards for Not just the code, but the explanation the commit message that explains what the code does Uh, a lot of projects think of one liner that says This is what it does. It's fine. We are not there. We we we want explanations But you want the explanations to make sense and then when you start looking at the patch You should not be surprised So lack of surprise is probably the best single sign of a good commit If if if the explanation says something and then when you read the code you go, what? That's that's usually the worst sign there can be But then sometimes There are good commits that can be one liners They can I mean a one line change is not necessarily a small change A one-line change can be very important if it's in a very subtle piece of code And you just noticed a really hard bug that has been around for years And it was something really subtle like the memory ordering of the cpu required you to add one line That can be a very important fix But a one-line change can be really important it can be really boring too But a hundred thousand line change can also be Both it can be something really important that adds fundamental support for a really big feature Or it can be something really pointless that just adds a lot of lines that describe a piece of hard work And the patch itself does not do very much So you can't really judge a commit by its size And you have to judge it by By the its downstream effects of that commit and and what it really changes But the one thing that you said about lack of surprise is something that I think is incredibly important I have seen commits from you actually that have a 45 line commit message And then change a single line of code And and it often seems like wow, why does he write so much about such a trivial change? And then you read the commit message and you say, oh This is what this does and to me that really is this moment where once you're done reading the commit message The patch below seems totally obvious That to me is really the the way I look at a good commit So don't get me wrong a lot of the times the good commits really are the boring ones the ones that are not One-liners, but they're also not a hundred thousand lines. They're just Doing regular every day work. They fix a bug that in retrospect the bug was really obvious So maybe the commit message is only three lines and saying hey, we had this bug because we were Stupid and we hadn't noticed this one simple special case And here's a 10 or 100 line fix for it And you you don't necessarily need to have a lot of commit message for that So and that's actually the bug that should be The daily bread and butter of your work There most of your commits should not be the exciting kind if you're If all your commits are exciting, there's something badly wrong We said earlier boring is good boring is good and now let's take this one one abstraction level up We talked earlier about The interaction with the maintainer and that the maintainer will push back and and will maybe disagree with you And one of the questions I always ask myself. So how long should you Do this this loop of of listening to the maintainer and Doing a new version of your commit and go back and forth and back and forth What's the point where you should just give up and say this isn't this isn't worth it? I should fork the project or I should give up on my feature I don't know It all depends on what you're doing sometimes and and Don't get me wrong Maintainers have been around for a long time But sometimes the maintainer is wrong too So sometimes what needs to be done is say This maintainer is not agreeing with me Maybe you need to talk to another maintainer Sometimes very rarely. Hopefully You may need to escalate it to me I try to avoid that because I hate bypassing maintainers Partly because it's more work for me Partly because it just makes for a mess when the maintainer is doing his own thing But the fact is Sometimes you also need to accept that maybe my idea wasn't good Maybe my idea was for something that For a problem that I have or that my company has But nobody else has and nobody else cares about And and then at that point One of the advantages of open source is saying hey This is An issue that only hits us because we're doing something really special And maybe we need to just maintain this set of patches And it's still open source and everybody else can use it But maybe nobody else is interested that happens And I think it's one of the advantages of open source is Is exactly the fact that not everybody is the same Not everybody cares about the same things Sometimes you need to just Go off on your own and say hey I care about this this feature nobody else seems to care And uh five years later Maybe you're still the only one who cares about that feature But the other possibility is also that five years later You are the industry leader because everybody else is wrong And an open source kind of allows you to make that choice I mean you and I ran into this in in one of the other projects That you work on where one of the libraries that we depend on would absolutely not Take a set of changes that we had developed And so we've been maintaining a separate fork of that project for I don't know three four years I mean this is how um a lot of open source development gets done And this is literally how kernel development gets done And and one of the design decisions I had when I started git was that One of the big designs designs are behind git is to make forking easy You want to allow people To just take your project And make make it their own and make their own version and and Allow them that freedom of showing the world that they actually have The right idea and they're doing a great job of maintaining it The other part that I think is very important is Make forking easy but make it really easy to also Then on the other hand say hey, you were right I was wrong You did a great job. I did not believe you But you did such a great job that now I'm going to to say hey I was wrong. I'll take your code after all Because I didn't make the right decision in the first place And I think actually that's something that we in the kernel community had been fairly good about We have avoided these So-called religious wars that we have seen in some other projects Where we've avoided getting To personally antagonistic against forks So I for example think that forks are a good thing And if a company ends up going their own way and and taking their Kernel in the direction that I think doesn't necessarily make sense for The mainline kernel I don't see that company as being Competition I see that company as just Trying to do what's right for them And that means that if you're not antagonistic and you don't Start this whole religious flame war over this is the right way When it turns out that somebody else was right wasn't the right and did the right thing They don't hate you they don't They don't mind you they actually feel kind of smug and say hey Look at me. I was right and when you pull from them and say hey, you were right Everybody is actually happy. We improved the system. This is how a lot of development in the kernel gets done But but there is a dark side to that So we have seen this in embedded linux, especially in phones in a few other areas Where it has become so easy to fork linux that people create Support packages for a specific hardware device for a specific Board for a specific phone And and those never come back to the kernel and the users are then stuck on an old version of the kernel What are your thoughts on that? I'm very unhappy about it at the same time. I it's I don't feel like it's our process problem One of the issues is in the embedded world. That is how people are so used to working I don't know how many of you are embedded people, but That's how you work on a hardware level too. You make basically One-time board knowing that five years from now That board will be throw away Because hardware will have Changed so much that what you make today Will not be relevant. I mean you will learn from your mistakes But at the same time You make a board once and you don't generally see it as Something that you continue to develop and that mentality then has clearly extended into the software space too Where people make one product And the hardware is seen as a hey, this is our hardware for this year or for the next five years And the software is treated the same way So you take one kernel version and one version of your ui and whatever else you use to to make a product And you don't try to see it as continual development on the software side But it's a throw away thing that you do once And but but that has potentially very bad side effects when it comes to Security bugs and the inability to update to a newer version of the kernel, right? It's a horrible model I'm not saying this is good. I'm saying This is what a lot of companies are doing And it is a problem for us because it does mean that they don't Actively try to feed it back to us because in order to then feed their changes back to mainline kernel So that they get maintained long term you have to do a lot of work You have to do all that communication work. You have to Send all these commits that have good explanations for why you're doing something and they need to make sense So that we don't go what? So there's a there's a There is definitely work involved in upstreaming code But once you are able to get your code into the upstream kernel It means that five years later when you make your new and improved Product whatever it is hardware software anything We will have maintained it We will have improved your code other people will have improved your code in the meantime And it will work better and you will not have the problems you had the first time around So that is the right model to do it is clearly not the model that everybody's taking I mean The learners community certainly tries to make this step easy greg who I think will speak on wednesday He's somewhere here in the audience greg is doing a fantastic job on working with harder companies And taking code even if it's not perfect and getting it into the kernel putting it in the staging area And and help make it easier and we've seen hardware vendors who have done an exceptional job at that if you look at intel had always always been The the software support for hardware was available before the hardware ever became Came into users hands to the point that at times drivers had to be removed because products were cancelled I think there is a lot of role modeling a lot of opportunity out there to do this right I also think that things are getting better I think part of it is people are After three or four iterations after you've done three or four different versions of the same thing You kind of start to see the problems inside the company and the companies themselves are starting to say, oh We've done this before it didn't work so well last time. Let's try to upstream at least the very core stuff We care about so that we don't have to do this again I also think that especially in the embedded world what is helping us is that the the hardware tends to also Become slightly more standardized. You you find that people are Standardizing around the more common SOCs that are cheaper and easier to get so there's a push From that end to also not be quite as wild and crazy As people wear five or ten years ago So jim showed a picture that that indicated the possible new logo for for linux. I'm very excited. I love pandas So if you were to stop doing linux, what would you do? You would you be a panda keeper or a dive master or I I don't think I'd be a panda keeper because there are very few pandas outside Slide china and I don't speak the language although maybe the panda doesn't care And I don't think I could afford to be a dive master I've always I mean, I don't think I have no idea what I'd do without linux. I had big plans when I was young to be Professor at a university And it turns out I don't like writing documentation and I really hate to write the papers And if you can't write or you don't like writing papers, you can't stay at university So I think I just need to find another Project to contribute to so I look forward to next year when you will tell us about your next project. Thank you very much