 I would like to start off by welcoming someone who I think most of you are all familiar with to the stage. We've got a great lineup of speakers today, but the first one I think needs very little introduction. He created Linux, he created this thing called Git, he created this great diving application that many of you may be unaware of. Please welcome Linus Torvalds. Thank you. And as usual, I don't know who talks, I hate preparing talks, especially when I don't know what people are expecting or interested in, so blame, Jim, if the questions are not interesting. All right, well, yeah, well, let me just, this is pretty much a puff piece interview here. We're gonna, this is not a hard hitting journalism, but you know, it's funny because you come to a lot of our events and I know that you like scuba diving, so you go diving around a lot of events. So you brought your skis to this event. No, no, I was carrying skis, but they were not actually mine. Oh, okay, so you're not? No, so usually I only go to places where I can do fun stuff, but then occasionally it's Valentine's Day and I go to places where my wife can do fun stuff. I noticed that she's up on the mountain somewhere now. Because someone pulled me aside and said scuba diving, downhill skiing, Jim, what are you doing here with Lena Storvalds here? And what I told them was, apparently, my mic is not working. But it's, apparently it's not skiing or scuba diving that are dangerous to you, it's the constant fear of being hit by a bus. That's what people always ask me about, yeah, I don't know, especially in the U.S. it makes no sense at all to me. It makes more sense on the European circuit, but. Well, we're going to talk about what happens if you get hit by a bus in a minute here, but one of the things that I think shocks a lot of people, particularly people who are new to open source or who are working on projects, is the pace of Linux development. So we asked Gray Crow Hartman for some statistics, and what he roughly described was that 10,000 lines are added to, 8,000 lines are subtracted from, and 1,800 odd lines are modified in Linux every single day, that's sort of the pace of development. Which is amazing, and you know, there's a lot of developers in here who just think about that, the pace of development and just are shocked. I mean, my first question to you is, you've been doing this a long time. How do you come up with a system that allows that kind of development pace to continue? Well, so the kernel is kind of special. It is the biggest project, it's by far the one that also has the biggest amount of code written for it, just the pace thing. And part of it is, we've been doing this for 25 years, and one of the constant issues we've had is people stepping on each other's toes. So for all of that history, what we've done is organize the code, organize the flow of code, organize our just maintainership so that the main pain point, which is people just disagreeing about a piece of code, basically goes away. So the kernel is literally organized at every level so that people can work independently. And it is slightly, I mean, what happens is people see that there's a huge influx of patches and there's a, we have, like, the statistics really are staggering, but then when you look at any particular one single piece of code, the development is not necessarily fast at all. It can take, I mean, some patches going just in a day or two, but it can take months, it can take a year for big reorganizations to happen. So each individual piece may not be, it's not a breakneck speed kind of thing. It's just that we've been able to really modularize the code and modularize the whole development model to the point where we can just do a lot in parallel. And that's what then allows this high flow in the end. But you're not, maybe you are, you're not reading all this code yourself. Like how do you develop the ability to, you know, when you do your final release, to be sure that that's code that's of decent quality and it's... It's a social project. I mean, it's about technology and that technology is what makes people able to agree on issues because technology tends to have very, they're not black and white, but at least there's usually a fairly clear right and wrong. So you can have people agree on the end result, but at the same time, the thing that just me insuring quality, it's not because I read through all the patches. I used to. And it became a huge problem because I ended up being, and this is 20 years ago, it became, I became a bottleneck. So I don't do that, but it's the social network, the trust and finding the people who can do the work and I trust that they may not do the same code reading either, but they have people they trust. And we have a very strong network and that's why we can have 1,000 people involved in every release because of those 1,000 people, a lot of them end up being these levels of trust where they're checking at every level, not of the code, but of, well, a code at the lowest levels, but as you move up the maintainership chain and the chain is often 3, 4, 5 people, you always have that level of trust and the level of just knowing people. Any other people I've worked with, I've literally worked with for over 25 years. There are new people coming in also, but there's this whole human potential there that we use very extensively to make sure the process works. And it's taken 25 years to get to where we are, I mean, on a process level too. And what's interesting is that we did a survey and I think 95% of the developers who work on Linux, particularly in the core project, are all professional developers who get paid by various companies, many of which are represented here, but the trust that you've built is with the individuals in that project and that's taken a lot of time. Yeah, and it's very noticeable too, when people move between companies, the trust does not stay with the company, the trust stays with the person and obviously it's not like people stay around forever. People come and go and that's, I think, one of the reasons people sometimes think it's hard to get into the kernel is because you think you can just come in and shoot off a patch and go away and that's not how it works. What really takes time is to shoot off a lot of small patches until you reach the point where the maintainers trust you and at that point you become something more than just the guy who sends patches, you become part of the whole network of trust. Yeah, it's interesting, we have some projects, Cloud Foundry is an interesting project in that they use a pair of programming technique, which is an interesting technique out of pivotal labs, where developers are programming together, same code, two screens, in order to kind of accelerate that trust and empathy that, you know, in the kernel takes a long time. I mean the kernel is kind of odd and what we do does not necessarily always make sense in other settings, just the fact that we, for example, cannot have people sit next to each other, just means that we have to work differently than you necessarily would do in some other company or maybe there's a lot of projects that are much more localized to than the kernel, even when they're open source and even when they use email or the internet to communicate. Most of the time a lot of projects tend to be inside companies or maybe inside universities or maybe it's just that there's a lot of, like there's the trust came from previously knowing the person, so there's a lot of advantage to being close by and being part of this inner group. And in Linux we never had that and I attribute some of it to me being from Helsinki, Finland, which meant that most of the people by far that I could work with were all remote. I mean I did not have the kind of local group around me that you find from like some of the traditional Linux projects like obviously the BSDs which actually happened inside a particular university. Yeah. And you, so also Linux in order to, the project in order to handle this pace, I love the social construct, but you also needed special tools and so you created Git to do that and now we're in this, you know what some people like to call this post-GitHub era where, you know, yesterday someone from GitHub was here and shouted out that there are 50 million repositories on GitHub. And what advice do you give to, you know, GitHub projects, first of all you're surprised about where Git has gone and then what do you, what advice would you give to projects that are on Git in order to help them build trust relationships and to do this kind of work at scale? So on the surprise part I'm really surprised by where Git has gone because obviously Git was written to be, solve the problems we had in the kernel community and nobody else or it looked for a long time like nobody else cared about the whole distributed environment and distributed development and in fact for the maybe three, four first years of Git, one of the biggest complaints about Git was how it was so different and everybody had to learn something new and people hated it, I mean people who came, who weren't inside the kernel and were trying to learn it after learning CVS or something like that. And then I'm really surprised by how it switched and apparently really people started seeing the whole distributed development model and having companies like GitHub just taking advantage of the distributed development model and making it so easy to then host Git repositories which was never true if you don't have distributed. The problem with centralized models tended to be that you had exactly one point that was the central one which means that it's really hard to trust a new or any hosting company because if you're a hosting company and you host the one centralized repository, you are so critical that it's really hard to even get there and being distributed just meant that now it's much easier to be a hosting company and give services and that value add and what I'm trying to say is it really took a long time and then everything like within a year, it went from having a lot of people complaining about how it was different to just growing like mad and it took me by surprise and I'm very pleased obviously because CVS has been around for way too long and there were a lot of other source control managers. Sounds like it still is out here. Well, I mean it exists and I'm not trying to say CVS was evil 40 years ago. I'm more saying that time had passed it by and no other source control system seemed to ever be able to really take over and Git seems to have done so and that's obviously a source of pride but every time I talk about Git I want to make it very, very clear that I was the maintainer for Git for six months and that's 10 years ago, right? So I came up with the design, I came up the basics and then Junya Hamano has been such an incredible maintainer for 10 years that's always something to keep in mind that all credit goes to him apart from that initial spark. GitHub no credit? No, I do think that the fact that, I mean without GitHub, there's other companies too but GitHub is clearly the 800 pound gorilla, without that kind of infrastructure it wouldn't have gone anywhere. I think the fact that GitHub existed and which there were a few projects that decided hey this model just works very well. All the Ruby projects switched like suddenly and I think that was one of the things that drove Git to be more than just the kernel people and the geeky people who really cared. And GitHub has done an amazing job of, and that's why people call it the sort of post GitHub world of just the phenomenal development that's going on. So you got the tools, you got the trust relationships. What about, one question that I have and then that comes up often for a lot of our other projects is for the kernel it seemed like for the first long period of time you were fast following an incumbent technology and requirements sort of flowed from that. But that has not been the case for a long time. In addition to how do you process so much code, how do you understand what requirements are out there? What features and functionality do you want to integrate in? Well I mean that's the real advantage of open source I think is that if you want to do something new, something that really is pushing the envelope. The only way you can really, I mean nobody can really say this is the way to do it. So what you need to do is just a lot of trial and error and I often compare software development to biological processes where it really is evolution, I mean it is not intelligent design. I'm there in the middle of the thing and I can tell you it is absolutely not intelligent design whatever you want to think. You have to have, I mean you have a lot of people who have their ideas of where things need to go and then you need to have a marketplace where you can try them out. And that's where open source allows all these different groups to just go off, do their own thing and then come back and say hey look I did this and it actually worked. And that's very hard to do in any other environment because if you're inside a company you can't afford to waste those kinds of resources. If you're inside a company you can't say okay let's just explore this whole space and put 10 different teams on it and see what works. That does not work, I mean I suspect a lot of companies have tried it but they probably only tried once. But in open source that's literally what we do, particularly in the kernel. We have competing projects, we have a lot of code that never ever gets merged because somebody went off, they cloned my tree, they did their thing, they could never get it to work or they could kind of get it to work but it only worked for their very particular platform which is fine too, don't get me wrong, it's these niche things that never go anywhere else and that never get merged back because there's no point. But you did at one point the kernel decided to do a long term stable release and then have intermittent releases in between, what brought that decision about? So we've gone through so many different models and they've all had huge problems until we then figured out about 10 years ago the model we're on now which seems to work really well. The traditional model is you take three years to make a new release and you aim for a certain feature. There's usually many different features but there's a few flagship features that you know, that's what you're working on and the reason it takes three years is it takes that long to get big new features done and in the meanwhile, so you have this development tree that all the geeks are excited about and that does the new stuff but it's also slightly unstable so most of the traditional users tend to stay back and that kind of worked but it meant it, we did that for probably five, six years and it caused horrible problems when you then did the switchover because so much had changed. So few people had tested the development kernels because they were considered to be unstable so only the most excited people or the most forward thinking people and there's very few of those were testing development kernel in order to make sure that they worked for their workload so whenever we then finished a stable release, we were always in huge pain and it does not scale, it does not work, it was a disaster but it took us a long time to figure out that it was so bad and then 10 years ago and nobody believed this would work, literally, everybody was like laughing in the room and at the kernel summit we said, I basically said I want to have an eight week release schedule, two months, that's it, right? And everybody knew I was like pushing for something unrealistically short so that maybe you could get it down to six months from three years, right, but you aim for two months in order to hit six months and now we've actually done the release cycle is usually two and a half months and we've done it for 10 years and it's lovely, it really works, it has avoided all the problems we've used to have, it's solved so many things we didn't even know needed to be solved so it's almost boring how well our process works and I say almost boring because it's so much relaxing, more relaxing when process works. All the really stressful times for me personally have been about process, they've never been about code, when code doesn't work that can actually be exciting, that can be fun, that can be the, okay you hit your head on a wall for a couple of weeks and it can be very frustrating but then when you solve it the feeling of accomplishment is huge, right? I'm hoping people in the audience know that, right? Code, technical problems can be challenging and frustrating but they're good things to have. Process problems are a pain in the ass. You never ever want to have process problems and that's why I'm so happy that our process has worked for the last 10 years and hopefully will continue to work because that's when people start getting really angry at each other, when the process doesn't work. Well last question, 10 years is a long time but you've been releasing the Linux kernel for 25 years. It's just an amazing amount of fortitude for anyone to do the same thing for 25 years. I understand you took a six month break it sounds like to do this get thing but it was actually more just two months. You're in a very short break for a minor effort but the last question is sort of you know you got almost 900 organizations that participate in all the different Linux foundation projects here that are creating code in almost every aspect of computing and these aren't just you know fast following projects. These are the core main projects for the automotive sector the telecom sector. What for you was there a moment where you thought ah-ha this is huge this is much bigger than maybe what you thought when you started. Was there any particular thing that you? I've answered that before there hasn't really been anything like that there's been a couple of moments that took me by surprise but the big moment was actually when it went from past being personal like and that was within six months of just releasing it. It was I don't even know the people working on sending me patches like know them personally I mean obviously now I know them by email but there's a ton of people I never met and just going past that initial personal phase where the project was just for me. I do want to kind of mention the whole 25-year thing. I'm a huge believer in just the 99% sweat, 99% perspiration, 1% inspiration thing. Any technology project the innovation that this industry talks about so much is bullshit. Innovation anybody can innovate. Don't do this big think different don't do this big innovation thing screw that it's meaningless like the 99% of it is get the work done right and there are I actually that's my least favorite part of the technology like new cycle is this constant innovation and new ideas and this will revolutionize all that hype is that's not where the real work is. The real work is in the details and I'm obviously one of those people who just I like to concentrate on one project I don't like flitting from one idea to another and you need the people who just flutter about and come up with ideas but they're not the really useful ones they're the ones who they end up being the ones who maybe give ideas to the people who then do the work so I hope you guys that's what anybody should take away from this talk is the people who actually do the work are the ones you should really listen to and these days I don't actually do the work anymore I I merge other people's work but I like conferences too to be less about the visionary innovation thing and more about the day-to-day what are my problems and how do I want to solve them. Well always inspiring and depressing simultaneously Mr. Torvalds so I think what it's fair to say is you know real innovation happens when you kind of get over yourself and then sit down and do the work which is fair enough we I think that's sage advice so thank you for coming it's always a pleasure. Ladies Torvalds