 So Greg we are here at open source leadership summit. Yes. What are you doing here? It's not a Linux, it's a 7-Open-Service. Yeah, it's Linux. 7-Linux is underneath all this stuff. To be fair, it is true. This is not, there's one other developer here, we're like, hey, a developer. But he's representing his company. I'm here to answer questions and talk about, actually how to talk about licensing for the kernel and SPDX and stuff like that. And then there's also a Linux Foundation board meeting later this week. So when you talk about licensing, what do you mean? So the kernel is GPL version 2, right? But the kernel source tree itself has like 10, 12 different licenses in it for different things based on what it is. We recently just, SPDX is a way to identify licenses in one single string, one single line. And ideally, you'd be able to just grab the whole kernel tree and know the license for everything. But it turns out it's more complex. There's six different variants of GPL v2 text we found out. So in doing this research, which GPL v2 do you care about? We have boilerplate text that is wrong in places. There's 700 different ways we save GPL v2 in the files of 25,000 files, right? 700 different ways. So part of the work is to go through and clean up all that. Recently, back in November or December, I turned out 11,000 of our files had no license text at all. So I did one commit to the kernel tree that touched 11,000 files and added the proper... So you have so much power? We talked about this in the kernel summit. We all got together and we're going to use the SPDX. Here's how we write it. And then after that, we're going through and cleaning up all this weird boilerplate and using just a simple, here's the line, here's what the text is. Because some files are dual licensed to the GPL v2 only or GPL v2 or anything else after that. So we just get the right list and we're cleaning it all up. So we're going to talk about licensing. Okay. I think you're working with Kate. So it's Kate and me and Philippe. Yes. Yeah. Well, I did a story about SPD a few months ago. It was a great year. Because people when they think about open source, they think it's open source. We don't have to worry about licensing or compliance, you know, but that's not the case. Right. It's really tricky. And how do you comply even with Apache and the BSD licenses require attribution? Exactly. So the really cool thing is the FSFE came out with a tool called reuse. And you run it on your repo and it sucks all the license out of it and makes a manifest that accurately describes exactly what the source code is for your thing. And so people that build products based on open source can use that tool to properly know how to handle what they're supposed to be doing, what their rules are, what they have to handle. I think the FSFE is totally different from FSFE in the US. Right. They are two different organizations. Yeah. Because I met Matthias and George who is also the founder. He's a very good friend. So we talked about the difference between two organizations. One is like Greenpeace, to beat everybody up and the other one is like more. I don't know. I've known both parties very well. I am a member of FSFE. Yeah. I do support what they do. So I work closely with them. And now I live in Europe. So it's easier to do that. It's even easier. And they're doing some incredible work there. They are. So like even just tools that we use, but then the whole public funds, public source. Yes, yes, yes. But I'm saying that word wrong. Yeah. But yeah, it's something like that. Yeah. That's a great, great initiative. And I know like the government of France has the rule that they have to release everything now. Right. But they produce all their data has to be even open, which is great. So the governments are changing and producing open source code and huge amounts now. And now since you have moved to France, and a lot of, you know, Colonel developers, you know, Leonardo and a lot of people, they are based in Europe. So how does change your working? I mean, you work for me. Yeah, it doesn't like I can live anywhere in the world. It doesn't matter. But there's three of us on the Colonel's security team that have to live in Paris. So we get together for lunch. You have to live in Paris? No, that we happen to all of them. So no, three of us are the 12 Colonel's security people on the team. We all live in Paris. So we get to the other for lunch once a month. Which is kind of funny. It's just really random. So do you talk about Colonel's stuff only or do you really enjoy French fries? We have good food. Good meat and stuff like that. So but no, we, but no, we talk Colonel's security stuff. We are all developers. But I think you're vegetarian, right? I wasn't telling the difference. Yeah, because you, yeah, I know. Yeah, when you can't read the language, you just have to accept sometimes what they give you. When you lived in Belgium because they speak French here, we were never able to order pizza over phone. Yeah. It was never there. So how is the experience living there versus here? Because I feel that there is much more awareness about open source and Linux, you know, all across the board. It is. I mean, so France specifically has the university I work with and Inria. Inria is a French government agency that computer science. And they've been a huge supporter of open source for the, they supported initial GCC development. There's a huge community of open source developers in there. They have the language OCaml, which is all open source. And that's a very French driven language, programming language that the research communities use as well as the finance community. So that's all open source. The universities teach on Linux. And there's a really, really, one reason I went to Paris was there's a really good group of researchers there that do systems development research on offering systems for that are applicable. So it's research that people can use. So Julia Lowell, who made Coconel, the tool that is fixed more kernel bugs than anybody else. And her team have done really good stuff. And they have graduate students and they do other research. And since you're there in France, are you also working with like any government organization or something like that to help them? I have contacts within different agencies in the French government that I have that do use Linux and I meet with them just because I've met these people and provide advice for them. One of them created a tool, a template for the office of the CIO of the French government that allows people to adopt an open source methodology for their company. And so we all created it with advice of lots of different governments. It was done in an open source way that to do group, a whole bunch of different people came together and wrote this template up. And you can take this template and it's been translated French, of course, and use that as a way to have, to control how open source is used. It like gives defaults of, hey, this is the licenses you should use and here's how you deal with community and here's the stuff. It's great template and it's all open. It's part of the to do group website now. So I was involved in that. Okay. So that was really good. Okay, now let's just switch gears from Paris and France to the real. I live there so I do know what it feels like. For the last one or two years, one of the things that has played Linux word has been security. Not just Linux word, but we saw the whole Intel before we go there. When we think about open source, we always think, oh, it's secure. The code is open. We have seen hard bleed and a lot of things happen. So what efforts does the Linux kernel community make? As Linus says, the software is part of, bug is part of software development process. So what steps are you taking to ensure that, okay, bugs will be there, but to detecting them early or to kind of mitigate them before they actually are released and they find the way in the system. Number one. So yeah, there's two things. There's one is a bug is a bug. We don't know if a bug is a security bug or not. There's a famous bug that I fixed. And then three years later, Red Hat realizes a security hole. But the other thing is the security, the kernel community realizes that we need to have mitigations. So hardening. We need to have hardening. So there's been a huge effort by Case Cook and others to take hardening features that have been traditionally outside the kernel and get them merged or adapt them for the kernel. So that when the bugs happen, because they will always happen, you can't do anything about it. It didn't hurt you. So like we have a simple one of, when you increment a number too much, it would overflow. So the reference count and you could leak memory that way. And now it won't stop. So things like that, we'll do things like that. Some stuff with protected memory and some other stuff. So that when the bug happens, the traditional old one was if you referenced zero as a pointer. Traditionally that used to be a security hole because you could access that. Many years ago we made it so we changed the way the kernel was laid out so that if that happened, it was just a bug and you couldn't do anything about it. It wouldn't hurt you. So in every kernel that gets released, Case Cook does a little summary of all the new hardening features that are there. That being said, you have to enable them to take advantage of it. Exactly. So I've seen, I've gone through and audited a whole bunch of Android phones recently. And some recent security bugs that we had like that would, if you had enabled the hardening features, wouldn't have even tripped it. There's a Bluetooth bug recently that it was a buffer overflow. If you enabled the buffer overflow checking and stuff that we have in the kernel, it wasn't mitigated so all your pixel phones were fine. They didn't even care about the bug. But other phones that just didn't turn that kernel configuration on were vulnerable. So it's part of getting into the kernel and letting people know that they need to enable the stuff. Yeah, that is the second part of the point was that even if you fix things, as you gave example of Red Hat, and we have seen and we also talked about earlier also that the system should be designed in a way that whatever changes or patches you're making, they should be actually applied. Yes, they should be applied and they should be used. So we release the stable kernels every week. I do a new stable kernel. Google announced that like one of the stable kernels 4.4, they wanted to support for six years. And they're doing that in a way because they want device manufacturers to take advantage of this. And then what I did is I went out and bought all the top of the line phones. They're based on the 4.4 kernel and see which one's actually updated. And so far only one company has actually updated their kernel. So the fact that we're releasing these kernels that are stable and update and have these bug fixes as well as so there's the carrot and the stick, they have security fixes and tons of bug fixes but they have some improvements. Like we sped up out of memory handling, we sped up a whole bunch of stuff. So you get benefits by taking these updates as well. These manufacturers haven't taken them yet. They're not updated. And so I'm working through the whole supply chain of trying to solve that problem because it's a tough problem. There's many different groups involved, the SOC, the manufacturer, the carrier. But the point is it has to push that kernel that we create out to people so they actually use it. But for the long term, you can't keep keeping up because there are so many bottlenecks as you gave exam. But why not? So today your Android phone gets a security update a month. That would be wonderful. Just update the kernel then. Yeah, but who's going to do that? Yeah, well, Google today already releases security updates once a month. But our Samsung, they... But then you got to push it. I mean, so there is a large ecosystem, right? So if you control the whole supply chain, it would be wonderful. But the way our ecosystem works, it doesn't... If we just keep these in consumer space, we talk about the server space where we have the CoreOS, which Red Hat has acquired. They came up with a model where it's atomic patches in the updates. There is a lot of... Yeah, that came from Android, actually. Yeah, that came from Android. But yes, yeah. They're constantly keeping up to date and update. And if you look even the enterprise issues, Red Hat and SUSE, even their big enterprise kernels do get updated all the time, too. But you just have to update them. Update them. But with modern systems, if you have lots of containers and pods and virtualizations, that makes it even easier because you're not relying on your 99... Your 9s of stability for your one machine. You can reboot this one and reboot this one and update it at the same time. And with that, it's actually made things better, easier to be secure than it used to be. Yeah, but now IoT devices are there, Google Home and Alexa. Yeah. I heard a story on verge that the author complained that, you know, I've only listened to music and there was an update on Alexa. They stopped my music and updated. I think it's a good thing that they stopped the music and updated because if you do a security patch, I won't mind, you know, my music getting stopped if the company is, you know, proactive and you know... Yeah, or do it at night or... I mean, there's... Except if you're doing a Tesla, you don't, you know, you certainly won't take over the system. Right, so like my TV, you know, it tells me, okay, I want to do this update. Is it okay to do it now or are we going to do it later tonight? It doesn't. It doesn't, yeah. And what about the hard bleed and, you know, Meltdown and Spectra? Meltdown and Spectra. It's a long day. Yeah. Seven interviews, so... Yeah, Meltdown and Spectra. First off, in the severity level of things, very minor. All you can do really with hard bleed... No, see, I did. Meltdown and Spectra is you can read data that you shouldn't be able to read from somebody else. You can't modify it, you can't change what they're doing, but you can get some information out of it. That being said, there's a really... Intel approached this problem that Google reported in a very different way. They approached it as a hardware bug because originally it was reported that, hey, the CPU is doing something really weird. So their hardware division was in charge of managing this. And then the software side got involved very late in the process. And then the way that Intel managed it, as everybody knows, was not really that good. To be fair, the Linux, by the time the leak happened a week early, Linux was fixed for Meltdown. We fixed it on time. Spectra was another issue. We got access to Spectra, that fixes and know about it about... We thought one week before we had to release, actually it was one day because it got earlier. Now the latest kernel versions are all fixed. So we fixed it all. The way Intel handled it was hard for the other source operating systems. The BSDs were not notified. Right. The OpenSlares was not notified. And that's that. And Intel is... I talked to them. I just had a phone call with them again today. They're reworking on how they approach security bugs and how they work with the community because they know they did it wrong. So that. So we do have a well-in-place process to do this. Security at kernel.org. You email us. Here's the bug. And we fix it. And we do this all the time. We do that. I fixed... We fixed security bugs that you could arguably say was more important like two weeks ago than what Meltdown Spectra was. Nobody noticed. We pushed it out. Everybody got it. Everything was happy. Just don't know about it. There's not... There's not advertising behind it, right? Right, right, right. That to be fair, Meltdown Spectra was a very unique thing that we fix bugs in hardware all the time. That's what a kernel does. But this just happened to be a piece of hardware that everybody has. Oh, yeah, yeah. And also it does... I mean, the mod... The bug was that it was an assumption that we thought the CPU worked this way. And it turns out it wasn't. So it kind of... Like the formal methods people are really interested because all of a sudden their model of what a CPU can and cannot do is wrong. So all their formal proofs for how things are supposed to work flew out the window because it turned out there's a side channel because they predict in the future and you can actually detect the prediction. So it was kind of a little side note that those people who formally approve programs are having to revisit everything. Doesn't it only need a new architecture? Or that's totally... No, I mean, so Meltdown was... Yeah, Meltdown Spectra, that's probably easy. Spectra, it's just the way CPUs were designed that in order to go faster you have to look farther ahead in the future. And so in some rare cases when you can modify what you're testing so it's very simple. If this happens, then do that. So that do that was actually executed before the if happened. And so if you can control what that if is testing you can sometimes see what that is. So you're looking... And then if you're looking from the side you can see that. So in the kernel or what you're doing is you're crossing a security boundary. So me as a process I can see what another process is doing or me as a virtual machine I can see what another virtual machine is doing if I can control data out there. And you can. So there's been proof of executions of how to do this or your browser. Your browser, like you run in secure you run on your run JavaScript and you don't want another tab to see what that tab is. So the browsers are fixed for it now. The kernels fixed for almost all the big spectrum issues it's going to be a long tail of minor things where we just we're sweeping the whole kernel of all. Are this data that comes from the user or that the user can modify and do we need to protect from that? And we have tools that are testing for this thing in static analysis but the false positive rate is like hugely high so we have to dig through it by hand. But all the big issues for x86 and ARM are fixed. Other CPUs are more interesting. So when you talk about all this security I mean you're actually becoming like a security expert. Are there like I mean you have some big maintainers there so are there like some maintainer dedicated to doing this or it's like whether it's Linus or you or Andrew or whoever everybody's doing everything. So we have a security team, right? Our security so like the kernel security team is just a bunch of people who know the core of the kernel when we get a report we then drag in the domain ownership. So say there's a bug in the sound subsystem. We say hey Takashi there's a bug over here so we grab him and pulled in then we get the local ownership. Sometimes on the security team we had some areas of the kernel that were always grabbing the same people so we made them just part of the security team to make it save that extra hop. So it turns out a lot of the core kernel people are on the security team we just fix the bug and everything's fine. But I mean all parts of the kernel have to be aware of these security issues because we are a trusted environment and you have to protect that. So these are just we have things in place the once we fix things we can put them in our static analysis rules and that way we never get it reintroduced. So we do things like that. But and then on the other side is like Case Cook and his work are adding hardening. So that's a other security type of thing and they're adding the features to do hardening and other bugs like that. Fix other things like that. As Linus said in the previous you know that we want the hackers to join us instead of attacking us. So how to kind of you know either encourage or incentivize people so that instead of kind of even if there are no such attacks or we have seen people are exploiting it how to encourage people to actually become part of the security community where they help the kernel community instead of. Well we always need help right. And so we have if you want to work on the kernel you can usually get a job doing that. On the flip side there's some companies that offer bounties for notifications of security bugs. I know Google does that. So we get a lot of reports from Google where someone's reported a bug to them and then we look and see and they think it's a security issue we fix it or not. So we get that a lot. So companies are willing to pay for that. Mm-hmm. So that there's other projects we have that so co-variety is a static analysis tool. For years they've been running on the kernel but no he's been fixing all the bugs he finds. So the common infrastructure initiative for Linus Foundation funded a kernel developer for a year to go through and just fix all those bugs. And he's doing an awesome job and he's got a job for you right. So to do all this kind of stuff and work through and fix all these bugs actually I need to get him to do some of the spectrophixes to think about it. So yeah so we have if you want a job to do this kind of work we have plenty of companies any company would love to hire you. I know some people who have started off fixing bugs and then can hire my companies just to do other development and things like that. Anything else you would like to talk about? Well I think about some you know I mean I can talk to you for hours. Yeah sure. We have a lot of questions. I don't know anything specifics. Yeah that's been consuming my life is getting all the updates to all the back ported kernels for that. But just yeah make sure you update your kernel. Take the update. Yeah if you make a device make sure it's always updateable. Right exactly yeah. So there's some interesting things with kernels wanting to be supported for a very long time because they're going into infrastructure. Like the Japan is putting them in streetlights. Yes you know. So that's a fun one. So making sure those kernels are stayed up to date and that we can test them properly in order to be able to push them out safely. I've been working on that. I'm talking a lot to chip manufacturers that way. Are you bored of this work? Are you? It's always something different. It's always something different right? And I still do the normal development and as far as review of the stuff that comes in. But no it's not important. I haven't got bored yet. It changes all the time. There's always new hardware. There's always new. I mean one really really neat thing about the Google thing with Spectre and Harplit is it opened up a whole new research into these type of side channel attacks. That was a brand new... It was a building on the prior art that the people have found but it was interesting in that it was hey there's something brand new here. But I said you know you're bored what I meant but you have been doing this for so long. I have. Yeah. And but the thing is that excitement is far from over. You know the new use cases are coming up. IoT is coming and so much is happening you know and everybody's using Linux for you know. Even Microsoft Azure is you know there's... Yeah Azure is ton of Linux. Yeah they're working with switches you know. It runs on Linux. So a lot of more work. But if you look at the core kernel community I mean I'm going back to the question that people keep asking is still the same people you know. You Linux and all those people. So how do you attract new people? Oh we have new people all the time. So every kernel release we have what? 150 to 200 new people every three months? Mm-hmm. That's a huge number right of people and we have what 2,500 people release. I don't remember the exact numbers. Mm-hmm. But so we still have a huge number of new people every year or every release are attracted and there's a side thing. So is when people ask we're getting older right new people. Is that a bad thing that people with experience and age are still working on a project because they still know what's going on? Mm-hmm. Or no I think that's really good. I mean look at the networking team they have been doing networking for 20 years right. That's a huge body of knowledge and we want those people. Now being said we have new people coming in to show their ideas or we need the help and we get new people in all the time. We have tons of work to do. We have tons of areas of the kernel that don't really have a maintainer. Um and we still have we always need the help so and we're still attracting them as long as we keep attracting new people. Uh with all the security things coming up and the explosion you know how much what is your big concern when you look tomorrow that oh you're still very oh that we should this is something we should be. What do I worry about? Uh-huh. We're about legal stuff unfortunately. What do you mean by that? Legal so we've had the copyright troll problem. Oh yes. So um which the latest thing just happened today in Germany that the um the current person who is doing that um withdrew his appeal. So he gave up. Honestly this one which was interesting. Um I helped out actually with that defense a little bit. And I put in uh it's not a friend of the court. I put in a little deposition saying here's how Linux is developed and things like that questions about that because what he was what he was proposing was not really the way Linux is developed. So there's legal threats that are um and trolls from outside our community are always my big threats right. What is um what that can hurt us and I always do joke Linux is so successful and is everywhere and so the only thing that can hurt Linux is Linux itself. Yeah I yeah you understand yeah. So what does make me worry is if we mess up if we if our development model crashes if we do something stupid and we push people away if we're not accepting still new developers if we our rate of change can go down it's fine our rate of change is so high but if we're doing stupid things that we're um driving people away from Linux because it's our fault and that's what I worry about and so every year at the kernel summit we re-evaluate what we do and we talk about our process and one of the things was like oh we need to do licensing in a better way because lawyers are reviewing this stuff this makes their life easier and they can make sure they're complying with our license easier great here's one thing that can help them out um new developers we look at that all the time um yeah so as long as we're still doing things well that's what we're doing and the external legal threats. Right I mean this could be off-record question I'll just ask based on your opinion then you said you know not to turn people away and you know uh sometime on uh mainly listen to Linux you know he has you know his own way of saying things but what people I'm not defending him or you know or but what people forget is that uh that treatment is reserved only for the high level maintainer you know the top maintainer that he trusts you know fully you know it's more like you're sitting in a boardroom with let's say Donald Trump you know and there are three advice the top advisor you know so there the language can be anyway it can be so I don't really know how much you want to comment on it or not but what I'm saying is that I see people misinterpreting because he I mean I've met him with the new people he like very friendly he explains things uh so do you want to comment on that or not so actually like this came up at lunch today with a group of people um it's um it's really hard to get Linux riled up so in the kernel development in the kernel community we have one rule and the one rule is we cannot break user space we made that guarantee over a decade ago called the Cambridge rule because we happen to be in Cambridge at the time um so Linus yells at you when you break that rule on purpose if you break that rule accidentally we're all human we fix it we move on or we break it and we can't figure out how to fix it we fix it and move on or we do something but Linus will yell at you when you're obstinate and you're saying I'm breaking this and I'm okay with breaking this and that's not okay um and it's a high it's a high bar to push him to yell at you I mean but some people are I mean developers are not the most um feedback friendly people that that feedback loop was not always there and email is a very tough medium in which to convey tone and expression something and sometimes you have to be very direct but if you are trying if I do something stupid and I break something and I know I'm breaking something and I'm insisting I'm allowing it to break Linus will yell at me for rightfully so and that's okay and it's always reported might say because fight I mean conflict is easy to report on and people are reading about it exactly it's more juice there it's juicy and like always said you should see emails for internal companies right I've been uh developers it's just totally separate I've been in meetings with some companies where people cry right some major companies used to have conflict as their major source of how they would resolve problems conflict with their way to resolve so I mean that and but so that's not a good thing at all but um I'm saying um what we do everything we do is in the open and tens of thousands of emails the problem is that's everything is an open so your frustration is also open so it's an open but and also I mean if I do something stupid or Mara does something stupid we know each other we exactly and it's friends so it's like I also used to term um you don't put a live microphone on a major league baseball or a major league football manager because he's yelling at his players right it's professional oh yes I know I messed up you're right because he told me that and I can take that for the what it's meant to be um yeah but we're doing in public right and it's not the little league you don't yell at the little league yeah I get it that way since we do have some extra time I want to because I talked to you earlier but that was not on the video format how did you start your Linux journey I've told you this before yeah but it was not on that's fine so I used Linux many many years ago I come from the embedded world so I used um Linux in a product we made where we were using sco Linux and running Oracle um Oracle came out on Linux uh we're like let's try and install it in a random wonderful so I'd use Linux for a while and then um I was working for a company um that we made barcode scanner and then um I was trying to test out my device with USB on all sorts of different operating systems um I accidentally made a barcode scanner that showed up as 102 key mouse so 102 key keyboard uh Linux didn't like that and Linux didn't have a working USB stack at the time so I was I saw this code I was like hey I can contribute and I fix some bugs and then it was fun and then um I've been messing around with a while and then my wife I was going away taking our daughter to visit a friend and she said hey why don't you write why don't you play with Linux this weekend and write that driver you've been talking about so I had this little weird USB to serial device USB to serial converter because I dealt with serial all the time and I wrote a driver for it and I submitted it and I wrote it and then um I learned from the Linux device drivers book which I ended up being a co-author on many years ago later um I submitted it and then instantly I swear within five minutes somebody wrote back oh this is wrong this is wrong this is wrong this is wrong this is wrong but I was happy I was happy I was like yes so I mean as a as somebody who is a programmer I know I'm always going to have to keep learning right then as you have to keep learning in order to get better and constant state of change you always have to keep learning and changing in order to adapt to the world so I was like yes I didn't know about s and p multiple processors what's locking and and what's this and oh yeah I guess that is totally wrong but that review process of I always like to code reviews and because you can get better and so I didn't take it you have to learn how to get rid of your ego and I can say I didn't have that ego and criticizing my code was great and I learned from that and I did better and I wrote that and that's fun and I had fun doing that and I wrote drivers and I wrote drivers for USB for a while and then I realized more people use the stuff I gave away for free the company I work for because the company I work for wasn't all that successful at the time and then I got a job doing Linux actually Linux security stuff so I started off in security oh okay and I helped write the first Linux security layer many years later but a hot plug and stuff like that so I've been involved in security for a long long time yeah so and how much influence do you have on your own kids are they all software developers or no my daughter's um no she has a degree in public health and she wants to be a doctor so she's applying to medical schools my son he knows how to program we learned the basics um he's now 13 but interesting thing is um you learned like learned the code and um code at the code academy you learn the basics and scratch and things like that and then after you can pass the basics there's no nothing to do exactly so it's like what do you and the same thing with the kernel you have these basic tasks but there's no intermediate tasks and finding a project on which somebody to work on it's hard because when there's minecraft you can build things and go away and that's a good distraction right I didn't have minecraft when I was learning how to program um so there's so many other distractions that I coming up with a solid program to make that hurdle to the intermediate stage is hard and I haven't I haven't figured out that problem so he learned the basics and that's good and that's everybody should learn the basics of programming just for whatever job you have what do you have what kind of hobbies do you have that kernel I used to um when I lived in the US I built a kayak wouldn't yeah yeah living in France I don't have a wooden kayak anymore so I really don't have any hobbies cooking travel we travel a lot more and see different parts of Europe it's fun right and what for the reason why you moved to France so I'd work with the university originally um in India they um did that I had gone over for a three month stay and um what's the family and we realized that and we wanted to move anyway and so we thought we'd try it a year and um because I can work anywhere because it doesn't matter and actually being in Europe worked out really well because I've done a lot of work with some German companies and things like that and going and visiting them is much easier right than flying across the pond and I've gotten to visit arm in Cambridge which is really easy and then we stayed a year and we renewed our visa we're going to stay longer yeah when you mentioned the German company you had been to Suze and Tumble Read was one of the project that you started and now it has taken a totally different shape where it's playing a very critical role in the whole Suze strategy yeah I'm really happy to see that so what do you think about it I love so I always I love that model that that continuing rolling update of model is a valid I always thought was a valid model for what a lot of people need it because you see people especially on doing the web stack and now containers are still that's the model you want to update that and you want to run the latest version of Ruby or latest version of PHP and um constantly update versus the old enterprise model where you backport things and you update on the two-year cycle and whatnot which works for some environments and the old traditional Unix model and the old traditional Windows model was that way I think the world changes faster and you want to do different things so I always like the rolling release that's my opinion I fully agree I have a rolling release so I mean our bodies are rolling really easy right yeah and also it's harder I've seen companies upgrade from major W and release this or to that and it's all this this harshness all the time we're like oh I have to figure all these changes in this happen versus if I have a change a week I can roll with that and I can figure it out and move on and then do it yeah it's much easier I think to do that and now with containers and Kubernetes and managing these things that makes it much easier to push out updates and keep things more up to date in a fashion that way I like that a lot better because I also use Arch Linux and but I prefer yeah Suze is a tumbleweed because what Richard Brown said that it's tested you know so that's kind of contrary to what rolling release is because they no you can always do a rolling release and test it yeah that's why he said you know it's well tested you know rolling release versus you know whatever code is coming just three out so I love it yeah no I mean it's always like I say you should always update your kernel to all your devices but test it always test it yeah because I test it in what we can test things for which are our tests are getting better and better Lenaro's doing actually an awesome job with testing stable releases on real hardware and putting a real load on them or just build and boot exactly so they're doing a great job with that so test for your environment and because you're always going to use it in a different way and then you can push it out but test and if you can't test then you have a problem no matter what if you if you if you update this year or three years later so five years later yeah so you need to build your systems in a way that you can test and and the kernel that's been an interesting thing is it's implicit that when we ever add new features like new system calls one that we add tests because we don't know if the feature you're adding is right or not so and like some of these backports when we're backporting these big security features backwards I'm kind of like pushing back and say how do you test this and they're like well we didn't test it but it built I'm like okay we need to actually test this to make sure that it did work and to validate that so it's test and validate and so we're updating LTP which is the old Linux test project for years was stagnant and it would have a bad reputation in that there's a lot of false positives in it so it was always is the test wrong or is the kernel wrong and most of the time it was a test a lot of people are now finally that's been revived and they're pushing patches to it and they're updating it for fix all the false positives and they're adding new features to it so they're going to test more and more of Linux and now it's running in Linaros testing and it's really good and Google's doing some testing in that area so we want to test all these updates because the kernel does move very fast so we want to make sure we are at least not breaking anything so that's added to our test infrastructure and the zero-day bot from Intel tests every commit that we make to the kernel and it's running all those tests on those as well they have a huge list of tests that they run on every tree that and everything that before it even gets salinus and it has to pass this before it gets there and testing is very, very good Well, we were talking about what are you running? Are you running Arch? Arch, you're a fellow Arch user I actually wrote Arch tutorial it's become so popular because people want to try it and because once I went to Arch it was just my heart for me to though my Plex server is almost always broken because I use Arch user repository or whatever you say it and but it's fun, you know Yeah, Arch is good I mean I've resisted really hard I don't want to be a developer yet another distro it's nice just to be a consumer of a distro that is always up to date I have had problems where Arch updated its compiler and the kernel wasn't I couldn't build things so I had to make sure the kernel could do that but that was fun Yeah That was my fault Because as a journalist and as a user also I like to stay on you know latest packages or whatever is there so it helps me I don't have to wait for a distribution to for two I don't understand why they have to wait when the project is open source your distribution is open source why can't you work in advance you know and you know test and do everything at the early stages Yeah, if you look at the Debian release model it's very old traditional release software release model and again it works so one interesting thing about Spectra and Meltdown I want to say Herpley was that the original Intel approached the enterprise distros and so Red Hat and Susan Knotical were all notified before the community and the problem with and then the problem with that was the majority of the world does not run those kernel Exactly So the majority of the world turns out runs Debian and it runs their own kernel from kernel.org So like all the cloud hosting providers and all that stuff like my cloud hosting provider runs just the kernel.org kernel and it's awesome So that was the big problem with Meltdown Spectra is the one third of the world's kernels were safe but the rest of the the community was not notified in time so we couldn't get stuff out to the other people so one thing that education wise is dealing with the old model of companies only dealing with other companies that's not necessarily true you have to talk to the community because the world runs on stock kernel.org stuff or Debian actually Debian is everywhere So that old model of the slow release for Debian does work and they're really good at backboarding stables to our patches and they go through a ton of work I don't like it because so that model but you can always run Debian waiting edge but it does work for a huge huge computing model and people are used to so it's fine and it's kept up to date and the Debian guys do wonderful security stuff so everything they're doing is kept up to date and it is rolling with the security effects but when you upgrade to a bigger release it's kind of hard then because of that yeah even I have not updated my Lenovo server to the latest because I was worried that something will break because I'm running couple of sites there with a lot of customizations so I didn't bother you know they still supported so when it will be out of that end of cycle then I'll update it so I wish I ran arsenic there so I run there arts there yeah I I mean I mean because now I'm like too busy with other things that I don't have time I will like at one point I'm like if I was running arsenic I'll just open SSH update it and done I don't have to worry about anything so you know the cattle versus the pet model so I think you have the same problem so my server instances are my pets because they're carefully too going off but if I had laptop disappears today it's the pet or it's cattle and I can back it up from from my server everything is on it nothing remains here either way everything is backed up everything's secure and if it's gone it's fine it breaks I move on to the next one and an hour of my actually less than an hour my new laptop's up and running fine so yeah so I need to make that so that my servers are treated that way and actually Kubernetes and the way you handle containers I've been starting to push some things to that and it makes that much easier so that model is much much powerful oh yeah I have a big build server for the my testing for the next kernel and it's now it's too I'm over using it it's so my builds are going slower because I'm doing so many builds so I think I need to just split it out and start using the cloud more and make it so what kind of machine are you using can you give us no I don't know it's some big it's in the it's in the data center okay so I bought me a big giant I got a big giant Xeon build server so do you go and sit inside just like science fiction movies sit in a big data no no no it's like when I do when I do a commit to the stable tree I how it commits and it pushes out and it fires off a test bill and I get a report from email depending on the load of it within less than five minutes but how secure is that server or data center who's it's just I'm building publicly available oh okay I'm just building it it's just a build you know what I mean and it's secure yeah how about Linus especially what kind of machine he uses he also uses the same yes I don't that's just my test builds make sure I didn't break anything so yeah I've committed everything to a public tree anyway okay so it's like I run test builds that make sure that I didn't break the build didn't break any warnings and things like that's all it is it's just a test load I just I could run it on I need to run on zillion cloud instances and go faster yeah so I run on my laptop mostly but I don't have any secrets right I got no secrets on my laptop I don't know yeah no I mean I mean I mean when you're doing open open you're doing open source the patches the changes that are coming to us are coming in public publicly yeah so with the exception of the tiny the tiny subset of security patches which only we only keep secret for a week anyway before we push them out so there's nothing is like secret there my wife's always saying the Russians are going to get into your laptop well it's fine it's like just there's a movie you know and we named K's laptop in the chain on the head it's all secured with that it's the next laptop no no we have we have two factor authentication so that when Linus and I push to our public repose that is two factor authentication so we do that it's so that's secure we encrypt our laptop drives and things like that so we have best practice actually so Constantine the systeminforerco.org has published a wonderful resource on how to do a secure Linux machine a developer machine and given it all to kernel developers it's public the systemin guide for how to do a secure systemin system because all the all the Linux foundation systemins have to do this and now he did a guide on how to do good GPG key handling okay and how to how to rotate them properly and secure them properly and what the good methods of care and feeding of your GPG key because we use GPG for when we do releases we sign them push them out that way so he's been publicly making that available to everybody and that's a really good resource what was this back back to the adopted after that one thing happened with the LKML the next archive was kind of compromised I think it happens so LKML.org is just a random website that gets an email archive of the kernel so that is nothing to do with our development our mail server is fine so that was just a think of it as a copy of all our public email it was just a mail server so um it's just a site you could search so we don't but there was something I because right now it's a long day I've forgotten there was something that was compromised then you guys had to rebuild everything or oh and kernel.org wrote down many many years ago yes yeah kernel.org got compromised many years ago so what is best practice about adopted after that oh yeah we have oh yeah so we have redesigned how kernel.org works amazingly and he gave a talk on a couple years ago with the kernel recipes conference in France on how kernel.org is designed and like when Linus does a I want to push a release so it's all publicly documented how we do everything and there used to be a Raspberry Pi involved in it too yeah but the Raspberry Pi did some of the the key signing because when we do the releases it would it would that because I was a secure little almost air gap system I was told that the Raspberry Pi is not in use anymore so I don't know what replaced that but it was something else but there's the steps that are involved with our two factor authentication system and when that and Konstantin comes from the Fedora community he was the system minister of Fedora and he's amazing and the other kernel.org systems are great so they built a system that is we documented how it all works and it's really really secure and like he says he wants to be able to make it if something happens to him or we need to pull we needed to revoke his access we can do it like that and it's still secure it's still here so we it's built and it's publicly it's public the design and implementation of all that is public as well so other people can be able to use it for their security. Anything else I think we covered a lot of topic today yeah it's fun talking it's fun talking to you all the time okay that was a wide range it was yeah which government from your your whole channel to your kids to friends to yes yeah from kernel to security to I enjoyed yeah we covered a lot thank you so much it was probably yeah I'm stuck it's always fun to talk to you let's go