 And I'd actually like to start by simply asking our panelists to introduce themselves. Then I have a bunch of questions. If anybody in the audience has a question, you can just raise your hand, and they'll bring a notepad to you, and write them down. And then by some mechanism, they'll find their way up here, and some of them we can use up here. So if you'd like to start, Greg? Sure. I'm Greg Crow Hartman, work for the Linux Foundation. I do drivers, devices, USB maintainer. I also do the stable tree, the stable kernel leases, and as well as the staging tree, which is the most bizarre crazy cause of the two extremes in the kernel. OK. I'm Will Deacon. I work for ARM. I primarily work on the low-level architecture stuff. So I'm really into memory ordering, and the really low-level TLB maintenance, or that kind of stuff that most people hate. I co-maintain the ARM64 ports, and I also maintain some of the Perf back ends, and the IOMMU driver for ARM. I'm Sebastian Hessebad. I'm a hobbyist attendant here at the discussion. I started working mainline Linux about a year ago, and yeah. OK. I'm Peter Zelschra. I am the scheduler maintainer, the Perf maintainer, lock depth maintainer, all co-maintained with Ingo. I work on the memory subsystem. I do IRQs. I do timers. I do architecture low-level stuff for various architectures whenever I have to. I get around a bit. I don't know. I work for Rahab. OK. Well, the program is a recall that the subtitle on this session was embedded in the core or something like that. So I'd like to start by talking about where the two of those come together. If you look at the contributions to the kernel, you see a huge amount coming now from the mobile, the embedded world, mostly in the ARM sub-tree, and so on. If you look at who's contributing to the core kernel, like the scheduler, stuff under the kernel, and MM directories, what you see are companies like Red Hat, like IBM, like so on. You don't see much yet from that community in the core kernel. So I'm curious about what we think about how those two communities are coming together, especially now that the embedded world is driving changes towards the core kernel. Are we able to work together and make one kernel that works for everybody? Sure. It's a fun exercise. I mean, scaling from the tiny little machines to the telecom graphics type systems out there. Well, Peter, I'll ask you this because I have heard frustrations from people on the ARM side saying they're having a hard time getting scheduler changes even looked at, much less merged. Ah. So what's going on there? So what I want to avoid is having a scheduler for every architecture out there. And in particular, we're talking about power aware scheduling, which is pretty high on the list for the mobile devices, which are battery-driven. Then there's the various architectures that are interested in, predominantly Intel and ARM are pushing this. And I'm missing a general overview. I mean, we've got the various subsystems that do do power, like CPU idle, CPU frequency, and the scheduler. And all of them must work together to manage the power planes of the actual CPU and how you distribute tasks on that. And I would like a coherent approach, a coherent framework for all of that, and not just random tweaks here and there. And it's been rough, but I think it should be possible. And we've got a mini-summit on this tomorrow somewhere. I think I'd add that it's not just mom, and it's not just mobile as well. They're interested in the power aware scheduling. Certainly there's a big representation there. But there are people from IBM, people from Intel who are really in that discussion. Yeah, there are. They also want this stuff, perhaps the data center. But for the data center, it's difficult to get a straight answer on what they actually need. At one point, they say, well, just get the numabits right, packing tasks doesn't really matter all that much. At the other hand, they say, well, we need to manage idle times. But because they've got shared, they cannot power down the memory buses until all cores are down. It makes things a whole lot more difficult. So it's interesting in that you get the various hardware vendors together, and they don't really want to talk to each other. They're also not used to talking to each other, right? You hide away in your arch directory because you know that it's familiar faces. You can probably move co-quicker in there. And maybe you can solve half the problem there. I mean, that's why companies start off doing drivers because they're self-contained and easy. They don't have to worry about. And then once you start realizing you have other issues that are cross-company and whatnot, you move into the core. And it's harder to change the core because everybody relies on it. So it kind of should be hard. And it's just that migration over time. If you saw Intel used to not contribute to the core kernel at all, it's taken them a number of years. Same thing with IBM. It just takes time. The companies realize that they can. They can evolve as a community. They realize how to work together and stuff. So it's the migration we've seen with all companies for the past 20 years. And it doesn't help that I've been out for a few months last year. But that couldn't be helped. So one of the interesting things the ARM side is driving is this whole big dot little architecture where all of a sudden we do not have symmetric multiprocessing anymore. And as far as I can tell, our scheduler wasn't really designed for asymmetric. And heterogeneous multiprocessing. Are we going to be able to incorporate that? I think so. Because at some point it's not really all that different from a symmetric system where you've got an asymmetric load due to, for example, real-time tasks or interrupt load. We already subtract time available to regular tasks that's taken up by real-time or interrupt load. And due to that, you get a skewed fairness perspective on the regular tasks as well. The big little system is, of course, different than that. But I think there's enough similarities to reuse a lot of that work. The one weird thing about big little is everybody focused on the speed and the power that the CPUs were taking up. But it turns out if you're running a small CPU that switches to little, you still have to have your memory running and your bus running and everything else. And those actually take up more power than your CPUs do. So I don't think it's a model that hopefully we'll see in the future. I don't think, I think it's an experiment that's failing. Just from the power, physical layer, the electrical guys are saying. Well, I don't know. Yeah, I don't think. No, I've not heard that either. So we can take it afterwards. Afterwards we can have that discussion. Actually, I'm an electrical engineer. I know and memory and buses are way beyond CPU power. Yeah. They take more, right? Yeah, of course, yeah. It depends how much you're using the interconnector But you're always using it. You kind of put the memory in self-refresh if you're actually running something. Right. What's really eating up your power is IOPATs and stuff, external world. Exactly. So Will, do we have proof that big, that little is worth it then? Probably, I mean, we'll see what people build. People certainly are taping out with these systems. They're putting them into products. They wouldn't do that if they didn't think it worked, so. Right, yeah, it's true. Okay, more in general, I guess, especially for the more armor-oriented folks, there have been certainly a lot of complaints in past years, mostly from far past years. The kernel has really developed for the needs of the enterprise computing world. Do you believe that's true, and if so, are there pain points that that's causing for you, or is this open to other uses as well? So I think that it probably has come largely from that background, and I think a lot of the initial decisions and perhaps initial maintainers, all the maintainers that are there now, were put in place by enterprise. But I personally haven't had too much issue with that. But again, I work largely under RCHARM, RCHARM 64, and drivers. Bugfix is mainly if I'm doing MMR kernel. I think that Linux is the main selling point of Linux is that it works everywhere, so it runs on all these devices. And if it's running more on mobile, and it is running more and more and more on mobile, then perhaps that'll start to get some more swing into those areas. I mean, you have said that over the time, I mean, what we have in our pockets now really was the enterprise system of five or ten years ago anyway. So all that work those enterprise companies did is totally what Embedded is now. All these phones you're carrying are far more powerful than the i386 Linux ever started with. Yeah, so that IBM's NUMA and S&P work is paid off for ARM. And I also say it's rare that you get put in a situation where you have choice X, which is make the core code really good for Embedded, and the opposite choice is make it really good for enterprise. That doesn't tend to happen. Normally, you air your concerns, and there's a compromise sort, and actually it tends to work pretty well, I think. I guess related to that, I recently heard Tim Bird here, a complaint that we're no longer interested in really small systems, because even our really small systems are fairly big now. And that as a result, we're perhaps going to miss out on opportunities and the whole Internet of Things and all that sort of stuff. Are we getting too big? Have we lost interest in really small? I don't think so. You still see the occasional bloatometer patch come by. Boris Love showed me one only today, with somebody stripping out X86 features, shrinking the kernel. There were some people recently suggesting we remove UP support, or at least strip down UP support, in that we make it look more like S&P, allocate all the S&P structures, and do all that just to unify the code a bit more. But given that Intel is still producing new 32-bit UP CPUs with the Quark stuff, or what have you, and ARM is still doing UP stuff, it is still relevant. We actually just merged, I don't know if it was 3.11 or 3.10, but we merged support for M-class cores, which are really, really, really small ARM chips. I think there was one where you was like the size of a full stop and it had a solar panel on it and it could power itself. Okay, you probably wouldn't run Linux on that, but we've just merged support for that. And I mean, that's going to be the type of stuff I think you might see in the Internet of Things. The trick is having enough RAM, because the RAM requirement is still quite high for these guys. You might only have 16K, and you can't run in that. People find hacks, they run it in the frame buffer, and you get a nice picture of whatever the hell Linux decides it's doing at that time. I mean, we are people that are running Mainline on those now. But I think as long as people want to run it and are willing to put the time and effort in, it shouldn't be too hard to keep it up. I mean, once UP really is gone, it'll be dropped very, very quickly. Okay. I can't wait for the date, but I'm afraid I'll have to wait a long time. All right, I'm going to shift topics just a little bit. One of the perhaps more controversial things that the Kernel Summit Committee did this year was to open up a few slots for people who identified themselves as hobbyist developers, people who develop on their own time. How many of you out there in the audience have contributed a patch to the kernel at some point or another? So, I don't know. Fair percentage, maybe a third. How many of you have done it as an unpaid, hobbyist sort of developer or started that way? Quite a few of you. Right, if you look now from as best Greg and I can tell, something over 80% of the people contributing to the kernel are paid to do so. But there are people who do it on their own time, one of whom is on the stage here with us. So, I would like to start by asking, why do you do it? When you're surrounded by people who are being paid to do that and so on, what drives you to use your own time to work on the kernel? Okay, first of all, it's interest. I like to discover the electronic devices and the SOC layout, but maybe I'm not that prototype of hobbyist because I'm hardware engineer, I'm building the SOCs, so I know what's going on inside. Part of my job is to write drivers for the hardware I've just built. So, I translated that to my personal interest in tiny devices and it started all with ARM or maybe earlier consumer electronic devices. You open them up and you find some interesting pins and you find access and then you start just looking at the kernel, how should it work, how can I make it do things I want to? That's the way. And then you're asking questions on the mailing lists and if you are lucky, somebody answers and you're fine, you're in. Is our community friendly to hobbyists? Do you get the answers that you need or do you run into troubles there? Well, it depends on your way to start the contact, of course. The people in the mailing lists are there long way before you've just realized that there is a mailing list, so if you start kindly, they will accept anything. You can be wrong, they will correct you and probably they have made it a thousand times before with other people because I'm not following the whole discussion. But in general, it is friendly, yes. From the other side of the coin, ask specific questions. They're easier to answer than the open question on how do I do blout. So ask specific. Always ask on list. If you mail me privately, you're on a very short path to death, no. And I generally try and answer. I don't always get around to it, but I do try. I'd say if you don't get a reply as well, don't feel too disarmed, maybe ping it again after a week, not immediately because the traffic currently on Linux ARM kernel, which is just for our charm, well, our charm 64, but there's not many developers there yet. For our charm, I think we're eight or nine thousand mails a month just for the architecture stuff. Well, we all try and review it, but it's definitely more people writing code than reviewing code, at this point, or anything even close to it. I have trouble just keeping up with the in-books. I read everything all the mail on this I do, except for Linux kernel. I filter Linux kernel. Read all the subjects. Read the subjects, yeah, there's some things. But other, my subsystem mailing lists that I have, I read all of those. So those are smaller, much less traffic. And that's the way with most maintainers of subsystems. If you figure out where your question is, there's a much smaller group of people to ask and a much tinier mailing list, and those are usually always maintained pretty well, hopefully. Yeah, it is. The ARM curcuit, the ARM Marvel SoC people, they find emails on the list. They answer. That works quite well, because if you pick something out, then you probably get CC next time around. A suitable subject is helping. So if you write curcuit, the maintainer will find it easy. If you miss to write the correct SoC name or whatever, it's difficult. It's a lot of mail. I skip days of mail. So when you look at where contributions to the kernel come from, one of the steady trends that I've been observing is a decline in the percentage coming from volunteers from hobbyists. And so I've been trying to figure out why that is and whether that's a problem or not. So any thoughts on why we might be getting less from volunteers? Is the absolute number shrinking or is the relative number shrinking? Percentages going down. So look at the absolute numbers. It could just be that there is just that many more paid people contributing on their free time isn't actually dropping all that much. I'd say one thing that has gone down is time to market for mobile devices. That's really short, a lot shorter than it used to be. So these companies are paying a lot of money to software engineers to get BSP support to get drivers in. So perhaps that it's the number of paid people. If you do five patches, you will get a job offer. I know at least five people who start off with tiny coding style fixes and the staging direct driver now have full-time kernel jobs. If you want a job, seriously it's usually been offered. You already have a job, so I don't know, but it's really hard to find kernel people with a lot of companies. They mine the list, they ask us, they ask me all the time, so I mean seriously I think it's that people just get hired. But that's mine. Some of them clearly are. Some of them are. I know some of them are. Which is kind of sad because we get them maintaining drivers and they get hired in a way and they do something else and they're not maintaining that driver. That's a personal problem on Linux. Or they get reassigned to a different project. Yes, they do. That's happened a lot. I hate it when I... It seems to happen a lot. Matthew Garrett yesterday talked about how he merged a bunch of buggy code and then got hired and he sort of left it behind and I saw you complaining about somebody who merged some code and then they're mail bounced. Yes, they were with the company. And so on. What kind of persistence do we want to try to insist on? How can we be sure that people will be around to maintain their code in this environment? I mean the problem as a maintainer of accepting somebody's code is if you disappear I have to now maintain that. So it's... For bug fixes and what, that's not an issue except a major body of code. I need to trust that you're going to be around to resolve it or I have to take that. The famous example is with the networking code a number of years ago. They merged a big, hairy piece of code and literally the day after that was accepted the email address disappeared from the web and it took Dave Miller and the network guys I just think six to eight months backing all that out and fixing the problems. So it's really hard to get core networking changes for that reason. You have to be around to do this work. The work from the body of code keeps increasing. My ability to do stuff is fixed. So you need to become part of the community in a way that then you know I can trust you to be around. The kernel development is a web of trust. I trust people not necessarily that they got it right but they'll be around to fix it if it's wrong. It's that simple. I don't know if people have been filling out sheets with questions on them. Maybe someone could bring them if we have some. Otherwise I'll continue to... Anyway... I'll continue to ask my own while I'm looking at these. Let's talk a little bit about device trees. Because that's an area of interest. A lot of the hardware that we have anymore is not like our nice PCs where you can simply ask every device what are you and it will tell you, right? A lot of the hardware that is being made now does not describe itself. So you have to give it a description externally and the solution that's been adopted for that is a data structure called a device tree. I know Linus's opinion of the subject that's about all. So we're starting to run into trouble with device trees because device trees are seen by some as a form of ABI that we should not break them as the kernel progresses and we are breaking them and we have various other issues with inconsistencies and so on. So the folks here who have worked with device trees and such to start with is device tree the right answer? What have we done wrong? How can we fix that? So I'll start with a caveat. And that is that the ARM Mini Summit which is going on now and tomorrow is currently in the middle of a heated debate about this stuff and it's continuing over the two days. Greg and I have just been talking about this very topic and we were arguing against each other. And certainly nobody has a complete agreement on what we should do about device trees. I was there when I started kernel development maybe a year or two before we started doing device tree. And I think that the amount of code going in since then has gone up astronomically and we couldn't have stuck with where we were. So device tree has definitely been an improvement in that regard. I think it's allowed a lot of SOCs to be supported. It's allowed a lot of drivers to go in. It's allowed a lot of code to go in as well. And we have also consolidated a lot of subsystems while we're at it. People will argue that we could have done that without device tree but I think the two are probably quite linked there. When it comes to ABI that's a religious issue. There are some people that are absolutely saying it's unstable, don't even think about putting it in a product. And then there are other people saying it's ABI, you can't change it at all. And they are people that have a lot of respect within the ARM community and there's just a fundamental disagreement. Hopefully we'll iron that out in the mini-summit. I don't know what your thoughts are on that. The ABI stuff, the stable stuff is very hard for hobbyist contributors. I have an SOC I find that one IP is using Synopsys designware of the shelf IP. You write some device tree support and then you have to invent a device tree if it's ABI, that will be sufficient for every possible variation of this IP. And you've just started to look into SOCs. But now you're forced or asked to think of every possible way to connect this IP and it's hard for them. You know that you can connect it in a million ways. And they'll break it in two million ways. And you will find somebody who didn't understand why not to do it this way. But they did. And there's also been a big trend to take what we had before, right? There's lots of static data in the driver and simply cut that out and then paste it into device tree so you have like property hash defined and they put the number in. That's really not what it's supposed to be about. It's supposed to be the phrase that gets thrown around and it's supposed to describe the hardware. I think people are now backing away a bit from that. So it's very much changing at the moment. And it's hard for, as a driver maintainer, I don't know what, if it's correct or not. They said it worked, great, I'll take it. And that's been a big problem because it wasn't right. And it wasn't the right schema. The number of people that actually know what it should and shouldn't look like is very tiny in it. We need a big education effort just for kernel maintainers and developers. And that hopefully after the two days we'll have something to hash out. But as long as you see the device tree as a replacement for the platform data, like the resource management or naming resources, interconnecting modules or devices, whatever. I think that is fairly compatible and not affecting drivers at all. Because the resources are passed automatically. You can write a driver that will be supported by device tree and not knowing of it in any way. Yeah, but someone's like oh, you used the wrong name for the company because it didn't match the name of the company's theory and then we get argued. It's like stupid things that I don't even know about. I usually choose the marketing name because that's what they are publishing. That's all you would know, right? But then other people would say use a different name. Marvel just did. I started the work on the Berlin SOCs which are in the Chromecast and the Google TV devices and at some point at the patches I've sent they said let's name this BG2 or something and remove all Armada marketing names. I said I can't. That's the name everybody will look for. It still works. It still works to be done on that. Just the other day I saw a fairly detailed critique from Russell King, the top level arm maintainer about the device tree abstraction. I didn't really have the time to understand it but I had a lot to do with the interrelations between devices which I found is already hard because a lot of device drivers aren't driving a device anymore. Instead you have a collection of devices and buses between them and so on. That's when we've just been occupying it together and make it work right. Is there a fundamental flaw there that we haven't... We have these complex devices especially the video for Linux guys have been doing this for years. We have different ways to hook up and make a core and now we're seeing it in other types of devices and then graphics devices that do we describe those interconnectivity of all these weird SOCs or weird IP blocks as a way in device tree or do we do it somewhere else? Because we have to describe it somehow. So what's wrong with making it all discoverable? That's another story. So for years the core kernel people have been saying make things discoverable and that and then for years hardware people have not been doing that and we have to support what they make. Obviously. But why can't we just take them out back and make them do stuff? You missed our talk about an hour ago. I think that was your conclusion. That was my conclusion. So that's a problem. Maybe device tree isn't applicable. Russell's big objection was for this very complex 3D graphics chip set it's all hooked together in weird ways and is that something we even want to describe a device tree? I don't know. Maybe that's something we shouldn't be. I think actually it goes further than that because even if we did have an ideal way of describing this to the kernel there are still perhaps some issues with how we deal with buses or how the whole driver model can't deal so well with if we've got multiple instances which all have to kind of talk to each other. Right. And the driver model is known for having problems. I did that. That's problems. But yes, maybe we need to fix things there That's a tough problem. The alternative that we're seeing is sometimes a lot of this gets punted and then the hardware knowledge gets pushed out into user space instead. You see that in video for Linux with the media controller thing where you really have to know what it is you're connecting. You see it with ION where you have to know what the memory requirements are for DMA buffers. We're starting to see more of that as our hardware getting to the point where we just can't solve it generically in the kernel. I don't think it's unsolvable. I just think it's not nice to think about and not nice to solve and it's nice to pretend that it's somebody else's problem. I think that actually you gave an example of pushing it into user space. It's also pushed into firmware. So we push it in both directions. We don't want to keep it in the kernel. Unfortunately, there are certainly cases now as we were discussing earlier where the kernel actually needs this information as well. It won't be able to operate correctly without it. Particularly more complex systems with intro route routing and things like that. Dynamic intro routing, cache coherent interconnects, all this kind of stuff where the kernel needs to know it. Hardware now is really a plug and play design model. You buy a core from somebody and buy another core from somebody, buy another core and you stick it together in your device and you ship it that way. We have drivers for those individual cores, but now we have different drivers that need to talk in different ways. So now we need to come up with more flexible ways of doing that the way the hardware guys have already solved. That's a tough problem on our side to do. So maybe whether you push it to user space or kernel, somebody still has to do it. So it's not like it's going to not happen. It's tough. We have a job for a long time. So maybe we shouldn't complain at the hardware developers too much. No. Discoverable buses, yes. Just make a discoverable bus, guys. It's really simple. I mean, years ago we told Intel to do it and they tried and they almost got it right for some of the really tiny embedded stuff. But it's just make us some static tables and we'll read them and then we'll find. But in our discussion an hour ago somebody was saying, well those static tables and the chip will be wrong. How do we fix that? It's like, well, then fix the hardware so then we're responsible for fixing the hardware bugs. But how many buses are discoverable? Lots. Not as many as there are embedded buses. Like I2C, SPI. I2C is not, but I mean PCI, USB, SPI no. Yeah, maybe that's it. But so PCI is a good example. Intel did this. They used the PCI headers as a discoverable way and it's just some static memory we're reading that describes the hardware behind it. You know how to enumerate the number of devices you have there. That's all we need. We just need something simple. Static tables. The way old Intel processors used to start up we had static tables that described the memory layout and all those fun things. Which were sometimes wrong. It was sometimes wrong and we couldn't believe in that but you got to start somewhere. But people don't want to put a PCI host controller in there. No, you don't need a PCI host controller. It's just a static table that looks, I mean Intel did this. It looks like PCI. It just gives you some memory, some values you read out and you go and they wrote a little tiny stub driver and it read it and away you went. Well actually even if I design a system at work and put it into an FPGA I have this static table there. It's lost after synthesis. Because I don't put it in hardware. I could. So I think the TI guys did something for HWMod which kind of did something like that. I think it slipped in the RTL. Should we talk to the hardware designer tools people? Well as a hardware designer is it easier just to randomly hook up pins or is it easier to... The address map? You decode addresses and say... But you use tools like VHDL which is a language here. So why don't we fix the tools they use to automatically do what we want it to do. You need to talk to Synopsys or Cadence I guess. Maybe we do. That's a good idea. We'll look for your pass Peter. We're drifting off. I'm going to switch to an audience question. Total change of context now. This is one some of us have heard before but it comes out often enough I think we might as well ask it and deal with it. Brace yourselves. SE Linux is a critical part of the kernel when it comes to security. I believe the NSA, the US National Security Agency has a big contribution to SE Linux. And the question is phrased are you still happy with the NSA? But I suspect that what they're really asking is are we really happy with SE Linux? Do we trust it? Whatever. It's a royal pain to say that. I mean I merged the Linux security model years ago and we created that as a way that Linux didn't have to rely on only one security model because the SE Linux guys were proposing this be the Linux security model. So there are ways to plug in different security models. So SE Linux is one and if you like it, I think it works really, really well. It's known to be a subscription for solving the problem that wants to solve. And I think it's good code. Whether you trust it or not, I mean it's user unfriendly in many ways and lots of people disable it, but lots of people run it. It's an access control type thing. Works. Other people don't use different I mean there's the app armor model, there's a lot of other models, there's security models out there that you can use. The smack is being used now in shipping devices. It's trust of different models. But these guys that wrote the code they're still part of our community. They've been around for years and I think they're trusted, known members of the kernel community. Yeah, also consider how many people are employed by the NSA. We don't know. It's huge and these few people we know them. The NSA has two hands. There's a protection side and a breaking protection side. These guys did come from because the government does rely on Linux and they came from the motive we want to be able to use Linux for our own systems and protect our own systems using this type of stuff and protect other parts of the government and other companies because it is if you have a known secure infrastructure that's a good thing to have. And they came out of the protection side. That's where these guys came from. In general, I had some stuff related to this because security and such is a lot in people's minds. So I'd like to ask what is our responsibility with regard to delivering secure systems, secure kernel to people? Certainly we can't at the kernel level secure the entire system. But we have our piece of it and are we doing that well enough or how could we be doing it better? We can always do better. The fun thing, I mean, and I argue it's what is determining what is and is not a security bug issue. I mean, as a responsibility, I'm part of this kernel. There's a security at kernel.org alias. Our goal is if you report a problem, we will fix it as soon as possible. I mean, that's the best that we can ever do is if we find a bug, we will fix it as soon as we possibly can. And I don't know a better way to do anything other than that. Yeah, and when writing code we just try and apply best practices and I'm not a security expert. I'm not ingenious or nefarious enough to actually do a lot of these things and I'm surprised always by the ingenuity of some of these exploits. Oh, they're great. And I try and learn from them and try and avoid these traps, but yeah, it's like Greg said, we're all you and we all make mistakes and the best we can do is try and fix them. We got good, I mean there's people doing research on static code analysis tools. I mean, Julia and Dan have fixed more bugs in the kernel than anybody with doing static analysis tools that are really fixing security bugs. So there's some really good stuff and now those are being run through automatic testing so any no code that gets checked in the kernel is automatically run through these static analysis tools and those bugs of known problems in the past are now fixed. So as long as we continue to learn from our past mistakes, I think that's the best we can do. And there's ongoing research. Julia's a professor in the university and I think we're doing the best we can. If anybody knows other ways we can do things better, let us know. We're open for that. We should do it. Because we're running everything now. It's scary. Okay, related question I guess. The very last 3.0 update came out this very morning as I noticed. The stable updates for the 3.0 kernel added up to something on the order of 4,000 fixes that went into that kernel over the course of two years. So one could argue that we released a stable kernel. With 4,000 known bugs in it. Well, I mean known after the fact. Not known at the time. So once again, are we doing as good as we can? Why are there that many things to fix? I don't know what's the alternative. We're not adding, we have to trust us that we're not adding bugs deliberately. Some of those fixes were adding new device IDs. We backfired some major new things. But that was, you're right, it was fixes. Now we're fixes that were done. But I mean these were things we learn over time that we did it wrong. So will 3.10 have fewer fixes by the time it goes unsupported? Proportionally no, because it's a larger code base. And now I'm doing it, I'm also doing it full time now as part of work at the Linux Foundation was the first year of 3.0 that was my hobby of maintaining it. So I wasn't finding all the patches and finding the pieces that needed to get backforted into it. I think 3.10 is going to have more because there are more people going to be relying on that kernel for a long period of time than 3.0 was. 3.0 was a neat experiment and seeing if we can do this in a two-year manner. But if you look at the number of fixes that affect a kernel over time, after a year it starts dropping off and then two years it hits the floor. If you look there was eight patches in the last release while there was in the last 3.10 kernel there was almost 100. That's a week worth of fixes. So the number of things that are applicable over time have fallen off. But also a lot of the patches where we know there is something wrong but the fix is so big that we try not tell Linux, hey pull this at Rc7 and then we let it cook for Rc1 or Rc2 and then pull it into stable if it turns out it actually works. The IPC worked recently. That was the big one. We fixed a lot of bugs in IPC and it ended up being like 40 patches that got backported to 3.10 which they went in over two kernel releases and then they finally all chuckled back in and the SUSE guys did a really good job of developing that and backporting it. So I don't know, yeah. 4,000 patches. One way to improve the metric is not to backport fixes. That's a good idea. People have to deal with that. It's a tradeoff for stability. It's a tradeoff for stability of not ever changing anything versus fixing things which implies changing stuff. Well, the world changes also. I mean, some of the things are things that people have never done before at the old time. Things change over time. Processors speed up. We have timing issues in this area now because they weren't there before. Linux changes because the world changes. If we stop changing, if our rate of change stops, then we are dead. We are going to have to keep changing. I mean, old OSes are there because their environment never changed and they're sadly dead. I don't want Linux to die, so we're going to have to keep changing. One thing is IP is reusable. So they reuse it over years, over tens of years. The Marvellous Hots sold today have Ethernet cores sold in PowerPC system controllers. I don't know when. 15, 10, 15 years ago. And it's the same driver. It's the same IP. And they get features. The driver doesn't know of the features, but it breaks the new IP the way it was written before. So yes, the world changes, and at least they get fixed. You get fixes. So to sort of circle back to the previous topic, you said that we're not how do we know that nobody is introducing bugs to the world? Because there are some worries about that, right? And I don't believe we've ever caught anybody trying to do that other than the really heavy handed thing in 2003, which was never going to work anyway. I don't believe we've ever caught anybody trying to do that. Well, the whole Linux maintenance model for the kernel, it relies on trust between maintainers. And if we don't have that trust, then the development system breaks down. So that's the backbone of it. You trust the people that you're pulling from. And essentially that's what we rely on. So we think that we're not adding things deliberately. If somebody tries to add a bug deliberately, then hopefully it will be spotted. Who knows if anyone ever has, right? But if it doesn't get spotted immediately, one would hope that it gets spotted later on. I think if you look at the known exploits we've had over the time, they can, as far as I know, get back to just stupid problems where we messed up. I mean, nothing's ever been figured out. I mean, I know I've introduced lots of bugs that have been fixed, but I've just messed up. Yeah, we all have. It's just part of code. But, I mean, people who have analyzed our code base do know that we have the least number of bugs per line of any known code base. So I guess we're doing better than everybody else. But it's a hard problem. It's interesting. And there's a tons of research out there to do. If you're looking for a master's or PhD thesis, please, there's people doing that. That's good stuff to work on. It's a good thing to study and help us. Even at the lowest defect rate, I mean, I think it's pretty easy to argue that we introduce enough bugs that there's no need for anybody else to kind of deliberately do it because that's just part of how things work. So, yeah, hopefully we won't have trouble with that. But when you see some of the things that slip through even in core code that is theoretically reviewed, it's really hard to catch things. You know, they're just mistakes, not... Yeah, but in our model shows, I mean... At least more people were reviewing my stuff. I mean, at some point, seriously, at some point, it's very few people that are around who actually look at your code. And some of the patches, I write them, Ingo merges them, I don't think anybody ever looked at them. And yes, I do make mistakes and I do get to clean them up, but... Yeah, so you and Ingo looked at... I mean, a lot of stuff can be traced to two or three people, every line, but the fact that you can be traced is a very good... It's a personal ownership issue, so that's a good thing that we know we can go blame. I mean, it's a path of blame, these kernel patches. Which is nice, so, yeah. In fact, when I looked a few releases back, a significant percentage of the code going into the kernel had no sign-offs other than the author. Nobody else had signed off. Were those authors subsystem maintainers? Often, yeah. Because they're the people who can actually pull that off. Do we have a review problem? How can we get more review? Because you never have too much review. So on the R list, I think we definitely have a review problem. Again, particularly by a device tree bindings are concerned. So we did establish a set of people to go and review device tree bindings, but it's not enough people. Some of them have since quit because there's too much email they're getting. They can't do anything else. They can't do that job. They can't go to sleep just to review device tree all the time. So, yeah, there is an issue with review. And the problem is that to get reviewers, you need the people who have all the expertise in that area, and they're either pulling code and dealing with that way, all the other ones that you want writing it. So it's hard to train people up. There's a bootstrap problem there. And once you've trained them up, they leave. Yeah. I mean, we have very few people that only do this as a full-time job as a maintainer. I think there's a number. I mean, Dave Miller said maintainers are like editors. You end up working with other people's stuff and getting it working. And every once in a while you have a side project of your own, which is kind of annoying, but that's our job to do that. Finding those people is hard. Reading code is a really good way to learn. So I try and get people to read code. In fact, reviewing code is one of the best ways to learn about it. It is. It really is. But I mean, I do think so I have a subsystem I wrote 10 years ago in a kernel, a USB serial. I maintained it. I've maintained it since then. I wrote some new code for that. It took me four tries because I got it wrong. But people reviewed it and rejected it and stuff and told me not to take it. Some subsystems are good. They do review that. So I was a proof of that it is still hard to get code in at times. Okay. Let's look for our future for a moment. BSD or OpenBSD in particular recently fixed their year 2038 problem. They did. They fixed it. The internal time T variable runs out of bits and wraps around. Happens in early 2038. We have not fixed that. Are we going to have a year 2000 type problem as we get closer to 2038? I was always hoping that by then we'd use 64-bit CPUs. I can tell you there will be 64-bit CPUs. 64-bit CPUs solve that, right? Yeah. Will we have no 32-bit CPUs? Well Intel just came out with a 486 new chip. I met one of the guys that said, yeah, I did that design 20 years ago. And we're still shipping billions. Yeah, you have billions. I don't know. That's tough. Somebody needs, I don't know. I don't know. I don't know. It was obviously somebody's issue that it should be fixed. OpenBSD fixed it by breaking their user space API. They can do that because they have everything together and don't care about anything else. We do care. We care a lot. So we have to do something clever which suggests that perhaps we should be thinking about it. I did talk to some of the tool chain guys about this. And they're really keen on breaking the API because there's a whole load of stuff they want to get in for libc and some unwinding stuff. So I think, to be honest, that's what will happen. There will be a new API. In our man, we rolled a new API for hard float recently. We have to go and rebuild stuff. So I think that's what will happen. The problem will be that people are deploying devices now. Satellites or weather stations or something stuff that we know have to be supported for 20 years. It's going into traffic lights. At least in traffic lights. But at least with the traffic lights you don't get that they run this old software on state-of-the-art hardware. I mean, we've got people trying to run REL3 on Haskell chips. Right. Now that's nice. That's a totally different issue. All right. So we haven't solved that one yet. So if anybody is looking for something to do with it, and you're willing to take a lot of darts. That's a tough problem. That's going to be a hard one. Okay. We're running out of time. We have just a couple of minutes left. So I want to sort of talk a little bit. Again, maybe future is about innovation stuff because we talked a little bit about device tree and how hard it is to get that stuff right and support it. We have at least one known critic of control groups on this stage who is just like that sort of stuff. And that's been another interface that we got wrong both internally and externally. Internally, we can fix it externally as harder. We're blazing a lot of new paths now. We're doing stuff that no one's ever had to do before. And everything we do, we end up committing to support. How can we do this? Are we going to end up carrying just tons of baggage and get buried under it? What's the alternative? We've tried this. We've had this problem for the past 10 years. Even with simple things, you need to write it at least three times to get it sort of right. Now, at the user interface, you'd have to try three user interfaces before you get it maybe right. It's a royal play. Everybody, every OS has API changes like this. Windows does it. Solaris did it. I mean, they had official ways to deprecate things and stuff. And we've tried those official ways and sometimes people yell at us and not to do it. That's good. So what we do is we create a second API and then just wait for all the users of all API to die and go away and then we try and sneak the code out and then somebody yells, but hey, I was still using that. And we go, damn. And then we put it back in and wait. Yeah, I mean, creating something brand new when you don't know how it's going to work over time is an insolvable problem. You know, you have to try it to figure out if it worked or not. And I think if you try to wait until you've got the perfect solution we'd never have a solution. You won't know if it worked. We have enough developers who are committed to supporting this stuff and working on it that actually we can kind of get away with the way we're doing it at the moment. Everything we're doing now is new. We don't have any model for what anything we're doing anymore. We're now the forefront and it's hard up here. Hard but fun. It's fun, yeah. There's lots of cool and fun projects, but it's something that is going to always happen. And we are now learning over the past five to six years how to deal with that. And we argue and yell. And the thing is we're doing it all in public. It's stuff that you don't see other companies. I mean, I know Microsoft has rewritten core APIs and user space APIs three to five times over the years because of the same problem. New hardware comes out with different device models. You know, 100 gigabit ethernet's coming and what is they doing to the whole networking core? It's totally having to change again. It seems like a high quality problem they have. Yes, it's a good problem. It's a little clock is actually beeping at me. I don't know if you all can hear it. This is just that we've run out of time. So what I would like to do is to thank all of our panelists. Can you give them a hand?