 My name is Greg Crow Hartman. I work at Noval, Sousa Labs Division. I'm a fellow there. I do kernel development for them. And I've been doing a lot of work on Moblin or Goblin. That's what this is. This is Open Sousa version of Moblin. So it looks a little kind of funny. That's what it is. It's a lot of fun. The talk is online, that location, including the slides and a big, long article that was written so all the words are there. So you can read it if you want to read it. So write and submit your first kernel patch. The Linux kernel last year had over 2,000 individual contributors. Now half those people only submitted one patch, then half the half submitted two, half the half the half did three. It's a little exponential curve. But those 2,000 people, that 1,000 people that did one patch, that's good. We need more people contributing. After this talk, you guys can contribute. So there's what? 2,000 people here. So we'll double our quantity next year. Awesome. So first thing in the Linux kernel, even if you don't want a program, you don't want to do anything, but you have questions about the source code, anything, the documentation, how to file. It tells you everything. And if it doesn't tell you, it tells you links to other files, other websites, other locations of what to, of what you're curious about. If there's any question that you don't, that you have that isn't answered by that or the other links, let me know. Let other people know. My email address is at the front. We maintain this. This is a very, very good resource. You have a question. It's there. We write in lots and lots of documentation, and nobody reads it. But hey, I'm trying. So what are we going to not cover in this talk? I'm not going to teach you how to build the Linux kernel. I wrote this book a number of years ago for O'Reilly. The Linux kernel in that shell, it's free. It's online. You don't have to buy it. I really get no money if you do buy it. That tells you how to build the kernel. It tells you how to install it, how to configure it, how to do it all. It's all online, free. That'll teach you how to install a kernel. If you can install your own kernel that you built, you're almost there. That's it. It's a very good thing. So read that book. I'm also not going to teach you C. I actually learned C at the time I'm on talk. I had Minix in college. I had to learn C and write a floppy driver as my very first assignment. It was hell. But I loved it. I did not do good in that class in university. But I loved it. Now look at me. Very good book. It's a very small language as far as the specification. Very, very powerful. People can call it a portable assembler. That's true. It runs on anything. I've written C code for microcontrollers that had one and a half registers. It can be done. That was a fun processor. All right. What are we going to cover? Here's what we're going to cover. Talk about Git, the Linux kernel source code control tool. It's a very, very powerful source code control. Our coding style, what the kernel does. How to fix a file once you find what you want to change. So talk about how to find something to change. How to generate the patch. And how to email the patch. That last one gets so many people. Email clients suck. Email servers suck. They like stripping off tabs. You guys see it, you're developers. Convert tabs to spaces. Spaces to tabs. Wrap lines. Don't wrap lines. Exchange server for Microsoft will not send Linux kernel patches through it. It just won't work. So much so. Microsoft has now contributed to Linux kernel. Microsoft has an internal Linux server that they use to send me patches. It works great. Intel has the same thing. IBM has the same thing. Novell has the same thing. No, we don't use Exchange. We use GroupWise, which is actually worse. Oh, this video? Sorry. Different division. GroupWise started on the Amiga. Anyway, so let's talk about Git. Git is awesome, if you guys don't know about it. So Git is a Linux source code control tool. The Linux kernel uses it. Lots of other people use it. It's very, very good. I'm not going to go over how to use it, how to install it, but I'll talk about how you can use it for the kernel. It's a very, very powerful thing. There's really good tutorials online to use it, so read them. There's very good books online. Some of them are open. Some of them aren't. There's some very good stuff. It's out there. But some things about Git that's really good. So first off, you have to know where you're going to get our kernel tree from. And it's at that tiny location. It's easier to Google it. But there. So you clone the tree somewhere. So I've cloned the tree, and I'm in this directory, in my directory, and Git status. Nothing's going on. And this is a little tiny, cheap laptop in Git rocks. It does have a good hard drive, though. So one thing, when you start using Git, is you don't want to be modifying the main branch. You want to make all your changes somewhere else, because you're not Linus, so you're not going to be able to modify his tree. He takes patches from other people, and then you pull. So you don't want to mess with the main branch of development. So what we're going to do is we're going to branch. We're going to Git branch. So let's make something called Git branch tutorial. So I've got two branches. The little star says which one I'm on right now. Master's the main one. I really want to switch to the tutorial branch, so Git checkout tutorial. And hey, it does bash completion. Switch to branch tutorial. And now we're on the tutorial branch. So anything I do is on my separate branch. I won't mess with Linus's, if I pull, it's good. We also want to be on a separate branch because we want to know what changes we have made compared to him, because we want to send all those changes off to somebody. So we want to know how to differentiate ourselves. So that's a nice thing to do. And then we'll always show you what branch you're on. I'm on the branch tutorial. So Git branch. Make your own branch. Don't be afraid of branches. They're good for you. They work nice. So let's talk about the coding style in the Linux kernel now, now that you guys know everything about Git. First off, you guys do a lot of development with other people. Why do we care about a coding style? And in short, it comes down to one thing, your brain. This is a study done a long time ago. You guys can read the quote. But what it really means is your brain sees patterns. And it's very, very good at pattern mashing and seeing things that are out of the ordinary of those patterns. So when you use the same coding style over and over and over, the format of stuff goes away. Your brain can't see that. You can see the metadata inside. You see the bugs. You see the logic inside much, much better. So only thing that matters about a coding style is it's consistent. I don't care where you put your braces, your tabs, your spaces, if you're going to do it for your project, but be consistent. If you're on a project that doesn't have a coding style, make one. Well, don't use a good new one. But make another one. Yeah, my brain hurts watching the new one. But that's good. Those guys know their brain feels good, and their brain hurts when they write Linux kernel code. So it works. You can switch between them. But be consistent. Seriously, if you're a one-person project, pick a coding style and stick to it, because you want somebody else to join, somebody else join. Hopefully you should. I hope there's no major projects these days without a coding style. I hope. It just matters. Because the real goal is I want you to fix my bugs. And you want me to fix your bugs. We want other people to read the code. Reading code is very, very important. When you first start off in university or learning how to program, you start writing code. But when you first start learning how to speak a language, you don't start writing it right away, or music. You don't start writing music right away. You read a lot. Reading a lot is very, very good. So having code that can be easily read is very important. The Linux kernel is one of the largest code bases in the world, largest developed project in the world ever has happened. And it's very, very readable. It's very good code. It's very easy to understand. Even at C, it's very easy to understand. So that's why it matters. It really matters. It's all down to your brain. So what do we do? We have it. It's in documentation coding style. You'll notice that's camel case. I don't know why. We don't do camel case. But in there, it's a very readable document. It's gotten longer over the years. People started being nitpicky. You can get too particular about a coding style. The best thing is have a few guidelines and stick to that. When you start getting where you put some other things, it gets too much. You can argue, like everybody likes to argue. So here's our rules. Use tabs. Yes, sorry. Yeah, but it's consistent. Hopefully you don't do that in your projects. And I won't contribute. But you won't contribute to mine. Use tabs like God intended. All tabs are eight characters and 80 character line limit. This last one I hate. And it's actually starting to go away. We realize our terminals are not 20 years old. They're not 80 characters only. So we do start taking things long. But the interesting thing about these combination, if your tabs are eight characters and your code starts going way over here, and you hit the 80 character line limit soon so your code starts wrapping and looking really bad, what that means is your code's getting too complex. You should be refactoring it, making it smaller and easier to understand. So by sticking to 80 characters is a very good limit on yourself to make easily to read code. Means you're doing something wrong in the logic. It shouldn't look like that. You shouldn't be running into that limit. Now long strings, long function names or long function declarations you can break. Strings you really shouldn't because you want to grab for them easily. But so 80 characters is, even though if we do bump it to 132, which we probably will, it's a good logic check. It means you're doing something wrong. All right, so people hate tabs. Braces in the sea like Kernigan and Richie show. Upper right, lower left, all the way over. That's it. There's nothing other to think. Fun thing, error numbers. All our error return values are always negative in the kernel. Zero means true, yeah. That's Unix for you. We never return a positive number. Very, very, very rarely return a positive number. So, braces like that. What we do with it, it's else statement. Again, wrap it there and there. Report error go-tos, we like go-tos. Go-tos are really important because in the kernel, we do locking. You lock because we're doing multi-threaded, multi-processor stuff. And when you enter a function, you grab some locks. When you exit a function, you unwrap some locks. Report error, if I was to do something, I want to jump to the place where I unlock everything. If I was gonna return here, I would have to unlock, unlock, unlock. And then you forget because somebody changed the locking a little bit. It's more important to jump to the end and do that. Go-tos backwards RMS. I wouldn't recommend that. The kernel does in a few places, but it's scary. The scheduler. I write drivers, not schedulers. So like that, that's it. It's a real simple coding style, real easy. Now here's the one that a lot of corporations get upset about. We don't do it for single argument if statements. That's it. Good party. Single argument if statements, no braces. We want to keep the lines so you can see it easier. That's as simple as that. But if one of these arguments, like the if or the else is multiple lines, then you can put the braces on both sides. It's okay. That's it. So, and the other place for braces in a function all the way to the left. Do something, return zero. Again, returning zero means true. Integer is where we return values. That's it. Small variable names, lowercase, underscores. Small, logically easy to understand function names that do something that say what they're gonna do. No Hungarian notation, no returning, telling you what type of the variable is. That's what the C compilers for. It will check it for you. Nothing crazy. So that's it. Look good. Good. We have a tool that'll check it all for you. Checkpatch.pl in the script directory. It's a fun little tool to run on the kernel itself. But it'll check your patch. If you make some modifications to the kernel, you run this tool on it, it'll tell you if it's good or not. If you've done everything you're supposed to do or not. It's a really, really fun tool. It's written by a Perl guy, so don't hold that against us. Checkpatch. You can do a file. You can just run it against a file. Terse. Let's pick a file. Drivers, staging, comedy, drivers. C. I have no cash. There we go. So it says, if you see at the very bottom down there, this file has zero errors, 39 warnings, those 312 lines checked. The record so far, I had a driver from a company that will remain anonymous. That was 300 lines along the driver, my small driver. It had 752 errors. More than twice the number of errors. Then there was, I don't know the number of warnings, per line of code. That was with comment blocks. Oh, man. We took the logic and we bleated it and rewrote it. Anyway, so here's what I'm saying. Warning braces are not necessary for any arm of that statement. So let's make that our change. So in the driver's staging directory, I maintain that. It's all the bleeding edge stuff. It's the crappy drivers that are not been merged into the kernel pre-properly. Every place in there, there's a to-do file. You can read the to-do file if you're curious about what you wanna do. You don't know what you wanna check. You don't know what you wanna do for writing kernel code. There's a big to-do list in there. Do one of those things, send us patches and we're happy, you do it. So cleaning up the code to make a check patch clean is always on that list. So we'll do that for here. So let's, and drivers, I can't see. So on this line down here, I'll move it up for the people on the bottom. Maybe not, that's the whole file. Let's edit it. Run it again. And there's one last warning. Nice. So hey, we made a kernel patch. Seriously, this is it. I mean, people do check patch, people fix spelling errors. That's awesome, we need that. People removed all the curse words, which damn it was not fun. I like those in there. I'm a hardware developer. I do embedded. You should do embedded, now I do drivers. So cursing at the hardware is my day to day life. So you like taking it out in the code. Working around hardware bugs, that's what I do. All right. So let's see what it does. Get instantly the whole 600 meg kernel tree told me instantly what was modified. That's fast. That's awesome. Mercurial also does this, very nice. Let's look at the diff. It shows I subtracted some line. I subtracted this line. Hey. Little boingy. Subtracted that line, added this, subtracted that, added that, subtracted that, added that. That's it, that's a diff. This is how we do kernel changes. We make diffs. Diff you send to the patch tool. We call them patches. This is a format. It's a very common format. It's been around forever. Larry Wall made it. Very, very powerful. It enables us to send changes through email to other people. You can read it. You can review it. You can look it up and say, hey, that's obviously right. I removed that, added that, good stuff. Oh, I guess we should build it, right? Little thing. If you want to only build a certain subdirectory, you can. Let me hold kernel. So let's just build that one. Builds the file. Hey, built. Ship it. Seriously. So I maintain the stable kernel releases. I do all the releases for the stable kernel. And people ask me, well, how do you test it all? I said a built, right? Then all the QA department yells at you. But I'm serious. The Linux kernel does not have a test suite. It's very, very hard to have a test suite for hardware interactions. The way my hardware and my laptop works is different than the way your hardware and your laptop does. So the best thing you can ever do for us is if you just build the kernel and tell us if you have a problem. That is our QA cycle. We rely on the community to do that. And that's the only way we can physically do this. Linux supports more hardware, more different devices than any other operating system ever. Currently does and ever has. It supports more processors than any other operating system. Sorry, not BSD. You guys lost a couple years ago. Many years ago, five. And all those drivers work on all those processors. Linux kernel is amazingly crazy. It's something that nobody has ever done. We scaled, we were the number one phone by quantity in 2008, and we were 85% of the world's top 500 supercomputers. Same year, same code. That's amazing. But we do it by everybody testing. Everybody tests on things. So if this thing built, it built before, and it builds now, logically I can look at it and that's all I can do. If it runs on my machine, I don't have this hardware. You don't have this hardware. So it's okay. But if somebody else does have this hardware, they can report the problem. It sounds bad, but dealing with a lot of these things is this is the only way you can do it is use it. Writing test cases for hardware is hard. I had a fun bug where if I plugged two USB devices in while streaming audio out of the third one, then I could reproduce the bug. You can't automate that. I have this giant box of USB devices that I test with. It's fun. So, okay, so we made the patch. So let's commit it. So we're gonna commit. We're gonna write, make a change into the kernel, into our git tree. So git pops up your favorite editor. Mine's the one true one. No, I don't care. You can do whatever you want. You don't tell me what to do, I don't tell you. Okay, so we're gonna write a commit message. So the good thing about a commit, well, here, let's show you an example. Here's a commit. So this is what's in the kernel tree. This is, you can look up this commit. The very first line on the top wanna make a nice short, succinct summary of what this does. In the short log, this'll show up. So it's USB OTG, which is on the go. That means USB is in the client itself, so Linux is running out there. Fix the bug on a remove path without transceiver. Looks nice. Nice, short, succinct. Then we have a few sentences. This one just says, hey, this gadget driver's removed. No transceiver, fine. It was a bug. And then we have a signed off by. Explain signed off by what we do in a little bit. And then that got sent to the subsystem maintainer for USB on the go, which is David Brinnell. He said act, looks good to him. He sent it off to me. I'm the maintainer of USB. I say I signed off by it, and then I got it into Linux's tree. So that's that. There's the patch removed. We wanna actually test if X is there or not before we access it. Otherwise it would have a nice little oops. And that's it. That's what the patch. So succinct line summary of what it does. Who wrote it? But that's signed off by. Let's see what that means. When we say signed off by, we are agreeing to this. We have something called the developer's certificate of origin. And it means this. You guys can read that. You made the change. You based it. Or you based it on somebody else's change, which was a compatible license. Or it was provided by me by somebody like that and I didn't touch it. So if somebody sends me a patch. And it's public. This is what in the Linux kernel. So every single line of the Linux kernel is reviewed by at least two people and they sign off on it. Saying legally, this is it. It's the best audited body of code in the world. I see, you can say who modified this line and it'll tell you. So signed off by is a very good thing from the legal standpoint. We also do it as the path of blame. If this file or this patch breaks something, you go, hey, Greg, you broke it. David, you acknowledge this. And Robert, you're the one who really is responsible. So all of us take ownership of it. If my name is on a patch that breaks something, it's up to me to fix it. Contribution is public. We also don't take anonymous contributions to the Linux kernel. If I know you're anonymous, I cannot take it. If I don't know you're anonymous, I can take it. It's just a legal thing. So that's all we do. That's all. And companies agree to this. You own your copyright on any files you have. Some countries like Europe are not allowed to absolve your copyright, so it's very good. You own your copyright. Your company can own the copyright. Everything's great. So that's what signed off by means. So we did something. So I'm going to say, in the staging directory, the column D drivers, the SSV DNP driver I fixed. A brace, I can say fixed. This one's kind of obvious, and I'll say signed off by. I have a little macro that does that. And let's commit it. Boom. You can see, if you do get logged, the very first thing, my patch, do that. You can see, if you do get show, there's the patch, in patch format, who authored it. When it was authored, the commit ID. You can see, get show, pretty equals full. Author and who committed it to a tree? Because authorship is who wrote it, but it might not be who committed it to a tree. You send a patch to me to send off to somebody else. I'll end up committing it to get somewhere. But you authored it. Authorship is very important. We preserve authorship everywhere. So we don't have really good ways of two people wrote a patch, but that's usually pretty rare. Authorship's very important. So there, patch. Cool. So, get diff, show the diff, commit, show it. Oh, format patch. Let's make this patch. It is a patch, so remember we had two branches. Tutorial master. So we want to take this patch and send it off to somebody. But we have to make it into a patch. So we have a nice tool, a nice thing, get format. And I want to say I want to do all the patches. If I had done more than one, I could just do it for one. I want to do from master to tutorial. And hey, it made a patch. It made one file. So if I look at this, it turned it into an email. Imagine that. This is good. It says who, when I wrote it, date, subject, fix the coding style, signed off by, what was modified, looks pretty, looks nice. Perfect. This is great. Now you have a patch. Now what do you do with it? This is hard. So email clients, GUI-based email clients will corrupt these. It's not very good. We have a whole file in the kernel in the documentation directory called email-clients.txt, how to configure your email clients to send patches, how to configure Gmail to send patches, how to work around Thunderbird's issue, work around all these issues. Kmail works really well. Evolution, OK. I use MUT. MUT works great. Send mail? You can pipe it to send mail? You can? Or you can do get send mail. But wait, who do you want to send it to? You've got to know who to send it to. So we've got something fun. We even wrote a script for it. Get maintainer. And you can send it a patch. Oh, wait, wait, wait. We're going to do Scripps check patch. It's good. You can send it off now. I've actually done this and it failed. Don't laugh. Get maintainer. So there's a nice script. It goes and looks at the kernel tree. It looks at the maintainer's file. There's a maintainer's file in the directory to who maintains what. And looks at who's actually committed to what file. So if you ever touch the floppy driver, tag you're it. It says to send it to Greg, which is me. Send to Bill's modified it. Test is modified it. And then to mailing lists. Because you want to send to mailing lists. We do all our development in public. So those are the people I want to send this email to. So let's send the email. I want to say to the allcopy.org and copy the kernel. Now you can do something called, oh, what is this? There's an option that you can edit. If you're sending multiple patches, you can say start a patch. The first patch you want to send is a little summary of everything. So you want to say, these patches are going to do this. I'm sending up one file. You don't really need a summary. This one is so simple, it's self-explanatory. So if you're doing these kind of changes, and I really recommend you do these changes, just send me one patch. But if you're sending more than one patch or a patch that depends on another patch, then you need to tell me what order in which to apply them in. Patches for Linux kernel need to do one thing only. Don't be a replace this subsystem. We want to see your work, like your old professor used to say for math. Show me your work. Show me the individual logical steps of what you did. They can be in order. So one depends on the other, depends on the other. And that's really good. Linux kernel works on the fact that other people review it. You have a 15 patch that patch number five didn't do something quite right. So we'll apply the first four, and then you'll take the review and work on the rest of them. And that's work. It's an iterative design process. I wrote the Linux driver core, along with somebody named Pat Michelle. When we came up with this idea, we wanted to unify the way drivers worked in the 2.5 series. Some other kernel developers saw all the work we were starting to do. They said, no, we can make that and make it even a more-based type, sort of a common type in the kernel called k object that all other types of objects can do. And then we realized, hey, that's reference counting. And then we can refactor that and make it a smaller thing. So we have a very, very flexible type system in the kernel. We do multiple inheritance object overloading in C. Virtual functions, hey, it's fun. Some days I wish I had a C++ compiler. But it's showing your work, seeing other people seeing the idea and saying, no, it could be made better. Recently, Google's been in the news for not getting their code into the kernel because they dropped big chunks on the kernel. And they didn't go through the review design, review process. We said, hey, but it wouldn't be great if you tweaked this and did this this way. And you can't just drop big chunks. You can add a whole new driver? That's fine. You're not touching anything else. Add a new file system? Great. If you want to change the core VFS, like riserfs4 wanted to do, no, it's not good. You have to go through review process. Changing little things in drivers is very simple. Changing things in the core kernel isn't so simple. But that's good. You don't want us to be able to modify the core kernel that everybody uses as easily as you can do other things. But we should always be able to take spelling fixes for things. So let's send the mail off. So it'll ask me, who do I want it to come from? Hey, it'll come from me. It guesses based on your IDs. Do that. And if you want to do a reply to, it'll do that. I don't want to do that. And then it'll ask me, hey, I'm going to send this to this person. Copy these people. It's going to look like this. And because it looks through the email, guess, is this OK? It's nice. I'll say yes. It talks to my servers halfway around the world. And it worked. Result OK? And then we do a little notification get. We'll tell you something. Yeah. Yes, how does get send mail know what to do? There's a zillion options. I have in my get config, I say my SMT server is my local SMMTP. And then that has its configuration that knows where to go. So if you Google or you look at the man page for send email, it'll do lots of things. It'll tell you how to connect to Gmail. You can use Gmail remote IMAP to do this, to send patches off. You can point it to somewhere else. You can point to your local send mail. It's extremely configurable that way. It's very good for sending spam, too. It was based on a Pearl script I wrote a long time ago that actually I have a rule in the spam assassin that's named after me that I used to send patches out, but other people got a hold of it. So that's it. So send email off. We'll look in a minute. My server doesn't refresh that fast. Let's see if it really got there. So get maintainer. Get maintainer's fun to know who to send the patch to, but it's also good if you think there's a problem in this file. So let's pick a file in the kernel. Pick a file. Give me a file. Keyboard, what? Which one? All right, driver's head, easy key. OK, Yergy is a head maintainer. Peter worked on the file. Yergy Slabby also worked on it. And you want to send it to the input list and then external. Boom. So if you have a question of that file, hit them all. The Linux kernel has been known to be a not nice place to play. But people respond well, hopefully in ways that might seem aggressive at times. But if you've ever seen internal email lists on technical people or companies, Linux kernel's nice. The thing, people will critique your code. They won't yell at you, hopefully. You don't go and say, wouldn't it be great if you do this? You show code. You do things like that. The best way to get somebody to do work for you, this works once a year. Take advantage of it, but don't abuse it. Make some changes to the kernel that you know are wrong. And then submit it. Send it to the maintainer. Send it to the mailing list. The maintainer will go, oh, crap, that's wrong. I can't take that, but I'll do it right. Because I have to do it right because that might go in instead. So you can do that. But here, you can see who wrote that. Each thing. So if you had a problem with that file, mess with the yearsy. I'll do the same thing. Stage R, drivers, hit. So before I said, remember, we can see who wrote what line of the code and what did it. There you go. You want to see who wrote every single line, who reviewed it. This all came in as one big patch. Oh, here's a chain from Peter. And then I can do get show that commit, and boom. I can see that Peter changed something to be an emit function. So there. And two people reviewed that line. I mean, that's an extremely, extremely powerful thing. Get is very good. And look how fast that. I mean, this is a cheap little laptop. It worked really good. I just picked something random. Anyway, it gets awesome. I'll refresh my mail, and go to the next thing. So here's a checklist for what you need to do. There's actually a checklist in the documentation directory. But it makes sure it builds. We almost forgot that one. Makes sure it's from the right person. Some companies don't let you contribute to the Linux kernel, so make sure you contribute it from your home email address instead. We track who writes what in the kernel. We do a report every year through the Linux foundation. If you send a patch in and I can't figure it out by your domain, you'll receive an email from me in a few months, just asking. You don't have to respond, but we try and keep track of it. We want to know who's doing the work. Other people reported 80% of the work done in the Linux kernel is done by companies, paid contributions. Some people think it's a bad thing. I think it's really good that 20% is done by people who aren't paid. This is their hobby. The problem with that 20% is, if you start doing patches, you will get hired, and that'll be your job. This used to be my hobby. Now I have no hobby. Other kernel developers have known to burn out and start other hobbies, and then they turn that into a company, and then they go back to kernel development. It's happened a lot. So make sure a concise subject. Explain the patch and signed off by. That's it. This is a very, very simple thing to do. Come on. Anybody can do this. Literally. Anybody? Let's see if I can show it. Aha, there's my patch. Sent to me, came to me. Cool. And there it is. Not going to apply it. So you send off a patch. Somebody said, how do you know what to do if nobody responds back? And that's a good question. Give people a week. Some people travel a lot. If they don't say anything in a week, send it to them again. Be persistent. You're not being annoying. You're being persistent. It's OK. I want to be reminded that me as a maintainer, I'm not doing my job, send it off. Keep bugging us. Persistence is the key. It literally is. It really is. Just be persistent. That's all I can say. It works really well. My inbox. Checklist. And Colonel Newby's group. If you're afraid of the next Colonel and you're afraid of asking a dumb question, colonelnewby's.org, very, very good resource. Wonderful Wiki explains what is in each Colonel release in a nice, concise summary. There's an IRC channel that's awesome. It has about 600 people in it, but nobody talking to us. The problem is people show up and they're like, ah, it is not for off-topic stuff. You asking a question, it will be answered. I'm in there. A lot of the Colonel developers are in there. It's a really good spot, that IRC channel. They have a mailing list. There's only one stupid question you can ask on this mailing list, and that's a question that was already asked this week. Read the archives. Just read the archives for a week. If you can't find it there, read. You can send it there. It is a very good mailing list for new people if you have a question about something. Sometimes people think their questions are too basic and too easy, and then they will actually post it to this mailing list like, no, that's a complex question. Ask on the real mailing list. Every single subsystem of the Colonel has their own mailing list. USB. You guys saw the input mailing list. Those mailing lists are very small. They're all open. You don't have to subscribe to them. Anybody can send email to them. They work really, really well. If you want a question from the people who are responsible for the code, who have been maintaining the code, they know it. It's very good in those things. So I suggest using those mailing lists. It's very good. And of course, ah, come on. There, my obligatory penguin picture. So that's it. You guys are all so many patches. If you do a patch for something in the arc arm stuff, do we still send it to you, or do we send it to Linux Orb? Send it to, well, run the get maintainer. It'll tell you who to send it to. If the staging, I'm responsible for all the staging stuff, that was me. There's a lot of work to be done in the staging directory. If you don't know what you want to do, grab something there. Otherwise, it'll tell you who to send it to there. Who's responsible? Hi. Thank you. You showed us the great, powerful tool for blame with Git. But what about those lines of code that were written before we had Git when we were living under the BitKeeper thing? Actually, BitKeeper is really good. BitKeeper made us change our development model. And we knew what we needed out of the source code control tool. But we've imported all the Keeper stuff into Git. It's all there. It's just in a different repo. We can track all the way back to the 0.1 release now. Yeah, we've imported it all into Git. We've imported the old history. It's all in the Git tree. It's a different Git tree, because we don't care. But you can find it. Yeah. How does you test your changes in the Linux kernel? Do you reboot each time you have changed a line of code? Yeah. Do I reboot each time I change a line of code? No. I would have no problem with sending that patch off right now after changing it. And I hope you wouldn't either, because you don't have that driver. So you can't know if it did anything. So you think it's right. It logically looks right. And it compiles right. Send it to me. If you're changing core stuff based on something that you know you want to do, then test it, please. Build it. We see a lot of patches that are sent to us that aren't obviously built because they failed the build. You can't break the build. Every single change in the kernel needs to not break the build. We can bisect things down, and we want to find the bug. Every change we need to build. The kernel changes in about five patches an hour, 24 hours a day, seven days a week. Impressive. I don't see any questions anymore. Anybody has any questions, you can email me. Thank you very much.