 Okay, just a little bit of a delay here. My laptop decided it wanted to reboot, so it was kind of fun. We have EL stickers here, and I was going to hand them out a bit later, but I'm going to toss a little bit at the end of each thing here if you guys can pass them down to the other people on the row there. That would be great. Hopefully I've given everyone enough. Okay, so sorry about that. Slight technical difficulties, and I just wanted to make sure that's working. I was just working on some labs for us later on, and unfortunately it looks like my document just got lost. So I have to do that during the break before we get onto the lab section, but that's fine. As you can tell, a lot of these things have been done last minute. We tried very hard to get all these things finished, but we've had a number of last minute issues and other things happening throughout our instructor, group of instructors. So I apologize for any roughness, and I hope you still enjoy our program we're doing this week. My name is Ben Webster. I'm an embedded consultant. I work for a company called Conversing Code. However, I do a lot of contracting, including contracting the Linux Foundation for both writing courseware surrounding Linux, specifically embedded Linux, as well as being instructor for the Linux Foundation. So I spend quite a lot of my time talking about these sorts of things to people like you. So what I do is what I call engineering-facing work. In other words, my users are engineers, and that's because I like talking to engineers the rest of the world is insane. So this is a whole lot more fun. I've been doing this sort of thing. Well, I've been in the industry for close to 25 years. Started off in telecom, moved to Datacom, more or less optical. Oh gracious, the gaming industry, I've done some defense work, medical, automotive, lots of different areas. However, always at the systems level, always engineering-facing, always doing things from board bring up through bootloaders, through kernels, drivers, compilers, various tools, libraries. I don't go much above that, even though I've done that work, I like stopping there. User-facing stuff is something other people are better at than me. But anything below that area is probably something that I've either worked on or worked with somebody on. So that's why I really like a lot of this low-level stuff. I'm also an early Debian developer. The app tool was one of the things I worked early on and other things along those lines, and what kind of questions you can ask. Today we're going to be talking about something that's rather interesting. I've been using Linux since the early 90s, and I've been doing it professionally and not behind the scenes because a lot of us use Linux despite the fact that we're supposed to be using Solaris or HPX or Windows on the machine under our desks or dual booted. I've been using it professionally since the late 90s, and my kernel work took a few more years than that, and part of it was because I didn't entirely understand this whole kernel module thing. I would make a lot of my changes to the kernel proper, reboot it, test things, and go back and doing things. So modules are actually your gateway to the rest of the kernel, and that's what we're going to talk about today. Who here has actually written a kernel module before? A couple people. Okay, what the heck is a kernel module? What does it do exactly? Why do we have it? So the big thing to get across to people about modules, a lot of people think that drivers are modules. Okay, like a driver and a module, same thing, right? The fact is that all the module is, is the way of getting information into the kernel. It's just bytes, a payload of bytes that goes into the kernel. That payload can carry whatever you want. Now it could be a driver, okay? But it could also be a library. It could just be a string. It could be a K-probe. It doesn't matter. Whatever you want to put into the kernel is what goes in there. And the neat thing about modules is not only can you put them into the kernel, but you can pull them back out again. Now that sounds funny that I said that, but in fact, adding code to a running kernel is easy. Removing it from a running kernel is actually a lot harder, okay? And so we're not talking kernel patching here. We're actually talking about adding code dynamically into various systems. So there's actually quite a lot of infrastructure and checking that has to happen before we pull code out so we don't crash our computers. The other thing that modules are used for is for automatically loading only the drivers we need. So most of you right now just looking around the room have PCs. Many of you also have Macs like I have. But the thing about all these machines is they're essentially the same. They're all Intel boxes. Some of them might be AMD, but they're all Intel boxes. They are roughly the same, right? There's a lot of ACPI glue and other things that make them look roughly the same. However, they are quite different. And there are things that – there's different controllers for graphics and for sounds and for lots of little bits and pieces. And we don't need to have all of those drivers in all the time. In the early days, we'd have a kernel that had everything compiled in. These days, if we did that, we'd run out of memory in a lot of machines, right? So we only load what we need. And so what we do is we build them as modules and then we use the initial RAM disk. It's a little memory-based RAM disk. And we load our modules from that. And so this means that we can dynamically figure what we need and only load what we need. That means we minimize the amount of memory that's actually used in the kernel. Part of the problem is because of the memory model. The kernel works. Kernel memory is what's called non-pageable. What that basically means is like a real-time operating system, when you allocate memory, it's there forever, okay? So we cannot use that memory for other things. In user space, of course, memory pages in and out all the time, right? But we have to be careful about kernel memory because literally nobody else can use it, okay? This is different than some other operating systems, but it has pluses and minuses. So automatically loading drivers, good. It also allows us to have more than one kind of driver where we load one or the other. Typically you can't load more than one because as soon as you take over a piece of hardware, another driver coming along could cause problems. It also allows us, of course, to architect our kernel in a very modular way so we can actually make things happen appropriately. So from an architecture perspective it works really well. In the early days, whenever we wanted to make changes to the kernel, you had to do it sort of the normal way you would expect. You'd make your kernel changes. You'd recompile. You'd reinstall. You'd reboot. It would fail. You'd go back to your old kernel and you'd keep doing that until it worked. And that was kind of slow. Eventually we had things like VirtualBox and VMs, basically, hypervisors. And this allowed us to basically reboot the VMs. And I'm sure everyone here, or many people here are using VMs. They're very nice in many ways. And they're really great for working in kernel code because, of course, when you reboot, you can reboot faster. Some other words, your failures are less awful. However, working in modules is even better. And that's because the vast majority of mistakes that you're going to make, you can undo by removing the module. Now, if you overwrite the wrong piece of memory or do something dumb, you're going to cause problems. Now in this class today, we're going to be doing some labs a little bit later where we're actually going to add code to the kernel. We're not going to be doing anything too horrible but do bear in mind that the code that you add to the kernel has the ability to pretty much do whatever you want. Once you're inside the kernel, there are some checks and balances, but in our particular case, there are none. In this particular situation, we're not using any kind of extra security infrastructures that can prevent things. So pretty much if you misstep, you're going to kill your computer. So it's a very powerful way of looking at the world. It's also very dangerous way of looking at the world, but it's a lot of fun as a result. Anyway, modules are great because you can add and remove them. It tries to protect you very much from doing the wrong thing. So again, as it says here, aren't modules and drivers the same things? Well, the thing to remember is this. Most drivers are written as modules because it's easy because we can do our work without having to reboot all the time. Developers, especially good developers, are fundamentally lazy. We want to do as little work as possible to get as much done as possible. And if you're not lazy, you're doing it wrong. So the great thing is that writing something as a module, and there's only a few places in the kernel that we can't do this. But if you write it as a module, not only can you write it the lazy way, but you can also make it so it's modular and dynamically loaded later if it's not there. You can pull it out. You can mess around with hardware by pulling it up, putting it back in again. There's lots of really great ways. And on top of that, by making it a module, you're using a very modular design, and in fact, your code is better as well. So there's lots of really good value there. If you look at the wireless stack, this is a really good example of where the 802.11 wireless stack was originally baked into every single driver. And eventually they realized, you know what, it's all the same. And they made essentially a library module that implemented just the 802.11 stack, and then everybody had their own extra pieces. You just depend on that one module and away you go. But this is the thing, most drivers are modules. Not all modules are drivers. So that's the thing to remember. And again, like I said, it's just a payload. It's a bunch of bytes that you're putting into the kernel. That's it. You can do whatever you want. You can just put a giant string in there if you want to. It wouldn't do anything useful, but you could do it. And yeah, as it says down here, you can implement driver, library code, K Probe, pretty much whatever you can dream of that makes sense to put in there, you can do. And there's all sorts of really crazy things that you can come up with, I'm sure. So here's a very simple module. It's off to the side here. It's a little hard to see. Unfortunately, the height is not great. But I want to walk through this here. So I think people at the back are going to have a harder time seeing it. But this is about as simple a module as you can possibly put together. It does absolutely nothing except print K. And print K is the way that we can print things off to the kernel buffer. So let's look at it in a lot of depth here, and we'll cover these points over here on the right. The first thing is if you look at the top, you see there's two include files. There's Linux module.h and Linux module.h, init.h rather. And the thing to remember is these are things we'll tell you what the rest of the file is doing. The first thing is module.h. This means it's, guess what, a module. This brings in all the machinery to do the kinds of things we're about to talk about. The next one, init. This means it has initialization software in it, or routines in it. And the thing about the init code is it again provides the machinery we're about to talk about that allows us to automatically load things and unload things and do the appropriate things that's in the middle here. So all the machinery we're about to talk about in these two header files. Looking at modules, the next thing typically to do is to go to the very bottom of the file. And that's because this is the conventional place to put metadata. And the metadata we have at the bottom largely starts with module underscore something. Now there's two things that you probably should put in there. Everything else is optional, but there's lots of other metadata beyond this. But those two pieces are module author. So we know who to be upset with when it doesn't work. That's you, by the way. And module license. And the license is important for later. The license basically ties into the internal linker which is used during intsmod. We'll talk more about licenses later. But you'll notice that we put gplv2 in there. This isn't a political statement. It's just that we will find out later on that there are certain things that don't work unless you actually have it using a compatible gplv2 license. So that's sort of the default spot for most people writing code. Unless you have a really good reason not to. And there's six licenses that we can use that are gpl compatible. We will talk about that in the future slide. So next, immediately above it, you'll see we have module init and module exit. And again, this will be easier to read in the slides. I was trying to upload them just before this, but apparently everybody came in and the Internet just slowed down to nothing. Sadly, they're sitting on my laptop right when my laptop rebooted. I was trying to upload them and that's probably what caused it to go over the edge. Anyway, so module init and module exit, these two macros tell, well, they're part of what compiles modules in such a way that those two function pointers are indicated as being the initialization code and the exit code for this module. So it doesn't matter what we call the edit and exit code. As long as we ultimately tell the system, this is my init code. This is what gets run when I'm initialized. When I'm removed, this is the code that needs to run. So it's just the starting point, just the function pointers. These are the two routines up here. And it's my init, in this case, and my exit. The names don't matter. They really have to match down here. Notice there's a couple of things here. First of all, they're both static functions. Who can tell me what a static function doesn't see? The best answer, there's a few that are close, but basically it changes the scope. It's no one good global. It would be what's considered global to the file, but statically scoped so it doesn't go beyond the file. So yes, scoped to the file is correct. And so what this essentially means is that when we link this into other code, essentially we cannot use these beyond the edge of the compilable object, the file. The reason why that is is because we don't care about other people calling this code directly. Who calls this stuff? The initialization system. How does the initialization system do it? By function pointer. So the name actually doesn't matter. Now, because you're good programmers, you're going to give it a unique name, ideally in your own namespace. If you're writing a driver on foo, you call it foo underscore init. However, because it's static, it largely doesn't matter. The next thing you'll notice is between these two is that one has a return code and the other one doesn't. So initialization code as an int as a return code avoid has a null return code. So in the case of an exit, we actually don't care about the return code. Why? Because literally exit codes are only called in a situation where you need to guarantee that the exit code will succeed. There's no such thing as a failure in exit code. Because exit code is protected by using a reference count. The reference count is at zero. You are saying in your contract with the kernel that the exit code will run successfully. So you have to architect your exit code so it will work appropriately. You should still check your error codes because again, good programmers do that, but literally you cannot report them to anything the exit code is still going to finish. Inmit code on the other hand, lots of failures can happen. That's important. So I'm sure everyone here has done some programming before. Not necessarily on a UNIX or Linux. It's slightly different depending on your operating system you're used to. What is success as far as a return code is in UNIX? Right, and that's because the return code means no error. So zero means no error. Error codes are what typically? Positive or negative? They're negative. And again, that's just convention. That's just the way it is. Let's say signed integer that we have. Positive numbers are typically things like sizes and so on and so forth. So zero is success. Negative number is an error. And we have a series of error numbers that are defined in user space that the C-libraries use. And the kernel uses exactly the same ones. In fact, that's where the C-library gets a lot of its error codes. They are shared between the two of them. It means if you have an error in your code you should throw one of those, not throw a rather, return one of those negative error numbers in your code. Why? Because the rest of the kernel will understand what it means. And when it goes back into user space, because presumably you've triggered it from a C routine, it will understand what it means. And the ideal situation is it will then spit out your error in English or French or Greek or whatever you've got your language set to. Because again, the C-library is internationalized. So the really cool thing is that by returning the appropriate error number, guess what? Something useful gets sent back to your user space engineer, which of course is you when you're writing your code. So the error code is important. So unless you return zero, your module actually doesn't get inserted. So when you insert your code, you run your init code. It returns non-zero. It throws away the memory you just added. All right? So what does this mean? In the kernel, it's not like user space. In user space, if you open a file, you allocate memory, you do something that uses up resources, and then you terminate your program. What happens? It's all cleaned up, right? All the files are closed. The memories return to the kernel and so on and so forth. That's not true in the kernel. So if you allocate or put something together or reserve something in the init code, before you return an error code, you need to undo all that stuff. All right? So keep your init code really simple, but you'll find that when you're messing around with things, you'll do something complicated. Just remember that anything that you don't clean up in your init code, you're going to lose. Or potentially, if it's something really important, you're going to crash your machine. Okay? If you allocate something that deals with code in your module and then your module goes away, when somebody follows up pointer, guess what? Non-pageable memory, you're going to crash. Okay? That's the way the thing works. I'll take questions, but just in a second after I go through a few more things. So in this case, we're just doing a print K and then we're returning zero. Nothing too exciting. Again, in our exit code, we're just doing a print K just so you can see it actually happen. We use this in the labs to make this work properly. So next thing, we've got these little things here, underscore, underscore init, and underscore underscore exit. And these are actually kind of clever. These actually use what are called attributes. Now, attributes are not something that are in the C standard. However, there's something that everybody implements and they allow you to actually attach metadata to symbols. Symbols, of course, are either functions or variables or what have you. In this particular case, these are telling the kernel where to put these symbols into the linker, so different sections of the file. And I'll show you that file in a second. Underscore underscore init will add this specific piece of code which is usually stored in the .text portion of the else file. It instead puts it in a separate section. I'm just going to go to the next slide for a second. This is what an else file module, this is what your module looks like, but this is an else file. You'll see we have different sections. We've got text, init text, exit text, RO data, data. This is read only data. This is data. Text, incidentally, is called text even though it's binary because really computers, of course, were interpreters and therefore code was text. These days, of course, it's just a digital thing that we still use, but you'll see we have init text and exit text. And this is what these two entries do. Basically this means this function, this symbol, is going to be put not into the text section but into the init text section. The exit code is not going to go into the text section but into the exit text section. Now this is pretty interesting. Bear in mind that every piece of code ultimately is loaded into memory. Everything that's loaded into memory is done page at a time and this has to do with how memory and hardware works. So it's not always the case, but pages are usually on the order of 4k in size. So this doesn't look very efficient, does it? We've got very little code here and yet we have three pages already. We've got text, we've got init text, we've got exit text. That seems like a silly idea. However, it's actually quite clever. So obviously the regular code has to go into the kernel. We can't run it otherwise. However, for the initialization text, do we need our initialization code after we've initialized our module? No. We're never going to call it again. We only initialize once. So remember I said it's non-pageable in memory. I know we have a ton of memory on big systems now but on our embedded systems, even a gig. It's not a lot. It doesn't go very far. So the more memory we can reclaim, the better. So after a successful module initialization or at the end of system initialization, in the case it was built in, initialization memory is freed. So this is kind of handy. So if it's in its own page with only things you need at initialization time, guess what? You can just reclaim it. That's all good. Now module code, module code can be built as a module or it can be built as what's called a built-in. In other words it can be put into the kernel to always be there. Now the thing I have is obviously in the case of initialization code at boot time once we're finished with initialization we can clean up that memory. That's fine. However, do we need exit code in a built-in situation? Why don't we need exit code? We're never going to call it, right? We can't remove that code. It's built into the kernel. Should we link it into the kernel and waste memory? No. So what do we do? When we link the kernel we throw it away. So we built it as a built-in. This exit code just is never linked. It's compiled. It doesn't get linked. The great thing about this is in the early days of modules we had to put if, then, else everywhere. Like if, you know, if-desk module, blah, blah, blah. It meant a lot of weird code that had to be done. In this case, look how clean it is. Just by designating something as exit it gets used appropriately. Designating things as init makes things appropriate and so on. We don't only do this for initialization code. We can do it for data, other functions, as long as we understand that everything marked in it is going away after initialization and exit code will not be there in a built-in situation. So let's talk about some theoretical here. Can initialization code call regular code? In other words, just the normal code that's in the kernel? Okay, I'm seeing nods. Yes. I mean, that makes sense, right? We have to be able to initialize ourselves. Can regular code call initialization code? That'd be bad, right? What happens if you call memory that's not there in the kernel? Remember, we're not pageable. So if we hit a page fault handler in the kernel, because something's not there, we die. Okay, so regular code calls initialization code, and we're past initialization, obviously bad things will happen. Or worse, something else might be there and we'll be running random code. Okay, that's bad too. So regular code cannot call initialization code. Can exit code call regular code? Yeah, right, otherwise we can't exit. We need those pieces of code to exit. Can regular code call exit code? Well, if you load it as a module, that's actually fine. It will work incidentally. Okay, in the case of a built-in, you'll crash your machine. Okay, so in both cases, this is what's called a section mismatch. And you'll find at the end of building a module, there's a step called ModPost. ModPost will point out these section mismatches. If you see a section mismatch, look for those kinds of problems. Okay, it'll actually tear apart your symbol table and actually look for those kinds of problems. So the reason why I like this code is it's so incredibly simple, but it actually shows you a lot of things that are there and happening behind the scenes. And we can actually look through the module, the macros that implement these things and such too. We won't, but you can actually go even deeper than this, which is kind of interesting. Okay, but that's essentially the template that you always start with. Alright, in fact, that's what our first slide will be, is building this and playing with some of the utilities for adding and removing code from our kernel. Now I think there was a question. I think over here first. No, that's a good question. That actually happens a lot. No, your exit code is never, it's brought into memory, but you have to clean up everything you're in it code. So I should have repeated the previous question. I will repeat this one. However, the question was basically, is the compiler smart enough to know what is in it and what is exit code? So it does the appropriate thing, and the answer is no. Compilers generally can do a lot of things. They cannot guess at things like that, nor would you really want it to. You have to explicitly label them with attributes to indicate what's in it and what isn't. It will check afterwards to see whether or not you've made a mistake as a section mismatch. But if code is only ever called as an init situation, it can't flag that. From its perspective, it's a single compilation unit. I mean, it would be potentially possible through static analysis and such to figure that information out. At this time, this is how we do it. Yeah, so that's that over here. So exit code is only called in a situation where we can guaranteed remove a module. So there's a reference count on the system. The reference count of zero means nobody references any memory in our module. So whenever we actually do two things, and we'll show this a bit later when we talk about LSMOD, but there's two kinds of dependencies. There's explicit and implicit dependencies on a module. They both increase the reference count. But when the reference count is zero, theoretically nobody references our memory and therefore it's safe to remove it. So theoretically, assuming you haven't made a mistake that hasn't increased the reference counter, theoretically it's always safe to call your exit code. And usually you also make sure that by essentially doing the appropriate blocking calls and locks and such to make sure the things you're dealing with are appropriately dealt with. But you have to write your exit code in such a way that it succeeds. Any other questions? Okay. So again, these are the important parts of an ELF file. It's important to understand that a kernel object or a module is just an ELF file. It just has a few extra bits in it. I've actually removed those extra sections. But again, you'll just see that we have text, which is regular code. We have, of course, our initialization and exit text. We also have things like read-only data that's put there by the kernel, initialized data down here, and a few other things, there's BSS and a bunch of other things as well. And again, you'll see that we have things like alignment information, how big things are, and other things like that, the offset into the file. So lots of really good things. We'll talk about that with the object dump. So again, you can mark two different kinds of initialization. You can mark both initialization data and initialization functions. Now, being a bit of a compiler geek, I just want to point a few things out here. You'll notice that for a variable here, right, you'll notice that we have the underscore init data at the end. So here we've got the specifier, we've got the actual type. We have the name of the symbol, and then we have the attribute at the end. This is because attributes are left associatives. In other words, they deal with the thing to the left. So strictly speaking, it belongs here. Now, the reason why this is is because we are saying this symbol is initialization data, which is what we want to say. Now, down here, you'll see we actually say void underscore init, some function, whatever. This, however, is actually associating with the type. Now, the way compilers work usually is that if you mess around with the return type, it, of course, drags the function along with it into that section. So making any part of a function init in this case will drag the whole thing along with it. Now, in the case of GCC, it will in fact work that way. So by putting init there, in fact, it will put some function into the appropriate section of the linker. However, Clang, which is one of the things I've worked on in the past, Clang, in fact, follows a stricter definition of how these kinds of things work. It sticks to purely left associative, except in this one situation. And that's because so many GCC people write code like this because it's pretty, and I'm not being facetious that they prefer it there. They think it looks nicer. It actually belongs here after the return type, and I have to say it doesn't look right there. That's because we're used to having it over here. Now, in the case of Clang, Clang does not allow you to put attributes on types only on symbols. However, they have an override. In this particular case, this is so common that, in fact, what they do is they make it right associative in this particular situation. So in the case of a function, they've got some weird code to make it work around. And what you'll find is in the kernel code, this is sometimes over here, right? And worse than that, sometimes you've got init somewhere in the middle of a type. Don't put something in the middle of a type. If you say const init, pardon me, const underscore init int, you're actually putting the attribute in the middle of a type. Const int is the type. Keep your types together, put your init attributes at the end. Otherwise, you're going to break compilers like Clang. So this is the agreed upon way that we do things in the kernel. Again, strictly speaking, this should be over here, but the kernel developers have decided they prefer having it over there. Again, just not in the middle of a type. Now, if you actually built things as a built-in, you'll probably see something that looks like this at boot time. There's your print k time off to the side. This says it's freeing unused memory, and here it's freeing some memory immediately after you usually see init starting up, right? The user space starts. But this is essentially all of the kernel module init memory that is being freed at boot time. And again, it's not that much memory these days, but unembedded, that's still quite a lot. Was there a question? I thought I had to go up, okay, fair enough. All right, let's talk licensing now. Again, we have a license that we apply per module. We'll see why in just a second. But there's actually six different licenses. Well, actually, strictly speaking, it's one license with some extra bits. But six different variations that we can use that actually make it compatible with the global license that's on the Linux kernel code, which is GPLv2. Essentially, if you were not one of these six licenses, you were considered to be not GPL compatible, and therefore proprietary. Now, strictly speaking, that isn't necessarily the case. However, as far as the kernel is concerned, you're not GPL compliant, and therefore it won't do certain things with you. And again, we'll see in a second why. So GPL, GPLv2, there's a distinction. GPL means GPLv2 or newer, okay? So it means if the kernel ever gets changed to GPLv3, or who knows, v4, it gets dragged along with it. So in this case, you're just saying it's GPL something. But it's at least GPLv2. In the case of specifying v2, it means it's v2 and v2 only. If somebody ever wants to change the kernel's license, they're going to have to come and ask you, as a copyright holder, are you okay moving to this new license? So it's a very specific license. GPL additional rights, this allows you to say v2 plus some extra things. So you're allowed to add your own addendums to GPL. And that's what this means, essentially. So it's not strictly v2, it's v2 and some extra bits. There's also three dual licenses that you can do. Dual BSD GPL, dual MPL, GPL and dual MIT GPL. BSD, MPL, and MIT are similar in their ability to allow you to use that code in a commercial context. But otherwise, you're saying you have a choice between using one or the other license. And in most cases, when you use it as a part of the kernel, of course you're choosing the GPL v2 side of that equation. Pretty much everything else is considered non-GPL and therefore proprietary. Strictly speaking, one could argue that it's not proprietary, but from the kernel's perspective it's not free. Now if you go down to the side here, you'll see we've got another column that says tainted. Taint is actually kind of interesting. It was added at a time where originally when the kernel came out, proprietary code was okay in the kernel, I should say. And the reason was because driver writers would write proprietary drivers. And the proprietary drivers would essentially came from the fact that everybody else was doing the same thing. When you've got a Windows machine, you've got proprietary drivers. And so everyone was in that kind of mindset. So initially, people were okay with it. But the problem that we have is that when a module or rather when a kernel dies, where did it die? If it died in a place where you can get kernel access or memory, in other words, the code for things, great. But the problem is that if you've got uncontrolled code in your kernel, what caused the kernel to crash? Was it the code that's in the publicly available part of the kernel? Or was it in the proprietary code? Was it one of your modules that they don't know about? So we needed a way to basically figure out what was going on in the kernel when it crashed. And if you actually look in an oops statement, in other words, they crash-stumped for your kernel, it will actually indicate what sorts of things that you've actually done to your kernel over time that are saved in such a way that when the kernel developer that looks at it and goes, well, actually, you had something going on here, try to recreate the problem without this load and get back to me. Because too many people were wasting their time on solving problems in a module when it ended up being their graphics driver from the proprietary vendor that knows what they're doing. And it's not bad. The graphics driver is great. You can play games on it. You can do all sorts of neat 3D stuff. It's all great. The problem is we don't know what it's doing to the rest of the kernel. It's purely a saving time and trying to get stuff done issue. So taint bits were added. And taint bits are kind of fun. So the original taint bit was the proprietary module taint bit. So as you can see here, in fact, we actually go just beyond the edge of a 16-bit value. There's actually 32 taint bits at the moment. We only use roughly half of them. The first one indicates that you have a proprietary module loaded. So the very first time you load a module that's proprietary, it will mark the proprietary taint bits in the module and globally in the kernel. Even if you remove the module, the taint bit is still set. Okay, that's the same with all the taint bits. The next one, a forced module. So it turns out you can load a module normally into a regular kernel. If you have kernel version magic turned on, you can actually load an older kernel module into a newer kernel assuming that the ABI of the kernel didn't change. Or the API, I suppose. However, you can force that if you know for a fact that the ABI change that's happened doesn't strictly affect what you're doing, you can say, stick it in there anyway. Okay, the problem that we have here, however, is one where you know that there's a potential bad thing. You've been told there's a potential bad thing, but you've still done it. Okay, so in this case, a taint bit is set because essentially you're doing something that is kind of dangerous. Just making sure I've got the time. I like to tell the story about Binky the Bear at this point. Anybody know of Binky? Nobody? Okay, so when I was graduating university there was a polar bear in Alaska at the zoo. His name was Binky. Look it up, it's on Wikipedia. There's actually videos in YouTube too. So Binky was a polar bear. Has anybody met a polar bear here before? They eat whatever they come across because there's not a lot of food up north. So in other words, they eat humans because we're just meat on two feet. So essentially what happened was at the zoo they had a fence. So they had Binky in the thing. They then had these bars to keep the polar bear back then. They then had another fence to do not cross. And then they had a third fence over here. And so the whole idea was to see Binky, you were standing behind three different fences. Makes sense, everyone's safe, right? Polar bears are cute. They're really fuzzy and everyone likes them. They drink Coke on TV. They do all sorts of really neat stuff. So what do people do? They climb over the first fence. They climb over the second fence. They take pictures sometimes through the last fence. And so what happened essentially was a tourist from the Southern U.S. went up to there and she decided to get a picture of the polar bear who was sleeping. And she climbed over both fences, stuck her arms through the fence, take a picture, looked back, looked back, and Binky was right there kind of pull her through the fence. And so they managed to get her away and there's this footage of people basically beating at the polar bear with some wood around and so on and so forth to try and get it to let go. And she was hospitalized, she lived. But the problem was is that a number of people were calling for the bear to be put down, right? It attacked a human. That's usually the bar for having an animal destroyed. All the locals sort of said, Polar bear, right? Of course it attacked her. Like seriously, right? So what I like to tell this story is that these are all Binky situations, okay? There's three fences telling you not to do this, okay? When you say force load a module, you're going right up to the bars and you're saying, I dare you to crash. All right? So if you crash at this point, it's your own fault. That's where we set the taint bit, okay? And there's all sorts of these kinds of Darwinian events throughout here, all right? CPU out of spec, okay? If you're overclocking your CPU, right? Again, you've crossed some fences here. Forced RM mod, okay? Remember I said your reference count has to be zero? You can actually do an RM mod minus F. You can force the removal. This is sometimes useful because you made a mistake. You allocated something that you should, you forgot to deallocate or what have you, and there's ways around that. But maybe it's easier than rebooting just to force remove the module. Maybe you know better. So I'll just tell you right now in my entire career, I've known better twice every other time my kernels crashed. Okay? So again, we will set that taint bit because you've done something that's inadvisable. Again, rebooting takes a long time. People are lazy, whatever else, right? Give it a try. But you can actually turn on and off forced module removal when you build your kernel. Machine check, right? If you get an MCE, it's a hardware problem, right? That isn't something you did, but literally your kernel is that much closer to failure because there's been a hardware problem of some sort. Bad page, bad memory, right? Again, same kind of problem. Taint user, you can set taint bits from user space. Taint die, in the case of different kinds of things dying inside the kernel, that one's set. Overridden ACPI table. Again, you're not reading the manual. You're saying, I don't know, I think it should be done this way instead. Again, sometimes there's bugs and so on. Taint worn, okay? So something happened that might be a problem, might not be, but you set the tape in anyway. And that's where we stopped after that. The next one is taint crap because it was for everything else. You did something that was awful, taint crap. Okay, they didn't want to define any more taint bits because they could go on forever. Okay, so you just did something dumb, the crap that was set. However, a few more things have been added in the last few years, and in fact every few kernel versions, in fact the new ones have been added. In fact, there's a 17th one that's been added down here that I removed because I forget what it means. The firmware worked around here. In this case, they mean firmware in the sense of boot firmware. This is an x86 boot firmware thing. In other words, again, you're working around a specific problem. You're not following the written manual. More or less, again, you're crossing senses. This is one we're going to actually set today. The taint OOT module or out of tree module. We're going to be building some modules out of the kernel tree. Again, uncontrolled code. You're putting code in from you that the other developers can't necessarily see. On sign modules, we have assigned kernel now. If you have assigned kernel, and you're allowed to put unsigned kernel code into the kernel, guess what? Taint bit is set. Soft lockup. This is where CPU basically, on a multi-CPU system, basically locks up and then sometimes can come back again. But again, there's been a potential problem with the scheduler, with hardware. Who knows? Again, we're marketing a problem. Live patching. This is kind of a neat thing. We can actually live patch running code. This is something I did in telecom years and years ago at Nortel Networks. Because you don't want to reboot a telephone switch because 911 goes down. Another thing that gets you fined. We have this now in the Linux kernel as for a couple of years. Again, we are actually changing running code. This isn't just a module. We're actually taking portions of the code that are actually doing stuff right now and we're changing it. We typically do this to get around really difficult security problems. However, again, we are massively moving on from what is normal in the kernel. We're doing some terrible things. Again, this is not a long-term thing. This is a get me to the next maintenance window thing. But again, TaintBit set. And last, TaintOx. There's a number of auxiliary situations where this TaintBit is set. We had a couple questions over here. If you have the original signing key for your kernel, yes. And if you are the person making your kernel, you presumably have that key. If, however, you are adding a module to a vendor kernel, some of you are running an Ubuntu kernel or a Fedora kernel. If you have access to that key, so does every other virus and hacker in the world, and that's not very useful. So essentially, by adding a module to your Ubuntu kernel or your Fedora kernel, basically you're tainting your kernel. You're putting code in there that you can't sign. Does that make sense? There's another question over here too? Yeah. So the TaintBits are just informational. Nothing reads them except you in your UPS messages. So you could have every single TaintBit set. And if you're still running, it's basically a miracle. Because quite frankly, it's very likely you've done something horrible enough that Polar Bear has got you. But literally, if all of them actually didn't matter, maybe your MCE was for something you don't care about. Maybe the bad page was memory you never touch, which is unlikely. But there are situations where it is possible. It's highly unlikely. But just because a TaintBit set doesn't mean a problem has occurred. Again, today we are going to set these two bits because essentially we do not have the signing key for the kernel we were using, nor do we build the entry. Okay? Exporting symbols. You'll notice that everything was essentially statically exported. And what that essentially means is if you actually want to have dependencies between modules, you actually have to explicitly export them. And the way we do this is first, by understanding how we implement this, there's several linkers involved when we actually build a kernel. There's the obviously C linker, right? And the C linker essentially does the initial act of putting the kernel together in a built-in situation. It will take multiple pieces of C code and make it into modules because although in most cases it's one C file to one module, in various situations we have multiple C files. Okay? To build the module, you're using the linker that comes with Benutil. However, when you're putting a module into the kernel, when you're actually linking those two together, there's actually a linker in the kernel that does that work. And that kernel as a result can enforce certain things, things like licenses, things like making sure the appropriate things happen, that ABIs match, and so on and so forth. So the in kernel linker is actually pretty smart. So what this essentially means is that most kernel code in fact is static. We actually make it all static in the C situation because quite frankly, as far as the kernel is concerned, everything is static unless you explicitly export it. And so in that case, we export symbol macro. And that doesn't matter whether it's a function or a piece of data. You actually tell the symbol table for the module, I want other people to be able to see this as a symbol. Now obviously in this case you need to choose a unique name. This is going to go into the global symbol table. Now what this normally means is it's generally the name of your subsystem or the name of your module underscore something. Just to be really safe. But you could in fact, in horrible situations in fact, provide something that was named something nothing to do with your module. And in the case of debugging, this is actually kind of an often occurrence. The fascinating thing about systems programming and writing viruses is that the skill sets overlap. There's a lot of very similar techniques used in both situations. This one is entirely legitimate. The other one is who knows. Export symbol GPL. This is how we actually indicate that a symbol is GPL only. What this means is it will only link to other GPL code. Now this isn't used a lot, but there are a number of internal powerful APIs that will only work with other GPL symbols, with modules. What this means essentially is if your module is not GPL, it will not in fact link to certain subsystems within the kernel. This is actually kind of an interesting thing because you're essentially saying anybody who wants to use my code has to play by the same rules I'm playing by. Now the really fascinating thing about this is this is primarily used by free software advocates to say I really like this code, only be used in situations where amongst other things companies are doing are following the GPL entirely. In practice I always argue this is something that companies should be using. Why should they do that? Because they are releasing their code under GPL so other people can use it. By setting it as a symbol of GPL they're basically making it so that all of their competitors also have to play under the same rules. So it's actually quite interesting. In this particular case this is actually quite an interesting thing for a company to do to make sure that everybody else is playing on the same level playing field. By not using this one, by using this one up here in fact you can write up a proprietary module that uses your interface. Maybe that's fine, that's all good. If you don't want it to be you can make your symbol GPL instead. You can also do this, GPL future. So if you've changed your mind, let's say it's an exported symbol but you'd like to make it GPL there is a code to follow and that code is you warn people that's what you're going to do and so what you do is you first market as being symbol GPL future. It's an exported symbol. However, you will get a warning in your kernel log that in fact it will be GPL in the future. This gives people time to see the problem and actually work around it or solve the problem or change their license or negotiate with you not to do it or whatever. So usually it's two versions of the kernel is the accepted one before you move it to GPL. Now some of you certainly in the embedded world might be using vendor kernels and you might go from a really old kernel to a really new kernel that skips many different intervening kernels. This is a problem. This is one of the reasons why using vendor kernels is dangerous. Tracking mainline and actually having somebody track mainline means you will catch these problems as they happen. I want to say problems changes, I should say they're not strictly they're only problems because you have to react to them. There's a bunch of other export symbol ones I didn't provide them all here but there's things to do with per CPU variables and a bunch of other stuff, unused variables and so on. But there's more than just these three these three are just probably the most useful of all the ones that are there. Kbuild. So in the case of Kbuild this is the actual build system that the kernel uses. It's rather popular and in fact it's so popular that these days almost every other embedded code base uses it except for one. Open embedded and therefore the Octo project do not use it. But UBoot, UCLibSea, BusyBox, Crosstool, obviously the kernel, Buildroot, everybody else uses Kbuild and the reason is is because everybody wants to use menu config. That's a menu system that comes up and you can choose what you want to do and so on. We're about to talk about it. It means your sub directory is kind of very simple make files. Very complicated things. You just list what you want. Again, we'll look at that in just a second. The configuration is very simple. It uses something called Kconfig to define what your variables are, how they work, what they do, some help and so on. And more than menu config there are actually several other graphical environments that allow you to modify your configuration options that are stored in a file called .config in your top level build directory. So menu config is merely the one I will show you because it is the most common. It's perhaps not the prettiest nor perhaps is the most mouse oriented one. However, it will work over SSH and everybody supports it. It's a de facto everyone does. The other ones do similar things but they don't always work depending on what subsystem you're on. So if you learn menu config, it will work everywhere. And it's based on end curses. Just in case people want to know. So this is basically how Kbuild works. You've got a .config file in it you have a bunch of different config operations. .config key value pairs rather. In the kernel they always start with config underscore something. Always in caps. And we then make them equal usually y for yes or build this thing in and for module. In other words build it as a module which is what we're showing you in this example. If you do not set it to anything it just becomes nothing, an empty string. However, some variables are set to numbers. Some are set to strings. There's a number of things you can do. You can do lists and so on. In your make file it becomes very simple. They start looking a little bit like this. There are several different lists in the kernel. Obj y and obj m. Everything in the list obj y is a built in. Everything in the list obj m is a module. However, you will notice that we have config foo up here and we have obj-config foo down here. Because this is an m we have foo.o being added to the obj-m list. Therefore we built as a module. So it's a very simple mechanism but this allows us to do is we can set this to a yes, we can set it to an m or we can set it to nothing which is a no. This allows us to configure how our kernel is going to be put together. Again, we can statically always add things. In this case we always think bar is always a built in and down here you'll see that in fact we're using it as a way of selecting a directory. What that means is we will actually recurse into that directory and look at configuration options within that. How do we know that? It ends in a slash. In other words, if this is set to no we don't even walk into that directory. This is why certain subdirectories in menu config disappear when other things aren't set. Kconfig. Kconfig is the language that we use to define what ultimately works and it actually allows you to define a number of attributes based on those configuration options. Again, in the kernel there are always configs something in other things like build root or busybox or br for build root busybox or busybox and so on and so forth. But you get the type the default value there's different health values you can set dependencies. When you turn a configuration option on what else gets selected at the same time there's conditional selects and other kinds of things like that. Which looks basically like this. Here's a very simple and silly contrived example. config foo in this case it defines the config underscore foo value. Notice that the prefix doesn't matter. We don't put that in there. So that's implicit by kbuild that it's config underscore. In this case it's a tri-state. We could have made it a bool, we could have made it a string, we could have made it a number. Tri-state it means yes, no, or module. And you'll see at the end, I'm missing a double quote here, this is the one-liner description this is what you see in menu config when you wander down. It's a very short I sometimes call this the Twitter version of the help. It's a very very short, very useless, not very helpful but enough information to know what the thing does. We have our default after this. In this case we've made it default m. We've said it depends on this. This means that config blah must be set in order for config foo to even show up. That's the first thing. We also have a select flupe. No particular reason I ran out of other names. This means that flupe is turned on. So config flupe is automatically set to whatever config foo is set to. Now a flupe is already a yes and you make foo a module good enough. But if it was no before it will be made an m at the same time. And down here we have select bar if not baz. This is the condition I was talking about. This will select bar only if baz isn't set. It's a rather complicated one but essentially we can actually set up a series of this conflicts with that by using this kind of thing. And it can be rather complicated. You can say and or this that the other thing and so forth. There's a whole mechanism for making sure that if you set a particular thing the appropriate other things are set it selects certain things and won't run in certain situations where that would be bad. There's a whole description for this. Documentation for this is all in the Colonel Big D documentation directory. It gets into way more detail than I'm talking about. Finally we have help. And help basically has a multi-line description of what the help is. And then the next config stanza starts when it finds the word config on the first line. Notice help is indented in. The reason why that is because white space doesn't matter is because if it has the word config on the beginning of the first line it thinks it's the next configuration option. Okay. Blah needs to be so the question was basically does the pen set blah? No, what it means is blah must be set for food to even be in contention. Will it warn you that you have this but not that? No it will not. Food just merely will not show up in there. Now you can actually search for food and it will actually tell you that it's set to a particular value. And then if you look down in the dependencies you'll see that dependencies aren't met. But there isn't something to tell you that it won't be there because it doesn't know that you want it. Okay. But again, very simple, very contrived example. This is menu config. When you walk in this you'll see we have a number of different things in here. I'll talk you through it the next slide. I talk about all the things I'm about to say. But here you'll see that we have a highlighted area. If you use the up and down arrows you'll notice it's a little hard to see but the plus here it means you can keep going down the page, right? The left and right arrows will take you across these buttons or tabs or whatever you want to call them across the bottom. Select allows you to change things. Exit of course will take you up to the previous level or will actually exit the tool. Save and load allow you to save or load .config files. Personally I never use these. And then help will tell you the help for that specific item you're currently on. I see a couple of things here. One is that we have these dash dash arrow things here. These means these are submenus. You'll see that we have stars in square brackets. These are bullions. So it's either on or off. You'll also see that we have a star up here with a dash on either side. You probably can't see that but trust me it's there. The dash means basically this is selected and you cannot change that. This is set by somebody else. In this particular case it's probably set by the architecture for instance. If they were angle brackets these are tri-states. And if you have round brackets like this it's a string basically that you can change to whatever you want. How do you actually change the value as well? Most people will go down to it and use the space bar and that will toggle. If it's on it goes off. If it's a tri-state it will go off module on. It will go from no, m, s, y. How do you actually change the value if it's a tri-state? It will go off module on. It will go from no, m, s, y. However, if you want to be really lazy and you just want to go to what you want to be at y, m, and n will also take you to that appropriate thing. Again usually just space bars faster. The enter key is how you get into a submenu. The way you get out of a submenu is you go to exit and it will pop back up again. Now at one point and there's a few people in the back that have been doing this as long as me at one point people remembered where everything was in the kernel menu. However, in any particular slice of the kernel that you're looking at you will find that there's on the order of 3,000 options. Most of them are hidden though. So it's 3,000 visible options. There's over 10,000 actual configuration options in the kernel. They move because people think that they should be in different places. They're hard to find and so on. Remember where exactly everything is really isn't possible anymore. It moves around a lot. Almost nobody does this. Instead what you can do is you can do a search. Now how many of you know VI? Pretty much everybody. How does one do a search in VI? Slash. Guess how we do search in menu config? Slash. Exactly. So you put slash in there. It doesn't matter about case. It doesn't matter about putting the config part in. Search for the string that you're looking for. It will show you the help block of all possible combinations. Just for time reasons I won't actually demo that for you at this point. But you will find that there are little numbers next to each one of the locations in that search. It will go 1 through 0 and then it will go a through z and so on and so forth. Now the interesting thing is this. It used to be that you do the search. It would show you where it was in the menu system and then you'd actually have to go and find it. You'd go to the appropriate subdirectories and then you'd have to look through the list. Which is not sorted by the way. So some bright bulb decided that this was kind of silly and instead put these numbers in there. So the cool thing is that if you see that number next to something you just press the number of the letter and it will jump straight to the value that you're looking for. And it means everybody's got lazy and we don't even know where things are even more now because we always use search to find things. So whenever you're configuring your kernel or one other system just use search. You'll find what you're looking for. Things are mostly named in a consistent fashion. And if there's more than one option that it finds basically you will list all of them there appropriately. Okay, so next, let's see now here. Okay, so the next thing is building a module in a tree. So how do you do this? And this is the ultimate way that you want to deal with things and you do this by essentially adding your code appropriately in the appropriate sub directory. You add yourself appropriately into the K config that you're looking for and what have you. You then make sure that your configuration is set up appropriately in your .config file, usually it's your menu config. And then you build your kernel. Okay, but building everything in kernel is a bit of a pain. It's a lot of problems and as kernels change of course you've got to bring things along with you and so on. Most people do their work out of tree just to make things easier unless you're an actual kernel developer in which case a lot of people just work in tree all the time. But everybody else works out of tree because it's easier. And the way you build out of tree is this. You use this line right here, make capital C and then a directory and then M equals and then a directory and then modules. And the way this works is rather clever. This case should be K root, not root, I'm sorry. This actually gives you the path to your headers. It doesn't have to be called K root, it just made it so it fit on the line. And the trick is this, if you actually look at your kernel headers that you're building against for the kernel, there's a few things in there. Your original static headers, they're what are called dynamic headers. They're built by the build system. When you prepare or build configured headers it's actually building those dynamic files. So things like version.h, autoconfig.h and so on. However the build system that was used to build the kernel is also here. When you build a module you have to use the same compiler the same setup, the same settings, the same compiler options as the kernel otherwise it doesn't work. So the whole build systems here, the configuration files here and everything. So when you configure your headers you have your static headers, your dynamic headers, your .config file, the make file, all the scripts that were used, all the options that were sent to the compiler, all the same. Because in the past when we had to do this manually we got it wrong most of the time. So over here we know how to build the kernel and therefore modules for this kernel. Over here we have our module. But we don't have the build system. So how does that work? We use minus capital C to jump. In other words this is a change over to the kernel directory. However we need to remember where our module is. So what we do is we set M equals and this is short had in hand for kbuild underscore module. That's too much typing so M equals. And PWD of course at the command line gives you the current directory that you're in. So what this does is it saves where it came from. M equals PWD. It then changes directory over here. It runs make modules but to build the code that's over there. It basically uses the make file over there to build things but using the configuration that's over here. Because it's easier. Literally because it's easier. You don't care where it goes in the past. You can build against multiple different kernel versions. Basically by building something out of tree you are building it in such a way that it doesn't have to be sent to a particular kernel or necessarily be upstreamed. Now that last part is bad and you ideally want to upstream everything but you can't always and sometimes it's big and complicated enough that building it in tree all the time means that you can't use it with random kernel that you want to use it for. So if you only ever do your work in kernel that means that's the only kernel it will work with. Building it out of tree gives you more options. Cross compiling modules. That's for natively compiling. In other words building on target or on your host. If you want to cross build from one machine to run on another you do this by setting two values. The architecture variable or arch and cross compile. It doesn't look a bit like this. You still have to set your root and your M if you're doing out of tree but you have to set your arch equals whatever the architecture is. Now realistically most of us build on x8664. Why? Because it's fast and relatively cheap for the power that you get. And so arch equals arm. This is literally the name of the architecture you want to build for. If you look in the kernel code there's a directory called arch and under it there's a number of subdirectories. This is the appropriate subdirector you want to build against. In this case we're building 32-bit arm. Cross compile equals and this is a triplet. It's called a triplet. Arm Linux. This is just a random one I chose. It's the one that my compiler tends to use right now. You'll notice there's a dash at the end. If you look in the make file it literally puts $cross compile in front of GCC and LD and AR and so on and so forth. So by specifying Arm Linux, Unabihf dash it means when you use the compiler it will be dash GCC. What does a triplet mean? Well just really quickly. The triplet basically tells you the architecture you're building for. Arm. The kernel or system you were building for. In this case we're not using what's called a bare compiler. In other words we're not going straight on to bare hardware. We're going to an operating system. In this case Linux. So it's understanding it's on a Linux system. The GNU-ABIHS is a little bit more complicated. This actually tells us the runtime that we're using. The GNU part tells us we're using strictly speaking G-Lib C. That's what it means. People have taken this to mean different things and in some cases people say it's the GNU operating environment. But literally it's effectively G-Lib C in all its other bits and pieces. The EABI part is because Arm has changed its ABI or its application binary interface over time. Guess what the old one's called? Anyone's EABI? The old one is? Oh ABI. Why old ABI? Normally we're better at naming. But anyway that's how that works. The extended ABI, pretty much everybody uses that now. You'll notice on the ARH64 or 64 bit Arm we don't say EABI why because there is no ABI. It's only EABI. So here we have to specify that we're using EABI and HF stands for hard float. In the past we had to do soft floating point because not everybody had a floating point operation unit. These days everybody has an FPU. So essentially it's faster for us to do that. So this basically is the big grab bag that tells us what is our user space running system that we're doing. But anyway that's how we cross compile a module. You do have to have the same compiler that was used to build the kernel in the first place. Okay so for installing a module. For installing a module. So essentially the way this works is if you use the command install at the command line. Does anybody know what install does? Install is a smart copy. That's exactly right. It copies the file from here to there with a specific mode and a specific user and so on and so forth. So install is a copy. The problem with modules is we have two concepts. We have an install and we have an insert. A lot of people think of intsmod as being install. It's not, it's an insert. Okay so when we say modules install, what are we doing? We are copying our modules into the, pardon me this should be lib. I will fix this in the slides. Lib modules uname minus r. This is the revision of your kernel. So all of your modules live under lib modules and the revision of your kernel. And under that it can be found by certain utilities. For instance modprobe. Okay we'll talk about that in a few slides. But when you install a module you are literally going from here to there, that's all. Nothing to do with the kernel. It's merely a copy operation. Okay so again you'll see that we are doing a make but we are doing a modules install. Now this has to be done as root. Why? Because lib modules is only written to by root. As a normal user you cannot build as root. We always build the kernel as a regular user but we install modules as root. Intsmod, this is the first of the commands to actually deal with putting code into the kernel. Again it does an insert. So it takes the module. It's not an install which is a lateral move. It's an insert. It takes a file and puts it into kernel memory. I'm moving that above because kernel memory is at the top of memory. And of course it causes the internal linker to add it appropriately. It adds it to the symbol table. It links it in all the appropriate places and so on. And the way you do this is you use of course what you call intsmod and the file name rather of the module. And that can include the path or not depending on where it is. There are options you can add after and so on. But to do intsmod something.co is usually what you use. Now it's worth noting that quite often we'll talk about module names and file names. Module names are the file names typically without the .co. The reason is because .co indicates a file and that it's a kernel object or module. However, once it's in the kernel we don't tend to use the file name anymore. We call it by the module name. By convention they're the same. There's a few situations where they're not. But in general they match. Other tools don't like using file names. In other words you don't put the .co at the end. However people are lazy again. The .co on it often still works. It'll just automatically take anything after the . away. What does this mean? Your modules can't have a . in it. If you insert a module with a . in its name you're going to be having a bad day. Lsmod allows you to list which modules are currently loaded. And so here you'll see there's a few things. The first thing is that the very top module on the list it was loaded the most recently. So the very bottom of the list is what was loaded first. Everything else is piled on top. Part of this is because of the data structure that is used and that data structure is a linked list adding it to the beginning of the list is very efficient. The other thing is of course is it shows you that in fact you are stacking modules on top of modules. Typically you have to remove things from the top of the list before you can move things from below based on dependencies. The way we see that is first of all you'll see that we have the name of the module. We have the size of the module and bytes. We then have the reference count. Now you'll notice the reference count here is a 1 and there's no list. However down here we have a 1 and there's the list of another module. That module is further down here. You'll see down here we have a 2. This is the difference between implicit and explicit dependencies. Now in the case of an implicit dependency in this particular case our UIO code used a symbol from UIO P-Drive Gen IRQ. By using that symbol it didn't say I depend on that module. It merely said I want to use this symbol. The system that built the module in the first place realized that there's in fact a dependency there and in fact marked that as being a dependency such that when the module is loaded it listed which other module needed to be removed before we could be removed. So it's merely there for our convenience. Why implicit? Because the compiler in the build system figured it out. So we can actually list it there. However we also have explicit dependencies. Now these are harder to deal with because they're done at runtime and so we don't actually have any symbol names or module names to look things up. Essentially those are just slow things down. So in these particular situations where these numbers are it usually means that a couple of things have happened but mostly it means somebody has a pointer into our memory. Who is that? We'd have to watch the memory table and figure it out forensically. So typically when you open a file for instance your file operations are cached in the inode so that the next time you do an operation it's faster and other things like that. So when you open a file you'll see on a driver for instance you'll see its reference count goes up and when you close it it goes down again for instance. This is why when the reference count is non-zero you cannot remove a module. So those are explicit dependencies. RMMod of course you use the name of the module to remove it so again it's sudo, RMMod and then foo. In this particular case RMMod can in fact remove more than one module but it will do it in the order that you specify. So in other words if you have this module it depends on this module being loaded and you try to remove this module and then this module it won't work. If you do it this way around it'll work just fine. So the order matters as long as you have it correct it's really dumb it says this and this and this and this. Wrong order won't work. DebtMod. DebtMod is there to figure out dependencies. This is the bit that actually as you install modules will in fact build a number of databases. Again I haven't shown you all of them. I've shown you some. There's a bunch of these so that you have binary faster lookup versions. They live under again under lib modules uname minus r and they're built by DebtMod appropriately. We'll just talk about these five files here the text files you can look at them. The first one lets us know what alias is so that we can essentially ask for things by alias and then the appropriate module is loaded. We can do this by configuration of modprobe. We have our built-in modules. Any built-in symbols that we have in the kernel we need to know where they come from and which modules are there. It's in there. ModulesDep tells us our dependencies between modules. ModulesOrder tells us the order in which in which the static order in which case they need to be loaded. Usually done for built-ins and so on. And we also have module symbols. This lists all symbols for all modules again so we can figure out dependencies. There's a few others but those are the nature ones. Finally we have modprobe. Modprobe is a smart intsmod. Essentially what it does is it allows us to intsmod a module and what modprobe will first do is figure out are all dependencies satisfied for this module. If not it will go and satisfy those dependencies for us. So to load all pre-dependencies it will also allow you to deal with options. Normally you have to specify your options at the intsmod time. For modprobe you can do that and when you come up in files which we're about to look at actually in these two places either modprobe.com or in a series of .comp files under modprobe.d you can set up static options so that every time you load the modules those are always set which is kind of nice it makes them persistent. And then finally of course modprobe is how we tend to load things like drivers. So when we need to make sure something's there we do a modprobe of the thing and it happens. Now just really fast before we only have to get to the break which I should have done ages ago. If you intsmod a module twice what would you expect to happen? You can't insert it the second time. Why? Because it's already there. What if you modprobe something twice? Why will it work? I've heard several different things you're basically all right together and the reason is essentially is because the question is different. In the case of insert it's saying will you please put this into memory? It's already there of course it's an error. In the case of modprobe what you're saying is is this satisfied? Have I satisfied this constraint? So if you modprobe something and then modprobe it a second time it's like super I have nothing to do. Yes it's good. And this is how essentially the dependency system works. Is this dependency set? Yes, this one is too. Good let's load. Second time through just everybody everyone's a yes. So for you devs to load drivers it's very helpful. It doesn't have to check whether something's already there. It just modprobes it. It takes care of it. You're good to go. Excellent. Let's keep going. So it's a different thing. It's just for modprobe you have to install modules first. With insmod you can put them wherever you want to. For gosh sakes if you're building a product insmod is not to be used in those situations. Install your modules. Use modprobe. Insmod is really intended only for people that are building code. It's meant for us. In far too many embedded devices where they insmod modules from weird places in the file system. Don't do that. It's purely a development tool. Modprobe is the right way to go. Finally in modprobe configuration let's look at a foo.conf file here. There's a few things you can do. You can look it up if you do a man of modprobe.d. It'll show you all the different things you can do. But you can blacklist particular modules they won't automatically load. This is great when you have more than one option for drivers. You can blacklist the ones you don't want. You can set options. For the module called blah in this case we're setting multiple options. We can also alias things. You can basically say, I want the sound system. What's the sound system? Well somebody's aliased it appropriately so the right thing gets loaded. Oh I'm sorry a couple more. ModInfo allows us to show the metadata for a particular module. This is a very simple one. It shows you the licenses and dependencies with built-in tree, version magic. Any parameters are listed afterwards. And so on. So that's a bit longer than I wanted it to be but do we have any questions before we take a quick break and then hopefully come back and play with some labs? We handle all the questions during it so let's take a quick bio break. I will do my best to come up with recreate my labs and we'll go on from there. Are we okay? We have many people in this room. Sweet. As long as you're cool with that I'm okay with that. I'm just letting you know. People can't really do the... The real problem is I don't want to push out anybody who's paid by having people who come in into that room. Agreed, agreed. We handled that poorly and next year we have to say you must sign up for... Pre-registration required. We could have easily done that. I don't mind if people listen. I just wanted to make sure that you were okay with that. No, no, no. I didn't think it was a problem either. I was just more concerned about people who paid by having a seat. That's the more important one. I've told lots of people that if they want to say that's cool but if we run out of seats they have to leave. That's pretty much what I've said. That's fine. I just wanted to make sure that I just counted. That gets really cool. Agreed. I lost everything. It's so annoying. Right before I had them all done and they're gone. It crashed. That'll take us almost to the end because it's at 4. This goes until 4. 10 minutes, 15 minutes tops. No swearing. Agreed. I'm afraid we're completely out of them. We have 15 slots. We sold out this morning. We will have them online for sale in the future but we're not there yet. We actually weren't sure we were going to sell out but at least you're the 10th or 11th person to ask. I apologize. If you look at this you'll see at the very bottom we have a Hello Module loaded app. This is in fact the print K that was in the module code. Unfortunately most of the beginning of this walkthrough was just for fun. Anyway, that's how that works. How do we remove it now? We do this by doing a to-do rm-mod-trivial. If you like command line completion you'll notice that rm-mod on a mod probe based system using bash in fact is bash completion which is kind of nice. On your average embedded system that's only using busybox you won't have that option. It's a fully functional system. And of course I can now remove it. Again if I do dmessage in this case I'm just going to run it through tail to give you the last little bit. You'll notice it now says buy. That's pretty much it. Let's try this one more time here. We're going to do a make clean and this time what we're going to do is we are going to copy trivial.c and we're going to copy it to second.c and what we're going to do is inside trivial.c we're going to add a new entry and maybe I'm going to do it from over here so I can see what I'm doing. So what I'm doing is I'm adding a function in trivial.c in such a way that it can be loaded from the other module. So here we have we now have it is updating the screen in a very strange way. We have something here that says foo, it's a foo module and I have no idea why it's updating like that. Let's maybe get out and come back in again. Alright well these are in different lines in my file. In any case the whole idea is that foo is going to do a print k and you'll notice I have not made it static. That's the most important part. Now what we're going to do in the other module which is second is now in the init code here before we do the return. Again it looks like the flow is a bit different. I'm actually going to make the change in my from my laptop here again just so I can see what's going on. Here I'm going to reload the file. You'll see here that I've done a couple of things. The very first thing, oh wow that's really strange. Let's maybe cap the file. Maybe the editor is doing some funny things to the two things I want to point out. The very first thing here is that I have declared a function called foo. It happens to match what is in the other one. Normally it would do this in a header file but I'm being lazy. You wouldn't do this in real life. Next you'll see that I'm calling that function down here. Now as long as everything matches it's all good. Now certainly speaking this is a very simple function so I actually didn't need to declare it. It's good practice. In this particular case that's all I have to do and then of course provide the function in the other file and again in a way that is not and theoretically it should work. Except for one thing that I didn't think of. And that is in fact we need to extern it. Again trivial.c In fact it can be static thinking about it. But I have to do an export symbol. Because again we need to tell the internal linker about it so it works appropriately. You'll see here that I've actually been able to make it static and I've called export symbol in order to make the appropriate function work. So what I'm going to do now is I'm going to whoopsies build it. And so I'll get out of here. I'll do a make. So the thing about externing is you only need to do externing for something where it's stored. So for a variable. Functions we provide prototypes all the time without externing so you don't need to externe function pointers. I do unfortunately need to add this to the make file so let's just quickly edit that and for some reason silly things. So let's do the build. Here we're going to make this will be the last of this because oh isn't a prototype. Yeah. I'll just get rid of that. We're not relying on the we're not relying on the kernel sorry on the compiler linker to do this so it can be static. The fact that we are exporting it using export symbol is sufficient. It's just that the compiler itself gets messed up so I'm making a mistake of some sort that I am not seeing trivial. I shouldn't have to. No but the thing is you don't need to provide a you don't need to provide it in this particular case. I guess the kernel does. Oh we will try that then. You're probably great. Yep. Function declaration isn't a prototype. Okay. Sadly we're out of time anyway so it doesn't matter. All I wanted to show you here was that in fact when you build the two together this is what happens when you try to root code on camera. All I wanted to show you was that when you have modules and one depends on the other when you insmod them out of order you get an error. I'm going to show you once we install them and then to use modpro it would have automatically loaded them in the appropriate order. Okay. Anyway that's it for today. Unfortunately we've just run out of time and we need to get on to the next presenter. The next talk that we were giving is on Uboot. But otherwise for those of you who are here to see modules in Kbuild I hope I answered your questions. I hope you know more about modules in Kbuild and I hope this now brings you into the ability to write your first kernel code. Okay. Yeah. And we have a couple of questions. I'll take two questions and we're going to have to move on. So it'll be under user source by default. That's where the package puts it. No you have to install the package first. So the package... Yeah. Yes. Nothing. We're going to give you those on a USB stick. So yeah, nothing you need to do. Okay. So the next talk is momentarily.