 Hello world By all means This isn't so much a VM as a machine that is less Underpowered than the netbook. I normally care around with me. This netbook has Many hours of battery life. It's extremely portable and it's powered up. It's powered by a wind-up toy So if I actually do test builds for you, we'd be here a while All right, I pretty much am forced to start Let me see did I decide to go with index or index to I think index to Yeah Six of one half dozen a other I'm pretty much forced to start with the concept of circular dependencies When you are building a Linux system If you've ever installed things there are often other prerequisite packages Well, when you get down far enough the dependencies become circular you can't Install a system without boot media a USB or CD to boot it from you can't create a boot media without having a running system You know way back in the dark ages We had punched cards and paper tape that you could physically write information to or the switches on the front of the altar that you Know you could flip here's the address Here's the here's the value the right button and you could literally toggle in a program if you had hours to do it That went away a long time ago. We now have you know millions of bytes of stuff and When you're putting together a system Something you're gonna hit a lot especially if you ever deal with something like Linux from scratch is you have to build packages before you can build other packages and In order to avoid bits of the host system leaking into the target You sometimes have to build something that you're not actually going to install, but you're going to build against So we'll get to all that But the reason I introduce with the concept of circular dependencies is I have spent two weeks Attempting to come up with a nice linear index of here's the topics I want to cover in this order and its circular dependencies all the way down So we're going to be explaining some stuff and then coming back to explain why that was incomplete or wrong a little later All right, so Circular dependencies So there are four There are four Basic components of a build environment of a Linux system that's capable of rebuilding itself under itself from source code Those four are a kernel a C library a set of command line utilities and a toolchain and a toolchain is what all the embedded geeks Call a compiler and linker and the make program the set of build utilities that actually let you compile stuff When you are creating a system though It doesn't have to be able to rebuild itself under itself your your tiny little embedded ones generally don't So your build time Packages and your runtime packages one of the ways that you break circular dependencies is by separating them okay one of the things I did as Okay, one of the things I did as backstory to this a long time ago is Starting like 15 years ago. I created a project called Aboriginal Linux from the Latin ab originae from the beginning Where basically I started with Linux from scratch and I went okay? What is the smallest simplest? Build environment I can get capable of building itself from source code those four packages. I talked about our conceptual packages I got it down to seven actual packages Which were the Linux kernel you see libc at the time make bash binutils gcc and I list them here and busybox and In order to make that work I wound up actually maintaining busybox for a while because I did so much work with you know If I want to replace the GNU version of said with busybox said It didn't work so I upgraded it to the point where you know I Upgraded their sort command. I upgraded their mount command and it got to the point where the previous maintainer just handed it off To me for a couple years, and then I handed it off to Dennis Flesenko But busybox has had a lot of work over the years to make it Provide the set of command line utilities All in one package and that was an explicit goal that took years of work the others You know the toolchain I used at the time was binutils plus gcc plus make so three packages there and By the way, even busybox at the time I was substituting in bash for it shell because I never got around to upgrading it shell That's why there's a seventh package there Busybox has had some work on it shell since then and it's possible that you could build Linux from scratch with busybox ash these days I haven't tried it recently This was an old version of gcc that had binutils and gcc as two packages now They've metastasized to five packages, you know, so the simplest system with modern packages is less simple than Previous systems were fun stuff Linux is still Linux for a while Linux needed pearl to build because Peter Anvin decided that was a thing in two six twenty five and It took me several years of submitting patches to remove pearl as a build requirement Before that finally went away in like 2013 Okay, so let me start By the simplest thing you can do to boot a system is Because I'm in the wrong. Yeah, I haven't the intro file Okay, so how simple what can we get here is a page where someone booted Hello world on the bare metal Okay, and you can easily Google for this or you can go to that URL. I'll post a list of the URLs I went to after the talk But basically what what he did is he has a little function that just repeatedly writes to the The output register of the serial port and has that iterate through a string and then Especially builds it with a little assembly header and with a linker script that puts stuff in the right place Does this stuff? Which you know the assembler GCC for the C part linker running a special linker script And then a program called object copy that takes it from the elf format and copies it to a binary It basically the elf format is a bit like zip or like tar It's an archive format for storing executable data So there's a variant of elf that stores shared libraries. There's an variant of elf that stores dynamically linked programs There's a variant of elf that stores dynamic statically linked programs It's basically the same archive with different header information And what this does is this extracts just the raw binary blob output format binary and puts it to test dot bin and one of the things I did is Hello hardware make dot sh The other thing I have to introduce is the concept of cross compilers. We'll get to that. I mentioned circular dependencies and So if I run make that sh which I ran on that machine and the paths are different, of course To so this is actually the machine I also cross make To let's try that now Hey, so that just compiled the The program and created a test dot bin and then run is copied from his thing where we call QEMU system arm tell it machine Versatile PB is one of the arm systems QMU knows to emulate. Does anybody here not know what QEMU is? Okay QEMU is an emulator. It's a program that creates a virtual machine and Runs the code you feed into it it has a built-in Bootloader where basically can go this file copy it into the virtual memory and Jump to the start of it. You know if it's in a known format it can actually read the header on it and go Oh, this is an elf file. I'll do basically what obj copy did Copying into memory myself and jump to the start information out of the header Here we already did the obj copy. So, you know the bare metal one The dash kernel test dot bin argument is basically telling it use your built-in bootloader To load this into the virtual machines memory dash no graphics says instead of popping up a window with The bitmap output of your emulated video card, you know, whatever the program that's running wrote into the M, you know, it tried to access the emulated hardware and its its driver Intercepted that and wrote it into the window instead of doing that just have a serial port on On the standard in and standard out of the QEMU program So that whatever you write to QEMU standard in goes to the serial port and Whatever the serial port, you know, whatever the program writes back to the serial port goes to standard out of QEMU So that you can actually, you know interact with it on QEMU standard in and you can actually do things like use tickle to script Emulator instances that way. It's it's very useful The dash M 128 M says 128 megabytes of memory is how much should be installed into the virtual machine This says basically what motherboard to emulate what board Board is shorthand for For motherboard it basically says put this processor input memory at this location All the information about you know all the information that would be in device tree when we get to that You know all the information that The kernel needs to know where to look for these devices will QEMU similarly needs to know where to put the emulated versions of them So we told it this is a common type of very old arm board from I think the 90s The verse the arm versatile board and then this is just me telling it Don't attempt to emulate an audio device because if you do it will spam Standard out with a bunch of I failed to to connect to OSS and it's like I just don't want that So when I do that I get hello world it booted a Virtual system which wrote something to the serial port and then just spun okay, so this is About the simplest thing you can get a machine to do and actually see the output and My fan is starting up because it is spinning. You'll notice there's a CPU pinned and The other fun thing about this is we're now going to tell you why it's wrong and the reason that it's wrong is Oh, come on. I have it in here somewhere The reason that it's wrong is that real serial hardware only works at a certain rate You know, it has a UART buffer that can fill up if you just spin Writing to its output register. It's going to start discarding data fairly quickly What you need to do is you need to read a status register That will have a bit set that says I am ready for the next bite or not and you can actually fetch that out of the Linux kernel source code in It's under drivers Yeah, TTY serial AMBA PL 011.c and then Put Yeah console put character It's doing a read from a register and ending it and while it's doing that, you know CPU relax is basically a way of telling the processor not to overheat It's it's a no op that takes a little extra time and then write the character once it frees up So, you know that a driver attempting to talk to real hardware is doing an extra step that the emulator is not doing and We're going to hit that a few times in here But the really fun thing about this this PL 011 console put care is a Linux driver that's designed to work Before interrupts have been enabled early in the Linux boot process early print cave we'll get to configuring the kernel in a little bit and That means that Before Linux has really initialized most of the other hardware it may want to print stuff out very very early on Which means if you know what your serial hardware is and where it's configured you can basically look up what these macros are and substitute in the Constance and cut and paste a liner to that let you stick print statements into Anywhere you can stick it into a bootloader. You can stick it into early boot You can truly debug almost anything with print statements, you know, so that's actually a nice little technique to have in your back pocket and We will get back to that so The other thing that's wrong with that is we're doing the simplest possible Linux system and clearly hello world is not Linux So let's actually do what it says on the tin and I should have done that beforehand because I want to build for 10 and I don't know how slow the you know the net here is going to be too slow. All right. Can you give me 49? I don't know when the next time at last time. I did a poll on this machine was all right, so The other thing other than circular dependencies is we're going to have to define simplest What I'm going to show you right now is the simplest way to build a Linux system Hello What are you doing? Where are you at? Random thing in 2016. Oh Right lovely Any others I have a bunch of others. Well, here's three five, right? Sure. That's ancient Still works the same way though Did you finish it yet? What? The downside of switching to a much faster machine right before the thing is my careful setup I ran through is right there. Yes the 4.9 4.9 there we go All right, so it's gonna boil down to We're going to do a Yes, I know All right, did I put it in here? Sure Okay, the theory behind building a kernel always boils down to you're going to do a make config you're going to do a make and then You you install and boot the result a lot of the times we're going to be cross-compiling Cross-comp the the difference between native compiling and and cross-compiling is Something that I'm going to explain On an x86 64 system. We're going to start with the available def config. So if you go make help grep underscore def config it should list For the architecture you've selected the available are you still checking out? No, you're not Come on. There we go. So it has I three I 386 def config Which is 32 bit an x86 64 def config which is 64 bit if I selected a different architecture There's a bunch of architectures in the arch directory if I selected, you know make arch equals Arm I'm going to get a whole bunch of def configs But for right now The simplest way of doing it is to build for the same architecture as this thing to not deal with cross-compiling at all first And I'm going to go make x86 64 Def config that's an underscore there. I don't think you can actually See it Seriously, there we go and then make dash j8 Since we're done come You can do it make j8 and This is going to build an awful lot of kernel and that's going to go for a while So meanwhile in another window reason I opened the second window Let's build a hello world binary the reason for this instead of exiting is That I'm going to run this as PID one when a kernel boots it needs First it needs a bootloader to have run to load it to load it into memory And there's a couple of times that it can load run straight around but we'll get to that Then the kernel needs to know the board layout You know where to find the physical memory where to find the serial port if you're using a serial console where to find the IO devices even if you have buses you can scan you need to know what buses there are and where to find them So it needs some starting data before it can even scan the rest of the system to see you know What's plugged into USB? Well, where's the USB controller and then it needs a root file system in order to load its first program from and It needs the first program to run and We're going to provide all those things in the little system. We put together here We are going to use a knit ram FS And we are going to use a statically linked hello world program as a knit And we're going to boot this under QEM you so that you can see basically hello world run in a different context So this is a second layer of useless before we start getting to actual useful Now you'll notice This is this is building it at a reasonable clip, you know This is an eight-way SMP machine and it's still building Quite a lot. The thing about def config is it's got how many symbols? So that's 1164 that are just set to y Actually Okay, there's the number that are that are set so 1,252 symbols Which Select a lot of stuff, you know, we're not going over all of these symbols. We do not have time So what we're going to do instead you know so Make def config is simple to build it is not gonna result in the simplest system I thought it would take less time than this and then to package this up We are going to We are going to do this little invocation find period to CPIO Dot oh dash h new C Pipe it to gzip and we're going to do that in the directory that Has our new root file system So I don't need I actually need the parentheses and then redirect that to the directory above this root dot CPIO dot gz. So this has created Yeah, I didn't I didn't strip that that's statically linked against glibc, which means it's flipping enormous and It even has the source code in there, but that won't That won't bother us When the Linux kernel boots it Starts by Mounting a root file system. There are a couple of ways you can specify Where the root file system lives you can statically link in a CPIO archive into the kernel binary Which will be extracted into memory in a file system called an it ram FS and it ram FS is an instance of either Ram FS or temp FS in more recent ones documentation file systems I Years ago I wrote documentation on this so if you want to You know read through all that this is just the very quick summary CPIO is an older archive format that predates tar but still works just fine They used it because it's simpler tar is actually a lot more complicated under the covers that gets extracted into ram FS ram FS is a Ram based file system. There's four kinds of file systems. There's block backed Which is the conventional ones you're familiar with like ext2. There's a there's a block device Which is a fixed amount of storage that gets formatted and interpreted through a second driver that tells it what kind of file system there is There's pipe backed which are called different things sometimes you hear them as network file systems fuses another example of this We're basically you're talking through a pipe to another program that speaks a protocol And that other program is providing your file system. So that program may be Samba that program may be you know There's a bunch of things it could be Pipe backed ones don't have to be network file systems But people usually think of that the point is that it's a serial protocol The third kind are the ram backed file systems where your your files are stored in memory Where basically the Linux disk cache the page cache and the D entry cache That store copies of the font, you know copies of file data on their way to whatever backing store There is because if you're writing faster or if you've read them in it doesn't want to redo the transaction with either the block Device or the server, you know The cache Storing the data in the case of ram backed file systems has been plugged so that there's nowhere to flush the pages to When you write data into a RAM based file system There's two instances of them There's RAM FS and temp FS in the kernel when you write it in there It it has nowhere to go so it stays and when you delete the files out of them It frees up the memory so that they automatically Resize themselves to how much is actually being used now note that the older RAM disk that you've probably heard of Isn't actually a RAM based file system. It is a block backed file system Where you have a device driver that turns a block of memory into a block device It exposes it in dev as slash dev slash ram zero or something like that And then you use a second driver as the you know ex t2 or vFAT or something like that as the lens you look at that Block of memory through you run a formatting program on it to put it in, you know The right format you copy files into it and then you look at it through a second driver That is a convention. That is a block based file system not a a RAM based file system So a lot of people to this day still confuse those RAM based file systems are actually a lot more memory efficient because the thing about a block based file system is you've got the Copy in the block device and then you've got another copy in the page cache so it actually, you know It it's actually less memory efficient to use the older format and a lot of people still don't understand the difference between those The fourth kind is synthetic file systems where there is no backing store when you read files or you write files You're talking to a driver that can do arbitrary stuff with them Proc is an example of this CIS FS is an example of this the contents of proc are Just hallucinated by the driver it makes it up and then when you write stuff into there It can call any arbitrary function in the driver to do anything it wants you know, so those are actually four different categories of file system and Generally when you are Telling the kernel, you know when the kernel is booting it can Extract a CPIO archive into the RAM FS instance that already has and this is called a net RAM FS The older way of doing it before Linux implemented any RAM based file systems It only had an IDRD the initial RAM disk the block backed version so It could allocate a RAM block disk and take something like an ext2 image and Copy it in there could take a G zipped ext2 image the same way that you'd you know You take a ISO image for example that you ripped from a CD and then you can G zip it well you could you could copy that into the the RAM disk and Mount it using the ISO 96 60 driver as the lens you look at it through And Then the third thing it can do is if you didn't provide either of these two or if you did provide them But it couldn't find an init program to run out of them In init ram FS it looks for file slash INIT it looks for file and knit in the top level Directory that is Execute that has the executable bit set if it can exec that it'll stop looking further and it'll just launch PID one in the In the older ram disk base one it looked for a different name Linux Rc Which is just historical interest don't go there and in? init main dot C the In the Linux kernel source code in init main dot C. There's a There's a function Start kernel So if you go into the Linux kernel source code and look in init slash main dot C function start kernel You can actually trace through how the Linux kernel starts up because this is the one that actually does it it has a bunch of Different places it looks as fallbacks It looks for s been in it it looks for et cetera in it it looks for been in it And if it can't find any of those it'll it'll launch been sh. This is for the fourth Sorry for the third place. It looks if there is a Root equals Argument on the command line. It goes ah, this is a block device mount it and then Try to find an init program in there that used to be you know That's that's the fallback root file system So those are the three different places that you can tell the Linux kernel Here's where to find your root file system and then within that file system. You have an init program so the kernel has finished building and At the top level there's a file here called VM Linux This is an elf program and I mentioned that Elf is an archive format and you saw that obj dump trick Well, they do a variant of that obj dump trick and then stick a little header on the front to create arch strisc boot The the file arch x86 boot BZ image and you'll notice I mentioned that this one's enormous We're going to go into kernel configuration stuff after this So what we do is we go QEM you Let me see I did a I did a So oh This little trick here make dash j dollar sign parentheses n proc There's an n proc command that tells you how many processors this machine has So if you want to automatically do SNP at the level that this machine does you can just you know That says run a command and turn its output into an argument So make dash j N proc will automatically do make dash j however many processors are in the machine That's what that little trick does it's a fun thing So we did the make config we did the make and then this is kernel command line Which is similar to the bare metal one QEM use system i386 instead of arm this time because I'm running an x86 laptop Dash no graphic is the same. Don't pop up a window instead. Give me a serial console on standard in and standard out Dash no reboot says when the system in here tries to reboot the hardware just exit QEM you instead rather than restarting it Here is where that kernel image. We just built lives dash a knit rd of A file and knit rd is as I mentioned there's the older Boot mechanism that did initial RAM disks. I mentioned that you can statically link a CP I o image into the kernel at build time well the method of Having the bootloader point it at an initial RAM disk image that usually was a compressed block device Bootloaders still know how to do that and the kernel still knows how to listen for that But what it does these days is first it checks the type to see is that a CP I o signature at the start Instead of ext2 or something like that if I'd identifies that you've fed it a compressed CP I o image It goes. Oh, I'll just extract this into an it ramfs and use an it ramfs instead of a knit a knit rd So it can auto detect the format. So what we're doing is we're taking that archive and feeding it in through the external A knit rd loader and the reason we do that is every time we change the root file system We don't have to rebuild the kernel because if it's statically linked in the kernel we'd have to rebuild the kernel in that slow and Then the last part which I screwed up There should not be a space there is Append this is QEMU ease for provide These extra arguments to the Linux kernel command line. What is the Linux kernel command line? there is a Another one of these kernel.org slash doc slash capital D documentation is The online place for the documentation directory in the Linux kernel source code Under the Linux kernel source. There's capital D documentation directory with Lots and lots of files in it. I've already shown you documentation file systems Ramfs root FS and it ramfs dot dot text, which is the documentation on it ramfs now. Let's look at They moved it recently it is now in Oh Okay, they moved it between 4 9 and 4 10. It's in a different place now, but it's still the old one here It moved into a subdirectory, but it's the same file name kernel parameters is A very nice file that just lists All the keyword equals value names that you can provide on the on the kernel command line to tell it to do stuff Again as with the configuration symbols, there's lots and lots and lots of them most of which aren't very interesting the ones that you are required to provide are Well the the main one and that's wrong too If you're using a serial console, you're going to want to say console equals The device the name of the serial device that you're using You don't have to say slash dev if you do it'll recognize it and chop it off But you have to identify which serial device should the should the serial console attached to because otherwise you won't get any console output and then panic equals one here says If the kernel panics reboot after this many seconds rather than hanging and remember that will make QEMU exit So that's a lot more convenient here. So we take this big invocation and We go and This will die because on this machine. I have the Root file system in another place, but let's just use the one that I built a moment ago Which is that root dot CPIO dot GZ that we just did so Let's go back to here and go root dot CPIO dot GZ and now in a moment we should get Come Hello, do I have a broken QEMU on this machine? All right, you're a much faster machine, but I'm going to go back to the one that I actually Did the initial set up on because this is silly. Oh That yes, that would be wrong that would be why and it didn't make it far enough to panic so it never tried to exit QEMU system x86 64. Yes, that's much happier. Thank you So here are you know, you'll notice there's lots and lots of boot messages What are you doing? Clock source switch to clock source TSE. Yes, I saw that Come on, you should finish booting Do I have a weird version of QEMU on here? Oh, yes, I do. It just it was booting asynchronously. So it yeah So it continued to spit console messages after the thing So as I said, this is very simple to build and pretty much useless. We need to make the kernel smaller We need to make user space bigger. Well the next way to make user space bigger What have I got in here is I have a copy of busybox checked out here and what I'm going to do It is Config prefix, so I'm going to go make def config Who here has not configured a Linux kernel Okay, we're going to get to configuring the Linux kernel, but busybox actually copied A kernel configuration infrastructure, so if I go make menu config I Get an interactive configurator program where I can cursor up and cursor down and I can hit enter to go into a menu And I can hit space to select a symbol or deselect a symbol or sometimes it'll it'll cycle through asterisk M and space where the M means build this as a module rather than statically linking it into the kernel and If you don't understand what modules are just don't use them Modules are a thing you level up to when when you need them It basically means that after the kernel has booted you can tell it Here's an extra chunk of code using the ints mod or mod probe program say insert this module into the kernel And you can also tell it remove this module from the kernel at which point it will hide it from you But not free up the memory Unless you've configured the kernel in a way people usually don't and provide an extra flag That will from that point on mark your kernel as tainted So that if you submit a bug report after doing it the kernel developers will explicitly ignore you Yeah, because Removing bits from a running kernel is just Not something that can ever quite be done safely or at least they haven't convinced themselves that it has and they just They don't want you to do that But people like red hat with you know large amounts of money from corporations where the system should never be down Even for maintenance they love you to do that So they'll argue with the kernel developers, but anyway this menu config program is basically just a way of configuring stuff And then when I'm done do I want to save the new kernel configuration in this case? No, well now if I go to busybox and busybox make menu config. There is a similar menu config here Where you know Here's stuff that would be in the GNU core utils package that was completely Reimplemented, you know here's you know, do you want the DOS to Unix command? Do you want the DF disk free command and they have a whole bunch of sub things of do you want it to have these features and Back when I was maintaining this sucker. I did a policy that there is There are a bunch of Yeah, make help on a lot of packages will actually provide here's various make targets That's a good thing to know and you can then grep for the ones that have config in their names death config is the default Configuration, you know, you'll notice that we did the x86 64 def config over in the kernel land Well in busybox make def config basically says switch on everything that is sane Don't switch on the weird debugging stuff Don't switch on like the se linux stuff that won't build if you don't have the right headers installed But if it's a feature switch it on Give me a configuration that includes pretty much everything. We know how to do So this is the biggest busybox, you know You can get and then you can go into make menu config and start switching stuff off to pair it down There's usually two starting points with any menu config base thing. There's all no config and All yes config all no config is basically if you went into menu config and switched off everything that could be switched off This would be your config all yes config is the opposite switch on everything that you can switch on and The reason that busybox def config is different than busybox all yes config is that in busybox all yes config is Insane and switches on stuff that no sane human being would ever want and usually whatever your system is it won't work Because I mentioned weird debugging stuff. So make def config is a good starting point So what I did is I did make def config and then We compile it Which shouldn't take as long as the kernel did and I don't know if this is a release version This is a random get snapshot, but it should still work. I've built it before If you're going to deploy it use a release version Okay, and then Make help grip install which has config prefix. So make config prefix equals I Can just do pwd is an environment variable. That is my current directory slash walrus Just some name that I know isn't already there So I told it install into a directory that doesn't exist and so it populated this new directory Oh Yes, you know My red-eye flight that I had to get up at 3 30 a.m. For was yesterday. You'd think I'd be recovered by now That's why I got this The drink that chases unicorns into the ocean nasty stuff, okay So so this popular director member I said the older Linux RC name that nobody uses anymore Well busybox has been around for a while and it hasn't necessarily Cleaned out all of its old stuff so What I do now is I do the same trick of find period pipe it to CP IO Dash H new C dash. Oh, I never remember that part. I wrote the documentation on this So I wouldn't have to remember it. Yes. Yes. Yes Okay Oh Pipe it to gzip and root 2.cpao.gz and Now what we do is we take our earlier System x86 64 and The cursor up and we tell it instead go to busybox busybox Root 2.cpao.gz and now we add one extra little thing rd init equals bin sh What I'm saying is instead of looking for a file called in it in Anit ramfs look for this file instead as your init program The name rd init dates back to when we were using RAM disks instead of ramfs, but it it listens for both So what that should say is do the same big long over complicated boot and No, it didn't Okay, and you know what I did wrong. I didn't statically link it. All right. Let me explain more Need more exposition so what it did here is it failed to boot it panicked at the end and The okay freeing unused kernel memory is the last thing it says before launching a nid in this case It's freeing three different chunks failed to execute slash bin slash sh error negative two error negative two in the case of The the file is user include SM generic error. No base that H if you you know know that and Eno ent You know negative two in this case Eno ent no such file or directory But bin sh is there. Yes, but the dynamic linker wasn't Because I forgot to statically link it or it is Ben sh is assembling to busy box busy box is a File that is dynamically linked and it is looking for the first thing It's going to look for is this file, which is the dynamic linker That is the program. It will actually execute and then this program will be the first argument to that It's kind of like when I have a shell script It has pound sign exclamation point and a path to the thing to actually run and then the thing that you actually run gets this as an argument so that Pearl or Python or the shell can interpret this script well dynamically linked elf binaries work the same way they have a dynamic linker program that runs and then Loads all these other Libraries in you know from the library search path into memory Links all the references out of this thing to them and then jumps to its start and what it was basically saying is this Misleading error message the no such file or directory was saying the dynamic linker in this thing wasn't there All it had was a number and the number did say what the problem was but it didn't say what file the problem was so If you encounter that kind of thing, you know you you learn You learn to debug this kind of stuff. Okay, so what we do is we build again and this time Clean make I don't have to do a make clean, but I don't remember the way to tell it to so LD flags equals dash dash static Make j8 Yeah Right and we build again and then when that's done I can test that the busybox file is statically linked While it's doing that. Let me point you at a couple other interesting web pages So yeah In current kernels not as much in current kernels I believe that a net ramfs will fall back to looking at a knit equals that went in like 4647 or something. It's fairly recent within the past year In knit equals Used to just tell the kernel what in it to look for when it was mounting a block device on You know stage three the root equals file back after it couldn't find a statically linked to net ramfs and then it couldn't find a an External rd image So it didn't have anything in a net ramfs or in a knit rd Then the third thing it would try to do was mount a block device and the knit equals used to only apply to that block device You needed the other argument to tell a net ramfs what in it to look for Okay, I think that that has recently changed so I could say a knit equals and I'm using a 4 4 9 kernel So we might as well try Let's see. So that will have okay, so What I was pointing you to here let me finish the thought I wrote a lot of documentation in that system that I have Since stopped working on but you know, I did this one for 15 years and I still answer questions about it But you know Aboriginal Linux our motto was we cross compile so you don't have to where it booted these the smallest simplest development environment capable of rebuilding itself under from source under itself from source code and Building Linux from scratch under the result and that meant you could natively compile Anything under QEM you that you wanted to without ever having to cross compile again cross compiling is very hard I wrote documentation. I meant to hand this out to the group I wrote an intro to cross compiling and Explains, you know, basically what cross compiling is and why cross compiling is hard This is very polite because this was written not to scare people away from cross compiling This was sort of you know, so you have to cross compile my sympathies. Here's how to do it If you can avoid cross compiling Try Because cross compiling is really really nasty You have two contexts. You have two sets of headers You know but one for the system that you're building on and the one that your system that for the system You're building for you have two sets of libraries you're linking against one for the system You're building on one for the system you're building for most systems go configure make make install and you know configure asks questions about the system you're building on and uses them to You know apply to the system you're building for and when they're two different systems The whole of configure is asking the wrong questions You know about half of cross compiling is figuring out how to lie to autoconf or To just get it to stop because it's like well Let me build a little test binary and run it to see what the output of produces Well, you can't when you're cross compiling it won't run on this system that's sort of the point of cross compiling and Your build system if you are building something locally that like generates a header file You know a lot of times you will run a small c program to produce code that that or modify code that then gets compiled So you need a native compiler you need what's called a host compiler and you need a target compiler To build for the target and so you have two separate contexts and have to keep them separate You have to prevent bits of your host system from leaking into your target system and what you wind up doing is You know just natively compiling on a bunch of different targets. It's like I tested it on arm I tested it on spark I tested it on power PC. I found slightly different bugs in each case But that's okay. There's a linear number of them. Well when you're cross compiling I tested cross compiling from this system To that system and I got slightly different bugs Every from system so it's hosts times targets So you're testing, you know attack surface is just Exponentially larger and then there's the fact that only about 2% of the developers will ever care about cross compiling at all You know Debbie and has over 40,000 packages in its repository Maybe a thousand of them have ever been cross-compiled on a good day with a tailwind and so it's just Cross compiling is a thing that can be done But the point of this entire project was native compiling under emulation is so much easier And then the trick I did was I fired up distcc and taught it to call out Through through QEM use emulated network to the cross compiler running on the host So configure ran fine The you know, there was one set of headers the header information was pre-processed before distcc sends it out to the compile node Linking ran fine because that happened inside the emulator I could install the result into the system so that my dependency recognition, you know worked fine But the heavy lifting of compilation was moved outside of the emulator to someplace that CPU was cheap and I had access to SMP, you know, so that was a that was a fun thing I did it for a long time and it's sort of out of scope here But feel free to read about it. What I am actually working on now is a project called make root which basically One of the things aboriginal Linux did is it built its own cross tool chain and its own native tool chain Well, there's a project out there now called Muscle cross make By the guy who does muscle libc rich felker Which is a tool chain builder that not only creates cross compilers, but creates native compilers Which is the thing that I couldn't beat out of you know build root will create cross compilers It doesn't want to create native compilers cross tool ng illa create cross compilers. It doesn't want to create native compilers, you know Code sorcery would give you cross compilers. They wouldn't give you native powers There were a bunch of places I get cross compilers from but if I then wanted to build stuff in the resulting system. I Didn't have a compiler. I could install into the thing. I could fish them out of Debian Debian has native compilers for all the targets it supports But they depend on Debian. They're dynamically linked not statically linked So I would need glibc installed for like super h and stuff like that And it's like yeah, well now I have I have a package that will actually give me the native compilers Which is why I started over and I went I can be much much simpler about building a thing and what this is is this is a shell script that builds a root file system Which I'll go through this in a little bit after I go through configuring a kernel I'll go through this is here's how you make a simpler root file system But it's it's a shell script that will basically populate one of these directories You then CP IO up and then run it out of an it ramfs and that gives you a nice starting system Once you have one of these things if you want to repackage it as ex t2 or repackage it as squash fs There's a make squash fs utility. You can read the man page up. I can walk you through how to do that, too Basically creating a file system for one of the other file formats is Either run a utility like make squash fs or gen ex t2 fs that works like tar does here's a directory Here's the output file. It just puts it into the thing or what you can do is you can use a loop back device Anybody here not know about loop back devices? Okay, loop back devices if you need to create a block device file system You have to do it as root You can Let's see didi if equals dev zero OF equals blah dot IMG BS equal block size equals one meg count equals let's say 256 so let's create a 256 megabit capital M Shouldn't have to care so that created a 256 megabyte blah dot IMG and then I can format it Yes, come on come there we go and then I can go mount dash o loop a Blah dot IMG Need to create a directory for it Mount dash o loop blah dot IMG subder and Then there we go and subder is That file system a a loop back device is a special block device that Creates It basically is it's another type of lens remember I said ex-t2 is a lens you look at a block device through to make it look You know like a directory full of files well This driver lets you look at a file and make it look like a block device It shows up as dev loop zero or dev loop one. There's a bunch of them You know these days it it dynamically creates more there used to be a static link number of them kind of like Pty's you know, they're dynamically allocated now. They used to be statically allocated They split the difference by statically allocating the first few and then dynamically adding more is you ask for them And what you can actually do is you can do LO set up Dev loop zero which I believe by default will tell you what it's associated with You know so if I say dev loop one it says it's not associated with anything But dev loop zero is associated with this thing if what the mount command is doing behind the scenes is calling LO set up To associate file name with loop device. There's a man page on LO setup you want to know and Then when you unmount it the unmount command recognizes Oh, this is a look. This is a loop back device and calls LO set up dash D to de-associate that loop back device with that file And the advantage of this is I can is I can copy a File into here Let's do that text in the sub der and then I can unmount sub der and then I can G zip blah dot IMG and I I have just created a compressed EXT-2 image of 256 megabyte size That's the basic procedure for if I want to create block device images now This works for Normal block devices. It doesn't work for flash devices flash devices aren't exactly block devices Because the way flash works normal block devices have a granularity of like 512 bytes or these days 4k And the file systems have a block size of like 4k This is the the size that they upgrade with if your underlying device has a has an upgrade update Granularity larger than the block size of a file system bad things happen Because if your flash every time you write to the flash it's updating 128k chunk And you think you're writing 4k sectors So you have ext 3 journaling and it's like right we write these things into the journal And then we update just this sector and we haven't written anything before it and we haven't written anything after it So we don't need to keep track of that and if we lose power in the middle of it All we have to do is replay the journal and that'll tell us you know What what changes we were in the middle of and then when we try to remount it Oh, the entire directory is gone because that 4k was in the middle of 128k of directory And all of it got zeroed and then didn't get written back because the power was interrupted and oops My flash file system is corrupted because I'm using a file system type that isn't aware of flashes a race granularity This happens It's a bad thing so they have what's called log structured file systems that are designed for use on flash That the first thing they do is they do a special IO control against the block device that says what is your block? What is your erase granularity? What is the update size and alignment and stuff? Where are your erase blocks? So that every time I update I am updating an entire erase block And they do things like they pack the data into the erase blocks so that they're not Doing extra rights to you know We only updated a little of this thing so we don't want to erase and rewrite the whole thing We want to save our rights and and you know round robin around the disk because that's better for flash because flash wears out You know so they they work in a completely different way And if you try to mount one on a normal block device or on a loopback device It goes I can't figure out what the erase block size is that's part of the format of the file system I fail and it refuses to mount So if you mount one of those you have to provide if you're faking it on something like a block device these days I believe you can provide an extra dash O option to tell the driver and Pretend your erase block size is this size So I can make a flash file system that I can then copy into flash But that's a whole area. You now know you need to look it up, but I can't cover it in any reasonable detail here Okay, so meanwhile going back to here we have built I Opened a lot of windows didn't I I? Wonder which one is okay. Here we go. So did I install no I didn't so make I Should still have it up here make There we go. Oh did you have a question? Okay, so we install into the thing now we and it's the same thing except Statically linked no dynamically linked darn it. Yeah, it should have installed over it anyway You want to try that? Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, I didn't actually do a make clean this time Did you detect that I changed the C flags and you're rebuilding everything? Statically linked. Whoo. All right. Let's do that install again. What are you compiling? I just built. Oh You're gonna rebuild the thing and I didn't do the C flags here Yeah, I Note that I haven't maintained this project since 2006 You can blame me for this, but I will Complain All right, so file Laura spin Busy box static whoo and let's do the CPIO thing again, which should still be in the command history and Now let us go to Wherever I had that kernel launch invocation To many open windows here we go and let's try it now All right, I Have a shell prompt so I have now made a Simple to make kernel and root file system The kernel is still six and a half megs Containing as you saw from the boot messages lots and lots of stuff We don't need and the root file system still has rather a lot of stuff. We don't need it's the biggest busy box You can make and also You know I well it doesn't have a it doesn't have an init script So I have to mount t proc proc proc Make their proc I didn't make the directories and then I should be able to if config and You know nothing set up the network, you know, we need an init script that actually does some stuff so actually I'll get to the kernel. Let me go to the root file system stuff first. This is that root file system builder. I talked about So let me walk you through what it does you can you know get hum.com slash landly slash make root There's one script make root where you can basically just Let me see if I Let me check out a fresh copy of it. Oh, no, just do I have no I do not have it extensively locally modified here woot January 31st is that current? Let me see. Yes. Okay, so the first thing it's building is When I left busybox, I started over from scratch in a project called toy box Because I felt I could do a better job toy box is now the standard command line of Android I am attempting to turn Android into a self-hosting development environment I'm using what I learned on Aboriginal Linux to basically make it so that What's installed by default on Android should be enough to build AOS P Let alone Linux from scratch so that having an Android phone is sufficient to develop for Android It's got USB on the end of it if you plug it into a USB hub you can add a USB keyboard USB mouse Use Chromecast to put it on a big screen and you know This is a gigahertz processor with at least half a gig of RAM You know it it's a reasonably powerful system and if you plug it into USB. It's gonna be charging itself. So I Gave a talk about that here at 2013 the Android guys went basically sure why not and started merging my stuff And that's taking up an awful lot of my time. Oh What I didn't do is I didn't tell it To use the current version of toy box, but I'll just grab the One of the things I did is I made it so that if you If you have an extracted directory in the packages directory with the same name It'll use that instead of extracting the tarball You know it's 300 something line script, but it actually does a lot So let me walk you through it So it's building toy box instead of busy box because I am migrating all my own stuff from busy box to toy box um It it's a somewhat cleaner implementation if you saw the the talk a couple of sessions ago on On shrinking the Linux kernel He he was comparing toy box to busy box The the really fundamental difference the reason that Android can use it and couldn't use busy box Even though busy box predates And you know busy boxes older than Android and you know you're not gonna wait for them to start shipping it It's they're they're not shipping it for a reason is that they have a no GPL in user space policy GPL v3 happened about the same time that Android did and they responded by throwing out the GPL v2 Baby with the GPL v3 bath water, you know now that now that there's no such thing as the GPL anymore And you have warring camps that can't you know the Linux kernel in samba can't share code Even though they implement two ends of the same protocol and each has to police their their submissions to make sure that they didn't get Code from the other one because that would be incompatible and they get sued it's like Android just went nope opting out and You know they rewrote They they rewrote their Bluetooth demon they threw away the one they had that was GPL and wrote one that's Apache license They're they're slowly going through and moving that so toy box is under a public domain equivalent license that looks like a BSD license to make the lawyers happy and You know that's why they can merge it Okay, so I did just build the thing and what that did is that created a Directory called root that has a bunch of stuff and in the build directory it created a root dot CPAO dot GZ and This is a mix of toy box and busy box. You could use just busy box But as I said my own personal development, I am migrating from one to the other I'm only using busy box for the commands. I haven't implemented a good enough version of in toy box yet So walking you through what this does This is the script on GitHub It starts if you you know if you don't run it with any if you run it with unknown commands It prints out a help message It figures out if you set the environment variable cross compile if so that is what cross compiler to use that says I'm not building it for The the host system. I'm building it for arm or power PC or whatever the cross compiler is so it You know figures out a couple of things from that and it makes sure that it can actually run the cross compiler Because if you pointed a cross compile, you know compiler that isn't in your path. It'll go up. No Those directory names I mentioned above you can you can replace them if you want to It checks to make sure that you can build static binaries because it turns out on things like Fedora Unless you install the package that contains libc dot a it doesn't install the static libraries by default So you have to install an extra package in order to be able to link stuff that Statically it when you install the toolchain it can only do stuff dynamically Because Ulrich Draper hated static linking and even though he left to go work at the Bank of Evil Goldman Sachs They haven't undone all of the damage Static linking is very useful. You'll you'll notice. I I hadn't copied the shared libraries into the root file system I could run the LDD command to see what shared libraries Do I need and copy them and then remember to run LDD against those libraries because libraries can link against other libraries? but That's that's a lot of work up to that The simplest Linux system as it says in the title of the presentation is not going to be dynamically linked because dynamically Dynamic linking adds complexity statically linking Dynamic linking is worth it once you have reached enough different files sharing the same libraries that You're saving enough memory to care and it that won't be for a while Okay, so this is a thing that basically just Extracts a tar ball into a working directory and it checks to see if there is an extracted version and it will use that instead It's only a few lines you can read that, you know, so and then cleanup is the one that Deletes it afterwards. This is basically tar extract and this is RM-RF Only they're you know kind of fancy versions But every time you see that one of them is tar extract this does a W get of the source tar balls from this URL and checks the sha one Some so these are the toy box and busy box packages we will be building and then the first thing it does is it sees if there's an airlock directory that contains the toy box binary because I Mentioned that when you are cross compiling It's easy for bits of the host system to leak into the target system And you have that combinatorial problem where it built slightly different on Ubuntu than it did on red hat that it did on this thing And instead of one set of things to test it's one for each target So it's number of hosts times number of targets and that is going to get out of hand And you will have stuff that you haven't tested well the way I solve that is I Spent years making sure that busy box Provided tools you could build a whole system with and these days. I'm doing the same with toy box every Tool that is in toy box def config is good enough to build a system with not all the tools are there But you know the ones that are there My said can actually run autocon that kind of thing. So what I do is I build toy box Install it into a directory and then I have a list of files I need to copy out of the host system because toy box doesn't implement a reasonable version and then And there will always be a few because I don't include a compiler. So I'm copying GCC. I'm sim linking to the host GCC I'm sim linking to the host binutils in this directory of commands And then I set the path to just that directory and this is called the airlock step Where once I've done this I'm using this Little tiny system I built to provide a known build environment to isolate against variations in the host system now Aboriginal Linux was doing a whole lot more sanitization. It had a white list of environment variables Where if it wasn't one of these recognized environment variables, it would unset it and it was doing a lot more cleansing and stuff and 95% of the time you don't need to worry about that. So this is you know, this is a little more than the 8020 thing This is like the 90% that gets you 95% of the thing So I this is creating an airlock step, which is strictly speaking optional But it means that I'm building in a known environment. I then This is basically just dependency checking because you know in case I don't need to rebuild stuff that skip that stuff Then what this does is this creates the directory layout this is doing a make-dear of All the standard directories in one line This is pretty much the directories that packages Expect to be in and with this set of directories I can install all the Linux from scratch packages Linux from scratch is an online book that tells you how to create a Very large very complicated about a hundred and ten megabyte system using conventional GNU packages, but it's the base of most OS versions it's nothing remotely like the embedded world uses and I as I said I spent many years making it so that a Megabyte and change of busybox could replace a hundred megabytes of GNU packages But that is the old reference Build that I use because you know if you want to use the big version of said You can mine should be a drop-in replacement for it But in order to ensure that it's a drop-in replacement I need to test against the old one and I need to test that it can build the old one in case I am missing anything So, you know, there's an eccentric directory at a TMP proc sys dev home Mnt root and then under user. There's been s been and lib and then there's a directory there and These are pretty much the directories that Linux expects to be there. There is a standard called the Linux standard base LSB that Unfortunately the Linux foundation took it over the Linux foundation was formed when the Went to black holes of bureaucracy combined to form Voltron of bureaucracy and One of those was the free standards group. The other one was OS DL and When they combined the The focus shifted from maintaining this standard to asking donors for money and Unfortunately what this meant is that the Linux standard base was so badly maintained that a year or so back Debian when the 5.0 release came out threw up its hands and said we're done We're not paying any attention to the standard anymore, which is kind of sad because there should be a standard for this stuff so There's still 4.1 LSB 4.1 and it does Describe the layout of these things eccentric directory is where a bunch of config files traditionally live Temp was the directory that Unix used for temporary files Proc is where you mount proc FS sis is where you mount sysfs. Those are two synthetic file systems that let you Give you an API to play with the kernel so that ps can see what processes are available and you know Ince mod can see what hardware is available and stuff like that Dev is the directory that the device nodes go in I mentioned that the the serial console device is one of the devices in dev dev Lo setup Sorry dev loop zero when I was showing loop act devices Home is the directory under which the user accounts live M&T is a directory for additional mounts. That's the traditional name for it Modern systems added one called media that does the same thing, but they liked it Root is the home directory of the root user. This is fall out from the days when home used to be on its own partition a lot and In case you hadn't you know in case if you ever trashed that partition It would give the and had to log in as root to administer the system. It would give the root user You know somewhere to have his files user bin S bin and lib actually exist because Ken Thompson and Dennis Ritchie on their original PDP 11 in 1971 filled up their first disc Their their root disc was a half megabyte very fast disc And then they had what was called an RKO five disc pack which was two and a half megabytes of external space basically like a USB disc today slower but much larger and when they filled up their their root partition They leaked it into the user directory where they had the user accounts and then when they got a second expansion disc They moved all the user accounts to the new disc and they called that one home And they left the user directory to be eaten by the operating system and then people retconned all sorts of Rationals for this but no that's why it happened and these days when I'm making a system You will notice I just sim link user bin to bin user S bin to S bin and user lib to lib so you know there's There's there's a bin an S bin and a lib at the top directory that I just Simlinked to the second copy of them collapse them together because I don't care. It doesn't matter You know, but I researched Why they did this in the first place and Dennis Ritchie wrote it up as a historical anecdote and I went Oh, that's why they did it now. I know that we don't need to do it anymore a Lot of this researching why we do things in the first place if you drill down into it a lot of times You can figure out it's okay to do it differently and it's okay not to do it at all once you understand Why it was there? So that's an example of that and then the their directory was It was a place to put things that should persist across reboots But should be writable because etc could be in a read-only directory, you know So config files versus dynamic config files So those are basically the the directories it expects to be there and you can just copy this snippet right out of my thing Or just run this file Then I create an init program now you'll notice I said, you know a knit is PID one it is the program you run when the system first comes up and I Actually wrote up a little Things I probably won't have time to cover but should okay so And knit is Weird and I wrote down why and yeah, these are a bunch of different examples of types of a knit you may encounter using BNSH is in it. You've seen me do that Your app the hello world was an example of that but a lot of internet of things thing will just run And you know they're a knit as PID one Here is a shell script running as a knit and then at the end it execs another program I wrote one called one it there's system five in it which was the conventional one everybody was using ten years ago Then Android uses its own in it program these days. There's a horrible one called system D and Then there's others like upstart and open RC when people were trying to move beyond the the main problem with system five in it Is it predated the widespread use of SMP? so it brought the system up in a single threaded manner and People wanted to paralyze paralyze the in it to bring it up to bring the system up faster and They couldn't quite agree on that because you know well this has to come up after the network interface comes up So suddenly we have a dependency resolver where everything has to know what else it depends on and People wrote up a lot of different things and red hat basically shouted everybody down by having more money okay, so PID one the first process that Linux launches is special and What's special about it is to start with? It has no parent it is launched directly by the kernel If it ever exits the kernel will panic you saw that kernel panic earlier of you know We couldn't find a knit because it was where we couldn't run in it because it was dynamically linked So we got a panic message and then QMU exited and otherwise would have rebooted the system Because in it should never exit And it has most signals blocked Specifically the way they implement that Not exactly, but you can think of it as SIG default is SIG ignore for a knit Where if you get a signal the default handler for it will ignore it rather than killing the program and If a program's parent process dies So that you have an orphaned child that when it exit it wants to send SIG child to somebody to notify it of its exit code But its parent has gone away. It will be reparented to in it and it is the reaper of orphaned children Okay, so and it does a lot of things If you run a knit equals slash bin slash SH your system will accumulate zombies if your shell does not know To listen for SIG child Which most of them don't there is actually a workaround if you set SIG child to SIG ignore in the signal handler tell it you know instead of having a signal handler Just set it to SIG ignore SIG child will be delivered and the fact that it was ignored is fine and then the child can exit So when you are running an a knit program if you're not going to handle SIG child Set the signal handler to SIG ignore and You have you have to actually do that explicitly because the default one Being like SIG ignore isn't good enough in that case So yeah And another thing is for example a knit can't reboot the system because the reboot system call Exits your process so the kernel will panic before rebooting So you have to fork a child have it call reboot and then you hang and wait Yes, I've done I I mentioned that I wrote one it one it is a tiny little a knit program It's in toy box You can find it on the busy box mailing list way back when but they didn't merge it where It does just the basic setup another thing about a knit is that it inherits File handle zero file handle one file handle two pointing to dev console It starts with a standard in standard out in standard error pointing to whichever dev console you had on in it ramfs If you didn't have it in it ramfs or if you created an in it ramfs But didn't add a dev console to it, which you'll notice we haven't in any of the ones we've done so far Those opens will fail and your standard in standard out in standard error in it will be closed Meaning anything you write will be ignored and If you try to do that as a shell prompt, you know if you try to then exec bin sh It will exit immediately because it has no input to read from and no output to write to and it basically gets a sig pipe and goes nope So what this a knit script does this a knit script, you know we start with pound sign exclamation point bin sh We then set a home directory and set a path because we didn't necessarily inherit one Our environment is mostly blank So if we want, you know some basic environment variables that you just sort of expect to be there we have to set them ourselves We mount proc on proc we mount sysfs on sys the reason that it's doing this is I sometimes use this in its script to change Root into a directory and when you change root into a directory The system may already have some stuff there So that's why I'm checking if they're already mounted Is just for change-rooted won't be in your systems I then mount dev tempfs dev tempfs is a file system that was added a few years ago that Automatically populates the dev directory with all the devices that are there So you don't need something like you dev or system system D to do it You don't need a demon listening for hot plug events and creating device nodes the kernel can do it for you And it took several years to browbeat Greg Crow Hartman and and company and to going you know The kernel can really do this for you, but what names will we use? Okay? You notice how under? sysfs sys class like you know mem There are names why don't you use these names? And eventually they went all right. We'll use those names. They are unique and it's like yeah, so You know You get a set of devices from Dev tempfs and Then dev pts is the pseudo terminals if you use the new pty format that auto You know that automatically allocates them rather than the static link from ones from way back when that requires a file system To you know every time you create one it shows up in this directory. So that's where it expects it So yeah, that that does amount of dev pts amount of dev tempfs amount of sysfs an amount of proc it then Does an if config of ETH zero and a route add default gateway for QEM you if you are using QEM use? emulated network this is the Address that if you ran DHCP this is the address it would give you and this is the gateway that we'll talk to the outside world through its basically faked IP masquerading that it does and then I attempt to set the The date in case I am emulating a system that doesn't have a persistent clock Because you know some arm or sh boards or stuff like that It's like this system doesn't have a clock so the kernel doesn't know what to set the clock to so I can do it once I brought up the virtual network I can do a transaction with a host that says hey, what time is it and get that set up I Then figure out when I you know the kernel command line is available. I think this is actually the Yeah, this is the kernel I just built So this is the emulated system actually that is still running under QEM you but and proc command line This is the command line that I fed in you may remember that from the QEM you command line so In proc FS I can see what my command line is and what I did here was I Ran said against proc cmd line to see if there's a command line argument if so I figure out what device it is and the reason I do that is Remember that standard in standard out in standard error that a knit inherited go to slash dev slash console Well dev console is a device That is an alias for whatever Actual console device you're using to talk to the outside world, but it does not provide a controlling TTY So if I hit control C it doesn't send a signal to my program and if I accidentally if I run like ping that just endlessly ping stuff and Doesn't exit and I'm going control C control C control C and it's ignoring me And I'm going right after reboot my emulator because I screwed up You know well what one it does is it closes the standard in standard out and standard error that it inherited and Opens the ones from the actual console device the serial device that does provide a controlling TTY So that if I'm connected to that one and I hit control C it sends the program a signal Okay, so that's why I'm doing that thing with the console You don't necessarily have to do that in your thing if you don't expect to have a command line Running as PID one, you know your init will also do the proper setup to that thing if you run sys 5 in it or something like that But I'm doing that here because trying to be simple And then I figure out, you know, okay, what program do I run next by default? I just run bin sh, but I can set the environment variable handoff in all caps One fun little thing about mm-hmm One fun little thing about this is Let's see any unrecognized keyword equals value argument on the kernel command line It will set as an environment variable So I can set any environment variable. I want to tell my init script to do stuff on the QEMU command line So that's you know, just a fun little thing now notice. I had to put that before the RD init line because init equals or RD init equals it will think that it's an argument to sh Okay, once you get to that point everything else becomes an argument to sh So it would have to it does have a positioning dependency. All right, so getting back to this script So I create my init script and here is the EOF of the here document If you know shell programming I just wrote this out as a here document so that the contents of the script could be in the In the script and then I change mod plus exit because if it can't Execute the init it won't run and you can't run a shell script unless it has the executable bit set I Then write out an etc password file to say these are the two users that are on the system And this is in the format that etc password has been in forever and I write out an etc group file I did not write out shadow files so there's no Passwords associated with this. I can't log in as any of these users, but su if you're switching user from root doesn't care and You know, I could have set up a shadow thing, but then what passwords what I assigned to them It's like I just didn't bother, but this lets me this lets them know UID 0 is root You know which UID 0 is always root, but this lets it know that the name root goes with that and if I log in as root my default shells should be been sh and my Home directory should be here and then I created a guest account to be UID 500 And I told it, you know, here's your home directory. Here's the shell to run That's just what the etc password format is. There's a I think there's a man page on it and Then I wrote name server 8.8.8 to 8 into Etc resolve.conf the file slash etc Resolve.conf is the file that the C library reads every time you attempt to do a DNS lookup It goes where are my name servers? 8.8.8 is the IP of the Google DNS pool So I don't actually have to even put in a second one You're supposed to have fallback servers, but like that's a pool of 30 something machines around the world That it does the automatic magic routing to the closest one through the through the weird backbone stuff And that basically means that should all if I can get out to the net That's a DNS server that should be up barring nuclear war and even then maybe So then you know I so those were actually the directories I created and to the The configuration files I added to give me a Basic system where playing around on a shell prompt if I do ls-l It should give me like user names for the files at least the ones belonging to root and group sort of thing Then I build an install toy box Which you know it it's only this and the only reason it's it's weird is you know I did a make you know set up for is the thing It's basically a tar extract of the tar ball make def config just like busy box I then changed one of the symbols because the muscle libc maintainer and I have this long-standing Disagreement about how no MMU should be configured if you're not using a no MMU system This does not apply to you at all and I've actually updated that but I haven't pushed it to the repository yet So it only does it for no MMU systems If you care about no MMU stay after we don't have time, but I've done a lot with them I Mentioned that elf is essentially an archive format for binary data Well, there's a slightly upgraded version called fd pic that annotates it with slightly more data and the reason for this is No MMU systems can't Have two processes using the same address space to see different things Because it doesn't have a memory management unit. It can't remap them It can sometimes have you know low watermarks and high watermarks where if you access memory outside of this window You'll get a seg fault But if you can access that address you see the same thing for every process Meaning if your elf thing wants to be loaded at address hex 1,000 and you have two instances of it running you can't Because the second instance would need to load at the same address as the first instance and they would get sad and The way most elf programs are linked the you know, there's like four interesting segments There's the code segment the data segment the BSS and the other one I'll think about it as soon as I stop trying but the problem is they're all put right after each other in memory and The addresses of them, you know this piece of machine code will go where I am you know or the start of this You know the start of the program plus this many bytes for the data or for BSS is the Ah our O data is the other one There's one code segment and three data segments basically one of the data segments is for data that you can That starts initialized to a value, but you can then modify it afterwards It's basically your global variables that are initialized to a value BSS is for your global variables that are initialized to zero so their contents don't actually have to be stored in the elf thing It basically m maps a segment of memory there that isn't backed by the file and then goes you know have at and then the Our O data is the stuff that's initialized to a value But then if you try to write to it you will get a seg fault because it is not writable You know and they're basically all glued together in elf as one big contiguous chunk of memory What fd pic does is it says these are now for can for Distinct chunks of memory load them wherever you can fit and that means you can share the code segment with other programs that Think they've you know loaded this and it they also it's a pic position Independent code normally we only link libraries that way where we say we have saved in a register or saved in some global thing That you know either ties up a register or causes an extra indirect You know load to slow your assembly down, but that means that Wherever you know it can be loaded at different starting addresses and that means if you have shared libraries Well each shared library doesn't magically know what all the other shared libraries you have running are so they can't at compile time Pick a unique memory location to be so they have to be relocatable because they could be used in combination with who knows What other libraries? Well, what they did is pi position independent executables are Executables built as pic so that they can be relocated and the security guys love this Because that way if you try to exploit it using an absolute address and it's loaded at a random location each time you load It's harder to exploit well the no MMU guys love this because it's like I can have three instance of this loaded at different Locations and they don't you know it's not going to conflict with other programs that think they're using the same thing But what FD pick lets you do is It lets you say okay the read-only section can be shared with other instances of the program The code section can be shared with other instances of the program the data section and bss You need your own instances of because you're going to write to that data So these two are shared wherever they can fit these two are unique wherever they can fit However, if your memory is fragmented you can't use the MMU to collate it But you can fit it into smaller chunks because we've broken the program into four different pieces that can load wherever you can fit Them FD pick is much much nicer for no MMU systems But it means you need a different kind of Program loader You know so it when you build the program you need to produce a different output format, so again went off on a tangent But anyway that me arguing with the muscle libc maintainer Yeah, so anyway It does a build statically linking and then it doesn't install and then clean up is that RM dash RF thing And then this is a busy box install where I did something called mini config And the the last little bit I'm gonna get to and I think I'm going over time probably but the last little bit I'm gonna get to is configuring, you know Drilling down into the kernel config This is the same format as the kernel config what I did is instead of a def config busy box I told it these are the symbols I want Only these symbols start with all no config and there's there's a bunch of them You know, but you know, this is less than a hundred symbols and busy box has hundreds and hundreds of symbols a Full config file for busy box, which I just built is 1097 lines this is It ends online 261 and it started online 191 so 150 symbols instead of a thousand and stuff and that's for selecting, you know, rather a lot of stuff and What this is this is a format called mini config, which I did write up documentation on once I Tried to send this upstream to the kernel of documentation on how to do it. You'll notice this was 2005 I Resubmitted it again a year later to see if they changed their minds and like nope bunch of people are using this Documentation on it is not submitted upstream of the kernel if you ever think about Linux kernel stuff Oh, well any interesting idea immediately gets implemented and done by the vast hordes of kernel developers I will point you at the fact that the squash FS developer the squash FS developers spent something like seven years Attempting to merge squash FS into the kernel He took a year off to full-time Push it upstream into the kernel after it was already in red hat Soussi Debian Gen 2 and like everything else it had been merged everywhere and the kernel developers wouldn't take it I spent something like seven years getting pearl removed from the kernel build environment from The first oh wouldn't it be a good idea to use temp FS for init ram FS From me first mentioning that on the mailing list of you know Is anybody working on this to me actually sitting down and doing it was something like nine years Okay, so yeah the the normal state of things is for somebody to do it off in a corner And then it's abandoned and it never goes upstream into the kernel if you know brow beating all these developers for you know Oh, you implemented stuff for your company and you've been using it in-house for years, but it is an upstream end of cartel That's actually fairly normal It's sad, but it's normal so Before that tangent. I was attempting to explain. Oh What was I attempting to explain I closed the window? Oh, I was trying to explain mini config where I did I actually did documentation Describing, you know all sorts of stuff about mini config, but I will walk you through it right now Basically what you do is you have keyword equals value config symbols like this in the same format They appear in the config file But the question you ask yourself is if I start with make all no config in the linux kernel in busybox and anything That's using menu config and I go Okay, I've switched everything off Now I want to pull up that menu config tool and I have a checklist written down of look for this symbol and Switch it on and every time I do so there's a dependency resolver that may switch on other symbols for me or change their values or whatever Well, that runs every time I poke a symbol in menu config switch it on or switch it off It may have fallout with other symbols in the kernel But I don't really care all I care about is I started with all no config And then I went down my checklist switching on these symbols and then I saved the result and I built it I know how to do that by hand Can I automate this process and yes you can that's exactly what mini config does and The magic line here is make all no config kconfig underscore all config equals the name of the file containing all these symbols It's a horrible name a Large part of what that patch I had for many years ago was having a make Mini config target that among other things would tell you if you tried to set a symbol that didn't exist instead of silently ignoring it it would go oops error you screwed up, you know that kind of thing um and There was bike shedding from a guy named Roman Zippel and even though he's not there anymore, it's I Tangent tangent so this is a fairly useful technique For configuring busybox a lot more, you know concisely and it used LD. Oh It said it is an environment variable not on the command line Make is terrible I'm sorry it people give Python guff about the significant white space and totally ignore the fact that make requires tabs not spaces In a lot of places like no it it combines imperative and declarative code in the same context where you have to care what order things happen in but Have no control over it. It's like No Tangent there's a whole rant on there Yeah, configure make and make install all kind of need to be replaced. It's just nobody can really agree on Well, I already went into the whole autoconf is you know Totally does not work with cross-compiling and half the things it's checking for specified by POSIX anyway And if your build environment isn't POSIX fix your build environment And make install of course has no idea what a package manager is so things Anyway, but meanwhile we do configure make and make install and config prefix just tells it where to install and I'm installing the unstripped version in case I want to You know run Gdb against or something and Then this was my attempt to copy the dynamic libraries out of the cross-compiler onto the system to give the option Of dynamically linking and just ignore this entire section because it doesn't work you can actually see that it's my second attempt and The problem is I can get it easily to work with any one cross-compiler getting it to work with all cross-compilers Which may be using different C libraries? There's a lot of testing there. It's on my to-do list so just Ignore the thing about trying to make that work and then if I want to build additional packages You know, I have a second example here that you know, what if I want to cross-compile more stuff? Well, here's cross-compiling, you know zlib and drop bear and installing those into the system Here's just examples of using the same infrastructure to do that with different packages in case you want to cross-compile more stuff and Then at the very end you may have noticed I kept pulling up this file to look at the end where it's like and here's the invocation to Turn that into acpao.gz file, you know So that is what this build script does and if you feed it one of the muscle cross make Toolchains which I've been talking to Rich Felker. He and I actually worked together To try to get him to do binary releases So I could just point you at a URL of download the binary of the pre-built tool chain extract it add it to your path and then follow along building the thing and It's not up yet because he's been really really busy and I'm not hosting it anywhere because it's gplv3 And I don't want to get any of that on me I Launched the first gpl enforcement suits that actually wound up in court suing people back when I was busybox maintainer I know a whole lot more about gpl enforcement than any sane person would ever want to and after a year of empirically going through and Proving that none of these companies that were shipping busybox binaries had any code We wanted not adding one line of code to the busybox repository after looking at a whole bunch of sorcery dumps. I basically went I I don't want to go there and I really don't but but it made me a lot more careful about license compliance stuff because you can't You can't spend a whole lot of time dealing with legal stuff without being really careful to make sure you don't have to again Okay, so that is an example of building a simple root file system, but building a simple kernel deaf config was terrible So let's walk through real quick because we're short on time Let's walk through kernel building So I actually I showed you the start of this this file before See I configure create to config file. I haven't really had time to talk about device trees, but when the kernel boots The bootloader there's actually two bootloader stages on real hardware you have a bootloader stage one that does Two minutes lovely, okay Ask me afterwards the the real hardware does things that QMU doesn't have to So QMU is a built-in bootloader. We've just been using that and explain it now so so we start with all no config and In order to do a standard x86 system. These are in theory the three symbols you need what board am I using and Where is my serial console? So I set these three symbols I then do the make I then boot it and I get no output And the reason I get no output is that printf is switched off Why is printf switched off because the symbol config embedded was broken by Kim by this commit Which was really stupid special case nonsense that that says when all no config happens switch this symbol on Which results in opening a menu so that a lot more symbols can be switched off and this this winds up switching off printf So what you actually do is? You do a special commit So you have a line That looks like this config embedded is not set which ordinarily in which is how you tell Dot config a symbol is switched off. Yes pound sign means a comment, but this special comment is functional So we add this line and then we get output, but it says no file system could mount root So we have to tell it to enable the check for external and it our D which are these two symbols and Then it says it couldn't execute bin sh. So I have to basically switch on config bin format elf so it knows how to read an elf binary and then Then I get to the point where it reads hello world in order to be able to do a shell script I have to switch on config bin format script and Then I can run my init script Except it can't mount dev temp. It can't mount dev temp fs so I have to switch on two symbols in order to get the file system that gives me the dev directory and I Could I could go into more but basically this is the kind of thing that you know the the minimal set of symbols that I was using for each Target back in for three He's github.com Landly Back when I was doing Aboriginal Linux the set of symbols that I used for each target Was in the targets directory. There were a bunch of things like here's the one for arm v5 In order to boot a system. I needed to know what arch equals value to tell the kernel when I built What path the resulting bootable file would would show up at that's for building GCC. You don't have to care about that What console value to feed it because which which console device am I using and then here is the all no Can here's the symbols I added just for this architecture for this board and the set of devices on this board and here was the QEMU invocation I would use to load it and that Set of symbols was actually added to base config Linux Which has more than you actually need but these were the common symbols that I would set on all the targets So you can actually read through and see what those symbols are You don't need a lot of these and if I if I had a little more time I'd go through the here's the list of the ones you need but unfortunately I am over time I can probably take one or two questions. I Cannot okay. Oh, sorry somebody needs the room. I'm sorry ask me afterward