 Hello everyone may I introduce you to Andrew Tannenbaum operating system hacker extraordinaire Author of minix and it will be talking about Reimplementing netbsd on top of a macro kernel Never shared the stage with the flying goat before but it's the first time for everything Okay, I wrote the first version of minix, but since then all my students and programmers have been doing all the work Three of them are actually here guys Lannell Ben and around stand-ups people can see you also direct all the hard questions to them Okay Let's go So the goal of our work is to build a reliable operating system. So Here is my definition of a reliable operating system, you know, your mileage may vary An operating system is said to be reliable when a typical user has never experienced even a single failure in His or her lifetime and does not know anybody who's ever experienced a failure. That's my idea of reliable. Okay My stereo is reliable for example You know my watch is reliable In engineering terms, we're probably talking about meantime to failure. Maybe 50 years something like that. Okay, I Don't think we're there yet. Okay So I think there's some work to be done yet. Okay This is what I call the television model at least before smart TV. So you go back about five years, you know, it's got Three steps one you buy the television To you plug it in Three it works perfectly for the next ten years. Okay, this is how televisions used to work until I became computers and everything changed Okay Now here's the computer model Windows edition You buy the computer You plug it in now two-thirds of the way there, right? No, just one step about it works perfectly the next ten years that we haven't got to yet Now you install service packs one through nine F and Then you install 18 new emergency security patches since nine F and then you find install seven new drivers somehow Okay, then you find the anti virus software and install it and then you install the anti spyware software And then you install the anti hacker software firewall And then you install the anti spam software And then you reboot the computer. Okay But we're actually we're not done yet. I just ran out of space on the sheet There's more. Okay, it doesn't work So you call the help desk Okay, you wait on the hall for 30 minutes And they tell you to reinstall windows Which is what you've been trying to do all along right So the typical user reaction to this is something like this The New York Times actually reported once the 25 percent of computer users have gotten so angry the computer. They actually hit it Okay, and you know, it's not the monitor. It's not the problem. It's not the monitor. You know, I mean, yes It's probably running. I mean it's more likely you're running lending since that's inside the monitor these days But you know ordinary people don't have the understanding of you know, like we have that nothing works Okay, um, so the question is is reliability important Okay, and it doesn't work. It's annoying You know, you might lose work if something goes down But also think about industrial control systems and factories, you know Some thing moving cars down the belt and you know, I don't like to stop that abruptly because you know All these cars Bang of each other, you know electric power grids, you know Those guys kind of frown upon computers failing from time to time Hospital operating room surgeons and a majority of surgeons do not like the lights to go out and the equipment to go out in the middle Of an operation. This is not one of the their favorite things Banking and e-commerce servers those guys talk about, you know megabucks per minute when something goes down Emergency phone centers in at the canum, you know crashes once in a while, you know control software and cars and airplanes, you know There's other You know other things that people don't like when it, you know, it sort of fails. Okay Um And so the question is can you make anything reliable? Okay, and well, we won't find out, you know, if we don't try So, um, I was very lucky in that the dutch rail academy of sciences gave me two million euros to go try So, okay, I'll thank you. I'll take it, you know Then the european union on top of that when that contract ran the european union gave me two and a half million euro To continue so we had a little funding for a while. Well, that's all run out now So we're trying can we make it? I don't know we're gonna try First of all is reliability achievable at all, you know, can it be done, you know But actually systems can survive hardware failures think about some examples, you know A raid can survive a failed disc. So if you have a raid system of some kind disc simply fails, you know, the the head gets a little bit too close to the platter It scrapes all the oxygen off the disc makes a big noise Raids can continue after that they're designed with full tolerance in them. They can go quite happily after losing a drive Okay, and many of these things build a bad drive out throw in a garbage put a new drive in hit the rebuild button somewhere Everything goes on merrily along Ecc memory can survive failures in memory. They can correct memory errors on the fly Tcp can survive lost packets, you know sends a packet times out doesn't gonna answer it sends it again, okay You know cd-roms, but you know the layout on the cd-rom drives about three quarters of the bits are air correcting bits Okay, they come in you know what you think of as an eight-bit group They think of as a 14-bit group with an error correcting code and every 2k block is actually about 4k with error correcting bits And you know this massive error correcting structure with all kinds of fancy codes that you can survive Um those things, you know, and so we should be able to survive if we can survive hardware failures for crisis We should be able to survive software failures as well Okay, so I think we need to rethink operating systems and the research about them You know, we have fairly powerful hardware these days on pc class machines. There's lots of cycles. There's lots of ram. There's lots of bandwidth You know Plenty of software has got tons of bloat in it useless features and all of you know about that You know, of course, none of the bsd's of any bloat, but you know Some unnamed software is a full of bloat So it's slow and bloated and buggy and you know, I don't know to achieve the tv model for the dumb tvs I think operating systems need to be smaller and simpler and more modular and You know more reliable and more secure And self healing which is I'll talk about quite a bit Okay, so I'm gonna give you a brief history Of uh work we've done, you know, some of you know in 19th probably mostly weren't born then but 1976 John Lyons wrote a book about version six eunuchs Describing in detail, you know, how it worked line by line. So like the towel mood, you know, this line means this, you know And so on and then 18t when they released version seven they forbade books on this a number of books You know, we don't want anybody to know about our product. That'd be horrible Um, and then you know in about 85 or so I decided maybe I could write a eunuchs like operating system From scratch and then it wouldn't you know, people could learn about it. So I started that and took me a couple of years long story um some other time So I wrote a book and I wrote, you know, put the The operating system in a CD-ROM back of the book. No that came later We released in a floppy disk initially because that's all that there was the first talk this morning There's also pictures of these five inch floppy disks. We use those also because that's what there was um in 97 there was Minix 2 became POSIX rather than version 7 conformant 2000 changed the license to bsd license 2004 I got the grant first grant started to work in reliable operating system 2006 third edition of book minix 3 sort of corresponds to the third edition of book I don't know if there'll be a fourth edition I don't think we'll change the name at this point Then we got the european grant and then the focus moved toward embedded systems and net bsd Okay, there's been three editions of the book different pictures. All right, you know um now a little bit about intelligent design um At least as applied to operating systems Of people other users, I don't know It's got a micro kernels about 13 000 lines of code, you know linux has got 15 million lines of code windows I believe xp had 50 million lines of code and I asked Dave probert once how big was like, you know, window 7 And he said, you know, depends on what you count, but you know, maybe 100 million lines of code something like that And we've got 13 000 lines in the code in the kernel Um, there's been a lot of studying studies of how many bugs per line of code It's kind of linear not quite but people have studied that you know Too many companies have bug tracking systems that keep track of this stuff and people have studied Some of those and like one to ten bugs per thousand lines of code is sort of the normal range You do a really good job and get it down to one bug per thousand lines of code But that's relatively rare and it doesn't decrease over time either because people like keep adding new codes, you know add new bugs um So minix you might have 13 bugs in the kernel because it's got 13 000 lines of code and you know Linux probably has 15 000 bugs in it and bsd, you know comparable Number not all the bugs are fatal and the bug might be typing error in some message somewhere It doesn't you know spelling error somebody got a comma wrong and some message at displays But some of the bugs, you know are more critical bash as you may have heard Had a bug in it recently that got a certain amount of attention. I've been in there for 25 years. Nobody noticed that You know these things happen You know drivers tend to have people have studied this three to seven times more bugs Than the rest of the kernel and 70 percent of the code is the drivers. That's not a good, you know a good ratio there um, I think good systems should be modular and You know example operating systems aren't the most complicated Things around like an aircraft carrier is much more complicated than an operating system Okay, it's a very very complicated system, but they understand that the systems there should be modular So, you know, um, if for example, one of the toilets gets plugged up It doesn't begin firing nuclear missiles Okay, because the missile system and the toilet systems are fairly decoupled You know and likewise if incoming missiles detected the toilets don't start flushing because They understand, you know, you want a modular system where the different parts don't interact all over the place You put, you know 10 million lines of code in the kernel and you know, who knows what the interactions are So I think modularity is a very important thing here um You know isolate the uh, the components all the little modules want to move everything we could out of the kernel Um, that means all the drivers all the file systems memory management process management pretty much everything's been moved out of the kernel Every module runs with um, the polar the principle of least authority So a module is running out of the kernel only has the authority to do those things it needs to do So if you're an audio driver, you need to access one way or another to the audio card But you don't need access to the disk Okay, so we've tried to make a very very big effort to limit what each piece can do And every time some use of land component does something is a check or you will authorize to do it If the answer is no you get back an error error code saying no permission Okay, so the next step was to isolate the uh, the i o So, you know, all the devices are isolated every driver is a separate process running in user mode protected by the mmu It doesn't even have access to its own ports Okay, it's got to ask the kernel if it wants to do actual i o So if this driver wants to do i o it's got to say, you know, write these values into these registers and tells the Microkernel don't do that the kernel checks if that particular process is authorized to use those ports If the answer is yes, it does it the answer is no gets back an error code saying no permission Okay, so somebody takes over an audio driver due to flaws in the audio driver It can make really weird noises But it can't fork a shell because it doesn't have the authority to fork anything Okay, so it gives you a lot more security goes to the modularization Okay, so step three Is um isolated communication So there's a lot of you know The system is all these processes in user mode for drivers and file systems and stuff Not everybody can talk to everybody if a wants to talk to b It's got to go to the kernel and say, you know, i'm a i want to talk to b First thing it does is check is a authorized to talk to b and if the answer is yes Then it'll send the message and if the answer is no you get back an error saying no permission Okay, so you can't just send to anybody whereas in a monolithic kernel any driver can you know talk to any other component There's nobody's checking And that again restricts You know damage on a sort of a need-to-know basis need to communicate basis Also kernel calls this the microkernel itself has a number of calls for very low level things like Creating a new stack and stuff like that But this is a different level of interface than the posix interface which comes to a much higher level Again, not every call is all available to every process So an audio driver can't do the things you need to build a new process So to the low level equivalent of fork because the bitmap for it for the kernel call You know the kernel checks the bitmap and if the bit is zero It says you're not allowed to to create a new process So the answer is no and so if somebody takes over Some component tries to do something that that component can't do the answer is no So that again restricts damage okay One problem we had was that um You know we originally had synchronous communication a sends a message to b waits for an answer Trouble is if a sends a message to b and b dies then a hangs because a you know You know the other half isn't there anymore. So we had to go to asynchronous communication. We don't like it Okay, now here's the the architecture of minix kind of in a in a nutshell um The bottom layer is microkernel it handles interrupts processing some of the schedule the clock Interprocess communication kind of the low level stuff. Okay next level up are user level processes all running protected by the mmu as normal processes for the drivers like the disk driver and terminal networking and you know printing and all other drivers running as um Usable processes above that are the servers which are sort of the real operating system You know virtual file server and any other Any actual file servers and the process manager and the memory manager and things you expect in an operating system Are all running a separate isolated processes. There's actually no difference Technically between those levels. They're just separate processes, but conceptually it's good to think about it like that And then above that are the user the true user processes running shells and make and you know other user applications um So the user mode drivers Each driver runs as a user mode process. It doesn't have any super user privileges So the operating system drivers and the other pieces have only the the polar principles principle of least authority They have exactly what they need controlled by various bitmaps in the kernel and so on um They're protected by the mmu so they can't escape their address space Um, they do not have access even to their own i o ports. They have to ask the kernel It's embarrassing if you're a disk driver. You can't touch the disk But the way it is, you know you as disk driver ask the kernel Can you please write these things onto the disk? You know i o registers and kernel checks and if it's okay It does it otherwise it doesn't okay um So user mode servers there's a whole bunch of them each server again runs as a separate process some of the Key ones or there's a virtual file server file Virtual file system, which you know sort of the top level of the Thing so basically when you do a read call Let's say as a user process there's a little library routine in libc And all that thing does is it sends a message to the virtual file system saying here are the parameters for read Go do the read and the virtual file system figures out which actual file system is and you know sends out the message and Go do the read So there's the actual file servers. There's the native minix file system. We have xd2. There's iso 966. Oh, there's you know various other file systems I don't know if we have fuse or not. I think we might have had that once I'm not sure if it's still there, but you know other just like any other systems vfs and lots of you know file systems under it Um, there's process manager. There's memory manager. There's a network server It's this thing called the reincarnation server, which is unusual its job is to reincarnate the dead Okay, that's a useful thing to have basically it sits there Pinging the other pieces. It's sort of the parent of everything it pings other pieces So the reincarnation server will send a message to the disk driver and say hi hi disk driver How you doing and this driver says doing great serve 47 requests last second You know and then a little bit later it says hi disk driver. How you doing? And it says doing great served 74 requests in a second and then a little bit later. It says disk driver. How you doing? disk driver, uh, how are you doing? I'll give you one more try How are you doing? If there's no answer, you know, it'll kill it start a new one and can regenerate it Okay, um, you might say how do you fetch the disk driver if you don't have a disk driver? Well, we keep the disk driver and ram And once you have a disk driver from ram you can get the rest of the drivers from the disk if need be So it can repair the damage which makes it Self-healing which is you know that and part of the scheduler runs in user mode too So the policy mechanism separation the policy runs in user mode and the mechanism runs in the kernel So here's an example On how things sort of normally work user sends a message to the file I've eliminated the virtual file system part from simplicity. Suppose the block you want is in the cache So you send some message to the file system file system, you know Just sort of kernel go copy this block to user mode, you know kernel replies. Okay copies the block to user mode user got his block, okay um Now suppose it's not In the file systems cast use it calls the file system file system says I don't have the block calls the disk driver says Please go read this block, you know disk driver tells the kernel go read the block eventually a notification I interrupt comes in from the disk that's turned into a message at a very low level disk driver gets a message saying hi, I'm the disk It doesn't actually have any content this driver then starts it looks at its registers figures out, you know What happened and then it tells the file system? Okay, the the dma is completed. You've got the block in the place you asked for it and then it tells the The kernel will copy it to user mode copies to user mode and users happy So there's a bunch of messages back and forth and these things take some fraction of a microsecond So there's some some performance hit here other people We haven't really focused much on performance other people like the l4 people l4 linux Have got the performance hit down to 5 or 10 percent. We haven't really Focused on that but it's doable, you know from their experience. So again, here's this reincarnation server It's the parent of all the drivers in the servers and when a driver, you know server dies It hears about it, you know trial died and it can collect it basically runs a shell script What I do now, you know, it can notify the administrator it can go pitch the new driver It can can log it somewhere can go all kinds of stuff and then go get a new one, you know And then in a sense it pings them frequently to you know, see how they're doing Okay, so here's how like disk driver recovery works just as an example Okay, so the user sends a message to the file system, you know, go read a block or go read a chunk of file File system sends a message to the disk driver, you know, he wants this block number so-and-so And this driver fails, okay hit a you know bad address or something Okay, what happens is the reincarnation server hears about that, you know, it's the parent It's child died. It's notified of that and this is oh my child died. It's horrible Let's do something which I do. Well, I'll have a creating a new one So in case of the disk driver, it takes a copy from the RAM disk In case of any other driver, it goes and fetches it from the hard disk Or wherever you want it to fetch it from It starts with a new one, tells the file system, file system then tells, you know It has to remember in its structure what it was doing, okay So it says to the new driver, okay, can you read the block and you know, you're back on course So the idea is that you can recover from actual failures. It's fault tolerant A fault tolerant means you can actually tolerate faults Which is a different strategy than trying to, you know, avoid faults In the ideal world, there wouldn't be any faults But experience shows there's one bug for a thousand lines of code No matter what you do, so they're going to be faults So we have probably as many faults per line of lines of code as anybody else does Only they're all separated and isolated And the damage they can do is less because of the system structure Okay, so the system is self-healing And that's a very important, you know, concept of It can fix its own errors to some extent Not for everything, but for some things We're working on more things There's also some security implications here Fewer critical lines of code means fewer critical bugs Total bugs are probably the same But they're less dangerous because they're all, you know, up in the user space So the trusted computing base is smaller than most systems Also, there's no foreign code in the kernel It's important, you know Other systems, you get a new device It means you've got to put somebody else's code in your kernel And as a fair chance, that code was written by some kid in Taiwan And his boss was beating him, finish, finish Let's get it shipped, ship the product, ship the product The kid didn't have time to test it, so they skipped that And you're putting that in your kernel, okay Here, you're putting that, you know, in theory in a user mode process Where it can't do as much damage if something's wrong with it And of course the drivers are very buggy when you know that We've got no malloc in the kernel It's all static, which is a little bit wasteful But it's also more reliable Because, you know, I don't have to explain to you guys what malloc and free And all the bugs you get from that So, you know, so putting bugs in user space kind of cages them You know, there's a number of bugs Probably isn't any different than any other system But they can do less damage because they're, you know, isolated Okay Interprocessed communication We use fixed-length messages We changed the message length For the first time in 30 years last week It's not 64 bytes It gives us more space to do things But fixed-length means no buffer overruns And there's one constant somewhere in the header Which says message size Actually it's not a constant It's a big, it's a bunch of structs for the different message types And there's a union of all these things And that determines the message size and it's 64 bytes So the variable size leads to buffer overruns And allocation problems, you know, of any of that Because it's simple We hit a rendezvous system A sends to B, you know, waits for an answer The answer comes back We eventually had to give up on that Because the problem is, you know, clients sent to the server Server did some work sent back And the client died The server was hanging with this message it couldn't get rid of So we had to go to asynchronous messages Sort of against their will to some extent But that was a good scheme while it lasted We tried to use it as much as we can But there are situations where you can't And so we had to add the asynchronous messages Interrupts and messages are unified at a very, very low level When an interrupt happens That's instantly by the microkernel turned into a message To whoever's, you know, sort of listening for the message So we don't see interrupts Except for a little bit of assembly code at the very bottom You know, as soon as an interrupt happens Somebody is getting a message And then they do a receive and they get a message And they can do whatever they want with it So we get interrupts out of the way very fast Okay Driver reliability Drivers, we assume drivers are untrusted Everybody else assumes drivers are trusted Because they're going on the kernel So our attitude toward drivers is you can't trust them Which I think experience shows is probably true Bugs and viruses can't spread to other modules Because each one has a separate process The drivers and the pieces of the operating system They can't touch the kernel code They can't touch the kernel data They can't touch, you know, other pieces of the operating system All they can touch is what's in their address space And the MMU keeps them from touching things outside their address space So they can't, you know, there's all kinds of leaks That can happen when some drivers read in critical kernel tables They can't, there's no way to do that Unless there's a kernel call Which they're making Which gives them access But then there's a bitmap Which says whether or not they can make that kernel call And only if they're authorized to do that Which is generally not the case Can they do that So we can limit what they can learn And the damage they can do Bad pointers or buffer overruns That kind of stuff Can only affect one component Because they're all, you know, isolated like that Infinite loops, you know If somebody gets into a loop And somebody else pings it And doesn't answer the pain Because it's sitting in a loop Well, the other guy will kill it You know, restart it So we can recover from infinite loops Most systems If there's a bug somehow In the critical kernel code And, you know, it's looping Well, you know, it just hangs In our case, it will get killed and restarted Access to the IO ports is limited Access to kernel calls is limited Access to, you know, inter-processed communications Everything's limited So it's very, very walled off Okay You know, other advantages It's a shorter development cycle Because developing something Is just a user mode process You know, use command Go start a driver And it starts it and use a mode And, you know, if it dies Well, you can try to debug it You can get a core dump To, you know, sort of the normal programming model For applications programs Which makes it easier Than dealing with the bare metal So there's basically no downtime If something crashes It's just, you know, process crass It's the same way user process crashed So debugging is easier And it's more flexible You can do the file systems That's why everything's just a user process So it's an easier model Than dealing with the bare metal Okay We did a fault infection experiment A while back We injected 800,000 faults Into each of three ethernet drivers And we basically I did it on the binary drivers So we weren't recompiling So, you know, we had a You know, a P-trace-like mechanism Which basically went into the binaries And made changes But it didn't write random junk It emulated changes That similar to programming errors Like there'd be a Sort of swap the source and destination addresses Or it would change a loop condition From branch less than To branch less than or equal You know, the kind of things a programmer might do So these are very carefully We built a whole framework To make very carefully orchestrated errors like that And we introduced like 100, you know, faults And then let it run for a while Waited for a second to see if anything crashed And if not, put another 100 faults Not through with you yet It's another 100 faults Well, eventually we got them all to crash But even though we brought down the drivers You know, all the time We never lost the operating system So, you know, the driver would crash If you were killing the driver every two seconds Performance went down I mean, you know, there's a hit there But if your driver is crashing every two seconds Probably you should try to debug it, you know So, anyway So now the story of the port to the arm We had a restructure of the source tree You know, it's funny, minutes one I actually had targets for the 68,000 For the Atari and for the Amiga And for the Macintosh And it had the spark And it had a whole bunch of different ports And it sort of got lost somewhere in the 80s or 90s And so we had a restructure of the tree again To handle multiple architectures The support for U-boots Which, you know, we heard about earlier We wrote the low level code for dealing with the hardware Because the arm is, you know, the very bottom code Interrupts and stuff a little bit different The contact switching, paging and so on Is somewhat different So that had to be changed The room did most of that The X86 segmentation had been in there from the beginning It started on the 8088 Which only had segmentation Didn't have paging And when the 386 came in We added paging But segmentation was still there We figured, you know, maybe it's time to throw it out It's like a thrown out Imported the NetBSD arm headers and libraries Build.sh came in Landed a little out of that Wrote drivers for SD cards And some of the other Beagle devices So there was a lot of change When we sort of went to the arm And moved many things over that way Then we began looking at embedded systems as a target This is the Beagle Black Some of you know that's mentioned earlier Today it's about, you know, the size of a small phone But it's quite powerful It's got, you know, it's an ARM V7 32-bit architecture Clock runs at a gigahertz It's got half a gig of RAM on it Four gigabytes of disk flash memory It's got HDMI, it's 1080p It's got 92 Lio pins You can control things from 100 megabit ethernet It's got a USB port on it It's open source ARM is open source The documentation is open source It's in contrast to the Raspberry Pi Everything's a secret So here, all things open source You can find everything you want to know about the Beagle boards And the black is $55 And I think the Beagle board white is $45 So they're fairly cheap And presumably if you order these things in large quantities They're a lot cheaper Now, I will admit that I was wrong On January 20, 1992 I posted the com.o except Minix Don't get me wrong This is when Linux was getting started And people were talking about it On the news group Don't get me wrong I'm not unhappy with Linux I'll get all the people off my back And want to turn Minix into BSD I apologize I actually do want to turn Minix into BSD It just took me 20 years To figure it out Sorry about that You don't want to talk into Linux, then Okay, so Minix meets BSD What's going to happen now So here's we have a logo I have a logo of some kind We sometimes just use the head Sometimes just use the whole logo And the M And there's BSD Thanks to Arun for this idea And the demon that you probably know Is copyrighted by Marshall Kirk McCusick And thank you, Kirk, for permission to use this Okay, or maybe I'm not sure what this mixture should be Something like this Why BSD? Well, because we didn't have much application software And we had sort of the version 7 software But not much else And BSD is sort of a proven stable Mature quality product And it seems to be You're going to join up with somebody That's like a bunch of people to join up with I think the code by and large is I mean, Linux sort of just grew by topsy And I don't think quality control Is high on their list of a lot of things There's a lot of things that do you very, very well But the code is a little bit messy sometimes And we have package source And package source is wonderful, as all of you know And there's a lot of packages out there Thousands of very good packages There's an active community out there There's a license compatibility issue Which is, you know, as we heard earlier today Not everybody likes the GPL version 3 We don't even like the GPL version 2 or the version 1 And so the BSD license says Go take it There's only one thing you can't do in that Sue us But other than that, do whatever you want So, you know, there's a license compatibility Why net BSD as opposed to one of the other BSDs It's a tough call But our feeling had been Because net BSD is so, you know Emphasized so much as on portability That, you know, portability forces you to have Relatively clean code If it's going to run on the kitchen sink, you know And everybody's motorcycle Then you can't use weird peculiarities of the hardware Because it's not going to be available On some other platform So you kind of have to, you know Keep it relatively clean We thought that would be true And from what we've seen it That's sort of relatively true And then maybe somewhat less so on systems Where the emphasis is on performance Or on, you know, other things So we thought net BSD would be a good choice We made that decision So some of the net BSD features The main compiler is Clang, you know LLVM toolkit Which is a very nice compiler And this is an aside One of my students is trying to do some work on Linux using our software And we need to compile Linux with Clang It's horrible Just horrible It's tremendous, you know Hundreds and hundreds of places in the code Where it doesn't compile Because they've used peculiar features of GCC Which are not part of the C language And they have to sort of go after them one at a time And there is a group trying to maintain Linux In Clang But it's a struggle And so anyway, we have Clang And we've stuck to official C definition For the most part And okay We have net BSD build system Build.sh Lionel's done the work on that It's a very good build system As I'm sure many of you know It helps with portability Elf file format Source code tree is kind of a model on the net BSD tree So net BSD person looks at our tree He's saying this looks familiar Headers and libraries are all stolen from net BSD We have package source We're able to build about 4200 packages out of the box This one thing we don't quite have X11 used to work But it sort of got broken somewhere along the line Where the process of fixing it Which I think is maybe a few weeks work And that'll open up probably another 3000 packages But anyway So we're almost back So here's the smallest things we don't have We don't have kernel threads It just got this long history of back Between educational system and it wasn't in there We do have pthreads in user land As a user land package So packages that need pthreads And don't care whether they're user land or kernel They work fine If it's got to be kernel feds Then there's a problem Some of the system calls are missing We don't have LWP or message or SIM calls I mean some of those could be added probably You don't have clone Some of the get-nioctal calls KQ and Ktrace aren't there Job control isn't there Sorry if you have X I don't see why you need job control But you know Some of the weird socket options aren't there But you know A lot of it's there We can build 4200 packages QA tests Bottom, you know There's 500 tests that fail But there's 2100 tests that succeeded So the bottom line here is That 81% of the QA tests passed So to get through 81% of the NetBSD test You're sort of 81% compatible in some abstract sense So we're not fully there But we're a lot of the way there 81% of the way there Whatever that's worth So here's the system architecture There's the microkernel There's the drivers, the servers That's all MINIX stuff And then on top of that Is user land The regular packages and package source and shells And that's all NetBSD So what we've done Is we've reimplemented the NetBSD user land And software and packages and shells And all that stuff On top of a different underlying substrate Which has these self-healing properties And reliability properties And all those other things That other systems don't have So it's a funny mixture of To the user level stuff It's just NetBSD Or 81% NetBSD I don't know if that's linear though Because the things we didn't do are kind of peculiar So I think most things don't need that stuff But there's a few that are going to need it And the bottom layer has all this reliability stuff And the bit of security and why not So it's an attempt to rethink The way one could implement NetBSD We have these three Beagle boards We've worked on It's the Beagleboard XM Beaglebone white and Beaglebone black And you can't read the letters But basically green is good and red is bad Okay And on the black Which is the one we've mostly focused on It's mostly green, it's a little bit of red There's a couple of strange things Like analog digital converter We don't have the driver for that And the yellow is the USB USB has just barely made it into the distribution Hot plugability doesn't always work But we're fixing that So if you plug a stick in live It might not work But if you put the stick in first And then boot it, then it works So we're not quite there But we're most for the black And for the white, we're mostly there Now with your role Like where am I here Well it's an open source project Okay So all the sources are out there And we hope some of you will join That's certainly people who have an interest in NetBSD And want to see a new way of thinking about NetBSD Hope some of you will join Here are some of the things you might do There's some system calls that are missing You know, we don't want to have every call Necessarily things that are really hard to do And mess up the structure And aren't that important You know, maybe it's better to do it without them Because I think simplicity is really the key here But there might be a few things That aren't that hard to do But which would be really quite useful And we didn't have the time Maybe they should be added Certainly more packages Some packages are blocked Because some library is missing Or whatever We don't have Java We have links Which is a text browser But you don't have a visual browser There's Dillow There's a bunch of little browsers out there Besides Firefox is gigantic But there's a bunch of smaller browsers out there That are reasonable It would be nice if somebody would find Some other graphical browser and port that We don't have every single driver for the Beagle board So if we don't have the analog digital driver There's a couple of the minor pieces we're missing So we could do one of those Get running on the Raspberry Pi and other platforms So, you know, that's the attitude As it runs everywhere basically People could try to make this run everywhere You know, Rump I was talking about Rump this morning It's certainly an interesting idea We're interested in Rump You know, we don't have any kind of a GUI really Just sort of When X11 comes back It'll be proper X11 To use it now The best way is you'll be to SSH into it From some other computer And then just work on the other computers You know, X or whatever, just X terms But there's a couple of libraries I think if you ported those It might well be that some of the GUIs would work After you get a couple of libraries ported And there's a lot of other things to do So there's lots of interesting projects around For somebody who wants to sort of hack on this So get involved So here's Minix in a kind of a nutshell You know, it's open source It's got a BSD license And as all of you know There are companies You know, I have one of my students used to work at IBM And they wanted to put, you know Help with Linux And he said, you know, IBM was really The people in the company said, you know, we're not really keen About putting a hundred million dollars into this And giving everything we did into our competitors You know, and the GPO of course required that And the BSD license If you want to tell us what you did That's fine If you know what that's also fine If you'd like us to put it in the system So it's there in the next release And you probably want to tell us about it But it's your call You don't have to if you don't want to It's one of the strong points of the BSD license As all of you know So it's quite compatible at user level 81% compatible Whatever that means With Net BSD It supports LLVM and GCC You know, the preferred compiler is LLVM Clang But we do have GCC also It uses package service Which is, you know, wonderful as all of you know We build 4,200 packages And it'll soon be about 7,000 When we get X11 running again Another thing which we have in the lab But it's not on the current release Is live update We have the ability We have the ability in the lab To replace the operating system on the fly With the next release While it's running Without affecting running programs Okay So you want to put in a new file system Which has got different data structures And you know, everything We can do that on the fly basically Okay So the way it works is We start up a process And put it into the new version The new version then goes to the old version And says I need your state Okay Because we're using LLVM The compiler has put information Metadata in RAM about This is a struct And it's so big And it's got these fields And this is what they're called This is what they're types and so on So the new one can go to the old one And say, you know, okay Give me the first data structure And then it'll give it the first data structure And tell you it's, you know, it's a struct And it's got 6 members And here are the types of the members Near the values and so on And creates memory That thing Including possibly conversions from Oh, you had a linked list before And now you want a hash table So the guy who wrote the new version Has to write a little function to do the conversion But it's in principle doable Okay And then it says Okay, give me the next one And one of the time you do that And when we're all done We convert it back to the old format And we can compare them See if they're the same Okay It's like you use Google translate And you convert, you know, say English to Bulgarian And then you convert Bulgarian back to English If what you get is sort of more or less The same as you had You can have some confidence that the Bulgarian Is sort of right Okay And we can do the same thing And there's a lot of interesting properties And we have good security by design The modularization The fact that each module Is limited to the kernel calls it can do It's limited to who it can talk to It's limited to which IO ports it can touch All it's controlled by little bitmaps And tables in the microkernel So, you know, there's a lot of modularity The second thing which we can do When the live update is there Is there's a lot of exploits In operating systems for somebody Who's very carefully studied the layout of memory And knows if I overflow this buffer By exactly 18 bytes Then I get to write of this word We'll overwrite some return address When it returns, you know Then I can, you know, return to libc And all that kind of stuff That requires knowing the exact layout Of the memory, okay Suppose we change that every five seconds Which we can do with live update We just update to a new version of itself But compile differently Because, you know, LLVM allows us to say Throw in some random fields and instructs, you know Move the functions around Do all kinds of randomization And every few seconds We load a new version of the operating system, okay Well, those attacks That depend upon knowing the layout Don't work when they're changing it out From under you every couple of seconds So it gives you a very, very strong defense Against a certain class of attacks And that's, we wrote a paper about it Asplows a year or two ago About that So, you can go out to minix3.org And try it Positioning of minix We want to show that multi-server systems We don't really push the microkernel thing It's a multi-server system Because it's based upon a bunch of servers That we want to show that it's reliable We want to show that it's secure That it's practical That, you know, driver's belong in user mode That we want to handle high reliability And full tolerant applications Maybe somebody will make a single-chip You know, laptop for $50 for the third world You know, small memory footprint Embedded systems and so on We have a logo It's a raccoon, you know Because it's, you know, cuddly Sort of, you know, wire raccoon Well, they're small They're cute They're very clever They're agile They eat bugs Which is important And, at least in North America They're more likely to visit your house in a penguin, you know Hey, mom Mom, there's a penguin in our backyard So, website is minix3.org It's a fairly simple website, you know Documentation It's in a wiki wiki.minix3.org You can help You can help document the system So if you don't, if you don't write code But you're good at understanding things And you're good at writing You can certainly help us document things better By writing into the wiki Here's some of the traffic To Minix 3 You know, we made a release of Minix 3.3 Last week and traffic shot up to, I don't know 75,000 hits in September And the download page Had, you know, 17,000 visits to the page And from the log About 14,000 downloads So 3,000 people went down But the rest of them actually downloaded them The total number of visits To the main page since 2004 And when we keep, again, keeping track Is about 2.9 million So we're probably not in, you know We're still not in any BSD territory probably But, you know, almost 3 million visits It's not zero either The total number of visits to the download page Is 1.1 million And the number of actual downloads Since February 2007 Is about 600,000 So it's not zero I mean, it's not, you know, way up in the sky But there's a community out there There's a news group That's the main communication mechanism It was a Usenet news group And Google bought Usenet Now it's a Google news group So questions and requests And discussions and complaints and everything Go to the noodle Go to the news group So the conclusion Is, I think, current operating systems Are kind of bloated, you know And it's intent to build a smaller More reliable, secure operating system Kernel's very small About 13,000 lines of code The operating system itself Is a little tiny kernel Plus all these, you know, drivers and servers running As protected user mode processes With very strong restrictions On what each one can do Each driver is a separate process Each component has, you know, A limited amount of power Of what it can pull off Faulty drivers can be placed on the fly We're trying to be able to replace The stateful components on the fly But it's trickier Some of the components, like I said, The audio, you know, driver has state But it doesn't have a lot of state It's got, like, the volume level And the trouble level And there's a way for it to say There's a piece called the data store An audio driver can, whenever the volume level Is changed, can go to the data store And say, here, store this number You know, it's 7 dB And if it crashes, the new one comes back Goes to the data store and says Give me my state, you know It doesn't work as well for the things Where the state changes rapidly Like a file system, but we're working on that too Okay, so live update, as I said, is possible That's not on the current release You know, Stay tuned It uses the NetBSD packages and everything One other thing We're trying to do a survey We, you know, we have like 14,000 downloads this month We've got no idea who they are What they're doing or anything So if you give it a try You know, there's a survey on the main page There's a little, this is the work Over here it says, you know Please take our survey So if you download it, after you've done Please take the survey and, you know Tell us a little bit about What you're doing with it We kind of like to know We're trying to build a community And, you know So we've had 600,000 downloads But we don't know who they are What they're doing or anything So we're trying to get a little bit wiser on that Okay, one last thing If you want to get a master's degree in systems You know, we have a really cool Systems master's degree That's almost theory-free It's completely theory-free Very practical and so on Just Google me, find my homepage Go there, there's a movie on YouTube Link there about the master's degree Okay, that was it And if you have any questions If I can try to answer them So we at our company Have some experience with L4 microkernel And such projects like L4 Linux Which is a Linux kernel Which is adapted to run on top of L4 So like a user-line process Do I understand correctly that In this case You don't have any NetBSD kernel bits And instead of it, you're Implementing for NetBSD API That's right The thing about the L4 Linux is If there's a bug in Linux Linux crashes The L4 kernel is still running You've got to reboot Linux from scratch The only thing is Rebooting Linux from scratch on top of L4 Is a little bit faster Than rebooting Linux on the bare hardware But one bug in the audio driver in Linux On L4 brings down all of Linux The same way it does on the bare metal All you get is a little bit faster reboot That's not true here So it's a completely different idea You know, the idea there Is they want to have Linux running as is But in terms of full tolerance It's no more full tolerance Than regular Linux is Because one bug in Linux Brings down all of Linux It's just that they can reboot it A little bit faster The L4 is still there How do you deal with poisonous requests In terms of restarting services? How do you deal with the situation Of a poisonous request Which kills Which causes a service to kill itself If there's a request that comes in That is absolutely If it's a deterministic bug Because the request contains something in it Which causes the component to die Then basically we can't handle it That's simply a fatal error That our experience has been That most of the I think probably everybody's experience Is most of the errors are not Absolute deterministic failures That always fail the same way But they typically are transient failures That under certain peculiar circumstances Something happens that nobody's counting on And if you run it again You don't get exactly those circumstances But if it's a deterministic thing If it doesn't, you know If it truly doesn't know how to carry out the read system call And you ask it to do a read again It's going to fail the same way again So we can't handle deterministic failures But those are the ones that testing Tends to find The hard ones are the ones where it's, you know Unlikely because it depends upon A bunch of coincidences And next time you're probably not going to Hit it even though it's still there So we can't handle the deterministic falls But they're relatively rare So you make mention of the issue of performance And that you're not currently looking at that But is that something that's planted As part of MINIX 3? Because that's the usual accusation Against microkernel architectures Given a very small staff and group We simply haven't had the time And also being academics We want to publish papers and, you know That kind of stuff You know, we're, I mean The work on L4 and L4 Linux In particular has demonstrated That even with the microkernel You can get the performance hit down To about 5 to 10 percent And so they've shown that it's possible Okay We haven't put in the engineering effort To do it But if they did it Presumably we could do it Because it's kind of the same You know, we have the same problem they have And all this message passing And if one went to the effort to optimize it My guess is We could probably also get it down to And maybe 20 percent or 30 percent I don't know where the number is But if they did it in theory But we should be able to do it too We just haven't had the manpower It's only been, you know, four programmers And these other more interesting research things Have taken priority But they've shown it can be done So there is a hit But we're talking tens of percent at most You mentioned you're using the MMU To protect the kernel from being tampered with By the user line and processes Including drivers Are you using an IOMMU To protect against DMA From the hardware Sent a driver We're still not aware of We haven't been in an hour, I don't remember It should be Ben Hello everyone So IOMMU is the IOMMU business We're aware of the DMA problem, of course But there's not a really neatly integrated IOMMU solution in mainline That's just something that we have experimented with We started looking at it It was very hard to get years ago When we started looking at it Everyone was different There was no documentation We got working on one driver And kind of abandoned for now Getting that neatly integrated with Minix But that would definitely be a good addition But so at this moment No, there's no good protection against Physical memory access by DMA But it's no reason it couldn't be done It's just an engineering effort We haven't had the manpower to put in But yes, it's only needed No more questions? Thanks Since the DMA question was brought up I'm also curious to know what's the state of DRM I'm sorry, of any graphics related buffering issues Like is there a short-term plan to bring up Something like DRM from Linux land? DRM? Yes, the direct rendering manager Basically the piece of software that is Kernel and user site that takes control I think you would map the frame buffer Into some process that had authorization to do that And so it could get at the... Okay, so basically the kernel stages is frame buffer And that's it We haven't put any other effort into the graphical part of it yet Okay, okay That answers my question Thank you Any other question? Thank you