 The next talk is running Debian on inexpensive network attached storage devices and your speakers, Martin Michelmaier, about whom I promise not to say anything evil. Enjoy. There is nothing. I don't think, yeah. Well, there is nothing evil to say anyway, but with NetE you never know. Next time she is definitely going to say nice things about me. So this talk is about a couple of different things related to NAS devices. So I'm going to start by showing a couple of devices which we currently support. And then since this is Debian Conf and I'm hoping to get other people involved, I'm also going to show what kind of changes if you want to add a new device to Debian installer, changes you need to make. Finally, I'm going to talk a little bit about the lessons I learned because I've been doing Debian on NAS stuff for quite a while and it's quite interesting to interact with users and to see what kind of issues they have and maybe we can think about how we can improve that. So what are NAS devices? So it's basically, I mean, they're really popular these days because everyone wants to store their MP3s or their data. So it's basically an external hard drive but with some more logic in it. So it's not just a USB disk but it's actually a full machine so it has memory, it has a CPU and so you can actually run a full Linux system on it. And that makes it interesting because if you look at, for example, access points, you can run open WRT on it, it runs from Flash but you can't really run Debian on most access points simply because they don't have the storage for a full Linux or Debian system. But with the network attached storage devices, I mean, they have plenty of storage so why don't you just put Debian on it? And it's nice because they're generally pretty power efficient. They're usually pretty quiet and they are quite cheap. I mean, there is a big range, some of them are quite expensive. Unfortunately, especially those, some of them which we support are quite expensive but those are quite nice but we actually support a quite range these days. And in my opinion, they make a pretty nice home server if you don't need too much. So now I'm just, I'm going to present two devices in detail and then just give a list of devices we support. And the two devices I'm showing, the first one is the Linksys NSNU2, the slug. And I'm showing that simply because it's very popular. There are many people who use it. That was the reason why the Debian ARM port became very popular and is the third most popular port at the moment. So to me this device isn't very interesting anymore simply because it's quite old, it's quite slow. It doesn't offer very much but it's very cheap and so it's very attractive for people. But fortunately now there is actually a replacement available. But so with the slug it has 32 megabyte memory. It has a CPU that does either 133 or 266 megahertz. So you can see it's really very basic. You can run Debian but it's not going to be very fast. I think Wouter blocked about that when he bought it and then he realized just how slow it is. I mean you can't expect magic. But since it's very cheap it is very popular. A nice thing about the slug is also the firmware can be upgraded over the network which is very useful because if for some reason Debian doesn't boot anymore and most of those devices don't have a serial console by default so you don't know what's going on. And so people can take a rescue image and flash that via the network. So that's a very good rescue option. Well there are a couple of disadvantages. The ethernet needs some proprietary microcode. So even though we have Debian installed support we need some unofficial images to actually have ethernet support and it's very slow. It doesn't have enough memory. So like I said it's not so interesting anymore. To me it's in maintenance mode. It basically works. So I would just skip that to the more exciting stuff. And the device I'm really excited about at the moment is the Shiva plug because it's a really good replacement for the slug. It's much faster. So it has a 1.2 gigahertz CPU. It has 512 megabyte RAM and 512 flash. And it has one USB so you can again connect a hard drive. It's really pretty small. So that's the Shiva plug. And it's available for $99. With that 1.2 gigahertz CPU and 512 RAM. So it's really a very good deal. It's pretty cheap as I said. It's very quiet. Doesn't have a fan. It actually has enough flash to run Debian. At the moment we don't support that. But there's a Google sum of code project. I mean they've been making those devices basically to promote the whole platform. And the nice thing is that the CPU they use they have over the last year or two years they have become very open because with Marvel a few years ago I saw those CPUs coming out and basically I said I'm not going to touch that because it's evil stuff. It's not documented. The Linux which they have is their own hacked version and there was just no mainline support. And suddenly Marvel started to hire two of the main kernel hackers and a couple of other people who know about free software and open source. And so now they have a mainline port. They have a couple of people working on it. They have open documentation. So it's a really interesting platform. And so this one is really aimed for people who want to do something who want to hack it. So you have a way. Is that. Is it. Okay. So it actually has a mini USB which has both Serial and Chetak. So Chetak can be used if you for example break the machine by overwriting the boot loader you can restore that. You can basically flash the boot loader again. So it's a really interesting device. And the only problem at the moment in my opinion is that it's not very available because at the moment you basically have to order it from the states. I know some people from Europe have done that and the the the import tax wasn't too bad but it's still you can't go to a shop and just buy it as you as you can with the slug. But I think that's changing. So my very recently made an announcement that they have made some deals with a number of big companies like Buffalo to produce those devices. The only thing I don't know if those devices are going to have the Serial and the Chetak because there is something called the Pogo block which is basically this device but without the Serial and the Chetak. So this is called the development platform. But that's something we will see. So the the status it's currently under development. It's it's a pretty new. So we have a kernel in the archive which which works. And on my web page I have a top all of Debbie and Lenny with with a new kernel and that can be installed very easily. It just partitioned the disk and packet make some changes in U-boot by the series. It's very easy. And Debbie installed a support. Most of it has been integrated. But it it needs a new release. So it will actually be available for people. And like I said before at the moment we don't support installations to to the internal flash. So you can install to use B disk or to a SD card. But hopefully in the future we will also be able to directly install to flash. So a couple of other devices which are already supported. So they're grouped into which platform it is. So the the first one is the Intel IOP platform which essentially isn't developed anymore. So those are all the devices. The the Glenn tank is only available in Japan. The Intel is interesting because Bouter just in the last few weeks did the port and just committed the final pieces to Devin installer. So that's very new. I mean new in the sense of that we supported in Debian but the device itself is is not new itself. So that's that Intel box. And then you have the the feakers which is actually what we use on most of our build these. So that has been supported for quite a while. The the next one is the Marvel Orion. And that's basically what what we support best at the moment in in Lenny. So we have the the D-Link DNS two three two three two three. And that support was added by by Matt Palmer. So that was actually added after Lenny or he sent some patches but they were too late for Lenny. But we actually added that as part of a point three days. So so that support in and that's very good because this device is pretty cheap. It's then we support the HB device. But the problem with that is it's not very easily available in Europe. You can get it I think in the UK and you can buy it in the States. But most other countries in Europe don't have it. And the the Kura box pro support was added by Pear who's here in the audience in a Google sum of code project last year. Again that works pretty well. But the the Kura box is is again more aimed at developers. The the device you can buy in the shop is called the link station. And it would be very easy to support because it's very similar to the Kura box. But no one has actually done that work. And then we have the various QNAP devices. So those are definitely the devices with the best support. The company has been very helpful. Very supportive. And the only problem with those is they're a little bit expensive. And the last one is the the Marvel Kirkwood line. So the Marvel Orion basically goes is about 500 megahertz. And the new Kirkwood is about one or two gigahertz. So it's it's much faster. It's much faster I.O. And that is currently being worked on. So I integrated them install a support. But again we need to actually make a new release for that. So how does the whole thing actually work. So if you're interested in in in adding support for a new device what do you need to do. And for for developers you basically need a serial console. I mean that that's something you need because otherwise you don't see what's what's going on. And you need to boot the kernel and you need to see if that actually works. So that's definitely something you need. Chatec I listed here but I have never needed Chatec. I mean you only need that if you if you override the bootloader and then you have to rescue the the system but I have never done it because we don't actually touch the bootloader. Another thing which is very important is that we need to have working kernel support in the main line kernel. So that there used to be some devices which were very nice but they only had a you know a kernel which which was a vendor kernel which was in the main line and that's just something which we can't support. I mean we the kernel team has a policy against keeping big patches around. And it's it's just a big nightmare. And obviously it helps if you have a good relationship with with the hardware company that there may be some things you need to know or that there are some things you can do better if you have certain information. But obviously even even without any contact you you can you can reverse engineer most most things and just port it. For users it's very different. So what what we have been trying to do is to make it really easy for users because if you look at some some other distros or some other how to is about how to install Debian. It's basically a long list of you need to do that. You need to partition the disk. You need to format it. You need to do that. And then you need to change the the the config of the bootloader. And to us it all seems very easy. I mean it's you know it's just a list. You do those steps and then it works. But what I found is it's not easy. For users you really need something simple like the Debian installer. And and the nice thing about this is that the installer on those devices works the same as on a PC. So you don't have to do anything manually. You just run the installer and and the installer does everything for you. So the way it works is that because those devices don't have any you know VGA output or keyboard the way it works is that we run an SSH server on the machine and you connect to it with SSH and then the installer starts and and you just go for everything. So you just need SSH internet. You don't need serial console. You don't need to do anything manually. So it's really easy. So the way it works is most of those devices I mean they they run some firmware and they have some firmware upgrade mechanism. So usually you go to the web interface and then you you can give it some firmware image and and then it will upload that and and reboot and and and that's the upgrade. So what we do we basically supply a firmware image for for those devices and each of those devices obviously have their own way of of packaging the the firmware. So that's something that needs to be done for each device separately. But we basically generate a firmware image which contains Debian installer. So when people upgrade their firmware what happens is that the Debian kernel is written to flash and the RAM disk containing the installer is written to flash and then they reboot the installer starts the installer reads the existing network configuration from the device. Usually that's stored in flash or maybe on the disk. It brings up SSH and then people can log in. And then like I said before it's a normal installation at the end of things so that the installer knows about those devices it knows exactly what it needs to do. And at the end the new kernel the Debian kernel and a RAM disk which boots Debian from from disk is written to flash or depending on some machines actually boot from disk. But again the installer knows about that and will just generate a bootable image. So it is really very very easy. So some of the the philosophy which we have is we try if possible not to touch the boot loader or the configuration of the boot loader. Basically we want to make it easy for people to also go back to the original firmware if they want to do that. And we certainly don't change the boot loader because if we did that we might break the machine because if you flash something wrong then the machine is just not going to start anymore. So we have a policy against doing that. And the other thing like I said we don't require any manual steps. I mean usually you have a firmware image and you just upload that via the web interface. On some machines you need to do a couple of commands manually but that's the exception really. And that's just to get the installer working. The installer itself is really automatic. So some of the tools behind that. So like I said we need to bring up SSH and we need to use some configuration. And so what we could do we could just use some standard IP address. But the decision I made back then was to reuse the network configuration which they have used for the machine because that seemed the most sensible approach. So I've written a tool which is called OLSUS pre-seed which basically reads the network configuration and nothing else and pre-seeds the installer with that. And again that's something which is very specific to the machine. So some of them use a Unix file system in flash. So you pass that. Some of them have their own config format. So that's really just something you need to look at the machine see how reverse engineer how it works and then write support for OLSUS pre-seed. And the other tool is flash kernel. So it's called that way because traditionally when I started we have devices that always boot it from flash. Nowadays we also support devices that boot from the disk. So nowadays flash kernel can either flash the kernel or write the kernel to flash or generate a specific bootable image on disk. So we do that depending on what how the machine is configured. So we support about 15 devices in flash kernel these days. It's very easy to extend. So I think Walter who just did it can vouch for that. It's just a few lines of code depending on the machine. It's pretty simple. And as I said before we don't change the bootloader config. We try to not make any changes to the system except for of course installing Debbie and then you know replacing the system in flash. But that's replaced again. So what we do because we don't control control the kernel command line. And usually they those machines usually boot from flash directly. So they have a RAM disk in flash. So they usually pass something like root is death mem to the kernel which obviously doesn't work because in our case the root devices is a disk. And so the way we have solved that is it's not very elegant but it works is that flash kernel contains a hook for in it from a fast. So when you generate the RAM disk the root device is directly written into the RAM disk. So that's loaded and then the RAM disk will see oh yeah I'm put from disk and it will do that. So how do you support how do you add support to Debian to Debian installer. It's actually it's actually not that much work if you think about it. I mean the major work is really testing everything and making sure and reverse engineering that usually takes some time just figuring out well how does the machine boot. I mean on our machines you also need kernel support and usually that's not there. And adding that if the platform is supported is pretty easy. But there are some things that need to be considered. So for example on ARM every machine has a machine ID. So the boot loader passes a ID to the kernel which says which device it is. It's nice it's a unique ID. The only problem with that is most of those companies don't understand that it's a unique ID. So they just use some bogus value usually from the development board. So what we actually need to do is you need to register a machine ID and then you need to override the machine ID passed by the boot loader. So those are just a couple of things you need to know if you want to work with those devices but it's not you know black magic or anything. And Matt Palmer actually has written a really good how to about some of those issues about how to get serial console access. How to put the kernel and things like that. But in terms of so if you have done that and reversed engineer how the firmware works how the config file what it looks like then adding them in store and support is actually pretty easy. So there are two things that need to be done. The first thing is you need to add support for the sub architecture. So that's sort of the CPU. And I'm talking about ARM because that's why I have the experience but something similar probably applies to power PC and other devices. But on ARM there are different platforms and different devices can use the same kernel if they use the same platform. So if you want to add a new platform you obviously need a kernel image in the Debian kernel and then what the installer does it takes the kernel and then it generates U-damps out of that kernel. So what you need to do the second step is to actually generate those U-damps. Then the third step is that base installer needs to know which kernel to install for each platform. So you just need to edit one file and say if it's this platform install that kernel but that's really a one liner. The other thing is you need to know for the platform which disk label is being used and so in the past you had to edit this file and say this platform uses this disk label but actually nowadays on ARM at least you don't have to do that anymore because Rico said well all of those devices use the MS-DOS partitioning label anyway so let's just assume that on ARM. So you don't need to change that but on other devices you do. The other thing is for automatic partitioning you may want to specify a specific disk layout, partitioning layout for that platform so that's another thing that can be done but again that's trivial. Usually you just copy the recipes from a different platform and then maybe make some changes. So most of those things are really like one line changes are very simple things and then the other thing is package lists. So when the Dabby installer generates the boot images it needs to know which kind of U-damps to include and that again depends on which platform, which kind of image and on the specific sub-architecture so you just need to add one file but again that's something you can just copy from different sub-architecture and the final thing is install a build config specifies which sub-architectures are being built so you just need to add the new sub-architecture to that but that's basically it and then to actually add a device in addition to adding the platform you need to so on arm. If you look at proc CPU info there is a machine string which specifies which machine you're running on. So you just need to edit that file and then put in that string and say which platform it is. Again that's a one liner. Then like you probably need all this pre-seed support. That's optional because some devices don't have any config file and so in that case what you can do instead you can just put a pre-seed file into the RAM disk which for example tries to do THCP and if that fails it will fall back to some static IP address and then obviously you will need flash kernel support so the machine will actually boot and then you need to add it install a build config again to build a specific image for that device and again that depends on your device and maybe it's very simple but that could also involve for example packaging some tool to create a firmware image for that device. So it really depends on the device. So what are some of the problems users run into? So I've been doing this for quite a while and I've seen many different problems but the most common are listed here. So some of those problems have fortunately been fixed in newer releases of the installer. So for example a problem we used to have is that also is pre-seed that for example if the problem with the installer when it brings up the network is it needs to know everything about the network so it needs to know things like DNS which don't really matter if you just want to log in with SSH you could always fix DNS later on but the installer is built in a way that it needs to know all of those things otherwise it won't bring up the network and so some of the things that happened in the past is for example if people have configured their device but they didn't put in DNS then the installer would start also is pre-seed would look at the config would see oh we have that IP address let's do that and then it would see oh we don't have DNS and then suddenly it will pop up the screen saying please enter DNS but obviously because people don't have a serial console and because SSH is not running at that point the machine is stuck and it's the same with the gateway and there was actually a bug that there was no way to pre-seed that you don't have a gateway so if you don't have a gateway which you can have then again it would be stuck but fortunately all of those things have been resolved so nowadays if the network configuration is incomplete or also if it's invalid it would just do DHCP and have some fallback IP address so that there was just something that we used to see a bit another problem is that users assume a different network configuration so the documentation is quite clear how the installer reads the network so what it does so for example if DHCP fails it will use that IP address but apparently some people get that wrong and they look for the machine on a different IP address and they say well I can't find it and that's pretty tough for us because there is no way those machines don't have any LCD so we can't show you know please connect to that IP address and use that password or anything so it's just use a problem. I should say it was partly maybe it was my fault because one thing I did was most of those devices have some default IP address so when you plug it in it would just come up with that IP address and I was trying to be really smart and I said well maybe in that case I mean that means they haven't really configured the machine so maybe in that case they don't want to have that IP address so let's do DHCP instead and that probably confused some people as well but again I changed that. Another problem is Devin installer it doesn't boot if you add a second disk so finally that has been is going to be resolved because Colin Watson recently changed the installer to use UIDs but that was a fairly common problem and obviously what can also happen is that you flash a non-working kernel or any drum FS so that's something which doesn't happen very often but we had a problem once a while for example once we didn't notice that on some devices the any drum FS that was generated was too large to actually fit into flash because there were so many modules and that was only called a little bit too late but then we just didn't build a couple of modules so there was never part of a stable release or anything but in unstable those things can happen. The most common thing I get is I turn the machine on it always worked but now it doesn't boot and I have no idea why. I mean I have a serial console for me it usually boots I've never ran into any problems but there are so many users regularly I get an email it always worked and now it doesn't boot and I simply have no idea what's going on and I really don't have any good way to find out because those people don't have a serial console and my suspicion is that in many cases it's just that the file system check is running and that they weren't patient enough and maybe rebooted the machine and then it was stuck and maybe asked whether it should repair the file system but that's just something that's not very ideal and that's something we should try to improve some way or another and finally the big lesson I learned I mentioned that before but manual instructions don't work I mean even if they're really simple or you would think so people are going to get it wrong and I know that simply because the way the installer on those machines works is it's a net boot image which means the installer image itself is very small and it needs to download additional UDAPs or installer components from the network which means when we prepare a new release of the installer things get broken because the old image assumes to find things on the network which are not there anymore and so we had that problem at some point that the old image was broken but the new image was delayed for various reasons and I think it was several months that we didn't have the installer working on those devices and so what I did instead was to create a table of Debian with some instructions you know partition the disk, make a boot, partition do that, untie the table, all pretty simple stuff and it's amazing how much email I got from people saying you know it doesn't work, it doesn't boot or I did that wrong and it's really amazing so having the installer I think that that's so good and I think Debian is really quite popular on those devices simply because of that so I'd really like to thank the installer team for making something that's so easy to extend like I said before I mean it's a couple of lines of code here and there and suddenly things work so the future what are the things we would like to see so one of them is persistent disk naming and that's actually done so that's very nice. Another thing some people have asked for is well wouldn't it be nice to have some kind of rescue system in flash so if Debian doesn't boot we could somehow start that rescue system and you know do something and in theory I think that's a very good idea the question is just well how do you want to do it and I think there are a couple of options I mean one of them would be to use the Debian installer but then you would need to both keep the RAM disk to boot from disk as well as the installer around and most machines don't have enough flash for that. The other option would be somehow to extend the Debian RAM disk with some rescue options and Riko did some work on that and the final approach which I kind of like but which I'm personally not going to work on is actually well do we actually have to put Debian in flash wouldn't it be nice I mean there are systems which actually run from flash and that are made for that so why don't we put something like open embedded into flash and use that to boot Debian if there is a disk so we could simply use something like KXAC to boot Debian from disk and if there is no disk well you just have open embedded but at least you have something so I think that that's a nice idea and from the Debian side it would be pretty simple because we would just make sure if there is open embedded in flash well then don't write Debian to flash or ask the user if they want to overwrite it or if they want to keep it but I don't know anything about open embedded so someone would need to do the other side making the KXAC stuff and checking if there is a disk and those kind of things but I think that would be a pretty nice solution. The other thing is why don't you put a SSH server into the RAM disk that would be useful for a number of reasons so for example my NAS machine is encrypted so I need to enter a passphrase so I can do that because I have a serial console but for other people well that wouldn't work but if you put SSH into the RAM disk you could connect and then enter the passphrase and the other reason is well maybe we could somehow see the boot process on SSH. There is also something called network console in the kernel which can broadcast the kernel output on the network but the problem is that that only does kernel stuff so when you reach userland it stops so it probably doesn't show why Debian doesn't boot and that's the problem. Another feature request is support installations to Flash so Paes is working on that and finally I mean there are always new devices so the Marvell Kirkwood platform is something I'm currently working on. There are a couple interesting devices like the shiva plug and some NAS devices and then you have the free scale, the OMAP. That sort of goes into the direction of netbooks but I think well why should support NAS devices? I mean we have that experience, we have the installer, we can also support other devices and I think netbooks, those ARM based ones are really going to be interesting and finally so all the work I've been doing is an ARM that has historical reasons that someone gave me some hardware and it's also nice because of that machine ID so you always know exactly on which device you run and then you can really customize it but I think even on things like x86 you could look at I think DMI information and find out something because we are seeing those devices quite a few are now actually based on x86 but you can't just use the regular installer because those machines actually boot from flash instead of from disk and that's something we need to support so we need to figure out in the installer which machine is it and then extend flash kernel. But I think that's it so I think like I said I think it's really, there are quite a few users and I think if you look at other distros I think Debian has much better support than anyone else so and it's those kind of things I mean if we go back to the question about the universal operating system I mean I think if everyone you know all of us just focus on one area and make Debian better than anyone else in that area then I think we will get there. So are there any questions? Why is anyone just hungry for food? I know I am. You said before that the SIVA plug was going to be shipped without the serial and the G-TAC port? Yes so that's something I'm not sure about so the SIVA plug is actually a couple of different things so because this design anyone can produce those machines so my family is really very open about that they just won't you know anyone to produce them so what's called the SIVA plug is the development kit and that has the mini USB with serial and G-TAC and the SD card but for example there is something called the Pogo plug which is essentially the same but it doesn't have the serial so there are people who have already opened it up and connected one but it doesn't come by default and so I don't know so I read that press release that Buffalo and a couple of other big companies are going to produce devices based on that design but I don't know if they're going to include the serial or not so my assumption is they're probably not going to include it and so again when you buy such a device you just have to make sure which one do you buy because they are very similar but with slight variations so the Maval SIVA plug development kit which is linked from my website for example that has the serial console but at the moment it's being shipped from the States so for people from Europe they have to usually pay import tax but hopefully that's also going to be resolved so I don't know if I told Maval about that I mean they now have the model with the European connector and the UK connector but it's still shipped from the States so but hopefully the next step is actually to ship it within Europe. Without the serial it would be very hard to hack on the device. Yeah that's right but I mean you could still open it up and connect your own serial but I mean that this device is so much fun particularly because it has all that stuff built in so yeah just make sure you get this one and not any of the variants and here are just a couple of links so it's also a pretty good documentation on the web. Do you know if the SIVA plug design or any other modern design also supports some faster way to connect the disk like SATA or external SATA or whatever? Well though the next device is for example the QNAP has one internal or two internal disks and it has external SATA as well but this one only has USB. It's funny because when it was announced everyone looked at it and said oh it's perfect it just lacks this one feature and the cool thing was everyone had a different request so there were some people who said well I'd like to have power over ethernet. The other guy said I'd like to have network over power. Some people would like to have two USB and you know if you produce all of that it's probably that device which actually just reminds me. So this one is really cool but another device which they have which I'm currently waiting for is also pretty cool. It's called the OpenRD Client so it's based on the same platform and everything but it has all those connectors you just asked for like external SATA and it also has VGA so you can actually use that as a thin client and I think that's going to be really cool but I don't know if I think they're actually being shipped but I'm not sure about that and I think it's about $200 or $250 so it's not expensive. So that would probably be what you're looking for. OpenRD. Okay so if there are no other questions thanks very much.