 or folder level with butter. It's cool from like a home hobbyist perspective. From a more professional perspective it can partially be cool and partially be just horrifyingly sloppy where you kind of can't keep track of everything that's going on. The other big thing that butter has that everybody has wanted for like forever is it's got on the fly a reconfiguration of rate. So I mentioned with ZFS you've got a pool and you can add VDev to the pool. What you can't do is you can't change a VDev really once you've created it. Once you've created for example a 6-disk grade Z1 which is basically like a grade 5, right? So you've got the capacity of 5 disks in the 6-disk VDev and you've got one disk worth of parity distributed through it, right? Well you can't add another disk to that. That's a 6-disk grade Z1 VDev from now until forever. You could add another 6-disk grade Z1 VDev into the same pool but now that one's immutable too. You can replace the disks in the VDev with bigger disks and once you've done that with all of them you'll see the additional capacity but you can't just be like oh well now I want that to be a 4-disk VDev not a 6-disk or now I want that to be an 8-disk instead of a 6. It's just not that mutable. Butter? Butter in care. With butter you can have a 5-disk grade Z1 add or I'm sorry a 5-disk, a butter grade one, add another disk and yay! Now it's a 6-disk. You don't even have to rebalance anything. You can rebalance it if you want to. You can even convert that butter grade one array that you already have into a grade five array on the fly. Just tell it, hey butter, that grade one make a grade five. It'll do it. You can keep using your computer while it's going on. Of course it may catch on fire the next day but you know that's butter. So you know portability, what systems can you use this on? You know again I hadn't really intended to keep making such a big difference out of people's ages in here but it's kind of relevant. So you know those of us in here who are in our 40s or more have probably been through a lot of systems, a lot of different file systems, a lot of different computers and hopefully some of us have data now that they couldn't read that exists on computers existed on computers that aren't even around anymore and it had to make a lot of transitions. So the more of those kind of changes you've been through, the more important portability comes to you, the more crucial it gets that you don't feel like, okay well if this particular operating system is no more, my data is going to be no more along with it. You don't want that. Luckily both VFS and butter are open source file systems which insulates you from that tremendous degree. But VFS is not licensed under the GPL like the Linux kernel is. Unfortunately it's licensed under the CDDL affectionately or unaffectionately referred to as the Cuddle. The Cuddle was created specifically by Sun Microsystems back in the day to license ZFS and Dtrace and some of their other intellectual properties under. Basically because Scott McNeely hated Linux. So they designed, it was based on the Apache licenses I recall and it's a very open license. It's not restrictive but it has just the right clauses in there. They felt like it would be impossible to combine Cuddle code with GPL code so he wouldn't have to feel like his hated high school rival Linux Torvalds would be able to use his stuff. Fast forward close to a couple of decades and there's a lot of controversy about that now. ZFS is very much available on Linux. It was available first as a Fuse file system which basically meant that it ran in user space not in kernel space which allowed you to separate the two so it wouldn't be a problem. It's also available as a loadable kernel module. The ZFS on Linux project is very mature. Those of you who are kind of interested in ZFS but haven't really gotten your feet wet too much may have heard a whole bunch of rumors that ZFS on Linux sucks that it's not reliable, that it's flaky, whatever. I can tell you right now that is not the case. I've been using it in very wide production for four or five years and it is awesome. With that said, there was always a question like, okay, there's no doubt that you're within your legal rights to use Cuddle code and GPL code kind of mush together. The question was, can you distribute Cuddle code and GPL code mush together? And everybody kind of assumed that you couldn't because that was what Scott McNeely was going for back in the day and everybody just kind of assumed that was the case. But ZFS and also Dtrace, they are awesome properties. And people have sort of been like, man, we really want to use this in the operating system that we actually care about. And they started looking at that a lot harder. And you have some folks like the Software Freedom Conservancy and the Free Software Foundation who still adhere to the position that no, it's not GPL. You can't use it with GPL code. Bad. You have a lot of other folks, actual intellectual property lawyers. I'm not just talking random people talking off the bar stool who have taken long hard looks at the code, at the licenses and said, you know what, I think this would stand up in court. I think we would totally be okay distributing this. Nobody has actually challenged it yet. The folks on the side of light, I have a very affectionate to Free Software Foundation and the Software Freedom Conservancy and all those folks. I'm a big believer in GPL enforcement. There are a few really good reasons that they haven't tried to challenge any of these ZFS projects. And one is that ultimately they are in favor of free software. We're all in favor of free and Libre software. The question comes down to whether you believe that endorsing these products is going to weaken the GPL or not. And again, opinions vary. What I can also tell you is that these folks are working very hard to lean on Oracle. If you don't pay attention to politics of the industry, a few years ago Scott McNeely had another hated high school rival and I'm using that loosely. I'm not saying they actually went to high school. Oracle. And Oracle won that particular grudge match and bought Sun and kicked Scott McNeely out to, well, you know, send them to that farm upstate to play with the other puppies. And so now Oracle owns ZFS and owns Dtrace and Oracle has been distributing Dtrace and their unbreakable Linux distro. And some of these folks on the GPL enforcement side have been leaning on them really heavily about that and being like, okay, look, you own this property and you really, really want to use it in Linux. So for the love of all it's holy, do it the right way. You need to do what you need to do and re-license this GPL so all these thorny issues go away. I have been told that they've been making progress on that front. They hope to make that a reality. But for right now, that's where we're at. We've got some folks fighting the good fight who think that it's wrong to distribute these things together. We've got some intellectual property attorneys who say, man, we have looked at that and we are really, really certain this would be totally fine if it got challenged in court, but it's kind of up in the air. Butter on the other hand, GPL. Part of the kernel. Don't have to worry about it. Just have to worry about it eating your data. One cool thing about butter is it scales down really well. There's a project, JOLA, Smartphone Project. The JOLA devs are like, I mean, no kidding, they are running butter on their phones to give you an idea of how well butter scales down. ZFS does not scale down that far. It's just kind of not really what the ZFS devs are truly interested in doing at this point. Most of the issues that prevent it scaling down could be fixed, but to give you some idea of where they are at, you can no longer really reliably run it on a 32-bit system. You really need a 64-bit operating system to play with butter. Now, with that said, there are some places that I won't mention, pretty bad forums, where you'll see people just tell you that you have to have this crazy unobtainium machine like, oh, you've got to have 16 gigs of RAM bare minimum and it needs to be 1 gig per terabyte of storage, so maybe more than that. And you've got to have an Intel processor. AMD's are terrible and you can't have a real technique. You've got to, things get kind of crazy in those forums. You really should not take that at face value. What you truly really need is a couple of gigs of RAM and a 64-bit processor. If it's AMD, that's fine. I can tell you I've run this stuff in production with real-world usage data depending on it, on like the cheapest possible AMD processors. Both the A3400, a little APU, cost about $40. There's also a little Tinkertoy processor called the Cabini, cost like $35 for the processor on a board. Most of them don't even have fans on them, if they do. There's a little teeny, teeny tiny ones that barely spin. ZFS works fine on those. So you don't actually need tons and tons of hardware. With that said, unlike butter, if you have more hardware, ZFS will totally make use of it. If you've got a ton of RAM, you've got a ton of RAM for that wonderful, wonderful art cache and you will feel the difference. So it's great to have it. Future proofing. So you don't ever want to bet against the GPL. I've said this in the original version of this talk three years ago and at the time I kind of interpreted that to mean I thought butter was just going to take off and leave ZFS in the dust because it didn't have the licensing problems. Unfortunately, the butter developers didn't really do what would have been necessary for that to happen. It's still not reliable, so it can't really take off. In the meantime, things have kind of gone the other way. Instead of butter getting closer to reliable, ZFS has gotten closer to being GPL. It's still something that we're looking for very hard to happen. In the meantime, just to give you some idea that, you know, if you've never heard anybody else say, oh, well, people are getting more friendly to the idea of mixing Cuddle and GPL code, you know, we had Ubicon, you know, here parallel with scale. We had Mark Shuttleworth addressing the audience out here. I don't know how much they talked about this, but the next LTS release of Ubuntu that drops in April, Xenial Zeras, ZFS is going to be in the main repos. You don't even need to hit PPA anymore. You can just install it direct out of the repos. I imagine it's probably a universe instead of main, but either way, I mean, it's coming direct out of Canonical. So that's pretty cool. Fodder may eventually be a default Linux file system. Three years ago, I would have sworn up and down that by the time I was talking to you today, it would have been the default file system everywhere, because I mean, once you get used to these next-gen features, it is like actual physical pain to have to manage a system that doesn't have all these things. So I thought, you know, surely it's already in the kernel, like we'll iron out the bugs and, you know, everybody, you know, Fedora, Red Hat, OpenSUSA, Ubuntu, you know, all the little tiny toy distros you've never heard of. We're all going to be running butter, because why wouldn't we? But unfortunately not. Still could happen eventually. All right. So we've got a few minutes left, and who wants to see a live demo? Anybody? Awesome. Everybody loves live demos because it gives me a chance to look like a complete, pardon my friends, jackass, and fall on my face in front of the big crowd, and I like living dangerously. So first thing that... Instead of trying to make this thing go mirror. All right. I'm going to bring up a VM here, and this is going to be fun, because I'm kind of going to have to type blind. Let's drag this thing over here. And now you know what, I lied. We're going to mirror this, because this is just painful. I think we've got time for that. Displays. Mirror. You know, let me mirror it. Now we'll just have to type blind. That will be fun. Urban2TrustEVM here. I'm sure this VM is actually on ZFS. I'm running ZFS on my laptop here. All right. Cool. So... I actually need to give that the focus first, fair enough. We've got our super... And now let's do something incredibly... I can't even see. Can anybody tell when I'm like actually over the terminal icon? Can I click now? Dash, dash. No, it was that in the dev file system, because I can't remove stuff from dev or proc. Now our next challenge... Now let's drag this over here where I can't see what I'm typing. Now this is on my actual laptop, not in the VM. A bunch of asterisks. Okay, now we're root there. And we do a ZFS list, RTC, Images, Urban2TrustEVM... Oh, I'm going to drag this over here so I can see what I'm doing. And then I'll drag it back so you can see what I did. If I could only mirror. All right. Well, Urban2TrustEVM working, that's the problem. The snapshot's down here. Not being able to see is miserable. All right, so I'm going to do a ZFS rollback to my last snapshot, which was an hourly, which was automatically taken at five o'clock. I'll be paced. I'll drag this back over there so you can see it. Or most of it, because man, the resolution is low on this projector. All right, so hit enter and you probably can't really see anything, but there we go. So the ZFS rollback, ZImages, Urban2TrustEVM working, blah, blah, blah, and it rolled it back and that's pretty much it. So now we go back to our VM, which I really should have forced off for that actually. I can usually kind of crane my neck and see what's going on better than this when you can see what you're typing. It really is. Hopefully that was kind of painful for me, but keep in mind that like literally I just restored a system that I RMRFNoPreserve root on while I couldn't see what I was doing to the side of me in a couple of minutes. So that's a lot better than having to go to tape or whatever else you might do for backup. So the other part that is kind of hard to demonstrate here is I could replicate the whole thing to another box as well. Esther in the back. Not on Linux, no. It's technically possible, but it's hideously difficult and would be a pain to manage. So I don't think that that's a win for me. On my own like workstations, I'll make like my home directory will be ZFS and obviously my VM storage is ZFS. I don't really try to mess the root file system because it's replaceable anyway. I mean it's a 15 minute OS reinstall away. So stuff that I really care about is like my configs, my actual data that might go in my home directory or an opt or like I said, storage. The question was would I make home on ZFS? And yes, I absolutely do because there's stuff that I really care about on home that I can't just get back with a quick app to get after a 15 minute reinstall. But better than that honestly is just don't wear possible. I mean so what you're asking is kind of an edge case for me. That's like my personal workstation. Really where the really important stuff happens is on servers and the whole server is a VM on top of ZFS. So the whole thing is an image just like what we just saw with that one there. So like everything's covered. Can you say that again please? Okay, so the question was it sounds like my deployment started on BSD and then moved to Linux and the answer is yes that's true. There were a couple of years where every time I set up a server I had the extremely uncomfortable question to ask myself did I want package management that was sane or did I want a file system that I could truly trust and I had to pick one or the other I couldn't have both. Once the ZFS on Linux dropped and got stable I no longer had to make that choice. I could now have apps package management. I could have the Linux kernel. I could have all those things that I really, really seriously wanted and have it on ZFS and that turned into a no-brainer. I didn't just immediately reformat all my BSD boxes overnight but I did stop deploying any new ones because prior to that my BSD boxes they had for a while. I used to do everything on BSD and like you know I would install Samba on BSD for file servers and offices and you know I would run Apache web servers and blah blah blah. But the issue that I came across was that while using the ports tree was awesome in the late 90s and early 2000s as the security situation got worse and as you know public internet access to like you know things that clients who want to publicly expose got you know more and more frequent I was less and less comfortable with BSD package management because it wasn't fast enough so I wasn't doing it often enough. I wanted to get to the point where I could literally just say you know after getting installed unattended dash upgrades and have security updates happen automatically while I slept and not have to worry about it and have it just work. And Debian gave me that and Ubuntu gives me that and free BSD didn't. So once I could have both those things on the same box it was no brainer for me and now everything goes on Ubuntu. So the question was is ZFS not kind of just a random collection of disks? And it's not really designed for that. I mean there's nothing stopping you from doing it but like you have mismatched sizes. It's fine to have mismatched sizes in a pool but if you have them inside a VDev the VDev is going to be sized to the smallest of your sizes inside that VDev so you're going to waste a lot of capacity. The other issue is if you mix a lot of different type of disks and now this is not unique to ZFS. This is true for any way that you're going to mix and match a bunch of disks and you know one big storage pool whether it's ZFS or butter or kernel rate or hardware rate or whatever the performance of that array of storage is going to track the performance of the absolute worst member of that group of disks. And the worst part is that may not even be the same disk every time. If you've got six disks even if they're identical one of them may have a hiccup on a given read. Now if they have a hiccup on let's say one out of six reads with an ounce of having a hiccup on one out of six reads you're basically having a hiccup that affects your entire read operation on every single read. So in particular if you have six disks that are really good and one that really kind of sucks now all of a sudden your entire pool is limited by that one disk that really sucks. So you kind of want to avoid doing that if you possibly can. So the question is what's the best practice for installing ZFS on a hardware rate controller and the best practice for that is don't do that. So here's the deal. ZFS doesn't want to be underneath hardware rate. ZFS wants to have direct access to the individual disks because it does the array management itself and it's frankly far, far better at it than you know some little chip on a rate controller somewhere. It has access to more RAM, faster processors, far better written code. And like for example if you were to set up a hardware rate 5 array and install ZFS on top of that remember we talked about all that awesome you know self healing stuff where you know it just rebuilds out of parity. Well ZFS can't do that because it can't see the parity as far as it's concerned it just has one big disk. So that sucks don't do that. If you have a rate controller and you're determined to reuse it some rate controllers can be reflashed with like an IT mode firmware that will allow them to function as just a regular host bus adapter that gives you direct hardware access to the individual drives and if you can do that, do that give the individual drives to ZFS and your gold. Otherwise pull that thing out, check up the trash. Don't check up the trash, sell it on eBay. Somebody will want it and get a nice inexpensive host bus adapter and put that in your box or just use the ports on the motherboard and go from there. You there. Isn't really good at warning you about much of anything. Basically with butter, I mean you better be monitoring your logs. There are, and I'm a little rusty on butter now because I'm not as enthusiastic about it as I used to be but it's possible to see that it's degraded, you know, using tools like Butterfly Show or whatever but it's a little obtuse, it doesn't always catch everything and usually your best indication that you've got a degraded butter array is that it won't mount because by default and the butter devs think this is a great idea and can't be convinced otherwise and I know because I've tried really hard to convince them otherwise. Like if you lose one disk out of a butter array, now all your data is intact but the array is degraded, right? So if you reboot your system, yeah screw you, it won't reboot because it's degraded and in an order amount at all you have to pass the dash O degraded argument which if you don't have that in your grub, yeah it's not going to mount, it's not going to boot, you're just going to be staring at a grub prompt with no way to get the thing back up other than like now you've got to boot from alternate media that knows about butter that can go through, that can mount it, that can get to your grub, I found that one out the hard way and the butter devs like I said, they think that's a great idea and they don't think that should be changed. Now the question was do butter and ZFS share a condition where you can delete a file and you still haven't reclaimed the space so you just get stupid folding and have problems? There's a couple answers to that. One is that butter has a really really bad problem with not being able to reclaim blocks once the file system gets really full. That is a bug, it's butter specific, it's not a byproduct of some intended feature, it's just like oops, that's not as bad as it used to be but it is still an issue. The other thing is if you've taken snapshots just deleting a file that's referenced by that snapshot, no that does not reclaim the space. If you want to reclaim the space that was occupied by that file, every snapshot that references that block has to be destroyed before the block is actually reclaimed. So yes, if you take a snapshot while your disk, if you fill your disk completely full and then take a snapshot and then rmrs everything, you won't have recovered a single...