 Welcome to the homelab show episode 31 file system layouts. I'm here and with Jay LaCroix Yes, we probably this is one of those fundamental building block videos that maybe should have been done sooner I don't know what you're doing it now Doing it now. That's all that matters better legion now Yeah, because this does come up from time to time especially when you're loading Linux just shows now We're gonna give some honorable mentions to some of the windows file systems But if anyone watched the most recent Linus tech tips video He did remove his window storage spaces and window storage spaces didn't scale to the large scale nature of some things And once you get into the enterprise market or having a really extensible large scale file system You'll find that windows isn't dominant there. This is not what the cloud platforms run on and If you're using windows is not because you're choosing between which windows platform file system you on it's well Because you have to use windows you'll end up using the file system that windows has So it's not like should I format my system with windows or not? So we're not going to spend too much time on the windows part of it Mostly the question comes up about different file systems is zfs butter fs xfs ex t4 Should we be using lvm? Should we use medm raid or manage it with zfs? Which actually can talk to the hard drives or butter fs that can talk to the hard drives and boy There's a uh, we got a lot to talk about. How's that? Yeah, we do we really do We feel like we could do like a series of like 30 episodes to go into the intricacies of each of the file systems So we're gonna summarize and bring up like the most important points of uh, each of our talking points and uh, that'll help Build the foundation enough so you can make a decision. But of course there is uh, plenty of forums and Debates and arguments over file systems over the years and even me and jay when we were trying to get a couple updated pieces of information You always got to check the year at each thing was written because some of them I mean with ext 2 versus 3 versus 4. Yes, there's definite version changes. But some of the other ones It's hard to tell what version they're talking about. They're just talking about zfs. You're just talking about xfs Uh, so there are features that have changed over time So, uh, there'll be a lot to talk about here But before we get started we got to think a sponsor of the show and that is linode And probably linode i'm gonna say doesn't run ntfs in the background. What do you think jay? I doubt it. I really doubt it Yeah, being and being uh, that they're a large linux based hosting Great place to hold your vms. We've talked about a lot of projects on here If you're listening to this podcast, it was downloaded from a linode server Which is even faster since I was at episode 30 we Before episode 30 j updated the memory make it a little bit bigger and give us a little more bandwidth because all of you Keep downloading this podcast and hey, we don't mind well knows pride in the band with us for it. Use it Click it download it We even make it just raw audio if you don't want to use an app you just want to listen to us raw all that's available And any of the projects or many of the projects we've talked about in here If you're looking for a place to run your servers linode is the place for that It's where this website's hosted It's where this podcast is hosted and it's where j has well currently all of his public facing stuff is all in linode as well So everything yeah, we want to thank linode to be a sponsor There is an offer code down below where you can get started yourself with linode and you know Get playing get some of those things out there use some of those predefined templates So you don't have to build it from scratch so you can test it out or then Learn how to build it from scratch and things like that use the use the pre-built ones as a reference because I've done that a lot man I don't know why something doesn't work when I build it But it works when I click the one click button and good ways to troubleshoot it So thanks again for the load for sponsoring the show and let's dive into file systems. Let's get into it What's the first one we want to talk about j? So, um, I don't really mind the order because I know we need to talk about lvm which isn't itself a file system But it's definitely a talking point. Um, but we should probably save that for a little bit later It may be Possibly go over ext4 first because it's developed on many distributions out there Yeah, ext4 is the fourth generation of the ext file system Now the one thing I like about ext and I think it's 100 backwards all the way Is you can import previous versions of ext into the upgraded version? I believe all of them can do in-place upgrades Is that correct j? I think so. I haven't done that in so many years I know that I'm was it ext2 to ext3 you you converted by just adding journaling. I believe yeah I believe yeah, so that was kind of like an interesting upgrade It's like adding the the feature there But um, it's been so long since I've had to upgrade anything from ext3 to ext4. I want to say what was it like 2008 Yeah, or something like that But it's unfortunate because there are some really old systems out there And so ext through all these generations has become a much more robust file system the way they distribute the journals on there Journals think of them It's all that data around where your data is and they spread them out across the drive This this isn't I mean we we could dive through and you know what we need is maybe we'll get Wendell to dive through the file system history with us We will do an episode on that in the history of fat file systems and how things were written the drives and Things were adapted in early days of Linux and unix with the basic file systems And someone says that's not the best way to do it hard drives are getting better And they added these journaling support to be able to have a journal of the transactions that occurred This makes recovery easier corrupted file systems was a big thing And you'll talk to older technicians that go oh man, there's so much risk of corruption It's actually such a minor issue in 2021 because of the robustness added for Things like the extra journals and the way they spread them out across ext4 ext4 is a Pretty solid file system. Uh, it is a default I think for debby and and all debby and basic distributions in my right jay Don't follow that pattern, right? I would say the majority. I mean you have um butter fs for I believe it's open Susa and possibly fedora. Um, it's like I review so many distributions that it's kind of starts to blend in After a while, but most of the distributions out there default to ext4 I like to think of ext4 is the general purpose linux file system because if you don't have a preference just go with that Yeah, and it works fine It has all the features you need of a modern file system So there's not any reason to not use it. It's solid. It's reliable It does not though and some of the other ones we get into it does not support any type of snapshots This is a contention point now. We're getting lvm. There's a way you can snapshot ext4 with lvm, correct jay We'll get a little bit later. Oh, yeah Yeah, but it's not a function of the ext4 file system itself And let's talk about the other one if you are over on the other side and you're wearing a red hat What type of file system would be the default over there? So I know I haven't used red hat in a while. So I'm thinking was it xfs. So it's a bit of a trivia. So I guess Yes, xfs is really popular. You'll find most of the documentation over there on the website for red hat now xfs addressed issues When it was when it came out against ext and when I see when it came out that was like 30 plus years ago So they they've been both developed. They're both longtime stable file systems and with xfs. There's also No snapshots natively built in there and me, you know, it's basically just red hat prefers using it There are some performance arguments of which one supports more directories versus which one supports more files in it There's lots of little nuance, but we're talking Like massive things that any home user was one never going to run into it It's only something you would think about on the cloud level scale of a massive System but both these have advantages now xfs seems to suffer a little bit more from small right issues But these are often contested in different tuning parameters I was reading through benchmarks to try to get a definitive it is not like there's a night and day difference There is a nuance difference depending on the workload What it seems that the xfs seems to handle some of the larger files So red hat uses it because they recommend it for like databases and things like that So there's there's a little nuance in there. So it's still a good choice It still works perfectly fine. And for most all your production environment, whether you're going exc4 xfs I think is a pretty It's a flip the coin. What are you most familiar with? What are you most comfortable using and are you using red hat? And it defaulted that anyways Right, that's a good point And I think it all comes down to what your use case is because if there was a clear winner Like a podcast would be like five minutes long We would say just use this one and we're done But you know, we can't say that because there's so many pros and cons And sometimes it comes down to the workload if it's like a database server Maybe one will work better for the other Maybe, you know, large files if you have like ebbobites I don't know if I'm saying that right of data like I mean one will be better for that But I mean if you have the much data like congratulations, that's amazing Yeah, and I don't really imagine too many people when you look at the maximum supported single volume size of exc4 or A xfs system I don't really imagine a lot of people are creating single volume sizes like that because when we get into zfs Anytime we see these incredibly large systems zfs is kind of the king of all that So we're we're spending a little more time talking about zfs But yeah, once you get into, you know, right now We have 1.2 petabyte servers on on the shelf on the other room over in air getting ready for a client project delivery next month and uh That's always going to be zfs for that type of storage. So is there existing storage? That's that big Yep So I think at this point It's probably a good time to get into lvm because I think some of the things we're going to talk about are going to kind of force a comparison there And lvm is not a file system. I want to be clear on that. It's like an abstraction You know underneath the file system. That's after the hard drive be you know beneath the file system That gives you additional features And it's one of those things that I say everybody should be using. I mean, if you're not using zfs If you're not using butterfs, I don't know if it works with butterfs. I haven't tried it but Not with zfs though if you're not using that then I think that lvm is a no-brainer because it's one of those things where you're not going to notice that it's there And maybe you'll use it and you'll never have a use for its features But it's not going to hurt anything And if you ever do need its features then it's there for you. What are the features? So One of the things that Is kind of a pain point for me is that when i'm working with a client, you know companies ask me to help them out their servers Right, so they're servers full their disk is completely full. Let's just assume. It's a virtual machine So if without lvm You have to rsync the volume to another volume if it's not the root file system And you know you add another virtual disk and you rsync everything over there and then flip them out point over to it There's going to be maybe a disconnect there. You're going to have open files, for example It's not going to be very easy to do this But you can do it in worst-case scenario. It's the root file system. You might have to shut it down And that's not good. Now, obviously if you're doing everything, right You should never need to resize anything because you've chosen the right size for everything Your virtual disks are the perfect size. You don't ever need to upgrade anything So if you're doing it, right, you'll never need it But more often than not you're going to run into this and I think as an administrator You're going to appreciate having the ability to resize a file system online So for example, you have a let's just say a 100 gig root file system and it's it's nearly full Now what you could do is add a second virtual disk add it to the volume group And then expand it you can actually grow it online And you don't have to restart anything and all of a sudden, you know, your your full disk alerts go away um, it's a great thing on a physical server, it's not quite as easy because Depending on the hardware, you may not be able to hot plug a hard drive into a running server You might actually have to shut it down But if you have a spare hard drive, you could just have it connected inside the chassis powered on but just not used So if nothing else, you can always add it to the volume group later. I've even seen some people that would essentially not have not use up all of the space in the lvm configuration have like, you know 10 20 gigabytes that are not even claimed for that one off. Oh my god. I need space because everything is falling over But um to kind of go back a little bit. We've got to talk about what lvm actually is because I just talked about the benefits So if you think of lvm, it's more or less an abstraction on top of the hard drive It isn't itself a file system. You can use lvm with ext4 xfs whatever you want So you have different terms with um lvm which stands for logical volume manager That are kind of like these, you know, I think they're hard for some beginners to remember Especially considering One of the terms is physical volume and when you think of physical volume, you're thinking about a hard drive You can hold in your hand. That's a physical volume. You have a physical disc in your hand, right? But with a physical volume in uh lvm is not that it's just a disc It could be a virtual disc. It could be a physical disc. It doesn't really matter So when you create a physical volume, you are basically saying this hard drive is a physical volume That's going to be used entirely for lvm Then what you do is you create a volume group. You could have multiple volume groups that group together logical volumes logical volumes Is closer to a partition So you have a physical volume You have a volume group and then you have a logical volume which you could then format So you could basically have your root file system as a logical volume Home partition logical volume And you could keep going and have as many as you need there And the cool thing about that is you can resize them and grow them independently of each other So if you have extra space not used up, that's not claimed biological volume You could actually just go ahead and expand it and that ability May not be as useful to a home lab person because Let's be honest if if your server's full are you going to have a bunch of people coming into your office and complaining about it? I mean the worst case scenario your kids say hey the Plex server is not working. Can you fix it? But it's still kind of cool to have and there's no reason not to implement it because again if you don't ever run into that situation It's just silently running there in the background And it's a great thing And you could also create logical volumes on the fly. You don't have to shut down the server You can actually create a new one if you have space mount it you're done But another feature I like is the ability for snapshots now. This is something that I think it's safe to say let me know if you agree that the majority of file systems out there don't have snapshot features And I feel like snapshot features in a file system is more or less the You know, it's just a value add that some of them have but it's not something you generally find In file systems. DFS has it butterf s has it But I would argue that butterf s and cfs are more than just a file system It's almost like a um file system solution application hybrid. Um, if you know Yeah, anyways, um, but a pure file system is just you know Blocks and whatnot how you store data and that's that's really it now with lvm you can create snapshots and One of the things that really annoys me is that when you use let's just say a boon to for example and you're installing it And you you say I want lvm. It's going to use 100 of your disk Period 100 you used so when you want to create a snapshot you can't because there's no room snapshots with lvm require Like you know some extra space that's not used by a volume group or you know, basically free space in the volume group for the Um, you're not formatting the snapshot your snapshot is just sitting in the on you space When you install the boon to you don't have on you space because it uses 100 of everything Which is annoying because if you want to use that then you have to use a partition tool To basically create a volume group and set it up yourself Before a boon to is installed and then you just point a boon to to the partitions that you've already created that works I think that's one edge that the debbie and installer has over a boon to because With debbie and you could tell it don't use the entire disk for lvm Give me some you know unused space at the end there so I I could use that if I need it later Debbie and let you do that and I think that's really cool because then you could use snapshots snapshots are I guess in lvm something you want to try to use only when you need to You don't want to have a snapshot sitting there because if it uses up it's all the space Bad things will happen. Yeah snapshots are temporary. You want to you you want to test out a piece of software Create a snapshot test out the software. I don't like that software. It doesn't work for me Revert the snapshot back and then it's like nothing ever happened. Leave that snapshot hanging out there You know, it's going to use up a bunch of space and it's just not going to be good. So Basically you want to keep your eye on that but one use case I like snapshots for as arch Linux It's like the ultimate way to run arch because arch is rolling all the time. You have updates every single day So what happens if you have an update that breaks something it does happen happens to me sometimes If you take an lvm snapshot before you run your updates No problem if something's break, you know something's break you just revert the snapshot back It's like nothing ever happened wait a couple of days try the update again. See if it works if it does um Make the snapshot permanent. Just um get rid of it. It's good. Everything's fine. And then you have a A greater level of stability on top of arch Linux than you would get normally Which is kind of like the secret sauce. I think with rolling distros just create a snapshot before you update anything It's like the best the best thing ever You can quickly revert back. I think a couple things that in I seen this in the comments It's something I want to address is let's say I have a drive and okay I ran out of space and I want to expand not add another volume group expand the main volume with lvm So I pop another drive and I expand it Where's the data it's spread across two drives if I lose one of those I've lost all am I correct? Yeah, I mean that that's kind of um It kind of also depends on how you set it up because One question I get asked sometimes is like Raid with mdadm versus lvm you can't compare the two because one's right solution one's a file system abstraction, but I personally I think I might have once set up mdadm raid Software raid in linux if you will manually never again um What I always do is I set up, you know lvm With debbie and or a boon to and then of course we have part We have mdadm underneath and then you have lvm on top So you have your raid and then lvm is how you split the partitions out So in that situation when you have raid, I mean already you have a situation where if it's like raid zero You lose a disc you lose data But if it's raid plus lvm Depending on the type of raid that you set it up with you know raid five raid six For example, I always prefer six, but that's just me you can lose a couple of drives and you're fine But if you have straight lvm With um, you know no raid at all you can't lose a disc because it's you know part of the volume group that bad things will happen and I've seen uh junior linux admins make this mistake where They try to snapshot a virtual machine And you know just habitually they've snapshot the main disc and nothing works. Oh There's another disc. They should have shot at all of the virtual discs Because maybe they have that one-off client that's using multiple discs or whatnot So you have to take that into consideration when you have lvm all the discs are part of a hole If you take one out There's a problem and I think the only way you can get away with you know losing a disc is if you combine it with raid Yeah, um in something else worth mentioning is the way when you set up, let's say a deviant system You want to have a password on boot and this works with a bunch of as well I believe it sets it up with a lvm and locks encryption So you got a small boot partition where it can load the kernel and then it prompts you for the password You can can figure this to have different volume groups locked that way if someone ever takes the drive It they have to have a password in order to boot this system up They have to have that luxe password you set So there's no way for people to get the data off the drive it it puts the splits it up And you can even maybe you want the system to boot at least to a level where you can remote into it So you let it boot but then it has another volume group where your important things are And then you have to unlock that volume group with a password So there's a couple different strategies by which you can deal with that Yeah, and you can have a luxe encrypted drive without lvm But when you are installing like a boon to and you choose that option for encryption full disk encryption They only it only gives you the option with lvm. I mean even though you could separate the two I mean why separate the two honestly like just use lvm um One other thing i'll mention about lvm like I think I've alluded to and i've mentioned already use it on everything Absolutely use lvm on everything because when you are you know for example working in a company if you do that And you have a server that's mission critical and it has lvm You're going to feel like a hero like a rock star when you are able to solve that full disk problem without the server going down Without losing any pings all of a sudden it goes from you know zero percent free space to you know 30 percent depending on how much you give it it just it's just so awesome And it's so much more frustrating to have to plan a maintenance window to shut it down because somebody didn't use lvm at the beginning Of course i'm talking about the enterprise here But yeah, it may not be that big of a benefit for the homelab user But I still think it's cool and if you don't have to reboot it why right just use lvm And if you need it, it's there. I make it the default on everything I do Well, there's also an advantage when you're doing this It's it makes it a little bit simpler when you're running virtual machine So if you build your virtual machine and you set it up with lvm And you're like oh, I need to add more space to the virtual machine Shut the virtual machine down expand the hard drive easy or if it's ice because you can do it that way less easy Well, just more steps is now. How do you expand it if you didn't have lvm? There's a few more steps not you can't there's just more steps in it, but the steps are Shut it down expand the size boot it up run the proper lvm commands to expand it You had the absolute most minimal amount of downtime to actually make that happen And I will add in and we'll leave it a link in the show notes for this You have a whole series on lvm. Do you not j I do actually it's all in one video that's all in one video guide You don't even have to have multiple versions. It's just one video and you learn everything yeah, so we'll have um We'll make sure we have that in the show notes that way you can just dive into lvm and learn about it I've I gotta watch it again. I I don't do enough with lvm because at least I usually so far my planning has been Hey, it's the right size. So I don't need to do much, but it's definitely a concern Yes, it is Another thing that I think I'd be remiss if I didn't mention this you can actually reduce instead of enlarge So if you if you have your you know lvm is using up your entire disc You can actually reduce and you know shrink that down to where you could get free space back I don't ever recommend anyone does that some file systems Absolutely supports reducing and shrinking volume sizes, but I've still run into problems I don't know why xfs has historically been one of the worst when it comes to reducing a logical volume um in pretty much you can't although I have seen some um patches and improvements that claim that you can now But I've still not seen it work And even with ext4 one time I reduced it and the whole system wouldn't boot anymore Even technically you can reduce but just don't um just start small and then go up if you have like a 500 gig disc For example, give it 400 gigs for the physical volume or I mean the volume group just leaves 100 gigs on use So you have that space to play with if you need it in some room to you know Oh my god, I need space right now. Okay, you have some space That's just sitting there waiting to be used and you're fine. Just don't go down just go up Yep, keep keep your life simple like that. Yep. All right. Where's the next one we should jump into So now that we have lvm out of the way Um, we do have riser fs on our list as well if you want to talk about that Yeah, I just want to give the brief history. This was an interesting one If you want to go through the history of things the riser fs was really an innovative and interesting file system written By a pretty small team. It has a more tragic history of what happened to one of the developers Which you'll find when you start reading on there But yeah, this was the first time and I want to bring you up in there Because it was the first time I ran into writing a piece of backup software years and years ago in the early days of linux Um, I said, you know, I can design this whole system to back up all these client computers And I ran into some of the limitations back then it would have been the x t3 because this was circa 2004 when I wrote this um, but that was one of the things uh that You know, I it's kind of interesting when you start running to with the scalability file systems And I thought I'd preface it with just mentioning because it's a fun dive into the history of what that problem went out to solve And how we ended up with both butter fs and zfs zfs being as we mentioned the king of all this And I just wanted to bring that a little bit of history. I think it's interesting Divers i'm a big nerd when it comes to the history of computer and technology Just like jay as we we get on these discussions like we did last night about the history of some of this stuff Yeah, it just really gets an interesting mention because it was a really clever and innovative file system, unfortunately with the tragedy And related to the developer of that with a primary lead developer That kind of fell apart on there. So it's also an interesting of what happens when a project falls apart doesn't have a lead developer and kind of goes stale now Another we mentioned that I think we talked a little about mda mddam. I'm saying it wrong mda dm There we go. So tom doesn't say it wrong the one thing i'll mention with mdm is that is Just a system by which it's going to manage the hard drives And I wanted to mention it because that's very different than the way zfs works or the way butter fs can work Now i'm much less an expert on butter fs But with zfs zfs isn't just a file system with a lot of features. It's also Directly talking to the hard drives to format them the way if you want to build a rate array with the xt4 or Lvm and then xt400 or xfs under it is first you would physically link all the drives together with the linux raid utility Then once it creates this new logical block device here, this block device then gets the file system added to it This is one step in zfs and this is where i'm dive a little bit into zfs here zfs is the Billion dollar file system as quoted many times by wendell and a few other people We've actually been trying to find the exact developer. I want to get them I want to get a clip of him saying it so I can just insert him It's because of the years of development It's kind of just one of those marketing terms in some ways But it's one of when that many large scale enterprise companies have worked on it for that many years That's the kind of money that's been spent on Really fine-tuning and engineering now the reason it was unpopular in linux is because Oracle is known for one thing more than any other and it's not actually java. It's lawyers If there's something oracle is known for it's being litigious So there was obviously concerns because oracle bought the right For zfs when they acquired some of the other assets they've acquired But either way that created some licensing issues and things like that now zfs was still a great system But the licensing around oracle created a lot of drama So this kept it out of the linux market, but it was compatible with the free bsd licensing Q years forward and thanks to a popular product. We've talked about here True nas and this is actually the parent company ix systems is a big sponsor and employees could a few people working on this This brings us all the way around to a system that works much better And has a better license compatibility with the modern versions of like open zfs It's stable and things like that. That's why we're starting to see it Finally get into the linux world as it goes to just the bsd world because now there's not that fear of the Giant gorilla in the room that is oracle and the litigious nature of that company. They're just not friendly No one likes them. That's cool to what they think of them Yeah, it's like watching them duke it out over licensing. I think they employ more lawyers than developers honestly, but anyways Yeah, I almost feel like you could create a netflix tv show with all this drama that goes in the background and you know suing an intellectual property and all that No, it's so many things and I think that red hat still Is kind of afraid of zfs in a way. I don't think I don't know if they've actually said that openly, but I mean the fear might be that red hat being such a bigger company that there They'd be a bigger target for lawsuits would they go after a smaller distro Probably not as much as they would go after one of the bigger companies Of course, that is all Just my opinion based on no facts just kind of what I've observed, but it does kind of seem to be the case Yeah, so it's it's one of those things and it that's not to get too far into the legal history of it but a lot of times what these companies do oracle definitely included is They sue the largest company because they want to set a precedent or sometimes the second largest They try to find someone they can set a case precedent with because if one individual Developing some distro or even a small team of them. Maybe they could set a decent decent assist But no one would think a whole lot of it and they probably wouldn't have the resources to fight it So when they try to set their sights on certain ones, this is that case precedence But back over to the point file systems and zfs So now that we're starting to see zfs on linux q the latest version of churnass scale just hit release candidate the other day but zfs Is one a way to manage the drives directly So you take the drives and you put zfs on the drives not another not an adjacent file system Not a anything else it is a file system format at that level So because zfs is talking to the hard drives This gives a lot more control over the developers for the extra features They want to add because you're getting to control things at a block level This is why things like zfs send where you duplicate a data set from one zfs pool to another It doesn't even care if there's a million files or two files. It doesn't matter. It always looks at everything from the block Jeff from craft computing actually just did a video on deduplication The reason it works so well and the way you can deduplicate data on zfs is once again It's operating at that block level. So this is another great feature So if you have duplicate sets of data, it is able to create duplicate deduplication tables to say nope That's the same data. It may have two different file names We know the data and the block level that it works at this is some huge advantage for Companies who go well sometimes we do have duplicated data across clients But we can't have the client seeing each other But we can't create deduplication where that data can reside there There are so many things we could probably do an entire video once again on just zfs Because there's so many things now snapshots. That's another feature zfs has that's outstanding It's very granular and on the fly You can just restore a snapshot and have things instantly reverted back and forth With granular control of how you want to do that Block sizes how you write the data to the derivatives matters how you write the data to a data set matters And there are when you're dealing with a singular volume in ex t4 Well setting the block size per folder. That doesn't make sense. That's not something it can do zfs can This is where if you have something that's going to do a series of small write transactions It would be better to set them to a smaller block size that matches the commit size of the database transactions But that same zfs volume can have another data set that's optimized for let's say video editing where there's larger chunks And it would benefit more from a larger commit for the way that does This is one of the reasons zfs is so used in the scalability of all of it You're talking about a file system that can grow Absolutely to maddening scale. I forget exactly what the upper limit is for it at the moment, but It's absolutely why it's used in enterprise and cloud environments Quite frequently because of its scalability and someone may bring up something like sef s sef sef fs I don't know why I've heard time saying it but these actually run in the background on zfs And you can check out 45 drives as a whole series on sef to kind of get a better understanding of some of those things That goes outside the file system type Swinging it back over to zfs There's also the ability with zfs to add Besides the drives you have that store your data what are referred to as data v devs zfs also has that advantage Of adding things like metadata or caching v devs or D duplication v devs these extra files This extra drive types that are attached to the file system It kind of gives you hardware level features that you're adding to a file system that is then presented Is just a place you store your data, but all the behind the scenes can be delegated out to the most optimized devices on there So as much as we thought here in 2021 spinning rust would be a thing of the past It turns out they're really good at keep re-engineering spinning rust drives to make them bigger, but these Small ssds might be better for certain caching functions metadata functions or the deduplication table functions And this is one of the reasons zfs because it can do all of these things and more than we have time to cover in this podcast It's one of the reasons it's such a popular file system and such a great choice but it's only more recently That you can really see things like installers with zfs built into linux. Have you tried any from j I have and ubuntu. I think they still kind of consider it beta even though it's kind of not I'm hoping that with the next release of ubuntu the next lts. They'll take the beta term off of it But the desktop installer server installer gives you that ability um I have to be honest though I think i'm a little bit mixed about this because When it comes to a server, it's a no-brainer use zfs. I mean that that's a great thing But when it comes to your laptop or desktop, I'm not really so sure like I don't know how you feel about this I know that the zfs cache has to come into play here and that you're you're kind of losing more of your ram Maybe not that much. Maybe they've adjusted it to make it work in a sane way But it kind of just seems like overkill on desktops and laptops But I know a lot of people are really wanting to use it there So I don't know if you have an opinion on that But I don't think I've noticed a big decrease in performance. But then again, I have really good laptops too. So I'm not really sure if I'd recommend it there. Um Also when it comes to proxmox so far, I have not had a good experience with zfs and proxmox I I don't think if there's anything wrong with zfs. I think it's just how it's implemented where I would go to see how much memory I have free. Oh, I have 40 50 memory free. I'm good to set up another vm. Wait a minute. Why is it crashing? Oh, because the memory that is telling me that I have free I don't actually have free and then next thing you know vm start falling over Um, there's there's definitely probably a way to better tune that I think I've done that in the past So on linux, I feel like it it can kind of be Depending on the context and what you're using it on but on You know trueness. It's just so great. Like it's just such a win overall Yeah, I as far as running it on the desktop, there are some goods and bads one all that complexity I talk about cool if you need that level of complexity on your desktop performance wise Well out of the box, you're going to have things like sink turned on that may not be The most optimal way you want to set up for performance on there That's a problem memory memory isn't expensive But it's the way it becomes implemented because zfs on linux is fairly new Hence the reason they still have like the beta tag on it for a bunch of You know figuring out and tuning how much memory you need allocated to zfs zfs by its default nature Says hey, if you've asked for this file once you might ask for it again So let me hold this much in memory and zfs if there's free memory zfs likes to grab that free memory and put things there I mean, you're not using it and this is the way it works over in trueness if there's free memory It's going to use that free memory to cache files This is why it works so well for trueness because what else we can do with the memory unless you've allocated to Maybe some different jails you have running there or in the case of trueness scales some different Containers that you've got running in docker unless you've allocated to other resources It'll use the memory for that This becomes a little bit trickier on a desktop or laptop a desktop or laptop environment You're going to have a different set of parameters and do you really need all that cool features that cfs has That's kind of going to be a personal choice on there and there is no matter What once you start talking about adding the complexities of zfs. No matter how much they tune it Well that copy on write file system type of journaling that it does does have a little bit of overhead on it So there is not going to be necessarily a one-to-one performance with the x t4 Because it has that extra data. It has to be written verified and synced along with all the other complexities That there's more code to run through so when that happens There is just more to do to ensure the integrity of the data which is amazing You know and save you from things like bit rot but is bit rot even a concern on a Desktop or a laptop. It's something you really want to focus on your nas. It's kind of up to you I don't see a reason truly not to run it But if you're trying to get the most performance out of your hard drive And you're only a single drive and as I said zfs rates the drive or mostly when people have arrays the drives If you're not building an array of drives in your desktop I don't really is I have a harder time justifying the use case. That's one of the reasons for me I'll still use a default file system I mean when it comes out, I'll do testing with it because I'm kind of excited Maybe I'll test a server with it, but it's not as likely to become the native thing I format it with when I build these Yeah, I agree. I almost wonder if they intelligently control what's being cached If we could get into a situation where the entire desktop environment is cached and your most common apps that you use are all cached in ram To increase performance that way would be pretty sweet if they go that direction, but one can only hope I guess Yeah, so it's a and someone pointed out and they're right about this if you the native default if you're running A zfs on linux is to only take up to 50 max of memory not a hundred percent like it is in bsd But that is a system tunable feature you can And that's obviously something that's going to go with that further certain distros may tune it differently than other distros So that's the default from prescribed by the standard, but people you know follow standards. What? We're writing our own distro. We got our own ideas of how things should be done Yep, we have our own ideas for sure Yeah, so that's kind of like I said without spending the entire episode talking about zfs That's what it is and it's as I mentioned briefly copy on right This is one of the final integrity checks is there's always a copy of your data and on right We confirm it. We commit it. We sync it copy on right means There's always an extra journal of your data part of that integrity checking is zfs And of course zfs is handling if you have multiple drives of parity checking on there And let's swing it over to the world of butter fs now I admit we read me and jay both read some on here neither one of us will it all claim to be butter fs experts Butter fs is also a copy on right file system My feeling is that butter fs was really pushed because zfs and the oracle debacle for years Butter fs has been around a little while Butter fs has also got the ability to but from anything I've talked to people that seem to be acknowledged on this subject They don't recommend necessarily having butter fs Directly manage the drives. Let me give you a great example of a company. I've talked to about this Synology Synology chooses to use the linux raid tool to manage the drives But then once they manage the drives with the raid tool the md adm tool From there the volume it creates gives you the option to ext for or butter fs But our fs gets you all these cool checksum self healing features journal features It's a copy on right file system And it seems like most of the complaints people have like for stability in regards to butter fs seem to relate to The butter fs directly controlling the drives, which is what the synology has chose to say Nope, that's why we don't have butter fs controlling the drives. We control the drives with the tried and true Linux raid tools We've optimized for it in the synology product line and it sounds not the only one by the way My understanding is there's quite a few other nas vendors out there that do the same thing They use the linux base kernel They use md adm raid to control the drives at the physical layer And then that creates a logical block device that you then can choose to format with butter fs And butter fs does have snapshots does have a lot of features And synology has done a lot of nice integration into it as of many other companies So it allows that extra feature set on there I don't think butter fs is necessarily that bad under that context as long as it's not directly controlling the drives It seems to be a pretty robust operating system but Without large scale popularity you run into Misconceptions you run into maybe less knowledge from developers implementing it So there can be bad implementations of it out there from less experienced developers Which can also lead to and there's much less documentation out compared to zfs In terms of years of use because of companies like true nas pushing zfs So it becomes a little bit harder to truly work on and understand And I don't know that it has all the same zfs send features and all the extra Drive vdev types that you support in zfs. So It's good. It's still a good system. It's not a way it's not reason not to use it But I don't know that compare honestly because now with the choices it was the open zfs I don't really feel the future of butter fs when you talk about a bunch of pushing zfs I'm gonna say I don't see it the really bright and shiny future for it May that's my opinion. Yeah, orange by the way Well, I mean my opinion is similar. I think that I could see Open zfs being the default for pretty much every server related thing at one point In the future. I don't know how far from now Maybe butter fs for the desktop because I think it's probably better for that anyway because it's I don't think it's going to have as much overhead I feel like there's a reputation problem And I've noticed that this in the linux community that reputation is everything and that's true of most things But even more so in linux like if you get a bad reputation in linux, it sticks for many years I mean, even though I mean, I don't even know how long How long ago it was that linux meant had that big problem where someone got into their database server and They had some bad iso images that were made available for download That was a really long time ago people still talk about it A long time ago ubuntu had the amazon shopping thing in the unity dash They don't even use unity anymore and people are still talking about that. So in terms of butter fs I think what I remember happening is that they declared it stable too soon And it's not just butter fs. It's not their fault necessarily because a lot of the tools like df and all those Standard tools for disk storage wasn't patched for better fs yet So people weren't even getting like an accurate representation of how much space they actually had free And then they would blame butter fs for that. Does butter fs have problems? Well, yes It does But now that some people have had problems and people started complaining about it on account of the fact that they declared it stable too soon I kind of feel like that caused it to really not be as adopted as widely as it could have And I almost wonder if like developers kind of abandoned it like the volunteer developers abandoned it because if it has a bad reputation Why am I developing for this thing that everybody seems to hate? And then because there's not enough people developing on it the progress slows down So I feel like in a lot of ways butter fs was kind of like a victim Um, and yeah, there are rough edges like you've brought up there always is but I almost wonder if like we'd be past all of that now If it wasn't for that declaration of 1.0 that they made prematurely Yeah, and like I said, I I'm not saying it's a dead project at all by far not dead It's just when I look at the 20 year down the road future not like the next five or 10 years I don't see as many people like I said with the excitement here But then again, there's a new file system underdeveloped right now I don't know what even was called but I guarantee someone's doing it And um, that could be what we're using five years from now because someone just figured out something better You look at any any new technology if it's really well written and just solves all the problems But then becomes a simpler code base and we'll use like the excitement around wire guard Look how fast wire guard became popular now It's still not the drop in solution for everything But a lot of tooling was built around it very quickly because you're like, hey That's a really clever way to solve a problem of vpn We may have someone working right now or some small team that just has the next answer to the file system right now I think that still is the fs but a you know, the world does change So that's also if you listen to this podcast and just a reminder it is 2021 so right it's five years from now boy tom didn't know that this magic thing was coming Yeah, I wonder it's like it's funny to think about this because when I go back to the reputation thing I almost wonder if somebody could fork butter fs right now Give it a different name change nothing all the code is 100 the same different name Oh my god, this is the best thing ever everybody has to use this thing And it's just because the reputation doesn't follow it and they just give it a different name Maybe a different steward is taking it over or something like that. I'm not saying someone should fork butter fs If someone wants to do that they could do that But I almost wonder if that would probably happen if that were the case because it's just about the reputation That I think is holding against butter fs. I think it's fine for the most part and I plan on diving into it I don't know when I don't want to over promise But it is on my list to dive in a butter fs to start doing tutorials on that So yeah, I think 2022 would probably be a safe bet Yeah, and you know the someone I seen in a chat here and this is a fair enough Fair enough ask is like what about if the Synology switched over to zfs could they I don't know I really the devs seem to know it really well. Synology is Selling a lot of high-end systems We know because we've delivered a lot of high-end systems Synology and they've been really reliable really performance-based the engineers there have done a great job of making all that magic work Engineers that are dedicated and building a stable product aren't quick to change So sounds you would keep developing it because dropping in zfs would be a big development change because well all their tooling is built around butter fs But I think said you look at something like qnap. I'm not a fan of qnap for security reasons Follow steve gibson and look for qnap for reference on all of them not to get too off topic here But the qnap has zfs options. So that's something to think about There are other nas companies building it out there and they didn't always have them You know it once again comes back to that linux compatibility thing now that it's linux It's going to see out there But uh, you know as far as zfs and i'm always a firm firm believer in true nas but I have reviewed zfs on like the 45 drive system They have their own implementation of zfs and linux you check out 45 drive channel They actually do some nice dives into that but I think both are going to go for it Someone said better fs. I've always called it butter fs. I've heard both I think it's supposed to be better fs. But butter always just sounds better to me Yeah, I think it's it's butter fs I wonder if that individual said better fs as like um a remark on my um joke about forking it Call it better fs, which is very similar to the original name. I think margarine fs I seen someone suggest that that seems like an even better I can't believe it's not butter fs. Yeah We we can just come up with a lot of names here. I can't believe it's not butter fs is a good one. Of course, sure Ah All right, I think we we ran down the list and hopefully we educated a little about file systems And if you're really confused the end of this just use whatever your os default is because they've optimized for it That's why it's a default Yep, that's true. Yeah, that's a really honest answer I mean, I think it was I remember someone said is there what should I change at your default and To optimize it someone said if there was better defaults, they'd be the defaults I was like, that's like the best another developer told me that if there's better defaults, why wouldn't I make them the default? And I'm like, yeah, all right. Yeah, that's true I think we live in a world where the defaults kind of are out to get us in terms of security because of default Password and well to no updates. I think that our mindset Kind of becomes we need to switch away from the defaults because they're bad. That's not really true security Yeah, okay, it's bad, but it's no when your distribution defaults to it is probably fine I blame microsoft because unfortunately Broken by design is almost a default for microsoft for a lot of their security implementations So a lot of people There is a list of things you kind of need to do to lock down the system because they leave, you know I to my knowledge still the latest version of windows server is by default Vulnerable to memecats to keep on some backwards compatibility for really old stuff That I think should be something you turn on because why do you have something that old in your network that needs this weird authentication? Fine turn it on but don't leave these things on default. So I I can see both arguments for it because this is something I heard from a development team developing a very popular well-secured product versus not something Microsoft's like got a lot of things you got to change by default. So there's reasons to ask it There really are I think um, they're firewall when they implemented that in windows xp service pack two It all it did was train everyone to hit the yes button to allow everything in so then everybody thought that just having a firewall Was fine while they're just allowing everything in I think it kind of hurt the Overall mentality of what a firewall is and what it's good for among other things Yep fun stuff and uh, yeah, we will Do dive into some other topics for some next shows but I think we beat this one up We've covered it all we've got all as we can possibly Eek out of us. I will leave links to jays lvm video because that's handy. Do you have video on lux as well? I do but it's in regards to arch Linux specifically for Installation, so I'm not sure if I'd recommend most people do it But you probably use the same commands that I use in there to create a volume Yeah, that way is at least one thing to think about where you're creating file systems and zfs supports this and uh, we've with lux and combined with lvm like mentioned, uh, do encrypt everything I'm a big huge fan of encrypting everything So it because especially laptops are extremely easy to wander off And at the worst time and uh, you want to make sure everything's encrypted very well So I still what no matter what file system you use setting things up that way is uh, yeah protect your bits. That is that is huge All right, thanks for joining and uh, see you next time send questions and Comments and everything else to them on this video or reach out to us on our website We got a contact form and we do read them. So that's how we build our next q&a episode. Thank you. Thank you