 Good morning. Good afternoon. Good evening. Wherever you're hailing from welcome to another episode of Enterprise Linux presents. I am Chris short executive producer of OpenShift TV I am here with the one and only Scott McBrien Scott. How are you doing today, sir? Doing great Chris looking forward to talking about some file system stuff and We'll see where that takes us. Yeah, and I know the topic seems Very rudimentary to some that might watch this channel normally But this is a foundational topic that will help everyone in the long run, right? Like Understanding the Linux file system and how it's structured Really helps you get a hold of Linux systems in general and manage them better And also don't store stuff in Vartemp. That'll be one lesson you should take away from this Well and in truth be told like if the audience wants to drive us down a rabbit hole, let's let's Brace it wholeheartedly. Yeah, I am all for it, right? Like consider this an office hour about Linux file system directory structures and We will happily take you down any rabbit hole you want to go down. We are ready. I don't know about happily Well, I'll happily do it because I enjoy it regardless So We've talked about this a while ago and it's like like said a foundational topic that a lot of folks don't always Have their arms around that figured we'd start with just talking about like what that top level of directory structure looks like So let me share my screen And you actually went above and beyond here I did. Yeah So Let's just take a look at that top level directory and What I did was I just took this list of directories and I put them over Here on another tab so we could just have something to like look at that's not Yeah, yeah, no that makes sense I like it and and putting Chris short on spot is always fun to do So I figured we would take turns talking about what we expect to find in each of these directories So Chris, I will let you go first pick pick any director. I'll just do it in alphabetical order, right? Like then Okay, then contains binaries That's one way to look at it, right? Like it's it's generally right like system level binaries There's other places to put, you know your personal binaries or you know Artifacts from your your your software that might be running but like this is where cat and LS and RM and all those you know like core Linux commands those binaries live here And I will see your binaries go other places but called bin and raise you Hmm If you take a look at the breath environment variable Every user if they log in with a bashell and use the standard Shell scripts that come with rel Will actually have their own personal bin directory Added to their path by default. Nice. That's amazing. So yeah crazy crazy town Um, what are you gonna pick next like you can run off of medically? Well, I just said I would start alphabetically It doesn't mean we have to stick with that So I'll stick with that theme. Okay, but you might get yourself in trouble later. We'll see we'll see I probably will So slash dev is where we expect to find device files these are a Logical representation of devices attached to the system. So this is where we find The these special kind of files That represent hardware and generally they're indicated on whether they're a character or black device and then there's like The major and minor numbers instead of number of links and some other pieces of data about them in their i-node But more more generically, this is where we find device files that represent the devices attached to the system Yeah, remember everything in Linux is kind of a file, right? Like that's kind of the idea So you can muckity muck with it on the command line as such but there are special kinds of files That are devices Alright your turn short. All right, so let's see. I mean, I'll just do home home is where all your users Live essentially, right? Like they will work inside home Their home directory is theirs. It's like unless you're root on a system one user can't look at another user's directory by default at least So it's kind of like your private playground to do whatever you need to do and you'll see Interestingly enough root does not live under home. It was under root. We'll talk about that later, but You know your home directory will have your files in it, right? Like if you use like the the GUI for Well, I forget. No, if you use no for example That home directory is kind of like where everything is for you your downloads your documents everything that kind of deal Yeah, and What goes in all these directories is dictated by a document called the Linux file system hierarchy standard and For for Linux, this is where user directories go for Unix at large That's not always the case right so for example on my my hyrax box that I I love dearly But was modern 30 years ago It shows to put home directories and users share people Oh, or yeah, yeah, so But home is is where most modern distributions will put user stuff And I would like to point out that in the official docs. It says home is optional just to give you an idea of You know how Sometimes there are systems that have no home directories and that should not be taken as an abnormality Yeah for rel we we include it right but but yes, there could be specialized builds of other Linux's that maybe would not include it But we we always include it in our in our file system So I'll do a twofer Okay, because we have lib64 and then there's also live and they actually both pretty much hold the same thing now But that wasn't the case always Yes, but they're both first shared libraries and the difference between them is that You know once upon a time we were combined with 32 bit Binary's and libraries and then at some point we started supporting 64 bit stuff so lib64 was created to hold those and Now we've kind of coalesced them back together again, but I'll talk about that coalescing a bit So lib and lib64 are for shared libraries So does this mean I should take mount and media? Or they are two different things. They are two different things, but they do use the mount command to get stuff there, correct? Yes, in most cases I feel like yeah, um, but anyways mount is the you know the facto point for mounting file systems of another variety, right? So You know, if you have your iSCSI network, this would get mounted here as a subdirectory underneath mount You know your nfs shares would get mounted for system use in the mount directory and notice it says mnt and I'm saying mount that's because Like they're one in the same kind of deal where media is a mount point for Removable media. So like you insert a thumb drive or cd dvd, whatever even certain kinds of Card readers that kind of thing right like that'll show up in media But it does use mount to put those things into media And media is kind of newer than Mount in the directory structure by number of years Yes, and and there's actually a newer one still that we now use in updated versions of known But that's right I can probably keep up with this stuff sometimes. I know and and that's one of the things like we Yeah, red hat Provide this file system layout for row based off of the file system hierarchy standard And there are sometimes where something exists in the standard that we don't really use Right, but we keep it because either Historically, we've used it in the case of melt for example Or maybe other people use it Which I think may still be the case today for media because I know that our stuff that used to use it has been moved elsewhere Um, but it's part of the standard. So we comply with the standard great um We've talked about the next one on the show if we're going in Line by line order here. Yeah, I'll find the link real quick for everybody So proc is a in-memory file system. It's actually not stored on disk So anything you look at in that directory is actually you're looking at um components in the kernels memory in the runtime Um, and so there's all kinds of stuff in there and we've actually done two shows on it one was just Overview and another one was a deeper dive and tunables that are there But um, it got its name because originally it was where we could find process information um So like when you look in there, you'll see all the pids of the processes right on the system home Home is where you make it That's a funny funny comment. I like that one There's the proc episode I'm dropping in the chat now and then the tunables episode. I'll grab here real quick. Come on youtube search How are you doing that? I'll just go ahead and yeah, please do. Thank you So I I mentioned that there was a newer place that um, gnome was using for file system removal media stuff and that's under run these are Four mount points that people need during the runtime of an application So when you plug a usb thumbstick into gnome, that's running it's running application It's like, oh, I need to make a mount point to mount this usb stick that just got plugged in for my user So I can run the files application So it creates this subdirect render run and then um attaches that device at that location nice All right, so you did run I'll do boot boot is a fun one, right? So boot is actually like Formatted differently as far as you know, your petition goes sometimes um and it's Supposed to sit in theory at the front of the disc But it's kind of hard like in the virtual world to imagine the front of the disc, right? Like It's especially with ssds, right um That was because on spinning media spinning rust as I like to call it nowadays um It would be faster access times So now with ssds and nvme drives like the access time is consistent across the entire storage device So it doesn't really matter as much but boot is where The init system lives, uh, well not all of it, but some of it um and like How linux starts on your system that lives here the kernel like lives here Uh, and everything gets loaded at startup from boot Yeah, and it used to be like way way back um, you put the Boot loader in the master root recorder disc, which was the first sector of the disc Right. We can't put a whole lot of smarts in one sector of space So, um, we had to create boot and make sure that it was at the front of the disc because You could put a little binary in there to be your boot loader, but it could only have so much address space So you had to store those files that it was accessing Within a certain amount of your disc because it just didn't have the address space to find them if they were stored later on um That this one is one that's kind of still a little funky um because in some places like Cloud images Oh, and a lot of the marketplace images and stuff have moved away from this. You just have the slash file system and that's it um, however for like Disastig security compliance Um, it requires that this be a standalone file system On a disc or partition um, and so when you're dealing with that, um, you also need to think about things like space so the kernel the binary executable part of the kernel not the modules Is stored here and so 10 years ago You could get away with a hundred megabyte boot partition and be cool because yeah, part of the kernel was like Five megabytes, right? Exactly. Um, that is not the case anymore so You may need several hundred megabytes for your boot file system So that you can apply updates on the kernel and not have to immediately remove the old one in order to have enough space In rel when you update the kernel you will have your old copy and your new copy here Unless you tell the kernel clean up itself. Don't clean up the known good one first Make sure the new one boots before you clean up any old ones Spoken like a true systems administrator. I have been bitten by this before. Yes Uh because the boot partition was so small and then the kernel that was upgraded didn't Work, so yeah Yeah, good stuff um So and the other thing that we find here is like the second stage of grub and You know the the larger boot litter components that are executed from the nbr record and old biases Um, so etsy is for the all the other stuff Not really so that's how it was that's how it was initially done um back in like the system five and Actually, it was berkeley that started that that directory But what has happened over time is it's become the place where we put configuration files Because like you put your binaries in the bin directory And you put other files and other directories and then it was like well, I need this config file Where's that go? Well, put it in dig set our directory and it over time That just became the place where all the configuration files went So if you're looking for config files, I would look in Slash, etc Yeah etsy containers is a great directory if you're using podman or a new container You know orchestrator or driver or anything like that The etsy containers containers.com will contain all your image registries for example um, so that's always good to be aware of and Uh, you know that order in precedence matters as well So we did live we did media we did media so opt so opt like For me op is I don't know. I kind of I have a love hate relationship Right like I get it opt is where You know software lands right like external packages, you know blah blah blah That's where your package lands or your provided thing lands and where it's consumed from what ends up often happening in opt Is you get like opt like opt bin and doc and stuff are usually in there But then you get opt package bin and people get confused sometimes right like let's say my package is You know a go binary And that's not where the go binary goes right like I think opt is for specifically stuff. You're deploying right like it's not necessarily the de facto place where stuff gets installed Which is kind of confusing to people um But if you look at the page for op there's some very specific Uh Ooh, there's a question in here. Sorry Does etsy stand for editable text configurations I thought it stood for et cetera like the the other stuff. Right. Let's see here. Um Doesn't necessarily say that in the docs I wonder if we'd have to actually go down into like the the bowels of berkeley unix Yeah, maybe it's in some old page somewhere. Maybe possibly it's entirely possible, but that's the case but I'm not seeing anything here to indicate that That is the case But back to opt back to opt uh intentionally it was for additional software to land so like again back back in the Another realm of time um Yes, the 90s otherwise known as the 90s um So if you were a unix producer You would put your binaries in bin and you would put your library's lib But if you were an isv or a third party developer You weren't supposed to put your stuff in those directories. That was for operating system vendor stuff only right um And so they created opt to be the place where if you were adding on stuff That's where you should add it on because it was optional to the system and so because of that complex history and the fact that Things like bin directories under opt were added to people's paths by default um That that line got blurred really fast over the need for Ease of use So there are still some packages that will do that like a lot of the add-on software you get from IBM or Oracle for example with their database suite Yeah, they'll put it down under opt and so there will be a directory in opt that says like You know name of software and then underneath of that will be Bin lib, you know yada yada yada for the software that goes along with that That's worth noting too that opt came from the unix land right and I don't in my early days of using linux. I don't necessarily remember opt being there It was it was but it's been there. It wasn't used just like it is now I guess probably the right description and even now. I don't think it's really used all that often like Typically where I see it is when people are building across multi-architecture and happen to support a unix flavor as well They'll use opt And then when it comes to linux they use opt too because it kind of makes sense for their packaging and sanity but a lot of stuff like if you install software off of the Extra packages for enterprise linux repository for example It's not going to put it not no, it's going to put it in bin or lib or you know, wherever else it it goes into the file system directory structure And that's kind of just the way it is yeah, so In there's not a one good approach like You know, it's an operating system, right? There's a thousand ways to be one thing Right now. I will say if you're installing additional software that happens to have the same name as system provided software Please put it in opt. Yeah, don't don't Know that like that's yeah, and we've even made that Not necessarily harder in rel 8, but we've we've made some very clear lines there with rel 8, right? Like this is where system stuff belongs this is where User stuff belongs right like we've we've made clear delineations there Yeah You're taking root. I'm taking root. So this is the root user accounts home directory We don't put it in slash home because Chris pointed out earlier. It's flagged as optional Also, a lot of times it's set up as a separate file system. So it goes to like different storage and if your system was having problems And for whatever reason that storage wasn't available when you log in as root You still need to maybe have access to your own stuff So slash root goes as part of the slash file system and it goes on the storage of the slash file system Other unixes will actually not put this out there. So when you log in as root on Solaris for example, you get dropped into slash as your home directory Which means when you download a file or your desktop wants to create things like downloads and pictures or whatever It puts the top most directory because that's your home directory for root so Things got kind of cluttered at that top level when you actually had someone operating as root on a box So in the file system hierarchy standard, they created this sub directory to kind of hide all that root user mess from the rest of people And and this is an interest like as we're going through these I'm playing up the docs. This is an interesting footnote on root We recommend against the root account We recommend against using the root account for tasks that can be performed as an unprivileged user. What a great idea And that it be used solely for system administration For this reason, we recommend that sub directories for mail and other applications not appear in the root account's home directory And that mail for administration roles such as root postmaster and webmaster would be forwarded to an appropriate user What does this mean To me, I read this as Root is cli only Don't install a gooey as root, right and have mail and all kinds of other stuff living under this directory because It is going to be a necessary thing for when stuff goes bump in the night You want that to be there, right and only really important things should just live in root For system administration, I mean Fair enough. Yeah, I would probably be Um Less murderous about it because I mean that's just me. Yeah, and I get it um but there's a lot of like work stations that people own And I could see them maybe using root on that But again for system administration stuff, I think you typically will operate on those boxes as a regular user for a whole host of reasons not just security and file ownership and variety of other things So we have Sbin No, we have a comment slash question. Um from rapscallion or eaves our friend And we might not be we might not be able to answer this because I don't know the answer Maybe you do Scott. Why are bsd and linux's file trees so similar but so different I don't get why bsd has user local bin and user local etsy for example Um, that's just their decision-making process, right? Like So I think it ties to the history of how we got here, right? So once upon a time, there were two large unix distributions Berkeley and system five and as they matured and they wanted to Um Woo users from one to the other or back again They had to build up this kind of cross populated directory structural horribleness so like if you were a Berkeley person right because you started on Solaris when it was solaris four dot something something And then your company decided that you were going to be an hpx shop, which was system five So you log into your first hpx machine. You're like, where is everything? So hpx or aix for that matter too from IBM They started to create these kind of additional directories to make it more comfortable for people coming from Berkeley And then son was like well, I mean what if somebody goes from aix or whatever to solaris? So they started doing the same thing with the system five directories um, and so When you look at the commercial unixes, I think a lot of them end up cross-populating between the Berkeley and system five standards Um, and then linux comes along and it's the same deal. They have to appeal to people who may be Used to working in a variety of unixes. So they created this much more amalgam of directory structures um but bsd is still a legitimate Unix distribution with a couple of different flavors and they've stayed pretty true to their Their file system hierarchy standard that they minted back in the day And so that's I think why you find like free bsd or net bsd have a different directory structure because there wasn't this, um pressure To create these additional directories to create comfort for people moving between dish trails right Cool, uh, so i've got a question here. Um Do you'll know this better than I do? uh When is Well, will rel migrate to butterfs? So we shipped, uh, butterfs as tech preview For the duration of rel seven right And then you'll notice that it's not included with rel eight So there's still work going on in butterfs. Yeah, it's certainly in the community more stable Yeah, like the biggest thing about file systems Uh, nobody likes it when their data vanishes So You know, you have to make sure that that stuff is rock solid like somebody was, uh, just recently was asking me about file system driver for linux And we do not ship that as part of rel You can get it in places like I think evil may have it. Yeah So you can get a module and it will plug into your kernel and stuff um But that's because there is a extremely unlikely um situation where You can lose data And so if you're mounting an ntfs file system into your linux directory structure and then data starts vanishing No bueno Yeah, um Yeah, people tend to frown on that especially in the enterprise Distribution world we like to make sure your data is as stable and intact as possible as we're going about making your operating systems So, yeah, I mean there's there's a lot of work being done on butterfs fedora is using butterfs I'm using that screen right there. I'm using butterfs. Um I don't have anything of significance that I cannot replace easily on that box for example though as a result And I don't think butterfs is like ridiculously unstable, but I mean even if it's a No, it's a very unlikely situation. Yeah Yeah, so I know that our product manager for file system Stuff is continue to keep an eye on butterfs um So I I don't remember if we've put it into centOS stream nine CentOS stream nine Was had their first build just like a week ago And that should be a pretty good representation of what we expect to land in in row next year when we release row nine um I have not looked for butterfs, but that would be if I was going to look to see if it was maybe going to land in there That's right first. Mm-hmm. Yeah, check out centOS stream. See we're Because that's the progression right fedora centOS stream rail Yep, tada Yeah, and centOS stream Keeps a pretty close relationship with rail. So when you're looking at centOS stream You're looking at a pretty good representation of what you can expect it in the next version of rail Makes sense. All right, so you're up with espin Oh so espin is Another one of those throwbacks I feel like Maybe maybe not. Uh, I don't remember which came first So you said bin was for things like ls Yeah, so espin is for things like fdisk there you go and Like system binaries right like stuff that's gonna muck with the system itself And again, this is another one like opt where it's like Was this here when I was using like redhead enterprise linux five when I first started And yeah, it definitely was and it came from The need to like Back in the day You partitioned the hell out of disks Um Nowadays you might not so much. It might be one or two partitions, right? So you would want espin to be in a similarly fast, you know file system slash space on disk as Boot was right because you're gonna need fsck at boot time sometimes right so like There was that for a little bit and People kind of went away from that as disks got faster So more generically it's for systems administration binaries So like partitioning disks fsck. Yeah, exactly And like way way back we could do things like change the permissions on it so that people couldn't access them But uh, we also for regular users don't include the espin directories on their path So when they're looking for commands, they probably won't find these with things like tab completion because they're not system administrators They don't get the espin directory and then over time We we figured out that that wasn't a great plan Because people would still add it to their path and find it anyway. So we started to do things like Building checks to the system administration binaries to be like if your user id zero Then please partition the disk If not start up in a read-only mode and tell the user that they can't do anything about lock But no touching Right. Yeah question chat here. Um Are sim links between bin and espin do they exist like they are between bin and user bin? That's a good question So I was gonna handle the sim link question at the end because that's a change that we made in rel seven Yes And I was going to flip back over like my actual box speaking which I should probably uh, yeah flip back over show us all right so if you look in our Our directory here, right? You'll see that bin lib lib 64 and espin are all symbolic links And they point to their user counter parts So this is a functionality of combining The linux file system hierarchy standard out of berkeley and system five and so in berkeley they put all of the Bin lib espin at the top level of the directory Whereas in um system fives directory file system hierarchy standard They put it under user for unix system resources And Then it became like well As time went on and we started to cross populate between those two directory structures Sometimes the thing wouldn't been and sometimes it wouldn't use your bin or sometimes the thing went into espin and other times I wanted to use your espin So we ended up having to start adding those directories to Like both sets of those directories to everyone's paths Um, and then in rel seven We just went ahead and merged them together And we're going to do sim links. So in reality everything in rel goes into user Um, and then if your package when you're you know installing an rpn or whatever says slash bin That's fine. It just follows the sim link and puts it in user bin So, um, we created those sim links to still maintain that compatibility And I missed this question from rat's gallant reese one thing I still don't understand about fedora 34 and butter fs Since compression is enabled by default if you check the encryption box during f 34 install Will that mesh you up? How can butter fs encrypt compress encrypted data? Let's and let's not conflate compression and encryption, right? They are vastly different things An encryption algorithm masks the data Using the you know prescribed algorithm Compression uses a different algorithm to make more space on the disk, right? Like that's essentially the way to look at it butter fs uses compression in the file system, which should be in theory more efficient. Um Then like using external processes to make that compression happen and then decompress and recompress in and out kind of deal um But encryption can happen before or after compression, right? Like there's no wrong order to do that in but you have to decrypt a disk Before you can load it. So it usually is compressed and then encrypted Oh, or am I wrong in saying that? I think I think that's mostly mostly right. Okay. Um The one place where you might get into some weirdness is if you encrypt first then compress Because oh, yeah when you compress stuff things like text compress really well Whereas binary does not text very well. And so I wonder if you encrypt something if it changes the data to the point where it doesn't compress very well anymore um, I think there's a point of diminishing returns, right Yeah, for sure um We're now we're getting two questions about the difference between x4 and butter fs What's more secure? What's more performant? So I am not a butter fs user I am I mean I am a butter fs experimenter at this point, right? Like I don't know enough about it to be like it's definitively more performant than x4 um, or xfs or anything else for that matter And I would say that I Just knowing myself I probably won't pay much attention to butter fs until it comes back in a row. Yeah So, I mean And the only reason I'm using it is because it's the default in the f 34 And 33 and 32 But yeah, that's a good question. Uh, I'll try to find some research on that for y'all So we should probably ask bob handlin Of long time watchers may recognize him from like our first or second actually a second or third show Yeah, uh, because he came on to talk to us about kpatch, maybe I think but he is also the data services product manager furrell So He's the one way I mentioned somebody was keeping an eye on it. He's the one who's keeping an eye on it And that might be our next stop to Ask these kinds of questions See like here's a performance like a really good one from furrow necks Uh, but it's from 2019. So it's like, I don't know if that data is still good Uh, let's see Yeah, and then in terms of like file system On rel By default we use xfs And but we still offer ext4 as an alternative Um, and the big difference there is things like I know to look up tables and numbers of files that one can have in a directory um xfs just does better with small files lots and lots of small files Um, then ext4 but is about the same for normal file workloads so we have um Some folks that like run their web store using rowel And there they have like a good jillion tiny files because like each product review is a file Each like suggested product might be a file And so for each product that they sell in their store, there might be like thousands of files underneath of it So we we use xfs because of that, right Yeah, I mean And I linked an article in chat folks from fedora magazine about choosing between butter fs and x4 So yeah, and it talks about lvm in there too and how you can use multiple devices So yeah, you can do a lot with the two file systems as you see fit Yeah, and if you're interested in things like compression and saving space You may also be interested in taking a look at video Which is the virtual data optimizer um that comes with rowel Essentially uses data deduplication Yes, um to right to to not store the same bits on disk multiple times But All right, so I think it's my turn SRv is one that we don't really use in rowel but like debian distros use a lot So it's for server or service files Um But yeah, we we don't really make use of that in rowel. Yeah, I mean it's Rational the main purpose of specifying this is so that users may find the location of data files for a particular service You know, there's there's a lot of caveats under here. Um Like some stuff should be in var lib Right like so. Yeah, just keep that in mind So the sys directory is One I am I will have to admit I'm not familiar with sys directory And I'm searching the docs for it right now So we talked about it a couple weeks ago. We're talking about tutorials Because like proc it is an in-memory file system, right? and so, um You have things there like um your preference for using huge pages in the kernel or power consumption settings or block device tools, right? This is the system settings for all the things. Yes. Yes. Sorry. Okay. Now it came back to me Then why don't you take temp? Okay temp uh is almost exactly what you think it's for it is for temporary files, but the different temp has a a relative in var temp Which will actually purge itself on boot So if you're putting files in temp They will last through a reboot where var temp. They may or may not depending upon the os Yes, depending on the os for well, they do persist across rights, but like temp Um, there is a service that's run as a cron job Right that will remove stuff from var temp as well. Mm-hmm um, and I use temp for like literally the output of Scripts that i'm running as like a diagnostic type thing or If i'm running an ansible playbook and I need to actually write a file to disk and then I know i'm going to get rid of it later Temp is where I do it, right? So Anything that I know i'm going to have to have some outputs That has to land on disk and then I want to erase it later. That's What temp is for? Yeah, and I think um by default stuff gets erased from temp every 10 days if it's not been used Oh, really? Thanks. Yeah, thanks to the temp watch service. That's why we threw cron So basically like goes into temp and looks at all the files and directories in there and what's the last access time stamp And the last access time stamp is older than 10 days ago. It gets killed off then moved Nice. Yeah Uh user we talked about a little bit already unix system resources and we merged that together with bin lib and sbin So this is where our binaries and libraries actually get stored for rel 7 and rel 8 Var is one of my favorite directories because there's so much going on in right like Uh, you know var logs var cache var crash var lib var lock like everything kind of like running on the system is Under like the outputs of that is all under var, right? So if you're looking for logs somewhere go look in var logs If you're looking for lock files that you're trying to get rid of go look in var lock But the the the purpose of like var itself it contains variable data files. That's literally why it's called var This includes like spool directories. So print jobs emails things like that administrative logging data transient files temporary files One thing to point out is that like some things Living in var aren't shareable right across uh systems So just keep that in mind right like you don't want to mount var log To the same thing on every system unless you have a way to take care of that output You know, right like your logs are structured in a certain way to identify themselves as coming from certain systems, right? Otherwise, it's just going to be a massive mess for yourself to clean up Yeah, and we actually have a service to to do that now. Yeah, if you use our syslog or syslog ng um, you can have The logging service on your individual boxes congregate all of those into a specific system And it takes care of like creating a subdirectory under var log with each system name or ip and then puts all the stuff that it got from them there But you know if you want to get a good Did you say temp is mounted in ram? No, we did not say temp is mounted. It is not mounted in ram. No, uh, it is all disk. Um Sorry, uh, there's questions that i'm missing because i'm so Just on you right now you're grossed in the conversation. Well, i'm i'm trying to keep my place in the slides here if that makes sense, um But yeah, like var Shoot Just slipped my brain. I I spent a lot of time in var Working with databases for example, right? Like That's where your data from your databases will land. Um, and you want to make sure that that var directory especially for that database is fast Very fast, right? Like you want var to kind of be your fastest disk at times for this reason Yeah, I think that that like finishes all of our directory. So let's um does so let's get All right, let's just see questions answered um Would you consider testing zfs on fedora? I mean Maybe uh so zfs is not um Open source Well, there's it's not open source with a license that we can Right distribute or something. There is an open zhs. I think that is making the rounds lately, but There's still some licensing stuff that needs to be taken care of around zfs I mean it I love the file system, right? Like we one of my First jobs out of the military. We had an oracle storage array. That's the only thing oracle we bought because it used zfs um, and it's a really impressive file system, but it is very tightly licensed in the sense of like it's incompatible with uh open source software to some extent um So, yeah, that's just something to keep in mind Linux has something for zfs like it's making its way out into the world Um, so just keep that in mind It's and I thought that butter fs like uh an attempt to reinvent that wheel in a more Friendly way. Yeah, like one that was not intellectually encumbered, right? Um, and that's kind of the idea But yeah, uh, oh boy, so one of my co-workers just got Another one of my co-workers riled up about butter fs So maybe they'll be joining the call here or the channel here in a second to talk about things Um, but yeah, so we'll see um One of the questions we got I'd really like to see butter fs be the new zfs. Yes, I would too um We covered the ram thing I thought exported things like nfs should come from slash srv Sort of like the opposite of mount Well, that's a good point Where do nfs mounts land typically So it depends on what you're trying to share, right if you were trying like Back in the day, we would do things like nfs share our user directory because we would install MATLAB On a server and we want other client boxes to be able to mount and execute it locally And so we we share user as an nfs share I think the biggest thing on rel for nfs is making sure that it's got the right sce linux contexts to be shared. Yes So Yeah, I think I still end up putting stuff under mount Yeah, me too just so that I can context it correctly and update my sce linux policy So that it knows that things in that directory gets get done like that, but you could do the same thing in in srv If that's what you want to do All right, I did have one it's kind of like there's There's the directory structure which follows, you know a thing a standard But you can deviate from that standard and some of that standard is optional and so forth so on Do okay. Here's another question. Do you use x4 or butterfs in your daily life? I don't I use that's butter. Yeah, I was I was about to say I'm pretty sure you're using xfs Just from previous conversations. So I have used xfs for a long time and I've only switched over to butterfs recently That's just because I'm using fredora more and you know rel less here in the house but that doesn't mean that like When I muck with servers, I'm probably touching xfs In general nowadays, I feel like So chris. Yes, sir. Looking at this top level directory. I'm using colorized ls output. Mm-hmm Do you know why temp is green? temp is green Because it has a special mount I'm sorry Just ls-ld temp and you'll see why It has the t at the end there the the wt Sticky bit. Yeah sticky bits. Yes that thing I forget I would remember it from the shamad but never like Not the numbers the words Right, like I memorized the numbers back in the day where other people memorize the words for change mod. Um, yeah So so temp because it is shared by a lot of different processes a lot of different people Is stickied meaning that Anybody can write to it anyone can write. Well, this is what stick it in this way. Yeah, anyone can write to it um, but the sticky bit says that Only the owner Of a file or directory may remove it So while normally anybody that has right permission on directory can had or delete files from it In a sticky directory anyone can add files, but only the user owner of a file can delete it There if I having the file stuck in the directory Until someone wants to get rid of it Yeah Um, doesn't this also mean that it will inherit the these settings all the way down no matter how many directories you create underneath it That is uh The ownership well right sg id. There you go. So yeah Yeah, that will that will propagate the group owner all the way through it When you get into sticky bits things get sticky. Yeah. Well, I'm like, how do they apply to files versus directories? Right, exactly. They behave differently and I will say that the sticky bit on files on xq goals in the linux is not honored All right, so so here it is. You ready scott? I'm ready. All right. Andrew Sullivan showed up I guess instead of christian her hand does So Andrew says I probably unfairly shy away from butterfs as a result of the now quite old Data corruption bugs. I stopped paying attention to it back then But I understand there's quite a few commercial products using it now including synology. He thinks Um, that's interesting to learn if that's the case If your sonology box is running butterfs, let me know I mean that's fair we we in technology tend to have really Long memories for badness. Yeah. Yeah, so things like I was just talking to somebody about uh satellite six Oh Yeah, okay, and they were they were like It's like but but it's fine now. Like that was now that was a long time ago six dot one or whatever Yeah Yeah, so synology has a page about how butterfs protects your company's data Next generation file system delivers data integrity and efficient snapshots The snapshotting is what everybody seems to like the most about butterfs Um, does xfs perform better than x4? Kind of depends on the workload depends on the workload every every performance use case is going to be workload dependent Yeah, so literally everyone so xfs does better for stupid amounts of small files At one point it actually was able to actually store more files at the directory level so like In in ext I don't remember the version You could only have 64,535 files under directory before it would like that. Oh, yeah, that's right anymore. Yeah and xfs like it's ridiculous amounts that I would Challenge someone to actually achieve It's like millions of files and that in a single directory level that can retrieve it, but because it's Better at looking up and retrieving those tiny large amounts of files That's kind of its sweet spot And it does about the same as the other file systems for just normal files to my help Can we touch on redirect on right? Uh, oh you're talking about data duplication. Yeah. Yeah. Yeah So I actually uh, I actually have a lot about that. Yeah, that's what I figured. That's why I asked It was written by uh, Sean O'Keefe. I think who is the product owner for For this technology So this we're doing reducing operational storage costs with the video um, and essentially it's like installing video and actually Using it to figure out how much data is really being stored when you um, are interacting with the the video devices so, um We don't have time to go through all that Yeah, but um video is implemented through a kernel module So you want to make sure that you are running the same kernel module that you have on your running kernel um, and so Whenever this provisions you'll see that very first command here is the yum install and It has a video version and I actually specify the version in the command and the k video Uh k mod also has a version that I specify because it all has to match the currently running kernel Right. So if I do a ashore um Well, I wonder if this will work. Let's see Well, there you go Something's happening. I think maybe not no Well, it may be installed. Let's see if it actually works The um the lab vendor actually rebuilt my labs without me asking them to a newer version a newer update release of rel 83 So it's like 834 um and I Did not go through and test all the labs after that We're about to update them to rel 8.4 because that was released today. Yes. I was waiting for you to say that It's been waiting a long time too It's like, is he really going to sit on this all the whole episode? Sorry not thinking No, you're fine. Uh, we mentioned it on our earlier show. So that's what made me, uh, like be like uoma mess with scott Um, but yes rel 8.4 was released today. Uh, just a few hours ago as a matter of fact So yeah, go check it out. Keep the tires on it on your desk systems, please It is working. So that is the right, uh The right kernel module and video version Normally when you run the video command, it'll like error if you have the wrong if you have a mismatch your module number Nice so um But yeah, the video stats is the command that like shows you how much you're using Um, and as you're like storing files, it'll report on how much is being used and that's where you can track The data duplication that's being Managed exactly Awesome Cool, so I think um Yeah, I hope this was to call it a day. Yeah, I hope this was educational for folks, right? Like if you have any questions Feel free to reach out to me short at redhead.com or chris short on twitter And you know, we're very happy to have you here for this episode and we look forward to seeing you tomorrow at 9 a.m to talk about Same show different name the data services office hour So yes See y'all then And thank you scott for a fun episode yet again. Thanks chris. Well next time we'll try and get a guest So in a couple weeks, we'll do a guest. Yeah. All right. Cool. See y'all then take it easy everybody