 Hi everybody, I am Matthew Miller and this is a Fedora council video meeting we try to conduct our business on email and IRC and chat and things like that most of the time sometimes in tickets, but it's also good to have these high bandwidth conversations so we do a video meeting about once a week we usually check in with some interesting part of the project. See what's going on there, what needs, what things Fedora council can do and just kind of to distribute the information to people from what's going on from something they may not be following. So, today we have Chris Murphy who's from the fedora workstation working group and he's going to talk about the butter fs change the butter fs or btr fs or better fs however you want to pronounce it is the new default file system for desktop. Operating systems in fedora 33 going forward but our 33 beta it's out already fedora 33 final in just a scant few weeks. So, this is a pretty exciting big change. And I know we've had a lot of enthusiast attention around this like people who like to run Linux for fun on their systems are very interested in butter fs. So, it's gotten a lot of attention so I'm sure there'd be a lot of interesting questions in this. And so, Chris doesn't have slides but he's kind of talk about it and I guess we'll kind of talk about where the idea came from to do this switch who is behind it. How, how it's going so far, and what we expect for future plans, I guess that's the basics. And with that, I'll turn it over to Chris. Thanks Matthew. So I've been a fedora user since maybe fedora 11 and active in mostly fedora qa since around fedora 16. And now I've been on the workstation working group for a bit over one year. And there's a little bit of butter fs history in fedora there's a long arc and a short arc to the story. And the summary that I'll just give away from the outset is that this is a surprise, and this is not a surprise. So, both of these are actually true. The long arc to the story begins in 2011. FASCO actually approved butter fs as the default file system way back then. And for various reasons it was set aside and then began a long period of butter fs isn't quite ready and things simmered for a number of years. But quite a lot of work was still going on behind the scenes ensuring that fedora continue to adhere to release criteria standards where butter fs has been a release blocking file system for about eight years or so. And then as part of fedora next the workstation working group had said that file system decisions were up to the working groups and a document that the workstation working group put together way back when said that butter fs would be the default at some point. So, you might jump into some background on that. Sure, go ahead. Yeah, just the fedora next thing was a strategic initiative we did about five years ago to kind of help chart the future path of fedora and one of the things we did was say that different. The groups that put together things like fedora workstation KDE fedora server are, you know, top headline additions would have more autonomy in what they do in each thing so that we didn't have decisions made for the desktop. People who are running fedora on their servers saying this is a desktop oriented decision. It's terrible for our servers. You can't do it. We wanted to and conversely, you know, server oriented decisions. People on the desktop saying why are we have all this server stuff affecting what our defaults are so we wanted to split that up a little bit so we could address these different use cases differently. So that also has allowed us to experiment with a lot of things. I think it's, I think it's been successful. So that's kind of where that, yeah, changing decision making process goes to. Sorry, gone. Yeah, good agreed with all that. So then about three years ago, the working group, the workstation working group started looking at changing the default file system. So that's the long arc of the story that goes back some years. The short arc sort of begins in parallel with that. It's compatible with that story, but it does have a different origin point, which is just this year. Earlier this year we took up the question again of default file system to solve a particular problem. That is somewhat common for developers running out of space on either the root file system or the home file system in the current or former depending on your point of view default partitioning layout and file system set up root and home or separate file systems. So sometimes people would run out of space on one or the other while there was still space left over on the other one. So I've been trying to get a particular but I ran into that a lot myself. Yeah, some it is interesting that some people run into this problem more often than others. And it is it is use case specific. So for those people that have that particular use case they run into it often, and then other people go what, but that's sort of a recurring theme in the default file system conversations is folks will say well what about and then they'll give their example and everyone will nod their head and go. Yeah, I can see how that could happen but then lots of other people guys never heard of such a thing. So, a lot of this has been juggling a variety of use cases. And trying to compromise. There. No, you were talking about the short arc. So, back to the short arc, I've been trying to get a particular butterfs developer on board helping us, Joseph boss. Joseph actually used to work at red hat at the time when for had originally approved butterfs as the default. And he's been one of the original butterfs developers for over a decade. So, Gumpa, who got multiple parties to commit pretty much all at the same time earlier this year, and they put together a presentation for the workstation working group that was, it was compelling. Because it turns out it wasn't any one specific thing that was the most compelling, but just the sheer number of nice to have features, perhaps overwhelmed the decision making process, and made it clear that butterfs was the way to go. So it really did happen fast. And it has been a long time coming. So it's both of those story arcs, I think are true. And that's pretty much all I have for the for background. You want to talk about some of those features that you found compelling the workstation working group on compelling. Are we seeing those features now or are they kind of like our people going to get them with the door 33 or the things that are enablers for the future more in those features. I think all of the above. The most immediate issue that is set aside that is resolved by the change is that users running out of free space on either the root file system or the home file system won't happen anymore. And that's because we will now have the one big file system approach. So there will be one big butterfs file system instead of separate file systems for root and home. There will be soft barriers to continue to separate root and home. So that folks that have a particular use case of reusing their home directories for reinstallations, you know, if you're reinstalling the operating system, you want to be able to reuse it. That is still possible to do with an account of the installer and butterfs. The two of them for a long time have had really good support for that particular use case to reuse home. Okay. Yeah, that would be top on the list. Go ahead. Because that's one of the things that people work the most is about how they're going to recover the home in a fresh install if they don't want to lose their data, you know. And the other question is that a lot of people is saying that normal comments like du or df works well with butterfs. So I, if I remember correctly in a magazine article, there are some kind of comments specifically for butterfs to check that this usage. So how, how do you think people are just going to react that they can type normal du comment. So, yeah, it's a good question. What I would say for the most part is on the desktop with single devices. We should reliably get information from du and df. There's a possible confusion where df shows the mount points for root and home because they're on separate sub volumes, but they show the same size. That is actually true, but it's confusing for those who are used to root file system and the home file system being separate and having separate specific size allocations. So even though that's different with the butterfs in the layout that we have, df is still telling the truth. And so does du. Where we start to run into possibly some difficulties with du. And I'm going to stand way over my skis when I say this, because it's a bit of a jargon territory that I haven't really sufficiently done any background explanation on. But we have this concept in in butterfly snapshots, ref link copies or efficient copies and deduplication. And those three things create a thing called shared extents and du is not aware of shared extents. So in the case where you want to know how much space is uniquely or exclusively taken up in a particular directory or sub volume, then you do need a butter of a specific du for that. So it's just a matter of perspective. The conventional du isn't telling a lie, but it isn't telling the entire truth. So there will be some of that as time goes on, we'll have some of those kinds of confusion, minor confusion, I think. Let me quickly explain those commands for anybody who doesn't know what they are, but hasn't been scared away yet. df and du are two command line tools that show you use disk usage on your system. df is, I think of as disk free, it kind of shows you the list of every file system. And it tells you how much free space is on each one. And I'm looking on my Fedora 33 beta system right now and I see that the slash file system and the slash home file system where all my user data, my personal files live both show up, but they both show that they're the same size and they have the same amount free. So that could be confusing because it now it looks like I've got 54 gigabytes free on two, two separate file systems. But the fact is really that's the same, same shared pool and that's not obvious at all in that tool. So that could definitely confuse people that's going to be aware of the other one is disk usage, disk used. And that actually tells you the actual like amount of physical block space that's taken up by a file as opposed to its nominal size. So a file could, for example, if a file contains a lot of zeros in a row, Linux is smart and doesn't actually take up the disk space with all those zeros called a sparse file. So a file might look like it's two terabytes, but only actually be, you know, a megabyte or so if it's got have has that sparse feature. So do you will tell you the actual disk space used normally. But maybe not on butter a fast. So I guess the question is, is there a reason we don't extend the standard d you commanded Fedora or replace the d you command with the butter fast war aware one, or what what is the command I should use for that instead, if the you isn't going to be helpful. That's a good question. In my day to day use of butter fast for many years now. I pretty much only ever use df and d you. I rarely use the butterfly specific ones unless there's something particular that I want to know that I'm aware of that is uniquely butter a fast. So those things tend to only come up with regards to snapshots, ref link copies, or do but I'm not doing any do for the most part on any of my system so it's just the first two. Yeah, so never the last do you just does still tell the truth that, you know, if I point to a directory or a sub volume, it's telling me what's in there it's just not separating out what is uniquely and only in that particular. So, if you then delete stuff from there, you might be surprised to find that you haven't actually freed up space after because it's correct. Yeah, and that can happen with hard blanks as well right so that's. Yeah, maybe. Yeah. Okay, so basically what I'm hearing from this is, if you are aware of storage or using it to do fancy things. You will also need to be aware of the tools to understand this which seems fair. It seems like normal users probably won't get tripped up for this very much. Unless we start doing fancy things which uses those things in the background without people being aware which might be some of our future plans and then we might need to keep that that in in mind better. Or sure, and that was actually part of a big part of why the options in the feature proposal there were three options. None of the three are in fedora 33 are being used in fedora 33. So, one of those compression by default has the sort of a effect where it just adds to the potential confusion of how much space is really being used on disk because some of the tools for example DF will take compression into account. Whereas DU will not take compression into account. Yeah, so folks will will have to do a little bit of experimentation. And if they're interested in this sort of thing and we'll take the feedback and see what sorts of things make sense and you know there's going to be another long arc to the story. As people actually use it. There are so many ways that butterflies is being used in production that the original designers weren't really expecting and that's in a good sense that it that it's being found to be extremely helpful in ways that that weren't anticipated. But yeah, in all the file systems are constantly evolving. They're constantly getting improvements, enhancements and features and butterfests is no exception. Cool. Anyone else have questions. I have another word. I want to give chance to other people to ask questions. Go for it, Edward. All right. Okay. I run for federal systems at home and I'm really interested to know if it's recommended to fresh install or is recommended to convert my current home from those systems to butterfests. Because I am a word that is there is a conversion tool for EXT for to butterfests, but I'm not sure if it's better to convert it or back up a fresh install and revert my backup. Yeah, it's a good question. There is butterfests convert tool and it works to convert X4 to butterfests or riser FS incidentally to butterfests. Don't do that. No. Hi. What I would say about it is the this is an upstream supported tool. The the the gotcha is that you know file systems are actually really quite complex and perhaps one of the most non deterministic things that we have in computer systems today. From the moment you start using them, they start taking multiple quantum forks in probability, you know, how they age and how different they are from each other. So the real world testing of the butterfests convert tool probably doesn't have quite enough for me to highly recommend it. I personally haven't run into failures with it, but I know that some people have run into problems with it. At least from a fedora perspective, I would say you should clean install and that's the expectation is that you will clean install so you'll back up your home directory and completely wipe away the previous installation or at least free up enough space to create a new installation with butterfests. And maybe at some point, you know, we get some bandwidth with users fedora users community and developers to maybe do a butterfests convert specific test day and get a bunch of real world aged X for file systems and really hammer on on making butterfests convert significantly more reliable. Yeah, I wouldn't say it's unreliable now, but it's sort of a yeah. One of the gotchas that you'll have if you're doing a butterfests convert in fedora is you won't get the fedora's installers layout, you don't get the separate home and root sub volume layout. You get a single sub volume hidden sub volume with butterfests convert. So it's pretty much exactly like one big X for file system. There's no sub volume layout at all that's created as a part of butterfests convert. To be clear, my interjection was don't use riser of s not to don't use that conversion. I don't know much about it. It's history will say in the meantime. Yeah, so my systems I did the reinstall because that dividing merging the file systems together home system was kind of the key reason I was interested. But people who are upgrading will stay on X poor, which to be clear is a perfectly good fine file system with a lot of history and reliability so it's not like your world's going to end for you if you don't convert but if you if you like to live on the edge converting seems like an interesting thing to do. And if you don't have a huge amount of data, could you do the convert and then create a home sub volume and then move things over. It might be worth writing an article about like how people could do that to get their systems up to date, maybe. Yeah, most things with butterfests are can be modified after the file system has been created including sub volumes and we can use reflect copies to efficiently move things over. So you don't have to recopy your data into the new sub volume. So yeah it is possible to do it it's a little say again. Does that even happen. So if it converts them with separate basically, as I understand that the conversion will be there'll be a separate home volume and a separate root volume that's not a sub volume. Can it rough copy between those two even or does it have to actually copy. So there's two parts to the question maybe if we're doing a conversion of the current layout where root and home are separate file systems. If we do a conversion, there's still separate file systems those would be two separate butterfests file systems. And in that case, there's no such thing as a as a roughly copy just like there's no such thing as a as a hard link. We can do roughly copies between sub volumes on the same file system. All right, cool. Sorry that might be in the weeds a little bit but interesting. Next topic is how are things going with testing you've talked about test days a little bit here how how tests gone so far. Do things like have we come across problems have we come across. Yeah, how widely tested this is how confident should we be at this point. I'm reasonably confident at this point in terms of the numbers I don't actually have solid numbers. I believe it was a little bit over 100 testers and over 1000 tests run. And that was the second test and it wasn't really a test day it was a test week. So the numbers are a little high probably because of that. Other than I don't have really strong numbers. I don't know that we have. Numbers generally in Dora. Do we. Numbers are hard. Yeah, numbers are hard. I have not gotten a lot of. Oh no this killed my system on the the. Yeah, anecdotally yeah we have had we've had one of those and it and we tracked it through it actually ended up causing two problems. For this user and it turned out that it was. Bad Ram. So I would not recommend butter fest as a memory tester, although it does. It does seem to have that side effect. I do have some numbers on the butter fs test day that I can share in the chat over here. And this is the test on the YouTube actually so. Can you send that to me separately and then I'll make sure we get it. Send it to me and Edward will make sure it gets on the on the YouTube. Links as well. Sure yeah and I can read them here briefly it looks like for the test week that occurred. August 31st through. September, let me see here from August 31st to September 4th. There were. 28 red hat contributors and 77 non red hat contributors for a total of 1020 individual tests run so that's actually. Looking at this probably the best outcome from a test day that we've had in quite a while. Yeah, I think a lot of like I was saying there's a lot of enthusiasts enthusiasm around this. So it kind of brought out people's interest in helping test out so that's that's cool and valuable in itself. Also, I would say that in on the QA side. They're quite a lot of tests that happen every day. They're automated tests. But they, you know, we've been doing automated tests on. They use butter of us as an install target for quite a long time. I mean, and since. Probably day 1 of open QA, I'm not sure how far back that goes probably. Maybe 5 years, maybe longer. Cool. Any other questions about the current state of things. Then let's talk about the future. I have a question. Yes, it's already ship in beta, right. Yes, yes. That's the current status is ship in beta. Yes, exactly. And we have quite a few people running the beta already people are excited. Yeah, so some of the future things we're looking at against our compression. Maybe some cool backup stuff. What about encryption? Encryption is, yeah, interesting topic. So right now, but our first can of course run on. DM crypt looks slash DM crypt, just like the same encryption we use right now for currently exactly what that's for. Sorry. I meant to mute my button, but didn't get to it and fine. So there is a plan and some work that's going on for. There's work going on for native encryption in butter fast and the native encryption uses. The kernel has something called FS crypto. And this is part of the work that was done to get X for an F2FS native encryption and they pulled it out of X for and F2FS and made an API kernel level. API for this that exports still uses an F2FS still uses and butter fast will use. So we are going to get native encryption. At some point, I don't have a timeframe for that. I, you know, I think that anything development wise in any file system is. An idea rather than a firm plan and that may also be true more so with butter a fast. The idea is that these patches should land by the end of this year and it's going to be a resources. Competition to make sure that the FS crypto folks evaluated and and review it and give feedback and, you know, whenever it gets merged and then. It'll need to be in upstream production of probably before we see it in a mainline kernel. So if I had to guess. You know, maybe about this time next year. We know if their plans for that to be able to be added after the fact like on an existing system. Yeah. Yes. Yes, sure. So the reason why I'm confident in saying that is because that's already the case with compression and all the other features in butter a fast. There are very few things in butter a fast that require. A new creating a new file system. It's possible to just set a feature flag for most of the features that have been added after the fact. An example of that is Z standard compression. And the pathway the code path for the native encryption is going to share the same code path is compression. So what I expect is that there will be a feature flag that gets set and that feature flag will simply tell the kernel. You know what the minimum kernel is needed to be able to read and or write to the file system. So maybe that there will be old kernels that can read the file system but not write to it. If it has encryption enabled because it will be possible to have sub volumes that are not encrypted and some sub volumes that are encrypted. So the ones that are not encrypted could be read by an older kernel and then the newer ones are I'm sorry the encrypted ones would need a minimum kernel version that supports the native encryption routine. So yeah set the feature flag create a new sub volume and then whatever the commands will end up being to set a key. And then encrypts the sub volume. So that's interesting, both for people who want to change to that later, but also, you know, when I got my new Lenovo think pad with Fedora pre installed on it, it was awesome to boot it up with Fedora pre installed. And then because I wanted to have it encrypted the first thing I did was reinstall so being able to instead have at that first boot say, do you want to encrypt your system back here give a passphrase. That would be awesome. Yep. And it's interesting actually also you're talking about the compression flags. So, yeah, if, you know, there are different algorithms for compression that have different trade offs and different advantages. So, if I decide one, like I'm not necessarily committed to that forever. If somebody comes up with an awesome new compression thing and gets that into the kernel I could actually just change and then new files would be compressed with the new thing and old files eventually converted over as they get access and rewritten. That's pretty cool. Correct. Yeah, the compression is at a. I'm not sure exactly what to call it. I'll just say it's at a block level. There's a block size for compression of 128 kilobytes. And so a given file plausibly will have some blocks that are compressed and some blocks that are not compressed. So, but our fast has the capacity on a per block level to have different, whether compression is on or off or what algorithm happens to be used. So, all that's baked into the metadata for that file, and it doesn't flinch. Yeah. Oh, so you could even have a file that has multiple compression for different blocks that's, that's disturbing, but all right. Yeah. This is super, super nerdy. Can you mark certain files as wanting to have bigger blocks compressed or is it always just the same block size. It's always the same block size. That's not exposed to the user at the moment. It's 128 kilobytes at the right now. There are probably some reasons why that's the case. I'm not sure what they are. I have some use cases from an old job where that would have been nice to set, but they can be smaller to me. They can be smaller than 128. So, Larger is what I want. Sometimes it's much larger blocks. You can get a little better compression, but yes, that's okay. That's probably again, not important to most people. All right. What other, what other future plans are there that you want to talk about what else and what else should we know as the council. Um, yeah, there are, there just, it's like this long list of features and what ifs. One of those that we're kind of looking forward to is, is while we've talked about encryption, we've talked about compression. One that I'm particularly excited about also is a little bit nerdy. And that is the resource isolation that we gain with Butterfest on the IO side. So resource control is a really interesting, highly technical. I understand maybe 1% of it. But there's this competition for resources. The resources being memory, CPU and IO. And using C groups of E2, we're able to provide upper and lower limits. So it's possible to assign a minimum IO, a minimum memory and minimum CPU to, for example, resources that we care about like the desktop itself. And say that, that those resources, that minimum resources guaranteed no matter what else is going on on the system. So if you have a particularly aggressive process, it could be a web browser, it could maybe with a runaway tab and script in it, it could be a compile that you're doing. We'll be able to do a better job of, of reserving a minimum amount of resources for the desktop to make sure that it stays responsive. And Butterfest is a part of that plan. It's not a required part of it, but it helps make that experience better. So that's something that I'm looking forward to and that's ongoing. I think some of that work has been happening in Fedora since Fedora 30. And now that we're moving to Butterfest, we'll get the IO part of that figured out. Cool. Yeah. So at some point, you know, the, the feature that we, we added at Fedora 32, nothing, nothing ever stays the same in Fedora it seems, at least not for too long. And Fedora 32 was early OOM, early OOM, OAM stands for out of memory. So we have a user space out of memory manager, very simplistic that will kill off wayward programs that are using too many resources. And it's likely that that will be set aside in a future, near future, Fedora, because of this resource control work, because we won't need to be killing off programs. We'll just say, hey, you get fewer resources. You're just going to go really, really slow, but the desktop is going to still be responsive and you'll be able to, you know, use applications. And then as a user, you can decide, oh, that's like whether or not you want to kill that or not. Yes. Yeah. There may be some more intelligence. There may be, we're looking forward to system D, OOMD. That's a little bit off topic for this, for today, but it loops and feeds back into the resource control, which Butterfuss is a part of. So there will be another, another round of features, as is always the case in Fedora. Cool. Well, thank you very much, Chris. I really appreciated this. I learned some details that I hadn't thought about before. So that's cool. And it's good to have those because there's a lot of enthusiasm around Fedora 33 and the interviewers are all asking me about Butterfuss. So I hope other people watch this video and learn things. And I also learned myself so I can help talk better about the work you've done. And thank you for this and thank you for all the effort you put into making this happen. You're welcome. Thank you. And thanks to the Fedora community for making it possible. Of course, that's always the case. Bye, everybody. Bye.