 Hello all. My name is Neil Gampa. I'm a Fedora project contributor and I'm here to talk to you about Butterfus, the Butter of Fedora. So, a little bit about me, I often call myself an open source advocate and a professional technologist. I'm a contributor and package maintainer in Fedora, OpenSUSA, I'm a member of the Fedora engineering steering committee, Vasco, for those who was more plugged into Fedora and knows what that means. I'm also a member of the OpenSUSA board as of January 1st. Yay! For my day job, I'm a DevOps engineer at Datto. I'm a developer and maintainer of package build and release pipeline systems using OBS. I'm a package manager and back porter just across the board. You know, that's sort of the thing I do, because I think that's pretty much what everyone in a corporate environment does for this sort of thing, but now on to Butterfus. So what about it? So, from the Butterfus Wiki, Butterfus is a copy on write file system for Linux aimed at implementing advanced features while focusing on fault tolerance repair and easy administration. Lots of words there, a lot of words there. Basically, what it is, is a file system that implements a optimization strategy where modifications of the file system are not written in place, but they're written to a new location so that you can have some kind of atomic integrity with every operation that happens to disk. It also makes it possible to preserve each instance of the file system and move back and forth through them. You know, with additional features like snapshotting, whatnot. It's not a replacement for backups, but it does provide some safety net that's not possible in traditional file systems. So, it's a 64-bit file system, 16 exabytes, I think most Linux file systems are at this limit. You're never realistically ever going to hit this. At least I hope you're not. If you've done something crazy to do this. One, I want to hear from you because I'm just morbidly curious. And two, oh my goodness, please don't do this. There's a whole bunch of other features in here, you know, space efficiency quotas, snapshots as I mentioned earlier. There's SSD awareness and SSD specific optimizations, integrated RAID, transparent compression, seeding for my other file systems, and there's a lot more to come from there. So sub volumes and snapshots are probably the most user-visible feature. Sub volumes are subsections of a volume, hence the word sub volume, they'll be independently managed. This is useful if you want to have different snapshotting schedules for portions of the volume. These snapshots are instances of sub volumes that can be preserved and with appropriate tools and configuration, you can do fun things like time machine style data management and recovery and save yourself from bad software installs, upgrades, catastrophic failures like power cutoffs and things like that. They're developed by a number of, you know, members of the community, Facebook and SUSE are very notable as a lot of the leading drivers towards a lot of the feature development. Western Digital does quite a fair bit. I think they recently contributed a feature for zoned block devices that just got merged for Linux 512 and Oracle, the progenitor of this. They do a bit here and there. They're still actively doing some stuff and they offer Butterfests as part of Oracle Unbreakable Linux or Oracle Linux. I don't know if they're even using a middle word anymore, but they do it. And there's a large user community as well. Of course, you all know Fedora has been using it since Fedora 33, but also open SUSE has been using it since 2014. They introduced it with SUSE Linux Enterprise and open SUSE Leap 42. And there's others such as Synology and Netgear as NAS appliance vendors that actually use Butterfests by default for these things. Rockstore is an open source NAS that was originally based on Fedora, is currently based on CentOS and is moving to open SUSE due to CentOS not having Butterfests in CentOS 8. And of course, Facebook, who uses it on their enormous fleet of systems and containers and all that stuff for their infrastructure. So why would you use Butterfests? It's a file system that's developed within the mainline kernel. So you've got absolute guarantee that it's going to at least be available at work. It takes advantage of the facilities provided in the kernel to be more efficient with operations and stuff. And Linux distributions basically, they have support out of the box with very little effort. It doesn't take much to start taking advantage of an advanced file system on Linux, because Butterfests is just part of Linux. So Butterfests in Fedora, obviously that's that's what a lot of people here are want to hear about, right, because hence the title. So the current state based on Fedora 34, which is coming out in a couple of months. It has been configured to install non-server variants with Butterfests. The disk images of the desktop variants, for example, for ARM will use Butterfests by default. They provide Butterfests based images. The boot mount point is still X4. It is possible to configure it with Butterfests. There are some caveats. If it works, I use it myself that way. Z-standard compression is on by default. This is new with 34. And disk encryption uses locks. So that means you can only do full disk encryption. You can't do partials right now. Now, going forward to Fedora 35, there's a whole litany of things that you'll see coming in the near future. There's a lot of things that you'll find live native disk encryption that's pending some upstream work. It'll actually work. You know, you don't have to reinstall to actually encrypt the system. Full support for Butterfests and OS build. We have basic support in place. There's just missing subvolume creation. Some features built around factory reset and recovery for OEMs. There's a lot for boot to snapshot and system rollbacks. And also, you know, a pet feature of mine, working on building atomic updates with transferable snapshots built on Butterfests using a more traditional technology stack package kit, microDNF, that sort of thing. And for those who would prefer to use that kind of technology stack for providing a stronger semi-immutable infrastructure based on Fedora. So kind of my future hopes and dreams here. I believe strongly that Linux has the potential to offer a superior storage experience for desktops and servers using Butterfests. The unified storage management interface and comprehensive feature set for all use cases enables lots of things like I personally use it for not just my desktop but also for doing container stuff, because Butterfests in containers, and Butterfests is the backing storage for containers actually makes it so that you can have a really slick deterministic experience. It determines the sense of IO performance. And you don't have weird, crafty things happening or unexpected IO behaviors because of, you know, quirks and overlay FS for example, and it's very easy to make it all the functionality. For example, for both an unprivileged and a privileged use case through very standard APIs and interfaces, which is makes it so that you can have, you know, a very strong guarantee of performance, because it's actually just part of the normal storage. And you can do things like backup container instances by sending receiving, you know, the data that's in Butterfests sub volumes and stuff like that. There are some resources here. Butterfests wiki opens is a Butterfests document just pretty good. Dusty may have another for Doran who used Butterfests himself he's actually got a cool setup for doing full system snapshotting and rollbacks using the snapper tool with Butterfests. And there's a fuller longer talk with more details from Nest last fall, as well as, you know, some YouTube videos and articles from Facebook's own descriptions of how they use Butterfests and if you want to kind of help with implementing a lot of these features and moving Butterfests forward in Fedora, we have a parallel project where we keep track of all this stuff and move forward and try to move forward implementing cool features that can benefit people in the Fedora community, leveraging the Butterfests technology stack and help drive innovation upstream for Butterfests features and technology. So any questions. Yes. We do have one question already on the Q&A session. And the question is, as someone who did a fresh install of Fedora 33 instead of upgrading for default. BTR FS, are new features on Fedora 34 or 35 going to happen with an upgrade, or do they require a fresh install? So, so for most, okay, I'm hearing. For most cases, doing upgrades will not automatically activate these features. We have ways to offer people who have upgraded from 33 to 34 and so on. We can give them pads to actually apply these changes to their systems themselves. It's kind of dangerous to muck around with the file system directly from the package manager level. But you can do the changes yourself. We could even provide scripts and tools to just automate that process of applying these changes after the fact. So yeah, it's possible it's just, it's just not something that I want to, I'm not comfortable considering that in scope for changing defaults in Fedora, because the interfaces for managing this stuff is super primitive and I don't know how people use their computers and I don't want to guess. All right. Thank you. Going to the next question. We actually have a lot of questions. So, yeah, that's great. The next one is, is the disk full issue resolved yet? Three to five years ago, I had issues with disk full, whereas there was still plenty of free space. I had to run some admin commands. Yeah, so this is one of those memtastic type of things where, because of the nature of the butter fs file system where it's copy on right. And because we do, we do things like by default we have a backup metadata tree inside the file system so that you know if you have issues then you can you can replay based on that. It's less predictable in terms of how much space you have for more advanced cases like when you're doing raid and stuff. For the single disk case and a lot of the simpler rate cases like raid one and raid zero. This is a lot more predictable now and this is all know it's very unlikely that you're going to encounter these kinds of things. I certainly don't encounter it on any of my machines and I've been running butter fs on hundreds of VMs and dozens of physical systems for about five, six years now, and it's going pretty well. So like these days, at least in the last two or three years, it's gotten a lot better because there was a huge focus on on resolving those kinds of UX pain points. The next question is, when can we expect mech OS time machine like experience in the non files now tools. Oh man. So it's funny that that's brought up because Sun Microsystems developed this feature for genome to like what I want to say like a decade ago for using in Solaris. And the feature never made it into genome itself. I don't remember exactly what happened then. But I would personally love to see this happen. Now that we have distributions like Fedora and open Susan now adopting butter fs by default and arch distributions some of them like Manjaro, Garuda and such are either considering or are doing butter fs by default now. I would love to see this implemented. I think it's more of a question of reaching out to the genome folks. And trying to get this to be a feature that they're interested in shipping. As of late Nautilus has typically been losing features rather than gaining them so I don't, I don't know what their philosophy here is about how how Nautilus is supposed to work, but I would personally love to see this happen. Alright, next one. If the RFS is default on Fedora and considered stable. Why did real team kick it out of real eight. Okay, wow. Okay, so I'm going to preface this by the fact that I'm not a red hatter I can't really speak for red hat because that's not employed by them. And even if I was I don't think I'd be allowed to. But I can offer a supposition based on public statements in the past, and that is red hat has no employees that work on the butter fs file system anymore their last employee that worked on butter fs directly quit in 2012. So they don't have any team members of the team in the file system storage group that work on butter are able to work on butter fs some of them don't even like it architecturally because they have different opinion of things like they prefer LVM and XFS and stacking layers rather than having an integrated intelligent layer. Whatever like that's that's sort of that's sort of where the fallout happened so if red hat had employees that were interested in working in butter fs upstream. I think that that would change. I don't know if it will anytime soon I personally hope it does. The butter fs developers are probably some of the nicest folks I've met in the Linux kernel in the Linux kernel upstream. And I think that it would be great to see how there's working in butter fs upstream again. Okay. Thank you. So, we have five minutes left. So, to the next question, which is, why doesn't fedora 33 server default to the TRFS. Okay, so a couple of reasons why I didn't change it for why we didn't change it for the door for fedora server. The first was, I had my hands full personally trying to just do the desktop stuff and the server stuff is a whole another can of worms. To there are people, or at least there were people in the fedora server team that wanted it to look more like red hat enterprise Linux. And so that was, I kind of decided to let let sleeping dogs lie there. I figured that showing it off in desktop would make people more interested in having it in server. And, and finally, it just time. It's, it's quite hard. It was quite hard for me to change everything in fedora 33 for just the desktop flavors. I had to go through and change a lot of different image tools a lot fix things and anaconda and stuff like that. Truth is fedora server works great with butter fs. It's just not the default right now and maybe it will be in the future. I hope so personally. All right. We have three minutes left. I think we can go for one more question. The question is, what are the performance improvement plans for butter fs. As much as I love it. It's still has superior IO birth. This can really be felt with, for example, next cloud or MX. It's not as bad on an SSD. So, so that's a great question because every file system choice has trade offs. So because x4 is a simpler file system. There's less operations happening in the IO path to disk and if you're really hammering it. This tends to be a little bit of a problem. Some of this is going to be mitigated in fedora 34 with the transparent compression being turned on across the board. There's also some other things that are going on in the upstream to improve the performance of processing IO operations. There was a refactor of the infrastructure in how I operations are processed. That landed in Linux 510 and was further improved in 511. So you'll just naturally start seeing these improvements just come down the pipeline. I believe that with 511 we're looking at. At least with 510. I remember looking at somewhere north of 50% performance improvement across almost every workload. And in some cases, almost 100 to 125% performance improvement, which is freaking awesome. And that's going to continue. I believe there's more improvements coming down the pipeline for 512 that improve it by another further 20% depending on what kind of IO demands you have. So I think it's just one of those continuous improvement kind of things. So I would just say check back frequently and see how it goes. Worst case scenario, if you make the folders that next cloud and Emacs operate in as mark them as no data cow. I get the same performance as X4 maybe slightly faster because it's a more optimized file system code. But yeah, like that's, I think you're just going to naturally see improvements across the board over time.