 Hello, my name is Joseph Bassack. I'm a software engineer at Facebook and a longtime kernel developer and maintainer of ButterFS. Today, I'm going to talk to you a little bit about deploying ButterFS at Facebook scale and some of the lessons we learned and the variety of ways that we use ButterFS in our infrastructure. So first, I can go a little bit of background on Facebook's infrastructure. There's quite a bit of different use cases. As you can imagine, Facebook is a very large company with a lot of different moving parts. So in order to help understand the presentation better, I'm going to kind of walk you through some of the basics of how we have things set up, some of the terminology that we use inside of our infrastructure. And so just to help make it easier to understand the different terminology that I use throughout the presentation. Then I'll go over some of the big use cases that we have for ButterFS. Obviously, every file system has its own pros and cons. This is not to say that ButterFS is perfect for all use cases, but there have been many use cases that we found it to be very valuable. And so I'll kind of describe those in detail and the things that we ran into and just some interesting tidbits and places where ButterFS really helped us versus places where it wasn't so great. The other thing that I will talk about is kind of the things that we did. Interesting things that we learned along the way, things that went really, really well. And also things that didn't go so well and some of the future work and just the direction that we're going and the things that need to be dealt with. So the first is just some background in Facebook infrastructure. Generally speaking, every service that runs inside of Facebook is contained within a container. This is to make it easy for us to migrate. So we have, obviously, we have a lot of machines, more than most. Because of this, it's difficult to manage. We have so many machines, it's impossible to manage in any sort of unique way. So we try to make sure that our machines are relatively consistent across the entire fleet. And thus, the services need to be able to move to any machine at any given point. We do things often like bring down entire data centers to validate our disaster recovery modes and that sort of thing. And so part of what allows us to do that is being able to containerize and ship services around anywhere within the fleet. When I originally joined Facebook in 2014, 2013, a long time ago, the every service had its own dedicated set of machines. So the web services all ran on web service machines. And if you wanted more machines, you had to go to the capacity people and say, listen, I need X thousand more machines or X hundred thousand more machines or whatever because of peak loads. And that's the other thing that's important in Facebook. Not everybody's on Facebook all the time. We have peak times. And so we have capacity set aside to handle peak traffic. But generally speaking, we're relatively unloaded. So anyway, before we had dedicated service, dedicated pools, and so there weren't really containers. It was just the web machines got their web, new versions of HHVM just pushed directly to them. HHVM would restart and they'd keep on going. Same goes for database, memcache, that sort of thing. As we have invested more and more time and energy into resource isolation and containerization, this has changed. So now our largest pool of machines is called TWShared, our container service is called Tupperware. And so the largest chunk of machines that we have is TWShared. And basically every service that doesn't require dedicated machinery exists in this pool of machines. So the web services, messenger services, that sort of thing, before web was on its own set of hardware, now it just exists like any other service inside the set of TWShared. Machines, there are some services that don't do this still like the database, things like MySQL, we don't containerize that. It's, they have their own machines with a special hardware, they get upgraded like any other kind of like small business sort of service that you would expect, upgrades, you know, restarting that sort of stuff. Not a lot of mobility, but that represents a very, very, very small percentage of the fleet nowadays. Nowadays everything is container. And even if you still have your own set of dedicated hardware, like our build servers, it's still built on this Tupperware platform where the service is containerized and ships around. This has obviously gone through a certain amount of evolution over the years. The original version that this looked like was a ButterVest loopback devices. So we had a base image of CentOS and then this base image would be generated with the service specific information and code. And then we would compress this all up and ship it around. And so any, so any box running a service would often have a three or four or five ButterVest file systems running on loopback device within it as part of the container. As we've made improvements, as we've done thing, this existed because as I said, we have a lot of machines and reformatting all of those machines is kind of a big task. And they were all running with them. EXT4 is the root file system. And so early on to validate using ButterVest in this way, it was simpler to just ship loopback devices as ButterVest around onto EXT4 root file systems. And then as we converted more and more machines to ButterVest root devices, we were able to take advantage of more of the interesting features of ButterVest. And just as a quick rundown of ButterVest, in case you haven't paying attention to Linux file systems recently or for the last 10 years, ButterVest is kind of an iteration on Linux file systems. We have built-in snapshotting and compression and individual volume management. So those are the sort of features that were very attractive to the various teams building out the container infrastructure inside Facebook. So one of the very first use cases where ButterVest was sought after for one of its core features was our build servers. Obviously we have a lot of code inside Facebook. This is not only like the webpage that everybody views of Facebook, Instagram, et cetera, but it's also the Android apps. It's all of the infrastructure code, all of Tupperware, any of our database, software, every service has it's in these various repositories. And because we have earlier thousands, but now tens of thousands of engineers working at a given time, we need to make sure that nobody steps on anybody's toes. And one of those ways is we have a relatively robust build system. So nobody ever commits directly to the repository, you write your code, you send up the patch, get your patch reviewed, and then you submit it to land. Before it lands, it has to be tested. So this is where we take, before what happened was we would have a local checkout on a machine of a relatively recent version of the repository, say it's a couple of hours old. Then what we would do is a Git checkout or a Mercurial checkout where I just use hard links into a specific directory for this patch that's being submitted. The patch will be applied, make sure that it applies. Then it would be built, make sure that it builds. Then the tests would be run, make sure the tests pass. All of this happens. And then that whole directory has to be removed. And these are hard links. So generally speaking, that's metadata operations for every file that you didn't touch. And then any file that was touched, obviously had to be removed. So the creation step was relatively quick because Mercurial and Git are pretty good about setting up their repos that are copies of each other. But the teardown was relatively slow, two to three minutes. In some of the cases, if you touched a core part of the infrastructure code, we have all of our, for example, we have all of our C++ libraries, Folly, for example, if you're touching anything in Folly, you're gonna end up with a lot of files changed. And those are like five to 10 minutes to remove those repositories. And while it's doing that, these boxes, there's only so many tasks and build processes these boxes could do at a given time. So if they're stuck doing removes for five to 10 minutes, that really kind of slows down. So like during peak hours of the day, it would take hours for individual code changes to land just because of this. So Butterfuss was looked at initially just for some of our more problematic repos. Android, I believe, was the first one. The idea being that instead of using the Git or Mercurial shallow copy, we would simply snapshot the repo, which is instantaneous. And then do the same build or whatever, and then delete the snapshot. Obviously, from the user point of view, the snapshot deletion is instantaneous. Obviously, it's really not because there's a lot of cleanup that has to be done, but the cleanup that needs to be done in the background is significantly faster than RM minus RFing and entire directory. So the repository cleanup, like I said here, was reduced by an average of 140 seconds. That's because that's about the average time for the Android at that time. It's 140 seconds and it went down to instantaneous. So it's reduced the capacity requirements for just that repo by a third. And the big thing inside of Facebook for services are their capacity requirements. So I mentioned before, we have a lot of machines. And so as programmers, we love to use every bit of resource that we have, but these resources cost money. And so you have to justify expanding out capacity. And so when we can help services reduce their capacity requirements, that translates into like dollar value savings. And that's the significant savings that can be passed like from our significant savings from our infrastructure budget. And so that was the initial use case. Obviously, once that turned out really well, it was expanded throughout the rest of the repos. Our build servers, it sounds like one kind of consistent thing, but each repository obviously has its own tricks and that sort of thing. The PHP stuff is different from the C++ stuff, which is different from the Android stuff and that sort of thing. So it kind of takes a little bit of retooling for each repo, but eventually, like this was a good enough initial result for them, that team to invest wholeheartedly in ButterFS. And now the entire build system is built on ButterFS snapshots for this reason. The next thing that was sought after was compression. Most of these are all tiny text files that are relatively easily compressed and ButterFS or Facebook spends a lot of money on flash. And we spend a lot of time and effort trying to make sure that flash lasts as long as possible. So compression not only helped us add more build jobs per server, but it also allowed us to contain the right amplification issues that we were seeing on flash and allowed us to have our flash, extend the lifetime of our flash, which again translates directly to cost savings, which is pretty impressive. This and this build service work was all done within Facebook, myself, Chris Mason, Omar Sandoval, other people work on the kernel team, we work on ButterFS, and we obviously help and we chip in other kernel areas, but we're the ButterFS developers. This work was done without our involvement and without really our direct knowledge. This was just a team that had a problem that they needed to solve, felt that ButterFS would solve it for them, went ahead and implemented it and told us afterwards. And I'm relatively pessimistic. File system developer tend to be pretty conservative. I was very pleasantly surprised at how well it worked that they had very little issues. Obviously, things can't come up, but generally speaking, they wrote the code and they rolled it out, worked great, they did it everywhere else, and we kind of like found out halfway through the process. Dev VMs, another big use case inside of Facebook. Every engineer inside Facebook has their own or multiple virtual machines in order to do their work. This is right now, we have 30,000 engineers, 40,000 engineers, so an unreasonable amount of engineers. So as you can tell, this kind of gets a bit unwieldy. Before we all had individual machines, can't really do that anymore. And honestly, a lot of the web engineers don't necessarily need a full machine, they really just need a box that runs HHVM in the most recent copy of the website. So one of the ways to address that capacity crunch was to implement Dev VMs. And these are VMs that are 800 gig VMs for the developers and they run, they're automatically set up, they already have all the repos loaded, they already have HHVM running, they have special sandboxes set up, they have access to their current website, the current website is installed on them every four hours or something like that. So they're having the entire Facebook code base installed on them to be displayed every few hours. And the entire Facebook code base is relatively large. So again, these VMs are backed by Flash, we wanted to be able to maximize this Flash usage, so compression was enabled to be able to maintain, allow developers to actually get their work done, because the other thing is 700, 800 gig VMs seems pretty big, but when you have all of the Facebook code and infrastructure that you need to do builds and that sort of thing, you start to run pretty close to full pretty quickly, which you would think would be problematic for Butterfest given our history with tight space. But it really didn't have too many issues with that, but compression was really, really attractive because it allowed users to not really bump up against that issue too much, because all this code compresses relatively well. And again, write amplification has turned out to be a really big issue with Flash inside of Facebook. And the less we write, the less write amplification, so compression kind of solves two pretty large problems, for us, in addition to allowing users to have kind of different access to the different Butterfest features, so snapshotting and that sort of stuff. It's allowed us to do different things for how we back up users data. It's not super interesting because generally speaking, users aren't going around nuking their VMs and having to restore, but it's an interesting use case. This was probably, this actually I feel like is the first coordinated effort inside of Facebook to have Butterfest as the root file system. And it went relatively well. Again, this is just something that happens. The only thing that I can remember with the dev VMs is again, you know, space related issues. You know, like I said, we have a lot of code and some service went wrong and started copying too many copies of the Facebook code base onto all users dev VMs. And this kind of put them in a tight bind and some of their boxes, some of them would fall over. So that kind of requires some manual intervention on our part and some code changes to Butterfest. But generally speaking, it's been relatively uneventful. Another use case, this is not currently in production yet. It's something that's being explored, but WhatsApp is reworking their offline storage. The way, because of the way that messages are sent on WhatsApp when the user is offline, those messages have to be stored for later. Currently, it goes into like these tiny little text files on XFS, they've been running into some weird scalability issues there. So in addition, not only with XFS, but with their older service. And so they've been rewriting their new service and they wanted to take advantage of the snapshotting features inside of Butterfest for ease of cleanup and removal as well as compression. Because again, these messages can be compressed relatively well. So this is, they've been evaluating Butterfest for this, which has been really interesting for us, because up until this point, Butterfest has not really been on a service where storage really mattered. So like, you know, starting a service or whatever, it's not super interesting from an iOS standpoint. And you kind of copy the service on, I'm sorry about the loads into cache. That's kind of it. The file system isn't really interesting there. And you know, the build services, like obviously they need to be able to read from desk relatively quickly. But again, once it's in cache, you don't really care. So from an iOS standpoint, these are not super interesting workloads. WhatsApp cares about read and write maintenance, for example. And so this is kind of the first major storage related service that's tried to evaluate Butterfest. So it's been going pretty well so far. They've run into some weird scalability issues inside of Butterfest. You know, you're talking about eight terabyte file systems that are mostly full. And because the files are sort of small, they're easily compressed and often end up in metadata. So, you know, of an eight terabyte volume, you're talking 800 gigs of metadata. So that just ends up with like, it's kind of interesting scalability issues. And also the way, because of how small the data is and how they clean it up, it's added a lot of, it's been a real stress on the allocator because of the fragmentation levels that it generates. And so thankfully we've been able to address these issues as they come up and it's been relatively smooth sailing. That being said, like we're still having issues and they still may not end up using Butterfest, but it's been an interesting test to see that so far it's worked relatively well. Because again, I'm relatively conservative and my view has been that if we care about latency at all, read and write latency, we use XFS because, you know, Butterfest is amazing, but it adds a little bit of overhead with, you know, online, with checksumming built in and that sort of thing that often store services don't care about because they have their own checksumming. So it's just extra overhead. And then finally our shared container pool. This is probably what we are most proud of if you go to like scale or any other conference where you hear talks given by our Tupperware team. This is the biggest infrastructure team within Facebook is the TW shared and Tupperware teams. They're the ones who handle containers and how everything works within such Facebook. So as I described earlier, originally this was with Loopback devices, Butterfest Loopback devices. Today it's much, much better. As we've validated and made sure that Butterfest is stable enough and fast enough to work for our systems, we invested the effort to reformat all of these boxes with Butterfest as the root drive and that's enabled us to do a lot of really interesting things and use some of Butterfest's features to really speed up and compliment our container strategy pretty well. So now instead of a Loopback device, the Butterfest images are composed with a base sub volume that is just a standard base sentos thing, which can be snapshotted then have the container packages applied to the snapshot and then these can be sent and received like the diff can be sent into a file. The file can be sent as our incremental feature where you can say, okay, give me a diff or give me this sub volume as a like an individual file that can be applied. And so these send images are generated which are significantly smaller than the actual service and then they are sent to the individual Tupperware boxes. They also have this base sentos image and so that's why we can just apply the diff. So it's just the service. So the snapshot is received. So the initial installation of the service requires significantly less IO than like copying the packages over and that sort of thing. That also allows us to do upgrades much easier because you're talking about just changing the bits that changed and not the entire image. So, and this also like this gives us the flexibility again to easily change different services. So the base image is the same everywhere and which means that if you want to tear down the old thing you just delete the sub volume, you snapshot the base image, you apply the new image and you just keep on going. In addition, we use compression again to reduce the right amplification in the IO load and space savings, that sort of thing. For our root devices, we tend to Facebook builds our own boxes. We don't use major hardware vendors which means for better or worse, we kind of for things like this, we kind of like cheap out for root devices which means we have to work around some really strange behaviors and ButterFS allows us to kind of get away with this with the compression and the send and receive. So we're just kind of not hitting these crappy disks that much and so it doesn't really affect services too much. In addition to the interesting features of ButterFS, we've also spent a lot of time getting it to work with C groups. There is a doing proper IO control with C groups in Linux has been relatively tricky because it's never just about the IO, it's about the entire path and there's a lot of weird gotchas that we ran into, I wrote IO latency, kind of the original IO controller for C group too and then Tage and wrote IO and so Tage and wrote IO cost which is the replacement IO latency was really brain dead simple, protect the main workload at all cost. You set a latency target, if it's latency target ever gets missed, we throttle everybody else and it's kind of like exposed a lot of issues within the entire file system and memory management and IO stack that had to be addressed but a lot of that work got done and then with Tage and its proportional IO controller, we're having even better IO handling but none of this stuff works with EXT4 at all. We've tried it with Exafest and there's still some issues there and we simply just not invested the time because generally in the places where we use Exafest, it's dedicated file system for a database, they don't need to IO share. So it's, ButterFS is nice for all that's other features but it's always the only thing that we've found that works with C groups. Not saying that EXT4 is off or bad or anything, it's just we've spent a lot of time making sure that ButterFS works and currently it's the only thing that does. So just kind of things that went well. These are just interesting tid bits that we've learned along the way of deploying ButterFS. Really, really early on we tried to use or what we did use ButterFS as our cluster backing store in some cases and we had this weird issue where ButterFS was complaining about checksum errors and again, I'm very conservative. If I see ButterFS throwing checksum errors, I just assume that ButterFS was broken. So we spent quite a bit of time trying to prove that ButterFS was wrong and when I say quite a bit of time like a week or whatever, I'm like really digging into this and then we started to discover that there was a pattern to the corruptions and it turned out that whenever there was a problem with the RAID controller, when you would reboot the box as it came up and probed to the partition table, it would rewrite the GPT header to the middle of the disk. And so when ButterFS came up and cluster started going and somebody went to read the file that had data in that area, the checksum wouldn't match and it would throw an IO error. This had been happening for years. These RAID controllers weren't only just used where ButterFS was, it was with XFS and it was clearly had been silently corrupting data and these cluster things were used for running AI tests, this is where big AI data sets existed and so they just had silently corrupt data and ButterFS was allowed us to notice it, get the firmware bug fixed across all of the fleet. And so that was a really interesting thing and that was the first time that I was like, oh, maybe we should probably trust ButterFS more than I do anyway. Chris has a very different view on this, but another thing that I think is really interesting another thing was we've found several micro architectural issues through our use of ButterFS. This is less like hooray for ButterFS on its own. It's more just because of the fast number of things that ButterFS does do, makes it pretty stressful on the CPUs with the checksum and the compression and the massive amount of off load threads that we have in order to handle all of this work means that we stress those CPU more than most and it provided us with a few pretty good solid reproducers to take to CPU vendors and get them to fix micro architectural issues and these issues had been plaguing us, again, we have a lot of machines so we see a lot of problems, a lot of problems and this kind of helped narrow down a problem and playing us for years where boxes would just lock up or hang out for no apparent reason or crash in interesting ways. ButterFS kind of helped us find that. Again, because I am relatively paranoid that ButterFS is gonna make the whole world burn down and it's gonna be my fault. I've spent a lot of time diving into what appear to be systemic issues with ButterFS. Because of the size of our fleets, we actively monitor every aspect of it and so every kernel message that ever gets emitted from any machine gets dumped into a centralized database that we monitor and things get pulled out so like ButterFS images, I have a little dashboard and I can go and say, okay, this is how many checksum errors that we've had all day or this is how many warnings and this is how many Enos based things that I've seen and so one of the things that have been bothering me was checksum errors and corruption issues and I've done several multi-week investigations into things, assuming that I was going to find a bug in ButterFS and so far it's hardware is worse than ButterFS, which has been interesting and been a good confidence boost. It's really kind of dispelled a lot of my paranoia around ButterFS and things that might be broken. That's not to say that I haven't been bugs, but it's mostly all of the things that seemed like really big problems that it's just like, wow, this is broken. No, it's been hardware issues or for example, with the ButterFS loopback devices, if you, because there were on any XD4 file system if the XD4 file system got full, it would return Enos base to the loopback device which would be translated as an IO error to ButterFS and then it would freak out and so there was a case where thousands of machines ran out of space on the XD4 and but we noticed because ButterFS started throwing thousands of warnings about corruptions and that sort of thing. It turned out to not be ButterFS, it was the underlying file system running out of space and just weird error handling stuff. And finally, like I've said on several of these cases, most of these were naturally developed by other teams inside Facebook. I have never once gone to a team and said, hey, will you please use ButterFS for this? Everybody kind of knows what ButterFS does. Obviously they know that we work on ButterFS and that's a big priority for us and so many teams have evaluated it and decided that, yes, this is what we want to build our service on and have gone and done it and we find out after the fact and not because of something goes wrong but because they start talking about it at performance reviews and it's like, oh wow, you did that and it worked. So it sounds like I'm negative about ButterFS but I'm honestly not, I just, yeah, I care deeply about the stability and success of ButterFS and I really hate it when people hit problems that are as a result of me making a mistake. So I want to be very confident that everything is right before people use it and so far I've been proved wrong over and over and over again. The, one of the funnier ones is I work on NBD for disaggregated storage services for a little while and part of that was using XFS over and that probably took me three to six months to stabilize just because it's a tricky thing. Most file systems are not really expecting your block device to just randomly disappear and with NBD that can happen and so there was quite a bit of time sunk into dealing with hangs and that sort of thing. I just recently found out a couple of months ago that some team had decided to use ButterFS on NBD and had been using it for two or three years and had no issues, which was just shocking to me considering how much time I had sunk into getting XFS to work well. Send and receive for containers was another thing. Send and receive, I know it works pretty well but I hadn't ever really considered using it in production yet because again, that's one of those things where if it goes wrong it's going to kind of go wrong in a weird way and then you want to be pretty confident it's working. Well, teams just decided to start using it and it's worked relatively well. And then with a compression, you know, I figured compression would be used for space saving issues, space saving things. I mean, that's kind of the main use case for it, obviously, but really it's been how we've solved our right amplification issues with all the flash and the fleet and this kind of translates to longer lives for our disks which translates to like real dollar savings. Problems that we've encountered, obviously, EnoSpace. I mean, it's not ButterFS if there's not an EnoSpace issue. That being said, it's been far less than I had expected. Again, this has been the thing that I've spent most of my career on unfortunately and it's gone pretty well. There's still some cases, there's some pretty bad cases that are left to be fixed that I'm working on but we're talking really rare occurrences in really bad extreme cases. So I've been relatively happy with that. Some different scalability issues. WhatsApp, like I said, has been pushing the file system to the limit in a way that we didn't really anticipate using ButterFS for ever. So those have been interesting to run down. They haven't been super difficult which has been nice, I've not yet run into a problem where it's like, well, ButterFS just can't do this. It's just been like, oh, we didn't anticipate this happening and get it fixed, so. And then there's just been loads of weird one-off issues. We've had a case where the checksum stuff was wrong. We've had weird cases with kind of weird corruptions that would happen. Happened to our, to Yen's Expo, the block maintainer for the Linux kernel. He had a pretty kind of work load that tickled it. So it's mostly just affected him but we fixed it anyway, we like him, so. But nothing really that stands out. There have been bugs with nothing that's been, oh my God, I can't believe this didn't hurt us or whatever. Things that are left to solve. Queue groups, quota groups inside ButterFS still, obviously we would love to use this for containers to be able to like fully contain space and that sort of thing for stacking services. The overhead for key groups currently is still just too much and frankly it doesn't, it breaks in a lot of weird cases. It's not something we spend a lot of time on, so I'm hoping to get to that in the next six to 12 months to really nail that down. Omar Sandoval is working on a subvolume encryption and hopefully that'll be in the next six months and that'll be really interesting. Again, for containerization workloads, we want to be able to encrypt containers and be really certain that we can stack different services with different privacy requirements safely. Balancing space in extreme cases, the way, this is just an extension of the InnoSpace stuff, I by and large, everything works fine. It's just weird extreme edge cases that are kind of problematic. So this is kind of another big focus of mine. And then RAID in five and six. The RAID striping and mirroring and striped mirroring works relatively well. It's pretty solid. Five and six, still there's a lot of edge cases and just more testing needs to be done. So that's another big work area for us next year to two years. Awesome. Well, that's it for me. If you guys have any questions, I'll be happy to answer them. Thanks. Okay, cool. So we have a few questions here. I'll just kind of run through them. So Nikolai asked, there are a lot of people claiming that ButterFS is not fully stable slash functional. What can you say about this? I mean, everybody has their own experiences, right? There's no doubt that ButterFS has had issues previously. That being said, Facebook is the largest production user of ButterFS, you know, millions of machines, you know, and then millions of machines depend on this for our everyday operations. And, you know, we don't take failures lightly. Obviously, you know, Facebook is far more fault tolerant than your average user. And so if something goes too wrong, right, we just reprovision and keep going. And, you know, a random Linux user is gonna have a very different view on those sort of failures, right? That being said, any sort of large problem or kind of systemic issues that you're gonna see would affect us in production. And we can't really afford those. We can't afford to have engineers running around fixing things all the time. So by and large, for Facebook and our use cases, which cover a large chunk of most people's use cases, it's been extremely stable and functional for us. Nikolai asked, how exactly does compression side solve write amplification? So this is probably more to do with Facebook's utilization of the flash. We obviously log a lot of things per disk. And so there's a lot kind of those constant stream of text being written to the disk just from logging. And so compression kind of handled this, you know, with the X-D4, we saw a significant amount of write amplification and the compression handles this by, you know, compressing the data and making sure we're not writing out as much. And the same goes for like file updates and that sort of thing. We do, like I mentioned in the talk, we push our entire website to our web services boxes pretty consistently four to eight times a day. And you're talking about 20 to 40 gigs worth of writes 48 times a day. And so compressing this really helps with how much we're writing, where we're actually hitting the disk with, which has drastically reduced our write amplification issues that we've seen in the fleet. Nikolai also asks, what compression algorithm do you use? I believe it's ZSTD. Well, yeah, I'd have to double check. I think it's whatever the newest one is that we added. We, one of the Facebook actually employs a guy who does compression algorithms and he wrote one and got it into the kernel and then we added that ability to ButterFS. And so that's what we use. How much, we also asked, how much space does that save for the repository clans? I wouldn't know. It's kind of hard to say exactly. I could probably go check, but off the top of my head, I have no idea. Oh, Chris just messaged me. We are in date using ZSTD. I'm sorry, I'm gonna butcher your name. Can you share performance numbers of ButterFS? I don't have any off the top of my head. So Facebook is really weird when it comes to performance. Every service kind of has its own requirements. I can say, and other services are better about testing their performance requirements than others. I can say for the web tier, it's extremely latency sensitive. Like I said, we love a lot of things and so the IO path tends to be pretty hot, weirdly, for the web tier. And so much so that latencies result in lower requests per second for the web tier. And web tier is kind of the first proving round after the build servers of will this work? And we consistently saw higher requests per seconds on ButterFS than EXT4 because we had generally speaking lower latencies because of the compression because we're not writing out as much. We were trading CPU for IO costs and with these kind of consumer grade solid state drives that ends up being a pretty good speed up for us. That being said, for what's that? For example, they're currently evaluating us. We see maximum read latencies on their disks at around 300 to 400 milliseconds, whereas EXFS kind of sees 50 milliseconds, maximum latencies, obviously the averages are lower. So ButterFS's averages are always significantly lower. So EXFS is around what I want to say like 10 milliseconds whereas ButterFS is around two to five. But again, for capacity sake, we care mostly about maximum latencies and so that ButterFS is for sure a lot more sawtoothy here. And so it's something we have to dig into. What is the future of ButterFS over databases? We will probably never, I won't say never, but we have no plans to move our MySQL databases over. That being said, we have a lot of services that use our LuxDB database. And that works pretty well because it's continuously appending and that right pattern works really well for ButterFS. So we have services to use them. If you use Docker, do you also use Docker as ButterFS storage driver? We do not use Docker, we have our own home ground system. Think we're just about out of time here. Let's see, is there a lower limit in terms of the use cases for ButterFS? For example, does it make sense on a small server or a high end desktop 32 CPUs, nine terabytes of HDD STD space? I wouldn't say there's a lower limit. Well, perhaps the, you know, if you start, we have machines that use atoms like atom CPUs and ButterFS is kind of CPU heavy just because of all our worker threads, especially if you start adding in compression. So, but generally speaking, you don't have to have a beefy box to run ButterFS. Anyway, I think I'm out of time. So thanks everybody. There's a Slack channel that you can ask questions and the hashtag to track Linux systems.