 All right, cool. I'm gonna go ahead and get started. This is sort of a rehash of what I talked about yesterday, but not really because I emailed Derek and got him to give me his list of complaints. So, some of the stuff on here, it's stuff that we've talked about. For instance, like the Continual Integration Testing is kind of a giant pain. We're all sort of setting it up ourselves. We can talk about this later in the FS tests, sorry, FS tests section. One of the things that he brought up is the stable thing. Are we actually expected to backport to all six releases of stable? We've talked about this in another session, so I'm not really gonna kind of rehash that. One thing that he did bring up that which hasn't been talked about is he's seeing an increasing number of people fuzzing file systems and then filing CVEs for fuzzed file systems, which triggers our security thing, which means then we have a time limit and to post a fix, you can't pull in the entire community because it's supposed to be only like the maintainer and maybe one or two other people. So I suppose this is a pain spot for him. I haven't experienced this, but you essentially get a CVE, now you're on the security process thing, which means that all development and review and everything is done very quickly without review and without the open community or whatever. Mike, can we please do something to address this? I think this is probably a wider kernel community thing. I have always been of the opinion that fuzzing file systems does not constitute a security problem. I know I'm probably a heretic in saying that because you have to be root to mount the file system. And every time I say that, somebody says, well, what if you plug in a USB drive? Turn off auto mounts. But anyway, I think that it is kind of terrible that like when we have these sort of issues that it kind of goes through the whole security apparatus. Have you experienced this at all, Ted? Yeah, so the main issue is if your company would like to sell into certain markets, like say the US government. The US government has this little interesting thing called FedRAMP requirements. And P1 bugs have to be mitigated within 30 days, I believe it is. And then they have like these different priorities that are based on the CVE severity score. And lower severity scores, you might have 180 days to remediate it. And that's a lot more time. Note that that's time to actually roll out the fix into production. Or you have to answer to the FedRAMP auditors about why in your environment it may not matter. And that's where you can make the argument about, well, we don't enable auto mount and we're not crazy enough to let random container people mount untrusted file systems on critical security systems, whatever. But it involves a communication with a government bureaucrat. Which the security teams don't appreciate doing. Can't imagine why. And so it's a real thing. I think the good news is most of the time a lot of the fuzz images aren't necessarily going to be a super high severity score because it actually still requires, if not local root, at least local login. The ones that are super, super scary as far as the CVE folks are concerned are the ones where you can actually remotely exploit the problem. And so folks who are dealing with network file systems actually have a lot more to worry about that product. But again, it only matters if your company happens to be trying to worry about FedRAMP requirements or you're selling into the financial world and they care about that for PCI compliance and whatnot. And that's really more of a commercial problem. But the good news is when that happens, it's a lot easier for me to justify headcount. Whereas getting headcount for, this XFS test fails one in 800 times. Yeah, good luck getting headcount for that. So it's a double-edged sword. Right, so this kind of sounds like a derrick. Yeah, I didn't think that Security at Kernel.org generally opened CVEs. I thought, yeah, Greg and company were fairly against that. So if the CVE is getting opened, it's from some other security? Yeah, it's by the research lab. And the problem is some of the research labs have an economic incentive to say we've opened so many CVEs that are high severity because if you then subscribe to that lab, you get early access to their security reports. And so they have an economic incentive to file lots of CVEs so that people feel motivated to pay lots of money to get early access to their reports. So I think that's the cycle that we need to get into in terms of communicating better what a CVE is and how to define them. And I just wanna say that FedRAMP doesn't really play into Security at Kernel.org so much. I mean, they're not caring about that. So it's more of the subsystem maintainer's decision, discretion about whether they wanna fix an issue before it goes public or not. So the net dev developers, for example, they're kind of known as not wanting to fix issues privately in Security at Kernel.org. They wanna fix it out in the public and things like that. The downside there is that it allows for zero days to exist for a small amount of time in distros and downstream products. So really, it's up to the maintainer. Do you trust your small circle of maintainers to fix something correctly, privately? Or do you wanna push it publicly and do it that way? So yeah. And I think a lot of it also depends on who your employer is, right? So if your employer happens to work for a large cloud company and you're not fixing the problem means that you need to explain to a VP why you're not dealing with this high security problem. Then that puts stress on that person. But to be fair, that's not a stress put on the maintainer. That's the stress put on an engineer who happens to work for Oracle or Google or Microsoft, right? Okay, cool means. All right, I think, like everything else, we kind of resolved in the conference that he mentioned, which I guess he'll be happy to see. So I kind of wanna talk about like the, like more expand on the things that I talked about yesterday. And Chuck and I had a good conversation about this is that like what do we expect from our maintainers, right? Traditionally, it has been, we expect them to merge patches, write code, test, review, all of these things. And as, what? Backport. Backport, right. And this is a lot of work for one person. And like a lot of us have kind of ad hoc come up with different ways to address this, making sure that our developers are reviewing the patches, not just the maintainer. Make sure we have good testing setups, this kind of thing. One of the other things that I wanted to call out and that I called out yesterday is like making sure that our maintainers are kind of actively involved in maintaining the community itself. Like the, we're all very driven and very passionate about the work that we do. And that passion sometimes leads us to get a little short of each other, right? And this, that's not really great for like good, maintaining good working, working relationships. And that's why we have stuff like this. Like it helps to be in the same room, to remember the other person as a real person and to kind of like resolve that. But I would love to see us move more towards taking care of these problems before they happen and not having to resolve them after they've blown up or having to wait till a conference to resolve them. You know, it's not possible for a maintainer to keep track of every email thread, of course, but for big things and big stuff that may be contentious, it is kind of easy to spot. And if we can encourage our maintainers to either themselves or find something in the community that's good for this play mediator to make sure that we don't have our developers picking fights, not picking fights, but like getting overly passionate and getting overly invested in the code. Because in the end, this is just code, right? We need to accomplish our tasks. We need to do the things that we're trying to do, but we shouldn't be getting upset with each other. And if maintainers can kind of be a little bit more proactive in keeping track of things that might be contentious and then making sure that people that are butting heads do it in a respectful way and in a way that they can continue to work together as time goes on, right? Sorry. So I had sort of two thoughts on that. One is that one of the things that I've actually found to be really helpful is we have a weekly EXT for developers meeting for which Derek also shows up. And that has actually allowed us to have discussions that will ultimately need to be resolved on the Linux file system developers list. But we have a chance to chat about it. Just Derek and I, maintainer to maintainer and the other EXT for folks are around. But it allows us to sort of work through various issues. Folks like Yankara is there and he actually adds a really, really useful perspective at times and then ultimately, of course, it has to go to the Linux file systems developers list because not everybody is on that particular call. But that is really, really useful. And I wonder if some kind of, I don't know, monthly, bi-monthly, quarterly sort of a chance for the developers to actually have a chance to chat might be helpful. I know we all need another meeting like we need another hole in the head, but it's helpful. It really, really is helpful that Derek and I have that chance to exchange things. And he just happens to attend the EXT for developers meeting because his work at Oracle, he has to worry about EXT for bugs too. But maybe that's something we couldn't consider expanding. I absolutely agree. I think some form of this would be, if this is in general useful for file system and be a best people alike, that'd be really useful. I think I've ran into this issue where I've had royal conflicts on the mailing list and my observation is that in very, very rare circumstances actually a third person comes in and is like, maybe you should all come down a little bit and that's a big problem in the long run. Yeah, and I think that the other, in your particular case, it's hard, right? Because I think for, I can think of one conflict specifically, like for that, I don't feel like I, like it's, how do I say this? We need to make sure that it's okay for random people to pop into a conversation and be like, all right, everybody calm down, right? Like, okay, I'm not the VFS maintainer, right? And I'd have no skin in the game or whatever, but it's like, if I pop in and say, okay, guys, sorry, let's all calm down, let's bring it together and kind of work out the issues, right? Like I wanna make sure that we're kind of encouraging that, like encouraging all of us to do this because it's easy for us to go and say, well, Linus should do it, right? Because I mean, you're probably closest to Linus, right? But I think expecting one person to do that is not awesome. But on the other hand, again, I'm not always paying attention, right? Like your stuff I saw well after the fact, right? And I think it, I wonder if it's a thing where we can kind of encourage people to say, okay, if they're in this, call for the third person. Call for a, you know, a maintainer or somebody to like, hey, we're having some trouble here. Can you help us work it out? And I think this also goes with the technical thing, right? Cause there's kind of two problems that I see that maintainers really should be stepping in on. One is the conflict resolution, but it's also like technical direction, right? Sometimes like you have two developers that like don't agree on what the best way forward is. And that's where I really want my maintainers to step up more and say, okay, we're gonna, this is the path forward. Don't care what it is, but like the maintainer has enough context on the whole thing and they can come in and say, okay, instead of letting these two developers or three developers or whatever continue to butt heads, the maintainer steps in and says, okay, this is the direction we're gonna go and gets agreement and we all move forward, right? Yeah. The other thought that I wanted to bring up is I do agree that in the ideal world, the primary goal of the maintainer should be conflict resolution. But I guess the model that I've always had, and I can definitely see how this can lead to burnout, is one of servant leadership. So the maintainer does everything that the rest of the community isn't willing to do, right? In the ideal world, I would have a test engineer working on tests for me. I don't, so I do test engineering work, right? And that's because I know it needs to happen and no one else is going to do it if I don't, so I do it. But, and the other thing that I do is I spend an awful lot of time recruiting people to my development community and encouraging other companies to staff people for that particular file system. So I'm working with an unnamed cloud company, not my own, to try to get some cloud engineers to work on EXD4 and hopefully join our weekly call. And that is probably something that other maintainers should think about doing is the cross company sort of inculcation and sort of encouraging other companies to try to bring people to the table and to the extent that other members of the community can help the maintainer in those two activities. I think it's actually a great way to try to reduce burnout, right? Because half of the problem is we just simply don't have enough people on some of the file system teams. Yeah, and what I really want to do is reduce the amount of things that the maintainers are doing so that they can focus on these things that are arguably what they're good at, right? Like Ted, you've been here forever and you are not argumentative, right? So conflict resolution and technical direction for EXD4 is probably what you should be doing the most. And the more we can develop tools in the community to do things like CI and automated testing that offloads the testing bit from you. And you already encourage your developers to review that offloads the review thing. And so like the more things we can, because we can automate a lot of these things, what we can't automate is the human parts. We can't automate conflict resolution. We can't automate technical direction. I don't know if you guys ever saw the mission statement of LTP project or in general how their maintainers interact, but their spirit is in general and specifically in the statement. If you want, just give us a demo of a test and we'll make a proper test out of it. And they keep constantly cleaning their tests and really if I submit the test, I submit tests that are already pretty clean, but they're doing the extra work afterwards and if there was something like that going on, if you could get headcount to do that work for FS test, because if you look at the people that clean up FS test right now, you will find several high level maintainers, high level burned out maintainers that are the top contributors and cleaners of FS tests, right? Yeah, I think that the better we get about running XFS tests in a consistent manner, the sloppier we can be with accepting XFS tests. So, and that would make it faster and easier and then people can come along behind and if there's things that need to be like, I really, you know, I kind of keep harping on this. I want to go faster. I want to merge many more patches per current release and in order to do that, I need to make sure I'm not breaking anything. And okay, is my FS test perfect? No, I probably missed some helper or whatever, but it's testing a real thing. Let's get it into the tree, so it's getting.