 Okay, so you guys ready for some pre-lunch excitement? Okay, sounds good. Okay, so if I stand over here, you guys can see the exciting stuff. Okay, now, what I wanted to talk a little bit about today was notify support. You know, I'm not expert in this, but I think as you guys know, there has been over the last three years some changes, some improvements. But a long time ago, there was some fun history about notify. So I deal with SMB a lot. I'm also a member of the Samba team. So Samba team was what drove the original notify into the kernel in the first place, Trig's work. It was kind of an irony with that. So I'm going to give an example of it. I'll talk about the problem and then how we might fix it. You guys may have better solutions, so let's dive into that. So as you guys probably know, if you had a Windows or a Mac laptop, it's very common whenever you open any file window, any explorer, right, or the equivalent on a Mac, it's going to populate with all the files in the directory and go read their icons and set little things at the bottom of the icon depending on attributes it finds. So the reason this is important is that if I'm editing a presentation and then I'm dropping it in some shared drive and Dave's editing your presentation, I'll see it. I'll see his show up automatically. I won't have to do anything. I won't have to pull. Back in the dark ages of OS2, they called this the wheel watcher. The wheel watcher, just they're pulling. But many, many years ago, they added notify support to Linux to avoid this. Well, ironically, when they changed the iNotify, that broke it for clients. And, you know, the remote systems connecting to Linux could see notify changes. The servers could see some of this, but it was kind of a mess because it changed the client API. So network file system clients or cluster clients have no way of waiting on these easily right now. Maybe they could hook it in user space. And what I do in the SMB driver is I have an iOctl that you can call. If you want to wait on a notify, then you have to call an iOctl. But then we'd have to change all the apps in user space to use a separate iOctl for each file system. So file systems are not told to wait on anything right now. So there are file systems that call out to wait on things, right? There's cache file systems that call out to wait locally. And there's notify events that file systems can call to notify their read has occurred or write has occurred. But there's no way that they can tell that their iNode has notify events queued for it. And this matters for Linux GUIs too because, you know, you've got, you know, people have the equivalent of, you know, we have an explorer here in Linux as well, right? So if we want to go look at files locally, that matters. If Dave changes or adds a file to a directory, I want to see it. Okay. So here's an example from Windows. So if you look at this, from a remote drive, I create a file junk or from a local drive, I create a file junk. It's automatically refreshed on the remote mount. This is Windows, right? It happens the same in most operating systems except Linux. So from a different client, I'm creating junk, but this client sees the junk that was created automatically without me having to do any polling. That's normally how you expect it to work. That's the whole reason for the API in the first place. But iNodeify doesn't call into the FS. So if you look at this function here, this is Update Existing Watch. You can see how Update Existing Watch works here. It calls into Find Mark, Container of, sets the mask, but there's no call out to tell Ceph or to tell AFS or to tell SMB. I don't remember, does NFS have an equivalent? I guess you could have a directory release delegation, right? Does NFS have a notify async? The protocol has it, but it's not implemented in Linux. Yeah. I mean, it's super common thing you see in other OS. And, you know, there's no point in doing it for Linux for NFS unless you actually had an API that could call into the NFS driver. But, you know, from the SMB perspective, it's not... It's very commonly used. There's also an FA notify. And I don't want to dive in too much to FA notify, but these file system events are also interesting. And a little bit confusing to me. There's these bit marks that you wait on, but FA notify and I notify are on the same directory tree. And you guys can look at that. Whether FA notify is as useful, I don't really know, because I don't know how many... One thing I couldn't find trivially last night, and you guys might know, is how many apps actually use FA notify? Obviously, there's some very important ones that use I notify. Steve. Yes. So, FA notify was... When it was conceived, it was used for and created by antivirus vendors for the permission events, which are blocking events. But since then, there's been a lot of work to add more functionality to FA notify. And it's relevantly recent, so it started coming in with 5.1 and more functionality later. So, in 5.10, there's... FA notify is almost a super set of I notify. Right? And, of course, it adds a lot more features, but it implemented most of the features that were available in I notify and not available in FA notify before. One very big feature that I added to FA notify that did not exist, I notify is watching the entire file system. Okay? So, instead of recursive watches, and I don't know... So, there are not many applications because it's new. Right? Yeah. And the applications using I notify are older. What I did do is I went and implemented for the library I notify tools, which is where the I notify wait and I notify watch utilities are. I added support for FA notify. So, now, the recent release, you have an addition on top of the tools I notify wait, I notify watch. You have the tools, FS notify wait, FS notify watch, which you can use to watch a file system or a file. It doesn't matter, but using the FA notify API. I think this is extremely useful. So, one of the key questions, though, is how it maps to, let's say, what's available in current file systems, you know, SMB, NFS, AFS, F, whatever. So, I'll show you in a second for SMB. We have the choices for notification of doing, you know, best effort, async notifications, or doing things like delegations or releases where we guarantee we'll be notified. 100% guarantee we'll be notified about the remote change. And, you know, obviously we, I think NFS uses it more than SMB right now, but we use it on the root directory and it's being, I just had some patches for 5.19 to extend it more. But those, you know, we can wait on any event relating to a directory or a tree. So, what this originally, what I notified was originally designed for was to support async, the best effort, it's called change notified over the wire. And that's an async thing, it doesn't sit there and wait for the client to respond. So, there are additional, one thing I'm curious about is here's the I notify flags in the Linux headers. And, you know, things, events like a sub file is created or you've modified a file or you've accessed a file, you've deleted a file, those are the most common events I would think. And are there other events that, you know, other than a file being accessed, modified, deleted, created, that FA notify requires beyond these here? What do you mean by acquires? Well, what I mean is that, I'll give you an example. In SMB, I believe there is a notify event on an alternate data stream or an X adder being created. Okay, great. I don't know if NFS could support that or AFS could support that, but there is an option to wait on other things than, I want to wait on this directory to see if a file is created, great. But do you want to wait on this directory for something that the file system can't support? There may be flags that you can pass in that can't be supported. And I don't remember exactly how the mask might be set up so a file system can say, well, I can tell you if a file is created or deleted, but I can't tell you if a file is accessed. Are you asking about FA notify? FA notify has a very similar set of masks. It has some additional events for execute and stuff like that, but I mean, if there's something missing, we can think about adding it, so here's what it is in the Linux iNotify headers. I don't have the NFS RFC pulled up to look at what NFS has or AFS, but those are probably a subset of SMB. For SMB, these are the things you can filter for or what can be watched for. You can look for changing particular timestamps, look for changing the file name, deleting, creating, writing, right? So these are the kinds of things that... So I don't know. I mean, these map pretty well, I think. I don't know if applications... I need to argue if there are applications that need to know specifically about the change of extended attribute. Yeah, but we don't have to use that. You don't have to. No, no, you don't have to. It's okay. It's okay if... But changing the accl, changing the rich accl or changing the posix accl, change security might, right? Maybe if you can show that there's interest, but then only CIFFs will support this event, so... I mean, you could have the posix local API, the posix accl API, support it locally. And personally I would think that I would want to be notified if somebody changed permissions underneath me without permission, right? So I got this... Here's a Ubuntu box here, right? I might have an event that I'm looking to see if somebody's changing accls under me. I don't know. Forget... But I think it's fairly close. Streams are similar to extended attributes, so we don't have to talk about that, but I think it maps fairly well the events you can watch for. And I didn't look... Like I said, I didn't look at the NFS-RFC for this or the AFS-1. And you can return these as... So you're filtering for this and what can be returned is you added or deleted whatever you're waiting on. So I think it maps pretty well. So... In terms of applications on Linux that use I-Notable notifications like this, it's mostly desktop, things like desktop file managers and stuff. So KDE, if I remember rightly, will start... You can set a watch through the KDE infrastructure and it will start to be a demon to do the watching for you. No one appears to have something similar, and it seems to watch for mostly directory change events, but also for permissions changes on files because it may be displaying the permissions all the times. And so it will update the thing if those change. And if notifications aren't available, I think it polls in the background. Oh, it's not that the notifications are not available, it's just that they're not granular. You get a notification for attribute change. You just cannot request for specifically... Yeah, but it is polling for some sort of attribute change. So polling or watching some sort of attribute change. And then it has to work out what changed. But there are applications in Linux that do this. Mostly desktop environment applications. But also Wine will do it, I would suspect. I use it in cache files. I'm sure there are other places that use it. I think it would make a lot of... We have a few questions for remote attendees, and I really wanted to use this opportunity to see how we can let remote attendees really participate because you've been asking for this still for a long time, but now a few developers also took a shot at implementing something like this. And let's see. First of all, Vivek says, what's the equivalent of a stream in FS Notify, in iNotify? What's a stream? Can you go back and slide? Yeah. So in the Windows world, a stream is associated with the file like... So a stream is like an extended attribute, but you can do seek, read, and write on it. With an extended attribute, you typically replace the whole attribute, right? Where with a stream, it's the same kind of thing. There are many people who dislike streams as a feature of other operating systems. The concept that you have a file that has its main data stream, is what you're used to. And then it has these other alternate data streams that are sometimes used. As an example, historically, what sometimes is used, you might have a file that had the English-language version of all of the help text was stored in one stream, in the German in another, and if they wanted to add Portuguese, it was in another. So they had an alternate data stream for each language that the program used. But it's really, I think, less important because many file systems don't support streams, and they just map them to extended attributes. Macs have a lot of backup code in the iPhones and iPads, and Macs, if you can't find a stream to use something else to use, et cetera. Okay, let's try to have Vivek speak on his behalf, if we can make it work. By the way, we did have some patches three years ago or four years ago from one of the guys, maybe somebody at... Yeah, Miklos sent some... some POC. But, yeah, it's not simple. So let me, before Joseph is trying to invite Vivek to speak on his own behalf, but what I can say is that there is an implementation, a national implementation by Ioannis that had iNotify, specific iNotify API added to Fuse protocol. Okay, Vivek. Okay, can you hear me? Yes. Yeah, so as Amir was updating, for the Fuse file system, so basically our goal was what you were trying to do with Samba and Sys, we thought, hey, can we do it for what Ioannis? That's where Ioannis started playing with it and did the initial implementation and because iNotify is simpler, fnotify, the semantics are much more rich, so a little harder to understand. So we started with something like, let's just try to do a remote iNotify and just posted one version of patches. So what we did is basically, because in our case, so basically what you were doing is whatever watches you put on the local iNotes, we forwarded it to the remote iNotes and so the what I offer just puts the same users iNotify and watches for the events and the moment the events happen, it forwards it back to the client and then client sends it back to the user space. So, and after that, the one feedback we got from Amir is, hey, this is like underlying, this is the fsNotify infrastructure, which is common and being used both by iNotify and fnotify. So what you have implemented is like, you are using this common fsNotify infrastructure, so try to make it more generic. Let's call it remote fsNotify so that you can support both iNotify and fnotify. So that's what Ioannis is trying to do now, like extend it to even support fnotify but because fnotify is so rich, in terms of semantics, we are running into a lot of issues. For example, like, of course, like Amir said, there are many things you cannot support. For example, the permission things we cannot support. fnotify has something where it opens a file descriptor by default, then for the file on which the event happened, then we realize that it requires a path information and all we are sending from the remote is iNotify information. And so like, we don't have the path and then probably can't open the file descriptor. So maybe I'm going into too much into details because Ioannis is still struggling with it and he was seeing many more events. So just taking a step back, like I guess that's what you are also looking for, like a respective implementation detail. Maybe like, I'm assuming that you are also looking at that, hey, let's look, let's try to implement this iNotify and fnotify for remote file systems and see how many of the events can be mapped, like whatever the protocol is supporting, whatever can be mapped, we will map and maybe reject the rest of the events if we can't map but don't support. Yep, that makes sense. I think it's very important to understand at a high level what Fuse might want to allow, what NFS might want to allow, what AFS, what SMB, can it work on files so it doesn't only work on directories? Because the alternative is really painful. The alternative is that we're polling and I was thinking about Dave's comment about FS-Cache. FS-Cache matters over every file system practically, right? I mean, there's a lot of cases where FS-Cache would matter over Fuse. There's a lot of cases over NFS or SMB and maybe this is a better way for FS-Cache. You know, we can tie in with delegations and leases, sure, that's great but there's cases where FS-Cache matters as well. Because right now... Yeah, sorry, Jan wanted to say something. Yeah, sure. So, hi. So my point was giving you, like, a hook into the file system when a watch, like, notify a watch is added is the simple part, yeah? You can do that and I can even imagine, like, the file system will, in the hook, have the right to say, you know, I don't support this so it will propagate the error back to user space, basically. The somewhat difficult part is more on these... So when the file system actually detects events and this is going to be different for different file systems, yeah? So for somebody, you have it mostly soft because the protocol already has the support of the remote notification, essentially. Vivek is struggling with this for the IOFS because there they have to actually develop the protocol to generate this notification on the file system side. So that's actually a separate set of problems, let's say. But if we now speak about the interface between the file system and the FS-Notify layer, like, telling the file system there is a watch added is the simple thing. But what's more complex is, so you have the event on the file system side and now you want to feed it back into FS-Notify so that user can read it from the, like, FS-Notify descriptor, essentially. Either I-Notify descriptor or I-Notify descriptor, but it's in principle the same thing. So because there, as Vivek said, you know, there are some expectations from user space about the information that's inside the event and, you know, you have to make sure that the file system can actually provide this information. So for I-Notify, it's relatively simple because it's going to be the I-Note number and the file name, essentially. And I hope that most file systems will be able to look up the I-Note for which the event happened so that we can give it back to user space, essentially. Like, we don't... It's not I-N-I, it's the watch descriptor for I-Notify, but anyway, oh, sorry. On that question, I looked at my grep for FS-Notify notifications, the ones that locally are passed up to the application and it looked like on open, it calls with the file, you know, FS-Notify open, on read, you get FS-Notify access or modify. So, you know, grepping for this, it looks like there's already things and they mostly just require the file descriptor, the file struct, whatever the... So, in many cases... Scriptor is definitely enough, you know. If you have file descriptor, you have more than enough, you know, for this file descriptor, you have basically everything, even for FA-Notify. Like, if you have file descriptor, then that's enough for FA-Notify in all the cases as well. Yeah, but for... Sometimes you have only... Yeah, sometimes the problem is you have only the I-Note, yeah, and then things start to get difficult or maybe you even don't have the I-Note. Right, that's the problem we have. Like, at least in our implementation, all we have is I-Note. No file descriptors, no path information, no mountain information, nothing. I mean, I have a similar problem. Right, I'll have a similar problem because it's a directory event, right, and in some cases I won't have an open file for it. So, there are going to be similar problems in SMB, right, because you're going to get a directory notification about a file being modified. Right, so, you may know... So, I was trying to look at these FS-Notify. If you look at, like, Removder, that's where you get a dentary, Flags, I mean, they're mostly... I guess they're mostly I-Note, but not all. I gave you some examples of file. I-Octol notifies for access and modifier files, but most of these are dentaries or I-Notes. So, we definitely need dentaries so that basically dentaries is a way to propagate the name into FS-Notify so that it can get reported. So, dentary is needed, but also the mountain information is needed for FS-Notify, for example. Because, like, we don't have to support this at least initially, but for us to be able to support a full set of FS-Notify features, you would have to also, like, tell the mount point. But I'm perfectly fine with just telling, you know, file system returning basically not supported for, you know, if the user attempts to use, say, the mount point watches or even the file system watches. You know, if we are going... Jan, we cannot support mount filtering for remote changes changes anyway, because it changes remote, it's not from a local mount. Yep. Yep. That's true. Also, we may be able to modify some of these helpers, like, you know, the ones that are files. Like, FS-Notify X-Adder is a dentary I-Node, right? So, FS-Notify X-Adder, you're writing to an X-Adder, it's using the dentary or the I-Node. Write to a file, but not write to an X-Adder, or use this file. So there may be a way to modify the functions that FS-Notify, modify dentary or something, right? So we may be able to avoid the need for a file by just... Yeah, it's possible. It's possible to do that. Yeah. We need to look into it, but it's possible. I looked at this code. Like what... Sorry, somebody's saying something. Sorry, I was just saying that I've looked at this code before, and usually these things are just wrappers around the main thing that does easily there. If you look at the helper, it actually breaks down the I-Node dentary, so... I mean, that's really good. And it avoids this problem here. If you look at the screen here, like, I don't want an app to have to call an FS-specific I-Octol. I mean, I'll offer it, so you can... We got user space tools, you can write for this kind of thing, but I really don't want this. I want people to be using something that's a little saner. And I have kind of PTSD from thinking one of my first jobs after graduate school was looking at the OS2 and Windows Wheel Watcher. That's so crazy. Polling every whatever just to look at the darn... And it's terrible for networks to be polling, so we don't want to have apps polling if we can avoid it. And anything we can do to avoid that, to sane use of notifies, to me seems like kind of a win. The hard part, though, was, if you look at the proposal thing, was it was just a little bit harder to figure out how to plum the registration of the events. It's not too bad, but the registration of events was a little bit tricky for I-Notify and a little bit trickier for FA-Notify. Not because of the mapping of flags, but just sort of, like, making sure that I understood where you'd have the check to see if a file system exported Notify, and if it did, also call Notify, because there's no harm in notifying locally as well as having the Notify from remote on that same mount. So, anyway, it didn't look too bad, but... Yeah, I think it's... the answer was reasonable to call... like, provide a callback to let the file system know if a watch is coming. I think that between you and Vivek, you can probably work out a reasonable interface. That sounds reasonable to me. But what about sub-tree watches? Because in Windows, there's an entire sub-tree. I don't know whether... I don't know whether you can do that in FA-Notify or specify that. Yeah, everybody wants that. Well, the good news is, as long as you're... for the whole mount, I can call a sub-tree watch on the mount, and the fact that Windows supports something you don't, it hurts Samba server, but I don't care as a client. The mount thing is, as John said, you have to know the amount the operation is coming from in order to filter by the mount, and that's only available if you have file-stract, not if you have dent-tree. But, I mean, if you're waiting on a file system mount, then we just call out and say, do a sub-tree for... tell the server to do a sub-tree watch on that whole sub-tree starting at the top. Several approaches have been attempted, several patch sets have been posted. Yeah, it's... It's not trivial. And I think the other thing to remember is that this is an asynchronous non-perfect mechanism using Change Notify, but NFS and SMB and every protocol also supports a strict approach where you'd have delegations, leases, usually not worth the cost to have all those, but it is an option as well to implement underneath in other file systems, but I think letting the file system decide on that is fine, but you'd have to just check to make sure the file system supports this operation and if not, just fall through the current code and not do anything different, there's no changes. But I think one of the things I want to mention, because we're all running low on time, is that what bothered me was when I went into XFS test, especially after the discussions that Ted and you guys were all having Lua, all these guys, like, do you see any tests? What? What? I was watching the chat, what? I couldn't find any XFS tests. Oh, the tests? The tests are not an XFS test. Okay. Yes, yes, yes. We maintain our tests in LTP. Yeah. Yeah. So, it's great, just something we need to make sure that we're aware of because we want to make sure we don't break obvious things. We have a feature and we have test coverage for bugs that we fix. Okay. You know, there may be cases, and we can talk about this in the afternoon, is like where we may refer an XFS test or the XFS test wiki to other test suites, but, you know, at a high level, I was like, well, how do I try this? Let me run a test case and show it, and then I'm like, oh, shoot, I can't find a test case. And I could have run, you know, a sample program myself, but I was being lazy. But I do think that testing if I notify, especially as it evolves, is going to be important because there are apps that depend on this other than just Explorer or the GUI, the desktop. Yeah. I think this is a historical artifact from the point of at a certain point in time, those folks who were most interested in XFS tests, which tended to be the core file system developers were using XFS tests because people who, if you actually look at denotify and inotify, it's in the core VFS and the people who were doing that work or maybe it was the LTP folks. Anyway, the tests are in LTP and I think I've never worried about the fact that there's no inotify or FA notify tests and XFS tests because I don't worry about inotify and denotify. I'm just merely stating that the historical reason was that at the point in time those people who were most regularly writing XFS tests and running XFS tests were very, very important and very important and very important and very important and very important and very important and very important so byDoesthingXFS tests were separate from those folks who were mostly doing I notify and denotify, and that's just a historical thing. We do have to be careful, though. Take your example. I mean, you call FSnotify. Now you're calling it on an event and you only have one event you're waiting on but even EXT4 has specific FSnotify code. I didn't realize this until last night like one in the morning. It's like, what the heck? We have notified code in it. Yeah, that's pretty recent. All right, let's move on to the next topic, which is you anyway, but we're about done on this one. Okay, sounds good, and we can follow up afterwards. Okay, so let's talk about another function.