 My name is Anurag, and I'm going to speak about writing flexible file system with Q's Python. Just a small survey before I start. How many of us here are file system engineers? Okay, so if I say anything wrong, anything just correct me there, okay? That'll be fine. So I'm a Python beginner, and I wanted to know how the OS works, how the file system works. This was a month-long project to understand the file system and see how to make a working file system. So there's this amazing library called Fuse, which allows you to write your own file systems in a high-level language. Python was the language choice because Python is good. So this slide is also available at this URL. If you are following in the laptop, just open this in the browser. So you can follow along at your own pace. Okay, so talk overview. I've segregated the talk into four or five parts. One is the Unix-based file systems. We'll see what file systems are, how they work, what are the basic components of a file system, what are the attributes that you need to define. Next is what is Fuse? So given the file system which we already have in the OSes, where does Fuse fit in? Why would you use Fuse? Then Fuse API overview, we'll see what are the basic functions that you need to implement, what's the class that you need to subclass, and how to get a basic file system which displays a directory going. Then there's a small demo about ToyFS. It's a Toy file system just to learn. So there are two, there's a controller in it and there are two modules. One creates a file system with directories and there's another module which creates a file system with directories and files which you can read and display some output on the screen. It's a very small toy file system just to learn. Next will be question and answer in the end. This is a very interesting comment by a professor Joseph Pfeiffer from New Mexico State University. This is very true, right? One of the real contributions of the Unix operating was the idea that everything is a file. You look at everything as a file. So for example, a keyboard or your USB drive or your display, everything is a file. So you read something from a file and you write something file. So it could be your printer or your hard disk or anything. So what are the common file systems which we already have? VFAT is there for Windows, HFS plus OSX. You have EXT4 on Linux, you have XFS. So there's NFS and Fuse which is mentioned with an S-Tris. So these are not on-disk files. You can say they are virtual files. They are based on an on-disk file system. They are based on top of it and they provide extra functionalities. So NFS for example is a network file system. Fuse is a very peculiar file system which allows you to make very interesting files. For example, you can make a file system which reads into a zip file and shows the contents in a directory format which a normal file explorer can view. If you do this cat proc file systems, it will tell you which file systems are currently supported by your kernel. So it supports, among other files, it supports EXT3, EXT2, EXT4. So the kernel is compiled for XFS also, but I don't see it here. So you can simply load it via modprobe. That will load it. When you do cat proc file systems again, you'll see XFS over there. So this way you can load new files which are compiled with the kernel. And now, this is, I'm sure many of you are familiar with this. These are the file system type attributes. So when you do a directory listing in a long mode, iPhone L, the first character that you see signifies what type of file it is. If it's a D, it's a directory, or anything else, it's a file. So directory is a special file which contains files, including directories. Now, fuse. Fuse, the full form is file system in user space. So f comes from there and then uac comes from there. So this is how the virtual memory is segregated. The first box that you see over there is the user mode, and the second box is the kernel mode. So user mode means all the applications that are running in, with the user's permission. So things like your demons, or the window system, or the browser or editor which is running on top of the window system, or any other libraries, they all run on the user space. They run with your unprivileged user's permission. So they all talk to the G-Lib C API, which provides you a set of maybe about 2,000 function calls. So they all talk via the G-Lib C standard C API, and which in turn connects to the system calls provided by the kernel, and gives you the ability to access the underlying process scheduling, or the inter-process communication, or the virtual file system. In this case, that's something that you're going to see, or the network. So this way, the memory is segregated. So you cannot directly access anything, management or virtual file system directly. You need to go via the kernel system calls. So what Fuse does is, it's usable by unprivileged users. What that means is, as a normal user, you can create a program which uses Fuse library, and make it behave the way you want. So for example, I mentioned the zip file system. So you can make something like a git file system. So you can write a program in Python in any other language, which does a git show ref, or a git ls, and the output that you get, it translates that into a directory listing. We'll see that into a directory listing, which is passed on to the kernel virtual file system, which goes back while the kernel system calls to the G-Lib C API, and goes back to the ls command. So when ls runs, it requests the virtual file system for the listing of the directory where it is in, or the parameter. So this travels all the way back from the kernel back to the application which is running in the user space, which responds with git ls, or git show ref, or whatever you want. So it's usable by the unprivileged users. So the most important point here is, the development cycle is slower, because it's a user mode program. You don't have to recompile the kernel and add a support for XFS, or any other latest leading edge file system. It's easy to install. For example, SSHFS is another interesting file system. It can simply be installed via appget install, or yum install SSHFS. What this does isn't, is it using your user name password, ssh user name password. You can connect to a remote server, and it gets a listing and transfers back. So you don't need NFS or anything. Simply an SSH connection to the remote server can get you a usable file system in your local machine. Multiple language bindings. We're going to see Python here, and then port it to other operating services as well. So this is what I was talking about. How the calls go via the VFS back to the application. So on the top box, you have your program running ls-l and a directory where your Fuse file system is mounted. Then that's the user space. So whenever you say ls, you want to know the directory contents of that directory, TMP Fuse, or whatever the mount point is. So the control goes back to GLIPC API. GLIPC API transfers back to the VFS via the current system calls. And VFS knows that this directory TMP Fuse is mounted over Fuse. Some other directory, mnt slash nfs, could be mounted over nfs or ext3, whatever it is. The VFS knows that. So it goes to Fuse, and it transmits back up to the libfuse and back to the program which is running. So dot slash hello is the program, and it's mounted over TMP Fuse. So the control goes here, TMP Fuse, and it goes at the hello program, and hello program will do whatever it wants to do, right? If it's zipfs, shfs, it will take action on the parameter, TMP Fuse, and it will give the listing back. It'll come back to the program ls. Then ls will display the contents, okay? So that's the diagram. We'll see how it works actually on a working simple file system, right? I'll set up two terminals. One is where I will mount the file system, and the second one I'll show the logs, right? ls slash TMP. It's empty. So we want to make a directory called TMP slash fuse where we'll mount the fuse file system, okay? This is empty directory because it's just an ordinary directory. Now we'll go to the toyfs file system. There you have a main controller, toyfs.py. We'll see that what's inside it in the latest slides. So to mount a file system, you simply have to do toyfs.py and the mount point, okay? I'll open the log here. A new log file has been created, okay? It's empty because we have not done any operation on that directory. Now when you see ls TMP fuse, you'll see it's no longer empty. So it contains some two directories, right? Hello and world. And on the log, you'll see the program is emitting get80tr, readdir on slash. So for the mounted file system, a program called ls somehow wants to know what's inside slash, right? So it responds with some directory listing. It's a dictionary. And then you get get80tr on hello and get80tr on world. So the directory contents is hello and world, and it needs to know what kind of file it is. It is a directory or what permissions are there. So because of which two get80tr extra are fired over there, right? Now let's see what's inside hello, okay? There's another directory called weld. Now notice here the get80tr command has changed, and now you'll see readdir hello and hello world. So it's getting contents from fuse hello weld. So same way you can go recursing into a directory and get the directory contents. I'll unmount this directory for now. So how do I install fuse? You need to install fuse-python, or if it's Ubuntu or Debian-python-python fuse. Loading fuse, same way, modprob that will load the kernel module. The way we did for XFS, you can simply do sudo mod fuse, and that should show the kernel module loaded, okay? So the first thing you need to do to implement a fuse file system in Python is create a subclass of fuse.fuse class, and implement a number of methods, okay? So we saw in the log there are two methods called, one is get80tr and one is readdir, okay? We'll see what that is. So as I mentioned, this file system contains two major components. One is the toyfs.py that's a controller, and the actual implementation is inside toy.py toyreadwrite.py, okay? So this is a glue code which can simply follow along. You have to import fuse time if you want to use it. I'll just import it. Setup logging, toyfs.log is where all the log is being printed. Then you have to subclass this fuse.fuse class, okay? I've imported toy as toy because I want to change this on the fly, make it toyreadwrite as toy and the rest of the code doesn't change, okay? Then you have to implement these two methods. Get80tr is getting the attributes of the files and readdir is reading the directory contents, okay? These are two basic functions that you need to implement to get a working file system. So what I'm doing here is get80tr, it's printing a log message and it's getting the stat object from FS. So FS is over here. We'll see next what FS does. And it calls the FS.get80tr. It's a mirror function. It does the same thing over there. And the actual implementation is inside the FS.get80tr function. Okay? I've implemented two more functions. We'll see that later. And in the end, you have to... This is for printing the... When you give the python and toyfs.py command without any mount point, then it prints the help message, okay? At the end, you have to set all these attributes and say toyfs.main. That will mount the file system, okay? And this... When you say python and then the program name and mount point, the program will quit. And then later when you want to unmount it, you can say fuser-mount and then minus-cube. That will unmount the file system. Yeah, we covered this. To mount the file system, you have to instantiate the file system class and just call the main, what we called in the end. Yeah, this is how you mount it, python, and then program name, which implements all this and then a mount point, okay? tmp slash puse or toy, whatever it is. Now, get it here, this function. Defining this is mandatory. Without this, a file system will not work, okay? Because a file contains many attributes, right? The file type, what permissions are there, who is the owner, okay? And what's the timestamp, create timestamp? All this is mandatory. So, you need to implement this for a working file system. The stat structure. So, these are the various attributes that you need to set. So, over here, you'll see that this is a mirror function. When you call, get it here, it will make an object of a toy stat. So, toy stat contains these attributes. So, sdmode, mode means the permission, its combination of the permissions and the file type, okay? iNode number, rest of it is mandatory. You need to set the user ID, GID. For this particular system, I've set it to 500. You can set any user ID, GID one. Or you can set this permission inside your file and then retrieve it and then set it, okay? This is static for this case. sdsize, you need to set what size the file is, okay? And then the access time, modified time, creation time, okay? So, these are just the explanation for all those bits. How do you view the stat of a file, okay? You run this command stat. When you do stat on a file, it will tell you everything about that file. So, permission bits, okay? Size, okay? File name or access time and everything. So, all this comes from the sd-stat member. How the mode actually works. So, in stat module, if you see the documentation, Python stat module. So, there are the variables, s and so if dir. And then you bitwise or it with the permissions. 755 is the permission that I want to give to this file. And do a bitwise or and set it to st-mode. So, that will set the file type and the permission bits, okay? st-mode can take these parameters. If you want to create a directory, then you can simply say if dir or link or anything else. And bitwise or with the permission that you want. This explanation is there in the Python stat module. Simply open this link. Next important function is readdir. These two functions are required. Readdir will give you the directory contents, okay? First you need to get the permissions of a file and then the directory content to be able to display the contents of the directory, okay? So, in this case, I've made a very simple function, readdir and the path. So, the hello and world of the directory that you saw earlier, it's coming from here. So, there is a content dictionary. And if you query for a slash, okay? When you do ls on the TMP fuse, it's querying for slash. So, you get this as the response, hello and world. Or if you query for slash hello, you get weld, okay? So, that's where it's coming from. And then you need to add this to dot and dot dot, which is the current and previous directory, combine it and return back, okay? That's the directory contents. So, you can, this is very simple file system in which the directory content is inside a dictionary, right? Or you can modify this. You can make it a SH file system or HTTP file system where you can do this lookup part and make a, let's say, a call to an FTP server, get the directory listing from there, and then append with dot and dot and return back. You can do all kinds of fancy stuff over there. Reading files, if you want to read files, so the first two functions, getdir and readdir as for simply getting a directory listing and getting the directory tree. If you want to actually read files, you need to implement additional functions open and read, okay? I'll just demonstrate a small modification to the toy.py file system implementation. So, I've made another program. So, in this case, what I'm doing is, first look at the readdir. So, in slash, I'm adding a couple more directories and files. So, if you notice there is a docs directory. In docs, it's returning a file or over the way, this is string called a readme.txt, okay? So, when you do a listing on slash docs, it will return readme.txt as a content. Now, in getdir, I'll just modify it. If the path ends with dot txt, then set the stmode as a regular file, okay? Or if it's a directory, or if it's anything else, then set it as a dir with the permissions bitwise odd, okay? And then return st. So, what happens is, whenever I go and do a listing on a directory which contains a dot txt file, it will set the permission mode as a regular file bitwise odd with the permissions. That's a small modification. And in this read method, if you do a read on a file, it has to be file, it can't be directory. On a file, it will return cell.data. Cell.data is there in init. It just is Lorentz and string, okay? So what should happen is, when I go to the directory, mount this again, and I go to docs directory, I'll get readme.txt as a listing, and when I try to read it, I'll get that string, okay? It can be binary, so it doesn't matter for this. So you need to write a program which can transfer binary files to the underlying LS program. So in this case, I'll just define a string. I could make a binary and make sure the function parameter calls and everything handles binary. Yeah, so I'll import, instead of toy, I import toy readwrite.readwrite as toy so that I don't have to change anything else. Just notice it. So what function calls are being executed, okay? If I go to docs, I have readme.txt, okay? So the if condition over there, it tells that whenever you see a string which end with .txt, set the mode as ifreg. When I try to cat it, okay? I get the lower-missing string back. So the condition what is there is whenever you try to read on an ordinary file, it should just retrieve cell.data and return it back, okay? So if I create any other readme.txt file over there, it will return this string because cell.data remains the same. So to add more features to file.txt, you can implement rest of these functions. There are more. So you can, for implementing symbolic link, for example, you can have like a link and unlink function for making anything as like a character device or a block device, you can implement the mkNord function. It takes the major and minor modes and the path or for managing the extended attributes, there are XATTR functions available. So you can go through the documentation and make your files some more feature-ful. So these are some file systems out there which are very popular. There is MySQL FS. So it just queries your database, right? You write a program which queries your database and saves all the files over there. Or there is SSHFS, which is very, very popular. There is Gmail FS, which was one of the highlights of Fuse back then. So what it did was, it used this username and password and logs into the IMAP server and gets the list of labels and the emails and exposes the entire Gmail as a folder inside your file in your computer. So you can simply do a CD to a directory and you see all the labels as folders. You see all the emails as files, right? So it just depends on what program you can write, right? So it's in user space. You are free to write or modify your program and LS will simply pass the file, the location to your program and it's for you to implement how you want to implement. That's the end of the talk and I'd like to take some questions. So when you mount other file system types, you can actually do it by amount by specifying the type by hyphen T option. So I believe it's some sort of alias. So how do I do it for this, for any file system that I implement myself? So I remember doing this once. You put it inside your ETC FS tab and you specify those mount point and the source. I'm sure documentation will have it. I've done it exactly once so I don't recollect the exact same type. But you can put in ETC FS tab and you can make it auto-mount and it will automatically mount it. And so does Fuse have support for asynchronous IO or is that something that the driver will have to implement in sort of... Since this API that was shown here was synchronous, completely synchronous. So let's say if I want to implement an API that is asynchronous in nature. So does Linux have that sort of abstraction available that Fuse or something else could use? It does, but I'm not sure if Python does. So mostly if you want to make like a real-world file system you would really want to use a C driver, which would support all this. Sorry? It does. But the Python driver has or not, I'm not very sure about it. So is the Python, it just calls the C, it's binding to the C API. That's what it is. Yes. Okay. Thank you. Any more questions? Okay. Is there any limitation on the size of the volume that you mounted or the partition table or anything like that? So as far as I understand with Fuse, there's no concept of size limitation. It's for your program to implement size limitation. There is however limitation on the number of the, I don't know what's called block size or something, which you can read at a time. So it's 32 KB or something. It's 8 KB or you can pass the option for big, big writes or large reads, which makes it 32 KB and larger size. So that's the limitation which is there in Fuse. Other than that, volume size limitation, I doubt it depends on how you write your program in the back end, in the user space. What is the size like if you can mount for, for example, ILO if I say other way embedded kind of area. Is it very, what is the size of this file system? Size of file system. There's this Fuse module, kernel module. Okay. Means you know like Idrap or ILO, all those kind of things. They'll have the very tiny ways. Yeah. Yeah. Means I want to see the better usage of this. Yeah. Like Raspberry or whatever, if somebody wants to try out, I'll go to the exact question because I missed the first. Yeah. The footprint. I don't have the figures for the footprint, but I'm sure the C library would be very small. There is Fuse module that you need to install in the kernel. That would be a standard side, known size, but I don't have the figures right now. We use NFS over the NFS module. Can we use Fuse? So you want to mount NFS file system over Fuse. Over NFS. Over NFS. So the backend file system is actually an NFS module. That, I don't think that should be a problem. You need to, so probably you need to implement the actual file system reads over there, maybe. I don't know the use case for it. Why would you do that? But still it's possible, right? You have a backend program, maybe in CE or Python, whatever it is, and you get a readDIR, getATTR, do actual getATTR on the file system and get it back. So NFS implements, I think, all the, it's POSIX compliant. So Fuse would run simply. I don't see why there is a problem. Hello. Is there a process which is running and what if that, so you said you did Python something and then you had the mount point. Yes. Is there some kind of process which is... Yes, let's see that. That's an interesting question. Okay. So there's a Python command which takes file as parameter and the mount point, right? So there you have it. What if I kill this and still try to access the mount point? Let's try that. Okay. Thanks for trending. Let's head for lunch. Jokes then. So all guys fed up, I mean, not fed up as in, you've fed yourself up, not fed up with the conference. That's not what I meant to say. I mean, you've had your food. Okay. Okay. So how did you find the conference? Just good. It was better the last couple of years. Why do you feel so? Too basic. Do you think you have expanded and you've raised your expectation? You've learned a lot over the few years, probably. Sort of expectation. Good, good. It's always been increasing and this year it has been. It's a dip. Okay. That's a harsh thing to say. Still, I would... It's always a mixed reaction, you know. You have a support. Okay. So how do you think you can improve it? Would you like to share experience? No, would you like to share some of yours and maybe share how would you like to... You would like to improve it upon it. Okay. That's excellent. That's excellent. So is there anyone who would like to share an experience right here? I mean, about anything, about PyCon, about your experience, using Python, something because, well, you've got this room, nothing's going on, and approximately for the next 45 minutes. People? Something? Swamiadeep? Got something to say? No. Okay. So I can tell about myself. Definitely. I mean, I've been doing it. Yes? Summary of what? Summary of what I was saying. I was just trying to... It was already in a summary. So, we were just trying to find out, I mean, how people liked the conference and what's going on and what's going on over here. So, we got one feedback that it's the performance has dipped compared to last time. So, just trying to find out why. Any more views on whether you liked it, how much you liked it? No views. Like, I... Let me sleep, dude. What are you doing? It's neutral. I mean, compared to what? We have already had that discussion. One point of view we have, I'm trying to get to more. Speak of people, it's a general discussion. I was just sitting over there, getting bored. I thought, let's have a discussion. So, how many people are associated with some sort of Python user group or community in their own cities, villages, schools, colleges, companies? Okay. One, two, three, four. Okay. So, you guys, is it your first conference by any chance? How many people have come to a conference for the first time? Okay. Why conference time? Any sort of... This kind of a conference where things are completely managed by volunteers. No corporation involved. Quite a few hands. So, let me tell you one thing. Like he said that, I mean, the quality of talks has dipped. But what do you expect from a conference when you come to one? What are your expectations? When you showed up yesterday in the morning, what were you expecting from the talks? Definitely not typical type talks. What were you expecting from the conference in general? The whole thing. What do you think the conference is all about? I mean, it's talks and it's workshops and it's sponsors for... Sure, but what else is there in the conference? Sharing experience. With whom? The audience. So, that's one person sharing his experience with the audience, right? Basically networking amongst the Python community. Exactly. Collaborate with people you reach. We meet virtually only. However, like we are a bunch of people who are sitting here trying to sleep. While there are thousand people out there who are there who have done... Possibly most of them have done amazing things and we are here trying to get charged up wasting our time. And I'm not accusing you of anything. However, that sort of bugs me. I mean, we can sit here and we can call it a car conference. But what is the effort that we are putting in to make it awesome? Ourselves. Good. Excellent. But that's the only participant. I mean, I'm like... Just trying to have a conversation here, guys. I mean, if you're too bored, just walk out silently. So, just one general request as in from somebody who's been attending Python for a very long time. The best experience is in talking to people. I have related this experience to everyone I have met. And trust me, the people I have met, they have helped me not only personally and professionally and they've helped me increase my intellectual level like trying to understand and help me get out of technical difficulties. And I've made good amount of friends over here. So that is what PyCon is as far as I'm concerned. And I hope you'll find good people to network with. And I mean, the next couple of hours you won't spend sitting alone just trying to slack off. Thank you.