 All right, good morning. So the rain has scared some people off, but that won't stop us in our journey towards file systems. So we finally get the last piece of the puzzle for the course, aside from the last thing we'll look at is virtual machines, which gets into fun page table stuff. So if you thought multi-level page tables were fun, well, with virtual machines, there is now nested page tables. But that's a topic for a later lecture. We'll probably spend this lecture and the next one on file systems. So file system, you're probably used to this right now. A normal POSIX file system looks something like this. So this layout of folders is actually a standard called the FHS, which is the File System Hierarchy Standard. So according to the standard, you have to have a special root directory. So that starts with a slash, and then within that directory are some defined other directories that must be there, like bin, dev, etc, home, and mount. And then within those, you can create whatever files you want. So the normal thing you do is in the home directory, each user gets its own directory within that that they should only be able to read and write to, which is what's kind of on the UG machines. And then usually when you use the file system or when you use your shell, you have a working directory. So there's a directory you're currently considered in and working from. So if I'm currently in the directory home and john, there's two different paths we can have. There's an absolute path and a relative path. So a relative path would be relative from where I currently am. And an absolute path would be starting with root. So if there's to do dot txt and I want to take a relative path to that, well to do dot txt is in my john folder and that's currently where I am. So the relative path would be just maybe dot slash to do dot txt or maybe just to do dot txt by itself. Then the absolute path would be starting from root. So it would be slash home slash john slash to do dot txt. And then for USB, if I did a relative directory for USB, well if I'm in john john I have to do dot dot which would take me to home then another dot dot which would take me to root and then slash mount slash USB or if I just did the absolute path to USB it would be slash mount slash USB. So that's kind of what we're all used to and we'll see in later ones that the dot and dot dot are actually entries in your file system. So there's kind of special file names and the only thing that's special about them is their name and how someone defined to use them. So within all the directories there will actually be an entry called dot which is supposed to refer to the current directory that all that this represents. And then dot dot is the parent directory and then the tilde, the little squiggly line is the user's home directory which is usually also an environment variable called home and then relative paths are always gonna be calculated from the current working directory which may just be set with an environment variable like PWD. So fun fact, does everyone kind of know on Linux or at least the UG machines that if you start a file with a dot that's like a hidden file. So that was actually kind of a bug that became a feature. So when someone wrote LS, well they didn't wanna show dot and dot dot because those are defined ones you don't really care about seeing them. So if you're display all the files to user well I would want to ignore dot and dot dot. Well if you're a very clever programmer you'll be like well those are two special cases I can generalize it. So both of those start with a dot. So that programmer just wrote that hey if it starts with a dot don't show it. Thinking that it would only cover dot and dot dot and no one would actually make files that start with a dot. But then of course someone made files that started with a dot and then they couldn't see them and that's just how it became. So now if you just start a file with a dot it's understood that that should be a hidden folder but it all stemmed from one programmer trying to be really efficient and handle both those cases at once. So that is the fun fact about how fun fact about how hidden files came about. So for accessing the content in the files well you can access files sequentially or randomly. So if you just open a file like what you've been doing and you read from it you're reading everything sequentially. So each time you call read it would advance the position in a file. So the next time you call read you continue reading from where you left off. And then if you write to the file you could be doing the same thing where writes just happen from where you left off but you might start writing from the end of the file instead of writing to the beginning of the file otherwise you might be creating a new file in that case. Then another way to access files is just like randomly. So you might just want to randomly jump all over a file and read and write in any order. And for that you would have to specify a position required for each operation which we don't actually know how to do yet because so far unless someone is doing something really tricky everyone has just been accessing files sequentially. So they just open them, they read and wrote. So the first call we're all familiar with that is the open system call that takes a path name some flags and then a mode. So in this by default you just get you get the positions that starts at the beginning of the file. So if you read from it you read starting at the beginning of the file and it updates the position. If you write to it it would start writing at the beginning of the file maybe overwrite some stuff. So usually if you're writing to the end of the file and you want to just throw that information at the end one of the flags you can give is this O append flag and that would just make that position point to the end of the file. So any write system calls you do will not overwrite anything it will just go directly to the end of the file. So the second system call is the lseq which is the system call you use to actually change that position so you can actually randomly access files. So lseq will take a file descriptor and an offset which is just a number to change the position to and what that change represents depends on what that third parameter is that whence. So it can take one of set seek cur or end and cur is just short for current. So if set if that whence is set is seek set then that means that offset is absolute. So if you choose seek set and they're offset is zero it's always going to go to the beginning of the file if you set seek set and then have your offset of 100 it's always going to go the 100th byte. Then the cur which is just current and end is relative to where the position currently is and relative to where the current end of the file is. So what you could do is if you wanted to just advance say 100 bytes you would set once to seek current and then your offset to 100 and that would jump 100 bytes from where that position currently is. So if you just want to skip 100 bytes in the file or however long or of course this could be negative if you want to go back. And then seek end is just from the end of the file so if you want to read the last byte of the file so if you wanted to do that for whatever reason you would set once to seek end and then offset to negative one or to zero to read the last byte negative one for the second last byte and so on. So they're both relative and so yeah current and end are relative and then set is absolute and then you have an offset. So this will actually change the position where it will read and write to that file next. Everyone good with that because it will get weird. So we saw this just for completeness that we accessed it in lab one so you had open dir, read dir and close dir and you'll see the actual structure of this possibly in the next lecture but it's just a special thing you open it and then you can call read dir over and over again until it returns null and that will give you some type of directory entry. So just remember there's like a D name and stuff and we'll see all that directory entry pop up in the next lecture but that's just so you know that it will probably come and we can get reminded of it. So now we can talk about what's actually in the process control block for more files. So within the, so every process has a process control block that stores things like its process ID, its scheduling information. If it, since it can context switch it would save the registers from when it switches. It would have page tables associated with it. So we've learned a lot more about the process control block than what we started with in like lecture four. So now we know a lot of what's in and the last thing we actually care about about what's in it is that file table. So the file table is stored in the process control block and it is just an index into this thing in the middle here which we'll see later but that name is the global open file table and that is maintained by the kernel. So within a process all your file descriptors just point to some entry in there and every entry in this table has three things. It has a position, a flag and a V node and a V node is just a virtual node. It represents something that you can read and write to. So in this example we have two processes, process one and process two in process one. It's got three open file descriptors but we'll only talk about the first two. So the first file descriptor zero could be pointing to this entry and this entry would have a position flags and then have a V node that points to some file let's call A and then it's file descriptor one would have another would point to something else that has its own position flags and a V node that could be pointing to file B and then in process two it's file descriptor zero could also have its own separate position flags and V node where that V node points to the file B. So two V nodes can point to the file B and in this case both of these entries have an independent position so they could both sequentially access the file both of those processes could read the entire file because they have their own independent position so each time they read from it they'd read from that file but because they have their own independent position they could actually read the whole file because they're not sharing anything but there are cases where things will get shared and that it will be a source of problems. So each process contains a file table it's process control block and the file descriptor is simply just an index into that table, that's all it is that's why it starts at zero and that's why the numbers are so low. So each thing in the kind of local file table will just point to a system wide global open file table and that is maintained by a kernel. So that is also why fork your file descriptors are all kind of shared because even if you copy the local file to table well you're just copying what it points to so both the processes are pointing to the same thing and that way they can share so there's like a layer of indirection there there's a reference. So the global open file table or goff if you wanna call it that holds information about the peak seek position and flags and then points to vnode vnodes are just anything that needs to support a read and a write so it could be an actual file we know from lab four it could also be a pipe which is basically just a buffer so a buffer is something you can read and write to and then we saw with sockets you can also read and write to sockets as well so they can be regular files, pipes, network sockets or anything else you can read and write to it's a really general thing. So again remember what happens on a fork so the process control block just gets copied on a fork so specifically for us the local open file table would get inherited completely so they both have their own local file table but at the beginning after the fork they're both copies of each other so if you modify file descriptors in one process after the fork it doesn't affect the other ones again should have gotten some practice with that in lab four but they both point to the same global open file table so this is what it would look like if say process one had a file open had file A open while whenever you open that file it would create a new file descriptor for you that points at a new entry so that new entry would have its position, flags and V node and then whenever we fork we would create a process so let's call process two just the child process to make it easy and process one the parent process so when you fork the child process is also going to have a local open file table where file descriptor zero is going to point to the exact same thing so now they are both sharing a position which will lead to some interesting things if you are not careful about your file descriptors so you could have a file you could open a file in a process so say you are opening like readme.txt or something like that you fork and then in both of the process so what would happen if in both of the processes I call read yeah yeah so there is a because they are sharing the position if both of them are trying to call read they both of them will not read the entire file so there is a few scenarios that might happen because they are sharing this position and the curl is going to make sure everything is read safe and everything like that so what could happen is one process reads the entirety of the file and then updates the position to the end of the file and then if the other process tries to read it it's going to get that end of file signal where read just returns zero so one process might read the whole file the other process gets nothing and it might also be switched and be the other way or if it's a really really big file what may happen is one process reads a chunk of it and then updates the position and then the other process reads another chunk of it so one process will have one chunk of the file the other process will have another chunk of the file and they'll be mutually exclusive so each one will only get like a bit of the file and it will probably make no sense to them so that's some of the gotchas for sharing so since you fork and you just you're pointing at the same global open entry that position is actually shared between processes and that leads to very interesting things especially when we have that seek system call before so remember the seek system call just updates the pointer so because they're sharing the position if you forked and then in one process you seek to the beginning of the file well that would also seek to the beginning of the file for the other process as well so you could interfere with each other it could just seek to the end and then neither process reads any of the file so you could get some very interesting things if you are sharing a file descriptor and another process decides to seek on you and do some interesting things because you're never going to figure it out so the alternative to that if you want to be independent so you want to have your own position is every time you call open that creates a new entry in the global open file table so you would want to perhaps just reopen the file or only open it after the fork instead of before the fork if you for sure did not want to share this position so we can look at that in an example so say we have something like this where we have a parent process that goes through it does open to do dot txt opens it as read only and then there's a fork and then after the fork there's now two processes so this is kind of review for processes and then both of them open b.txt so assuming there's no previously open files not even the standard ones how many file descriptors does the parent process have open to and what files are they? Okay, yeah, to do mb.txt what about the child process how many files does it have open? Two as well, to do dot txt and b.txt but how many global open file tables entries are there? Three, right? So if we have this, so if we have open to do dot txt whoops, fork and then open b.txt well in our original parent process so we'll just call this process control block one parent so if we assume no file descriptors are open then, or no file descriptors are currently open then open so you can just return the next unused one so it would return file descriptor zero and then an open will always create a new entry in the global open file table so here is the global open file table so it will have a position, some flags and then a vnode and that vnode is actually going to point to to do dot txt and file descriptor zero is just going to point to this entry here and then when we fork that would create a new process, a new child process so it would create a new process control block for the child and it would copy the table so the local open file table so it would also have a file descriptor zero which points to the exact same thing so now they are sharing that same position so if one read from zero it would advance the position then the other one read it would start reading based off where it picked up from or if one of the processes did that seek system call it would update the position and then it would affect the other process but because we have the open b.txt after the fork both calls from the process are going to create an entry in the global open file table and because they're processes we're not sure which ones are going to run first so let's just assume the parent runs first and creates an entry in the global open file table so it would have a position, flags and then vnode and that vnode would point to b.txt so and then the file descriptor that would be returned for that process would probably be file descriptor one so file descriptor one would point to that entry in the global open file table and then the child would at some point execute and then do its open call which would create a new entry in the global open file table and it would have a position, flags and then its own vnode that also points to that file and then it would get file descriptor one that points to it so each process has two entries in its local file tables so there's two entries here two entries in the child and then in the global open file table there's a total of three entries so any questions about that so there's two entries per process and then three entries in total and here's again so you have it on the slide so you don't have to write it down okay so the next topic would be hey I need to store these files somehow so we kind of saw that the structure of the disk devices are they give you a bunch of blocks that are kind of like pages they may or may not be the exact same sizes pages but you can allocate them like pages so if you we want to store files how should we store them so one thing we could do is something called contiguous allocation which might be the first thing everyone thinks of so say in this example we have a green file a red file and a blue file well to keep track of where the files are the green file you could just say hey it starts at say there we'll call these big things of information just blocks you can say hey it starts at block zero and it's three blocks long then for the red one hey it starts at block seven and it is six blocks long and then same thing same deal for the blue one so would that be a good idea for files which hint this is kind of related to memory allocation yeah yeah so wouldn't it lead to a lot of fragmentation so that is the huge issue with this because also unlike memory allocation where you just give some memory back and it just uses it well files can grow and files can shrink and files can get deleted well memory can get deallocated but files can grow and shrink kind of at will so in this scenario what would happen if I wanted to make the red file bigger yeah so I have a few options here if I wanted to make the red one bigger well assuming this is all my blocks I couldn't actually move the red one because if I move it down oh no I could move it down so I could move it after the blue one and then give it another block or I could move the blue one which is probably smaller and then give the red one another block but I'm moving possibly a lot of information here and there's no guarantee that's gonna get any better anytime soon and the other option would be hey your operating system can just be like yeah sorry I can't do that that's too bad which if your operating system did that and you worked on your lab you made your file bigger and then you tried to save it and your kernel just said yeah sorry can't do that you probably wouldn't use it anymore so probably a bad idea especially we know that that fragmentation issue is going to be a very large issue and we wanna use our disk as efficiently as possible so it's gonna be fast as long as there's no modifications and we can lay things out properly and it's space efficient if we need to keep track of the contents of a file well we just need to know what block it starts with and how many blocks it takes and that's all we need to represent a file and then we could just read a file by chasing down those so if we wanted to access any block you would know exactly what block you need to access by how big the block is and what byte you're accessing so if I want to say my blocks in this case are the same length as a page so they're four kilobytes well if I have to access byte 100 it's gonna be on the zero width block and then if I wanna access byte 5000 it'll be on the second block and I know exactly what block I need to go to by just what byte I'm accessing so that's really fast as well but you have the problem where files can't really grow that easily there's gonna be lots of external fragmentation when files are deleted or truncated we might have these big gaps and you probably don't want them and we can do better for files but in this case you're gonna see this pop up every time that for file systems generally they only care about blocks and files may not nicely fill a block so there's always going to be some type of internal fragmentation because a file might not perfectly fit within a block and waste some information so for all these we're going to have some internal fragmentation as long as we just care about blocks but our goal is to not have any external fragmentation so if we don't want any external fragmentation we can do the same idea that the Linux kernel has for just giving pages to processes and do something like a link list of free pages so this is called linked allocation so unlike the kernel where the kernel just has to keep one track of it and then allocate it as it's used we would have to do something like that for every single file so it might not be the most efficient but this is one thing you could do that would actually work so what I could do is within a block so say all these green blocks represent that green file all you would have to specify in order to sorry all you would have to specify to say where that file is is you would specify the first block and then within each block would have a pointer to the next block so there'd be the blocks would be a little smaller in terms of usable size so say they were sizes of pages again so like 4,096 bytes well if you wanted to store the pointer to the next block on the block that and the pointer is like 4 bytes well then you could actually use 4,092 bytes for storage and then the 4 bytes are for the pointer to the next block so in this case they would point all and then once I hit the end of the file that pointer would be null or something like that to indicate that there's no more blocks for this file so it's a bit better so we got rid of our external fragmentation using that and kind of recycled the idea we got from the Linux kernel it's really space efficient because for every file all you need to store is the first block and then you can figure out what the rest of the blocks are from them but that is also the drawback where your random access is going to be really really slow so to traverse anything well I can't just go to block I don't know block 5 of the file directly I would have to start at the first block and then traverse all of it until I got to the block that I want to get to and we know this is going to be even worse because let's assume they were just in memory and this was pages well every time I access a pointer there because everything's so far off and all the pointers are on different pages it would be like having a TLB miss every single time so even traversing it is going to be really really really slow so what would I do if I wanted to speed up that traversal so if I have to access it on different pages and that's slow could I do something a bit better hash map so we can do it not being that fancy so if all the pointers are stored on different blocks and I want everything to be closer together where should I store all the pointers yeah I could essentially store all the pointers in an array unlike a single block itself so that's what a file allocation table does so all it does is move the pointer from the block itself to its own its own space and then each array element represents a block and it would have a pointer to it so in this case before I had block zero point directly to block six and that pointer to block six was stored in block zero but instead of doing that I would store all the pointers in another in just a big array so if I access block zero or if I access block zero I could access its pointer directly and then just jump a few bytes so it would point to six and then from six I could follow that to point to two and then from two points to 13 13 points to nine 18 and then nine points to 18 and then that's the end of the file so it's the same idea as linked allocation I'm just moving it to a separate table so who here has formatted a disk before and saw on the options there's an option called fat seeing a few people so this is fat so fat is file allocation table and that's exactly what this is so fat 32 is means that the pointers to blocks are 32 in size 32 bits in size so that's all it is so each file would have its own file allocation table that essentially has a link list of all the blocks that are used so this is a bit better than linked allocation we again have no external fragmentation still have internal but we have we've sped up our random access time by a lot because that file allocation table can be held in memory and then we get all the advantages of our caches and everything like that the big disadvantage is that the fat size is linear to this size so every single file is going to have to be able to reference every single block in memory and most of the time that file allocation table is going to be empty minus the blocks that are actually used to do the allocation but no matter what every file is going to have to be of a reference every single block so the file allocation table is related to how big the disk is so the bigger the disk the larger your file allocation table is and the less room you have to actually store files so this is still using the idea of a link list so what could we do if we wanted to do something even better than a link list so what is generally faster and easier than a link list a tree what's simpler and faster well a stack what about just an array an array would be simple why why am I storing the entire like list of pointers for everything why don't I just store that for this file directly block zero is block one block one of this file is six block two of this file is two block three of this file is thirteen why would I just store that directly in an array and then I can just look it up directly I don't need a hash table I don't need a tree let's use a good old simple array so that's exactly what indexed allocation is screw this why would I use a link list try and figure out what block is which I'll just store which block is which so indexed allocation is for each file it would store just an array of what what block represents what block of the file so you would actually use a block to store the array which steals an idea from our page tables where we have a page table that fits on a page well we store our indexed allocation map on a block so this red block could store our array pointers to blocks and then we would just index them in the order that our file needs them so if our file takes six blocks here well we just look them up directly so block zero point to block zero on the device block one of the file would point to block six on the device then block two the file would point to block two in the device etc so that's going to be way faster that's an array we got all our nice random accesses everything is really really fast and we don't have a such a problem anymore because now this indexed array is only proportional to the size of the file which makes sense so we have to keep track of how many blocks represent the file but it's not proportional to the size of the disk anymore so we got rid of that weird dependency so that's good so for index allocation we still have all your nice advantages our files can still grow and shrink at will still have no external fragment or yeah no external fragmentation still have internal fragmentation because that's the trade-off we use for these big devices but now we have nice and fast random accesses but now we have another disadvantage is where oh if we're using that trick of storing all the pointers on a block well we have a limit so with file allocation table a file could uh... a file could span the entire drive if you wanted to but now we're limited to the size that we can the number of pointers we can fit in that block so let's take a look at that scenario so if we assume we have a block that we used for indexes so that block will just store pointers and that's it so no no other information whatsoever will just make it simple where it just stores pointers to blocks well let's assume a block on our disk is eight kilobytes in size and then a pointer is four bytes well this will look a lot like page tables where we can kind of calculate the maximum size of a file that can be managed by a single block so if our block size is eight kilobytes well like page tables powers of two are probably good so what is eight kilobytes in powers of two yeah two to the thirteen so our block size is two to the thirteen and then our pointer size was just four bytes what's four in powers of two two to the power of two so knowing that how many pointers could we fit on a block yeah two to the eleven so that's how many pointers we could fit in a block because two to the eleven well that's the same as our block size divided by our pointer size and since we know fun exponent math that's just the same as two the power of thirteen minus two which is two to the eleven so I can fit uh... two to the eleven pointers and then each of these pointers points to a block how big is a block two to the thirteen so what is my maximum file size sixteen megabytes yeah or if I not skipping steps it is to the eleven which is the number of pointers I have times the size of our block so it is two to the thirteen so that's the size of our block so that together is two to the twenty four so if we want we can rewrite that in megabytes so two to the twenty is a megabyte so it would be two to the four megabytes and MBI again is the powers of two ones the megabytes if you just write it it's like somewhat ambiguous depending on who you talk to but MIB is definitely powers of two so this would be sixteen megabytes so if we do that the maximum size of our file is sixteen megabytes would anyone like to use that operating system so maybe that was cool you know in the seventies or something like that or possibly before that but not gonna quite do it today so that will be what we discuss in our next lecture and hint we'll probably borrow some ideas from page tables so you thought page tables ended at virtual memory well we use the same idea all over the place and you might actually use the same idea yourself when you have to design one of these things so we'll see that in the next class but here again that's just so you have reference so our maximum size of file sixteen megabytes and no one wants something like that because you know everyone wants to see some videos or store your games or whatever and they're gonna be in gigabytes so so we talked a little bit about file systems and they're again one of the things that enable persistence so this are the hardware part and file systems are the software part so they just describe how files are stored on disk which will go more into detail with next lecture but we saw some allocation strategies and ways to keep track of what blocks belong to which file today but API wise we learned a little bit more you can open files and that actually creates a new global entry into the kernel and then for each process just points to that global entry that has its own position and then when you fork when you fork both processes are going to point to the same entry so they're going to share their position which may mean weird things happen and then we saw some different allocation strategies contiguous linked fat which is the file allocation table which was the same thing as linked but we just store it all all the pointers together and then we saw it indexed and then we'll see some more interesting ones next lecture so just remember I'm pulling for you we're all in this together