 All righty, welcome back to Operating System, so we get to discuss file systems now. So, me telling you I was curving down on the midterm, I didn't even scan them yet, that was a joke. It's fine. So all the midterms scanned as of this morning. TAs are doing that, they're grading as we speak right now, so hopefully it'll get back to you sooner rather than later. Usually they turn around times pretty quick. They're grad students, so they're kind of like you, they're nocturnal, so they'll probably grade at night and on the weekend and things like that. So, hopefully it'll be done by Monday, we will see. Usually they're pretty quick. So, now we get to talk about file systems. So, we've all used file systems before, this is probably somewhat of a review, but the layout in POSIX looks something like this. There is actually a standard called FAS, which is the file system hierarchy standard, very, very creative name. So, the file hierarchy standard says that everything must begin with a directory called slash, and that special directory is called the root directory. So, that root directory, it has to have a bunch of other folders in it, or directories, and according to the standard it says what should be in those directories. So, there's supposed to be a bin directory that has all your binary files, all your programs, all your executables, things like that. There's a dev directory that's supposed to represent all the devices on your machine, typically the kernel will represent all of them. So, this will represent things like your hard drives and things like that, because everything is a file. ETC, which will generally have all your configuration files. Home directory that will have a directory for each user on your system. So, if you're using the UG machines or something like that, you'll have your own home directory. It might be in a different location, but it follows the standard. For me, it'd be something like home slash home slash john inside of it would have some files. So, for example to do.txt, and then there's also this mount directory which should have things in it. Like when you plug in a USB drive, it should be created in this mount directory. So, all of our processes will also have a working directory. So, currently if my working directory is slash home slash john, well we have a little bit of practice or hopefully some experience with absolute and relative paths. So, if I was in slash home slash john, what would I need to type to get to to do.txt? Yeah, so it's a file. Yeah, so if I wanted to access it or just its path is just to do.txt, or have we all encountered the special dot? What about if I started with the dot? If I started with the dot, what would it be? Yeah, dot slash, so dot is supposed to be this current directory, so I'm currently in home john. It's a file in there so I could do dot slash. What about if I'm in home john, what about the relative path to this USB directory? Close, yeah. Yeah, so if I'm in john, if I do dot dot that should be up one directory. So, I could do dot dot which brings me to home and slash dot dot again, which brings me all the way to root and then slash mount slash USB, which is longer than absolute. So, absolute paths start with that root character always. So, starting at root, how do I get to to do dot txt? Well, it should probably just be slash home slash john to do dot txt. And to get to USB would be just be slash mount slash USB. So, any questions about that? Alright, so there are special symbols which you will discover what they actually are in lab 6. They are not actually that special, but they are actual entries in your directory. So, there is one called dot, dot is the current directory and dot dot is the parent directory or one folder up. So, this is where it comes into fun history land. So, have you noticed on like link systems, you can create hidden files if you just start them with a dot. Anyone know why that is? Hmm, so, whatever you type ls, typically you don't want to see dot and dot dot. Alright, so if I wanted to ignore them, if I wanted to ignore just there, those I'd be like, well if the directory is dot or dot dot just don't show it. That's not efficient. So, some smart programmer was like, well I can make this a lot easier. I can do it in one line of code. And his one line of code was just if it starts with a dot, which covers the case of dot and dot dot, then just don't show the directory. And then after that, oh, guess what? Well, if you just named a file starting with a dot, it also wouldn't show. And that's how hidden files came along. Someone was trying to be clever and that bug became a feature. So, that is literally a case of bug becoming a feature. So, if you're on Linux, you can thank a lazy programmer for making dot represent a hidden directory. The other special one is this tilde character and that's always the current user's home directory. Might be in this environment variable called home. And relative paths are calculated from the current working directory, which is given by this PWD or print working directory, which is a property of the process. So, for files, so far when we've opened them and read them, we have only read them and wrote them sequentially. So, for sequential access, each read just advances the position within the file. So, if my file has the contents hello world and I read a few bytes from it, I would read hello and then if I do another read system call, it would continue where it left off and I'd get world or the remaining part of the file. Right? That's how we've been using files so far. Anyone disagree with that? Because things probably will get weird here. So, how writes happen? Similar way, but they're just appended to whatever the position is and it's updated afterwards. So, if I open a file and I write hello to it, well it would advance the position, write those five bytes or however long it is, and then if I do another write system call to it, it will just append it to the end of that and keep on going and I can grow it on and on and on and on again. So, you can also access a file randomly, so I could just open a file and go to any old random byte and start reading that, but we don't know how to do that yet. So, luckily there is a system call for that. So, here is our friend opened. So, open like takes a whole path name, flags, so if I want to open this file for reading, for writing only, for reading only, if I want to open it for reads and writes, whatever I want to open it for, those are all the flags and one of the other flags you might be able to put on is this oh append flag, so that will move the position to the end of the file initially. So, if I just open a file and then I write to it, by default I'm going to write to the beginning of the file, I'm going to overwrite whatever was there before. If I open a file with this oh append, and then I do a write system call to it, it will just append it to the end of the file and I'll just grow the file. So, there is this l seek system call in order to change the position that I'm going to read and write from the file. So, it takes a file descriptor that I want to manipulate, an offset, which is just the number of bytes from the position that I want to go forward, backwards, or stay the same from and then the third argument is this once parameter, which is basically just a slightly scuffed C enum. So, it has three choices. It can be seek set, seek cur and seek end. So, if once is seek set, that means the offset you give is relative to the beginning of the file, so it's absolute. So, if I have once equal to seek set and an offset equal to zero, that means go to the beginning of the file. If I have it seek set and then offset to 16, that goes to the 16th byte no matter where that position currently is. If I set the parameter equal to seek current, that offset will be relative to wherever I am currently at in that file. So, if I'm in the middle of that file and I want to just back up four bytes, I can set when equal to seek cur and offset equal to negative four and I'll just back up four bytes. And then similar to seek end, I could set to seek end, if I set the offset to zero, I go to the very end of the file. If I set to negative one, I go to the second last byte of the file, so on and so forth. So, any question about that? Because this... basically there's an internal position of where to read and write from a file, and things will get weird. Yeah, the offset. So, based off this once, it's where I'm going to move the position to. So, it's just the number of bytes to move it, depending on what this once is. Yeah. So, you have to move somewhere that's valid within that file. So, if I do end and then one that's beyond, it'll probably give me an error. Or maybe if I'm lucky it'll grow the file, probably it'll give me an error. Pretty sure it gives you an error. Yeah, so depending on what it is. So, if I do seek set, so that's relative to the beginning of the file and I do negative, it'll probably give me an error before the beginning of the file. Yep. Alright. Any questions with that? So, basically there's a position. It's going to get weird once we bring Fork into the picture. Because, guess what? Fork is always in the picture. So, this is what we did before. So, you get to explore what a directory actually is. Because internally this opendir crapper, well that will open a file descriptor and a directory just is a file descriptor like anything else. It doesn't open system call but C tries to, or the standard C library tries to abstract it away. We did this and your lab was a one, I think. So, you're going to figure out how this actually works as setup for lab six. And it will be cool. But for that, we need to talk about what is in the process control block what the file descriptor actually is and be a bit more explicit about it. So, each of your processes in the process control block will have a file table in it. And so far I just said think of that as just a pointer to a file. Now we get to figure out what it actually is. So, for each process it would have a file table and it would point to things but it doesn't point to just a file it points to some other information. For instance, in process control block one file descriptor zero points to a position so that's what L-seq will manipulate where I currently am at in that file. Some flags that I used when I actually opened the file and then a pointer to a V-node which sounds a bit weird. And what that represents is just anything you can read and write bytes to. So this can represent, you know, a file, a real file a pipe can represent a whole bunch of stuff they just in order to abstract it well they just called it virtual and it's anything I can read and write bytes to. So in lab six we will see what the structure is called. Well we'll actually do that in the next lecture. So if this V-node represents a real file that is on the disk well it's actually called an I-node and we'll see what an I-node is in the next lecture but for now we'll keep it as a V-node so this V-node points to the actual file that I want to manipulate through this file descriptor. So you might notice things get a bit weird because also in process one file descriptor one could point to a new entry in this table with its own position, its own flags and its V-node could point to file B and then I could have another process, process two and its file descriptor zero could point to a completely independent entry here with its own position, its own flags but this V-node points to the same file B so that way one process does not interfere with the other process. So that position if process two changed the position it wouldn't affect process one even though they're actually accessing the same file they could be if they do a read they might actually read different bytes from that same file still go to that same file. So any questions about that? Alright, so a file descriptor, all it is is an index into that table and the kernel will keep a global system-wide open file table sometimes referred to as a GoF because GoF, why we called things what we call them we're just not creative. So the GoF holds information about the position, the flags and then points to a V-node and that V-node just has to be something that actually supports reading and writing bytes to. So the V-node holds the actual information about the underlying resource that it represents. So it could represent a regular file which we'll see how that's actually laid out. It's called an I-node next lecture but it could also represent pipes, network sockets, shared memory sometimes anything else we used as a file descriptor in this course. I think that was about it. Anything to read and write bytes to. I think that was pretty much it that we've encountered so far. Yup. Alright, so now we have to talk about forking because this will get slightly weird, slightly maybe not weird. So when you fork the exact copy of the process that whole process control block you get a copy of it. Specifically for us that local open file table will get inherited so it will just get copied. That's why all your file descriptors are the same numbers and they point to the same thing we just weren't super clear about what they point to. So we said they point to a file which was kind of true, kind of not true so they do eventually point to a file but they also have a position in some flags which point which are stored in the global open file table. So in this case this is what happens when you fork. So if I fork and process one has file descriptor zero and it's pointing at this so always to the left here within the process control block these are local open file tables and in the middle is the global one that the kernel maintains. So in this case process one and two point to the same entry so they'll be sharing that position and they also that entry also points to file A so now this is what it would look like after I fork in which case they're sharing that same position so say I had two lines in the file what would happen if process one and two both did a read system call what could each process see hmm yeah it kind of depends on how many bytes but is it ever the case that both processes will see the entire file no because they're sharing that position right so it kind of depends how much I read since they're sharing it well if one of them reads it could read the entire file in which case it would update that position to the end and then if the other process tried to read it it would get read of zero saying that you've read everything from the file nothing else to read so one process reads everything the other process reads nothing or it might be in the case where one process reads one line and then another process reads the other line and you get like half and half or it might be any combination then depending on how many bytes you read and write to which seems like an odd behavior for forking but it's actually been useful so far because well guess what when you fork and you're sharing you know standard out standard in and everything for your terminal well this is actually the behavior you want if you type a you don't want every single process to actually consume that a and do something with it it's only the one you're typing in so only one of the processes should actually consume what you've written instead of every single process that got inherited along the way so might seem weird but this is actually useful so any questions about this one although if it represents a real file and then you only read half the file that might be like kind of hard to debug and in fact I guarantee at some point in your life if you if you take this course to heart and you're forking you'll probably encounter this problem and you're trying to access the same file in your forks and they're actually sharing the position and you're going to get some weird results and then you're going to have to be remember this lecture and be like oh yeah crap I should probably just reopen the file and that way I have my own independent position so whatever you do an open system call you'll always get a brand new global global open file table the only way to share an entry is through forking all right so yeah yep yeah so pipes are just the exact same thing so a pipe would have a position some flags and then put that vino would just point to the actual pipe so yeah a different three bytes so this is the same thing as if it represents a pipe so if I create a pipe and I have two processes and they both call read on the pipe I don't know which one's going to read information but they're not going to read everything I put into the pipe so one might read a little bit one might you know so you won't get any duplicates and it's kind of like a data race well it's kind of just non-determinism because you don't know because essentially it's whoever calls read first which is up to the scheduler which it's fun all right so yeah just remember if you fork you're sharing that position and also well other fun things that could happen is because you're sharing that position that lc system call that updates the position well it would update it in your parent as well because you're pointing to the same entry so any change to the position would be shown in both processes so you could really screw with your parent if you want just kind of like in real life so you could fork since you have a copy of all the file descriptors you can just throw it to the end of the file and it won't read anything ever if you want to make people not be able to debug anything ever you can start doing that and get some job security because yeah they probably don't know how that works so yeah and if you open the same file in both processes after forking then well you have two independent entries in the global open file table they just point to the same file so let's see if we can do this so given my program is something like this I open so say it has no file descriptors open for whatever reason if in my process I do open2do.txt I fork and I open b.txt how many local open file tables do I have per process and then how many global open files do I have in total so I will give you a second to think about that and then we can go over it alright any guesses yep yep yeah so two local per process and then three globals right everyone get that alright I won't even draw it then so I'll just go to the next slide so in each well actually no I'll draw it so if we have process 100 that starts executing well it opens to do.txt which would make a new global entry whoops so as part of the global entry it would get a position get some flags and then a pointer to a V node and then that would point to to do.txt wow that's great writing well let's call it to do so I don't try and cram it in so in process 100 well file descriptors will just do the lowest index first so it would have descriptor zero and it would point to this entry so now when we fork we create process 101 and process 101 will be an exact copy of the parent at the time of the fork so it would also have the exact same local open file table which points to the exact same global entry and now after the fork I don't know which one will execute in this case doesn't really matter they will each create a new entry in the global open file table with its new position new flags and then a V node which points to B.txt and say process 101 open first so file descriptor one would point to this entry and then eventually if process 100 executes it would have its own unique position its own flags and then its own V node and then this V node would also point to B.txt and it would get its own local entry one so any questions about how that works so now both of these processes well they can read B.txt its entirety because they have their own position but they are sharing file descriptor with the same global entry and file descriptor is zero so if they both tried to read from it they would read some bit of the data and they would update that position and then the other process if it reads would just carry on from that point so any questions about that one so that's really what file descriptors are so that position will become it wasn't really important before but once you start moving around and stuff it will become important especially if you're trying to if you start creating multiple processes and then you're reading from files that were already open this I guarantee will surprise you so that's why it's in this course alright so now we need to go back to file systems and how we actually store files so in this case we kind of looked at what an ssd is it has a bunch of pages which are like 4 kilobytes the generic term usually we refer to them as blocks because they might not exactly be 4 kilobytes they might be 8 kilobytes or something like that so we just have blocks of storage we can use that will be like 8 kilobytes or something so given those blocks we have to be able to form those blocks all together to form a file and describe our file so in this case each box is a block say it's like 4 or 8 kilobytes something like that and I have a green file a red file and a blue file now if let's say we give the green file a name just call it like the green file how would you describe to me where this green file is on this disk yeah yeah the first 3 blocks what about the red file kind of looks like an array right how do you describe an array yeah starter end or starter how big it is something like that because it's all contiguous you don't really need to tell me that much just start how big it is so if I was to describe the screen file and these are all blocks that are in order I could say it starts at block 0 and it's 3 long for the red file starts at block 6 index 6 and it's 6 long blue file starts there so fairly concise way to describe a file what about if I want to make the red file bigger yeah I'd have to move it or I could move the blue file if I wanted but I just have to get it out of the way because everything needs to be contiguous so moving files this might actually suck for that you would especially not want to do this if that red file was several gigabytes and you're trying to add a little bit more then you have to move a gigabyte and then you add a little bit more and then you have to move it again and move it again move it again probably a bad idea especially since files are generally big so it's space efficient if we want to describe a file because it's like an array you only need to tell me the starting block and the number of blocks we need to store it it's got pretty fast random access so say I want to access byte 10,000 of the file well since everything is contiguous it's like an array index that calculation is really really fast and really easy to do in this case what block it is on is just whatever the offset is plus the block size and then you take the floor so if they were like 8 blocks were like 8,000 in size and I was looking for byte 10,000 well it's not going to be on block 0 it's going to be on block 1 and I could just do that calculation for any offset no problem but this is kind of bad files can't grow easily and we have to talk about fragmentation I think it's the first time it showed up in this course whenever you see the word fragmentation it usually has to do with storage or memory and it basically means wasted space so internal fragmentation it's relative to the blocks so internal fragmentation means I'm wasting space within a block so for instance if my blocks are 8 kilobytes and I need to store a file that's only 12 bytes well I still need to use a block to store that file because I can't do anything smaller than that so all the other bytes aside from those 12 in that block are essentially wasted space what you call external fragmentation is wasted space wasted blocks or anything between blocks that's wasted so in this case let's say I made the red file smaller by one so there was just a whole of one between the red file and the blue file well that space is going to be wasted if all my files are bigger than a block I can never fill that whole up it's just there forever just wasting space what external fragmentation is and it becomes a problem especially once we go later and talk about memory management or basically how malloc works because malloc has the same issue as this alright, whoops alright so if that isn't great well how can I store a file slightly better than that so I kind of ruined it already so I could just store the file as a link list so we like link list or at least evidence seems to say that from like lab 4 we kind of like link list so we could store a file as a link list so this green file now it doesn't have to be contiguous I can just given a block I can just point to the next block in the file and eventually it'll end point to nothing and then I know I'm at the end of the file so if I was to describe this green file what would you need to tell me to describe the green file so think of like a link list where do you have to tell me about a link list to describe the whole thing well at the end of the day just where the head is if you know the structure of the link list and where the pointers are you can figure it out from that but to describe this I could say hey the green file well you know where the pointers are just follow it any disadvantages to this yeah yeah real slow access too so what if you know if I wanted to go to block 5 well I have to start at the beginning keep on going all the way to block 5 and why is this also especially bad and hint the answer is performance so in this case I store the pointers on the block itself so yeah yeah well in this case typically you cache like blocks at a time but each time I access a pointer if it's like this it would be a new block which would nush so every time if I want to go to block 4 of this file well I have to go to block 0 that wouldn't be in the cache read the whole thing in read in like a whole block but oh to figure out the next one I just read the pointer from that have to go to the next block read the whole block and it's really really slow how would I speed that up so if pointers on all different blocks is bad where can I put the pointer can I put the pointers closer together somewhere yeah yeah basically yeah yeah well I could basically just take all the pointers away from the blocks and just shove them all together right just shove them all together so that way it kind of sucks but if I have to read one well hopefully I cache the wrestling they're all in the same block I don't have to reread it again so this linked allocation is really slow random access because it's a linked list I have to keep on going through things but it's kind of nice because I can grow and shrink files really easily because one block is any other block so I don't have any external or external fragmentation because if there's a one gap pull that's fine I can just point to it and I can point to another block doesn't have to be contiguous and that last question there to increase random access yeah well if they're not cached because they're always in different blocks just move the pointers all to one single block so they're all beside each other that's our first file system that's actually used so same idea but it just takes all the pointers and throws them all in a block or throws them all in just a giant table and that table is called the file allocation table have we ever heard of a file system called FAT32 this is FAT32 so FAT32 means file allocation table and the size of the pointer for the size of the pointer in this block is 32 bits so that's what it means every entry there that represents a pointer is 32 bits that's what FAT32 means if you use Windows you're using FAT32 even if you don't know it because that's the file system that Windows boots off of and it has to be that and you're definitely using it even if you don't know it so if you open up disk explorer or whatever file manager whatever the hell it's called in Windows and you look at your drive and how it's formatted you will find a FAT32 partition or it might be called a system partition it's this so what's kind of bad about this yeah you need more memory well in that case all these pointers would be like on a block somewhere but the size of that table depends on how big my disk is which is kind of bad so if I have a bigger disk I need a bigger table so it doesn't really scale that well because it gets worse and worse the bigger your disk is and that's why it's only used it's simple but that's why it's only used for your booting drive because it doesn't have to be that big and that file that file allocation table doesn't actually have to be that big so it's not that big of a deal which is why you don't see it on any real file system but that's the idea towards it so any questions about that first real one so if this sucks is there anything I can do that's better yeah I could make it a bit better and have link lists to contiguous blocks might be somewhat complicated but it would work it would probably be a bit better what about if I just go completely out of the box and be like well this is based off link lists and link lists suck what do computers like more than link lists they don't what are hash tables arrays hash tables are just hash tables are actually just arrays with smaller bounds hash tables are actually just arrays we like arrays so we could do arrays and arrays would be better oh yeah so sorry for file allocation table similar to linked allocation just it's fast so that was our idea to increase random access speed instead of having linked lists I could just essentially have an array and kind of flip the problem so I can use a block and in that block well I can have an index for every single block of that file so in this case this red block I'll just store pointers in a row that represent this file so I'll take this red block and then to represent this green file this is just a giant array so block 0 of this file points to block 0 on the disk block 1 of this file points to block 6 block 2 of this file points to block 2 so on and so forth so now I transform it into an array and how big does this array need to be kind of yeah sorry yeah the number of blocks are being used for this file so it's only as big as it needs to be for the file this will become somewhat familiar to you because it actually uses the same idea as page tables it looks kind of similar right blocks kind of look like pages and this kind of looks like a smaller page table that fits on a page right so in that case that's exactly what this is so it could use an entire block and just fill it in pointer with pointers and they'll be indexed in the correct order and they can just point to individual blocks and this is something called indexed allocation basically just an array so has all the same advantages so files can still grow and shrink because we're just pointing at blocks we're just instead of using a link list to keep track of it we're using an array so we don't have to keep on going following the next pointer over and over again if we want block two of the file we go to index two and then it points directly there we don't have to do any messing around but the file size is going to be limited by the size of the block so right now I'm just assuming that I'm using one block just full of pointers so given that let's do a bit of math so if I have this scenario where my index block all it does is store pointers and nothing else so just stores pointers to blocks and each block is 8 kilobytes in size and a pointer to a block is 4 bytes well what is the maximum size of the file that can be managed by this index block yep 2 kilobytes close so that 2 to the 11 is how many pointers could fit on a block right and then if each of those pointers is pointing to a block how big is a block so I can point to that many 8 kilobyte blocks I'm bad math in my head so let's see our block size here is 8 kilobytes so let's just write in powers of 2 because we should be somewhat used to that right now so that's 2 to the power of 13 4 bytes is 2 to the power of 2 thankfully they are both in bytes so if I want to figure out how many pointers I can fit on a block well I just divide them so it's block size divided by pointer size so that means I can fit 2 to the 11 pointers block so now if each of those pointers can point to a block and the block size is 8 kilobytes that should mean the maximum size of the file could be 2 to the 11 so the number of pointers times how big the block is which is 2 to the 13 which should be 2 to the 24 or otherwise 16 megabytes so that would be the maximum size of the file if we did that are we pleased with that? would we like a file system where you can't do anything bigger than 16 megabytes no probably not so we will tackle that in the next question or next lecture but guess what if this was page tables and I wanted to access more virtual memory what would I do? yeah that's spoiler alert that's going to be the same idea we use if we want to make this bigger but there's going to be some different tradeoffs here so if we wanted to we could instead of having pointing to a block that points directly to the blocks we use for the file well it could point essentially to like another level of index blocks and then we can make our file way way way way bigger so any questions about that would again we'll get to that later so no questions alright so now we have file systems help enable persistence so it's actually how you describe files on your actual physical drive your drive only cares about blocks or pages terms are generally interchangeable so api-wise we can open files change the position so we learned about the position today which is in the global open file table and each process has local open file table entries which are basically what the file descriptors are sometimes it's important to know whether you're sharing global entries or not because that means you're sharing a position and when we're actually laying out files on an SSD well there's multiple allocation strategies we could use contiguous is fast but kind of a pain linked which is good because it got rid of that external fragmentation but it was slow so fat was just the faster version of it where we just crammed all the pointers together and then indexed which is basically the same idea but instead of a linked list we used an array so with that just remember we're on this together