 Welcome back to 353. So we have made it over the hump. All the hard topics are gone. Yay so You might have noticed it got a bit easier So let's continue our journey on somewhat easier topics and talk more about file systems So we've used file systems before I mean your virtual machine has file systems There's folders or directories and files So the typical layout of like any POSIX type system. So like linux the bsds things like that They're actually defined in something Very creatively called the fhs or the file system hierarchy standard that just defines what directories should exist Very exciting name. So that like Defines that hey, you should have a root directory just called slash at the very top Inside of it should be a bunch of other directories. So there should be a directory called bin that has all of your binary executable files a Directory called dev that has the fake files that kernels managing that you can interact directly with hardware or maybe Make up some terminals things like that. So they would all be in this folder There's like etc, which is supposed to store all of your configuration files There is the home folder which is supposed to have Another folder in it for every single user on the machine and it should be private And then there's something like the mount directory or while they shortened it to mnt And that is a place where if you plug in a usb drive or something like that That's typically where you should be able to access that drive. So if I plug in a drive Maybe on my machine it'll get mounted or I'll be able to access it through a directory called usb And then in here maybe You know from the root directory. I can get to my home directory Maybe inside that there's a directory for me called john and then maybe in my john directory I have a file called to do dot txt So your file system will form a direct acyclic graph because everyone loves dags and in here Whenever you're using your terminal or in terms of processes, there's always something called the working directory So that is typically the current directory your process is actually in and Then any path I give it or any file I will give it will be relative to that directory so If my current working directory is home slash john, so I'm currently in the john directory Well, then the relative path to to do dot txt is just to do dot txt, right? I don't have to go into any other folder. I'm in the correct folder and The difference between that would be the relative path And on the flip side of a relative path is an absolute path So the absolute path is just the full path name starting at root So the full path name for to do dot txt would be Slash home slash john slash to do dot txt Yay And then if I wanted to get to this usb directory. Well the relative path from The john directory I could do like dot dot so dot dot sends you up a directory, right? So if I did dot dot that would bring me to home and then if I do dot dot again That'll bring me to root and then I could do slash mount then I could do slash usb So that would be the relative path to The usb directory from home john and the absolute path To the usb directory would just be slash mount slash usb boom All right questions about that. We've been using this pretty much the whole time This should be more or less review So Here's the answers for that. There are some special symbols and we will get into How these are actually implemented. They're slightly special but not terribly special So there are some special symbols where a single dot means the current directory So instead of just to do dot txt. I could do dot slash to do dot txt So that says well dot is just your current directory And then dot dot is the parent directory And then this little tilde thing that's usually a shortcut to the user's home directory So in my case if my user is john, it's probably home slash john So all the relative paths are all calculated from the current working directory So also funny story has anyone noticed on linux like all the hidden files are just any file that starts with a dot Anyone know why that is? Because guess what it was a complete accident that worked that way Okay So turns out in a directory there is like a dot and a dot dot usually when you ls something You probably don't want to see dot and dot dot every time right? You just you don't really care about them. You assume they always exist So if you were implementing ls Well, you could just have like an if statement. So if The current file is a dot skip it Or if the file is a dot dot skip it right to if statements to skip it seems reasonable What would you do if you were a very clever clever programmer to get that down to one line? Yeah Yeah, just check if the first character is a dot and then skip it So someone implemented that and then guess what if any file starts with a dot you can't see it And then that feature or sorry that bug just became a feature easy So that is why hidden files on linux are just any file that starts with a dot Because some programmer wanted to save a single if statement and He they totally meant to do that. So they meant to implement hidden files So fun historical fact with that So when we access files we can access them sequentially or randomly So sequential access just means that each read would just Advance like some position inside of a file So if I read five bytes Well, if I call another system call read on it for another five bytes I just get the next five bytes until I'm eventually at the end of the file Right, that's how we know how to read files if we just do a read system call We just get the bytes in order from the file And then similarly if we just did write system calls or how we're doing them right now The writes are just kind of appended together And the position is set to the end So if I create a new file and I do a write system call and do like write hello Then hello would be written to the file and then if I write again and say there Well, that just gets appended to the end of that So then the total contents of my file just say hello there or something like that And then I can optionally Just open a file for writing and just stick my writes onto the very end at the beginning So we don't know how to do anything other than that So the other option is to have random access So that means you can read and write anywhere in the file in any order you want And you get to specify which byte of the file you want to read at any given time So how you use that there's our old faithful the open system call That gives you back a file descriptor takes a path name some flags like do I want this file able to read to it write to it things like that and then some modes for permissions And then there's this new system called called l seek which takes a file descriptor And then an offset so that tells you which byte you want to be able to read or write to next And then the third parameter is this wence which tells you what this offset is relative to So some options for wence is seek set So that says just go directly to the byte in the file that I have told you So if I give once seek set and set my offset equal to four that means it's going to move The internal kernel position of that file to the fourth Bite so if I read or write to it, I will either read the fourth byte or modify starting at the fourth byte Then the next one is current so that's relative to wherever I currently am in the file So if I read all of the file Well, maybe I want to go backwards a little bit and read some more information Again, so I could set wence to seek current and set my offset to like minus four So I want to go backwards by four bytes and read the last four bytes again Or if you're in the middle of a file you want to back up and go forward You can do that And then the third option is seek end. So that's relative to the end So if I want to go directly to the end of the file I could just set wence to seek end and offset to zero and then boom It will just point me to the end of the file If I read something it's not going to be there if I write to it Well, I would start writing from the end of the file. So new system call we get to get and This will make our lives Kind of fun once it interacts with work and all these things because there are Subtle things that go on here So other things just so we have functions like gall get out of the way So you've done this before to access a directory So this is the api You open a directory. You can read directory. You've kind of done this already in lab two Print the directory contents all that stuff just so you have it here We won't really use it in the examples, but hey, this is probably where it should be in the lecture notes. So there we go so Now we get to talk about what file descriptors actually represent So each process essentially has a file table in it and that would be stored in that process control block And that file table is basically a giant array And the file descriptor is an index into that array Which is why I called it a pointer and said think of it as a pointer at the beginning So I said just think of it as a pointer to a file turns out that was Not a lie, but also not the whole truth So what it is pointing to it's going to point to a few things So it's going to point to all the information about the file and like the current permissions you are You have with that file So one of the entries it will point to is like what is the current position of this file descriptor in that file So I could open two file descriptors to the same file But I could they could be in different places in the file So the position would be part of the file table entry And then flags whether or not I can read and write this File descriptor or not and then there is this v node pointer So it's short for virtual node and it basically represents anything you can read and write Bites to so could represent our terminal could represent a socket could represent Well, it could also represent a directory, but here let's just assume it represents a normal file So there'd just be like a v node for some file a so that would actually be the contents of the file and We'll explore in the next lecture exactly. Well towards the end of this lecture. We'll explain how we store a file so In process one it could also have file descriptor one which points to A different position flags and v node that points to some file b And then maybe in another process process two its file descriptor zero Points to an entry that has its own independent position own independent flags And its v node points to the same file b So they can both read that file starting at the beginning to the end because well, they are both independent here so Each process We'll have this file table as part of its piece process control block File descriptor basically just an index into this table What actually happens is each item it is just a pointer But it points to a system wide global open file table So this thing in the middle here So this thing in the middle here would be like our global global open file table So that would be managed by the kernel. So the kernel knows how many processes are accessed within what file And well if it was a pipe, it would know how many processes have access to the pipe more specifically what it cares about How many processes have access to the right end of the pipe? so That is the proper name for that. They call it a goff table Global global open file table why they wanted to call it a goff. I don't know sounds terrible to me But basically tells you the current seek position Some flags and then points to a v-node so which is just anything that supports standard reading and writing bytes to it so Would hold yeah the v-node would hold information about the file They can represent regular files pipes network sockets directories all sorts of different things and Towards the end of this lecture like I said we'll get into how to actually store the contents of a regular file But now we have to remember what happens during a fork. So the process control block Oh, yeah Yes, yeah, so in the process control block for each function They would have a Local open file table that just points to an entry in the global open file table yeah Yeah, so this is where it gets complicated with forks. So in this They're kind of independent But if I were to fork process Two in this case. Well, it would copy the file table and The new processes file descriptor zero would point to the same thing So if one process read it would move that position and then if the other process read Well, it wouldn't read what the other one just read, right? It would already be moved because It is a global entry So that is where we will get into fun and where Any confusion of this Stuff may occur. So yeah, so again What happens with the fork process control block gets copied Specifically for us that like local open file table that's part of the process that just points to a global entry gets copied So both process control blocks point to the same global entry And that would look I mean something like this you have process one process two if they're copies Well, both of their files descriptor zero point to the same Global open file table entry which has a position has flags and has a v-node pointing to some files So there are going to be some gotchas with this so That means oh, yep So copy so the question is uh, how does this relate to copy on right or how does copy on right come in here? so When we fork We just create a whole new like we just create a copy of the global open file table Copy on right is only for virtual memory And just copying is just how we deal with the virtual memory part of that So remember both processes have to be independent. So to make them independent they get their own They get their own file table and then we also have to do something with their virtual memory We would have to essentially create a new version of a thread for each of them too And they'd need to look like exact copies that are independent after the fork So, yeah So, yeah some fun gotchas so that current position is shared between both processes so if you did like an l seek in one process while that affects every other process that forked because you're pointing to the same global open file table entry and Otherwise if you open the same file in both processes after you fork that would create Two independent global open file tables. So whenever you do an open system call that creates a new entry in the global open file table And you can essentially only share them through forking so If I had something like this So in my program, I did an open of to do dot txt and then I forked And then I did open b dot txt How many entries would I have in the global open file table and how many entries would each process have in its own file table Assuming that both neither process have any file descriptors open. So before this line executes in our first process We got nothing open. So no file descriptors So hopefully what should happen is let's say process one runs It does an open So that would create a new global open file table entry that the kernel gets to manage And then I fork So now I have two processes that are exact clones of each other They are both have one entry and they're both pointing to the same global open file table And then afterwards Well, after the fork I have two processes. I don't know what one is going to execute next In this case, I don't really care. They're both going to do the same thing So they both independently after the fork open b dot txt. So that would create two local file or Yeah, that would create a local open file table in that process and also a new global entry in the global open file table So each of them were have their Each of them would have an independent copy of b dot txt so you could go ahead and read the whole thing So that kind of clear Yeah So the so the local one has to point to a global one So the first process is going to open to do dot txt. That's going to create a global entry And then you fork so the new process is also going to point to that entry So So both of them would create a new Global open entry. So what what it would look So to do dot txt would be a different file descriptor than b dot txt So what it would look like probably something like this So each process has its has two entries in it So the parent process well, it would have file descriptor zero that would point to a global open file table entry That would have a position flags v node that v node would point to to do dot txt And then after the fork the child is going to look exactly the same So it would have file descriptor zero which points to the exact same entry And then afterwards they both independently open b dot txt. So let's say the parent ran first It would get a new entry in the global open file table So it would have its own position flags and v node that points to b dot txt And then in the child whenever it executes that would be a new entry in the global open file table With its own independent position flags v node and that v node points to b dot txt Yep, if we dupe the file descriptor Then we would just change file descriptor Zero to point to this one, right So so the the parent can't point to this one. There's no way to access it Oh, so like As long as one process is pointing to the global open file table Then it's fine, right if no processes are currently pointing to it the kernel can go ahead free it up Use that entry for something else So let us see here. So Here is a fun little program just to illustrate exactly what happens here. Let's see if we know what will happen here So I have two files a and b. They are very exciting files So the content of file a just says this is file a The content of file b says well, this one is file b And then in my program In my main I open I do an open of a dot txt Open it as read only Then I fork and then I open b dot txt And then I have this function called read file that just creates a buffer Does a read system call And then make sure it's just null terminated because i'm going to print it as a string And then it will print off the process id of the current process How many bytes it read and then the string it actually read So if I run this program, what will I see? And I don't care about byte counts So yeah, so So I see the contents of the file essentially Two times each What so one time each? So I'll have two processes each printing two files, right? So one process is just going to print the contents of the whole file and then the other process is going to print zero Yeah, so this should print file a this should print file b And but I have two processes, right? Because i'm doing a fork so Will both processes print out all of file a I don't know yeah Yeah, so this is where it actually matters. So here I do an open before the fork So that creates a new global entry that has its own position flags all of that stuff And then when I fork Well, the new process the new child process also gets Looks exactly like the parent. So it's pointing to the exact same global entry So both processors pointing to the same global entry for this a dot txt And then after the fork, I don't know which one's going to execute but they both Open b dot txt which just creates A new global entry for each process Just like we had here So they would each get their own position flags and everything So what should happen is since file descriptor one is shared between the process Assuming that my read system called just reads the whole file I don't know which one's going to run first But whatever one reads runs first is going to read the contents of the whole file The position is going to be set to the end So it's done. So it just prints the contents of the file And then if the other one tries to read it, well The first process already read it the positions at the end of the file So it will get zero bytes. It won't read anything But since fd2 that essentially is my open entries for b dot txt Since they're independent, I should see the full file in both of them because if I Read the file in one while it's a different global entry it might update the position there But since they're not shared doesn't matter. So if I run that That's what I get so This process happened to run first. It read 15 bytes. So it read this is file a and then the other Child process probably because its process id number is higher. It read zero bytes While for b dot txt both files read the entire contents of the file because they had their own global entries Yep Yeah, in terms of the pids the parent ran first read and then we context switch over the child and then the child executed Pretty much all the way and then we switch back to the parent and then parent ran Uh, is that because read is a slower operation? I assume There's a different order So doesn't matter just got unlucky So this one the child The child went first. So not even predictable. The child went first read all file a Then the parent got nothing and then we switched back to the child It read all the file b and then it switched back to the parent. We got all the file b so fun, right If if you didn't know the global entries and this happened to you where you forked And then you tried to read a whole file Um, and you just got nothing in one process. You'd probably be very confused so This is why you have to be careful about it. So imagine Like I couldn't read all of a dot txt in one system call What might happen if this essentially was just You know just a while read does not equal zero just keep on reading over and over again And I just opened the file descriptor fork and then both of them had that like while read just read the data Do I know which process is going to read what? No, all I know is that one process Like they're going to be mutually exclusive. So one process might read The first half of the file the other process might get the second They might get some random assortment of characters. They might interleave. You have no idea So this is the cautionary tale to be careful if you are forking with file descriptors Sometimes you want that behavior where it's shared. So like For the terminal. So like copying Standard in standard out all that is fine. They're all sharing the position. So No matter what process is running while it writes to your terminal And then essentially just gets appended to the end and you don't see anything weird And you can't even seek with the terminal to begin with So any questions about that fun thing? So that is your cautionary tale for the evening So now we have to talk about how we would actually store a file on a disk so If we were actually to store a file on a disk Well, how would we store it? We might think of storing it like memory So like everything is contiguous. So here we kind of have to just explain How we would describe how to get to the contents of a file So if I just think of it the same way as I don't know a pointer or malloc or something like that How would you describe an array to me if I was a complete dummy and you want to uniquely Tell me everything about an array with just two pieces of information Yeah Yeah, where it starts and how and how big it is essentially, right? So that could be how we represent a file. So Assume this is our disk and our disk is divided up into blocks Well, we could just describe file green So file green could say it starts at block zero and then it's three blocks big This red file starts at block one two three four five or zero one two three four five six Starts at block six and it's six blocks big That could be how our file system works Everyone thinks it looks like an array So if that's how I represented files What would I have to do in this case if I don't know say the red file Was I don't know my stream recording and it got bigger What do I have to do to make this file one block bigger if it has to be contiguous? Yeah Yeah, I have to boot Yeah, I have to boot the blue file the hell out of the way and then I can go ahead and move space like That's what malloc might do if it runs out of room or just move the red file here So it has more room, but For memory that's not that big that might be okay for file systems that are maybe that red file is like I don't know 10 gigabytes then probably a bad idea to do that so We could do that turns out to be a very bad idea with files So if we had contiguous allocation It would be really fast if there's no modifications and space efficient So to describe a file all I need to tell you is what block it starts at And how many blocks it needs And then for random access I can figure out Exactly what block I need to get to by just making a single calculation So you just take the offset and the block size so that basically means let's say our blocks were four kilobytes Well, if I want to access byte 4000. Well, that's on block zero I can just calculate that immediately if I want to go to byte 4098 Well, that would be on block one. I know exactly how to get to it. All I have to do is do a simple calculation bad things are that Files can't really grow that easily. So now we get to talk about This fun word fragmentation, which basically means wasted space So internal fragmentation Just means like within a block. I waste some space So for file systems, typically we just deal with blocks So if there are four kilobyte block and your file is only 10 bytes large while you're wasting 4086 bytes But we call that internal fragmentation because it's within a block and we don't really care about it And then external fragmentation is going to be wasted space between the blocks or essentially wasted blocks So we might have external fragmentation when blocks are like deleted or truncated So like let's say I made the red file one smaller Okay, while there's this like Hole of one of one block just kind of in the middle of the red and the blue file And if all my files are two blocks, well, I can never use that space ever again. So it's essentially wasted So everyone in this course loves linked lists So we could store a file using a linked list, right? So that is called linked allocation So I could tell you Well, I guess I don't have to tell you but To keep track of linked allocation You just have to tell me essentially like what the head of the list is so what the first element is and then to get to anything I could just follow pointers to the next element and Then it doesn't really matter what blocks. They don't have to be all in a row So if I want to make this file bigger Well, I grab any old block doesn't really matter and I just make my next pointer point to it and That's great easy to grow But what is bad about this? Yeah Yeah, if I want to read the whole file, well I don't know exactly let's say I want to like go to the end of the file And I don't want anything else. Well to go to the end of the file I have to access every single block and hop all around and make it to the very end which Is bad so really really slow so pros space efficient so To describe a file. You just need to tell me what block starts the file and then I can go ahead follow pointers but Yeah, end files can grow and shrink. There's no external fragmentation. I can use one block as good as any other block No risk of wasting a block Like with file systems. We just accept internal fragmentation. So not using the full contents of a block, but Now we have really bad Random access speed so we need to walk each block and each block might be far away on disk So Like loading a block might take a lot of time. So can anyone think of an idea to Still use link lists because we love link lists, but make this go faster So like in this case In order to walk all them. I have to read this block which points to this block read this block Then read this block. Okay. Was it point to this block? I have to read it Then do this block read it. Oh this block read it. How could I speed that up that? Yeah, how could I speed up my description of a file? Yep Yeah, perfect. I just have a block that all it does is store all the pointers And then I can use my beloved link list and I just access a whole block It's all full of pointers. So if I need to go to the end, I at least don't have to read another block I just keep following them and I don't really have to do anything special So they're all there. So That is what a file allocation table is and if you have used windows Well, that is what fat 32 stands for. So file allocation table There it is so All it does is move that list to a separate table and the table Represents like all of the blocks on the disc So the size of this table is going to be proportional to how many blocks are on the disc and then That is what your file allocation table is and it is just a list of pointers Basically, it's a link list same idea, but they're just all located together So this could be the contents of my disc all the usable blocks I can use for files Name zero all the way to 24 So maybe to describe my file Still same idea. I can say the file starts at block zero But instead of having to read this block then going to block six then reading this block and going to block two Well, let's assume that this big file allocation table is in memory. I can be like, oh it starts at two boom I go to six. That's the next block. Okay after that I go to two Okay after two I go to 13 after 13 I go to nine after nine I go to 18 And then that's the end of my file so If any of you have a windows machine Even though You know your c partition might be a different file system The file system that you boot off is this So this is really really simple And It's essentially linked allocation, but Smarter and more performant. So questions about that so Same benefits as linked allocation files can grow and shrink at will don't have any external fragmentation Still have internal fragmentation, but that's pretty much every single file system. And then we have fast random access because that file allocation table Could just be held in memory. You could cache it. You could just read it But the size of this table is going to increase linearly to this size. It can be very large So that's why you're just they use it for your boot partition because typically that's fairly small But if you use it on like A 12 terabyte drive Probably a bad idea because while you need as many pointers as there are blocks on the disk so How could I make random access speed go faster? Hint Don't use link lists. So if I don't use a link list, how could I describe a file smarter? Yeah, same thing with page Yeah pointers going directly, but So there's link lists and then there are fast starts with an a Arrays thank you And we should use arrays. We've never used a link list before, right? All right, so An idea could be well, I could just use an array to describe a file instead So maybe I do that so each file I just use a block and it is an array of pointers to blocks So then I don't have to traverse anything so this let's say I use this block as Just storing an array of pointers So this red block would describe my file and then for that same green file well Maybe The pointer at block or the pointer at index zero would be what block zero I need to read in the file So maybe that points to this block on disk Then block one of the file is associated with block six on the disk Block two of the file is associated with block two So on and so forth that the that the duh So Would this be a lot faster? Probably right so What um Yeah, what is the main drawback with this one though? Yeah, lots of memory Kind of yeah Sorry a lot of Yeah, you need an index block for all the files you have But it wastes about the same amount of space as a file The file allocation table. Yeah Oh, how do we know how many blocks are in each file? Well, you could just fill this up with pointers And hope it and then that would be the limit of the size of your file Right. Is that what you're going to say too? Yeah, okay. Well If we just have one block full of pointers to describe a file Well, how big could our file be wait? First let's summarize this though. So for index allocation Well file still has all the same benefits So files can grow and shrink at will we have fast random access because it's an array and The benefit or the main drawback is that the file size is going to be limited by like the maximum size of the block so We have a little problem now. This kind of looks like an easy version of page tables So let's assume an index block just stores pointers to data blocks There's no other information And then a disc block. Let's say it's eight kilobytes in size just to be fun And then a pointer to a block is four bytes so What we should be able to answer is what is the maximum size of a file represented by this block Or represented by this index block Oh, yeah Yeah Yeah, yeah going ahead 16 megabytes Why is it 16 megabytes? Well So the block size is going to be two to the 13 Pointer is two to the two So the number of pointers I could fit on a block is essentially two to the 13 divided by two the two Which is two the 11. So that is whoops So that is the number of pointers per block And then well, that's how many pointers I can point or how many blocks I can point to So I can point to two the 11 blocks and then each block is two to the 13 So that means the maximum size of my file could be two to the 24 Which sounds kind of big right except that Well, that's the same as two to the four times two to the 20 and then this Is a megabyte and two the four is 16. So That means the maximum size of the file would be 16 megabytes. Is that good? No, you'd probably be pretty pissed right if you went to Store, I don't know any file or anything. It's like yeah, your maximum size is 16 megabytes too bad. So sad Probably be pissed. Yeah ah so yeah, if I want to Support a larger file Well, maybe I just grab another index bit right. Okay. Well now it can be 32 megabytes Sounds lame. So What we'll get into in the next lecture is like same idea with page tables I could just have a block of pointers at point two another block of pointers And then suddenly well that would be like two the 11 To to the 11 if I just do like two levels and then suddenly Well, that's that's a lot bigger of a file. That's what would be like 32 gigabytes should be slightly better But yeah, that's going to be the idea for the next lecture So that's why we do page tables first because that idea constantly gets reused So File systems that is what enables persistence. So they describe how files are stored on disk We will go over how most systems actually store file systems on disk, but Spoiler alert. It's pretty much like the idea of multi-level page tables But there's some trade-offs because well, we don't want everything to be slow So unlike with virtual memory, we don't have a tlb or anything to help us out So we'll have some tricks up our sleeves for performance reasons But api wise, right? We can open files change the position to read and write at We know that each process has like a local open file table that points to a global file table entry Could cause some surprises with forks if you don't If you don't realize that and then we saw some allocation strategies Contiguous not used link not used file allocation table over fat Well, that's used sometimes and then index isn't used, but that will lead us into our next idea Again in the next lecture. So just remember pulling for you. We're all in this together