 All righty, welcome back to operating systems. So yeah, before anyone asks, the curving down or curving up or whatever, that was a joke. I didn't even scan them yet. Like I didn't even actually look at them aside from like a few. So don't worry about it. So all the exams got scanned this morning. TA is currently grading it as we speak. They'll be back to you ASAP, hopefully by Monday, something like that. I don't know. They all have their assignments. They are all currently grading. They might actually be asking me questions right now, but I'm doing a lecture. So today we get to continue our journey of file systems. So we've all used file systems before. So there's actually a usual layout we have for POSIX systems. It actually has a name. It's called the FHS or File System Hierarchy Standard. Yes, we are very good at creative naming. Look something like this. So at the top of your entire file system, you have a directory called root, which just begins with a slash. And that is an actual directory. Then past that, the directory has a bunch of other things in it. They're all given by names. Likely in this case, these are all represent other directories. So within root, there's a bin directory. There's a dev directory. There's an ETC directory that usually has config files. Then there's like your home directory. So if you're on the UG machines, you'll have your own home directory. So in this case would be a subdirectory of that. Mine would be called john and then you'd have your files in it. So like todo.txt or something like that. There's also a specified slash mount and that's where whenever you plug in USB drives or something like that, they should show up in this directory. So all of our processes, they have something called a working directory that we've probably been dealing with forever. So in this case, if my working directory is currently slash home slash john, so I'm in that directory, what is the relative path of todo.txt if we remember what a relative path is? So if I'm currently in the folder john, what do I have to do to get to todo.txt? Yeah, dot slash todo.txt or just todo.txt because I'm already in that folder. So that's relatives directory. It's relative to wherever you are. Dot represents the current directory and dot dot is one directory back. So what about to the USB folder? How do I get to the USB to folder relative to john? Yeah, so relative to dot john, if I do dot dot, I go up to home, then slash dot dot again. Oh, sorry, FM? Yeah, yeah, then slash dot dot again brings us to here and then we can go to slash mount slash USB. So it's a bit of a trick, but those are relative directories. So to do dot txt relative, it could be dot slash to do dot txt. Sometimes you want absolute paths. So these are all called paths. Absolute paths will begin at the root and then get you to whatever file you want. So the absolute path for to do dot txt is slash home slash john slash to do dot txt. USB relative was dot dot to go up one, dot dot again to go up to the root and then slash mount slash USB or in absolute terms it would just be slash mount slash USB. So any questions about that? We've kind of dealt with file systems the whole time. You'll actually see what these things represent in lab six and while we're talking about it. So there are special symbols here. So dot, like I said, dot represents the current directory and dot dot represents the parent directory. So do we also know that in Linux, have you noticed that when you ls and stuff, any file that starts with a dot you can't see, it's like a hidden file. So in your git repository, you don't see it, but there's a dot git directory that holds all the information about your repository and everything like that. And in Unix or Linux or macOS, everything like that. If the file name or directory starts with a dot, it's like a hidden folder. Well, that was just one big mistake. So in ls, there's always a dot and a dot dot and typically you don't wanna see them if you just do ls, it's kind of useless to see them. So you just don't want them displayed if someone types ls. So a programmer was like, well, that's like two cases, that would be two if statements. That's silly, I see some commonality there. So I'll cover both these cases with a single if statement. So I'll just check the first character and if it's the first character, if the first character is a dot, I'll assume they meant it's dot or dot dot and I just won't show it and that was it. And that's why anything that begins with a dot is a hidden file because some programmer decided to be clever in like the 60s or the 70s and that was a bug that became a feature. So fun fact. So the other special symbols is tilde and that usually represents the current user's home directory. It's also held in an environment variable called home. This case, it would probably be home slash john and relative pass are calculated for the current working directory. Each process will have its own current working directory and it will also be in this environment variable called PWD or print working directory. So when we're accessing files, so far we've just accessed them sequentially and we haven't talked about how to randomly access any byte in the file unless we like M map it if we paid attention to that which we didn't really have to but for now we just know how to access a file sequentially. So whenever you open a file and you do a read while you are advancing a position inside the file so every time you call read you just read more bytes from that file and you don't get any duplicated bytes so it just keeps on going and going and going sequentially and if you do a write well they're just appended to the end wherever you currently are in the file. So if you do a write system call and you write hello and then do another one that says world well then you'll get hello world it'll just be appended on to the end and the kernel has to keep track of where you need to write those bytes in the file if it represents a file. There is a way to do random access and for random access that means you can read and write anywhere in the file and for that a specific position the file is required for each operation so you need to say if you're doing random access what byte in the file do I actually want to get read or write. So we kind of well we went over the open system call before so it just takes a path name some flags which can be like open it for read write open it for writing, open it for reading only and there's also another flag here that it's called oh append so by default when you open a file and you write to it it just writes to the beginning so if there was data already in the file it would just overwrite it so if you add a flag called oh append that means that any writes you do to that file descriptor that represents the file will just go to the very end of it. So the only way to modify essentially the where in the file you read and write system calls are going to read and write from is with this else seek system call which we haven't seen yet so it takes a file descriptor and an offset so an offset is just the number of bytes either to move this pointer so either negative, zero, one, two, whatever how many bytes and then the third parameter is wence so wence just means what this offset is relative to so there's a few options for this wence parameter it's basically just like an enum in C but it's kind of bad so there's three options seek set is like an absolute offset so that is just from the beginning of the file no matter where the pointer currently is so if I say wence is seek set and I set the offset to 16 that means it's going to change the position 16 bytes into the file from the beginning if I do seek current that's wherever I'm currently reading from so if I set it to seek current and hit an offset of minus two it means wherever that current position is I'm just going to go back two bytes and then I can reread that same information again and finally the other one is seek end so that's relative to the end of the file so seek set is relative to the beginning of the file seek end is relative to the end of the file so if I want to just go to the very end of the file I could go to seek end and an offset of zero so it takes me right to the end of the file so any questions about this, yep so the question is if I do a negative offset will I get an error or will loop around to the end so you can do a negative so if I do a negative offset but I'm relative to the end of the file that would be within the bounds of the file it'll give you an error if you're not within bounds of the file yeah so if I did seek end in a positive offset that would probably give me an error but if I did seek if I had a positive offset and I went to the seek set it would probably be within bounds unless I went over the file but it will make sure you're within the bounds of the file so any other questions with this because this makes it even more fun once we involve forking again so forking is the bane of our existence so here is the API for accessing a directory which we kind of which you did do in your lab so open directory, readdir so we're gonna see how these actually work under the hood in lab six but here's just an example just so we remember how we print directory content so this opendir well it wasn't anything special it actually opened a file descriptor if we S traced it and then this is how we just read everything in directory so in lab six you will be making your own directories from scratch and you will be told what file system format to actually use so here is what is stored in the process control block so remember that's in the process control block that's information about your process so that would be the processes page tables it's threads and it's open files I just said before so here is where we become explicit so inside your process control block there's going to be file open files and they're just going to be a big array so in this case I have the process control block for process one and it has three file descriptors zero one and two which our friends standard in standard out standard error and they actually just point to things so I told you before you can think that they just point to things so this is where we can see what they actually point to so they will point to a structure that keeps track of the current position that this file descriptor is currently at in this file the flags so if I have it open for read write write only read only and then this thing called a pointer to a V node and a V node just represents any resource that you can actually read and write bytes from so the full file descriptor represents a whole bunch of different things the V node would in the case of a real file point to that real file on the disk so you can have multiple file descriptors that point to the same file that have a different internal position so if I access it through this file descriptor it might start accessing it from the beginning if I access it through another file descriptor it might start accessing it through the end so in this case file descriptor zero of process control block one points to file a at some position with some flags file descriptor one points to different position different flags and a different file in this case file B and then process control block two while it can point to a different position independent of process one with its own different flags and its V node could also point to file B so file process one and process two could have file B open with different positions and when they read and write to it they have their own independent position so they would access it serially or sequentially if they just if they didn't do any L-seqs or anything like that so in this case one process would not affect the other so any questions about this? because yeah yeah so what dupe does in this case is it will take say you do a dupe two file descriptor zero and say two and one what that will do is essentially make file descriptor one point to the same thing as file descriptor zero so yeah so just point to the exact same thing the exact same position the exact same entry everything points to the exact same thing so yeah in that case if I read from file descriptor zero and then read from and then read like hello and then read from file descriptor one while it would keep on reading from that point it would read like world or something like that it just keep on going because they're sharing the same position all right any questions about that because it gets it's gonna get kind of weird once we start forking yeah so V-node is just a pointer to some structure that we can read and write bytes to so it could represent a real file, a socket, a pipe something like that it's just called a virtual node to represent you know it's virtualization so we're just representing something we can read and write bytes to the concrete version that we'll get into next lecture for real files and directories or something called an i-node and we'll see exactly what an i-node looks like in the next lecture but the super generic term that represents literally everything is just called a v-node or virtual node because well we like virtualize the things it seems to be one of our major solutions here so each process has its own file table in the process control block and a file descriptor is basically just an index into that table and that table is just a table of pointers that point to like a certain position, certain flags and then a v-node to represent what that file descriptor actually represents so what actually happens is when I pointed them to something what they're pointing to are entries in a global open file table and the kernel will maintain this and these are all the open files across all the processes in your system so that's how the kernel keeps track of literally every file descriptor that is open across all processes and how it would note that oh, does a process have the right end of the pipe open because it would be in the big global table? So we shortened the global open file table to a goff why we chose that name I don't know we're really terrible at naming so the global open file table or goff holds just information about like the seek position the flags and points to that v-node so which is just some resource that supports reads and writing bytes too so like I said again we'll see what that structure actually looks like if it's a real file it's called an i-node and that's tomorrow so the v-node again can represent lots of other things pipes, network sockets, shared memory whilst did we go over I think those were the main things that file descriptors represented right anything else I'm missing I don't think so yeah the kernel just depends entries to it so it's just dynamically resized so it's just however big it needs to be all right so now we have to discuss what happens with forking with this silly thing so remember what exactly what happens during a fork the process control block gets copied so specifically for us it would copy that local open file table so just that big array of pointers that point to entries in the global open file table so both process control blocks would point to the same global entry so that means if they're both sharing the exact same global entry if one process changes the position it changes it in the other process because while the position is part of that global open file table so that may or may not cause an issue usually it behaves how you want but it might surprise you so in this example both processes point to the same global open file table so both process one and process two their standard input file descriptor zero points to the exact same global open entry so if this represented like a file or something like that well if both processes are trying to do read system calls from that file and they're sharing the same open global open so say that file has it has two lines in it well what might happen is process one does a read system call it reads the first line which would advance the position to the end of the first line and then process two could do a read system call and then get the second line and then we're at the end of the file both processes would assume that that file is now empty so we read one copy of the file because they're sharing the same global open entry but one process gets one half and the other process gets another half so this might happen if you're actually forking and then sharing the same file descriptors which may or may not be what you want so any questions about that one because that will probably trip you up at some point in your life and you will have to remember this course and be like oh right they're sharing the position so the two processes neither one of them is going to open the whole file or you might get lucky and one process reads the entire file and the other process reads absolutely nothing so that also might happen so these are the gotchas so remember the current position that's part of the global entry and if you fork that gets shared between the processes so and also what will happen since that position shared if you do that L seek system call that changes the position well guess what since you're sharing the same entry if you change the position the other process it would also change the position in the other process because you're sharing the exact same entry so you could fork and then one of your processes could just set the position to the very end of the file and the other process might not know of it and then it would just assume that the file is just empty so sometimes you may or may not want this if you open the same file in both processes that would create multiple global open file entries so if you just call open or however you get a file descriptor through socket or something like that that will create a new unique entry in the global open file table so let's look at this example what would happen here if say for example this is one process it opens to do dot txt and then forks and then after the fork we open b.txt and assuming these are only open files how many local open files do we have in each process versus how many global open files we have and what is their relationship so I'll give you a few minutes with that and then we can discuss yeah so specifically how many would I have in each process so each process will have two local open files and in the global file table we'll have three entries so they'll be sharing to do dot txt and they'll have their own independent b.txt so just to go over what that looks like so let's say we have you know processed yeah let's call process 100 process 100 comes open has no files open whatsoever well if it doesn't open it would get a new file descriptor probably file descriptor zero if it has nothing else open so file descriptor zero would point to an entry in the global open file table because a new open system call creates a new entry in the global open file table so this would have its own position its own flags and then some vnode pointer to vnode that actually points to to do dot txt and that looks ugly so it has one entry in its local table that points to one in the global open file table so now if we do a fork while we're gotten used to this this would create a new process called 101 and since it's an exact copy of its parent it would have the exact same local file table so it would have file descriptor zero which points to the exact same entry so any questions about that? Just our good old friend fork so now at this point we don't know what process is going to execute next we got process whoops we got process 100 and process 101 they both want to just open b.txt so in this case they are now independent because they are after the fork so standard rules apply they would each just create a new local file descriptor called one and they would point to a new entry in the global open file table which would have its own position its own flags and its own vnode pointer to a vnode which would point to b.txt and then if we went and we executed process 101 it would open its own file descriptor one which would point to a new entry in the global open file table with its own position its own flags and then whoops its own vnode which points to the exact same file so any questions about that? yep so standard out standard error they would have entries in the global open file table and if I'm typing to my terminal well we saw we could do that ls of command to see what the file descriptor is actually point to so if you're using a terminal file descriptor 0, 1 and 2 all point to a terminal so there's a fake device that looks like a terminal that you can read and write bytes to so whatever you do so because you're fork and then you're all sharing the terminal that's why since they're all sharing this position if you type a character so you type a and hit enter that also means that because they're all sharing the same position only one process is going to be able to read that a you typed otherwise every grand parent would see that a as well if they read from standard in before you and things would get weird so that's also why the standard file descriptors work because while they all point to the same position they multiple processes won't read the same thing because right now if either process reads to do.txt since they're sharing position it'll advance it in both but for both of these they could since they have their own unique entry pointing to b.txt they could both read the entire contents of b.txt because they're not sharing a position alright any questions with that because it's yep yeah so the local ones are just this table here and that's what we call the LOF or local open file table so all they are is just pointers to some entry in the global open file table and their index is the file descriptor you get so the file descriptor is basically just an index into this table so which is why it doesn't actually have to represent a file can represent anything so any more questions about that? alright so here was the answer to that so draw nicer than my chicken scratch writing so hopefully that is more understandable so you have it so now we have to actually store files on an SSD or something like that so we have to figure out a way to store these files and how you store these files is exactly what a file system does so remember SSDs well they kind of behave like memory they have pages so I could represent a file as just a series of pages so how would I store information about the files so for instance if I have to do .txt how would I describe where to go on that disk to actually read the contents of that file so if you were to tell me how to use an array what two things would you need to tell me about the array in order for me to use it yeah the start and how big it is so in this case if I have three files here so maybe I didn't explain that well enough so each of these different colors should represent a different file so if I have contiguous allocation if you wanted to tell me where the green file was you could say well it starts at block zero or page zero so I'll probably use typically when you say blocks you talk about pages on an SSD and it's like a file system related thing so I'll probably use say the term block but it's just probably the same size as a page or slightly bigger so in this case for the green file to describe it I would say it starts at block zero and it's three long well and then if I describe the red file well it's zero one two three four five six so it starts at block six and it is six things long and the blue file starts right after that and it is four blocks long so is there a problem if I do this to represent my files so what would I have to do if I want to make the red file larger yeah yeah so if I assume everything's the same so this is like the beginning of the file and this is the end of the file if I allocate a block backwards I'd have to move everything else anyways so in this case I'm screwed I can't grow it because if I grow it by one block since everything needs to be contiguous in a row I can't grow it easily because it would interfere with the blue file so I would have to move it and make it bigger so is this a great idea if you are constantly making files bigger deleting files and everything like that is this a good way to store files in your file system probably not we probably read and write files a lot of the time if you have to move so say that red file you know was your 50 gigabyte Blu-ray or something like that and well I guess you wouldn't modify it all at a time but say it's my stream recording it's getting bigger and bigger all the time I don't want to keep on moving it and just essentially wasting my SSD and things like that so we need to do better with it than this the system is going to be really fast though because well it's really easy to describe a file everything is all in a row you don't really need much information so just what starting block and the size of it and we also have fast random access because well no matter where I'm trying to access in the file since I know where it starts and say I want to access I don't know byte 10,000 well if I know where byte 0 is and they're all in a row I just start at the beginning and go to byte 10,000 and I can know what block that byte 10,000 is at really easy with just one simple calculation so it's just like the offset block size take the floor of that so if my blocks are like 8 kilobytes well it's probably on block 1 so not on block 0 block 1 so the problem with this is files can't grow easily so they have internal and external fragmentation so that's a new fun term that we will also get into once we talk about memory allocation come up with it here so fragmentation basically just means you are wasting space internal fragmentation means you are wasting space within a block external fragmentation means you are wasting space between blocks so I have internal fragmentation because well if I need an entire block for a file and say they were 8 kilobytes but my file is only 10 bytes I just write hello world or something like that it's 12 bytes well in that case I still have to use an entire block so most of the space is going to be wasted and that's internal fragmentation and then external fragmentation is well when I can't actually use blocks because they are not big enough so in this case if I made the red file bigger by one well I would have a hole of one between the red and the blue but if I never make a file that is only one block big I can never fill that hole and it's going to be wasted forever so that's typically a problem once you come into memory and yeah this is the same problem once we get into memory allocation so like things like malloc would also have to deal with this problem so we can do a bit better so everyone likes link list especially based off lab whatever what was it for everyone likes link list for some reason so what about if we did that for file system so we just do a free list like we had a free list of pages what about to describe a file we just link a bunch of blocks together so in this case if I have the green file well oops I would also save some bytes at the end of each block to point to the next block so in this point if my green file is six blocks large well it could start a block zero and then the index zero block could be this block and then the next block could be this next block could be this and the final block could be this one so pros and cons for this so how do I describe my what data do I need to describe my green file here yeah so if I assume that each block just has a pointer to describe this whole file I just need to know where to start where the head is so that seems to be fairly space efficient and in this case I don't need contiguous blocks at all one is as good as the other so if I want to make this green file bigger I could pick any old block and just point it to that one right so growing files is easy shrinking files is easy in this case because all our blocks are the same so any questions about that part alright what's can anyone guess what is really bad with this part and to give you a hint it has to do with performance yeah yeah so these blocks are pretty big oh yep yeah there's a few things so if I want to go to block 6 or block 4 or something like that I always have to start at the head and keep on going to block 4 that's not good I have to keep on jumping through all the pointers and like you said the pointers are really spread out so one pointer is on one block another pointers on another block which wouldn't be in cache so I'd have to read it we can assume if we're having an SSD reading each block this is fine as any other block they don't have to be sequential but I still have to read every single block every single time I want to access a pointer which would be slow so how would I make that go slightly faster without fixing your issue use a cache I could what about if instead of could I move where I'm storing these pointers to make them closer together yeah I could just put all the pointers on its own block if I really wanted to and just have a gigantic array of pointers so that should speed up things a bit more so that's a good idea and we'll see that next but first you know this is slow random access so space efficient to describe a file I just need to say where the beginning of it is and then the blocks to store pointer the next block in this case the blocks would be slightly smaller but it's probably not a big deal easy to grow and shrink files we have no external fragmentation because well one block is as good as another I can just point to it we still have that internal fragmentation issue where we might not use an entire block worth of storage but in this case our accesses are slow because all our pointers are on individual different blocks so like you said we can just all on a single block put all the pointers closer together and at least we can have slightly better performance and in fact this is the first actual file system we use so all that array of pointers is something called the file allocation table and essentially is that idea of just moving the list to its own block so if you have ever heard of the term fat 32 file system or something like that that's what it stands for file allocation table and it is exactly that and if you are using a windows computer you are using fat 32 because your boot drive must be formatted this way because windows told you so so you are using this if you are using a windows machine so that just as the same idea moves the pointer out of the block and then just it creates a gigantic table of pointers so that everything is closer together and we get a bit of a speed up yep so there would just be a gigantic it would just move all the pointers to just a giant array and that array would just be a giant global array so it is shared between them all so you would have to keep track of what they are pointing to and if they are actually in use or something like this using the exact same thing in my giant array whatever is at index 0 is the pointer of block the next block to block 0 so here block 0 points to block 6 block 6 points to block 2 and same idea as our link list thing so this kind of what you alluded to this is kind of bad because well I need just a bunch of pointers and how big is that array well depends on how big my disk is so if I have a bigger disk that has more blocks I have more elements in this global open file sorry why did I say that more elements in this file allocation table so it just gets bigger and bigger and bigger the bigger your disk is because it is essentially storing all the pointers for it so it is pretty simple and that is why it is used as a drive like it is the file system for booting but you don't really use it in the real world because it is kind of silly and kind of slow and the bigger it is the worse it is so any questions about that one yeah it might use more than one block if it needs if it is that big which is not that great right so the bigger your drive is the bigger this file allocation table is but that is the first one that is actually used and you probably use it without knowing it if you are using a windows machine so has all the same pros and cons of that link allocation but it is a bit more cache friendly so it can actually hold that table in a cache if it wanted to too but the size of that table is going to be linear to the disk size can become really large so is there anything we can do here to further increase random access speed because while this still has to go through pointers and everything like a link list can we do anything better well let's see what we can do index allocation just make it a big array so instead of just having a gigantic table of a gigantic table like a link that represents a link list of blocks I have to follow the blocks why don't I just have essentially a table per file that tells me which block I have to go so in this case for the green file the green file could just have its own array that is as big as however many blocks it is and then it points to exactly what block is being used so in this case block 0 points to block 0 on the disk block 1 of the green file points to this one block 2 of the file points to this one block 3 and now I have nice random access because it is essentially a gigantic array well it is an array that is as big as it needs to be to store the contents of the file so link list is kind of terrible the idea here is to just use an array so any questions about just using an array yeah isn't this bad if I just have one massive file that consumes a whole disk but if I have a massive file I need to keep track of that number of blocks anyways right so what can you do you have to keep track of it somehow otherwise you can just guess and you can't just guess yeah sorry so in this case so I made this red box this is the block that stores a gigantic array and this kind of goes back to our page tables it kind of looks like a page table right like it's pointing instead of pointing to pages it's pointing to blocks so it's kind of the same idea here so in this case if my file is if I have just a block that's storing all of the pointers well it has a limit right it can only store so many pointers if your file is bigger than that you would need another block or what would you need if it was virtual memory and page tables if I want more to address more virtual memory what do I need to do add another level yeah so here same idea that we'll get to later you can add additional levels to this so for this oh yep so as part of the description for the file before we would say what block it starts at if it's like a link list here we could just say what block has the array of pointers so we could describe it by saying exactly where is the red block so if I want to describe file the green file I'll say its array of blocks is at the red block alright so how big could this file be so let's say that this index block just stores pointers to data blocks only there's no other information and say our disk block size is 8 kilobytes and a pointer to a block is 4 bytes so what is the maximum size of a file that can be represented by this index block 2 kilobytes so why 2 kilobytes I'll probably have to go back and forth I'll have 8 kilobyte block 4 byte pointers so in this case if we want to know how many pointers we can fit on a block what do we need to do just yeah just divide them so 8 kilobytes is 2 to the 13 4 bytes is 2 to the 2 2 to the 11 pointers on a single block and if each of our blocks is 8 kilobytes well we could multiply them together to see how big our file could be so in this case if we multiply them together we get 2 to the 24 or in other words 16 megabytes so if your file system could only support files up to 16 megabytes would you be pleased some of you might be but some of you don't watch movies I guess or anything remotely complicated or even have images so in this case probably need to do a bit better so we want to see how big of a file we can support so if we use one block for the pointers we can store 11 pointers and each of those points to a block so we multiply by the block size to see how big our file could be so that's our idea this is file systems more persistence and we'll get into how to improve that later because that will do with our actual file system so just remember pulling for you we're all in this together