 It's green! It's green! Yeah, people are here. So everyone's done with assignment two. How many people are done with assignment two? All right, cool Simon three coming out today very exciting. So today File, I wish I had a guest in class every day, you know, like John Stewart I wish I could be like on today's show. We have a guest special guest. I Know who that person would be though. We could have a file system person All right, so today we're we're gonna talk about cash right last week we talked about file systems and what I realized as I was getting toward the end of this is that Caching and consistency kind of make nice friends together part of it as they both start with the letter C but they also have you know caching has implications for consistency and then we're going to talk about journal in this of specific approach to providing consistency in the face of caching and disc latencies, right? And also just you know failures which can happen at any time, okay? So a couple of announcements Simon two is due today midnight You guys have five late days, okay? And let me explain how the late days are going to work right because maybe this isn't wasn't clear And I think it's a little different than then maybe how some other classes work. So One of the things I don't want to happen is I don't want you guys to take late days on assignment Two and have those late days kind of come back to haunt you for assignment three, right? So this is how it will work assignment three is tentatively do the last Midnight on the last Friday night of the last week of the semester if you take five days from there You get to midnight of the last day of reading period. So the last day before exams I think that technically that's the latest that I can let the assignments go, okay? So what will happen is if you take a late day on assignment two or two late days on assignment two that will push back Your due date for assignment three for two days So you will have the same amount of time for assignment three regardless of how many late days you take for assignment two, right? So my suggestion is unless you want to finish the class early, which is a completely laudable goal You know use your late days. There's no penalty for for for using, right? But does that does that make sense to people? So if you take a late day currently the assignment three is due on Friday If you take one late day on assignment two then your assignment three will be due Saturday Does that make sense to people? All right, and then you will still have those four late days that you can take on assignment three as well, right Jason This will be up on the website. So that's a feature. We're going to add today, right? So the idea will be when you go to resubmit a patch The website will tell you if you submit a patch at this time You will have this many late days for me, right? And that'll be up to you. So so don't mess that up, right? Like don't upload something again You know and these deadlines are hard, right? Like this isn't me. This is the computer, right? So if the computer thinks that the time is after 11 59 p.m. Tonight, it's going to charge you a late day, right? And you know that's just how it's going to be so you know the website may get a little backed up at that point So maybe it's good to start submitting like 11 55 or something, right or maybe 11 50 or 11 45, right? You know if those if those 10 minutes if you really think those 10 minutes are going to make a huge difference in your Performance on assignment to my suggestion is take a late day, right? And take a late day and make sure that whatever it is that you're trying to finish up really works. All right all right So last week was file system week or week two of file systems, I guess Questions about file systems, right? So I want to go back to kind of doing a general review on Monday I know the weekend was long. I'm sure lots of exciting things happened And so, you know some of the brain cells that you were storing that information in have potentially expired, right? So any questions about file systems to generally right before we do just a little bit of review All right, so who remembers what the design goals for file systems? Well, let me start over here in this corner of the room with with with this group Keith Alex But that's too bad because this was actually covered last Monday So that's not a good excuse for your file system design goals. Good try though Right, right. I've got to be able to find files. I need to translate file names to file contents. That's good Do you remember Monday last Monday long time ago? You were one week younger. It was warmer. I think Nothing, let's see. This is this kind of room. Maybe I've chased people over to this side Okay, I'm gonna start standing over here more. It can cause a migration back in that direction. Let's see. Should I pick on today? Scott It's a single files, right? So I think there's something in the middle So I need to so the second thing is I need to support the file interface I have to a lot of files to grow shrink change move, etc. Right? Yes, optimize access to single files and then along those same line Dodging okay be efficient with memory. That's that's a potential design goals not the one that I'm looking for What's what who can do a one word modification of design goal three? To get and the next design goal Jason think I heard you mumbled multiple files, right? So if there are files that related I want to optimize access to those files, right? So today my goal is to get through caching tomorrow. I want to talk about FFS Finally, I know that I've been hinting that we're talking about FFS for a long time But I want to actually go through a couple of file system case studies to kind of close out our file system unit So we'll do FFS on Wednesday and on Friday We will do LFS a log structured file system Which is kind of a fun take on file system design something fairly fairly different right and debatable something that caused a lot of A lot of debate within the Fauston community, right? And then finally particularly pertinent for today surviving failures, right? So maintaining consistency in the light of you know people tripping over the cord, you know Maybe sectors failing on the on the disk, right? So essentially trying to survive failures particularly sort of power down kind of failures, right? Where the machine does not shut down clean, right? This is kind of a critical role Alright questions about file system goals design goals, right? Okay, so Let's say I want to actually perform a right so now we're kind of getting into Wednesday, right? What do what do I have to do? Right? Who can who can who can remind me how I'm how I need to do this Malik? So I need to locate an empty disk block right I need to mark those as in use, okay? What's the next thing I need to do? Calvin do you remember? He does not who wants to help him out front of the room Sean Michael Michael Carl What's that? Got to associate them in the file, right? So I've got to you know link them in somehow and we talked on Friday about multiple different ways of Associating data blocks with the file which I'll ask you about in a second. What's the next thing I need to do? Adjust the size of the file so that probably yeah That probably means updating some information in the inode right along with the data blocks associated in them Which might mean updating the inode or some sort of index structure, right? And then finally what do I need to do? write the data right and You know the implications for consistency are that from the perspective of a process all these things need to look like they happen together Right, but perspective of the disk these are potentially multiple implications to different parts of the disk different disk blocks right and depending on when a failure happens I could have all sorts of problems right which we'll talk about in more detail today. All right Finally so rewrite right who Who remembers so rewrite are essentially the goal of the file system is translating the offset Into a data block. I need to find the data block associated with the offset of the file We talked on Friday about three ways of doing this who can give me the first one we talked about How do I associate data blocks with a file? Scott link list Starting at the inode right so the inode contains a pointer to the first block and then the blocks contain pointers to the next box Etc. Okay, so that's one way. What's another way? An array flat array in the third way the more a little bit nicer clever way that we talked about a Multi-level index right and we talked a little bit about the pros and cons of these Which I will not get into in the interest of time. Okay So brief overview kind of jogging your memory about the stuff that we covered last week Questions on this material before we go on all right Slow crowd again today. It's good. All right So, you know, we got into this a little bit. So we're doing this is also going to be a little bit of review, right? Canonical way operating systems make a slow thing look faster. What do we do? Use a cache right so we take a smaller faster thing usually with different properties and We put it in front of the big slow thing, right? And that's intended to make the big slow thing look faster by exploiting properties of the small fast, right? In the file system, we're using memory, right? And so memory has these nice properties is very fast access. What's the problem here for? Consistency sake what is the difference? What's the main difference between memory and disk? Why do we have both? Disk is persistent memory is not right. So caching has some serious applications for consistency, right? We talked about how operating systems use memory both as memory Right to hold, you know data in a process address space and also to cache file data, right? So this is the canonical case where I have a portion of the system memory that's being used for the file caching a portion that's being used for actual pages and process address spaces and Balancing those is something that the operating system is kind of doing continuously to try to maintain good performance, right? These are competitive uses of memory obviously the more I use for the buffer cache the less I have to use his main system memory, okay? So who remembers big buffer cache small main memory. What what happens here? What what so so here's here's one question when let me turn this around when would this be a good thing? When would this be the thing that I want? When would I want to have the buffer caching up, you know taking up most of the memory? Yeah, Ben? Yeah, yeah, yeah, so maybe I'm running something like an NFS server, right? That's that's caching file contents in a in a huge chunk of memory and the server itself is fairly simple And so it doesn't need a lot of of data in memory, and it's a single application machine, right? Another case where this would probably happen would be with a database, right? A database would probably want to map or cache Massive parts of the files that where it actually stores is persistent data, right? And if I had the database running is the only thing on the machine eventually over time I might see the buffer cache get really huge without there being much memory and used by the application or applications, right? But what what sort of problems could this cause let's say I have a different Maybe more typical scenario. What what would what's the worry here about performance? What could happen? Swapping, right? So if I drain so much memory out of the memory pool that the processes are going to disk constantly to Because of page faults, then I can produce thrashing in the memory subsystem. This could be bad, right? Okay opposite case Small buffer cache large main memory. So so again, when would this be a when we could this be potentially a good thing? Well, I'm not using much disk and and what's what what can you can you come up with an application that might not use? Files that much Yeah, some big computation it might read a little bit of data from a file Start some massive computation that sprawls all over memory and then just like write a small result, right? So some some big computation intensive thing might not actually use the file system that much while it's running, right? So it might be okay to trim the buffer cache quite aggressively in order to provide it with a huge chunk of its address base that Can use efficient, right? but If this gets if the buffer cache gets too small then I lose out on swapping but my file axis is extremely slow So anything that's dependent on IO is is going to start to slow down, right? so it turns I mean I was thinking about this and it turns out I don't really know what sort of Algorithms or heuristics modern systems used to tune this balance, right? And I would love to find out I'm sure there's been some sophisticated Approaches into figuring out sort of where you know, where do I how do I draw this balance to essentially what am I essentially trying to do here? What's the overall goal, right? What's the overall goal? I'm trying to find some balance between the buffer cache and main memory that reduces what? to what? To the disk, right? On some it's not it's not a perfect analogy because throughput can depend on other things But at some level you can you can see this is trying to find this balance to reduce disk access because the disk is just so Terribly slow right and if I can find a place where I have enough memory that I'm not swapping that hard And I have enough memory in the buffer cache that I'm not hitting the file system that hard then then I'm in a good place Right, so this is kind of the overall all right And on Linux we talked about how there's a swapping this parameter that has a fantastic name And in my like as far as I can tell it's a little bit mysterious as far as how it actually operates, okay? So the last thing we got to just briefly was was a discussion of kind of where where are we going to put the buffer cache, right? So what were my two options, right? So so here's this this canonical view of kind of what the file system looks like from the perspective of the operating system I have the file system interface that's provided to processes, right? This is the system call file interface open close read write do whatever, right? Those calls in some way or another depending on the operating system I'm running on have to get actually mapped down to a concrete Implementation right and this is usually done in a flexible way Because I want to allow myself to support multiple file systems, right? If I baked this in to the system and I coupled these two together Then I would essentially only be able to support one file, right? So I would ship Linux and it'd be like if you don't like ext4 It's like when Henry Ford made the you know the first model tees, right? You know you can have any file system you want as long as it's ext4, right? So so and that would would not be great, right? and then So these guys, you know have their own implementation and you know They do their own things and they arrange things differently on disk, but fundamentally the disk interface is the same, right? So what are these guys doing they're they're doing all sorts of complicated logic in here and accessing their own data structures But from the perspective of the operating system what they're doing is they're making read and write block request to the disk, right? Disk interface is very very simple very thin So where are the two places that I can put my file system buffer cache? Where are the two places I can sneak in here to make sure I'm on every file path? So I can put it on the bottom. I can put it on top of the disk. Where's the other place? I can put it so I could put it up here, right? And so I could put I could put a buffer cache essentially below or right above the virtual file system layer Or I could put it down here, and I could put it above the disk, okay? And I think we got into the implications of this so if I'm above the file system, right if I'm up here What objects am I caching? Conceptually what objects in my cache? Files, right? I mean the operations that I'm seeing are operations that are on files, right open-closed read, right? These are file level operations. They operate on files or the file abstraction, right? Okay, so I'm caching files and directories, which are you know, of course just files, right? The buffer cache interface is what? What's the interface that the buffer cache has to support? The file interface, right? The file system file interface open-closed read write whatever, right? These are the calls that the buffer cache is going to see and it's going to rather than going to disk try to return contents from memory That's the goal of the buffer cache, right? The buffer caches is supposed to avoid going to disk when possible Okay, and so how does this actually work, right? So how if I'm at the in the buffer cache and implement it above the file system, how do I perform an open? What do I need to do to open a file? Okay, so I'm gonna look in the buffer, but let's say this file hasn't been opened before What do I need to do? Can I perform this operation in the cache? No, I need to just pass this down to the underlying file system because I don't again I mean the file is new to me too, right? I need a file to operate on and open is what creates a file, right? So the buffer cache unless the file is already open in which case I can return the contents that I have But if it's a new file, I just need to tell the the underlying file system Hey open this file, right? Because it's the one that understands how to do name translation and all these things, right? What about read? Right, so so far so far I'm struggling, right? Like the only thing I've done is pass the call through to the file system I'm supposed to be caching stuff for, right? So can I do better with read? How does read work? So right, I mean here here's when I actually start to be able to cache things, okay? I look in the buffer cache if it's not there Then I need to pass it down to the underlying file system so that the contents can be loaded, right? If it is in the cache, then I just return the contents I have, right? This is the idea of a cache What about write? What about write? How is write a little bit different than read, John? So I can write to the cache if what? If the file is in the cache, right? And how do I get the file contents into the cache? So I Probably need to do something like a read, right? I need to get the contents into the cache and then I can modify them in the cache, right? This probably, you know, remember the file system interface allows you to write one byte Right, so, you know in order in order to actually do that properly I probably need to load, you know, a whole this block potentially, okay? So so essentially at the files on in the buffer cache I'm going to load the contents in the buffer cache and then I can modify them if the file is in the cache Then I modify the cache contents, right? And I'm done, easy Okay, what about close? What do I need to do on close? Can I just, so on close I just throw the file out of the cache, no problem Yeah, yeah, I better make sure I flush the contents through the file system if the file is dirty, right? So if a write has taken place to the file, I better make sure the file system actually knows about those contents Because otherwise the next time I do a read I'm going to get the contents on disk and the contents on disk won't reflect the modifications the processes make, okay? Questions about this Right, this is kind of, we haven't really talked about how to do a cache I guess I kind of thought we had because we talked about the TLB, but the TLB is a little different, right? So does this make sense, conception, all right? We'll walk through another example when we talk about disk blocks Although maybe we won't, I can't remember if you're not Okay, so what are the pros to putting the file system above the You know, what it's sorry, what is the, what are the pros of putting the buffer cache Above the file system, right? And having it support the file system interface, right? Remember, we're comparing this against putting it at a lower level and having it cache disk blocks, right? So what's what's one pro here? What's that? So I see file objects, right? Why is that so so if I'm let's say I'm down at the Disk block interface, right? Do I have any idea and it says oh do a read to block 32 Do I know anything about what that block is or what its contents mean or what they are? No, I have no semantic information, right? So the nice thing is I see file level operations, right? So for example, if someone does a read from a file, I might be able to actually, you know What we talked about is I can load the entire file, right? Because I know that it's a read to this file and I can ask the file system to actually read the entire file maybe rather than just the contents that the Process asked for meaning that now I have everything in the cache and further reads if I'm expecting there to be further reads We'll hit the cache, right? What are the cons? Particularly when it regards to consistency what what what's what's what's the biggest problem with this approach? Anybody any ideas? The biggest problem here is that I'm hiding file operations from the file system Right so these these calls to things like read and write are not actually some of them are not going to the file system Right the file system never sees it Okay, so now what I especially with right read who cares right reads are on some level very different than rights because they're not Not they're not modifying anything on disk, okay? But because I'm hiding these file operations that the file system can't actually do any consistency Checking and and you know if I do a right to two different files on two different file systems Things that get updated on disk might be very different Right and and the file system may want for example for consistency reasons to flush certain things to disk at regular intervals We'll talk a little bit about this today, so this is the biggest problem with this approach And I think that's why it's not really used, right? What's another? What's another potential problem here? What kind of data can I not cache? What kind of data won't be in my cache? What's that? What's that? Well, I don't know about that what so remember we divided the on disk We divided blocks on disk into two groups right there were data blocks that stored file contents And then what's the rest of the disk being useful? Metadata right the block bitmaps. I know bitmaps. I know tables super block all that stuff. Is that stuff going to be cached? No, right. It's not a file. It's never actually written as a file It's updated by the file system by modifying disk blocks, but it's never going to be in my cache, right and and so Finally, what's the problem with not caching those blocks? They're they could be really hot right they might be really hot blocks Sounds like a like a hot stone massage or something right they could be very hot blocks Right meaning that you know every inode operation might have to hit the inode table right every time I access a data block I have to update the the block bitmap right so these get hit a lot and if they're not in the cache And they're always having to be done at the disk level then that could be kind of terrible, right? All right, so let's talk about Below the file system. So let me back a little bit Right okay, so now we're down here right So now we said okay putting this up here is a little problematic We're hiding information from the style system. There's a lot of things we can't cache So let's go below What are we cash? What is the cash? What are the what are the objects in the cache? They're disc blocks, right? They're just disc blocks 4k disc blocks. That's nice, right? We like that something nice and consistent about it, okay? What's the buffer cache interface? Read block write block. It's the disk level Interface right, you know read this block with some block ID write this block, right? We're like we're literally we're just up we're just telling that you know We're passing these commands straight through to the disk, right? So the disk doesn't know about files It doesn't know about super blocks. I know tables whatever the disk just knows here's an address And I know where to seek to to get that block, okay? So pros to going below Right, what can I do below the file system that I couldn't do some of these are as you might expect just the Inverse of the cons of the other So now I can cash everything right anything that the file system is using any buck that is read or written I can cash right and so I can cash the super block I can cash. I know tables whatever, right? This is great Okay, what's another pro? So okay, it's independent the file system and also allows the file system more control, right? Because the file system code is actually being called on every file operation, right? So every call through the file system interface will be processed by the file system, right? And so for example and usually cash is provide ways for the file system to force things to disk, right? That so when I had the cash above the file system there Let's say I did a right and the file system wanted to make sure that that data was on disk for whatever reason, right? It's potentially there's potentially no way to do that. I mean I might be able to tell the cash don't Ever cash this file or something, but it's kind of gross here What I can do is in the process of doing the right I can issue commands to the cash to say, you know flush this data block Right like you can keep this data block in the cash, but I want you to write the contents to disk a medium, right? So and the cons are essentially what we what what the pro was for the other approach We can't see file semantics all we see is this stream of block IDs Which means it can be very difficult for the cash to provide certain types of performance improvements So let's say I wanted to do read ahead, right read ahead is a pretty common file system operation It's predicated on what assumption, right? So so what what do you think is a common access pattern for a lot of files? particularly things like media Sequential access right when you play an epi three file the contents are ordered, right? I mean why not they might as well be they don't have to be they could your MP3 file could have its own internal index And it could have stuff all over the place, but but why do that? That's weird, right? So I mean stuff's just laid out in the MP3 file start to finish You know the samples that that correspond to the start of the tracker at the beginning and samples that correspond to the end of the end Right so when I process that file when I you know when iTunes reads that file It starts at the beginning reads the metadata and starts processing samples and essentially if you look at what it does It's just reading sequentially over the entire file If I can detect that then I can load the future contents, right? So I may watch the first couple of reads and I may see that a pattern is emerging and then the file system may start to Pre-fetch stuff right so it may start to say hey You know I know that he just read the fourth block of the file and and before that he read the third block and the second block And the first block so when I go get the fourth block I'm going to get blocks four through ten right and then if I see this pattern continuing I make it larger and larger chunks Why would I want to get large chunks of the file at once? then In general what we'll and we'll talk about this I guess we didn't talk too much about disk head scheduling in general when you're operating on when you're when you're operating with disks the more The more operations you can give them at once the better Right, so if I have ten operations to do right if I tell the disk one at a time Which operation to perform what's going to happen and it's going to seek where it's going to bounce all over the drive Right, let's say I have ten random blocks to get and I tell the disk one at a time get this block get this block at this Block so it's just going to go there go there and the head's going to be bouncing all over the place And it's going to be slow. Okay now. Let's say I tell the disk. I want all ten of these blocks now What happens the disk and planet and it can probably can do those blocks in one path Right, so it orders them from the inside to the outside or vice versa And it just does all the seeks in one direction and it gets everything right and actually We'll I have later in the semester planned a kind of a neat case study of one Technique used on windows that actually exploits this very proper. It's kind of a fun Fun combination of some things from file systems and some things from the memory management layer. It's a fun little thing, okay? So but below the file system and this is essentially what what modern operating systems do this is where the cache is Okay, it's a block disk block level cache, right? So now let's talk about consistency So I've got this caching thing and the cache is great, right? And and I think I think I told you guys this last time But it's too bad We're not doing assignment for because the buffer cache provides this incredible performance approved, right? Like when people did assignment for normally it was it would really kind of blow them away How much faster file system tests would get once their buffer cache board, right? At least in order of magnitude maybe two orders of magic, right? It really helps memories fast the disk is slow, okay? What's the problem here though for consistency, right? We talked about this earlier. What what why why is the cache? Why does the cache matter for consistency? What what can the cache cause to happen? then Right, so Objects in the cache are lost on failures, right? I mean they're stored in in volatile memory So power goes down those bits are gone and Remember this is this is you know, this is why this matters, okay? Every almost every file system operation involves modifying multiple disk blocks You know reads writes a pens creates whatever, right? You've got all these on disk data structures, and I've got to touch the disk in a lot of different places and If some of those and essentially on some level the operation is not complete until all Those operations complete so you know if I'm adding a file to a directory if I'm creating a file and added to a directory Until I made all the necessary modifications and those modifications are actually on disk That that that overall operation is kind of like a database transaction, right? It's not in a consistent state and so if the cache Holds some of these pieces of this operation in a not in in volatile memory Then it's possible that when the power goes out You know I'm going to have a problem reconstructing what was happening, right? So on some level, you know caching doesn't create this problem right if I have to do multiple disk block modifications in Order to maintain consistency. I always have a window of time when a power failure can cause an inconsistency to occur But what caching does is caching can potentially expand that window of time, right? Because as things sit in the cache I'm giving a bigger target, right for you know your little sister or my dog to trip over something and whack the computer Right. I'm giving them a longer time interval when this can happen. Does this make sense? Okay So again remember I mean if I was creating a new file I had all these different things to do right I had to allocate an I know I had to allocate data blocks I had to associate the data blocks by modifying the I know I had to so I had to modify the Directory and then I had to finally write the blocks right and then we talked a little bit about what can happen if I get stuck or if something fails at any point here, okay, and I just said this okay, so right I can leave things in inconsistent state and caching exacerbates the situation by giving me a longer time Alright, so let's so let's say okay, you know I've put the fear of of you know of power failures in you Of any sort of failure and now your approach is I want a consistent file system What is the safest way to achieve this keep flushing buffers or what does this mean? What does this mean from a caching perspective? What should my cash not never do have any ideas? Anybody have an idea? Yeah Flush the cash, but what does this mean? What do I need to flush? Don't buffer rights pretty simple, right? Don't buffer any rights. Just let the rights go straight to disk, right? Anytime I make a modification to anything that modification does not now it may modify the cash, right? Why would I modify the cash even if I'm going to? Write the data immediately out to desk So the reeds work right because the next time I read this I want the data and the cash to be to be fresh, right? But so so we there's a there's an approach to cash, which is called a write-through cash It simply means that rights are never buffered in the cash, right? They they they sit there. They sorry Getting ahead of myself. They go straight to disk, right? Okay, now let's say you're let's say you're like I don't I don't care Man, I keep wanting to use more colloquial and less video appropriate language So let's say you say I don't care at all about consistency. I just want my file system to be really fast Now, what do I do? What's what's what's the other pole? What's the other position in the universe that you can take here that will maximize performance on some level? And for how long? So you close the file or I would I would argue that that you can I mean you can cash all operations into blocks Or evicted meaning they're removed from the cache for whatever reason Maybe they hasn't been modified for a while and I need space for something that is actually actively in use But you know on some level you can cash these things until the machine shuts down, right? I mean that would be the longest period of time that you could leave things, okay? So I think it's clear where the trade-offs are here, right? The more data I cash the better things are for performance the more often I write things to disk the better things are for Safety, okay So what about and there are plenty of middle grounds here, right? So so what's what's one middle ground between you know? Buffering until things are evicted and writing back immediately. What else could I do? What's kind of a straightforward approach? Yeah, you're getting closer Luke. Do you have an idea? Then So so in general I want to do that even with the right through cash right because I want reads to work right but but again So one option a write immediately option B wait as long as possible. What what's the what's the what's the trade-off here? What's what's a middle point John? Yeah, so I could definitely be be it's similar to what we did with paging, right? I could be writing out dirty cash box periodically or I could just ensure that what? This is a variation of this What's that so so I'm still not I'm still not sure what you mean by this Oh Okay, yeah, so I could I could wait till a certain number of modifications happen to a block But what is that a variant of? Trying to get it something very maybe too specific for for an in-class question. I can just have a timer Right, I could just have a deadline I can say the longest I'll let rights sit in the cash is X right and you know With with a minimum interval X I'm going to make sure that every block in the cash gets flushed to disk right so now what I've done is I I have a Kind of a bound right I can tell you if you make a modification the file as long as the machine stays up for 10 seconds That change will be reflected on this right so that this is kind of another right All right The other thing okay, so the other thing I can do which which a number of file systems do do is I might not allow metadata to sit in the cash right because this stuff is important Right, we talked a little bit before about if I forgot to if if a modification to the superblock or To one of the disk data structures doesn't make it to disk that can cause a lot of problems, right if some you know if some read that or sorry if some write that you do to you know your Assignment to patch file doesn't make it to disk who cares right like you know I don't know at some level the file system will come up and at least that file will exist and there won't be some orphan data Blocks somewhere right and you'll just have to reconstruct the contents right so it's possible that I keep file system Metadata structures like the superblock. I know maps of bitmap those get those get written immediately Those are right through right through and everything else is is held in the cash Right, so this can make sure that again the file system data structures are consistent the file system data may lack right and then finally I just want to point out that there is an interface to Forcing the file system or asking the file system really to sync synchronize itself Right, and there's a Unix system called called sync There's also them called f sync right sync syncs an entire file system that says to the file system Please make sure all your stuff is on disk now I guess on some file systems this actually might not do that right but you know You're asking politely and and some on some biosystems You would hope that it actually does do that and then if I want to sync a specific file I can I can request that a specific file be be written to disk right through a separate All right, but let me let's talk about a little bit of a different approach to consistency right so We've been talking here about I mean, what's what's what's at the root of our troubles here, right? What is not atomic? So I'm making these modifications and usually involves what and that's not atomic right what what's not atomic from the So writing multiple disc blocks, right? I mean that's conceptually not atomic right I asked the disc to write a block and then I have to ask it to write another one in another right What what potentially might be atomic from the perspective of the disc? writing one block right one disc blocks nice Okay, so I don't know why I find it so funny. So it's already in a disc block might be a top right and How many how many people ever read Tom Clancy books? Is it just me? Oh my god. I feel so alone Anyway, so One of the main characters and you guys are gonna care about this who cares I won't tell you Kathy Ryan He's you can go look it up But but on some level, you know when I was a kid and I was reading these books and someone thrall I thought this like you know this she was the first person to say this right which is probably false but you know what one one way of trying to ensure file system consistency is simply To to obey this maxim. Okay, if you don't write it down it never happened now. That's true, right? I mean that's that's our problem right and the problem is that certain things aren't getting written down to the disc But the question is can I keep a more compact representation of things that allows me to record what operations? I did and didn't do okay, and this approach is known as journal I did during journaling is a modern file system technique. It's in use by a lot of file systems and The approach is actually quite simple. Okay, I maintain a special data structure on disc. It's called the journal When I make changes to the disc, I write the changes that I'm going to make in the journal, right? I essentially the first thing I do is I tell the journal what I'm about to do Right, and then when I get around to actually doing it right and that may be delayed by cashing or other things I mark those things as completed in the journal if I fail I Load when I load the file system I open up the journal file and I look in there, and I try to see what things hadn't I done yet Hadn't I done yet. That's also not grammatically correct. I'm getting tired What what things had I not done yet? That still doesn't sound right. This is the best I can do And and I try to redo those operations right so I can use that information to figure out what needs to be done Okay, so so let's so let's make an example of this right and here's my little like dear diary sort of Clearly this is not what you would write in the journal right but but but this is what it would mean Okay, so let's say I'm going to create a file. What do I need to do right? So I say dear journal this is what I'm about to do right or here's what I'm going to do today I'm going to allocate this I know right and I put down the I know number right I'm going to associate These data blocks with this new file right and then I'm going to add this file to a certain directory Right to the directory with I know what up. Okay, so I mark this in the journal and then I say that's it. That's one operation Okay So now the question is let's say that I wrote that down in the journal and that's the time would buy and finally all of those Data all those blocks that were involved with this block 567 blocks 55 87 98 and 33 so what happens when they're flushed to disk? What do I do to the journal? What's that? Well, I yeah, I marked that that I did it right I say hey I'm finished with these things right and at some level when file system data structures get flushed to disk We mark that in the journal as what's called a checkpoint right a checkpoint means that the journal up to that point is Now consistent with things on disk. Okay, why do I why do I mark this? What am I going to use this checkpoint for? When I replay right so when a failure occurs What I'm going to do is I'm going to go back to the last checkpoint and work forward Right, so let's say I had a catastrophic failure and the machine starts up again, right? And here's what I find in the journal So I've got this message here that says above what you don't see I had you know essentially written all this to disk Right, so this is a checkpoint and I have a nice horizontal rule in there to show you And then I have this entry in my journal. I say here's what I was going to do today Allocate the sign out so again I'm creating this file and and this is what I find in the journal so now what do I need to do on recovery? I need to replay this specifically. I need to go through operation by operation and See if these things have actually happened It's possible that some of these rights actually made it to disk Right, not all of them were guaranteed to make it to this so what I can do I can look on the disk and I can see okay I know 567 is already allocated. Okay, so that means that I had gotten around to doing that, right? What do I do with the next entry? What do I need to do with this entry? What's that? I need to look at The blocks that are associated with I know 567 and make sure that these data blocks are actually linked in And it's possible that in this case. I hadn't made the modifications to the I knowed yet And so I need to associate these blocks with the file, right? And of course I would have to put down Exactly which part of the file that they were they were storing right so I could associate them properly and then finally What do I do with the next entry right that says that I know 567 shouldn't be in should be in the directory with I know 33 so how do I how do I check the rest of this entry? What's that? Well, no, no, no, so it's possible that things I'm pretty sure it's possible in this case that things can actually These things can happen in different orders, right? That's something that I could support So it's possible that I actually did number two, but didn't do one and three, right? There's no requirement that these things happen in order because of caching they might happen in different orders, right? All the checkpoint ensures that everything up to that point happened, right? That's a gear That's sort of a guarantee, right? So what do I need to do with this last entry? Malik Not sure okay, so I need to read I know 33 and I need to figure out You know is is I know 5567 in this directory, right? And I would have to have a path in here, too, right? So I'd have to know what's the path name that I was adding this this I know to the directory for, okay? And then I'm done right so this is all that's in the in the journal then when I boot this system up I replay the journal and now I can essentially produce a new checkpoint here indicating that all the on this data structures are up to date Okay, so what do I do with an incomplete? This is the last thing we'll do today. What do I do with an incomplete? entry Let's say that this is all that Ended up in the journal What what what do I have to do here? What's that I? Can't do this right I essentially just have to toss out these changes Meaning that if there's an incomplete journal and why do I have to do that? Why do I have to throw this out, right? What would happen if I actually process this state? I would leave the file system in inconsistent state and I meant to have an answer to this question, right? But yeah, I would essentially you know, I might have Well, what would happen here, right? I would have an inode. I would have a file. That's not in a directory, right? So that would be not good now. I like to have files and directories fine Right, so if I process these and that's why I need to have a start of the commit message and an end of that message in the journal Okay, so so let me let me just get through this. This is the last slide. Okay, so The night what's the nice thing about journaling at some level it seems like I've just pulled some sort of parlor trick on you Right, I told you that I have to make these multiple operations to disc blocks and now I've got to make multiple operations to the journal Okay, so what's different here? It's right up on the slide So so the metadata updates I can represent them Calvin got part of the answer for not the part I was looking for I can represent them compactly And so I can write them quickly anatomically to the journal, right? So writing down that I was going to allocate inode 567 to the journal Doesn't you know doesn't require touching the actual inode bitmap data structure, right? What what about data blocks, right? So I've got data blocks That I might have changed right what are the two things that I can do with data blocks So data blocks are different because with the data block. I've actually got a whole block of data, right? So one approach is that I can put them in the journal this kind of Is is stinky, right? It's kind of stings because it means that eventually that data block is going to be written again to the actual data block on disk And so it's going to be written twice The other option is like an exclude them from the journal, right? meaning simply that the on disk data structures will be Up to date they will be consistent, but some of the data for files may be missing This is similar to the approach where I was always writing through my my bitmaps and things like that to the disk, right? So it's possible that I'll lose some file contents, but not file data. Sorry I'll lose some file data, but I won't lose the structure of the file system, right? I won't have blocks that are marked allocated that I can't find or you know files that aren't included in any directory All right We're done on Wednesday. We're going to talk about I'm going to skip this slide on Wednesday, we're going to talk about FFS, right? So FFS was a An early file system design did a lot with the geometry of the disk and will be a good kind of case study for us to learn about How some of disk layout considerations were taken into effect