 Good. OK, sweet. So, sorry. OK, so today we're going to, so on Friday we talked about this classic file system design, right? The fast file system. And the FFS is sort of, you can think of the Godfather to a lot of classic file system designs, right? There's something particularly fancy or aggressive about what it's doing. It's a nice design. And parts of it have survived fairly well over the years. Today what we're going to talk about is something called log-structured file system. So in contrast to FFS, LFS is an example of a sort of radical paradigm shift in how disks were designed. And it's a nice example of a file system that's responding to technology trends. So a great deal of the reason why we keep redesigning systems over time is because technology changes. And we'll talk a little bit about some of those technology trends later in class. OK, so assignment 3.2 is due on Friday. Weeks are just spinning by. So at this point, hopefully many people are to the point where things are pretty much working and you're sort of hunting down corner cases that might get caused when your core map fills up or other things like that. So we will, Rajesh is going to do another one of his all nighter office hour type things on Thursday night, which sounds like fun. I wish I could be there. I think I'm going to be out of town. So we'll have office hours this week. Please come and get help. And that's where this is. OK, so for Wednesday we're going to start. So the rest of this class is sort of a mixture of some lecture material and then a few chances for us to read some papers in the systems research area. On Wednesday we're going to do the first one of those. And this is a great example of a paper because the raid paper, I mean, academics sometimes worry about having impact. The raid paper spawned like an entire multi-billion dollar industry. So that's the best case scenario when you write a research paper. So instead of having you read that, I posted a link. You can tell you're reading the classic paper where it's been scanned in on some ancient, the scanned copy has clearly been photocopied like 62 times and then scanned. It's barely readable. So rather than reading that, look over the Wikipedia article about raid. And then I haven't read this paper yet, but this looks like a neat retrospective about raid written by one of their original authors of the raid paper, a guy named Randy Katz, super famous at Berkeley. So take a look at that. It's only a couple pages long. It looks like it's kind of a light read. So please read those before you come to class so that we can talk about the paper. And on Wednesday, we'll also talk a little bit about kind of how to approach some of the other papers that I'll assign for the rest of class. All right, any questions about logistics? Questions about FFS before we move on? Oh, and then we're handing out midterms. But you guys know that. OK, cool. All right, so FFS was a file system design circa 1982. Now we're going to fast forward to 1991. How many people were alive in 1991? Yeah, very few. OK, at least it looks like more than one hand up. 1982 was zero. So what's different about 1991? 1982. Anybody? So what's that, nine years? Nine years? Yes, good answer. I like that. That's very literal. So yeah, I mean, music has gotten worse. This is clearly a degradation. I and the Tiger is a great song. That other song is, I don't know, I don't want to talk about it. I grew up with this stuff. So now you can see how I've entered the state that I'm in. I had to listen to those same things. Movies are still pretty good. Silence of the Lambs, classic, classic film. I was actually just looking up Jody Foster the other day because we've been watching the Americans, and there's that scene where Reagan gets shot by Hinckley. I didn't realize that he had an obsession with Jody Foster. That's so weird. Anyway, the things are weird. The world is a weird place. Anyway, Silence of the Lambs, good film. Gandhi, has anyone seen Gandhi? Why? Did you see it in class or something? Did they show it in high school? Yeah, good way to kill off a few weeks of class. Am I allowed to do that? Can I just show a movie for the next month? Social network. I can queue up a bunch of movies. They're really informative, useful stuff. Clearly high school teachers are getting away with this, so maybe I should be able to. Silence of the Lambs? Hopefully everyone's seen Silence of the Lambs. Classic film. These are good movies. All right, but anyway, obviously that's not what we're talking about. What's changed about discs or computers during that time period? Well, things are getting better, right? Now, disc bandwidth is improving. So my ability to get data back and forth to the disc, the buses that I'm building to actually communicate with the disk drive, are getting better, so that's good. So I can get, remember that weird thing that FFS was doing where it was skipping sectors so that it can do continuous, I don't have to do that anymore because I can actually read and actually send stuff fast enough down the bus to keep up with full disc bandwidth. So that's great. Now, computers, one of the big technology trends that LFS responds to is computers are trying to have more memory, like 128 megabytes, right? I feel like you guys should just, you have a, should have a little card in your pocket that has, for each decade, kind of typical computer specifications, just so you can remind yourself about how far the field that you're studying it has come and how lucky you are, right? 128 megabytes, I think a single tab in Chrome consumes that much memory now, right? This is the entire machine. So this is, again, this is sort of like, this should remind you a little bit of your Sys161 virtual machine. That's a lot of memory, though. That was a substantial increase in the amount of memory in previous machines. Unfortunately, seek times for discs. So remember, discs don't, I remember seeing a great talk by the guy who started iRobot, whose name is escaping me right now. And he pointed out that a lot of the trends in computer science in terms of faster and faster processors and more and more speed and more and more memory come down to the fact that you can take a bit and you can make it smaller. You can cut in and deliver little pieces and each one of those things becomes a bit. So that's, for some reason, on some level, that's what we did for a long period of time. That's where we got some of these exponentials that processors wrote and that memory wrote. Is there an active shooter, or is there someone moving a thing out there? Sorry. I like the retroactive trigger warning. I don't know. It's the second time today I've heard random loud noises. So anyway, just trying to keep us all safe. Okay, so, but disks, disk seeks are not getting faster because we're making things into smaller pieces. So disk, what do you think has happened to disk seeks? I mean, a lot more memory in the computer, disk seeks are still what? Really slow. This is still, and the problem is this is becoming a bottleneck. So before the bottleneck might have been the bus or maybe the bottleneck was the amount of memory you had in the machine or something else. But as some of these other parts of the system are evolving, disk seeks are starting to stick out as something that's really a problem. And so people are starting to think about what can we do about this? Okay. So we can use the lot of, if we can solve the seek problem, right? So this is something we'll come back to when we talk about performance. When you try to improve the performance of the system, focus on the thing that's causing the problem. The thing that's causing the problem are disk seeks. So it can become up with the file system design that tries to avoid disk seeks. Oh, and the other hand, and the other thing that's nice about this is let's put all, maybe we can find a way to put all the spare memory that we have, right? All the spare memory of that 128 megabytes that you have in your computer. Maybe we can find a way to put that to use as well. So these are sort of the design parameters of LFS. And it's, again, responding to technology trends. I've got more memory, and I've got enough bandwidth to get things back and forth the disk quickly, but disk seeks are still really slow. So I'm trying to find a way to avoid seeks. So obviously, we've talked all semester about one of the ways that we wanna use this memory is as a cache. So I can use a big cache in front of the disk to soak up some of the disk activity. This is part of my goal, right? You know, and we know what this is. This is called the buffer cache. It's something we talked about last week or the week before. And this is something that starts to become part of file system designs. And LFS really counts on this feature of file systems, which is that I have a reasonable sized memory cache sitting in front of my file system, it's right in front of the disk, excuse me, that's going to absorb certain types of traffic. Okay. Now, so the nice thing is, as the cache gets bigger, its ability to soak up reads improves. So remember, reads are totally safe to cache for as long as I want to. There's no potential harm for caching disk blocks in memory if they're just being read from. So as disk blocks enter the cache, once the cache gets large enough, the number, this is sort of like page fault rate, right? The number of times I actually have to go out and read data off the disk starts to fall for a given workload. All of the files, all of the disk content needs to make forward progress start to fit inside the cache and I'm good. So this is sort of one, again, one design parameter of LFS is counting on a reasonable sized memory cache to soak up a lot of the read activity. But unfortunately, we still have to do these writes. I still have to do writes to the disk. Remember, I mean, changes that I make to the disk still have to make it to disk somehow in order to persist. So I have to do these. So this is sort of what LFS focused on. It said, we're gonna design a system that assumes that a large cache is gonna soak up a lot of the reads. We're gonna focus on reducing write activity. Okay, so now we've talked about, again, your minds are mired in the world of traditional file system designs. But try to reverse that programming and answer this simple question. So what's the best writes as possible and write them all out together? So ideally, I let writes accumulate in the cache for a long period of time and then I write them out at full bandwidth. So the more writes I can pack together, this is always true of any disk. The more operations that I can bundle together at once and send to the disk together, the faster the disk will go, particularly spinning disks. I think, I actually flash drives, it may not matter, but spinning drives, absolutely. The more I can tell, if I can, because the other thing the disk can do and I don't think we, did we talk about disk scheduling? When we talked about disks, Carl, I'm pointing to you. Okay, yeah, no, I don't talk about that because I think it's kind of boring, but the disk is, just so you understand this, spinning disks are also doing their own scheduling internally. So if I give the disk 10 data blocks to read from, the disk looks at that set and says, okay, I know where the heads are and I know which data blocks you wanted. I'm not gonna get them in order. I'm going to try to optimize the movement of the heads so that when I'm done with my operation, I'll have all the data you wanted, but maybe I'm doing one pass across the disk, for example. So if I give the disk larger and larger and larger sets of blocks to read, it can read them all in one pass and it can actually get them pretty quickly, right? Whereas if I send them one at a time, it's like running here, running there, running here, running there. So the more I tell it at once. Same thing with writes. Does that make sense? Okay. So, you know, and I can, so again, I wanna hold writes as long as possible to get the full benefit of the short seeks that I get to do with the law. If I start to write things back, if I start to do, well, sorry, if I start to do more of a write through approach where I'm writing immediately for whatever reason, whether that's for consistency reasons, then I start to lose some of the benefits. I still get some. But the benefits of FFS are greatest when I can do lots of writes at once, because lots of writes are run all over the disk are slow, lots of writes at once to one part of the disk are way faster. This is something else that we just pointed out. LFS inodes move around. And so locating LFS inodes is a problem. So LFS now needs more complicated data structures for finding files than other file systems did. I mean, look, the AXC4 approach of just having fixed locations for inodes is really nice. It simplifies inode lookup, which also simplifies path resolution because that's a big part of taking path names and translating them to inode numbers is doing that thing where re-directories map the directory component to an inode, read the next item. So having this Part B performant is important. LFS does a fun thing. It caches an inode map that stores the locations of the inode, but it also logs that inode map periodically. This is just another piece of metadata that LFS is allowed to log to the disk. And so to find an inode, I read my inode map in, again, that's probably in the cache most of the time, and I use that to map from an inode number to a block on disk that contains that inode. All right. Yeah, pretty much all of the file system data structures can also be logged. So all those things we talked about that we haven't really talked about very much, like the free block bitmaps and the inode bitmaps and stuff like that, the sort of basic file system internal data structures, I can log those two. Anything can be put in the log. Now here's the issue. So this is great. First time through the disk, everything works awesome. There are no issues. Then I get to the end of the log. So you can imagine starting the log at one side of the disk, one end of that flat array, and just I increase it until I get to the end. So what happens at that point? Do I tell the user disk is full, no more space? No? Okay. What happens at that point? Yeah. Yeah, just because the log hits the end of the disk, there's a lot of junk. There's a lot of dead stuff in the log. You know, if I made multiple changes to a file, I've got maybe multiple copies of inodes for that file that are sitting in the log, only the latest one is up to date. Everything else is wasted space. So this is this issue of what's called an LFS, log cleaning. And this is where we get to this controversial part. It's like one of these mirages that you can only maintain for so long. Like I can only, I'm trying to think of an analog, but I mean there's certain things in life where you can get away with kind of faking stuff for a while and then all sort of builds up and adds up and at some point you just have to go back and clean up your mess, right? So the first time through the log, awesome, everything works great. Now I've got to clean the stupid thing and this gets tricky. Okay, so now again, conceptually you can talk about cleaning any portion of the LFS log, like the entire disk. I can clean the entire disk at once. Typically for performance reasons, LFS file systems don't work that way. They clean parts of the disk because what would be the problem of cleaning the whole disk? I mean, we haven't talked about cleaning yet, but cleaning sounds like it might be slow, why? What's slow? What part of the computer is slow? What does cleaning involve a lot of? Reading and writing to the disk. So yes, cleaning is going to be slow. So let's say I wait till I hit the end of the log to clean and I have to clean the whole disk. What happens? Like you're in the middle of trying to submit assignment 3.2 and your computer's like, sorry, I need to clean the log. Come back in six hours, right? Like you don't want that. So we try to do it in little pieces so that I can keep allowing the user to use other parts of the file system where I still have some space while I'm cleaning out other parts. So again, typically LFS will break the file system into chunks. Once one chunk of the log fills, I clean that part and then I, but I allow the log to continue into some other part that's already been cleaned. Okay, yeah, to some degree, yeah. I mean, I don't know what defragmentation in Windows does, right? Have we had this conversation already about defragmentation? I think we did, okay, sorry, I'm not gonna say it again, but yeah, other than move colors around on the screen, I mean, and generate a lot of disk activity. So it's possible, yeah, it's very, yeah. No, because I want to put everything at the end of the log, right? Once the log goes past a spot, I don't want to back up, right? I mean, all LFS cares about is the free space in front of where the log happens to be. Stuff in the past, let it be until I clean it and then we'll come back. Yeah, good question. Yeah, I don't back up. I can back up for reads, right? I can pull stuff out of the log for reads, I have to, because that's where all the data is, right? For writes, you know, I'm very fixated on getting all of my writes into one spot so I can minimize this. Okay, so how do we do this? So here's an example segment. This segment contains some active data. Maybe the red stuff is file system metadata, like iNodes and lock bit maps and stuff like that. The green stuff is actual file data. And so this is like compaction. This might be a lot like defragmentation, actually. So I find the stuff that's in use, I write it all to the beginning of a segment and then I find other things and then I'm done. So when I'm finished, now what does this require? Based on the diagram. What's that? Yeah, I need two segments to do this, right? There's probably clever ways to do it in one but that's usually not what they do, right? Usually what I do is I find a completely clean segment or a mostly clean segment. I can clean into the, like this could have some stuff at the beginning of it and I could write it to the end. But essentially what I'm doing is I'm taking one segment that's fragmented. I'm walking through the entire thing and just rewriting all of its data to the end of another segment and when I'm finished, everything should be pretty compact. Yeah, I don't remember actually, I have to say. I think it was supposed to be two different types of data, like file data and inodes and other sort of file system metadata. So when I'm done, I have a clean segment in this case and I have another segment that's mostly clean. So I could keep logging into the end of this one as well. So this is where LFS starts to go from being an incredibly good idea to being a debatably good idea. And this is what people debated about LFS for a long time. So if you look at the debates and the arguments that people had over LFS, which got fairly heated, I'll show you some email exchanges, they're pretty awesome. Then this is the part of the systems operation that was difficult for people to reason about and could account for big performance differences. So for example, let's say I run a test on LFS, a benchmark on LFS, and it never does any cleaning. The performance is gonna look awesome. But the problem is, again, it's all a mirage because as soon as I loop around and actually have to start cleaning, remember the first time through the disk I can get away with not cleaning. At that point, I've gotta clean all the time. I've always got dirty segments that I've gotta be compacting, or frequently, let's put it that way. And so on some level, the first time I go through when all of my segments are clean is like this grace period where everything works really well. And as soon as I have to start doing cleaning continually, performance is going to degrade. How much it degrades is what people debated. So, and there are also lots of questions about integrating the clean operation into the rest of the LFS system. So when's a good time to run the cleaner? There's no right operations, okay, let's extend that answer. It's a good time to run the cleaner. Yeah, okay, can we like merge these things? Well, there's no operations at all. The disk is quiet, right? User might not be using it, but I might be running like a virus scan or something like that. So I want to find a period of time where there's no disk activity. And then I want to fire up the cleaner and get to work on scrubbing some of these dirty segments. But on systems, so this is one of these questions that started to cause people to debate how good LFS was. So on user-facing systems, you can make arguments that there are long periods of idleness. So for example, your smartphone says like hours a day. I mean, maybe it does, unless you're one of these people that has a serious smartphone addiction. I sat next to, I was on a plane recently for like seven hours. And I sat next to someone on the plane who the whole flight simultaneously watched a movie and played this like game with like little colored, like I don't know, fruits and stuff like that. Do you guys play these games? What's that? I don't know what it was, but this person played this game for like seven hours straight. I'm not kidding you. I kept looking over me like, are they gonna stop? And they never stopped. And they were also watching TV at the same time. It was amazing. I was like, I don't know what your problem is. I hope that this is the only time in your life that you're playing this game. Like just get it all of your system on this one flight and then you can go back to being a normal person. But anyway, it was strange. Has anyone ever? Well, anyway, I won't ask. I don't want to know the answer. It just seemed like a weird game to get that into. Anyway, I don't play computer games so I'm not familiar with the stuff. So anyway, so unless you're that person, your smartphone probably has some idle periods where it's not doing anything. And I could sneak this cleaning in there and get some useful file system work done. But there's other types of systems that are not designed to be idle. So big cloud service providers do a lot of work to make sure the machines are not idle. It's idle machines are a waste. You talk to people at Google, they'd be like, we paid good money for that machine and we wanted to run as hard as possible until it falls over and dies. That's what it's for. If it's sitting there idle, we did something wrong. And of course, you can't always make this happen. There are big diurnal shifts and things like that. You can imagine the Google's data centers in North America are a lot quieter at night than they are during the day. But to some degree, once I buy a server, I paid money for that and what I purchased is a certain amount of computation over a certain number of years and I wanna put all of that to work. If it's sitting there doing nothing, it is not useful. So on some instances that don't idle much, this maybe, it's harder to find times to clean. What size segments do we clean? So here's another interesting trade off. Large segments give me the most bang for my buck. I get the most space left when I clean a large segment. But it also takes the longest period of time. So if I'm trying to do something while the user is at home, if I do a large segment that might take an hour and the user comes back after five minutes, I'm in trouble. On the other hand, small segments increase the probability of finding this glorious segment that the log cleaner dreams about at night, which is the segment where everything is dead. If I can find a chunk where everything is dead, cleaning it is a trivial operation. I don't have to write a single block. The amount of work I have to do to clean a segment scales with the amount of live data inside the segment. I shouldn't say it scales. I still have to read the whole thing, but the amount of rights I have to do, the amount of data that needs to be duplicated is a function of how much live data there is in that other segment. The biggest problem with the cleaner in terms of the systems community, reasoning about the performance of LFS was that the cleaner is very workload dependent. So you can imagine thinking about this, right? Really, read heavy workloads don't generate a lot of right traffic, and so the system doesn't need to be cleaned as often, whereas write heavy workloads or other types of operations that generate any kind of right activity. So for example, renaming files. When you guys start to look at file systems benchmarks, there's all these weird benchmarks, like benchmarks that take a file and rename it millions of times. This is a way to put stress on a particular type of operation that file system has to perform. Renames in LFS potentially create right traffic because I have to re-log the inode, the directory that the inode was in, and things like this. So again, this is one of those things where two different groups could take two different benchmarks, run them on similar LFS systems, and one would claim LFS is awesome, and the other group would claim LFS is a piece of junk. And people did this. So there were also issues with reads. I'm almost done with all these caveats, right? If the read, if the cache doesn't, let's say the cache doesn't soak up as many reads as you were hoping. Let's say there's still some substantial read traffic that's going down to the disk. What does EXT4 do? EXT4, I know EXT4 sounds boring now. I've got this static layout. LFS is so much sexier and cooler. But what is EXT4? Remember the cylinder groups in FFS that made their way into EXT4? What is the goal behind doing that? Why divide the disk into these smaller pieces? What does that help me do? I know I'm going to have to do some seeks to get from the inode to the data blocks and things like that. But those systems that do more static file systems and do more static layout, what are they trying to do? Yeah. You try to keep stuff close together. You don't make these promises that all the writes, like these pie in the sky promises, like all the rates are going to go at the end of the log. No, you're like, you know what? I'm just going to do average, right? We're just going to try to put stuff sort of in the same place on disk, which is fine. The problem is, let's say LFS, again, let's say I have to do a lot of reads on an LFS system because the cache isn't working as well as I thought. What problem has LFS created? In order to get this great write locality, what does LFS do? Yeah. Four sakes reads me. Well, four sakes read locality, remember? Like, stuff can be anywhere on disk. There is no high level planning about where things go in the log. If I have to start doing lots of reads, I could be seeking way back to the very beginning of a log if I have to get something that hasn't been modified for a while. So I have totally abandoned any attempt to keep reads close together because I'm pretending that the cache is going to soak up all this for me. If the cache doesn't do as good a job as I wanted to, blocks can be anywhere. And so all the improvements I get by being able to stream reads, sorry, stream writes at full bandwidth to one spot may be lost if I'm suddenly having to go and run all over the place to grab reads to fill the cache. So this is another questionable issue. And again, something becomes very workload specific. If my workload fits inside the cache, I might be good. If the workload spills out of the cache or touches a lot of disk blocks, then I start to see problems with reads. All right, so just to sort of this fun little timeline, so 91 was the original paper about LFS. Now there were also sort of two camps here. There were the FFS guys that were building kind of a more standard design file system. And then there were the LFS guys who came in who swooped in on their high horse and were like, this stuff is stupid. We can do a way better job with this clever trick. And so these groups went at each other for a little while. So Margot Seltzer was involved in this, the person who also led to the creation of the OS 161 torture chamber you guys are enjoying. So as soon as it happened, it's over a period of a decade, LFS would publish numbers, FFS would make some improvements. They would publish numbers claiming they were superior to LFS. So, and these are like fighting words for scientists, like poor analysis. That's essentially your stupid in scientific terms, poor analysis. What else is there? It's all analysis, right? If you don't do analysis, well, you're not a good thinker, you're done. Yeah, oh, they also implemented it badly. I like that. This is great. This is like, this would just ruin my entire world. You're a bad hacker and you're stupid, right? Nice work. And you can't choose benchmarks, but that's just like a, that's just the cherry on top. So now 95, another part of paper by Margot. So, and now we're getting a little bit more specific here. So look at the critique, LFS offers superior performance on small file rights for large files. The performance is comparable. If I tune FFS properly, so one of the arguments I think the FFS community made was when the LFS people were comparing against FFS, they didn't do a good job of tuning the FFS file system so they weren't seeing representative performance. And so now I'm seeing a little bit better than LFS and comparable. So in the best case, I'm a little better at LFS at the thing I was tuned for and the worst case, I'm comparable. So all of these magic improvements that the LFS file systems were promising are not happening. So, and then you know, I'll just describe this paper as misleading in many ways. You lie. Okay. I need questions about LFS, this is kind of a fun. Now there are definitely aspects. You know, I think there are some features of large structure file systems that are made their way into modern file system designs but not really whole hog as far as I know. Yeah. Clusters of logs are adjusted based on workload. Yeah. It's hard though, right? Because to some degree, the two things are sort of intentioned with each other. If I'm gonna write everything into one place, then I lose a lot of control over where things end up. And that's the control that these other file systems are trying to get at. So even more exotic FFS variants, things that use fancier data structures, I still, to get good performance, where those data structures are on disk starts to matter a little bit, right? Now, if you think that now there are some, I think I had an exam question a few years ago. There was some upswing of interest in LFS for flash file systems, flash-based file systems, simply because LFS can amortize writes and for certain types of flash devices, that's important. So there are flash devices where I can read from like a 256 byte chunk, but I've actually wanna modify a byte. I have to erase an entire 32K sector. So that makes writes extremely expensive. And so being able to amortize writes in the same place in an LFS manner might be useful. I don't know. I mean, it would be a great thing to look up. I mean, this is like, and this is LFS vanilla. I suspect that some of the same people worked on this project for a while and I suspect that there were variants and offshoots of LFS that tried to solve some of these issues. But I think on some level, the locality thing is really fundamental, you know? Do I have a top-down design that allows me to do sort of best case or average case performance given certain assumptions about files? Or do I go forward and try to put all the writes in one place and then lose control over where things are on disk? Yeah, it's a good question. Yeah. Yeah, that's a great question. So I'm assuming that when you're doing that process, I need to know things like, where is the current inode for this file? If I read a data structure and it's an inode and it's not the current inode, then I don't understand. I'm probably something similar for data blocks. Yeah, yeah, that's a good question. So you were saying I could jump, I could jump through. I mean, I lose some of the benefits of being able to stream at full speed because I still do have to seek from place to place. So the seeks, you know, potentially I might have to seek across large amounts of active data. So that becomes an issue. I don't think other than that, there's any real reason that's not done that way. They could certainly keep track of where things are in the same way. Yeah. Yeah. I don't know. I suspect the answer is yes. I don't know exactly what techniques were used, right? Yeah, that's a great question. How about that school stuff? Okay, I have a couple notes about reading papers. So we will consume the next minute of class. So it's time for a meme. Now that I'm saying it properly, thank you. Okay, so, okay, so look, I mean, a lot of you guys are gonna go on, you're gonna build cool software projects and you may think, who cares, right? Why would I ever read a research paper? And that's fair. And one of the things I sent you guys to read for Wednesday is a Wikipedia article, which is probably better to read than the original rate paper. It's actually way better because the original rate paper was original, right? I mean, it hasn't aged very well, right? Papers don't age well. On the other hand, big software companies, even if you wanna have some understanding of how things at Google work, how do some of Google's big cloud-scale services actually work, a lot of that stuff is an open source. So you're not gonna be able to find the source and read it. The documentation, in many cases, is a research paper. So things like Bigtable and all of Google's flywheel services and a lot of things that Google works on. Periodically they publish papers at scientific conferences that give you a glimpse into some of the design principles that Google is applying to build things like this. Or Facebook, VMware, does a certain amount of this. Microsoft publishes stuff all the time. So this can be a neat way to learn about these sort of systems that are actually out there in the real world solving problems at a scale that is almost unimaginable. And reading code is, I mean, reading code is something you guys should all get better at, but I understand it's pretty tedious. Like if you had to read the entire code for some of Google's services, first of all, it would take you several months. And you don't necessarily get a nice view of the high level design principles that can inform your own software practice. So the cool thing about computer systems, I mean, maybe you think computer systems is sort of like archaic old stale discipline. But what's neat about computer systems is it's a story about software responding to technology trends over long periods of time. And a lot of the papers we're gonna read have that flavor. So this is why research is ongoing in this area is because technology changes, hardware changes. And so as hardware changes, we need to respond by building different systems. So end of moral loss scaling, does anyone, everyone understand what this means? The end of moral loss scaling, how many people feel like they know that? Okay, I'll post something on discourse. You guys should all understand this. So at a certain point in time, we were writing this exponential and things were getting faster and faster and faster. And at some point, that stopped. Do you guys remember that stopping? You guys were alive when that stopped, maybe you were little. The Pentium 4 computer that maybe you had, did anyone have a Pentium 4 other than me? That was sort of like the end of that era. Do you remember the heat sinks for the Pentium 4? Like that big, right? They're not that way anymore, right? Has anyone installed like an i7 or an i5 recently? Why is the heat sink like this little thing? It's because at some point we ran out of the ability to make transistors smaller and smaller and smaller. And so we started using concurrency to make things faster. So now your computer has 32 cores. It's a very, very different design trend and produces the need for different types of software. Flash, obviously a very new storage technology, cloud computing, battery powered devices, fast networks everywhere. Again, people are irritated now when they can't connect to things and get the airlines are the last ones that we're gonna have to come after and beat into submission so that we can get connectivity. That's pretty much it, right? I mean, that's one of the only places left on earth. So the papers we're gonna read are going to be designed, sorry, left on earth that doesn't have internet access for free. So the papers we're gonna read are responding to these technology trends. So please, on Wednesday, read that short retrospective by Randy Kass. I'm sure it'll be interesting and look over the Radar Wikipedia article and we'll talk about it right on Wednesday.