 OK, the song is not going to play today. That makes me a little sad, but that's all right. Oh, there's a new table here. Change something in the matrix. OK, so today, first of all, congrats on those that are done with assignment three. I had a couple of new names up here recently. So a couple more days. Actually, you can keep submitting on the leaderboards as long as you want. Leaderboards don't know anything about the deadlines. So yeah, hopefully, we'll get a few more people up here over the next couple of days. And of course, I still need to click outside of here to get past this. OK, today we're going to talk about Amdahl's Law. We're going to finish talking about performance. And then at the end of class, I'll start talking about the paper that you guys are going to look at for Monday. So as a reminder, there is no class on Friday. We will have all-day mega office hours going on in and around Davis Hall for finishing up last push on assignment three. And then Monday, Carl will talk about the rest of the paper that we introduced at the end of class. So you guys haven't read it yet. I just sent it out today. But please look at it over the weekend. Hopefully, I can encourage you, inspire you today, to actually look at the paper. Because this is a fun paper to read. The lessons are timeless. Some of the examples have gotten a little bit crufty, the paper was written a long time ago. But there's a lot of good advice in this paper, some of which we've talked about throughout the semester. But Lambson does a nice job of organizing it all together and presenting it well. OK, any questions on Carl's? Well, Carl talked about on Monday, sort of the start of our brief discussion of performance in Benchmark. Carl was excited about sharing some of his own experiences with performance testing with you. So hopefully, you got something out of that. OK, so let's talk about statistics. So once you've actually picked a benchmark to run and figured out how to measure things about the system that you care about, at this point, what you actually have to start doing is analyzing the results. And this is a place where computer scientists can get into trouble. Can I talk? Thanks. Yeah, there's only one right answer to that question. So black results start competing some statistics. And I mean, how many people in this room enjoy math? Really? Why? OK, anyway. That's not the response I was expecting. Yeah, hands down up here. How about how many people enjoy statistics? OK, still something wrong with you guys. No wonder you're still coming to class, demented people. Maybe you guys are coming because you're waiting for me to talk about math and statistics. Well, today is the day, you guys. You don't have to come to class anymore. Anyway, I'm definitely in computer science because I was too dumb to do other things I wanted to do, which included at one point physics and then math. Well, I guess math was first, maybe, and physics was second, but whatever. That's why I became a computer scientist because I was better at it. So I don't particularly like math and statistics, but these are really useful tools like other things that we do when we build systems. So we have to learn how to use them. On the other hand, our approach to these things is not necessarily always ideal. So averages, for example, like if I want to summarize a data set, hey, I'll run the experiment a few times. Well, first of all, I have to have a data set. So that's kind of the first thing. Actually, on a bad day, I just run the experiment once. I get one number, move on, because clearly that number is exactly right. On a good day, I might run the experiment a few times, see a little bit of performance variation, and try to summarize that data set using some sort of summary statistic, like an average, which is a fantastic statistic. About 10% of the time, and the other 9% of the time is a miserable, terrible statistic. And if I'm feeling really frisky that day, I've got some extra time, something good happened to me in the morning, I drank a couple of extra espressos or something, I might actually put some air bars on the graph so people have some idea about the variance in the underlying data set. So we're going to talk today about some of the right things to do, right ways to approach this process, particularly once again. So we're at the point where we know how to measure things about the system, and we know the things that the system is going to do while we're measuring it. So we figured out how to measure things, given the limitations that Carl talked about Monday, and we have some sense of what this workload should be, what are the benchmarks we're going to run that produce behavior that we're interested in studying. So this is something I always tell my PhD students, and I think this is really an important part of this process that people don't always do. The first thing before you start running experiments is make predictions about what's going to happen. In the scientific method, people called this a hypothesis. I know we're computer scientists, right? What do they say about science? Any science that has science in the name is not an actual science, right? Political science, the social sciences in general. This is true, and I didn't say this, don't blame me, but you're a computer scientist, right? So you feel the need to emphasize to people that you do science. What that means is that you don't formulate hypotheses before you run experiments. So why don't we try doing that? And actually, this is really an easy thing to do. So before you produce a graph, a form of a hypothesis, it doesn't have to be like third grade science. You don't have to write hypothesis, colon, blah, blah, blah, blah, blah. Draw on a whiteboard a picture of the graph that you think is going to result from this particular experiment. And that's your hypothesis now. Then you collect results, graph the data, and if the graph has a shape and features that match your original hypothesis, then something about your understanding of how the system behaves has been validated. This is always a good thing to do because this is a way to help build your intuition. You know, a lot of, particularly when you build computer systems, computer systems takes a lot, you know, building computer systems, if you guys have discovered, takes a fair amount of time and energy. And so something that you build up over time is an intuition about what things are going to work, because we can't always try stuff. Trying things when you build big computer systems sometimes takes years. If that thing doesn't work out, that's time wasted. And so having some intuition about what's going to happen, how systems are going to behave, and what are general good design ideas is something that comes with time, but is worth trying to build up. So make a prediction. I just said you can compare graphs, you know, compare your sketches of what you thought the graph would look like with the actual graphs themselves. Has anyone actually ever done this at college? Have you taken a class where you did experiments? Just curious. Okay, are you guys like, are these computer engineers over here? Is this like them when they make you to chemistry or something like that? Yeah, there we go, okay. Yeah, like other fields do this. It's not a weird thing. I run a chem experiment. Once you start doing what we're talking about, benchmarking and experimenting with systems, this is important. And it's not something that we necessarily always talk about enough in the context of computer science. Okay. And then when Carl talked on Monday about various tools that you can use to experiment with systems, predictions are also good ways to validate models and simulators. So if the simulator doesn't match up with your intuition and produce results, at least along the lines that you predicted, then there may be a bug in that system that you need to fix. Okay. Understanding your data. Once you have data, working with that data, understanding that data, you know, Oliver in our department works a lot on data cleaning, thinking about how to take the data and massage it into the format that you want and then mine it for information. One of the biggest mistakes people make is that they, in my opinion, is that people are drawn to computing these sort of summary statistics too soon. Why do, so for example, when I talk about summary statistics, I'm talking about things like means, medians, standard deviations, things like that. These are single numbers that are designed to summarize an entire set of data. How many people have computed one of these things before in the last few years? Okay. How many people did that when it didn't involve like a grade in a class? Okay. There's a few hands that didn't go up, yeah. So, you know, there's deep explanatory power in some of these statistics when they're used properly. They also can be really misleading if they're used improperly. And there are assumptions about the underlying data sets that are implied in the ability of some of these statistics to actually effectively summarize the data that you're trying to use. Why do people tend to compute these so quickly? They give you a data set, you know. First thing you do is compute. If I give you a data set, a lot of things, a lot of you, the first thing you would do, or if I said compute one interesting number about this data set, you would probably just compute an average, right? Why? Yeah. It's a good representation of the data. It's a good representation of the data. It's a representation of the data. It's easy. It's easy. It's actually pretty easy. You know, there's a very easy mathematical definition of average that I can compute. You know, even if Excel makes me vomit in my mouth a little bit, it will probably help me to do this if I poke it in exactly the right way. And it also avoids doing things like actually graphing data sets. That's complicated. I don't want to do that. But some of these things, looking at full data sets before you jump to summary statistics is actually really important. So let me give you the canonical example of this. These two experiments, these two data sets produce the same mean and median. So if I computed the median and the mean of these data sets, common summary statistics, I get the same result. Now clearly, there are different things going on with these two data sets. You know, it clearly, they look different. One's a one-hump camel. The other's a two-hump camel. But other than that, what is, like, if you were a computer systems person and you saw a data set like this, let's say that this is the page fault handling time for your VM. Like, what's the difference between these two? Or what's a difference? What's one direction that that second data set that has two clusters in it starts the lead doing? Yeah. Yeah, there's something else going, there's two different things going on here. You know, maybe this is one of the code paths in your VM fault handling. And this is the other code path. Maybe you didn't know that there were two code paths. So this is useful information. Maybe you thought there were three code paths and it turns out there's actually only two. Because maybe there's some dead code in there that never gets run because one of the corner cases you thought to handle never happens. Or didn't happen when you ran this particular benchmark. So, trust me, I mean, there is nothing that's gonna make you feel dumber than, you know, going to somebody like your boss in the future and presenting them with these high level summary statistics and then having to go back a few days later and be like, actually whoops by the way, the data actually looks like this and it turns out that there's this completely unexpected thing going on in the data that I didn't realize. Does it make sense? The graphing data, I mean, don't ever compute summary statistics on reasonable sized data sets before you look at the data in some way. Histograms like this are a nice way of looking at larger data sets. If you have small data sets, just plotting them on a scatter plot or just looking at them. What's another, I thought at some point when I was thinking about new classes to teach here, I thought it would be fun to teach about it. You could teach a whole class on misleading statistics. What's an example? Anyone have a favorite? Yeah, what's that? Starting in access. Oh, in axes. Oh yeah, you see that all the time, right? You ever seen graphs where it's like, it looks like there's this huge change in a variable and then you squint down at the access and it turns out axes. Axes? How do I say that? Axes? You really messed with me. I think I'm saying it wrong, right? You look down at the scale. Let's put it down, I can avoid the word. And yeah, the scale starts at like 5 million, right? Anyone else? But my favorite statistics people talk about, like highly misleading statistics. What else? Poor statistical practice. Oh, come on. I mean, how many people in here follow sports? Like there's a fair amount of numbers in sports these days, right? And of course, some of the movement in sports like baseball is away from these statistics that are really misleading on certain levels. A batting average, for example. Yeah. Well, Tom Brady wins a lot of games. So that's just, okay, okay, yeah, yeah. Well, so if you don't play baseball, for example, something like a batting average has been considered to be a poor statistic because it doesn't include times where the runner got on base without actually being given a chance to hit the ball. This is where the stuff gets weird. My favorite is average income in the United States. Yeah, look at the average income. Who cares with the average income? There's some people over there making billions of dollars that are dragging the average way over in that direction. I'm serious, that people talk about, oh, you know, the average income in the United States went up. All that means is that Bill Gates made 10 times as much more money, right? Like Bill Gates affects the average way more than I do. What's a better statistic for a data set like that? Median. Median, oh, the median is an underappreciated statistic. You know, someone said before, said the average is a good way to describe a data set. Almost never true, actually. Average is a good way to describe data sets that have a particular distribution to them. A data set, what's so nice about a median? If I told you the average of a data set, what do you really know about the data set? Very little. All I know is that if I add it up and divide by the number of members in it, I have this. If I tell you the median, what do you know about that data set? Half are bigger, half are smaller. So if I give you the median income, in this country, you know that half the people in the country make less, half the people in the country make more. That is a useful thing to know about a data set. Not the only thing you might want to know. Obviously, again, both these data sets have the same median and mean. So the median is not a, you know, using medians is not a magic, you know, thing that's gonna save us from looking at the underlying data itself. Okay. Outliers. What's an outlier? Yeah, so a weird data point. It's like most of my data points are over here and then there's one data point that's 10 times larger, 10 times smaller. What do people like to do with outliers? Yeah, ignore them, right? Whoops! Something must have gone wrong, right? Something must have gone wrong. That just happens sometimes, right? Like sometimes, you know, the sun doesn't rise at the time that you think it's gonna rise. It's like an outlier dying. Just ignore it, you know, just go on with your life. Yeah, so outliers are tricky. I mean, it is possible. I'm not saying that you have to, at all costs, preserve outliers in your data set, but it is possible that outliers are real. They could just be some sort of noise in the system. You know, you ran an experiment at a time where somebody else was hogging the machine and was consuming all of the system resources, right? I would argue that's not a particularly well-designed experimental environment, but that can happen. But outliers sometimes can represent behavior that you really, really need to understand. So some sort of corner case in your code, some sort of very, very poor interaction between different system components. And here's the thing. So I think you guys have probably figured this out by now. By some magic of the configuration of the universe, test 161, when we run it on the server, exposes lots more bugs than it does when you run it locally, right? And this is the case of a general sort of law of the universe that says, whatever the worst thing your software can do, it'll do it when you give it to the client and say that it works, right? That's the point at which it'll start to act up. And there's actually some reasonable reasons for that. They're gonna run it with a different configuration. They're gonna provide more load. They're gonna use it in some way that you didn't understand or expect. But it's also just sort of one of these laws of the universe. So if you see something like this when you're testing a system, this is also sort of like those race conditions that you guys run into when you're testing your kernels. I've been helping people in office hours and they'll hit a race condition like this and they'll start immediately shut down the system and start, I'm like, you're never gonna see that bug again. It's gone. They're only gonna hit that bug once every million times. Of course, again, when you run it for a gradient, we're gonna hit it once out of one times. So, but don't stop. Like you have valuable information. Like this is a really, really critical moment. Just leave it alone. Take a deep breath, start debugging. Don't stop and hope it doesn't happen again. It won't, right? Again, until we get it. Okay, so understanding outliers and data is really important. Okay, any questions before we go on? That's sort of a data analysis. All right, so now we're gonna, I mean, in terms of like material in this class that has significant deep life implications. That people tend to ignore. That, you know, if there's advice that I can give you to take out in the world that's gonna improve your life, not just in the computer programs that you write, this is probably the best stuff coming up here in the next 10 slides. Okay, Omdol's Law. How many people have heard of Omdol's Law before? Okay, good, we're gonna talk about it again because we should talk about it more. So, once you've measured things, once you understand data, remember that we're talking about performance improvement here, making things go fast. Deciding on what to improve is the next step. What part of my code am I going to try to rewrite in order to make it faster? Now, even if I thought I should work on the slowest part of the code, which is not true, this is really hard to do. And again, I think by now you guys have been developing on OS 161 all semester. You may have started to see some of these patterns emerge in your own behavior. So, for example, how many people here have just known that a certain part of their code has a bug in it? Like, if there's a problem with your code, you just kind of know where it has to be, you know? It's in that janky thing I wrote at three in the morning that's got I, J, K, and L as indices. I did see this in someone's code, by the way. They know who they are, four level page tables. Yeah, like the part that has no comments for 300 lines of code, the part where I decided to stop using the syntax I use everywhere else, like the part where I didn't put brace statements around my bare if statements because I thought, why bother, right? I've only got a certain number of keystrokes. I've said that, so I'm gonna just stop using brace statements, braces. Yeah, like we have this intuition, and this happened to me all the time when there's bugs in my code. I'm like, oh, I know where that bug is. You start debugging, you know, and it turns out, you're wrong. Even if you, in non-times, you have data in front of you that should have led you to some other conclusion. But the problem is you were, you're nervous about this one piece of code because you don't understand it fully or you forgot what it does or your partner wrote it. And you're like, eh, it has to be the problem. And so you start looking there and sometimes you rewrite it and sometimes you spend time, you end up trying to, you end up wasting time cleaning things up and stuff like that. In the meantime, the actual problem you're trying to solve is somewhere else. You know, and so programmers frequently approach performance evaluation and performance improvement in the same way. They say, okay, I'm gonna work on this one part of code because I know it's the slow part. Now that's where I did a linear search through this array. And I mean, that's gonna have to be the part that's slow. Okay, but let's come back to talking about Amdahl's Law. So here's an example. I've got two functions in my code. One is called foo, which takes five seconds to execute. Sorry, five minutes to execute. And the other is called bar, which takes five seconds to execute. So which function is slower? Now that's your question. So foo is slower. Foo takes 60 times longer to run than bar. So clearly I should start working on foo right away, right? Okay. Somebody has trained you guys a little bit better than in the past, so that's good. The progress is happening in the world. So what two things do we need to think about here? Yeah? How many times do I have to execute? Yes, so the first one, and this is one I think you guys have probably already told them for, how much does foo matter? How many times does foo being executed? Maybe foo is some sort of recovery logic in your file system that only gets run when the file system crashes. Now I'm not saying that that's not an important time. We talked about journaling and why people do that now to speed the process of propping up file systems that have crashed. However, if we're trying to improve the common case, then code that only runs in special circumstances is totally irrelevant. There's something else to think about here. This we can measure. This I can run tests and I can run benchmarks, I can figure out how much is foo contributing to the performance of the system. There's something else though here that is a little bit more intuition driven and that is how much slack is there in foo? How much can I really get out of this? If I worked on foo for an hour, how fat much faster am I really going to be able to get it to go? And this is not something that your benchmark is going to be able to tell you. This is something that you're going to have to use other information to figure out. You might have talked to other people on your team and say, hey, has anyone ever tried optimizing this? And when everybody on the team is like, yeah, I did it, yeah, I did it, yeah, I did it, yeah, I did it last week. Then you're like, okay, there's no slack there, right? Unless they're really terrible programmers, in which case I would find another job first before you optimize this code anymore. So how hard is it going to be to do this? You also look at the code. I mean, if foo is like two lines of code, that are calling into some library that's provided by somebody else that you don't have control over, then good luck, right? I mean, that may just be nothing that you can do. So, but because we're talking about performance improvement and how to use benchmarking and analysis, we're going to talk about significance. But again, difficulty matters. So here's the statement of Amdol's Law and we'll look at several statements of this. This is sort of the most mathematical, which is that the impact of any effort to improve system performance is constrained by the parts of the system, not targeted by the improvement. It's kind of weird, right? This isn't my favorite formulation of this, so I have a couple more. What this means is that if the rest of the system, remember, when you work on improving the performance of something, you're working on part of a system usually, the rest of the system is staying the same. That's what this formulation of the law is saying. I can only get the system to go as much faster as the contribution of the part I'm working at because everything else is unchanged. So going back to our example, if I can reduce the execution time of foo from five minutes to one minute, or reduce the execution time of bar from five seconds to four seconds, what should I do? This is more obvious, right? What's that? Right, we're still stuck here. We still don't know what to do, right? And this is where the human psychology starts to play a role, okay? Because you're gonna feel better about yourself at night if you work on foo, right? You're gonna be like, honey, I made a piece of code today five times faster. Now, hopefully the person that you care about doesn't care about that sort of thing. They're like, whatever, let's have a beer. But the improvement to submit, it's like, oh yeah, I spent all day working on this stupid piece of code and I only got one second faster. That's just not as glorious, right? Unfortunately, it still doesn't matter because if, as Stephen pointed out, if the program doesn't ever call foo, then all of your glorious victory has been for naught, right? I remember once at Microsoft, like reading this totally gnarly, nasty piece of code that had to do with weird page table structures for really large memory architectures and then starting to scratch my head and wondering like, I just felt like I was on a wrong path. And so what I did is I ran the pre-processor over the Windows kernel code that I was looking at and that file vanished, right? So for the types of machines that I was working on, that file didn't even get compiled in, right? So that was great. I'm glad I wasted those few hours of my life. I remember that experience. I remember nothing else about the code. So again, I mean, in this case, and because in this case, the improvement to foo is so dramatic, you really have to push these numbers pretty hard to get this to happen, but this happens, okay? When I worked at Microsoft, I worked on a desktop performance group the first time I worked there. We had people in the same unit that worked on server performance and the server performance people spend a lot of time thinking about these hot paths. So if the server performance people can find one instruction that they can get rid of on something like the page fault path, the system call handling path, they all get several months off. They go on vacation. I'm not kidding because these code paths get hit so often and usually they're pretty tight already. People spend a lot of time working on it. Okay, so this is my more colloquial formulation of Andals law, which is ignore stuff that looks bad. There may be parts of your code that you don't like. There may be algorithms that you don't like. There may be data structures that you don't like that you think are embarrassing and bad and they may be, but no one cares. What they care about is whether or not stuff works and goes fast and so if you fix the parts that are gonna cause things to go fast, the thing that's doing the damage, the parts of your code that are actually causing the entire program to feel slow, that's what's going to eventually make the most difference. And again, this is something that you can apply to your life. Most of us spend a lot of time optimizing things that don't really need to get optimized. So fix the thing that's causing a problem. All right, and then this is, so this is another corollary to Andals law, which I think is really interesting that's also important, which is that performance improvements to a particular part of a system have to be part of a process because the longer you improve a particular part of the system, remember Andals law, the rest of the system's not changing. So even if you're working on 20% of the problem, you can only make the system 20% better. At some point, you have to loop back, start over, run your benchmarks again, analyze your data again, and pick the next thing to do. I've seen people get really wedged into improving one particular thing about a system, and they're totally missing the bigger picture. It's like that thing stopped mattering a long time ago. You won, you made it better. Now stop, go back and pick something else to work on. And of course again, there are sort of deep psychological reasons that people do this. I understand this piece of code really, really well. And I know everything I wanna do, and I've got this whole checklist of improvements I'm gonna make, and it turns out that I actually already fixed all the things that were a problem, but think about all the new cool things I can do to this, and just don't do that. At some point, stop, take a break, go back to square one, start running experiments again, and pick the next thing to work on. All right, any questions about this? Okay, so what we're gonna start talking about now, and Carl will pick this up on Monday. So this is one of these classic papers. This paper is old, fun, venerable, written by a very famous computer scientist. So this paper contains a variety of hints, as he puts it, and Bollin-Labsend is very open about the fact that these are suggestions. Some of them are self-contradictory. There is no internal consistency. Some of them tell you to do one thing, and other ones tell you to do kind of the opposite of that other thing. And taken as a whole, these express a lot of the different tensions that exist in improving the performance of systems. One thing that Bollin-Labsend is very correct about is that systems are more difficult to both understand and improve than algorithms. Systems contain algorithms, they use algorithms, but systems themselves, the design decisions that you guys make when you figure out how to factor your VM system, that's not exactly an algorithm. I mean it is kind of, I mean at some level, if you really want to go there, everything's an algorithm. But systems as a whole are more complicated to think about, analyze, and improve than algorithms. And one of the things about Bollin-Labsend does early in the paper is he talks about why this is the case. So for example, the external interface is less precisely defined. This is, particularly when you're building systems for actual people to use. An algorithm frequently has a very, very specific thing it's supposed to do, sort an array of numbers. Take two sets and figure out what members are common to both of them, things like this. Systems, the external interface, particularly if it's a human interface is subject to change, and less precisely defined. So this is one of the challenges. Systems have a lot more internal structure to them than algorithms do. Systems have subsystems that talk to other subsystems. I mean you think about something like Twitter. I mean Twitter probably consists of dozens of different components that are talking to each other. Rather than one single piece of code or single sort of piece of pseudocode or single approach. How to get things to work properly is frequently less clear. Now for certain types of systems, there are certain constraints. For example, if Twitter didn't deliver tweets for several days, that would be a problem. So certainly Twitter and other types of systems have constraints about things that they have to do. But the overall design success of some of these systems is a little bit less clear. Okay. So, Butler Lampson wrote this paper because he had accrued wisdom from working on a variety of different types of systems. And this is probably the, in my opinion, the weakest part of the paper at this point in time. Because 40 years have gone by or 30 years have gone by. And some of the examples from this paper are sort of great computer history lessons. And so actually, hopefully he won't bother me, he won't mind me teasing him about this, but this is a great paper for Karl to talk about right now on Monday. Because Karl may have actually used some of these systems. So, Butler Lampson doesn't claim that these are original hints. These are things that a lot of system designers apply sort of naturally. His goal is just to collect them together and organize them. Well, I can't ask you that because I haven't read the paper. So, Butler Lampson organizes the hints into three categories. So one is, and these are three things that you think about when you build computer systems. One is, does it work? Does it do the thing it was supposed to do? This is probably the most important feature of any computer system. No one really cares how fast an operating system is that doesn't run user programs. No one would care about the performance of Twitter if you couldn't send tweets. That wouldn't be interesting. So until the system works, the other two things are sort of irrelevant. So, getting systems to work. Once systems start to work, then speed becomes interesting. And you can substitute speed for resource consumption today. At the time this paper was written, cloud computing was not a big thing. So now, big cloud computing providers, big companies that build big systems think not only about how well they work and how well they perform but how few machines they can be run on. Because that affects your bottom line. Fewer machines I can run things on. Even if, you know, so for example, if you went to Twitter and you said, I have an approach to improving your computer system that will allow you to use half as many machines but will only increase the delay of a tweet by half a second, they would probably do it. I shouldn't put money on that. But, you know, all these companies are making trade-offs between performance and the amount of machines they have to rent, amount of rack space they have to use, stuff like that, capital costs associated with maintaining these big data centers. So fault tolerance. Final thing he talks about is, does it keep working? How does it recover from failures? What are the failure models? How are things allowed to fail? Okay. And then in each part of these tasks, there are three, sorry, in each part of these goals, there are three parts of the process. So part one is ensuring completeness. So for example, with functionality, making sure that the system does everything that it's supposed to. Choosing interfaces. So this is a task that affects all three of these. So how fast does it go sometimes has to do with what are the interfaces that you've chosen to build between various parts of the system and actually designing the implementations themselves. So, you know, making sure that you've done everything you need to do, figuring out how different parts of the system are going to talk to each other and then figuring out how to implement the parts of the system that you now need. I mean, how many people had to design an internal interface when you did assignment three? I hope that you guys don't know what I asked you. So how many people wrote their entire VM system as inside VM fault? Just one big function, okay, Gus is laughing. So at least one person. Yeah, I mean, I hope like you guys wrote some little helper functions. Did anyone write a helper function? Okay, there we go. So fine, you used an interface. Maybe you didn't think about it that way. Few, okay, good, I feel better. Yeah, I mean, you may not have thought about this way and unfortunately you may not have designed it very carefully, but once you start to refactor your code and write helper functions, essentially what you're doing is creating little internal interfaces that you're using. And it's usually better to think about those interfaces beforehand, spec them out, talk about them, make sure that everybody on your team understands how they're supposed to work, but sometimes they just sort of emerge naturally. That's okay too. All right, so here is a summary. This is a figure that's actually drawn from the paper of all of the hints. Let's see here. And some of them end up in multiple categories. So this end to end principle is connected through several different places. Let's see here. It's a good idea again, divide and conquer, cash answers. So Carl is gonna pick up with these on Monday. Does anyone have any questions about these before we stop for the day? I know it's a little bit of a short lecture. So again, this paper is rich with useful hints about how to build computer systems. Please look at it over the weekend. Of course, after you finish assignment 3.3. We'll have office hours all day Friday. Good luck wrapping up assignment three.