 Laser pointer, a learning guru, and I'm ready to start class. All right, so welcome back to class. It's Wednesday, last full week of class. Today, we're going to try to get through performance so that we can talk about virtualization on Fridays. I think it would be sad if we didn't do any virtualization. I might move a little quickly today. But yeah, so our goal is to talk about benchmarking and then get through a really important computer science concept, which I actually think is applicable to everyday modern life as well, which is called Amdahl's Law. How many people are familiar with Amdahl's Law? A couple of you, OK. How many people ever remember hearing Ross Perot say that if you have one penny and someone gives you another penny, then you've doubled your net worth. But you still only have two pennies. I think that's a variant of Amdahl's Law in some way. So we'll talk about maybe some colloquial ways to apply Amdahl's Law. All right, so the exam is a week from Monday. At 8 AM, here, I think it's here in this room. So stuff's on the calendar. The format will be similar to the interim. And we have an all-release last year's exam for people to look at. So last year, unfortunately, in the hackathon, we did not have a winner from this class. So that offer to release an exam question early did not play out for you guys. However, at least last year, there was some sort of have you guys got any sort of feedback form regarding this class? Oh, OK, so they haven't sort of bothered me about that yet. But last year, I think we had an agreement where I need to look at what the numbers are. But if we get a certain percentage of feedback from the class, then I will release one of the exam questions early. So I'll email out, we'll decide what that threshold is. Actually, I think last year, I didn't offer to release two of them early, but they didn't quite make it to that threshold. They did make that threshold after the exam. At that point, it's like, why bother? But so yeah, I'll send out something on Piazza about that. And then next Monday, so we did this last year, and it was kind of fun in a sort of masochistic way. But we'll do exam review next Monday at 8 AM from 8 to 10. So normal class time plus the extra hour from 8 to 9. You can think of this as like practice waking up on time for the exam. And we have a little bit to talk about. So anyway, I'll bring coffee and donuts next Monday. And we'll have fun. We'll just go over some of the material, maybe go through the last year's exam. And all this is on the Google account. OK, so questions about sort of, we didn't actually talk about benchmarking out, but questions about the performance analysis stuff that we had talked about on Monday. OK, so who remembers? I mean, what other than everything was hard about this so far? What were some of the challenges that we had encountered once we started to think about how to improve the performance of our system? What was one of them, Satish? Yeah, so measuring the system at all. We talked about just even getting timing information from the system wasn't easy. What else? What else was another challenge? Yeah, I mean, you have so many choices. You could just say anything that's involved. So yeah, picking what to measure. What is the actual yardstick that we're going to use? And then what is something else that we were concerned about? When we start talking about benchmarking, Robert, what are we really talking about in terms of system performance? We're going to measure something. That's going to mean we can measure it. We're going to measure it. We're going to measure something. But then we also have to determine what else. Tim? Yeah, kind of. I mean, what we really need to do is figure out what is the system going to be doing? What tasks are we going to give it to do? So is this going to be, are we going to measure it doing like 100-yard dash, or are we going to measure running a mile? So we need to figure out what the benchmarks are. So we had this approach of measuring the system, but how are we going to collect those measurements, that wasn't necessarily straightforward. And then what is the system going to be doing while we do that? We'll start to get into this a little bit today, but this gets to be into a part that maybe at least computer scientists, I don't know, computer engineers, statistics, no problem. But keep your scientists are kind of like, eh. And then this is probably the most important thing we're talking about today is in terms of once you've identified the slow parts of your system, or once you're able to start to profile the system, to examine the relative performance of each component, how do you decide what to work? We talked about pros and cons of models and simulations, so models allow us to make these strong guarantees about system performance. But they usually end up, in order to prove things about the system, in order to model it mathematically, we usually end up having to make a lot of really unrealistic assumptions. Simulators, on the other hand, if they're well-designed, can offer us a significant speed up, maybe. So Guru's been doing some work on an architectural simulator that does not offer any significant speed up. In fact, it's an order of three orders of magnitude, at least slower than a real system. So there, what we're getting out of it is flexibility, or we're giving up is speed, because it's way slower than hardware. But in the best case, you can build a system that abstracts way enough details that improves performance and allows you to run more repeatable experiments. But in the worst case, you end up losing some important features of the real system along the way, and that makes the results not really that useful. All right, so. And then we talked about different metrics we might use to compare different parts of the system. And in a lot of these cases, we've talked about this over and over this semester. I mean, there's a lot of interesting design decisions that get made when we design operating systems. And also, when you design hardware components, and a lot of those design decisions involve trade-offs between one thing and another thing, both of which are desirable. So who can give me an example of a case where the design of an operating system component might involve a trade-off between two desirable properties of the system? Spencer. Yeah, that's a great example. So how much memory do I allocate to the buffer cache, which is going to improve file system performance, versus how much memory do I allocate to process address spaces? So that is a nice example. What's another one? Wembley. What about in scheduling? What were some of the things we wanted from schedulers? And how were they in what's that? Types of parents that get scheduled options. Well, what were some of our goals for scheduling? Who remembers? Peng, what was one of them? What's that? Yeah, so we were concerned with throughput. Like, we really care how much work the system gets done. But what does throughput frequently come into conflict with? Andrew. Yeah, interactivity. Interactivity requires a switching back and forth between things to maintain this illusion of concurrency. But every time we do one of those switches, we're sacrificing some throughput, because we're spending the time and energy to reconfigure the system for a new threat. So that's another case. And there's several others. Most of the things we've talked about this semester are involved in some trade-off like this. Yeah, Jeremy. Correctness for holding B? Yeah, so there's another good one with modern operating systems, correctness. I would like the system to not fail. And so what was one example of that from our discussion of operating system architecture? What was an architecture example that tried to achieve correctness potentially or stability potentially at the price of speed? Yeah, Paul. What's that? OK, so Dijkstra system, yeah, for sure. But what's a more modern design alternative that tried to do this? Greg, you remember? That's not what I'm thinking of. Yeah, the microkernels, right? Microkernel's idea was if I reduce the amount that's inside the privileged kernel address space that runs with kernel privilege, this system is going to more likely be more stable. The less code I have that can crash the system, the less likely this is going to crash, hopefully. But the cost of that was I have a lot more structured communication between different parts of the system, and that causes slowness, right? So there's another example, and there's a lot of these, right? And so most of the time, when we end up measuring things, what we care about is some important complicated combination of things that are good in and of themselves, right? I mean, a lot of times when you're talking about two different disk drive designs, you're not talking about which is better, right? That question has no meaning. What's important to think about is better for what, right? What are you doing with it, and based on what you're doing with it, how well we'll perform, right? All right, and this is, you know, this class, right? So when we start talking about benchmarks, what we talk about when we talk about benchmarks are pieces of code that are designed to isolate, potentially isolate, or essentially allow two different things to be compared against each other, right? So common benchmarking suites are released, they're licensed, you can buy copies, you can run them on your system, you can compare the results, right? And they're designed to try to give you a yardstick that you could use to compare different systems, right? We can break them down in a couple of ways, right? So, micro benchmarks typically try to isolate and focus zero in on one part of the system's performance, right? One specific thing that the system might do well or poorly, right? What would be the kind of thing that I might wanna try to isolate with a micro benchmark, Sarah? Like, what's one thing about a system that I might wanna test, like one very small component of performance I might care of? Yeah, okay, so that's an example, right? You're getting, but you can even get more micro than that, right? Like, what's one operation on your system that you might care about, maybe for like assignment three, something that you're working on that you might care about measuring the performance of? Jen? Yeah, so maybe swapping page to disk. What about choosing a page to swap, right? So how long does your page replacement algorithm choose? Take to choose a page. How long does the scheduler take to run? How long does a context switch take, right? For example, if you're doing IPC and a microkernel, you might care a lot about how long it takes to deliver a message from one process to another process. So these are kind of examples of micro benchmarks, right? We're trying to zero in on one specific piece of the system. Why would I do that? Like, why does that matter? Why not just, you know? I mean, when we start talking about Omdel's Law, we're gonna realize that these micro benchmarks can be quite misleading in certain cases. But why would I care? Like, why do I write micro benchmarks in the first place, probably? Yeah, so I might use a micro benchmark to try to locate the thing that is the slowest, right? So in general, micro benchmarks allow me to take the overall performance of the system and slice and dice it into other little pieces. Yeah, Jeremy? No, the answer was headed in a good direction. Yeah, it's what you have to do. Oh, okay, yeah. Okay, well, that's okay, you can just read the slide. Sometimes that's the answer to the question. Yeah, so, but if you're the, like, let's say you're the developer, right? Let's say you're writing IPC mechanisms, right? A lot of these other, a lot of the other types of benchmarks we're gonna talk about aren't going to express to you the impact of changes that you've made, right? You're now the developer of the Linux kernel subsystem. You care a lot about how long that scheduler takes to run, but if you just run these, you know, big system-wide benchmarks, all sorts of other people's contributions and changes are gonna start to muddle in there, right? So if you really care about one thing, and if that's your job, right? If that's the part of the system that you're responsible for, then a microbenchmark can be a good way to go, right? Cause you make a change, you wanna isolate the effect of your change, right? Macrobenchmarks are kind of the opposite, right? So macrobenchmark involves, you know, trying to look at how an entire, you know, potentially an entire system or entire subsystem of our operating system works, right? So, you know, sometimes people call this sort of end-to-end performance, right? Because at the end of the day, you know, all these little microbenchmarks combined together to produce a complete picture of how well the system performs. And again, we start talking about Amdahl's Law. I mean, if there's 99 parts of your system that are really well-optimized and one part that isn't, but if that's the part that's getting run over and over again, then the system may be very, very slow, despite the fact that the microbenchmarks you run may show very good results, right? And then application benchmarks are usually geared more towards one application, right? So application benchmarks, you can almost think of as a more specific form of a macrobenchmark. They can sometimes be microbenchmarks too, if a particular application is incredibly, the performance of a particular application is incredibly dependent on one part of the operating system that might end up being a microbenchmark. But frequently, you know, an application benchmark might test a certain pattern of behavior associated with a particular application, right? So for example, database systems, you know, have these database-driven benchmarks that essentially test, you know, how good are you, Mr. Operating System, at doing what I want, right? Because databases have particular access patterns and particular sort of types of behavior that cause them to perform well or poorly on different operating systems, right? And actually, I mean, in the database community, there's been a lot of work on, you know, essentially like what to do about this meddlesome operating system, right? Like how do I get it out of the way as much as possible because there are cases in which it's, some things that the operating system is trying to do may directly interfere with database performance, right? All right, so, again, we went through some example of these, right? So, you know, handling a single page fault, maybe that's a little bit more macro, though, because maybe that starts to encompass a couple of different things, right? These are relative terms, right? You know, so macrobenchmarks, we might look at the page fault rate itself, right? Or the aggregate time to start handling a page fault once the system is under high load. And then, yeah, I mean, you can think of something like triple sort as an application benchmark, right? If you were writing an operating system that was designed to support sort for whatever reason, then you might care about being able to run lots of parallel sorts, right, on your system, right? You know, that might, you might care about, right? All right, so what are some of the weaknesses of some of the, I mean, we started to touch on them a little bit, right? But what's the problem, what's the problem with running, you know, let's say you were asked to do some performance analysis, what problem might you get into if you rely too heavily on microbenchmarks, Sam? Yeah, so if you miss, right, if you picked the wrong tree in the forest, right, then you don't necessarily get a sense for how well the system's going to work, right? So if I, yeah, if I end up running these micro, and this happens, right? So people start with assumptions. So if you do bad performance analysis, right, you frequently start with assumptions about what you think are wrong with the system, right? If you use those assumptions to choose microbenchmarks, then those microbenchmarks are not gonna help you determine what's actually wrong with the system because it turns out that there's this other thing, right, that's actually ruining performance, and it's kind of like you're totally ignoring it, right? Because you've already decided before you even ran any tests that that wasn't important because you didn't choose a microbenchmark that isolated that part, right? So to some degree, using microbenchmarks forces you to decide a little bit too soon potentially what's actually contributing to system performance, right? What about macrobenchmarks, right? So what's the danger of relying to heavily on a macrobenchmark? Yeah, not sure. Bethany, you wanna? Yeah, I mean, so this is kind of the opposite problem, right? So first of all, microbenchmarks end up merging lots and lots and lots of different parts of the system together, right? So the overall aggregate performance now starts to depend on lots of different pieces, right? And if you're trying to, if all you're trying to do is kind of figure out what's the fastest system, then you may not care about this so much. You may just wanna run some end benchmarks and you may say, well, for whatever reason, this machine is much faster than this other machine. I don't really care why, right? Maybe it just has a really, really good cash eviction algorithm, but I don't know and I don't care, right? And in certain cases, that's okay, right? But if you actually wanna do any more detailed analysis and or improve things, right? Then you start to care about what's the problem, right? Not just what relative performance is, but what is actually the problem, right? What about application benchmarks? Dan. Yeah, so I mean, there's this potential here. This isn't always a bad thing and we'll talk about this again on Friday when we talk about virtualization, but to some degree, some tuning that you can do for your system may improve behavior, may improve the performance for some applications while hurting the performance for other applications, right? And our application vendors that go so far is to say that we will only guarantee you a certain performance for this application if and only if it is the only thing running on your system, right? So you buy some sort of database server from, I don't know, some sort of database vendor and they're like, you bought this, you paid thousands of dollars for it, right? It's really specific, but if you run something, if you run your file server, if you run your web server on the same machine, who knows, like, sorry, we can't make any more commitments about performance at that point, right? And so you start to see a little bit about where hardware virtualization came from, right? And the type of the ways that people have used that to get around the problem, right? So Jeremy was hinting at this last time, right? The probably the biggest problem with any sort of performance analysis, right? And this is something that I think we're all subject to, right? It's just bias, right? So, you know, I was remembering, I mean, how many people, how many people remember the days of PowerPC Max? Do you guys ever own a PowerPC Max? Did anyone know what I'm talking about? Okay, good. So the back, you know, in the dawn of time, right? Max had this separate architecture that they used on their machines and they clung to this for a long period of time, right before they finally suddenly, inexplicably, gave it up and went to x86, which is fine. But there were always like these comparisons, you pull up, you know, the glossy computer magazines and oh, look, you know, here's, like you could be on one page and you could be looking at a graph of how, you know, Mac OS 9 on the MacBook Pro 2004 was just creaming, you know, creaming these Dell laptops at everything, right? And then you turn the page and it'd be like, Dell laptops are killing everybody else at everything, right? So it's like, you know, the benchmarks that you run, right, start to matter, right? And there's a lot of benchmarks out there and if what you care about is trying to make a claim about performance, and you know, I don't know, I don't wanna be too black helicopter here, right? But yeah, I mean, people, you know, let's put it simply, right? This is human nature and people have been known to choose benchmarks that make their system look good, right? And once you start doing that, it's very difficult to have a rational conversation with somebody because, you know, you and them are living in parallel universes, right? Like you're not, you're not, you're not talking about the same set of physical reality, right? It's like, oh, my system's faster, well, so is mine, right? So, you know, before you can actually start to have relative conversations about performance, you need to agree on benchmarks, right? You need to agree on how they run and what matters about them, right? And that's difficult to do. Yeah, and again, once you start, and then once you have communities that are choosing benchmarks, right? Then you do a lot of work to improve its performance and you're kind of like, you're falling more and more in love with that benchmark, right? You're more and more committed to that set of benchmarks, right? Because you're devoting time and energy to that relationship, like you want it to work, right? If you're trying to get it to, you're trying to improve it and over time that can be very difficult to say, well, you know, actually this benchmark suite isn't very good and I know that I've, you know, it's time to break up with it and go over here, right? And I have to do all this extra work, right? To get to know this new benchmark suite, right? All right. And the other thing that is also usually the case, right? Is that making, you can make any system faster for a particular application by simply making it less gentle, right? I mean, what you guys own and what we sell and what the computer industry supports, at least on consumer devices are these general purpose operating systems, right? That are really designed to be very flexible and run a large variety of applications, but making them faster it can be done if you're willing to sacrifice generality, right? And this already happens on the system you guys, the systems you guys are buying today, right? So what's one example, you know, can anyone give me an example of a case where we started to make a different trade off here, right? So a case where, you know, a system that's being optimized for something that in a less general way than we might be used to, yeah. Yeah, so various types of graphic optimizations, right? That start to force people to program in certain ways. Jeremy, what was? Yeah, okay, so server hardware, but even just like the general purpose hardware you guys are buying, right? What about Chrome OS? Uh-huh. Yeah, so okay, so maybe you might argue that that's just a reflection of like some change in the way that we're gonna design systems, right? Storage is dead, right? The cloud is the cloud is the future. You know, I'm thinking of things like hardware accelerators for, you know, decoding movies, right? So, or decoding MP3s, right? I mean, you can write an MP3 decoder in software, right? But you can write a really, really much more efficient, much faster, much slower power one and just implement it out of a little hardware chip, right? You know, Wi-Fi chipsets, right, are an example of this. To some degree that can't necessarily be done in software, right? But that's why it's done, right? So there's always this trade off, right? The more general the system is in general, the slower it is, right? And this is not a perfect trade off, but you can certainly usually make things faster by customizing the hardware and making, you know, building more of a system around a single application, right? All right, so let's, we talked a lot about sort of what not to do, right? So let's start talking about some suggestions for good benchmarking and performance improvement, right? So the first thing is it's usually more, it's usually better to have a more specific goal than improving the performance of some piece of code, right? So why would it be, like, why a priori, right? Why before you've done any measurement or any analysis, why is it not potentially a good idea to start with this idea of trying to make a particular piece of code faster, right? What trouble would that get you into? Yeah, what's that? Yeah, I mean, so in general, how do you know that the code, the blob of code that you've decided to work on is the problem, right? You might think it's the problem, and this is the source of a lot of headaches for people, right, it's probably the biggest mistake people make when they're doing performance improvement is they start with this idea that this must be the problem, right? So why do people start with this idea? Why is this kind of a natural result of programming behavior? Josh, okay, so that's not a bad answer, right? I mean, how many people, if, you know, I asked you to improve the performance of part of your system of your OS 161 to this, and I'm trying to think of something specific, right? Like something in assignment two, maybe the, you know, how many people have I asked you to improve the performance of some of your system calls? How many people think they would know where to start? How many people think that there's this dodgy piece of code that they wrote that's sitting there in the tree that, you know, that it's not very efficient and it's kind of broken and, you know, it was done in some sort of dumb way, right? So usually when you start programming, you have these things in your code, right? Like that one part that you were kind of tired when you wrote it, and so you just did it in some sort of stupid, you know, really lazy, slow way, right? And usually people think, oh, well, there is the part to improve, you know, there's my linear search of that big array and I need to go fix that. It may or may not be the problem, right? So, yeah, so I mean, now Josh's answer is pretty good, right? So we have instincts about what's wrong with the system, right? Our instincts are rarely correct, right? Until you get good at this, right? Actually you've done this for years and years and years, I think you've done a lot of this. I'm guessing that trusting your instincts is not a great approach, right? So we talked a little bit about this, right? So when you start choosing models and simulators, validate them before you start changing things, right? So if you build a simulator, build a simulator and match it up with the behavior of your system before you start, right? If it doesn't match when you start, then you can't really make changes, right? You have to be able to understand the baseline performance before you do things and then use these tools appropriately, right? So use model and use simulation and run real experiments and this is something where, now I can tell you about it right here but this is something that you just get better at by doing, right? And I wish that we had a performance part of one of the assignments, but we don't. Maybe next year. All right. And then you can use these different stages, right? So typically modeling is a little bit easier and faster than building the simulator. Building the simulator, hopefully, is easier and faster than actually implementing it and pushing those changes out on a real product, right? So you can use this to kind of stage your ideas, right? If you can't convince yourself analytically that the idea works, don't bother with the simulator. If it doesn't work in the simulator, don't foist it on some poor customer, right? Or some poor person out there in the real world because it probably won't work in practice. All right, so, so far we talked about how to measure things, right? We've talked a little bit about benchmarks, what they are and different types of benchmarks that you might choose, right? So now, okay, so we've figured out how to do the measurement, right? We've decided what the system is going to do while we measure it. So now what? What's the next step here? Cal, Jen. Yeah, so now, like, potentially I might actually have some results to look at, right? So now I've collected some results and now this kind of gets a little bit more fun, right? Because now I have some data and now I can try to figure out what to do, right? So collect some data, try to use it to figure out what to improve, right? And then we go back to the top, right? Well, there was a beer drinking in here somewhere but in general, like this, and then we go back and, you know, you don't always have to choose new benchmarks every time, right, but this is kind of the loop that we try to get into, right? Okay, so this is just me, right? Maybe I have a tortured relationship with statistics. Maybe you don't. How many people feel good about their relationship with statistics? All right, okay, well, that's good. At least there's a few of you. So this, I was de, like, well, that's actually not true. I started doing physics because I wasn't smart enough to do math and then I started doing computer science because I was still too dumb to be a physicist. So that's, you know, that's how I made it here. But again, I mean, some people maybe just go straight, I could have gone straight from math and computer science. I probably would have been better, right? Just cut out the middleman. So, and, you know, again, this starts to, it starts to become socks, sort of embarrassing, right? Like, you know, so this is the, yeah, I'll run my experiment a few times and I'll collect some data and the variance will be huge, but I'll just make the variance go away. So I'll just compute the average and it'll be awesome, right? Because that's what averages are for, right? To get rid of variance, right? And yeah, maybe if the variance was small, right? Small enough to not be embarrassed by it, I'll put the airbars in my graph, right? If not, I'll just forget to put them on there, right? Because, again, that's what averages are for, right? Make variance go away. Yeah, so you guys should try to do a little bit better than this, right? And there's all these great tools out there now to help you with this, right? So you don't have to, you know, there's a lot of nice programs to help you with these types of things, right? So one of the things that I find helpful when you start doing data analysis is try to predict, try to use your intuition, and I know I just said not to, right? But you're not usually to make a decision, try to use your intuition to predict the results you're gonna get, right? So before you run the experiment, before you graph the data, right? Say to yourself, what should this look like? What do I think it's going to look like? I know something about the system, you know, what would I expect it to look like, right? Then when you collect some results, you can decide, you know, you have some idea of do they match my intuition, right? So you're testing both your intuition and the real system. This is one of my favorite things, right? So, and again, people are fond of these summary statistics because the summary statistics are typically, and I shouldn't say typically, like summary statistics are designed to try to condense information about a system, right? So that you can talk about it in a slightly more rigorous way, but if they're very badly used, right? Then they can lead you in all sorts of wrong directions, right? So for example, here are two, so here is a set of, here's a variable that, imagine I collected some measurements of a variable, right? And this is what it looked like, right? This distribution has a particular median and a particular mean, okay? And maybe those are the results that you would report, you know, to your boss, right? Or the guy who's leading your product group, right? This distribution has the same median and the same mean as the first distribution, right? These distributions are very, very different, right? Yeah, and these systems are very, very different, right? So if you don't look at things like this, you're going to, so there's a story from a, you know, from a researcher I heard that there was very much about this kind of thing, right? They collected some data, right? They had aggregated it all together, you know, they were talking about it. I said, oh, it looks pretty good, but she asked her students, she said, well, why is the variant so high? What do you think he said? Anybody remember? What? Cash effects, right? Like, oh, that's cash effects, right? So she said, well, why don't you produce the histogram, right? So we can look at it together. And this is, well, this is what they found, right? The variance wasn't high because of cash effects, right? The variance was high because there were two different cases, right? There were two different things that were happening that were mixed together in the same data set, right? And as soon as they started looking at the raw data and they grafted appropriately, they started to understand this behavior, the system much better, right? This is a system where there's kind of one thing that's happening, right? And maybe there's some delay. So for example, this could be the average, well, it shouldn't be negative, right? But imagine this is positive. This could be the average time it takes your page replacement algorithm to run, right? And this indicates a certain thing, right? And it indicates that the running time is fairly predictable, right? This indicates something else. It might mean that the running time is pretty predictable, but there's two different cases here, right? Maybe, I don't know, maybe this is a case where maybe there's a bug in your out, right? This might be the consequence of that, right? And it's just a determined by, do you hit that bug or not, right? So this can be, these sorts of graphs can be really helpful to look at. So don't rush into this desire to compute these summary statistics, right? Look at raw data as much as possible, right? Okay. So outliers are, you know how outliers are our favorite thing to try to ignore, right? Because if we can throw out outliers from our data set, we can reduce variance, right? And then we will believe our summary statistics more and more. But outliers are, you know, outliers might, so outliers can indicate two things, right? In certain cases outliers are just some sort of problem with your measurement setup, right? Like, I don't know. I mean, there's all sorts of things that go wrong when you take measurements and if you see a measurement that's 10 times, let's say you're measuring a certain latency for a certain operation, you see one measurement out of your data set of 100 that's 10 times longer, right? You might say, well, you know, maybe I just messed something up, maybe. Now, if you can find some bugs in your measurement harness and I produce this, maybe you're okay. But the fact is that, you know, you have to respect all the data that you collected, right? So if you have that data point and that data point is really 10 times larger, then it's possible that that data point is real, right? It's possible that that really happened on a real system and that's something that you'll need to deal with. So there was some research that was done a few years ago on scaling commodity operating systems to large numbers of cores, right? So there's been an ongoing argument within the operating systems research community about whether traditional operating systems, sort of shared memory operating system designs scale to machines with like 48 cores, right? So can you take just sort of standard Linux and make it perform well on a machine with 48 cores, right? And what these researchers did is they took Linux and they started running on a machine with 48 cores and they found all of these performance problems and they just started fixing it, right? But a lot of them were these interesting cases where when you start adding cores, right? So if you have two cores, there are these certain very, very unlikely cases where these certain synchronization primitives could have very, very, very ugly behavior, right? That involves sort of coordination between cores. As you start adding cores to the system, those cases become more and more and more common, right? So what looks like noise in your measurement when you're on two cores, once you get to 48 cores starts to become the signal, right, that's all that happens, right? There's always some sort of collision between the two cores every time they try to access this data structure, right? So part of what they did is they just started running on a machine with a lot of cores, they started to see more of this and they started to fix those problems, right? But someone who was testing that system on a machine with only four cores would have been very tempted to just say, oh, that's probably just noise, right? Like that one time that that lock primitive took like a second to get a lock, oh, that's not gonna happen, right? That must have just made some mistake, right? Well, it turned out, again, once you got to 48 cores, that started to happen a lot, right? And that was really, really affecting the behavior of the system, right? Okay, so let's finish with talking about Andal's law, right? So once I've done these measurements, you know, I've run into a mixture of different benchmarking approaches and if I'm lucky, right, I've identified the slowest part of my system. And now, clearly, the next thing to do is to improve that slowest part. Yeah, so this isn't that simple, right? So I just like this aside, right? So again, frequently, even if there was a slow part and even if it mattered, right? Like a lot of you guys would wanna work on that ugly part that you just feel embarrassed about, right? Like that part of the code is, oh, yeah, I don't want anyone to, I'm gonna go work on that. Your boss is like, no, I want you to work on that. I'm pretty sure that's not the problem, you know? We're in this other thing, right? But as nothing points out, right? Like without some relative understanding of performance and a profile of how the system runs, it's impossible to choose which one of these functions to improve, right? And there's a number of things that go on here. So it's not just how often things execute, right? So imagine, again, I have food that takes five minutes to execute, right? Some function food, I don't know what it does. I don't know why it takes five minutes, but that's how long it takes. And then I have bar, right? Bar takes five seconds, right? And so immediately I should get to work on improving food, right? But there's two things, actually. So Intel's law focuses mainly on significance, right? How often does food run, right? How much is food contributing to the overall performance of the system? But there's a second thing here too much I think is equally important to think about, right? Which is, can you improve food? You know, like, is there slack in food that you think that you can get rid of? Is food really lazily implemented, right? Is it using a lot of really simple and long-running algorithms that can be improved, right? So these are two things that you wanna start to allow to guide you when you start choosing, actually, what to work, right? So, but when you actually start, so the thing that we know about, right, after we've run these experiments is usually significance, right? How much is food contributing to the performance of the system, right? So Amdahl's law has a mathematical statement, right? But what I like is the more colloquial formulation, right? Which is that the impact of any effort to improve system performance is constrained about the performance of the part of the system not targeted by the improvement, right? So essentially, if I pick a piece of the system to work on, even if I make its runtime vanish, the system will still take some time to run and the time it takes to run is now constrained by everything else, right? So who can give me, I mean, I just came up with one off the top of my head. Who can give me, who can come up with a colloquial way to apply this that doesn't have anything to do with computers? Well, what's an example of an application of this to our modern life? Sean, okay, yeah, no, that's interesting, right? So the effort to optimize your drive time to school by speeding, I'm sure nobody here does because it's against the law, right? That is limited by the time you spend, spend a stoplight. That's a good one. Anyone else want to give a try, Robert? Hybrid cars? So, yeah, okay, so how you, right, so how you operate things matters, right? If I'm flooring my hybrid Prius out of the lights, probably not gonna help. You know, I don't know, maybe it's just my political tendencies, but you know, the effort to reduce the United States budget is constrained by the fact that like 90% of it, or 95, something like 90% of it is stuff that you can't actually get rid of, right? So you can have arguments about, you know, reducing spending, but they're, you know, unless you're really gonna carve into things that people don't like to give up, you're not really gonna get it, right? You can reduce discretionary spending by 50%, right? And you've reduced the federal budget by 5%, right? So anyway, so let's go back to computer science, right? So imagine that, you know, I have, now imagine that I've done, you know, somehow I knew, somehow I knew that I could do this, right? So I have the option of reducing the execution time of foo by 80%, right, from five minutes to one minute, and reducing the execution of bar from five seconds to four seconds, right? This is a 80% improvement, this is a 20% improvement, does anyone, can anyone predict the speed up that this will have on the overall system? Anybody? No, nobody can, right? And, you know, well, okay, no, I can, right? I'm gonna say foo is better, right? It's, come on, man, I mean, like the, so just let me warn you, right? You're gonna run into people in the world who think this way, right? Well-educated people, right? With degrees from like prestigious institutions, many of them will be degrees in business administration, unfortunately, right? Sorry, that was a low blow, right? But yeah, your boss is gonna be like, come on, man, like, just come on, just go work on foo, you know? 80%, you know, like that's something I can make a PowerPoint slide out of that number, right? You give me a 20% improvement, I don't know, you know, I might need some fancy animations to sell that. But yeah, look, like, this is gonna be in the glossy magazines, right? Like your bar just went like, wow, cool. Unless the program spends over 95% of its time running bar, right? And only 0.1% of its time running foo, right? So you can do this math if you want to, we don't have to go through into detail, but, you know, the speed up that you achieve by optimizing bar turns out at that point to actually be better, right? And these are pretty huge differences, right? Like this is a function that's taking, you know, 60 times longer to run, right? And the improvement is four times larger, right? But still, if bar is my hot path, right? If bar is the thing that the system spends all of its time doing, then bar is what I work on, right? So if you go, you know, do server, I think I mentioned this before, right? If you go do server performance optimization, right? They actually spend time, you know, looking at assembly code that's run by the system frequently. And if they can find like one place where they can like reverse a few instructions, you know, maybe I get something into a delay slot and it's like sweet, you know? I just sped up windows by like 1%, right? With a single instruction, that's an exaggeration problem. But, yeah, so you start to look at things that, yeah, yeah, look at this, who wrote these slides? So, yeah, so, and this is another sort of, you know, colloquial, even more colloquial way, maybe this is a way that you can use to talk to your future bosses about Amdell's Law, right? It's like, look, this thing's hurting us, right? Like this little piece of code, I know it doesn't look significant. I know, you know, you don't think it's that important. I know the improvement that I can offer is maybe now this big, but it's killing us, right? That's the thing that's actually hurting us. And then here's the other, here's I think the other sort of, I don't know if it's unfortunate, but it's kind of an interesting corollary to Amdell's Law, a result of Amdell's Law, right? So, okay, let's say you're good, you run your benchmarks, you've done your statistics, you've found the right part of the system, optimize, you start working on it. As you improve that part of the system, what happens to the overall system performance? How do I have to reapply Amdell's Law? What's sort of the continuous version of Amdell's Law? Nothing. And so what's, yeah, so essentially, the more, you know, what Amdell's Law will say is that the more you improve one part of the system, the less likely it is that you're still working on the right part of the system, right? And of course, this is frustrating for people, right? Because they spend a lot of time working on this, they think, oh, I can get 10% more if you just give me another week or two weeks or a month or a year. But the problem is that they're not working on the bottleneck anymore. You know, this is the result of success, right? Like you've done your job, like that part of the system has now, and it's kind of like whack a mole, right? Like you whack that mole, right? Like it's down, that there's other moles popping up all over the place, right? So you've got to go after those guys. So yeah, so I think this is the other thing to keep in mind, and it just means that, you know, you have to stay agile, right? So as you're working on systems, as you're optimizing performance, you have to keep in mind that you have to repeat, right? So you have to continuously go back to the top of your loop, right? You've done some work, you've optimized a certain part of the system, you've replaced some sort of, you know, inefficient algorithm with a more sophisticated one. Now you have to reassess, right? Because now it's going to turn out that some other part of the system is actually needs your attention, right? All right, so on Friday we will talk about, we will do a lightning tour of virtualization for fun, because virtualization is cool, and I will see you on Friday.