 All right, Friday morning. Good morning, everybody. Good morning. Good morning. Is everybody excited about the weekend? No. Why? Project. Project. Look, so sleep, sleep is good over the weekend. So, I mean, we could have had the, there are other due dates that are available for these assignments, right? Let me throw out some due dates that you guys might like less. For example, Friday at midnight, right? That's a due date that I've actually experienced in the past, right? Monday at midnight, right? Equally fun. Equally capable of destroying an entire weekend. So, okay, Wednesday. So, I meant one is due on Wednesday. Okay, so today we're going to kind of do a fun lecture. It has some cool content to it. Partly, this was motivated by the desire to introduce you guys to a real schedule. So, we talked on Wednesday about some sort of these basic sort of toy scheduling algorithms that we talk about in operating system classes just to introduce you to the idea of scheduling. But I just don't, I don't think that's super, you know, let's talk about how it really works. Let's talk about how it works on the big stage. And this is kind of a fun story, right? So, there's some personalities here, and there's some interesting tensions, and we'll talk a little bit about Linux development and how these things play out in the context of the Linux community, which itself is a pretty, pretty cool place, okay? So, and this is going to stop working right now. Firefox wants to install an update. No, thank you. Let's do that later. But that's good, so I've got to turn this on. Awesome. So, at this point, if you haven't started the synchronization problems for assignment one, then you're behind, okay? So, how many people have started the synchronization problems for assignment one? Okay? The rest of you guys are behind, pretty seriously behind at this point, okay? And I don't want to, I can't emphasize this enough, okay? You guys need to start the assignment, right? You know, I am not planning on answering a lot of email over the weekend. I don't expect the TAs to either. You guys have had two weeks to do this assignment. There's really no reason to leave it to the last minute, okay? So, you know, if you guys, you guys have the rest of today. You're up bright and early. It's 9 a.m. I'll be done with you in an hour and get working on this, okay? There's office hours right after class. Anodipa is holding them in the usual place. I'll be in my office, but I have other things to do. I'm happy to answer some questions, though. So, look, I mean, you know, I've spaced these out as best I can, but I can't force you to start them, all right? So, you know, this is how it is. And we're not going to make special accommodations next week for people that have decided to wait till the last minute, okay? So, people didn't have their hands up before, please start the assignment, all right? The earlier you start it, the more likely that you're not going to have a train wreck at the end, right? And this is not a hard assignment, right? This is not designed to be a hard assignment. The assignment that you're going to do next is a hard assignment, and the assignment that you're going to do after that is a pretty hard assignment, right? So, you guys need to get your butts in gear, all right? I think I heard some people talking about the assignment before class, which is great, right? Any conversation I hear in this room about the assignment is awesome, right? It's in English, it's cool, it's kosher, and as I told you before, please do help each other with the assignments, because that's important, all right? Any questions about this sort of stuff, logistics, et cetera? All right. Questions about scheduling algorithms that we covered on Wednesday. So, we talked about a couple of simple scheduling algorithms. We talked about different pieces of information that we might want to incorporate when we do scheduling. Any questions about that material before we talk about a more, somewhat more complicated? Karlie, you're looking at me like you might have a question. Pensive. Pensive. Oh, great. Okay, good. I like pensiveness. Pensiveness. Energy. Some kind of energy. Okay. So, yeah. Sorry. So, if there is a process with one thing and another one with multiple things, what's the difference between them? Right. So, that's a good question, and there is, you know, there is some sense in which you would assume that the operating system would want to, let's put it this way, we don't want to give a process the incentive to create a lot of threads, right? We want processes to create as many threads as they need to effectively do what they need to do, okay? And scheduling algorithms are built to try to make sure that, you know, there's not an incentive to create huge numbers of threads, right? Does that answer your question? No. Can you repeat your question? The ones that we talked about, no, right? They don't do that, right? But we talked about very, very, very simply schedule that, because what I'm saying is that you can imagine that if you design a scheduling algorithm for a real system, that is one thing that you might want to take into account, right? So, again, so you don't create the incentive for, you know, a process that wants to grab a lot of CPU time to create a billion threads, right? Now, of course, creating a billion threads creates other problems for the process because it has a lot of state that needs to synchronize, right? So, there's trade-offs here, right? Good question. Any other questions about scheduling, the simple scheduling algorithm? All right. So, what kind of information, information, I don't know what that is, what kind of information could schedulers use to choose what thread runs next? What do we talk about on Wednesday? There's a variety of different things, right? Anybody have an idea? What's that? So, what sort of information, does Round Robin use any additional information? Not really, right? So, one answer to this question, in a way it's kind of a trick question, is none, right? And we talked about some schedulers that don't use any information about the threads, right? That just look at the run queue and do something simple, right? We'll review those in a second. What else? But what other kinds of information do I want to incorporate? What the thread's about to do, right? If, per chance, I could know that ahead of time, which I probably can't, but that would be nice information. What else? What sort of other information would I want to use about? Priority. Priority? So, I might incorporate some information from the system or the user. I might give the user the system a way to influence my schedule and decisions by saying, this process is more important than this process to me, right? It's entirely relative, Ben. What the thread has done? What the thread just did, right? So, something about the thread's history might be useful to figure out how to schedule it in the future, okay? So, this is good, right? What will happen next? So, the irracular scheduler. What just happened? Typical schedulers that look at past behavior. And then, what does the user want, right? What did the user tell me to do? What did the user tell me was important, okay? So, give me two examples where I've already had one of them of the know-nothing schedulers. Round robin and random, right? These are the ones we talked about, okay? These are the simplest. I think random is simpler because random doesn't even... Random is just picking a random number. That's all it is. That's the only thing you have to do and that random number determines what thread you're going to run next, right? Round robin actually has to maintain an order in the run queue, right? And that's something that random doesn't even have to bother with, right? Okay. Know-it-alls. Well, you know, we just talked about this. So, if we could predict the future, what might we like to know about the thread, right? We'd like to know what it is going to do next, but specifically what? How long it's going to use the CPU? Is it going to exhaust its quantum, or is it going to block pretty quickly? What else? Anything else? Maybe how long the IO is going to take if the IO length is predictable, right? So, give me an example of a predictable or semi-predictable IO length, right? What's an operation that I might be able to at least assume it's going to complete, right? Read from a disk. Read from a disk, right? One example of an IO operation that I might not know when it's going to complete. Network. What about a read from the keyboard? Waiting for user input. No idea. You know, you might have gone away to get coffee. So, that thread might sit there for a long time. So, you know, if we could know these things, that would be nice. In many cases, a lot of this information is unknowable, right? What's the other big reason that this information... So, one key source of uncertainty here is you, the user, right? What you're going to do, the system doesn't know that. But what's another source of uncertainty? Why wouldn't I be able to, you know, to estimate how long a thread is going to use the CPU? Because the control path is very difficult to determine, right? Depends on a lot of other data, et cetera, et cetera. I mean, the best way to find out how long the thread is going to run would be to run the thread and see how long it takes, right? But that's what I can't do, right? So, in general, you know, code is complicated. There's a lot of different control path issues, so that makes it very difficult to predict. Or to have... Because you could have the thread say, oh, I'm going to use the CPU for 10 seconds, or, you know, that'd be a long time, 10 milliseconds. But it's unlikely to get that right. And we're unlikely to care because we don't trust the thread anyway, all right? So how long it's going to use the CPU? Is it going to block? Is it going to yield? All right, that would be nice. And how long will it wait if it's going to wait for something, okay? So, instead of predicting the future, we...what? Use the past to predict the future. Everybody repeat after me. This isn't important. If there's... If there's only a couple of things that you remember about this class, this...I would like this to be one of them, all right? So, ready? Everybody together. We use the past to predict the future. I'm not hearing people say it with me, ready? Use the past to predict the future. Okay. Very...you know, again, I mean, we're systems people, right? We're not, like, really deep thinkers, okay? But this is what we consider to be kind of a deep thought, right? So, let's use some past history about what the thread's going to do to try to predict what it's going to do next, okay? Use the past to predict the future. Yeah, good. There's some blue, blue color of highlight. We talked about one such algorithm, one such scheduling algorithm that we called multi-level feedback cues. So, how do these work? The first thing we're going to do is we're going to choose a scheduling quantum, right? This is the longest we're going to allow any thread to run uninterrupted before we stop it and decide if we should run another thread, okay? What else do we need to do to do MLFQ? What else do we need? What's the next thing? We need cues, right? It's multi-level feedback cues. We need levels represented by cues and we need multiple of them. Otherwise, we'd have single level or unilevel feedback cue and that would be kind of dull, right? That would look a lot like some of the other simpler, know-nothing scheduling algorithms that we talked to, okay? So, we established some cues and we're going to choose threads from the highest level cue first, the simplest incarnation MLFQ. Now, what do I do? How does this algorithm work? Who remembers? Give them penalty and rewards. What's that? Give them penalty and rewards. I give them penalty and rewards. How do I do that? So, what's the loop that I start into, right? So, I'm going to choose a thread. Like I said, I choose threads from the highest level run cue first, okay? So, I choose a thread from the highest level run cue. I run it, okay? What do I do if it's quantum expires and it's still running? What's that? I push it down in the cue. I push it into the next lowest priority cue, right? And what happens if it yields or blocks before it's scheduling a quantum expires? I promote it up, right? If the blocks are yields, I promote it. If it has to be preempted, I demote it, okay? And this is designed. So, what is this designed to do, right? Who gets, I think this is here. Who gets rewarded here? Iobon threads. Threads that sleep a lot. Threads that don't exhaust their quantum, right? And what about CPU bound threads? Where do they go? Down. They descend into the depths. I get to do this again. There's so much fun on one side to side. And these guys go up, okay? Questions about MLF cue, right? And then again, why would we, so we're going to, we just decided we're going to prefer these Iobon threads that sleep a lot, that block a lot. Why? They're more likely to be interactive, right? That's our claim. It's more likely that they're an interactive thread. But what else, right? What can I do with their weights? I can parallelize them with other threads' use of the CPU. So while somebody's, if all, one thread has to do is run a couple of instructions to initiate a read from the disk. If I start him first, and while that disk read is completing, I allow another CPU bound thread to run. Then I'm making better use of all the resources on the CPU, right? If I switch the order, then, you know, that disk is idle while the CPU bound thread is running, right? If I, if I schedule the thread that does the IO first, then both of those resources are scheduled efficiently, okay? Okay. Questions about scheduling algorithms after our little review? Any questions, questions? Okay. So, so today let's, so let's talk about Linux, right? We haven't really talked much in this class about Linux. You know, and I think it's, it's kind of weird, right? Because as, as, as computer scientists, which all of you are, and, and as people who, who might, maybe when you started this class, you had some interest in systems and I managed to totally kill it off. But as, I'm going to presume that you have some interest in systems because you're in this class. Linux is, you know, I think Linux is one of those things that we kind of take for granted, right? We don't really pause to think about how incredible Linux really is, right? I think when anthropologists look back on our, on our culture and our society, Linux is going to, the Linux community and the evolution of Linux and the rise of Linux is going to be something that people are going to, are going to be really interested in, right? Because we talked before about how operating systems are, you know, the most complex piece of, some of the most complex software that people write, right? And, you know, the most complex software, big, you know, has to, has to work really, really critically important. And yet, a bunch of people built this free operating system for fun, right? And it works really well. That's the thing. I mean, you guys are using Linux on your VMs. I used to run Linux on my machines and now I, now I run Mac, so I guess I had to hang my head in shame a little bit. But, you know, the point is that this, you know, this is a really, really cool store, right? So just stop for a minute and think about the fact that, you know, especially in this day and age, right, where everybody, you know, this is a whole cultural mem that, you know, you should only do things that you're paid for, you know, and I want to be really well compensated for my work and I want to get, you know, squeeze every dollar out of everything I do because I'm worth it, right? And yet, like, these people are having fun and they built this incredibly cool thing, right? So I think it's a really, really interesting thing to think about that this is possible, right? And not just possible, it actually happened. Yeah. So Linux, if you know anything about Linux, Linux is an operating system that tries to implement the POSIX standard, right? The POSIX standard is an agreement among different UNIXs about how to implement system calls and various expectations that the system is expected to provide, right? So Linux provides a very UNIX-like environment, right? But Linux is distinct from other communities that work on things like BSD or, I don't know what else is out there, Solaris, I mean, I don't know if Solaris is really politics-compliant, but, you know, there are other groups out there who are doing this, right? A lot of them were affiliated with academic institutions. Does anyone know what BSD stands for? Berkeley Standard Distribution, right? University of California, Berkeley, right? That's where that took off. There was another version of UNIX at the time that was being developed by IBM and one of the reasons that BSD got started was because people wanted a non-commercial open-source alternative, right? But the Linux community really, because of this open-source collaborative global development model, really took off in a way that some of these didn't, right? And that's why it's one of the more widely used, probably Debo's widely used Linux distribution that's out there, right? So I think that the story about the scheduler really provides as fast as... So again, I mean, this is an awesome story. The Linux community is this really incredible thing, and I think it's a very, very strange and inspiring thing to think about. At the same time, I mean, this is a big community with people in it, right? And it's not this utopian place, right? And this community struggles with a lot of issues, many of which are the function of its success, right? I mean, Linux now runs on all sorts of things, right? I mean, tiny little, you know, sensor modes and, you know, big corporate mainframes, right? And phones, right? I mean, the Android phone platform, which I do some research on, is this exploding phone platform, and it runs Linux, right? So Linux is really sort of out there everywhere, and it's one community, one operating system that's trying to support a really, really sort of diverse group of targets, right? Lots of different environments, lots of different drivers, lots of different people get involved, right? And then whenever you're doing something in a project this big, you know, there's this tension between true collaboration, a truly open environment where everybody contributes, and actually getting code out the door that works, right? And at the end of the day, a community like this lives or dies by the success of the products that they produce, right? That's what it's about, right? If Linux was, you know, producing buggy operating systems that didn't work, nobody, it wouldn't be a great story, right? No one would care, they'd be like, oh, a bunch of people wasting their time, you know, developing some sort of crappy buggy operating system, right? And then, again, I mean, in running a project like this, right, you have all these people who want to contribute. I mean, anybody in this room, now, when you finish this class, I mean, you guys can start hacking on Linux. That's one of the reasons I'm telling you this story today because it's a cool story about someone who did something just like this, right? And someone I don't think is, you know, is smarter than everybody in this room, right? Who had a day job, you know, who actually was doing something else. So, school. And I think the other thing that's fun about this story is just this challenge that we touched on a little bit on Wednesday of, you know, benchmarks for desktop interactive environments, right? What does it mean, right? What does it mean for your desktop to feel sluggish, right? How do you quantify that, right? And it's difficult to do, and that difficulty plays a role in this story, okay? So, and then again, yeah, incredibly contributions, man, this is like typo day. Incredible contributions made by hackers that, you know, that we're doing this for fun, right? Okay, so let's talk a little bit about Linux. 9.2 million lines of code, right? It's a big, fat, mature system, you know? Half of that, device drivers, right? A sign of this incredible diversity of hardware that Linux supports, right? 2,400 developers involved in this project. It's a big group. And on average, I guess, there's a new kernel release about every three months, right? So, rapid, you know, evolution. And these are not necessarily huge major releases, but there's actually a lot of changes that go into this. A lot of new drivers that appear, a lot of fixes to old problems, et cetera, et cetera, right? So, this is a really, really sort of active dynamic place. So, most of this, I would say all of it, all of this stuff is stuff that I sort of found by googling around and reading stuff and finding stuff on the interweb, right? So, I'm not speaking for experience. I don't know any of these people. This story is not intended to cast aspersions to anybody in this community. I think that these people are all, you know, heroes in many ways for what they do. And I don't know anything, right? So, just caveat there. I want to make sure that I'm not, you know, I'm not like sent buggy versions of Linux by angry Linux developers after this, right? So, here's in general how Linux development works, right? Every file has a maintainer, okay? If you want to contribute to Linux, one of the things you have to do is maintain your contributions, right? Because this community works, you know? If you want to push code into the tree, if you want that code to run on the millions of devices that Linux runs on, you've got to keep it up to date, right? You've got to maintain it. When people have problems with it, they're going to come to you and they're going to say, I don't understand how this works, or I think there's a bug in your code, right? And you're the one who's going to fix it, okay? Every big kernel subsystem also has a maintainer, right? So, things like, you know, the file system interface and the memory management and the scheduler and things like that, they have kind of a head maintainer who oversees all the development, right? So, you can start to see this as kind of a tree, right? This hierarchical tree that goes up, and then at the top there's these guys, right? So, that's Linus Torvalds and on the right is Andrew Morton. So, between the two of these guys, they approve, I'm trying to remember, there were some numbers on a website. I mean, both of them, so whenever changes get made to the Linux source tree, somebody signs off on them, right? There's a maintainer who signs off on changes that are put into the mainline tree. And both these guys have, like, twice as many sign-offs as the next highest person, right? So, a lot of the stuff that goes in the tree goes through them, right? So, they get sent patches, they review the patches, and when they're confident with it, they push stuff into the mainline tree, right? So, what is the mainline tree? Well, the mainline tree is what's used to generate official Linux releases, right? I mean, if you, you know, get a, you know, distribution like Ubuntu, what you're probably running is some official Linux release, right? That's sanctioned by the community. It may not be in a vanilla release, Ubuntu may have made some changes to it or applied some patches or done some configuration, but there, you know, there has to be some canonical version of Linux, right? And that's what the mainline tree is. So, other developers are allowed to, and I think they've been probably encouraged to maintain their own kernel trees. They can try things out. And in some cases, you know, when you start, you know, you say, I have this problem with Linux and I want to try out this new thing. What you end up doing is going and taking a stable version of a Linux source tree that's not the mainline tree, compiling that and installing it on your machine if you want some sort of feature that either isn't in the mainline yet or maybe will never be in the mainline, right? And as you guys use Git more this semester, you'll realize that Git makes this whole process very easy, right? This is one of the reasons. You know, Linus built Git out of his experiences maintaining this code base, right? So Git is really designed to facilitate a lot of this kind of interaction. All right, so let's talk a little bit about the one example of sort of the tension in this big development, right? So, you know, and this is a very, this line is not so hard and fast, right? But in general, I mean, Linux runs a lot of places, but one of the places that it runs is it runs on servers, right? And another place it runs, you know, maybe, now with Ubuntu and these other really nice distributions, maybe this is starting to catch on, I don't really know, I haven't looked at the numbers, right? But it does, people do use it on desktops, right? And on phones, right, which I would say is a class of computer that is more like a desktop in terms of interactivity than a server, okay? And these are really, really different environments and they differ in a lot of ways, right? So think about servers. Servers frequently, when you set up a server, especially if you're in corporate America and you've got this thing that the server, you know exactly what that server is supposed to do, right? Like, this is one piece of our overall web architecture and it does this one thing, right? It sits there all day doing this one thing, right? Over and over and over and over and over. It's SQL transactions, you know, file serving for your web server, whatever, right? And so, you know exactly what it is and you also frequently have tools that you've designed or other people have designed to measure exactly how well that server does that one thing that you wanted to do, okay? And this is important because if you, you know, if you get a new Linux distribution and it's suddenly 20% slower, you may say, you know what, I'm going to back up, right? I need that 20% of performance. The other thing I think is that, you know, imagine you're in a company, right? You have a bunch of people that work in IT and some of them are probably in regular contact with the Linux community, right? They talk to people who maintain the code that really matters to them. They email back and forth to people who write device drivers that are really important to them and so on some level there's this sense of, you know, I wouldn't say belonging but, you know, frequent contact between them, people who do, who rely on Linux on servers and the people who develop Linux, right? Because the people, the part of it is just there's less turnover, right? I mean, people stay with companies for years and years and years and maintain some of the same tools. What about desktop Linux, okay? So here's scrappy desktop Linux, right? I mean, Linux, you know, there's big story on servers. Desktop Linux is kind of like still, you know, still client its way up and just a very, very different kind of environment, right? You've got these really poorly defined goals. I mean, one of the things that the guys we're going to talk about today did was try to develop some benchmarks and this is like five years ago, right? This is not 15 years ago. This is still new. This is still happening, right? So he's trying to develop benchmarks and some of these things so that you can measure desktop interactivity. You know, you got this user community that's perceived as cheap, right? Maybe, I don't know, whatever. You know, oh, yeah, I don't want to spend money on windows or, you know, I don't want to bow down before Microsoft or whatever. Who cares, right? I mean, it's just, you know, I remember every year when I was at Harvard as an undergraduate there would be some flame war that would come up over our house email list about Linux versus Microsoft. I don't know. I guess it was fun at the time. Now it just seems whatever. Use what you like. If you want to pay money for windows and you like it, fine, right? And then, you know, these are people that they're not, like, a lot of these people hopefully, I mean, to some degree I think part of the problem with the Linux desktop communities are too many hackers and too few grandmothers, right? Maybe that's starting to change. Some of the new distributions work really well right out of the box, right? I mean, your grandma could use this, right? Really all she needs, right? And maybe Skype and maybe, you know, I remember my grandparents it was like their big goal on the internet was to play this one game, you know? Like that was most of what they wanted to do online. So, and these people they're just like, what's development? You know? Even if there was something wrong with their system they wouldn't know, I mean these are the people who call it Microsoft. You know, my computer crashed because most of the crashes in Microsoft are caused by device drivers, right? But people, you know, it's Microsoft Windows that crash it, so they call it Microsoft, right? And, you know, so the Linux community has this great mailing list called the Linux Coronal Mail List. Has anyone read anything on the Linux Coronal Mail List or subscribed to it even maybe? So I lurked on this list once for about six months, right? Oh my gosh! It's incredibly hard to understand, right? Like, I think I have a statistic up here and this is very scientific. Yeah, yeah. It's probably like 99.99%, right? Like, there's these because you're stepping in the middle of these incredibly technical, detailed discussions about these subsidies. You're like, what is that thing, you know? Like, again, I mean these millions of millions of millions of lines of code and people are debating the like one line changes, like I think there's a bug here. It blows your mind, right? I used to spend several hours a day trying to keep up with this, right? And it was just reading. I wasn't even contributing. And most of the time I was reading, I was like, don't understand that. Go to the next message. I think if you understand what's going on in this mailing list, it probably consumes 10 hours a day of your time, right? Just to read and understand, okay? And, yeah, I mean like, let's say you're some guy who downloads a bunch of for free, right? Okay. And then it's slow, or like something, Firefox, something in Firefox doesn't work and you go to Linux for the mail list. It's like, hey, my computer doesn't work. What kind of response do you think you're going to get, right? You know? And unfortunately, you know, this is kind of like, this is one of those things where this is a very serious mailing list for serious people involved in working at serious problems. And, you know, I mean they may be nice people if you like met them, you know, in a bar or at a restaurant or out on the street but when they're trying to like figure out how to change, you know, line 34 .c. driver that we think is causing some regression in this test that was reported by this big corporate company, they don't have time for newbies, right? They just like go away, right? You know? So this is a really scary place to go for help, right? And people don't, right? All right, so with that in mind, let's talk about Linux scheduling. So, you know, there's been several, the Linux scheduler, it's kind of interesting, when I was working on this, I did put this up there, but there's some quote from Linus, actually, and, you know, as you would expect, what he thinks about things carries a lot of weight in this community, right? It's named after him. And there's some quote, I guess, from him from years ago where he basically says I don't think schedulers are that interesting, you know? Like, scheduling is one of those things you think about once in a while, you get right, and then it's just done, right? And so, but despite that fact, there have been, you know, a series of Linux schedulers. So, I can't remember exactly when the 01 scheduler dropped, but earlier versions of Linux, like pre-2623 or something, used a scheduler that scaled with on, where n is the number of tasks, right? So, as you added more tasks to the system, the scheduler took longer and longer and longer to figure out what to do, right? And as we talked about last time, this is not good, right? Like, we want the scheduler to do its job and get out of the way, right? It should not be bogging down the system, right? So, there was a rewrite of the scheduler during the 2.6 era that aimed to fix this. And what was implemented by Ingo Molnar, who's the Linux kernel scheduler maintainer, so he's the scheduling subsystem maintainer, right? One guy. He implemented this 01 scheduler, right? And the 01 scheduler, as it implies, took a constant amount of time to select a new task to run. Now, the 01 scheduler, I'm not going to get into the details of the 01 scheduler, but the 01 scheduler essentially combines two priorities to come up with an overall priority for the task which is then used to make scheduling decisions. The first is the static priority. And the static priority is the priority that you assign to the task, right? When the task runs, if you don't assign anything special, it runs with some sort of default priority. And then Linux uses a tool called NICE to allow you to adjust that priority if you want to, right? So, there's a static priority and that's straightforward. Then, I have this dynamic priority, okay? And the dynamic priority was intended to essentially do something similar to MLFQ. It was trying to reward... It was trying to identify and reward interactive tasks. And the way it did this was by boosting their priority, right? So, you know, it would basically say, okay, I'm looking at the system and I'm trying to figure out which of the tasks are interactive and if I think they're interactive to a certain degree and now they're degree of interactivity, I'm going to boost them, right? The problem is that this boosting code was really bizarre and awful, I guess. I never looked at it, but it got to be really complex, you know, difficult to understand, difficult to maintain, and it was, you know, one of the consequences of writing really complex code is that it gets really hard to understand whether it's doing the right thing, right? So if you're trying to have a simple algorithm that's easier to model, right, we'll come back to this later, then to have a complex algorithm that's really, really hard to figure out, right? So this code got really, really sort of... Wow, it really is typo today. I need to start. Yeah, okay, anyway. So, okay, so, now, like, I think it was his first... I don't know when he started working in the Linux community. I won't say, right? But this guy is an Australian anesthesist. He's essentially, I think it's the equivalent of an American anesthesiologist, right? So he has a job, right? His job is making sure that people don't wake up during operations, right? And maybe this gives him time to think about other things. I don't know. But he gets interested in Linux, he starts hacking on Linux, and at some point, he starts getting interested in scheduling, and specifically, he starts getting interested in scheduling on desktop systems, right? And so here's an example, and I'm not going to read this whole thing, you guys can look at the slides, but this is an example, I think, of him, you know, being really thoughtful about thinking about some of the issues that arise on desktop systems. So this is this explanation at one point, I don't know where this originally appeared, where he's trying to distinguish between responsiveness and interactivity, right? Both of which he claims are important but he sees the distinction between these two concepts, right? And you'll notice at the bottom here, he starts writing benchmarks, right, to try to evaluate how well a system, how interactive a system is and how responsive a system is, right? So he wrote something called contests, which is obviously a little bit of a joke, and then interbench, right? So these are, you know, benchmarks that he was trying to use to understand the performance of the system, right? And we're going to talk about performance later in the semester, but this is a really, really wonderful example of what to do, right? When you are trying to address a problem, first try to understand what the problem is and be clear about what the problem is, right, to find the problem. Because if you don't define the problem, then you don't know if you've solved anything, right? And then you have to find a way to measure things, right? Because if you can't measure, you can't evaluate. If you can't evaluate, you don't know if your change has made a difference or not, right? So he's going through this really nice process of doing these things, right? He releases what he called the Rotating Staircase Scheduler, okay? And the Rotating Staircase Scheduler chopped out essentially 500 lines from the current scheduling code that he considered to be, you know, black magic, right? Like this really weird, hard-to-understand interactivity test that we were talking about before, right? And he replaces them with 200 lines of code, right? So this is usually a good sign when you're writing code, right? When you can take 1,000 lines of difficult scheduling code and replace it with 100 lines of, you know, easy-to-understand code, that's a good day at the office, okay? And there are some similarities to this algorithm with MLFQ, but let's not talk about the Rotating Staircase because in 2007 he comes back with a new version of the schedule which he calls the Rotating Staircase Deadline Scheduler, right? And this is his description of this, right? Starvation-free, strict fairness, scalable, and essentially interactivity as good as the above restrictions can provide, right? So look at what he's offering here. No interactivity estimator, right? This is one of the problems he had with the O1 schedule. It had all this complex code to estimate interactivity. He said, no, no more, right? Simple fixed accounting, right? It allows us to reason about things and also allows us to take care of the scheduling accounting very quickly so we can get another task set up, right? And let's see. The prime concern in this design is to maintain fairness at all costs, determined by a nice level, yet to maintain as good interactivity as can be allowed within the constraints of strict fairness, okay? So let's go through this as an example, right? Because this is pretty cool, all right? So the Rotating Staircase Deadline Scheduler has one parameter, right? Again, he's trying to simplify things, right? So if you simplify things and you have a bunch of gobbledygook parameters coming into it, that's not a good sign, right? And the interactivity estimator actually had a bunch of sort of magic numbers, right? That people didn't really understand what they were. So he wants this upon this. He says, one parameter, right? The parameter is the round robin interval. You could think of that as the scheduling quantum. That's the longest I'm going to allow any thread to one, you know, basically in one epoch, right? But we'll talk about how this combines with priorities. The other input is a priority, right? So we're reusing the original priority approach and the priorities are the priorities that are set by nice in the usual way, okay? So let's go through an example of how this works, okay? A long-winded example, alright? So let's say I have... So the first thing that he does is he establishes queues at each priority level. So I have three threads, priority two, four threads of priority one, two threads of priority zero, and I establish a queue for each priority level. So so far this looks pretty similar to MLF queue, alright? The second thing I do is that I use the round robin interval to assign each thread a quota, alright? I'm going to talk about how that quota works in a minute, right? But you can see here, imagine the round robin interval is five, I assign each thread a quota, right? And the quota is five actually for each thread. Now you're probably wondering, well, you know, how does this work? I have priorities and I have quotas, it doesn't seem like. And remember, one of the things that we talked about with priorities, I may want to use priorities to give high-priority threads more CPU time, right? And this design does that. It does that in a really interesting way, okay? So let's walk through how it works, right? So, okay, this is very like MLF queue now. I'm going to take the thread. Okay, so, okay, sorry, I forgot one thing. Each queue is also assigned a quota, right? And here the quota is essentially the equal to the amount of time for all the threads that were in that queue when I started one epoch of the schedule, right? And you might wonder, why do I need a separate queue quota and thread quota? I have an example of why you need it on the slide, but I'm actually not sure it's correct. And what I found is in later versions of the scheduler, there are no priority queue quotas, right? But just bear with me, right? Because this will make more sense, all right? So now what I'm going to do is very MLF queue, like I'm going to take off thread five, it's at the head of the highest priority queue, and I'm going to run it, right? And let's say it runs for its entire quota, okay? And blocks, right? Now, what would I... In MLF queue, what would I do with this thread? Push it down. And that's what I do here, too, okay? But what's interesting is, notice, and it's hard to see with the colors, this is still a priority to thread, okay? So I actually have not changed the base priority of this thread at all, and when we come back to how the system re-initializes, you'll see why that's important. The other thing that happens is it gets pushed down to the priority queue, but it's granted a new quota at that level, right? So what happens is, you know, I pushed it down here, but I renewed its quota back to five, right? And you'll see, so essentially, it has a quota, and now it has to compete with all the other priority one threads, all right? So let's say I run the next thread, and let's say that thread, you know, I think, let's see, this thread yielded, right? So it used only two ticks of its quota, so it goes back on the priority to queue, right? Now let's say I run thread two, and let's say thread two blocks, okay? So it's red. Now I put it back here because when it returns from whatever it's sleeping on, it's going to return to that queue, okay? Now let's say that thread one runs again and also blocks, okay? So now I don't have any runnable threads left in priority to queue, right? And what do I do? I go down, right? I start running threads from the priority one queue. I run this guy, let's say he blocks. I run this thread, let's say he also blocks. I run this thread, let's say this thread uses its entire quota, so where does it go? It gets pushed down into priority zero, right? Now I'm going to run thread, and it gets a new quota, right? So its quota gets restored in priority zero. Now I've got this priority one thread, thread nine waiting, so I run it, let's say it also blocks. Now what's happening? I'm going to run my priority two thread again, right? It got pushed down, but it's still runnable, and it still has quota space five, right? So I'm going to take this, I'm going to run it, it's going to exhaust its quota, let's say this is a CPU bound thread, right? Exhaust its quota priority one, where does it go? Priority zero. And in priority zero, it's again, a new quota, right? Now remember I said that priorities and quota combine here, right? And the idea is that a priority two thread can run for up to 15 ticks in any round of the scheduler. Five ticks at priority two, five ticks at priority one, five ticks at priority zero, right? So this establishes a direct relationship and an easy to conceptualize relationship between where threads start and how much CPU time they can use in one round of the schedule, okay? Any questions about this? So let's go quickly through priority zero, right? So I'm going to run this guy, let's say he exhausts his quota, now it turns out he gets pushed off the bottom, right? So he's gone, but he's still there, he'll be back, right? But I realize at some point I didn't have a bottom to this slide, and this is like animation 35 on this slide, so I decided not to fix the 34 that preceded it. Okay, so this guy also blocks, he vanishes, this guy also blocks, he vanishes, and then let's say this guy also blocks. Sorry, doesn't block, these guys have all used their entire quota, okay? And so they're done. Now I've cleared every level, right? So notice this about this scheduler. In every round I will run every thread that was runnable when the scheduler started running, no matter what run queue it was on. So I can guarantee by the mechanism of the scheduler that I will never starve a thread, right? Every epoch I run every runnable thread from every level, right? Okay, so this is again, this is a really nice property. And actually if you remember how we started I can place a bound on how long it's going to take me to get to the threads in priority zero. It's the number of threads in priority one. You know, it was the number of threads in priority two times the quota. Imagine all those threads, use their quota and go down to priority one, I've got to run all those threads again. So basically I can come up with an equation that it will take to get to a thread in priority zero is x, based on the number of tasks in the system, right? Why is this important? Responsiveness, right? Interactivity. I want to be able to say, you know, no matter what happens on the system the longest it will take, you know, the longest between the times that a thread down here because let's say this is actually an interactive thread and these other people have just been boosted up by nice or something, the longest it's going to take to run every thread on the system is bounded, right? That's awesome. That's a really, really, really nice thing about the schedule and there's parts of the schedule that are really carefully designed so that the thing is easy to model, right? And if you look at Koliwis' notes about this, he goes through examples, right? Here's an example where I have these threads of this priority and you can easily calculate using simple arithmetic what the longest the schedule it will take to reach a thread at a certain priority level is, okay? So now what happens, right? So now this is one of the things that's really, really important about the schedule and makes it different from MLFQ, right? What happens is now that I've completed so this entire process he refers to as a major epoch, right? I worked all the way from the top to the bottom and I've basically cleared the entire queue. This will always happen, right? What happens is that threads get restored to their original level. So this is different from MLFQ, right? Because MLFQ I pushed a thread down and that's where the thread lived, right? Here I pushed the thread down to get to run again and when I restart it jumps back to the queue it started, right? So this thread was still runnable. It's restored to priority two and it's given a new quota, right? And the priority two queue also has a quota update. I had a priority one thread that I pushed all the way down. It's restored to priority one given a new quota and I had these two priority zero threads they're restored to the queue and they're given new quotas and the queue's quotas are also updated, right? So again this is the big difference between MLFQ, right? Threads always pop back to whatever their static priority level is and they descend from there. So the rotating staircase, right? Falling down the steps. Finally I ended up in the basement and then you drag everybody up again and it kind of kicked... It's almost like kicking the little past children down the stairs, you know? Like, maybe that's a terrible analogy. Anyway, so... And then I just start this up again, right? So now here... So I have to admit, I spent about two hours reading the description of how this works and I wasn't exactly sure what happens in this particular case, right? What I think happens and what I'm going to show you is that, remember, I had set the priority to a quantum to five, right? This is when I started the epoch, okay? I start running this guy, let's say he uses two ticks but let's say at the same time the thread two that was waiting on something returns to the queue, right? As far as I can tell I think that the quota does not get updated, right? So he still has quota three but that's from the previous epoch, right? So what happens now is that I've got two threads in the queue that both have quota three, right? But the queue only has quota three. So this guy will run, let's say he exhausts his quantum he's going to move down and thread five is also going to move down. So once the level runs out of quota if there are still runnable threads in that level they all get pushed down into the next level, right? And again, this is so that I can bound the amount of time I'm going to spend at that level if I couldn't do that I might be able to create a starvation case if threads were coming back from doing that, right? All right, so that's the end of the example. Okay, so again, nice features of this approach, right? It's very easy to tell how long I'm going to take to get to a particular thread we talked about this, right? Simple fix to counting as he points out this is all just like basic little math that I have to do from time to time. So more recent versions use this interesting interleaving idea, right? So what happens is a task so this is a nice 19 task that's the highest priority level turns out that for whatever reason this has always been true but on Linux lower priorities are better, right? So negative 20 is the highest and what this means is that a thread the negative 20 priority will run in every... at every level, right? A thread with this, this is negative 10 so that's a little bit lower will run, it looks like, you know, every fourth level, right? Now with using the example that we talked about what would have happened is that he would have run in all the levels, you know basically he would have weighted and he would have run every level, right? But doing it this way allows us to achieve even better interactivity potentially, right? Because I'm interleaving things but it's essentially the same approach, right? So again, go back to our relationship between sleep and interactivity, right? So this is Galivis the design relies on the fact that interact to tasks by their nature sleep often, right? Most quote-unquote fair scheduling designs end up penalizing such tasks indirectly giving them less than their fair possible share because of the sleep and have to use the mechanism of bonusing their priority to offset this, right? So this is what the 01 scheduler did, right? His scheduler does not do that if you sleep, your quota is still there, right? And you can use it as soon as you wake up, right? All right, so now what happens, right? So this is kind of like the interesting personality part of the story, right? So one of the things that was difficult for Galivis was to really demonstrate at least to the satisfaction of the kernel developers that this was an improvement, right? You know, Linus said some nice things I guess on the mailing list but eventually these patches never get integrated, right? So he's maintaining his private Linux kernel tree. There's a couple of nice features that are in there. Those features never make their way to the main line. And simultaneously, the Linux scheduling maintainer decides that, oh, well, maybe this is a good time to update our scheduler, right? And maybe while I'm updating our scheduler I'll use a bunch of ideas from the rotating staircase, right? And it turns out that the completely fair scheduler CFS, which is what Linux ships with now is even more complicated. It uses a red-black tree and stuff like that and I didn't want to talk about it. So we're not going to. But the point is this is interesting, right? So on some level, you know, this work didn't move the community, right? It just didn't move it exactly to that particular place, right? And finally, you know, Khan gets frustrated. You know, you can find interviews with him online where he talks about the fact that he was really annoyed that none of his patches ever made its way in and he, you know, he leaves the community, right? So let me go, anyway. So one question is should you use a single scheduler for every device, not clear? But let me just cover this last little bit because this is kind of fun, right? So this is why I think the morals of the story are, right? Yeah, I believe. I mean, the idea of fairness, right? I mean, something that Khalifa was struggling for with his design was fairness, right? And to some degree, fairness is guaranteed by the fact that every thread that's runnable gets a chance to run in about an amount of time, right? Every time through the schedule, right? You know, I think, I haven't looked carefully at the new design. I'm assuming that some of those ideas made its way to the new design, but to some degree in the open source community, people don't hold on to stuff like that, right? And again, I think it's a victory for his ideas, even if it wasn't a victory for his code base, right? Yeah, I don't know about that, but it doesn't surprise me, right? All right, so this is an interesting thing, right? So now 2009, right? Khalifa's returns, right? He couldn't, you know, in this interview, he's like, oh, I'm gonna, you know, I'm gonna take up a new hobby, I'm gonna learn Japanese, right? So apparently it took him two years to learn Japanese, right? And then in 2009, he decides to start hacking Linux again. And he returns with something which he refers to as the brain fuck schedule, right? Or BFS. And he's still riding the same horse, right? He's like, I want to achieve interactivity and I want to do something that's demonstrably simple so that we can reason about it, right? This is his description of it, forward-looking only, rigid fairness, nice prior distribution, and extreme scalability within normal load levels, right? So one of the things that Khalifa started to push back on is this interest in the Linux community of like scaling to 128 cores, right? He's like, look, my phone has one core, maybe two, right? But I think we can write a nice scheduler that works well for a two-core system, right? Without, you know, worrying about having to support 128 cores. Maybe it's time for some schedule and diversity, right? So as I like this, he gets this list of reasons why this is called the brain fuck scheduler. It throws out everything we know about what is good about how to design a modern scheduler and scalability. It's ridiculously simple. It performs so ridiculously well despite being that simple. It's designed, this is good. It's designed in such a way that the main line would never be interested in adopting it. Which is how he likes it, right? It'll make people sit up and take notice of where the problems are in the current design, right? To some degree, he already did that, right? I mean, RSDL did that. Led to the completely fair scheduler. It throws out the philosophy that one scheduler fits all, right? I had a slide about this. He proposed a pluggable architecture for schedulers that would allow you to use multiple schedulers, right? Or to choose which scheduler you want to use in a more flexible way. And finally, oh, sorry. It actually means that more CPUs mean better latencies, that's good, right? And because I must be fucked in the head to be working on this again, right? So this is my favorite. So next week, virtual memory. We're done with the CPU. We're moving on. We're going to talk about memory. I love virtual memory. I think it's an awesome topic. A lot of elegant ideas. A lot of cool stuff. Come Monday and we're going to kill it. Let's see you then.