 One of the things I do at the University of Buffalo is teach a large and I think very fairly popular course on computer operating systems. The course is known to be difficult, it's also known to be a lot of fun and give students a chance to learn quite a bit. So like, you know, with my group stuff, like with phone lab, like with my other classes, everything is online. It's taken us kind of several years to build out a nice website for the class. The website's not interactive or anything, it's just sort of static content. And I can't take any credit for the assignments for the course. These are drawn from the OS 161 assignments that were developed by David Holland and Margo Seltzer at Harvard. I think we've put together nice write-ups for them. We have lots of explanations for students. We've built a tool chain that students use that's based on a vagrant virtual machine, makes it very simple for them to install all of the different pieces of special software that they need to learn and to take the class. On the other hand, this material is mine, so all the exams for the class are online along with all of my lectures. So I've been recording my lecture videos for about five years now, I think. And you know, if, okay, I'm going to subscribe to my own channel here. So my, you know, all of my videos are up on the YouTube channel. And these, this site has access to all the slides in sort of a nice format. This is slides from when we were reviewing a paper. You know, this is probably a more representative set of slides. So this stuff's all up online. There are also the slides themselves. You know, we, you know, have some little bits of fun before class and things like this. So all this information is up online. One thing that we have built for the class, which has been fun, is a new testing tool. This is called, let me go back to the About page, this is called Test 161. So we got tired. When I started, I originally built some automatic grading tools for the OS 161 assignments. They weren't necessarily fantastic. And over time, we decided to dramatically improve them. This is a project that my students Scott Hasley and former student Yee Hong Chen were very heavily involved with. And this is awesome. This is the OS 161 testing tool that I have always wanted. It pursues kind of an interesting design approach, which we consider to be sort of distributed automated grading. I think in many cases, we're seeing more use of automated grading tools, which is fantastic. But there's still a lot of centralization involved and I think there's two reasons for that. One is that, you know, people don't necessarily know how to build sort of distributed software testing tools like this. And there's also a concern about giving students access to the tests themselves because people worry that students are going to gain the system and just reproduce the output of the tests. And so we've got around that in a very clever way. What we do is we actually inject secrets into the code before we compile it and run it during testing. And that allows us to verify that the code is running the tests that we wrote. So we make sure that students don't bypass the test by inserting their own code. Then we make sure that the code is producing output and students can't generate their own output because they don't have our secret keys. So we're actually doing this in the context of testing a small operating system. So students have a lot of control over the system that makes it difficult. We don't even trust printf because students have to implement it themselves. So this is a fun tool. We have students can submit things. There's a reactive interface here that when they submit will show them the results of testing. The Test 161 tool is really intelligent and knows about dependencies. It allows you to declare new tests if you want to. So here's an example. So showing the output from the system as it ran. This was some system simulator as it booted up and I ran a particular, here's the part where I ran a particular test and it was supposed to panic and it did panic and so I got credit for this test. And students see this updated live as the tests are running and then they can browse through this when they're done. There's also a leaderboard as you always need to sort of encourage students. Here's the assignment three leaderboard from last semester. We can see that there's some students that did extremely well. So this is fun. This class is very portable. It can pretty much go anywhere. I think the OS 161 instructional operating system that we use is excellent. It's a great balance between a system that is kind of too simple. Having students do OS-esque assignments that aren't in a real operating system. And then having people hack on Linux as a learning curve like this. And already has a lot of things implemented that you want students to be able to build themselves. So this is fun, continue to maintain this site over the next few years. And there's a lot of very helpful content up here. And we're very, very proud of the test 161 tool. I think it's a fantastic automated gradient approach.