 Okay, good afternoon everyone. Welcome to OGBB3. This afternoon we have Tom Clarke presenting what should a system administration students homework look like. I haven't done anything yet. Okay, so I think before I start I should take a second to kind of explain what I am fully aware as a long and ungainly title for a talk. But the reason is the reason that I hope to convince you of is that if you want to teach system administration this is the crux of the issue. This is the question you've got to answer. What are you going to assign the students to do? How are you going to assess the quality of their work? It turns out in teaching system administration I think that's the tricky issue and that's the issue. I'll kind of talk about what I've done towards solving that problem, towards answering that question today. At first I was thinking you know observing different talks over the last few days and I'm using on this and thought about well what are the attributes of a good conference presenter and really I think there are two. And the first one is of course really strong expertise about the particular topic on which we're presenting and fair enough that makes sense. And the other one, the one that I hope is more important, is a real genuine enthusiasm for the topic, the sort of enthusiasm that it's being really can convey and share with his or her audience. And I think for the next few minutes I can promise you at least a second of those two things because I am really enthusiastic about the idea of teaching system administration. I mean I like system administration, I've been a systems administrator, but I really like teaching it. I think that it is something that is worthwhile, I think that it is challenging, but I think that it's fun and I hope to share some of that with you here. And almost that, I mean I haven't been teaching this stuff for that long. I've most have been working in industry in my career and it's only a few years ago that I started teaching as my full-time job and started teaching system administration. So I think it's worthwhile to take a minute and think how did I, how did I get here? How did I come to be enthusiastic about teaching system administration? And as it turns out that's a little bit of a funny story or maybe an embarrassing one we'll see. And yeah, so it turns out I run my mouth a lot and interesting challenges, that may be happening right now. It certainly happened, I can tell you, what about three years ago or so when I was interviewing for my current position at Otago Polytechnic. Because apparently at some point during the interview, now my memory of these events is somewhat hazy, but the interview panel insists that I said this and we have to take them with their words, said that at some point during the interview I said, you know what would be cool? Like an advanced Linux systems administration class, that's not the kind of thing you see very much, that would be a neat addition to the program. But I said this, I've got to own up to it. So I got the job, that's what you can see. And I show up, so my first day on the new job and what do we do? We have a department meeting and the topic is what are the classes that we're gonna offer in the next academic year and how we're gonna organize it. And so at this meeting where, you know, my very first few minutes on the job, my new colleagues come to me and you say, that thing you talked about the interview, that advanced sysadmin class, we like that idea, we think that is cool. We'd like you to teach it first semester next year. And something like, oh, I've kind of got myself into a good one there because that's gonna be a tricky thing. But why is it such a tricky thing? I mean, presumably I'm reasonably well qualified, you know, to be a lecturer, teaching stuff is what I'm supposed to do. Why is this one so hard? And so let's talk for a few minutes about why it's so hard. The real problem, first off, is that teaching sysadmin administration itself is pretty new. It's not traditionally part of a CS curriculum. It's not traditionally CS, even though I would argue that it's very relevant. But because it's new, we don't have a lot of examples. We can't really look at a lot of things and say, oh, that's an example of a good systems administration class. And it's worth it to take a second and contrast that with situation and teaching programming. Because a programming situation is a lot different. If we say let's set up a new programming class, that's relatively easy. We have plenty of good examples to work with. There's lots, of course, material out there available. Why? Well, for one thing, because yeah, we've been teaching programming for a long time. And so we at least claim to know what we want to teach. We at least claim that we know, we say what we want to teach. What are the big ideas? What are the really lasting themes or whatever that we want to convey to our students? We can do that with programming. And we can at least claim that we have some idea about how to teach these topics. And certainly it's the case. I mean, the really nice thing about teaching programming, so I'm going to teach you programming. What am I going to do? I'm going to have you write a computer program. And by the very nature of the you have produced an artifact that I can examine and evaluate its quality. I can look at your source code and see whether you've done a decent job. I can run your program and see whether it satisfies the requirements of the assignment. And I can assign you a mark based on that. And that's all great. But we don't really have an analog like that for assistant administrators work. You don't produce quite so much of an artifact. I mean, we could say, oh, I'll look at the config file. And that's fine. But if you think about it, there's something about the way config files are set up that it doesn't really provide a good measure of the quality of a student's work looking at a config file and the way they're looking at source code does. So that's one problem. But another problem is, again, I mean, assistant admins, they don't really write config files for a living. That's not really the task. The task is to successfully run systems in a way that makes them available and then performatically and that sort of thing. So it's not about producing a thing that we can look at later. It's about doing a thing. And then if I, as a lecturer, I want to assess your performance on it, I need to watch you do the thing and see how you do it in real time, if you will. So that's the big challenge we have to sort out. There are some other challenges there. So, yeah, I say there are a lot of examples to draw on. What examples there are in terms of classes and materials for teaching classes in systems administration. Those things tend to be vendor-driven or certification program-driven. And I mean, those classes are fine up to a point. They have their place and that's all right. But that's not really the thing I was looking for. You know, in a particular which you get in those classes, well, you get two issues. And one is it tends to be much more about how to accomplish a specific task, usually on a specific platform. And that's not really a lasting valuable skill that we want to teach in school. That's not something worth paying a fee to be taught how to do. That's not really the kind of thing that we're shooting for there. And, you know, in a way, the skills, they're just a little bit too small, a little bit too narrow to really capture what systems administration is. The other problem is that, of course, a lot of these certification-based classes, which are really being trained how to do is how to pass the certification exam, which is not the same thing as administering systems. So, that's not going to quite work. So, anytime we're going to teach something, teach something at a tertiary level, what we really want to do is we want to think about what are the big, lasting, durable principles of things that are going to work year after year after year. I mean, I graduated from university some number of years ago. But the things that I was taught back then, even though I don't do those things in exactly the same way today, I mean, the tools and practices have changed, but the big ideas have held up. They worked for me. And so, in system administration, it's a fairly new topic to teach. One of the challenges is to say, well, what are those core principles? What are the things that are going to hold up, even as the tools and techniques change? I don't know what to state that is that really what you want to learn, you want to learn why you're doing a thing, because if you understand why you're carrying out a task, then you're in a position, you're equipped to develop the kind of judgment that you need to decide for yourself how to do it. And that, that is education. That is something worth paying for, going to school for. So that's the kind of thing we want to do. So I guess that's the statement. That's the outline of the problem that I saw as I'm trying to, to design this new class in systems administration. I need to solve these kind of problems. But how specifically am I going to do it? Well, whenever I'm confronted with a big sort of professional challenge, what do I do? I reach out to my network. I talk to peers whose opinions and views I respect and value. And I kind of say, what kind of ideas do you have? And presumably you do the same kind of thing, because that's why you come to conferences like this one. So that's what I did. So in particular, I went out and I talked to working systems administrators that I knew, and I posed them the following question. I said, let's suppose that you're hiring an entry-level systems administrator so your desk is piled high with these freshly printed CVs of recent graduates. And one of the things I can do in this class that when you see it on my students CVs, it's going to catch your eye and cause their CVs to rise as if by magic to the top of the stack. Because if I can do that, I mean, that's some measure of success. That's a good idea of what to do. And there was a pretty clear and rapid consensus amongst the people who I talked about. One of the things that's important to point out is they said, we need classes like that to be taught. When we look at recent graduates, they're not really being taught the sort of things that we need. So that's great. What I've learned early on is that the project seems to be worthwhile in the eyes of the people whose opinions I'm seeking. And this thing I pointed out, in a way, which they said, look, it's no big thing to configure Apache on one server. Great, you can do that. That's not. But we really want to do the kind of lines we're running in which are services that have multiple tiers and they're running across multiple services. As one person said, they have a lot of moving parts for often flying in different directions and visualizing and understanding that. That's the kind of thing we want to see. Okay, so in my eyes, this is pretty good news. I think Bob Young, right at the keynote, he would say, oh, this is a problem that aligns with my business model, I guess, is the entrepreneurial way to say it. Or I would say that it aligns with the way that I want to teach. This task works. So one of the principles that I want, that this is aligning with, one of them is that I don't so much like talking about things. I like, well, okay, actually I like talking about things. I mean, I'm doing that right now, that's not quite right. I, but I would rather that my students do things rather than talk about things or even worse, listen to me talk about those things. That's the thing I want to have happen. And that's great, because right away what the, my colleagues who I talked to did was they said, okay, here's something to do, something worthwhile. I want a consistent thread to the whole class. Or to put it another way, what I don't want is, hey, we're gonna, this week we're gonna do this topic and we're gonna complete it and then we're gonna set it aside and look at another topic. I want a theme where it says we're gonna, throughout the course of the semester we're working towards one big goal. And the work that we're doing early in the semester is leading towards that big goal and the skills that we're developing early in the semester are skills that we're gonna apply later in the semester. That's the kind of thing I want to get. And I guess what I'm looking for is some degree of what I call realism. And the quote marks there are very intentional because when I talk about realism, it's sort of a Disney-esque theme park sort of realism. You know, not realizing the sense of push you off the pier and see if you can swim, but real in the sense of that I'm gonna present you with scenarios that are carefully constructed so that you'll have the sort of experiences that I think will help you learn. And that's what I'm going at. And in particular, whether I'm real is I'm gonna accomplish that by giving you real tasks, the kind of tasks that system administrators are gonna do performed with real tools, the same tools that you'd use in the industry. And I highlight this one because that's really what I talk about, which is the assessment. I wanna assess your work on some sort of realistic standard. If I'm gonna assess you on passing this exam, then you're gonna direct your behavior towards passing the exam. You're going to learn how to pass the exam because I've told you that's what's important. So instead, I wanna say, well, I'm gonna measure your performance based on the same sort of things that your performance might be measured on in a workplace. Because then you're gonna direct your behavior towards that sort of thing and you're gonna develop some sort of realistic skills. So that's the thing that we're gonna try to accomplish. Okay, let's get concrete. I've got 16 weeks in my semester. I need to do something. So in particular, I need to start, remember I said I wanna identify four, well, I didn't say four, it turned out to be four, sorry. I wanna identify some fundamental, some big themes, some topics that I think are gonna hold up independent of how we execute those things. And so I hit on these four things. And I'll point out here that when we look at these things, I, in parentheses, talk about the software that we're gonna use in demonstrating those ideas. Not because the software is important in particular. It's not that you need to learn how to use Puppet, Chef's Fine, whatever you wanna use is fine. But because if you, for most of us here in the audience, if you know what pieces of software we're using, you have a good idea of what kind of tasks we're talking about doing. That's one of the reasons why I'm telling you about the software. Another reason is because these are pieces of software that work pretty well for me, so maybe they'll work for you if you're trying to do a similar thing. So my first big theme is something I call the sysadmin professional practice, which really comes down to things like knowing how to work with a ticketing system, something that a typical graduate has never seen before, and how to use that to organize and document work, and how to, and they need to learn how to write and maintain documentation. We'll see, they don't always do that well at it, but now they do the rest of us. And in our case, we use MediaWiki for that. It's a good tool for sysadmins. And then we get on these three, kind of what I call more technical topics that I decided were important. These are configuration management, I'm using Puppet for that, system monitoring, we're using Nagios for that, and backup and cover with Bacula. Now in particular, any proper old school sysadmin is gonna look at this list of these three topics as I've constructed and say, ah, my great, great beard, you've got this wrong. I mean, backup and recovery is the important thing. That should be first. And you're right. And in fact, the very first time I taught this class, I did backup and recovery first, and did configuration management last. I mean, I don't do that anymore. And there are two reasons for that. The first is, if I teach you configuration management last, then I teach you configuration management after you've already done the lion's tear of your configuration. And so that doesn't work out so well. It doesn't give you much opportunity to apply the skill. And so that's one reason why that goes first. Another thing is that in general, just observing student performance, the students seem to have the easiest time of those three topics with config management. So let's start with that. And then monitoring was the next thing, difficulty wise for them, and backup and recovery turned out to be the thing that they had the most difficult time way. So that's fine. We'll present the topics in that way. And now we see what the quotes are around realism. The quotes are the shorthand for my certain sort of artistic license for how I can tweak realism in a way that leads to a good class. That's what I'm going to do. And in particular, what we're going to do this is we're going to put the students in pairs. Right at the beginning of the semester, say here, these are your virtual servers. And we'll start early in the semester by getting these things up and running and then putting them to work later on in the semester. Okay. But remember from one of my earlier slides, what I said, real tasks. So if you set up puppet so that you kind of install and configure Nagios and then use that to monitor puppet, that's starting to get a little bit contrived. That's not where we're after. And remember, this problem's already solved because early on people said, no, no, no. I mean, find some interesting application to run, something that's got a few bits and pieces that's interesting enough and where the functionality is distributed across different services and different servers and that sort of thing. And now we're getting close to this real assessment that I'm after. So the service that I found works pretty well as own cloud. It's got just the right degree of complexity, the right number of parts and that sort of thing. And seems to work out pretty well as a service to have the students run. So we'll have an application server and it's running the own cloud web attend and we'll have a talk to a MySQL database. And while we're at it, we'll have it off its users against an active directory. Because even though I said it's a Linux system administration class, it's good to look at this in the context of a heterogeneous environment. And again, the feedback that I get from SysAdmins is you need some active directory in there because real world, that's something that's valuable. So okay, we've got that. So they're gonna deploy and operate this service. And now they've got something to use, puppet, to use Nagios, to use Bacula4, to use RT and all that sort of thing, to successfully operate the service. So what in particular are we gonna do? So as we get later in the semester, in the first parts of the semester, we're just getting all the infrastructure in place. And then finally we say, okay, we've got this two week period coming up and this is your big assignment. This is your final project. So at the beginning of this two week period, in fact, first thing Monday morning, you need to have own cloud up and running and ready to go. And over those next two weeks, you need to keep it running. You need to, in fact, a half of your mark is just gonna come from up time. In particular what I had hit on, is I said over those two weeks, you can have two hours of downtime and it won't hurt your mark. And more than two, and I start to take marks off. Which seems kind of generous, but I remember these are students who were doing this for the first time. And actually the arithmetic worked out in a convenient way. And it seemed to work okay, but we'll see more on that later. There certainly are ways that you can fail to hit that desired uptown mark. Yes. This needs to alter the audio settings. Oh, right. Sorry, that's for sure. Thank you so much. I don't know any jokes that don't violate the code of conduct. Is this the first system in the past? No, they do take like an elementary Linux class. How is that? How is that? Why? Why? To tell you precisely something. How does the sound level work for you guys? I can hear everything great, but. Let's see, where was I getting? Oh, so how are you gonna miss your uptown mark? Well, one is you're gonna miss your go live date. You're not gonna successfully get on cloud ready to go by the first thing Monday morning. And one reason you might not do that is because you, for example, you haven't got your puppet sorted in such a way that you can efficiently get things deployed in that sort of way. So that can happen. The other thing is that it's gonna be some problems over those two weeks, part of what I'm paid for is to make sure those problems occur. And so these problems are gonna lead to downtime and you're gonna need to resolve those things quickly. And again, here's the thing that's important is that I'm not presenting these problems to torment students, not that I don't enjoy that part. But because that's the way that I measure, like are they successfully using Nagios and that sort of thing. Well, let's take a look at that. But actually too, so that was half their mark. The other half, right, which is they're gonna be presented with issues like tickets, all sort of role play the role of a typical user, all might role play the IT manager and all say, hey, you need to get this thing done. And so besides measuring uptime and you're doing their mark on that, we're gonna look at their performance on closing tickets. Are they closing them in a timely manner? Are they meeting a certain sort of rudimentary SLA that we've spelled out at the beginning of the assignment? And are they effectively solving the problems and are they documenting their work properly and are they communicating with the effective parties? So when they say, well, I'm not closing this ticket as quickly as you would like, but at least I'll talk to you about why. Because that's worthwhile. So that's the other half of that mark. Tickets-wise, what are we talking about? And how does that fit into my grand scheme of things? Well, you know, a trivially, you know, an ordinary user could ask for a password reset or something like that. Not all that interesting, but realistic and make sure that they've got the process sorted out right. Now, restoring a deleted file is perfect because if they've got their backup and recovery operating properly, if they have documented the recovery procedure correctly, then they will very quickly restore that deleted file and their performance on that ticket, I mean, will be clear that it's excellent. So they will have demonstrated that they can apply, they have applied the principles of backup and recovery correctly. That's the kind of thing I'm after. In my little IT manager role-playing thing, one of the things I like to do is just give them a big list and say, oh, I've got these 500 new users coming online and I want them up, you know, start a business tomorrow morning. Um. And what I hope the students are gonna do is they're gonna knock up a little nice script and get that thing sorted out. They don't have to. I'm not gonna mark their performance on, I'm not gonna look at the script. I'm gonna see did they get the 500 users online in the deadline that I gave. And either way I went, because either they wrote a script and successfully completed the task, or they didn't write a script, in which case they learned the hard way, which is the best way to learn anything, that next time they're gonna write a script. Okay. So that works out pretty well. And another important thing, so remember when I was talking about programming, I said, look, you know, when you do a programming exercise, you produce an artifact that I can look at later. Tickets are an artifact that I can look at later. Any good ticketing system, I can go back later on and look through the log of the tickets. And now I'm like, ah, I've got something I can look at that tells me about how you did. It says, oh, how long did it take you? I mean, again, did you carry out the process, the procedure in an effective and reasonable manner? So that's working well for me there. Okay. So yeah, now this is get the fun part, which is I just get to go, you know, vandalize things. That works out pretty well. Yeah, yeah, so of course there's an easy way around that, which is I don't log in as my normal user, but oh well, still, it was worth making a note. So yeah, I'm just gonna go into break stuff. So yeah, I could just go in and quietly shut down my SQL D, no big deal, but again, this is perfect because if they've got Nagios running right and doing what it's supposed to do, they're gonna get alerted in a way which both shows that something is wrong and in fact zeroes in on what's wrong. So if Nagios is doing its job for them and if they've done their job in getting Nagios set up, then I can shut down my SQL D. They're gonna have that sorted in just a few minutes. About as fast quickly as they can get the alert, they can solve the problem. And if they don't have Nagios sorted very well, well, not everybody gets the A, I guess that's why. But yeah, I mean, I'm not always gonna be that gentle. It's just not in my nature. So at some point, I wanna do something nasty. And I do warn students, I say sometime in the two weeks, I will do something unpleasant. And so in the 2014 instance of this paper, what did I do? Pretty much I just went under their systems, picked some strategic directories and did RM minus RF, wiped out some stuff to face their website, generally mucked things up really well in a way that they wouldn't be easy to recover if they're not doing their job right. And yeah, I did it around 3 a.m., what the heck? Yes. I'm assuming you screwed that. Did I what? Did you speed it? Oh yes, yes. Yeah, I did, although there's a more on that later, I guess, because I would like to do a better job of that. Okay, so great. So now they've got real downtime. And now that, are they gonna hit that to our window? Now this is a real challenging thing. And all of the components that we talked earlier are gonna come into play. Have they documented their procedures correctly so they can quickly resolve this issue? Is Puppet sorted outright so that it can restore a config? Are there backups and recovery ready to go? If so, then they're not gonna have trouble. They're gonna recover from even a mess like that in the window that we have. And in many of them do. Now how am I gonna observe all this? Cause I remember I said there were two parts of the problem. One is what are the students gonna do? And the other is how am I gonna observe it? Cause really I need to observe it as it's happening, but I can't observe it over a two week period around the clock. You and I don't drink that much coffee. So I'm gonna of course have my own instance of Nagios. I mean, you know, eating my own dog food, I guess. So tracking the uptime, that part's easy. We get that. And again, I said earlier, look at these RT ticket logs themselves are a terrific artifact. The Wiki, one of the great things about a Wiki, a Wiki is a good documentation tool. It's a good teaching tool because I can go look at the edit history and see there and see, ah, look, there are a whole lot of edits like right before the deadline. So yeah, again, these tools are working for me. They're helping me to observe and assess performance in a pretty good way. And then what I'm gonna do with each student group is after that I sit down and have this post-mortem session where we're gonna look through the RT logs together and talk about let's see how that thing went and figure out what the mark is. So all right, I'm pretty happy with this. This thing's working okay. So in particular, how did it go? So remember, the big first measure was uptime. And I say it went surprisingly well. So if we look at that 2014 instance. So I went in and trashed a server pretty hard, made it pretty useless. And nearly every student group who could get a passing mark had no trouble getting their system back up and running inside that two-hour window. In fact, they didn't have quite enough trouble. If I look and say this is something I need to improve on, I need to make it a bit harder, which I guess as an instructor, that's a good thing to have. Or maybe I'm just really good. I think I need to make it a little bit harder. I actually had one student team who after trashing the server, had everything restored and functioning in approximately 10 minutes, which is pretty remarkable. Now when the porous motor to me, it did come out that a large part of that was good luck, that one of the students was at that very moment acting something. And so immediately there was a problem and immediately started restoring from back up and that worked out great and good on him. I mean, that's fine. But yeah, really as it turned out, if you, you know, any team that was capable of getting a passing mark actually resolved this, what should have been a difficult issue in about an hour? And I said they had about two. So this is an area where things need to work. So what could I do? Well, one is again, maybe I'm satisfied and that's okay. Another is maybe two hours is just too much downtime to allow. Maybe I should shorten the window. But I did say that the arithmetic worked out really nice, the two hours worked out to be a nice percentage. So I don't quite wanna do that. And in fact, my suspicion, my hunch and the way I'm gonna approach this is the third way, which is that I need to mess with their systems a little bit more or a little bit more often. And in a slightly more unpredictable way, and we'll talk about that in a few minutes. As far as resolving sort of issues, like tickets, student performance on tickets, as far as like closing tickets, they did pretty well. Now in a lot of ways that makes sense. I mean, one of the very first thing we started at the beginning of the semester was using ticketing. And in fact, throughout the semester, whenever the students had to perform a task, whenever I signed something, I signed it by opening a ticket. So by this time late in the semester, they've got, for the most part, they've got this ticketing process down and they're resolving issues promptly. But one of the things that they're not taking on board, or at least they're not demonstrating to me that they've taken on board, is that I'm trying to say, look, the ticketing system is both a way to organize your tasks and prioritize what you need to do. But it's also a document tool. It's a build up a little knowledge base of how to solve these problems. So when a ticket comes up and you say, that's really similar to what I've seen in the past, I should be able to go back and look in the ticket logs and see how to do it. Some, but I don't think enough student groups were really taking advantage of that feature. So that's something to sort out. And documentation, otherwise, nah, no. No, that didn't work out very well. Let's be honest, how many of us are really good documenters? Yeah, well, but I mean, it needs to be better. And how can I make it better? Well, let's take a look at the Nagios. I was actually pretty happy. The student groups did well with Nagios. When I talked to them after the semester, student groups had a good impression. They really got Nagios. They understood the value of Nagios. That's something I was really happy with. Why don't they understand the value of documentation? Well, one is they hit writing documents, I understand that. But another is, with the Nagios example, there was this clear and broad path between getting Nagios working right, perform well on the assessment. Then everybody got that. And what seems to be missing is in that documentation thing is that that path is neither clear nor broad that good documentation is leading to good performance. So that's something I need to work on. And quite frankly, I'm not really sure how. I mean, I'm not sure how I'm gonna form that broad, clear path, but I wish I did. I will suggest one. You will suggest one? All right. Now to repeat the question to the microphone purposes, get the teams to split up a mix halfway through the process. Yeah. Repeat the question. Right, so Jimmy just suggested that having students, hand off and use somebody else's documentation on their servers would be a way to work. Yeah, that is a good way. I have a suggestion as well, actually. Yes. Microphones. You can have a final exam as well that is worth some number of marks where within a time limit, they have to follow their documentation to rebuild the exact same or a very similar server configuration. Yeah. Yeah, I search a sand. Yeah. Another might be to have as part of your documentation stuff that they need to document for users, which you can then, you know, you log a T-ticket to say, hey, I don't know how to create my own blah and the documentation for that. Once a user has asked that, you put it on the wiki, there's your documentation, there's a clear path. If you don't see that, then you log another ticket. Oh, I like that. Yeah. My suggestion is to emphasize the use of playbooks and so to actually have that in the wiki and particularly break a system in the same way twice so then you can see how well people actually filled out their playbook the first time and how well they actually used it the second time. Yeah, very much so. And certainly we talk about playbooks, but again, I was just saying that's not a message that my students are taking on board. Yeah, so certainly if you, you know, you're saying that they use RT to document how they fixed it if you actually kind of emphasize you should have a playbook for this, write your playbook and then reference in RT how you used it, then you might find that a lot better. Yeah, that's a very good idea. And again, it's certainly something I talk about. I just, I don't think I'm being heard yet. Anything else? These are some good ideas though, I like that. This is, I went, you have an idea. All right. What I've made my students do is for each thing we do, go around the room and it's your turn, it's your turn, it's your turn. You're the guy writing the documentation for this step this time. And then social pressure occurs. And if you do a bad job and you have to hand it out to all the other students and all the other students will see the bad job you did. Yeah. And then just stand back and let social pressure occur. Hmm. Yeah. The tough thing, I don't know how your students are but mine social pressure often works against my goals not toward them. But oh well. Okay. But so yeah, there's a problem. I do appreciate, oh yes, another one. Just one small addition to the suggestion before about having a final exam. What you could do is do it in a lab condition where they do not have the internet and only have their own wikis. Yeah. And they have to go from there. I like that one too. Yeah. Yeah. But then you could also maybe lock off the internal documentation for the different products as well. So you can't get the Nagios read me or anything else. So they're literally on their own. Yeah, these are all good. But that's just me being a little bit statistic. I was gonna say I hope somebody's writing these down but somebody's recording them so even better. Okay. They just don't run in touch with them. Yeah. No, this is, I mean this is all great and this is exactly why I'm here. Yeah, I do love to talk but I really actually like to get good ideas and I'm getting them. So thank you very much for that and hopefully we'll get some more of that going on. A couple other notes. Some interesting things that I observed about student performance. So it's just like somebody when we were having mic issues, somebody said well is this their first systems administration course? They are required to take a more elementary Linux course earlier on in the curriculum. So it isn't their first time. But in general my observation is the typical student is not entering this class with the level of Linux proficiency that I would like to see. Now I think there are two possible sort of expertly, there are two possible solutions there. And one of them is to say well we need to integrate more Linux earlier in the curriculum so they're more familiar with using it before they get into this class where they're using it under a considerable amount of pressure. And that sounds like a lovely and well reasoned thing and I'm actually skeptical that it will work. And in fact, I mean I do think that we should integrate more links into the curriculum and those sort of things but there are a lot of issues around that. And quite frankly the reason I don't think it will work is that I think another solution is that maybe I actually need to lower my standards. And I don't really mean that in a pessimistic way. I mean one of the things that we often lose sight of. So I don't know, I've been doing this stuff for a long time and you have to stay in touch with. I mean the way that educators like to state this is you have to meet your students where they are and maybe where they're gonna be is they're not gonna be at that level of Linux proficiency. For what it's worth their Windows proficiency isn't anything to write home about either. So it's not just the Linux thing. And really maybe the thing is to say look at this stage in your education you're not gonna be that proficient and I need to incorporate it into my plans. And probably the solution is a balance of those two things. I need to have realistic standards and I need to work to make sure that those realistic standards are being reinforced throughout the curriculum. I think that's probably true. Another bit there, so interesting thing that happened and I've run this class twice, 2013, 2014. And 2014 groups set up a Facebook group to share some information. It was that they didn't tell me they set up a Facebook group. They didn't invite me to look at it. And if they had I would not have done so sort of a, what is that, a Heisenberg-esque sort of way I would have been concerned that by observing the thing I would have changed it. And I didn't want to do that. And in part the goal is great. That is they're collaborating and they're sharing information and the thing that I'm telling my students all the time is you need to collaborate and share information. So great, they're doing that and that's fine. The trick there was that some of the collaborating and sharing information might be, ah, looks like Tom just killed by MySQLD. You might want to go look at your server and see if he's done the same thing. So in the way that I was introducing problems I think that I have been too predictable. Easily defeated by this Facebook sort of approach. Facebook, whatever happened, I or C. But whatever. So that's fine. The solution there is not that I think the students have changed because I think they're doing the right thing. It's that I need to change and be more unpredictable. And so we'll talk about that a little bit. And finally, students seem to enjoy the class. Now, I mean, generally when I teach the goal is for me to enjoy the class. And I did too, but students did enjoy it. And no, I mean, that's, that really is a useful measure. And I'll tell you one reason why it's useful. And anybody else here who works in the education field maybe tell me whether you see this too. In my degree program, I very much see that the first class destination for students, the good students are supposed to train to be developers. And that system administration has traditionally been seen as the exit path for students who for whatever reason didn't see much future for themselves as a program. I mean, that really happens. I mean, I think that I think I've done some work to try to defeat that image, but I'm still not done with that one. But the fact that students are enjoying the class and they're telling other students that they are enjoying the class means that I'm introducing to the students the idea that systems administration is itself a first class career destination. And some of the students are taking that on board. So if I've had a particular success with this class I'm proud of, it is precisely that. That's something that I'm happy about. Yeah, it also turns out that some of the students who are working said, yeah, they're taking what they learned in the class and putting it to work in the workplace. Which actually tells me two things. One is you're working somewhere and they're not already doing these things. But I guess if they were, I'd be out of a job. So okay. And again, what it shows is that the work that I'm doing seems to be relevant and that sort of thing. So I'm happy with that. So what do we need to do from here? Academic talk, you're supposed to end with a future work slide. It's the obligatory last slide, or second last slide. So I've got two issues here. So one, I said I'm too predictable. So yeah, I've got scripts and things to produce the problem. But the problem is the way that I'm applying it means that I tend to give everybody the same problem scenario at approximately the same time. What I really need to find is some sort of software tool that lets me automate it and schedule these kind of things in a more staggered, less predictable way. A configuration anti-manager, as I call it. And I think that's right because I think the most likely thing is that I can take one of the system management tools that's already out there and adapt it or perhaps pervert it to meet my goals. I think that's a private approach. Maybe I need some modification or maybe I need to write something myself, but I suspect not. And that is the very next thing on my to-do list. That is the thing I will do when I get home. Is that a five or is that a time? That's like two messages up there. Oh, yeah. Now how does this works? There we go. So some of the stuff you talked about, there is a very real need for some of the stuff you're talking about, an information security and infosec. Yeah, yeah. So this could basically be a training, like an introduction course for infosec. And I would maybe suggest that take something where you're more advanced students and take them onto like an infosec 101 and set them loose on your juniors. Ah. I'm gonna reply to that. I know I don't have a microphone and that's fine. I've done exactly that, they're too evil. You can't allow juniors to attack freshmen, it goes bad. Enthusery is evil. Yeah. I think that's what you're saying. And I think there's something in there. And again, that's definitely something I've taken on board. There really is, remember the Disney theme park bit, which is that I want it to be predictable in a way. Predictable, like I want to know they're gonna be presented with the right scenarios. I like that approach, I just need to think about how that fits into this big picture. But that is the very next challenge I don't work with. And then finally, what I want you to do is I want you to think about how that fits into this big picture. But that is the very next challenge I don't work with. And then finally, what I have adopted as a matter of course is that all my teaching materials, as I produce them, they go on GitHub. Both so that I can share them with colleagues and that sort of thing and so that I can foster some open collaboration because that's what we're about. And that sort of thing, for that matter, that's how my students get their course notes. I'm like, you want the course notes, you pull down the repo. And so that's something to go on. Be aware that I follow the open source, I commit early and often, and I don't worry that there are bugs. They will get fixed over time. If you spot an error in my course notes, I like pull requests. So that's what I do. And the last bit of future work is this idea of, yeah, I want to foster that sense of open collaboration. I want to seek out and help build a community of practitioners who are working on the same sorts of things I have. What I've said are these are some of my ideas around core, I'd like to hear other ideas or maybe I'd like to hear people say that my ideas are great. Either way, I'd like to be in the community. And in part, that's what I'm doing here. That's what we're all doing here. And so that's the last bit of future work that I'm working on. Okay, great. There's my lovely questions or feedback. Is there, is there time for that question? Yeah, we have time for a few questions. Okay. One observation, one question. Observation is that we, I think I am incredibly fortunate to have an entire stack of all those things that not only can, do the students get to use free in their lab environment and whatever in the university, but they can take home, they can use on their own systems, they can use elsewhere, they can use in business. Yes. And that is something that I think is an amazing competitive advantage for open source software that universities need to keep pushing because it's all very well for proprietary companies to come in and say, we'll give you this license for free, but usually it comes with a bunch of conditions about where those students can't take it. Yes. The question I have is, did you find any of your students saying that what you were doing by being able to log in as root and do whatever you like was unfair? No, and I think that's a really good question because if they had said so, I could see that point. But no, I got really good feedback and the students appreciated being confronted with these challenges even if I made it easy for myself by making sure I had root on their servers. No, the response from students is very positive. I'm happy with. Apologies if you already covered this. Are your students undergrad or post-grad? These are undergrad, so they're in the third year of their undergrad program. And do you require a programming type introduction in addition to the Linux introduction before it? There's a programming introduction built into the degree program, so I don't need to do anything extra. All students in our degree have taken at least a full year of programming. I'm just curious how much hand-holding you were providing them with things like the Nagio setup. And if you get in the two-hour window, did you sort of suggest that they had to have an on-call roster or anything like that or? Yeah, I explicitly said you need to have an on-call schedule. You need to account for 24 hours a day over that two-week period. So yes, very much so. Hand-holding? Yeah, I mean, the earlier parts of the semester was they're setting on Nagios, but yeah, I mean, I was very much there as a resource to help them see how to do it. And a paper like this in the third year, a lot of it is you take a stab at it and when you start to have trouble, then I'm ready to help you. But I usually like students to try it on their own first and seek help when they decide they need it. So I'm just curious, how much, I don't know if you actually looked at things like bash history to see what methodology they've applied because obviously commenting on tickets to keep track of their own workflow wasn't working as well. How many, did you see many people actually applying problem-solving methodology as opposed to just resorting to a backup restore? Well, a couple things. One is I didn't systematically look at bash history now and then I did just as an actual effective marking thing that doesn't scale. So that's a problem with it. Frankly, problem-solving versus restoring from backup, I mean, both are valid approaches, so I'm not sure. But yeah, basically there's only so much I can do and that kind of really detailed looking at it, I don't think I can do that. I don't think time-wise that works. Even though, yeah, if I could think, if we could come up with a way to automate it, we could do that. Slightly non-technical, but have you started edging into looking at the mental health issues associated with the industry? Mental health issues? Here it is. Well, I do, early on, so remember one of the big themes was the idea of professional practice. And one of the things I'm making clear, one of the ways I phrase it is that they need to carve out a space for them to say no, to push back and say, you need to accomplish a reasonable kind of balance in your work. So that is something that we talk about in the semester. Yeah, and I think it is important. Okay, so we have time for one more question. And, sorry? No. Okay. Is yours very long? We might be able to fit it in. Okay. That's great. The way I automate the breaking of students' stuff and the way I add unpredictability is I make a list of 10 things that could break on your server. I cut them into little pieces of paper. I sort of randomly throw them at students. And I tell each student, you go find somebody who you hate and break their computer in this way. Hmm. And it's random in the sense I'm just throwing them randomly at them. And it's automated in the sense that I don't have to do it, they get to do it. Yeah. I want to see you taste the sign of the 10, right? You do the whole thing. Yeah, there's a point there. All right, we need to wrap up. But I do want to add one last thing in terms of that future work. So I just want to make a little plea at the end. If you are here and in some way represent or if you're a part of a group that employs, say, recent graduates who studied system administration, I really want to talk to you because I work with a lot of bright, you know, highly engaged young people and I would love to see them work. And I'd love to see them get work. They're kind of clueful organizations that send their people to something like LCA. So maybe we can make that one happen. All right, thanks.