 And we're going to kick it right off with Rob Olson. Howdy. So if you're here for duckies in the middle, sorry. I wanted to see that one, too. That was on my list. So instead, we're going to talk about cybersecurity education standards, which are still kind of a cluster, right? I mean, yeah. So a little bit of background about myself. I'm currently a lecturer at RIT. I wear a bunch of different hats there. I teach two to three classes a semester, depending on what else I've got going on. I've been active in curriculum development for quite a few years, both at RIT and in previous lives, previous institutions. And at the moment, I'm incoming as the head of our undergraduate curriculum committee and as our undergraduate program coordinator, which puts me kind of in an interesting position to see what's going on with a lot of the cybersecurity education standards. And there's a lot out there. So before we start comparing them, we need to do a little bit of sort of how the sausage gets made type. It's messy. Like I think the sausage process is cleaner. So what we see is a number of different agencies, a number of different groups involved in the creation of these standards. ABET, and I'm blanking on what ABET stands for, but ABET is the main accrediting agency for the computing discipline. They have traditionally accredited computer science programs, information technology programs, information systems, and now they're starting to kind of dabble in cybersecurity standards. They've been going through a pretty lengthy process. They've been working on this for about two years, I think, trying to build out their own cybersecurity accreditation, which I think is largely intended to compete with NSA, the NSA CAE, that's the Center for Academic Excellence in Cybersecurity, the CAE programs. And NSA has two, CAECD, which is Cyber Defense. And that comes in both a two-year and a four-year variant. We'll be talking about the four-year one mostly here. And there's also cyber operations, which comes in, I think, fundamental and advanced variants. And when I talk about that, I'll be talking about the fundamental one, not the advanced one. Now, accreditations are kind of a buzzword in higher ed. And it's important to understand what that means. So accreditation refers to an external reviewer coming in and looking at your program and saying, this meets some objective standards that somebody besides the institution has put out there. So it's basically a certification, if you will, an independent certification that your program meets some kind of standards. But there are two kinds of accreditation in higher ed. There's accreditation at the institutional level and accreditation at the program level. Accreditation at the program level or at the institutional level, you see organizations such as middle states, which are huge and basically are used to verify that a college is on the up and up. If your institution as a whole doesn't have an accreditation, you should probably be concerned. If your program is not accredited, that's an entirely different thing. There are many institutions that are themselves accredited that have unaccredited programs there. And we'll talk about some numbers for this in a second. But just because a particular program accredited is not accredited does not mean that it is not a high quality program. All it means is that it does not necessarily adhere to a particular standard. NSA has this thing. They don't call themselves an accreditor. They call themselves designators. They call their things designations. I don't know that there's any practical difference, except it's not the keyword, the buzzword, that academia uses. So how does this come about? We start off with ACM curricular recommendations, and nobody has to adhere to these. In some cases, that's, I actually think, not a bad thing. If you look at, for example, the ACM computer science curriculum, at least the 2013 version, I think there's a new draft, but I don't know that it's been finalized. The ACM computer science curriculum only has three to nine instructional hours allocated for security. That's three to nine lectures, one-hour lectures, to learn everything a computer science graduate needs to learn about security. If an organization strictly adhered to that, I think they'd be underserving their students here. So those recommendations get taken into consideration. There's a little bit of influence by industry, and those get turned into ABET requirements. Now, both of these things, and industry input, goes into actually creating a program or modifying a program to meet or adhere to these standards. Largely speaking, it's ABET that is the prime mover here, the main mover. Industry tends not to have a huge voice in this space, which is good sometimes and sometimes not, because industry tends to be very buzzwordy. Like I've heard, for example, of industrial advisory boards asking places to create entire courses on blockchain. It seems like it may be overkill to me. But they do have a little bit of influence, probably not as much as they necessarily should, because academia tends to go the other direction and think entirely in terms of theory. Now, once this program gets made, the name of the game is assessment. And this is how things happen in academia. So ideally, you're supposed to have some kind of academic content, and you develop and revise continually these things called learning outcomes. You develop them, you develop these yourself, for the most part. They oftentimes have to adhere to or closely relate to learning outcomes in these standards. But these are your goals that you set for yourself, and you have to periodically measure them through by developing assessment metrics. You have to collect and report data for these and analyze and so on. You often hear accreditors talk about this idea of closing the loop, and that's quite often because a lot of academic institutions stop at collecting and reporting and never go through the whole revision process. So I think this is a pretty important thing because this is how change gets made in academic content or should get made in academic content. One of the things I think we should do is take a look at how these standards talk about learning outcomes. So I'm gonna have some metrics for comparing these programs. First is adoption rates. This is a little bit of a soft metric because there's a number of confounding variables here. It doesn't take into consideration information such as the prevalence of particular degree programs. For example, there are many, many computer science programs, very few computer security programs. So naturally, we would expect to see more accredited computer science programs. And it doesn't take into account necessarily the difficulty of acquiring a particular designation or accreditation. Some of these accreditation designation processes are multi-year endeavors. It takes a lot of time. I think another thing to look at is required technical content. And this is where you see me sort of borrowing some ideas for how industry talks about certifications. So if you look at industry certifications, there are some that have really a good rap, like OSCP. So I did OSCP, it kicked my ass, and that's how it works. And it was highly, highly technical. You hear, on the other hand, things like CISP, NCEH, et cetera, being knocked around a little bit because they are largely non-technical and people think they ought to be. I'm not gonna delve into whether or not that's the case, but it is a point of data that I think we should use to compare academic programs in relation to security. Are they highly technical security programs or are they more aimed at security analysts, at producing security analysts? So we'll take a look at required technical and non-technical content. And also the kinds of skills that are measured in learning outcomes. So again, these learning outcomes are sort of the lifeblood. What do these standards think, what kind of skills in the security realm do these standards think students ought to come away with? Are they gonna be soft skills such as analyze, discuss, develop? Or are they gonna be harder skills like build, implement, create? So let's start off with adoption rates. The ABET Computer Science Curriculum is by far the most widely adopted at 335 accredited programs. I think this is worldwide. Honestly, ABET is awesome with their data. So kudos to them. You can download, there's like a one-click download for a spreadsheet with all of their accreditation data for every school. It's easily sorted, it's fantastic. By comparison, there were only 50, what, 59 and 55 information technology and information systems programs. And there's zero security, which isn't surprising. The security standards were all, the security accreditation standard that ABET put out, well, I should say ABET slash ACM, the relationship isn't clear there, that they put out, it was a few months ago, that they actually finalized it. So it's not terribly surprising that no one hasn't got because this is probably gonna be at least a year-long process for the first institution to get the first accreditation. NSA's CAECD has about 154 programs. Now, one of the interesting things about this is how this CAECD has changed over time. It used to be that as long as you offered these courses, as a particular set of courses that I'll show in a second, at any institution, or at the institutional level, you got accredited. They have since really tightened that up, and now it has to be within a particular degree or a particular track or a particular program that the set of courses are required. So I suspect this number is on the decline because institutions are finding it harder to tighten up their curriculum to meet that particular requirement. And for the NSA, CACO, we see 20 schools. And when you look at some of the learning outcomes for NSA, CACO, it's not terribly surprising why that's the case. Honestly, some of those learning outcomes are crazy. So, as far as required content, as far as looking at content metrics, I'm gonna talk about is the content required, or is it optional, or is it neither? Okay, is it basically, do students have to see this material, and how much time is dedicated to this material in the curriculum? And feel free to disagree with me on this one because this is entirely a subjective categorization on my part. I kind of ballparked it at about 35, what is it? 35%, in roughly the 35% marks. But there's no way to measure this. This is just me looking at these standards and creating a subjective judgment here, okay? And finally, is this topic related to a learning outcome? Since again, learning outcomes are the thing that drives academia. So, let's start off by looking at curriculum. And here we have the ABET CS curriculum, which I'm sure is too small for everyone to read. But just to give you a very high level view, there is coverage of algorithms, CS theory programming languages, a deep dive into one programming language, exposure to architecture, organization, networking, OS, parallel distributed computing, a required computer-based systems project that looks at multiple layers of abstraction. And that's it. There is no mention of security at all in the ABET CS requirements. There is a ABET-wide, very general security learning outcome that says something along the lines of, the program must cover security in an appropriate way. Yeah, so I hear laughs. I'm right there with you, okay? Yeah, what does appropriate way mean, right? That's up to the institution to define. As a side note, just an interesting anecdote. I got into security because I was teaching in a program that got dinged for not offering either security or ethics. So we created a class called security and ethics. And I was doing AI before this, and I wanted to teach the ethics part, and security ended up being a lot more fun than AI. Okay, so the IT curriculum. Again, there is very little in the way of security here. So we see coverages of interesting topics related to web, information management, systems like technologies, networking, et cetera, system integration, which is nice. We do see system administration here, which is an interesting thing. But we don't see security specifically called out in any way, and just heads up, the information systems curriculum is no different. None of these three curriculums call out security as they relate to the particular discipline. They all rely on that one general learning outcome that says cover security in some appropriate way. So I think ABET realized that they had a problem to their credit. ABET and ACM realized they have a security problem to their credit in their standards. And they've looked at this by coming up with a CSEC curriculum that's entirely separate, which I think going back to that diagram where I said industry has a little bit of influence on everything, but not a lot, I think this is probably a place where industry might, where academia might have benefited from higher industry input, because it seems the industry trend is to add more security into particular parts, basically integrate security more into other things rather than pulling it out and siloing it. This seems a strategic error on ABET's part. But if we look at this breakdown, it really feels a lot like CISP. It feels like academia sat down and looked at everything, all the topics, did a two year study, and came up with CISP, a four year long CISP, which honestly, may not be bad, right? I mean, if everybody knew in-depth everything in the CISP-CVK, I think we'd be a lot better off, particularly if it had hands on lab components too. But for better or worse, this is where we are. Even if you do deep dives into these topics, you'll see it feels very, very CISP-y at every layer of abstraction in the CSEC curriculum here. So NSA, CAECD, by comparison, kind of divides this into two parts, and we see a lot more hard skills, okay? Databases, network defense, programming, intro to crypto, basic scripting. So these are required classes. It seems like a much more technical implementation. So I'm gonna fast forward here. Just, it has a ton of electives to make it very wide, okay? So that a lot of people can get it. CACO has a number of very, very technical subject areas, including low-level programming. One of their learning outcomes is implementing telnet-like application in assembly with no external libraries. And that's what they expect for your student, for your undergraduate to be able to do by the time they're done. To me, that smells of wanting people to be able to make implants. There's some electives here as well. So I'll leave this up just for the slide deck and move through this pretty quickly. So for technical content, programming, is it required? And the answer is the only programs that require, about half of these require it. So ABETCS requires it, CD requires it, CO requires it. Only it's CO and CS, is it significant? And only for CACO and CACD are there learning outcomes. So networking is a little bit better, but again we see learning outcomes pretty much in the CAE check boxes. System administration, it is a required topic. It looks like I have an error. It is a required topic in ABET IT, that no should be a yes, but there's no learning outcome connected to it. It is a requirement in NSA, CECD, not in the CSEC curriculum. It is in the CSEC curriculum, but it's only mentioned. Crypto, similarly we see the yeses at the bottom. We see very few learning outcomes. So for non-technical content, risk. ABET, with the exception of the CSEC, doesn't cover risk. NSA does. Security policy, again we only see this required at the bottom levels, okay? Security management, we see this covered in CSEC 2017 and CACD. Privacy, it's hit in CSEC 2017, CACD, not in CEACO, which I think is an interesting thing. Ethics, again we actually see a lot of yeses, a lot of things here, a lot of yeses for ethics with some caveats, for example, CACO, which again is the NSA's offensive designation, requires significantly more content from the Geneva Conventions than it does ethics. Okay, learning outcomes. We see the soft skills, ABET kind of falls along the soft skill line and the NSA, CACD, falls along the hard skill part of the spectrum, okay? Which I think, and I'm on my last slide, okay? So some conclusions here, I think CSEC 2017 does a better job at hitting core security content, even though it focuses mostly on soft skills. I think CSEC 2017 is an indicator that ABET wants to do more security and that it wants security to be its own discipline, but I think the accreditation is actually structured so that anybody could get it, which is counterintuitive to I think their goal, or counterproductive to their goal. CS, ABET CS is the most prevalent, but it has the fewest core security skills. NSA, CACO pushes technical skills, I think, at the expense of soft skills. And CACD has a lot of content. Very, it hits a lot of security information, but it's very, very broad, okay? So I've got some references here. Again, I'll just leave these up for the video very quickly and that's it, all right?