 Alright, good afternoon ladies and gentlemen. I am Captain Nathan Flack, current student at the Air Force Institute of Technology. I'm doing research on multi-domain operations education. So I'm here as a research student collecting data on ECSC but also have the chance to present some of the research that I'm doing and to get feedback from you all. So this will be a pretty informal presentation and I invite you to throw out questions or comments as we go here. And really what I want to get at is how are we thinking about teaching multi-domain operations? What does that look like? And what ideas are out there for really engaging people on this topic? So who here knows about Ender's Game? Yeah, got lots of hands. Okay, so basic story, Ender's Game, he uses the military at the time, they use games to teach their students almost exclusively. They're in a classroom environment, in a military training environment, but they're using games. So what does that mean for the military? It was interesting, this article if you go out and look it up, Chip's article, it talks about the military strategy of Ender's Game, and back in the 90s they would have Marine lieutenants and captains read Ender's Game to pull out some of those military strategy type of things. And this is actually something the military has been doing for a long time. We've been using games to train people. So here's, if you can see, here's some airplane spotter cards. So this is not the World War II edition, but they would distribute these cards in World War II. They were just basic plane cards, but on them were the outlines of airplanes. So you would, as you were playing, you would learn what the Allied and Axis airplanes looked like from the front, from the side, from the back. So a way to use cards, a game, to train military folks. Also ran across this one recently, this is Art of War, so that book turned into a card game. It's also just a basic deck of cards with your different suits, but it has some military strategies on there. So interesting, we've been using cards to train people. So this talk is about interactive learning tools in Air Force education and training. So I'll kind of explain some of my research, and also talk about a product that we've put together at Hazard. So here's some background. So one thing that we're working on right now at the Air Force Institute of Technology when I got there last August was this thing called Cyber Education Hub. And essentially, this is a cloud developed platform where people at the lowest levels of our Air Force and DOT can contribute to the content that's out there that we're training on. So it's essentially, it's kind of like a YouTube or a Netflix where anybody can contribute and anybody can consume content and creates social networks around what we're learning and what we're training. So that was something that they were focused on a lot when I got there. But one of the things they were looking for is we're going to have this great platform where people are going to be, but what kind of interactive hands-on, as much as you can be hands-on with a computer, but what kind of interactive training and education tools are we going to be able to place on this and integrate into this platform. But at the same time, you have what's going on over at ATC with the Continuum of Learning initiative and they're not going to remember the name right now, but essentially they're learning ecosystem that they're putting together. And they're trying to make this ecosystem that has all of these apps plugged in. As I dug more, there's not a whole lot of apps that they have right now. They're kind of laying the foundations and looking forward to that. But what are we going to put in that? And then at the same time there's a lot of this talk about MDO. One of the big things obviously conferences like this and the Chief of Staff of Air Force and then also on the Army side talking about multi-domain battle, multi-domain operations and multi-domain command and control. So try to build a research project that took all of these things into account. So first of all, when you talk about educating and training on multi-domain operations, we have to define what it is. So I went to a conference in Europe the European Cyber Warfare and Security Conference. These are some of the same slides that I shared there, but I had to get really basic with the research students there is an academic conference about what is multi-domain operations. That's kind of where some of this comes together. But the goal really is to, that the Air Force is looking at, is mastering that multi-domain command and control and have a few quotes that's going to get at that. But kind of, this might be something we look at and say, okay, it's all of these domains connected together. But in fact it really turns into all of these domains connected to every other domain. And that's where you get a lot of the complexity. So when we think about training and education, how do we simplify this for students so that they can start to grasp these concepts? I'm going to watch that video. Alright, so when Chief of Staff, General Goldfein came into office he sent out a focus area, he sent out three focus areas. The third one was multi-domain command control. And these were some of the quotes that he gave. So the ones involved there. So he's talking about military capabilities working in concert across all domains. And he really padded home, we need a new way of thinking and a new way of training. Then he followed that up with the MDC2 implementation plan that was signed by three top leaders. And it really focused on interoperability but also hands-on training and education. Giving everybody, giving as many people in the Air Force hands-on learning in multi-domain operations, multi-domain command control. So trying to bring all that together, what are some of the challenges for multi-domain operations in educating an entire force in it? So three, these six things, I'm sure you all could probably come up with more or come up with this exact same list. There's a lot of complexity to it and then things are constantly changing. You're trying to take cultures and knowledge across different domains as well as different services and try to integrate that together. And obviously General Goldfein's been talking about this a lot but he really sees it as we don't need a new technology to make this happen. We need a new app. It's going to be the applications that connect all of these shooters and sensors across those domains that are going to make this happen. So thinking about this, where are the gaps today? So I see kind of three gaps that I put into this. So kind of lack of understanding of MDO. At least it's hard to come to a definition of MDO that kind of everybody can agree upon and gather around. Availability of collaborative interactive trading tools is kind of a gap where we're moving towards filling those holes in more areas than just this but that's definitely still a gap. And then personnel are overloaded with training. You know, we've heard death by power point right? So how do you cut through all of the training that's out there to really engage people and get them to think about this topic? So what's the need? We need more learning tools to engage our personnel. We need experiential learning that's going to make people put these ideas and concepts in their own words so that they can understand it and they can express that to others. And then resources that are ready to be integrated. So the big thing about the MDC2 implementation plan was that a tag ATC said, hey, you need to start integrating MDO education into all of your current courses. Now, it didn't say all of them, but it's starting at what we already have out there today, integrate MDO there. So how do you do that? How do you go to an instructor who's already you know, over tasks and doesn't have enough time and say, okay, now integrate MDO. They have to go out and do the learning. They have to read for themselves. So how can we build a tool that somebody can take their classroom in an efficient way? So here's a proposed solution. So to build a serious game to bring these concepts together. So both an experiential learning in the classroom and something that can be integrated by instructors at various levels. Serious games, there's a lot of research out there in the commercial world about them. It's been a growing area of research in the last about 40 years. So they provide hands-on learning. They engage personnel and the build long-term competency is what we've been seeing from the research. So we wanted to take that and build a serious game that would use all the different morphine domains and integrate them together in a way that a student can take and to use in a classroom environment. We wanted to paint the big picture so people start seeing how these little things connect together. Relatively easy to learn in a classroom. As you'll see, probably get the idea that the game we ended up building ended up being very complicated. That's one of the big pushbacks or critiques we get about the game. But I would say I'm very much open to suggestions about how to get after integrating six games together in a game that's not a complicated. If you have any ideas, please let me know. You see the Venn diagram on the right. We're going to spend some time in that. Where do serious games fit in our world today? You'll see the Venn diagram learning, games, and simulations. Games, think of Starcraft 2 or Settlers of Catan or throw out your favorite video game or board game. As I'm going, maybe start to fill in your head examples of what you see out here. Obviously learning is going to be very much classroom instruction. There's a lot of different types of learning, but that's a big one. Simulation and getting into the simulation games a little bit would be like a flight simulator. So you have those that very much are on the game side of that spectrum. Very much on the training real-to-life side of that spectrum. You bring in VR and some of those other capabilities. Here's one that's more towards the training and exercises. Command, modern, air-enable operations, if anyone's heard of that. Very much tries to integrate in a computer game, a real-world military capabilities. They're in the middle between learning and simulation would be more games. BogSat, who's heard of BogSat? I got one bunch of guys sitting on the table. You have a bunch of people in a room thinking of your tabletop exercises that are playing out in a scenario and trying to think of the different actions an enemy or your friendly force could take. Over on the other side between learning and games you have an edutainment. How to do newware that came out in the last 10 years. So I think that we're in the world that's coming to San Diego. You're thinking of edutainment. It's entertaining and also has some real-world learning built into it. Serious games kind of sits there in the middle. I wanted to share Foldit. If you've heard about that, it's a game that's real-world protein folding. Foldit came out in 2008. Three years after its release, players of the online puzzle game Foldit helped decipher the crystal structure of the Mason-Pfizer monkey virus and AIDS-causing virus. Although this solution had troubled medical science for the preceding 15 years, the combined effort of thousands of players produced an accurate model of the enzyme in only 10 days. So they turned this problem that they had of, hey we need to figure out how these proteins and enzymes fold together. I'm not a microbiologist or anything like that. But they built it into a game so they could spread out that the brain power that it took to do this and made it in a fun way and engaging way that people wanted to spend time figuring this out. So we built a collectible card game. And some of the things that kind of got us to the collectible card game was it has the ability, you have different cards in the game we built. One version has 57 cards. This version that's up here has 40 or 54 cards. But you could take one military capability or one capability to distill it down to a card. We've got a picture and some things and I'm going to pass around the cards here in a minute so you can kind of see what they look like. And then you can put some traits, you can put some ideas about how that card interacts with the environment around it. And then you can bring in another card that does the same thing. Distills a capability down into its essential form and kind of outlines a way that interacts with other cards. And then when you keep on adding more that you're building this environment that now you have to think about okay, there's this one capability. How does adding this or how does using this in a way work itself out and affect that environment? And that's why we focused in on this collectible card game. And they're actually very popular. So Earthstone might be one that you've heard of. It's an app or a computer game. Probably can't see the numbers in the app. But 1.5 million reviews and 4.5 stars. UGO is another one that's very popular. It's got 1.7 million reviews and it's almost got at 5 stars. So a lot of popularity. Pokemon might be another one last year. So it would have been 2018. There were 2.1 billion Pokemon cards sold worldwide. So this is a very popular type of game and so we wanted to leverage that and match it up with the military capabilities. So we built the game called Battle Space Next. So I'm going to pass a few of the decks of cards around. So the game was actually the game was built by Colonel Allen Lin who's an instructor at AFFIT. He's since moved on doing some other things. And he was actually gone before I got to AFFIT. But I picked up his game that he developed and tweaked his ruleset and tried to engineer it to be able to work in a classroom environment. And that's where Battle Space Next came from. So it seeks to integrate cards, capabilities for all these different domains. So we included some electronic warfare capabilities as you'll see. Also information operations. And then we focused on the cyber side of the game. And really that cyber integration piece is where I think we get a lot of value out of this game. So this game is coming out of the Center for Cyber Space Research at the Air Force Institute of Technology. So we were really interested in that cyber education side of the game. So what we did is we took a cyber kill chain that's pretty commonly known out there. Trying to remember the Lockheed Martin I believe was the first one to come out with it. But distilled that down into some basic steps. You've got your scanning and reconnaissance that you've got your gaining access and then you have your exploits and attacks. And so we tried to come up with real world capabilities like NMAP and like distributed denial of service, firewalls, rootkits, and try to put those in that construct to help students learn that. Here's some of the lesson objectives that we've put together for this game. And you can kind of read through those. This would be something that an educator as somebody who's doing a class right now is going to be very interested. They're looking at different training resources out there. How can I use that to meet my requirements of what I need to teach my students? I think a big one, and it goes back to game theory, is the anticipating, adapting, and responding to an enemy or to your environment. I think that's a takeaway from the game that we're already seeing in some of the preliminary data we've been able to gather. So really the game is about integrating hands-on MDO training and education into whatever military environment would have a need for that. So this kind of gets into the research side of it. I'll go through these pretty quick just because it's not too relevant to you, relevant to me right now and my advisors and things like that. So trying to get at the efficiency, that's the integration piece into a classroom, and the effectiveness. That's how the students are perceiving it. So if they're engaged, if they're having fun, if they're learning, that's good. That's really good over here. But for the instructor side, does this meet my requirements? Does it fulfill my learning objectives? And how much time is it in energy and effort and money is it going to take to get this thing, whatever it is, might not be a game, and get it into my classroom. So I'm doing a human subjects research experiment. You'll see on the next slide we've run it through several different organizations already. Like I said, I'm here at Air Command and Staff College with their multi-domain operational strategist program, their ENDAS program over there with Dr. Jeff Riley who presented at the conference yesterday really getting his students feedback on the game to improve it and also to see how it's performing in the classroom. So here's some of the different engagements that we've had with the game. Getting some good feedback. I think I've put together probably about four different versions of the game now. I'm guessing the version that's being passed around is probably not the final one. But different versions of the game have gone through different organizations. An exciting one just towards the bottom. The newest career field in the Air Force, at least at the time, was the 13 Oscars, which is the multi-domain command control career field. So these got to go down in their first month of their very first course to their initial skills training and run them through the game and collect some data. They had some fun with it. They had some good critiques. They found some mistakes on the cards that only a 13 Oscar would find. So here's some of the environments that have already been targeted with the game and some other potential ones. So part of this presentation might be, hey, there's this research going on to AFFIT and maybe you want to leverage that to take a picture of your environment. So thinking you might not be in a military classroom but what about unit training? Is there a need for the people that you work with, people that you lead to have MDO experience maybe you bring in a game like this and you can actually it's kind of a fun thing. Just kind of have people go and do it but then in the end bring out the discussion points that are really important to you. So I'm a poor researcher so I'm looking for other environments around this experiment to collect data so if that interests you I'd love to talk afterwards. And here's some of the resources that we're building putting together at AFFIT to make it integratable into other environments. So Instructors Guide is going to give my reading list essentially for MDO and what I've been able to find and put together and leverage from other people along with videos of how to integrate this into a classroom maybe some example debrief discussion questions to have afterwards things like that that's going to help somebody take this and plug and play into their classroom. Here's some of the research efforts that are ongoing. So short term right now we're working on a digital game so taking a card game and putting it into a digital environment that has slowed down over the past couple months unfortunately with changing priorities and things like that but I think there's a lot of potential there for taking this because you're able to reach a lot more people especially as we move more towards the continuum of learning environment. You're going to have people that are sitting at a computer away from other students and this would be a way for them to engage with the game. And then long term I think there's a lot of potential for a game like this and you know I think training on the electromagnetic spectrum and how that influences military operations I think there's a lot there that we haven't explored yet. We get to the cyber piece of that but as General Holmes pointed out there's this other thing over here called EMS that we really need to start figuring out and might it be more the umbrella term for cyber operations so more exploration of that I think a game could be a good way to attack that. I was hoping so maybe it might be you know a year or two year and a half it might depend on the next student class at AFFIT if anybody picks it up and carries the baton from what I've been able to put together I'll show you in the next couple slides the digital game has kind of redesigned it we're putting it together now. We have very rudimentary cards moving around on a screen but that's about it. Also when you're looking at a game you can play an AI engine so you have an actual computer player that you can play against but then also you can have AI on AI there's a really interesting video out there about Starcraft 2 and an AI that they built to play Starcraft 2 and that kind of opens up a world of possibilities when you're looking at military training and how are we helping decision makers and strategists and planners to do this the best way. So here's an initial design of what the digital game could look like so you kind of have your hand down there at the bottom and then you have your playing area where you're playing different cards than other labels of things. It's a little hard to there's a lot going on on the screen for the crowd that doesn't know about the game and how to play. Alright so in conclusion we looked at some of the background, foundation of the game gap analysis and our proposed solution and then we got into how we're going to answer these questions about efficiency and effectiveness and then we launched into future research so I appreciate you all coming out. I'm not going to keep anybody around too much longer because I know this is the last presentation of the day so I'm sure there's some things that you're all looking forward to get into tonight but I've got my email up there, I welcome any feedback now or later on through email and I know some people have been asking about how do we get this game, things like that so I would love to keep a discussion going with you. So I'll open up now for questions. Yes sir? So the card game, the cards are they available to anybody who would like to use them to get them to use content? You do that, how do we? Yeah so we've got like 30 sets of the cards right now that I'm kind of taking around and doing research with so that could be one avenue, we could get interaction with the cards but then also they're PA approved and so we can actually take them to commercial printers and we can work on that so we could work out something where I can get you the card files and then you can take them to whatever printer you would want in order to print those off. We ordered 30 decks of cards, each one has 54 cards and it came out to about $10 a deck, so pretty reasonable if you were looking at ordering a lot first squadron. I guess they alluded with the cards there were some kind of instructions or something to make sure the whole process. Exactly, yeah so it's definitely still in flux because as I'm doing my experiments I'm learning more about actually how to do it but we do, there's like a very complete instruction guide for how to play the game. I can provide you with today if you wanted it. How long did you work with Dr. Riley? So I got to meet Dr. Riley at the MDO Working Group that we had at PathFit and then we kind of had a back and forth about using the game in his classroom. So it's only been probably about five or six months now. Actually before I met him I did some of his articles on Over the Horizon which is a great resource for MDO education. Yes ma'am? Did you do any research on the No, I'm not aware of that. I'm assuming it's modeling and simulation through like the Air Warfare Center or something like that. Good, thank you. It'd be great. That's one thing I'm exploring as I go to now write all this down for a thesis is I think there's a lot of evaluation that has been going on in the war games and simulation and trying to pull the goodness out of it. Anything else? Does the game currently only have American assets? Yeah, good question. So it's all US military assets not because we couldn't do other things. There's kind of a requirement but we'd like to keep that at the young class level and publicly releasable level because that gives us a lot more ways to get in, especially when you think about recruiting, ROTC, some of those environments that could really benefit from a game like this, we want to keep it at that level. But if we're taking it to a different level I think having it publicly releasable you could have some of the assets have a card deck that's actually modeling the enemy. But actually from our senior leaders it's kind of been more like that's kind of an aggressor squadron type of role. We actually want the common person out there to be more thinking about our capabilities and how to leverage those. So that was interesting to hear. Good question. You could take this to the TS level if you want. I build cards with a PowerPoint file. So you take the PowerPoint file, it's got the templates in there, you type up the different things, put the card parameters in there, and you've got a card that's ready to go to the printers. So there would be a lot of possibilities there. You're thinking who's going to print it? I actually, early on in my development I print out paper copies, cut them up, do my little arts and crafts thing. And then they make the gaming world is great. They make these little cards and you just tuck them in there and you can play with the card sleeves. So there's nothing commercial printing that doesn't stop us from using this in an environment that might not be publicly releasable. As far as ideal audience that you would do some research with your recommended group? Yeah, so I've done RTC cadets, so sophomores and juniors in college all the way up to multi-domain strategies and the 13 Oscar career fielding. So that's pretty wide range. I think as we're seeing the game part of the research is to, okay, where's it be the best place? I think RTC, OTS, and maybe SOS the junior captain's kind of level because you start, you want people to kind of think of the big picture. And anybody could take the cards and kind of look through the cards and it would be a learning experience for them but who would actually be able to engage with the strategy part of the game, which is that higher level learning on the blooms pyramid. Yes, sir? Have you ever thought about considering coming down to the 39 O.S. schoolhouse where we teach IQT for all of the systems in Syro? Yeah, so haven't made a specific connection with that, but there's actually been some connections with Keesley Air Force Base and some of the training squadrons there. So later on in the next quarter, hopefully going down to undergraduate cyber training and then some of the other cyber tech schools for some of their training squadrons to run and experiment with them. So that's on the radar. But I'm willing to work with anybody to try to hit that. Training environments are going to be a little different, right? So I would say that the tool is more engaged towards an education type of thing where you're talking about the big ideas, the big picture. When you're thinking about training, you want very tight restrictions on whatever tool. So I think there could be possibility of taking the template of the card games and things like that in the rule set and applying it to a training environment but with different cards in a way that's going to focus in on one or two learning objectives. So there's definitely more discussion there. And whenever you're thinking about trying to fit the whole world of video into a card game, there's going to be some complexities that just get lost that aren't going to, you know, a trainer's going to look at that and might not be that valuable to them. How long does the game last? Yeah, so the ideal would be about 30 minutes. So I had some play testers that were looking at it a couple weeks ago and seriously these guys played for three and a half hours going on. Like guys, it's not really what it's meant to do. So, you know, that was a concern. So the experiments, I say, four hours of interaction with the card game to really get a handle on what's going on. And that's from very first opening the box, reading the instructions all the way through to playing a few games and then doing the debrief and discussion part of that because that's such an important part of these serious games. You know, just play it and then everybody goes home. You know, talk about the experiences that the different players had and try to draw out the real world implications of those things. So that debrief discussion on the back end is a very important part, especially when thinking about a classroom. So that, yeah, that four hours. Is it only 1v1 or can you make a multiplayer? It is 1v1. The way I do is when people are first learning it, they have a strategy partner that they're going to look at the cards together with them, think through it together, bounce different strategies off of one another and then play a demo game where they play each other and they, you know, see how their strategy fares kind of thing. And then they go and play the opponent, which is just another person in the classroom. So probably for future research, but in the mid 90s, I got to go and check out the Virginia Tech ROTC unit and they were, the colonel that was ahead of that unit that created a fight simulator, you know, aggressor versus Air Force in your red blue. And then built an Excel based POM process so you had to POM all your war fighting assets for that you were going to take into the fight. So playing maybe more off the 1v1, but you actually didn't have to build your units that you go into that strategy. Yeah, that's good. Something for future researcher for the next thesis. That sounds very similar to something that a 13 Oscar instructors pointed out. It's like, hey, we could use, like really use this for planning and like even authorization of different cyber, you know, weapon systems and things like that. Where now you're having to go and make a case to whoever used the card in your games. So that's a little more long term, which could be a good application of this. You give a deck to all the students on day one and then you use it for this learning objective, you know, the second week and this learning objective, the fourth week. And then, you know, in the end you build up to a complete strategy. It's very similar to what Dr. Riley is doing. He says, okay, we're just going to focus on ground. We're just going to focus on maritime and then they're building the students up to bring it all together. Anything else? Alright.