 Hey, good afternoon, my name is Evan my company is called point three security and we help organizations Identify cultivate and measure their cyber security talent. I want to thank the I could see organizers for inviting me to speak And I'd like to thank you for the opportunity to share with at least I find to be a pretty fascinating topic Recently there have been there's been increased energy and attention devoted to game fight ecosystems Particularly how to help solve challenges related to recruiting retention training Talent benchmarking and other aspects of workforce development And I'd like to use the next half hour or so to share with you some of the research and attention in the field The talk is entirely ready player one how game of fighting persistence will save us all feel free to interrupt Feel free to heckle. I'm pretty flexible. Okay, so I don't believe that I'm super naive, right? If anyone's not familiar with the Gartner hype cycle you should really familiarize yourselves with it. It's it's a pretty fascinating Mental mental trick So like I've seen us wander into the peak of inflated expectations We're I think a lot of people are placing game of fight ecosystems right now And I know what likely comes next is the trough of disillusionment where everyone just turns against the technology, right? Blockchain and AI were just super in vogue just a few short years ago And now everyone thinks it's not going to do neither of the technologies are useful at all Eventually that'll write itself, right? And I'm seeing just past that slope of enlightenment and well into the plateau of productivity Because if you're going to take away any one thing from today's talk, just know that game of fighting ecosystems are not a fad They increase your team's productivity levels in ways that translate to enterprise level metrics I'm talking about recruiting retention job satisfaction employee confidence response times and close rates all measures of productivity and Game of fight ecosystems are already excelling here in the productivity world where results matter both to your mission and to your organizational culture and some important research studies that we will review today corroborate that marketing hype so Question how are we taught in the u.s. Today and in much of the world we can thank Queen Victoria Mashing a global empire with a seasonal farm and mercantile-based economy sprinkled in with a little industrial revolution It's efficient with rows and rows of empty chairs in the front With rows and rows of students each being told the same information on the same schedule One teacher can cover the needs of dozens of students and thanks to technology The Victorian model can now put one teacher in front of hundreds of students in large lecture halls And you can even stream content online or webcasts to thousands Think about the cost savings to the school houses It's not effective as we all know Information is crammed and then forgotten as soon as that multiple choice test is over the curriculum either teaches to the lowest common Denominator slowing down your top performers or those who fail to keep up wash out either forever Thus wasting time tuition and a seat or in the case of a retake must start all the way back to chapter one When the course starts up again, even if the student passed through those early chapters He or she still has to start a chapter one. It's all or nothing So different question from how are we taught? How do we learn? Since the dawn of time we have gained skills via apprenticeship by a master craftsman imparting knowledge to an apprentice The young Padawan engaged in an authentic experience on the path towards becoming that Jedi and it's very effective Historically though, it has not been very efficient Anyone that could not land a limited slot with that blacksmith or tailor didn't have much in terms of job prospects Recently, however, there has been new study into this and if we can define a model for apprenticeship Perhaps there are ways to adapt it to the modern workforce So Vanessa Pasdenin from Florida State authored a paper defining the critical aspects of cognitive apprenticeship into six areas In summary distills basically into an experience of watch me do it now copy me now do something close but not identical No, let's talk about and this process is repeated until mistakes aren't made and the student surpasses the sensei So let me take a quick three slide interlude to explain how I got immersed in the world of gamified learning So my company point three was brought into the Pentagon a few years ago to help them with something They called the cyber operations Academy course for co-act The purpose of the co-act was to offer a schoolhouse using this cognitive apprenticeship model instead of the Victorian style schoolhouse currently in production for the cyber workforce today So to facilitate this we built a learning management ecosystem to track student progress and provide access to a cyber range housing very very carefully constructed Authentic and modern hands-on challenges in a variety of topics reverse engineering Vulnerability research network forensics post-based countermeasures instant response automation and so on so right the skills Essentially that are required to be relevant in today's cyber security workforce All right, so the content on the slide is lifted from a talk at this year's mod sim world given by dr. Shane Gallagher Dr. Gallagher's talk was titled games the solution to build the cyber operations workforce Mod sim conference. That's modeling and simulation conference Is for real nerdy data science types full disclosure? I'm not a data scientist But when I look at these graphs Clearly something is happening. So How do we cleanse? the Institute for Defense Analysis Booz Allen Hamilton and the advanced distributed learning initiative observed this co-act course and these charts are in a unit of measurement called Cohen's D that's a zero to two base scale on Mathematical derivatives from data at specific intervals essentially it's used to measure learning on the left You see a before and after of our students' ability to perform applied skills on practical hands-on assessments Now some of those assessments were point three So there is some bias in what you're seeing on the left, but many of the benchmarks were not We utilize Carnegie Mellon cyber states live and Pico area securities capture the packet Circumstances project areas the Department of Energy's cyber fire and the industry's OSCP certification is independent benchmarks of learning a Normal college course produces a zero point five change in Cohen's D and an effective college course produces a zero point eight change Coax students on average experience gains of one point three one Cohen's D And on the right you see those who have gone through a cognitive apprenticeship style of learning compared against their Victorian style counterparts So three years later. We have 75 students have gone through this program We're looking at north of 90% mission-affiliated placement within the community our former students have retained what they have learned They use what they learn in their professions and their outperforming peers that have not experienced the cognitive apprenticeship Coac-related personnel have gone on to win or placed in national capital flags to include Raytheon's Game of Pwns Sands is net wars deaf con the cyber a3 name a few and so here's the cool part If you think about classic apprentices those blacksmiths those tailors The journeyman to apprentice ratio was like one to one maybe one to a few we have 75 alumni So how did we scale something that historically has not been efficient? Formalized education theory the study of learning models not really my jam I used to work in the intelligence community as a computer network operator So it turns out like we by accident push the students through a game of five learning environment And I'd not heard of such a thing Turns out these are all rage in academic circles these days. So what makes a game? on the screen is the Definition of a game defined by two MIT researchers in a paper called rules of play design fundamentals If you parse the definition you end up with four key aspects of a game features mechanics Dynamics and aesthetics Features evoke a particular gameplay, right? This is your raw emotion Anything that contributes to problem-solving ability or two executive functions Mechanics are how you motivate and invoke engagements. So examples include player feedback goals badges leveling competing community points Dynamics pertain to that autodidactic behavior the drive to complete on one's own whether the game is solo competitive or collaborative in nature And aesthetics is that sense of immersion the degree to which one becomes part of the simulated world And it turns out that games work This image was lifted from a paper written by Simone Coombe with the coolest title ever It was called playing Super Mario induces structural brain plasticity So if you check out the fmri on the screen What the image shows you is that one's gray matter volume is affected by gameplay So I'll say that again, right your brain literally Physically changes composition in areas related to spatial navigation strategic planning working memory and motor performance The games have a persistent effect on researchers are now calling fluid intelligence and non IQ form of measuring potential That is the ability to analyze novel problems identify patterns and relationships and perform inductive and deductive logical reasoning processes This problem-solving ability increased by gameplay is of prime interest to cybersecurity employers Employers such as the US Department of Defense So the general cybersecurity and information systems recently published an article called learning cyber operations through gaming an overview of current and Upcoming gamified learning environments and they profile for gamified learning environments in use by the military today Full disclosure point three's escalate platform is one of the four profiled While as a small business owner, I totally appreciate the exposure. That's that's not why I cited here Like this really is a key document It was authored by those responsible for readiness and preparedness for the military and the article cites evolving strategies on how to build a pipeline of talent The DVD has to experiment with many approaches and the article has just candidness about the sense of immediacy of addressing the problem The current style of Victorian model learning in school houses It's just it's simply not working that Victorian style takes longer to produce qualified personnel and is more costly So reading explicitly out of the journal article quote gamified learning environments are powerful tools for engaged and motivated learning experiences They can be an essential component of the strategic and comprehensive cyber security training strategy, which is a national imperative and These environments will help produce qualified personnel to meet the demand signal and This gets to that point where we have this like skills gap that we keep hearing about right recruiting and upskilling assessing and retaining These are big issues with the military right the military doesn't often get like Harvard grads and certain unover-generous salaries But the military must staff cyber security positions and do so with individuals capable of becoming experts in a very short amount of time And one of the best ways that has been identified to ensure this is by a gamified learning and industry players agree as well They have the same challenges talent identification cultivation measurement retention at the RSA conference last year Grant Borsakus then the CISO of McAfee Let a series of workshops and presentations around a commission study about gamification solutions Pulling managers and technicians only from organizations with 500 employees or more. We have some data. This is fairly significant Small companies are traditionally at right like large companies and organizations take a while to change So the study is only limited to those large firms that are typically not agile leaders with response to policy shifts So what do those large companies say? one There is a clear and strong correlation between gaming and cyber security activities Gamers and incident responders and sock analysts it all overlaps They all continuously look for clues develop perseverance work logically based on observation They don't threats Those pulled site the mix of gaming and automation as directly reducing threat detection times 57% report that games increase awareness and knowledge of how breaches occur You cannot be a defender if you don't know how the bad guys are breaking in and 77% of senior managers agree that their organization will be safer if their organizations leveraged more gamification 77% of matters to There's a clear correlation between gaming and retention Responded say gamification forces a teamwork culture Respondents who are extremely satisfied with their jobs work for organizations that run games or competitions throughout the year multiple times and Satisfied employees are more productive the study sites 11 hours for happy employees to identify a breach While dissatisfied employees take 23 hours to identify similar breaches and a lot as you know can happen in a 12-hour difference And three there's a clear correlation between gaming and recruitment Three-quarters of senior managers of large firms consider hiring a gamer even if that person has no cyber security experience Again, you can train the technical aspects using internal workforce development programs, but it's harder to train passion and drive So at those companies who utilize gaming solutions You're seeing happier employees the time-to-id threats decreases turnover decreases and employees are more likely to say positive things about their employer And the small and mid-sized business market is getting involved too the next few slides are derived from a LinkedIn work product Based on a 4,000 person interview So executives are tremendously afraid of the looming skills gap as many in this room know You study new hotness and like boom. It's old and busted just a few weeks later So there's limited utility in investing in a specific training on a singular technology Rather the number one concern cited is soft skills problem-solving organized and collaborating those indirect elements One gets from a gamified approach to talent cultivation the number one concern The second core message from hold executives is to meet your employees where they already are and we see this all the time We talked to prospects and it's the technicians who tell us about the big training systems that the senior execs bought that could teach you everything from Python and cooking to a foreign language currency trading right but no one uses any of that because Nobody wants to use it. It's not interactive. It's a bunch of videos It's CBT which has a terrible stigma We hear that videos in the browser is not as compelling as in our industry pulling down old capture the flag images online and going home to Tinker so it's nice to see some realization from management and that order room that Gamified learning is the way that you know the direction the company should take And ROI the study asked executives What would be the measure of success of a training program the top three responses? One talent retention. This is unsurprising one of the greatest costs to any organization is in recruiting onboarding and ramping up personnel Smart leaders will do whatever they can to reduce turnover To an increase in productivity, right? This is also fairly unsurprising You measure productivity before the training measure productivity after the training if the gains outweigh the cost There's your ROI right check out success measure number three employee feedback So is the training applicable is it compelling would you do this if you weren't forced to do it from management? That's a measurement of culture right our employees part of the career progression Conversation and done right this can be very powerful And that's backed up in the same study which also called employees not just employers 94% of employees say they would stay in their organization longer if it invested in their career development That is an incredibly high response rate your employees want to be loyal and productive. They want this to be a two-way street So consider the next question Anyone in the audience have any thoughts as to how most employees will respond to that question What would lead you to spend more time learning? I know what's that? Okay, no one else. Why would you spend time learning? passion Cool, I like it and now I'm stuck because somebody always in the audience screams money and none of you did so now So it's not money right a lot of times I've done speech a couple of times and often I'll do the training if it's gonna give me a payback, right? That's not a thing right so less than half of the respondents cited incentives or promotions as the reason to endure training the number one reason cited Was it was part of the organization's culture to learn and grow and progress people want to feel value They want to interact with their learning environments and not simply consume boilerplate mandatory videos and take compulsory mandatory multiple choice tests So what does gamified cybersecurity look like for you and your organization? How can you get involved? Broadly speaking, I believe there are currently five viable market offerings for gamified solutions Really you need to do some soul-searching So whether you want your gamification strategy open to the community or limited to key internal stakeholders This largely depends on how you budget for this is it training human resources recruiting marketing There are many ways to implement the gamified strategy that suits your needs and budget So the first is bug bounties right these can be internal or external Barely simple in concept You invite people to break your product if they succeed you serve them with a prize and not with a cease and desist And part of to be proprietary tech Custom web app and mobile app. What are you? Some are by invite only to key employees Some are broad appeals to a global community of researchers and some are in the middle We partner with a really cool company called SINAC. I know they've done stuff like hack the Pentagon Some other military-based exercises Crowd source this right have a global pool of talent that can look at your products for you and help you out Hackathons are also internal or community-facing You give participants some resources come out of your hardware maybe a license to your thing for a day or the weekend You state a general problem Teams work to solve the problem and prizes can be awarded based on novelty feasibility or commercial viability And since you what you're doing here is leveraging the gig economy and you're really only paying people for their success And so a little bit, you know if he but right like it has benefits Participants not just those winning participants Particularly as employers start steering their human resources departments to focus on skills over knowledge We've seen hackathon participants receive job offers just for playing At best and at minimum just a resume boost Capture the flags. So we've hosted these intercompany red team versus blue team east coast versus west coast What have you we've done them as community-driven events, you know Battle of Montgomery or my school cyber club is better than your school cyber club, right? They're really good for social engagement. This is a pure community building play You can ID talent before you recruit you can do a post event debrief on winning strategies and broad benchmarking Capture the flags matter they matter to the hosts and they matter to the participants Continuous learning environments is the fourth viable option So this in my opinion has the biggest growth area Employers hate paying for certifications Right, your employee is gone for a week or two weeks doing nothing for you. They come back with a cert And I think we're looking for a new employer, right? There are smarter ways to engage your employees and nurture their individual career growth Companies can can tailor continuous learning environments to create incentives for their employee for employees We have one customer right now using a gamified ecosystem for job titling right passing these x challenges and your title as junior Solve these y more and now you're a senior if you solve z you're a master operator, right? Prizes are awarded at each level, right? We've seen companies pull employees out of help desk or out of finance and into cyber security roles So a gamified ecosystem works well for all stakeholders employees gain skills And grow which makes everyone happy and productive And the last market offering is tabletop exercise, right? These are often internal only functions Usually with very limited participation your senior level execs Maybe a manager or two from your IT or cyber security shops Goal of the tabletop is to see how your company responds to stress conditions It's scenario driven, right? How would we respond to ransomware if our comms go down or if some proprietary databases out on a paceman site? They're run like a fire drill Don't panic We have a three ring binder with our contact for our pr firm our data backup solution You know our lawyer if applicable like we've got this right? That's the purpose of the tabletop So interlude number two IBM coined the term new collar in op-ed Probably something five ten years ago. They went on a huge media blitz And have since authored a series of fascinating white papers Specific to cyber security and IT in general And in short as many of you know, there's a reported skills gap hundreds of thousands of vacant jobs And the problem is only expected to get worse The non-profit ic squared organization claims the problem can be as many as three million unfilled jobs in the coming years And in my view there appears to be a huge disconnect between what companies need How human resources recruits that talent and how academia prepares that talent New collar fills jobs where academia is not supplying with talent as is IBM did a huge breakdown comparing professionals who have college degrees professionals who have technical certs and professionals who don't buy it And you want to take a guess about the correlation? It's weak doesn't exist shocking. I know but it is possible To perform technical work without certifications or without degrees It's possible to be productive to employers without certifications and degrees It's possible to have a work ethic and be able to participate as a team player without certifications and without degrees And IBM now claims that 20 percent of its cyber security personnel hired in the last few years fall under new collar That is they entered those positions by non-traditional hiring strategies They were tapped for their interest and then upskilled internally for IBM's cyber security shop demonstrates the talent trumps the ink on one's resume And so I have one final question Why is there a skills gap? Why can't organizations identify, cultivate, measure and retain cyber security talent? I believe the answer is that there is not a skills gap. There's a gap in hiring and placement practices With gamified ecosystems. We are seeing the following One you already have talent within your personnel gene pool You simply need help identifying the key attributes of likely performers So you can reassign them into cyber roles where they can learn the technical aspects and trade craft and thrive for you on mission two or You can leverage the gig economy by bug bounties and hackathons engage the community only pay for results It's pretty compelling, right? We're seeing traction here again with hack the Pentagon and other out firms that are similar in nature Or three and probably most importantly Your team may be completely overlooking entire populations of incredible talent Simply because those individuals do not possess technical certifications I was at the NSA as an operator for several years. I was a team lead I didn't have any technical certs and wasn't a discriminator for me then and I don't understand why it's become a discriminator now in all cases A gamified ecosystem is able to help your organization attain the readiness levels for your critical missions And that is why gamified ecosystems will save us all Thank you Talks that I mentioned are on the screen. I see some people taking pictures of slides now It's probably a good time if you're interested in the sciencey aspects Um, and if not, you can just get in contact with me and I'm happy to get you a copy of this brief I see some cameras up And then this is my contact information so I try to make the phone large enough, but if Can't see it just Comes to the high behalf of the briefing. So we got some time. Anyone have any questions? Very cool. Yes, sir Do you know if the DOD is already planning on moving to more gamified environments for training or anything along those lines? So I I can't speak on behalf of the DOD. So the question is Do I know if DOD is Moving towards a gamified approach? I can't speak on behalf of the DOD as a business owner and one of a small handful of of gamified providers and our business is Increasing particularly with with the DOD. So I'm personally seeing a trend higher Um, I know our Other folks in the community in the gamified ecosystem community are also seeing the same kind of trends Thank you. Dan. Yes, sir So yeah, it sounds to me so much of the ideal area concept of moving towards And coming into the basic The basic training The set of cyber When you get to your unit they're gonna Give you a Training Yeah, so Roughly we're talking about like pipeline management. How do we shape? individuals into specific roles so like the way that our company got involved again was You know the Pentagon says like hey point three like we're training all these people We're sending them to like mission operation units and those people aren't necessarily producing What's going on? I'm like, well, what are you doing? I get like, well, we sent them to certified ethical hacker bootcamp security plus bootcamp CI says people can't hate plus bootcamp type plus bootcamp and they got all the things they should be experts now Why can't they do things on computers? I was like, well, I don't know I don't hold any of those certs But I'll tell you if you give me six months with somebody instead of sending them to multiple choice tests over and over again Just let me have your folks do no emails. No meetings. No taskers. No pt Which is positive negative, right? But uh, but let me give them an authentic just jungle gym for hackers, right? Let's break into a couple computers Even if you've never done that we can both of that and it's going to take time because I'm not going to teach you the answer You can teach yourself and it takes time You're going to be frustrated and you're going to think you've gotten nowhere And you're going to have an aha moment come back over the weekend and figure out and move forwards We're going to work on teams around that tests And again, we have that team-based hands-on learning This is the day to show it like it's more effective than learning from a book or from watching a video or from attending a talk and so um It helps right we were able to reassign a couple of individuals I'll one from the Air Force who was a cable dog He pulled cat five through drywall four operating bases and now he's reverse engineering malware That's a better fit for him. It's a better fit for the nation So you can identify that kind of talent When individuals self-identify by putting in the time and it's a gym, right? Like that's what all these solutions are. It's it's a gym, right? Everybody wants muscles. They all want the beach bodies But then they expect you to like spoon feed them the answers and it doesn't work Like you have to invest in yourself and do those push-ups over and over and over again And you hate it while you're doing it till you look in the mirror Game of five learning is no different You hate it while you're doing it because you feel like you're not getting ahead And then you solve something you feel really good. You never forget that you remember your first exploit, right? I wrote that shell code. I did the thing like I didn't have to cheat. I didn't have to copy It's way more effective when the experience is pure And I think that's why you're seeing a trend into game of five learning Yes, sir two questions one is the the crux of your talk talk a little bit more about um creative ways to to train people to a level or to a skill I'm wondering if you have experience with or you're familiar with research that deals with more like medic cognition, you know education learning So what did you do with that? Okay, so the first question is uh I'm probably not going to summarize this right but What is my familiarity with medicognitivity and other ways of looking at learning? Okay, so I can very positively say I know nothing about that. Um, dr. Gallagher who I referenced in the talk He is all about that about, you know, did you get the right answer? Did you know before you hit the submit button that the answer was going to be correct? How confident were you? Are you guessing? Are you sure like that kind of stuff? He would be fantastic contact for you for for that kind of stuff. I don't want to think about that My second question is Say hypothetically you work in an organization that's responsible for The online education of every Air Force officer Okay, and you are thinking very critically about how to incorporate gamification into capstone level of dense at Captain major lieutenant colonel level learning Where do you begin? Okay, so the question is How can you leverage a gamified ecosystem to create internal classes of seniority? No, it's way more. I guess at different levels of learning or different Seasons of leadership How do you even begin to think about gamifying a capstone education experience? Yes, it's hard to do a capstone if you don't know what the learning objectives are for the thing that led up to the capstone Um, I'll say like one of the strategies that we as point three use in our ecosystem is the the user experience is not linear So um anyone I always akin to netflix like you log in sometimes you're in the mood for like horror or romantic comedy Sometimes you want to do like now reverse engineering and never forensics. I don't know whatever Since any individual can choose his or her own learning pathway What that does is as like a mission commander you just have your junior people go do junior level stuff Your senior level people go to senior level stuff your officers go to officer level stuff or whatever your warrants do warrant stuff And then from a managerial perspective you can see and track Who's doing what how is that individual progressing and how does that individual or Team units benchmark against other team units that have gone through similar experiences So it the way that I would do it Like we map everything against the nice 2.0 framework And so what I would do is I have to try to figure out like what what does success look like for your particular program And then let's work backwards and figure out one of the challenges that it takes to get someone there And then again, how does how does that team compare against similar individuals under similar situations? Any other questions?