 Welcome to DefCon 2022. I'm a moderator for this session. My name is Andrey Veselov, and me and Richard Philo are moderating this. This session is live, and there will be a time for your questions. Use the Q&A section, please. And from this moment, I'm giving one to presenter and see you soon. All right. Thanks, everybody. I'm David Duncan. I'm a principal partner solutions architect at Amazon. And my role as a partner solutions architect is to work with our open source platform partners. And I spend a lot of time working on Red Hat in my day job, and then in my volunteer time, and whenever I have opportunity, I work on CentOS and Fedora projects, including the Cloud Six. So we're gonna talk a little bit today about Amazon Game Day and the event engine and how this all comes together to make a gaming experience. And one of the things that is exciting to me is that Game Day was created in the context of the partner team almost from the very beginning. We had a small group of partner solutions architects who were involved in, actually they were just called solutions architects at that point. The Obama for America ops team. And if you don't know, that was Ryan Nall on top of Amazon Web Services and was a super exciting experience. But Amazon.com had what they called a Game Day. And Game Day was this operational stress testing and validations that they did for new hardware or new system setups. And that continues today. That's a part of the sort of the culture of the Amazon experiences. Is there is what they call a Game Day for pretty much anything that is new or has not yet been tested by customers where they just hammer it. But we created a Game Day out of that. And there was some really amazing people. Scott Ward and David Rockamora are two of the people who originally worked on Game Day and just amazing engineers who happen to be part of the partner solution team. And the way Game Day works is we break up into teams of four and or break people up into teams of four. Usually there's a lot of people who are playing the game. And the tenant is that there is no right answer that absolutely there's no way for you to be wrong outside of just not having enough points to win the game. And it's a hands-on experience. Everybody gets to touch keyboard. You get to look at problems, try to solve issues. Some of them code, some of them just operational and determine how you can do that. And the goal is to build communities around the technology, right? It's a really easy way to do that because this is a time when you are put in these groups of usually randomly assigned, right? We try to break as a rule, the goal of the game is to break up groups of people who really know what they're doing and put them with people who really don't or maybe don't have a significant significant amount of experience around a platform. So then you'd have some people who are senior developers but have never touched Amazon and then other people who are maybe junior developers but have a tremendous amount of experience and then all ranges, right? So every functional ability and role from business development managers to product managers, to seasoned developers, seasoned DevOps groups or individuals, so it's flexible. And one of the things that I really love about Game Day is that you have this opportunity to build in whatever can run on top of Amazon infrastructure which includes lots of partner solutions and we'll get into that a little bit later. That's a duplicate. So this is what a Game Day looks like. This is three people from the Game Day advisors they're walking around making sure that everybody who has questions gets their questions answered or if they're looking for hints in as to whether or not they're on the right track. These are the guys who are standing around talking to them and this gentleman in the red shirt is a participant who is asking a question and this is a group of partner development or partner solutions architects who are answering his questions. Now, we've done this in the past where we had some Ansible modules and those Ansible modules fell out of support for the program and this could and has been Red Hat engineers but it could be CentOS engineers or it could be Fedora engineers who wanted to be to participate in building and supporting this type of environment and work. And so here they all are working away in these big teams most of them probably may not have ever seen each other before in their lives, right? And this is why I think this is something that I want our community to participate in the game day. Everyone who is part of this experience comes out of it feeling better than they did before, right? And they're upskilled on related technology. So when we know that people who are having a great experience getting a little trophy with a unicorn on top of it and by the way, these are probably just horse trophies where they taped or glued a toothpick to the front of the horse. That's typically how we do that. Look at the statistic, it says 92% of the people who do this want to do it again, right? And I think that speaks for itself. And this is what a grassroots experience like this gets you, you get a team of folks who are super excited about having just won a plastic trophy with a toothpick on the head of a horse and some stickers. And this thing can happen anywhere. This happens to be in Austin, the one before this that I was showing that happened to be in Las Vegas. These are, this can be a very large event. And people, when they talk about it, they say that it was the most amazing thing that they've ever experienced, right? They helped me feel more comfortable using the console. I want to do this again, right? Is one of the things I love. And the framing device of stepping into a startup to clean up someone else's mess is an entertaining and realistic experience, right? So the way it works is it takes about 30 days for us to set up the infrastructure environment, make sure that we have everything together, that there's no conflicts with any of the other events. And then the setup, basically the event is set up and then once game time arrives, everybody gets to participate. But that's not why we're here. We're here because we want to know how this thing works and how we can participate. So the linchpin of everything is what we call the AWS event engine. And the AWS event engine is a sort of a living space and code base in an Amazon account that provides the framework for all of the things that go on, that these participants interact with. It creates an account structure for each one of the individual participants. So one of the, or not participants, but each one of the individual teams. So each team of each group has an isolated account structure that doesn't have anything to do with anyone else's accounts in the room, right? So we've separated everything out. And then the engine itself will create that kind of a configuration in a way to support game day or now we can support workshops. With that same functionality. So if a group wants to do an immersion, they end up using this as a way to isolate infrastructure and make a sandbox. And then at the end of the day, they clean up the entire event engine. The event engine has the event, right? Has a finite life. And when you close it down, all of the resources are destroyed. A little bit like deleting your account at the end of the day. So stay tuned to this space where all of the workshops that are being created now, they are being done in this 2.0 version. So all of the things that I'm saying are probably going to evolve. I mean, for every system here, there's a next generation and this is it. This is what it looks like. But this one provides a source forage support and makes it possible for us to have publicly available accessible content for the whole AWS experience. And that's something I'm super excited about and why we're here today. This will be more publicly accessible. So an event engine creates for any event an AWS organization and an AWS. So Amazon has account structures for web services. And in that account structure, there is a single account, a single account for an event that is called the central account. The central account has access to all of the team accounts. Each one of the individual team accounts has resources. The central account through cross-account access is able to manage and make modifications to the team resources. And that makes for some fun antics during the game. So there are obviously there's development, but then from the perspective of the game itself, the operator is king, right? Somebody has to go in and set up an event and start all of the work. So an operator is a role where someone sets up a blueprint, they look at what it is that they're playing to do with the teams. There's some very specific information that's associated so that we know what kind of resource limitations to put on each one of the accounts that's created. And then the user gets a little dashboard of their own and once this event is started, right? Once the operator goes back and says, okay, I wanna start this game, then the user, the team gets a dashboard that they have access to that provides them with console access for their temporary account and then credentials for logging into the instances that are pre-created, pre-installed. And the system. So now those users, those participants are participating in the event, right? So there are lots of different types of events. I've put down some suggestions here for the types of events that we might have in the future. And then I've shown that this is made up of modules. So I've also put down maybe some modules that exist and some that maybe should exist. So let's say we had created a module for the Open Data Hub Machine Learning with a game, with some sort of a game angle to it. And we added that to a game day blueprint. The blueprint would then be made up of a group of these modules. Once that blueprint is created, the blueprint sort of aggregates all of these component parts together. And we can assign that blueprint to one or more events. So from the back end, we're able to take the modules and the modules are really what it is that we're focused on just to let you know for the deep dive on code and create the blueprint. The blueprint then can be used in it's, in whatever state it's in, in a specific event. And you can have multiple blueprints that are associated with a single event, meaning that if you've got specific versions of specific pieces that work well together, then you can maintain those together and you can maintain them, maintain the groups of blueprints so that you can use them to create larger or smaller games. So looking back at the team and the components, when we create, when we create a team, it's usually one person for a workshop. You know, the size is usually one person for a workshop but for four people ideally for a game day. And a module has the ability to, well, a module provides a CloudFormation template, we call it the central template. And it's the central account that issues that and generates the, generates a team's resources. A team has an IAM policy and that IAM policy is used to generate roles for the actions that resources can take. And then those resources and the functionality of those resources contributes to your score. So blueprint has, so blueprint for a module would have a read me file for the operator instructions, a team role for IAM policy, pre provisioned infrastructure, which is in the central template. And this is where I'd love for us to be able to create resources like OpenShift Clusters or even rail configurations for different tools, possibly, you know, control hubs for Ansible configurations could all be a part of that. And it's where we have interactive experiences like creating outputs, inputs, stuff like that. Then there are what we call checkpoints in scoring and I'll go into this in detail in support for remote events. So let's start by looking at the read me. Read me's are usually just marked down, super simple configuration. So we create it, we give an overview of what goes on in the module and we talk about exactly what happens in this, you know, in this lab, you know, we're gonna be doing such and such a thing. And then we have more detail and then there's a quality checklist which kind of identifies exactly what's going to go on in the module and how you will understand, how you'll know that it's fully functional. And then there is, you know, there are guides, guidelines for building and developing these workshops. So the policy module is a really big deal. This is kind of the first and the first actionable thing, right? If you have any resources, there has to be a policy that goes with them. And this is, if you're looking at the policy, this is an IAM policy or module policy. I'm sorry, that's backwards on the diagram. So the module IAM policy in this case is one that's very relaxed for S3. Obviously, this is a configuration that's pretty easy. And you can set up, so the thing that makes this great and the thing that makes this easy to do from the outside, right, as someone who wants to participate in the game day experience is that there are these variables that are used in our configurations so that if you have a module policy, module policy can be set up in one account and run in one account and there's a mock environment that we can use to get the experience of doing this as if it was in the event engine. But then we can replace a lot of these things in the CloudFormation template, account information, region information, region specifics, role names, et cetera, with these variables. And then those can be the event engine itself can transform those variables into the content that they need to be. So if you have multiple modules, you have multiple module policies and those IAM policies are aggregated through the event engine. So we consume these modules here into an event. That event is managed and deployed by the event engine and the event engine takes all of these policies, combines them together to create one, well, it doesn't technically combine them together but it applies them and they all have, they all are associated with the team and the team's resources. And basically that gives the access control that a team has inside of the account. And the same thing happens for building those resources. So the collected resources that are built using the central template. And this is a, I showed the policy in JSON but this is a CloudFormation template in YAML. When you look at the configuration of the CloudFormation template, it's setting, this is setting up the parameters for the variable, or for the variable parameters for the template. And the parameters for the template are filled with the variables that are part of the configuration of the game day. So a lot of that is determined by the engine. So the things that are set up in the event determine the values of the event, like the event ID, the generated team IDs. You have to tell it how many teams you want to associate assets bucket for the event. So any assets that are leveraged by a module, they would have to be available through the central account. And so there we usually be a central account, or in a policy to make that central account access work. But all of this again, can be done in the context of a single account structure. And the goal here is to set up all of the, to have all of the resources that are necessary for the game to work preset for at the time that the game starts. And that way people can come in, we set the game up, they enter their IDs and that identifier gets them into the game and off to the races with the resources that are built here. And those resources could be OpenShift Cluster, they could be OKD, they could be Fedora Cloud instances, they could be an entire configuration, an entire solution from whatever open source experience that we wanted to bring to it. And the interact, so don't just stop there, each one of these modules, we have the ability to create these kind of behind the scenes, interactive moments inside of the game. And that can be something as simple as determining a way to pre-populate an necessary bucket in a central account. So suddenly during the game, a necessary bucket has different information than it had before. And if they're, this can be used to kind of level set whether or not they're doing proper ingestion from the location or if there's a trigger or something like that, we can make all of those things happen. And this could be, again, part of any sort of centralized part or solution component of anything we can do in Fedora, Fedora solutions, anything we can do with CentOS as a base, anything we can do with Red Hat as a base. All those are available and something that we could deliver and get an experience where people are learning to use it and getting comfortable using it in this context where they don't have anything to lose. And this interactivity makes it a sort of a living experience. And that kind of comes, falls directly into the next part of this, which is the score. So scoring is really one of the biggest and most important things that you do. And the checkpoint, just to keep this conversation going around the checkpoints, we set those up and make sure that if things are going properly, if at this point we can determine that one of the players or the team has set this up right, we can award some big points. Now, that's a great way to go, but then points and scoring can't just be a one-time thing, right? Can't just be suddenly it came out of the air. Scoring has to come from somewhere all the time. And this is kind of a blurry picture. I'm sorry, I even, I literally cropped this out of one of the pictures from earlier. But what you're looking at there is a list of teams and their scores and those scores are going up and down based on what they've done. And that scoring process is an iteration. It's a Lambda function that goes out and retrieves a list of all the teams and iterates through them. And checks if the team has reached whatever checkpoint it is. And then if the checkpoint's set, it will dispense the points. And the same thing happens for other sort of scoring models. So we can have functions where if a system is responding, as it's supposed to. So let's say we set up a microservice for a customer, in the game we're setting a microservice up for our customers. And there's a way to have the other teams be your customers. So you have a system that is a customer, you have a system that is both a customer of all the other teams and is a provider for all the other teams. There can be a scoring mechanism that is part of the successful completion of a share or the failure of a share. And that's those kinds of adaptive models for scoring are super helpful and very easy to implement in the context of the game. So we have lots of scripts that can be just added in to make that scoring work. If it's not clear what I'm saying, it's I'm kind of alluding to the fact that you could create a simple service that would both send and receive from others of its type and provide a way of making that work and making it score. But then also we can add interactivity and adding that interactivity means setting up a service that will at some point in the life cycle make a change to the system that they're using, right? So we might break the security group rules and then they have an operational problem instead of taking on the functionality of the microservice. The microservice is fine, but something we can deliver a different challenge over time that might cause them to lose points in a way that they did not expect. Okay, so now I feel like we have a pretty good understanding of what it is that we're looking at. We've got three things that I think are really important. One is we have the central template. The central template creates the content or the resources. And the central template is run in the team account to generate that, to pre-populate what it is that the team members will interact with. And that's done with the central template. And that's done in a CloudFormation template, along with a group of scripts. So part of what happens here is resources are established, an S3 bucket is created, that S3 bucket can then include things that happen on those resources that are created or serverless functions or configurations for services. And that can function just the same way in inside of OpenShift as it can on some bespoke pre-existing service. So now I have some gratuitous use of colored slides and this is my call to action. I don't have, there are no Fedora, CentOS, or Red Hat OpenShift based game modules. And so I think that this is a deficit. I really think it would be super fun to know that there were people out there in the world who are getting as just as excited about the community experience as I am and just as excited about being able to work with OpenShift and container-based technologies that we've grown to love over these past few years as much as I do. There ought to be a SIG. And I think that that's really something that we don't have right now that we really should is I think we should be working on some really fun ways for our new users and old users alike to gain insight into the technology that they are interested in and especially the technology that we're building. And then just I wanted to end with a look at what kinds of things, what kinds of events and blueprints are being created and the target groups that are there. And we can take these same kinds of paradigms and build our own event configurations and find ways to engage groups based on that. So we have these blueprints, each one of these blueprints kind of fits to a group, an audience and a duration. There's almost none of these experiences that go less than fewer than four hours. But this basically tells you a little bit about how each one of these game modules works and what it is that they're targeting for the users. And you can see that everybody's getting a lot of introduction to very specific services in this case and very specific content. And some of these are super simple, right? They can be as simple as just I have a website and on that website, if it's working correctly, it will reverse the word that I send to it. And some can be as challenging as trying to determine the specific security vulnerability that is being exploited. This will be a fantastic compliment to OpenSCAP and this is probably the shortest one. I was looking at energy focus. And then I'll end here. But this is some real world discussions. There's a series of YouTube videos out there to show some of the strategy. So there's one where two solutions architects are playing the game and they're actually going through. It's actually a Twitch stream that's been captured. And then there's another really good sort of breakdown of the financial services game day that I think is amazing. And then the last link is just the link to the game day itself. So thanks everyone for paying attention and being a part of this. Super excited about this. And I will say we can take questions now. So Neil asks, what do you think would be some good first modules? I think some really great first modules would just be building out some functionality on top of simple components. And maybe having scaling issues. So finding a way to set something up so that you need to respond faster than one, faster than a constrained pod can provide and then have the team have to figure out what to do. And effectively any basic best practice is gonna be where I would like to go with that. And then let's see, a SIG for building conference games for these technologies and Fedora sounds like an interesting cross between mind share and technical governance. I have, and that's part of what I'm doing here, Neil, is I really think that I'm kind of trying to gauge the interest and generate a little bit more community drive. But yeah, I think it should be more formal. Mark, so how does the scoring work? So scoring works two ways. One is you can have a scoring service touched by whatever it is that you're doing. So you can, you know, your little application. Let's say you have an application that has a reverser in it, right? So I'm running a web, a web application with a reverser. If the reverser doesn't work or if the web service doesn't respond, I can say, okay, well minus five points for, I didn't get there, minus five points for, it didn't reverse my word. Or I can say plus five, I got there. And then plus five, it reversed my word. And that works behind the scenes for us, that works with CloudWatch and we can typically, we typically have a communications agent through the event engine that will go back and filter through all of those scoring modules. And then the other one is the checkpoint. And the checkpoint is also hidden inside of the event engine, but it's basically you say, you identify a specific condition. And usually that's a CloudWatch metric or some sort of something that you can, you can poll or whatever to determine that you've got a successful run. And that can be, that can be done through an interactive experience too. So a customer might be, they might input a hash that was generated from a service that they, that was broken and they had to fix. So it can be interactive from the, with one of the team members inserting it into an input field on a web form or it could be a Reaper that just goes out and determines whether or not the functionality is correct. And each one of those is considered to be what we call a checkpoint. And then there's the interactive scoring. Yeah, so yeah, the OpenScap I think is like, that's one of the ones that I feel like we have a really easy way of putting it together and having people understand best practices around it. One of the scenarios that I can imagine almost immediately there is a customer who is getting a report that their extended update support service version of Red Hat was out of compliance because they were using the wrong SCAP feed, right? Or, you know, same thing, something similar, excellent. Well, I'm glad I could be of service there, right? It's a great, it's been a fabulous friend to me for a lot of time, fighting back people who don't understand the Red Hat and the RPM packaging model for handling security updates. Any other questions? And if there was a repository out there that you could participate in and a testing model, a testing framework that you could participate in, that fit into the model, do you think any of you might have interest? Yeah, yeah, I think that that's a reasonable answer, Neil. I mean, it is a lot more, I think it would be more fun to, I think the fun comes in finding out that you've got something that is super simple and then putting a scoring model to it. Yeah, I think that having the scoring modules and being able to see that you're getting 100 points or five points or whatever for the experience is probably great. And then I meant to run a game day here, but we couldn't negotiate the short amount of time from acceptance all to completion. But I would like to do that. I'll definitely gonna try and get one set up for, I've already, in fact, got it established and ready for Summit and also for devconf.us. For devconf.us. And if it's accepted, then I would love for us to all participate in this game. But I'm gonna ask for a little bit of a commitment. I wanna do at least two hours in the workshop so that we can all get some big points and we can watch somebody walk away with a nice unicorn trophy. That's right, horse with a toothpick. Mark, I've had really positive experiences with the virtual game days. So you can still, that's true, they were pre-COVID times. So the way we do a remote, I didn't actually put this in the slides and I should have, the way we do a remote one is we have, we use the Amazon Chime as the basis and Amazon Chime has a web client that works. Well, with Linux, but there is also a Pigeon plug-in that was made by David Woodhouse that works really well for just basic connectivity. But teams have their own, they basically have just a conference call together like a conference call together and they work through the problems. And that gives you chat and a lot of times chat is even better. What we found over the years is that the people who participate don't necessarily come from the same place. So some people who speak, if the game's in English, a lot of people have English as a second language. So the remote experience turns out to be better for the people who have an English as a second language experience because it's easier to part, it's obviously easier, I think for all of us in our second language to write than it is to recognize the idioms of native speakers. And so the experience is really super positive especially for people who might not have been as included in an onsite experience. So we found that that works out pretty well. The people that I think have the best time are the people who just sit down and enjoy learning. And one of the things that I thought was the biggest success in running a game day was seeing a group where about four of the tables had just abandoned the game but they were all squirreled around one person who really understood what was going on and he was just basically giving lessons, right? And he was teaching everyone about access controls and security configurations and just going through each one of the things that he was doing. And most of the people had just stopped playing the game but they were having a blast just watch it, just standing behind him. And then obviously an entirely different group won the game day but it was, but I felt like the experience that they had where they spent so much time learning was just as good if not better than winning. Anything else? Anyone else? Yeah, the teams are, so we try to make the teams random mark. That doesn't always work out because some people are adamant about sitting together, being part of the same team and there's no reason to stop that, right? I mean, this is not high stakes. We're talking about people who are getting their names taped to a statue, right? We're not talking about, this would not make sense as a high stakes experience, right? This is a learning experience, a hands-on immersive educational experience, not a tournament. Awesome. Yeah, thanks for being here and thanks for participating. Thanks, Radek. And I can tell you right now that part of my world will be making this happen. So I'm super happy to be a point of contact and someone to help you get through all of the detail that's necessary for you to participate successfully. And I really appreciate everybody. I'll leave it there. I think we're at the top of the talk. Thank you, David. Thank you. Thanks to the speaker and thanks to the audience. It was a great presentation. It seems like we finished a bit sooner and it's good. So the next session will be in nine minutes and I think see you soon on the next session.