 So, everyone, my name is Nick Vogelblum, and I'll be your host today, I suppose. I'd like to begin by acknowledging country. I'm on a beautiful Wurundjeri country where summer has finally arrived, and I would like to pay my respect to their elders, past and present, and also to acknowledge all the many Aboriginal lands we're meeting on today, one of these spaces, give us a kind of virtual space that's held, but we're all on unceded land, so thank you for joining us. To any Aboriginal and Torres Strait Islander colleagues with us on the call, we're very welcome, and thank you for joining us. Hopefully, you're in the right space. This is part three of four in our design and evaluation learning sprint from the Australian Evaluation Society, and I'm part of the, I suppose, organising group or committee. I don't think we're a committee. We're a group of people who are interested in it, and if you are interested in joining our group of interested people, please do reach out. But we've been running these sprints for the last, I think this is like the third year we've done them, and we find that bringing in different people who are at that kind of intersection of design and evaluation, whatever that looks like, to kind of share some of the things that they're learning about and the things that are keeping them busy and working on. And we've got a really great speaker for you today. I do encourage you to sign up for tomorrow's session as well. And we've got one more left in the sprint with Luke Craven on systems thinking and design. So that's there as well. These are all free. We give our time for free. Niko's giving us time for free. So, you know, bear with us and hopefully you have some some questions and some some engagement in this. The the kind of, I suppose, the program as it stands is that I'm going to introduce Niko and then Niko's going to give us a presentation and we've I'd love you to gather questions and thoughts as you go through. Please feel free to use the chat to kind of give those questions over or wait until the end of the presentation. And then we'll try and have a bit of a discussion towards the end of the session. So I'm really pleased to introduce a different speaker, I think, for the Australian Evaluation Society. This is Niko King joining us here. He's the co-founder and director of Chaos Theory Games. Hopefully you've read a bit about him anyway by signing up to this. But Niko does all things on AR, VR, mobile games for different organisations. And I think organisations that many of us have worked with this world world food program, DFAT and Samsung as well. So kind of from all different sectors. He's going to talk a bit about. Oh, I don't know what he's going to talk about. He's going to tell us, but but Niko kind of comes in that space between the educational and psychological benefits of games, which I think is a really interesting space for us to be talking about in evaluation as well. So I'm going to stop there and and hand over to you, Niko. I think you've got you should have sharing permission. So go for it if you've got slides and anyone if you've got problems with tech, just reach out to me on the chat. OK. Is that coming through? Awesome. Thank you for the introduction, Nick. And thank you everybody for having me here today. I'm excited to talk to you all about video games and whether or not they're effective tools. My yeah, my background and experience is in starting a game development studio that specialises in creating games that solve real world problems. My my experience isn't in evaluation necessarily. But what I'm hoping to do today is share with you some information about why games are really powerful tools, as well as how we might measure specific data within games and how that can be used to evaluate the efficacy of games themselves and show you a few example projects of what we've done in the past. So starting off a little bit of background into myself. I'm one of the co-founders of Chaos Theory Games. We specialise in creating games that solve real world problems. And our vision is to improve quality of life and inspire a more sustainable future through the magic of play. As Nick mentioned, we've worked with organisations and large companies all across the world. We've created e-learning experiences, games for health, social change projects. So all of them are really focused on some sort of core problem that games can be used to help solve. We also have quite a bit of experience in just the purely entertainment game side of game development, where the collection of our mobile games has been downloaded over 70 million times. And we've found that working in that space, just pure entertainment, we've picked up a lot of tools and a lot of experience that we can bring into the sort of transformational games industry and that can be used to solve real world problems. So there's this really good relationship between those two industries where there's a lot of innovation happening in entertainment games and we can bring that into more social good or education focused projects. Throughout this presentation, I'm going to be using the term transformational games. It's the same term or the same concept as serious games, which a lot of people would be familiar with, or games for good or impact games. It's all the same focus where you're looking to solve a real world problem using a game. The primary purpose lies outside of pure entertainment. We really like the term transformational games as we think it's more descriptive of the sorts of experiences that we create. We're looking to create some sort of transformation within the player or within the broader world. And it's also just a bit more inspirational or motivational within serious games. Serious games can sound a little bit boring. So an introduction into transformational games, I think video games is so broad and so diverse as a field. And it's almost an oversimplification for the amazing technology we have at our fingertips. We can create, we can have experiences where it's a text based narrative adventure where you're exploring the future of climate change or you've got a rocket science simulation where you're looking to learn about rocket science and play around with how to build rockets or you're creating a VR horror experience that's just designed to make people terrified. So what do those look like? Up in the top left is the Financial Times Climate Game and it's a text based narrative adventure where you see the impact of different decisions now and how they play out in the future. Below that is Among Us, which is a social deception game. So you're really just looking to deceive other people and trick them and lie to them. And it's all about that sort of social interaction between the players. Below that is Kerbal Space Program which is a rocket science simulation game that was built as an entertainment product but is also enjoyed by people who work at NASA and has been used in the classroom. So it's a really fun and interesting game that I really enjoy. Candy Crush is up in the top left. That's a mobile game that I'm sure quite a few people would be familiar with, played just for fun. Below that Monopoly, I'm sure everybody's familiar with that one. Interesting story about Monopoly is it was originally designed by a communist who was looking to show the negative impacts of capitalism where one person would just go around the board, get all of the assets and then have all the money at the end of the game. Everybody else ends up with nothing. And it was originally called Landlord and considering that it turned out to be the most successful board game all time and make Hasbro a lot of money. It's a little bit ironic how that one turned out. And below that is Alien Isolation which is an absolutely terrifying virtual reality experience that's just built purely for horror. Then there's two of our games in the middle. Kengizu is a game that we developed for DFAT and it's a wildlife simulation game where you go around and explore Australia's national parks and collect and rescue and rehabilitate animals and then release them back into the wild. Below that is Shamilla, which is a game that we developed for the United Nations Board of Food Program and it was designed to help educate and train their staff how to design better policy and make sure that they're helping people. And so that's a sort of safe and simulated training environment where people are free to explore different ideas and different decisions without the sort of lives of people weighing on those decisions. So it really lets them experiment. So yeah, I think the reason that I wanted to run through all of those is just to show you the sort of broad spectrum of what games can be. And whenever I'm talking about games in this presentation, I'm talking about anything from a text-based adventure through to a simulation, through to a board game or something that's just played for fun. In terms of the different types of transformational games, I just wanted to go through some broad categories as it might help put into context the sorts of experiences that we would typically develop that you might be able to apply to a use case. So educational games is a huge one. We do a lot of work for the education industry. That might be for sort of school students, but it can also be education and training for the workplace and games can be a really good use there. Training simulations, that's huge in the workplace and lets us practice real-world skills in a safe simulated environment without the risk of failure, essentially. Or without the risk of failure impacting things outside of the game environment. Behavior change games is an area where games are particularly well suited for. So whether or not we're building new behaviors or changing old behaviors that are negative or that we wanted to get rid of. Games have a lot of tools in order to form new habits. Mobile games in particular are really good at tapping into our motivation and creating some habitual play of you come back to this app at the same time every day. And that can be good for when the behavior change that you're looking for takes place over a longer period of time. Games for health. This could be games for fitness or personal health but also for rehabilitation or in a medical context. Games are often used in hospitals to instead of pain treatments or an additional pain treatments. In some hospitals when kids are getting band-aids changed they get a game and that's been shown to be a lot more effective at reducing pain and reducing anxiety and stress. When compared to other options. Empathy games. So games that are looking to help develop empathy with another person or another group of people. Games are particularly well suited for this area because of the active nature of play where when we step into a game we experience it in the first person. We say I completed the level or I walked down the corridor. You're not observing somebody else go through that experience. It's really active and you're an active participant in that. So games have the capacity to let us walk a mile in somebody else's shoes and it makes them quite well suited for empathy building and empathy development. Citizen science games. So games can be a really good tool for if you're conducting research and you want to get a good data set. I know of some examples of games that have been used to gather water level data and river systems and then that's been used as part of science. Sorry, part of scientific studies. So yeah, citizen science games is another good category. And then impact games is sort of a patch all if a game is looking to have a broader impact social change games might fall into this category. Games that are looking to address some issues with climate change or saving more energy could be considered an impact game. So when do we use games? For the next section, I just wanted to look at some of the benefits and the challenges of games. So when are games useful and what are the challenges of implementing a game? Games certainly aren't a silver bullet and aren't applicable in every single use case. So hopefully through this section I'll give you a bit more information so you can make an informed decision of is a game right for this particular use case. Games can be really good for boosting engagement. They can translate complex topics into engaging and memorable stories or worlds that kind of immerse us. The human attention span has dropped drastically over the past 20 years because there is so much going on on the internet and there's so many things jumping for our attention. So if we needed to connect with an audience to hold their attention for a longer period of time games could be a good way of doing that. They can motivate behavior change. I touched on this earlier where games have a lot of tools to draw the player in and keep them playing for a long period of time. If anybody's played any mobile games you'll know that you get these notifications every day and if you get in the habit of playing that game every single day then it just becomes part of your routine. So that can be an area where games can be or those tools can be applied to healthcare or fitness or eating the right foods. All those tools can be used for behavior change. The practical application of skills and skills development. So games are great at training problem recognition and problem solving skills as well as increasing social skills or simulating different social situations. There are quite a few studies online of people who play online competitive games and the ways in which games improve those communication skills and make them better leaders. And games also kind of follow the mantra you learn by doing and then you get to apply that learning and see the results of the application of your knowledge and use that to update your hypothesis about how our system works. And that process, that experiential learning process is one of the strengths of games. Prolong of attention. So this is similar to motivating behavior change. If you have a particular use case where you need to keep an audience engaged for a long period of time, a game could be a way where you can get the person emotionally invested in the experience and get them coming back. You could form that habit over a long period of time. We're working on a project in the adherence, medical adherence space. And it's really important for the people who are gonna be using the experience to come back and use this game every single day in order to maintain adherence over the course of years. So that's a case where games can be quite beneficial. Widespread platforms and reach. Digital games can be distributed to billions of people pretty much instantly. So app stores, the app store or Google Play or through websites or through steam and consoles and other platforms. There's a huge potential audience. So if you need to get this project out to people and connect with them and distribute the project, they have a huge capacity for that scale. Instant feedback. I think this is a huge strength of games where if you're going through an experience you can know right away whether or not you're doing an action or activity or whether or not the answers that you give are right or wrong. If you were training a particular skill in a simulation, the potential you're using VR and you need to use a tool in a particular way, the person the very first time that they're practicing that skill, they know whether or not they're doing it right or whether or not they're doing it wrong. And that prevents them from forming bad habits or have bad behaviors and can let them adjust and I guess shorten that learning feedback loop where they can improve and make sure that they're doing things the right way. Personalization at scale. I think this is a really interesting aspect of games where it is a very intimate and personal experience when you're playing a game and the game can respond and be tailored to your particular skill level or where you're up to in your learning or development journey. So a very simple example of this is if you've got 10 levels that cover the learning syllabus, somebody who knows the learning syllabus would be able to move through it very quickly and then maybe they get to the harder levels much faster and then they're challenged and they're at an appropriate skill level whereas somebody who is maybe not familiar or doesn't understand some of the core concepts, they get stuck on level two and the game can recognize that and acknowledge, oh, that you've been here for a longer period of time than you should be. We've recognized that you don't understand these fundamental concepts. Here are some links to some external resources or here's a video of some of the learning content that you might not understand. So it's really that ability to respond and adapt and personalize the experience to each individual participant can be a really powerful use case. Measurable outcomes. Games are great at tracking all sorts of data about how we're using a game or our performance within the environment and that can be good for evaluation but it could also be good for the player and their learning or their development where they get that feedback, they understand their results, they understand their marks in real time and they can adjust their approach accordingly. Gathering valuable data, I guess the depth and the type of data that we can gather within a game environment I don't know, I'm not gonna say it's limitless but you can gather a very amazing amount of data. So if you wanted to measure how long people dwell in a particular space or where they take particular actions or lots of information about demographics or usage, there are just a lot of different data points and then that data can be analyzed, processed and reported on in a somewhat automated fashion. So some data points that might be too challenging or time consuming to record within the physical world might be easier to do in a virtual world. And I believe this is the last point, a safe environment. So games allow you to experiment, they allow you to fail, they allow you to try options that you wouldn't normally try in a real world environment. And this also allows people to sort of push their limits. So if you were training to land aircraft, if you're in a 747 simulator, you could push the limits and see like where the breaking point is, where if you were training in a real world scenario, you wouldn't be able to push those limits, you wouldn't be able to see where the breaking point is. So you wouldn't learn as quickly, you wouldn't know when to expect danger as well as if you had gone through some of those scenarios where you had failed and learned from those lessons. So those were some of the benefits, but there are also challenges that we face when we're using games. Games can be, it can be very difficult to teach highly specific or technical content. I guess when we're developing a game, we need to program in or figure out every single use case or every single possible answer or configuration of the way that people are going to interact with it. And if it's a very specific or technical subject matter, it might be that there are 100 different ways to answer this question or do this activity. And it might just be that developing a game or a simulated environment for this isn't going to be an effective use of time and just doing in-person facilitation or more traditional learning methods might be more effective for that particular use case. Referencing, so games don't work like a textbook. With a textbook, you can kind of flip it open, flip to the page that you know that the formula's on, look at the formula, write it down and then get on with your day. It's a very quick process. Finding a particular section in a game where you learned a particular concept can be really challenging. It'd be common to, oh, you have to open up the game. Maybe there's a level select tool where you can go and find the right level, but then maybe you have to play through five to 10 minutes of a level in order to find that one section. And at that point, it's all too difficult and I'm just not going to bother with it. It's also difficult to reference it if you were doing like a research paper or if you needed to send a particular section to somebody else. You can't just send a section of a game to somebody else. You could potentially record a video and send that across, but because of the interactive nature of games, maybe that's not going to be as effective or actually achieve what you're looking to with that. Personal preferences, just like with everything else, people have personal preferences and that applies to games. Some people will like a particular style of game and because it's interactive and it's kind of driven by the player, those personal preferences are more exaggerated in their enjoyment or their willing participation in the experience. If somebody isn't like a logical problem solving sort of person and you sit them down in front of a game that has a lot of puzzle elements to it, they'll be frustrated and annoyed by the experience. Is there anything else about personal preference? Oh, this also applies to maybe a similar point but digital literacy. So this is a big one for the projects that we work on where two people might have the same personal preferences but one of them is very adept at digital literacy. They have played a lot of games and the other person might be very new to games or has never played games in their life. Trying to design a game that works the same for both of those people is very challenging because the person who has low digital literacy might be confused and overwhelmed and the person who has high digital literacy might be a little bit bored or think that it needs to be more challenging or have more features and functionality in it. So it is really good to understand who your target audience is so you can develop an experience for them. And there are certainly ways around the digital literacy component. They can be expensive to produce for a niche audience. So there is certainly an overhead in developing a game. There's a lot of design work and software development work and art production work that's required. And if you've got a niche audience of 30 to 100 people that are gonna be playing this experience it's probably just gonna be more cost effective to do some other form of engagement in person physical facilitation or whatever it needs to be. Transformational games aren't created fundraising. We've been approached for a few projects where the desired goal was to use either a sales price or microtransactions in the game in order to fund the experience itself. It's certainly possible, but essentially you've got a split focus where you're looking to raise funds and then you're also looking to have some sort of change or solve some sort of problem. And you need to have a priority on each of those. So which one is more important and that splits your focus and you end up usually doing both poorly. And you're also competing with entertainment games where the only purpose they have is to generate as much money as possible. So the option of what the player engages with and where they spend their money usually goes towards the projects of the games that are specialized in a particular area. Unique capacity to iterate and it usually requires a long timeline. So if you've got a month to put together a project, games probably aren't the right tool. There's just, again, an overhead in the design time and building and developing and iterating on the experience where you need at least sort of four months minimum is what we would usually recommend. But for a more complex project or when you're dealing with a very sensitive subject matter or other situations might need longer, maybe it's 12 months. Those are sort of common timelines. And it's also very important to be able to iterate, to be able to get feedback from users to be able to update your approach, learn from the feedback that you're getting and then improve the experience based on that. You need a distribution plan. So developing a fun, exciting game with a noble purpose isn't enough. You really need to understand how to communicate with your target audience and how to communicate the value of what you're working on to that target audience. It's easy to think that a fun, exciting game is gonna get people's attention but there are a lot of other fun, exciting games out there and you really need to understand how you're gonna get it out into the world. So those were a few benefits and challenges of, games have a different time. Those are a few benefits and challenges of games but games don't need to just stand alone. They can be part of a broader strategy and work together with other initiatives or activities in order to achieve your objective. So you've got the digital game, maybe you also have text-based learning resources that do explain the dense, complex learning content beforehand. Maybe you've got a video series that has somebody playing through the game and explaining it for anybody that has low digital literacy. Maybe they could engage with that video content instead. Maybe you've got some sort of workshops or in-person facilitation for, I guess condensing that learning and letting people teach others about what they have got out of this game. So yeah, whenever you're designing a game you can always rely on these other tools, these more traditional methods in order to create a more effective approach for your initiatives. So I thought it'd be good to talk about a few example projects that we've worked on to put some of this in context. So this came as the project that I mentioned previously that we developed for the Department of Foreign Affairs and Trade and the objective of this game was to appeal to international students and get international students interested in studying university subjects in Australia, being interested in ecology and environmental science subjects. So the way that we did it is we created a game that was appealing to the target audience. You adventure around Australia exploring our national parks and rescuing native animals, rehabilitating them and releasing them back into the wild. So the game itself then has a lot of links out to external resources where you could learn more about these animals or you could, there were characters in the game that studied at Australian universities and had bios and there was information about the different universities that are researching these areas. There's information about what research was happening. There's an encyclopedia filled with entries and fun facts. So it's really about capturing that initial interest and getting people emotionally invested in these animals in this environment and then providing avenues for them to further that learning and that education. So through the game, we measured a 10% increase in intent to visit Australia after 20 minutes of play. So when we can distribute this to thousands of people across the world, we can really, I guess, change people's minds about a career or potentially change people's future trajectory for what they were going to study. The next project that I wanted to touch on was In Their Shoes. So this is a game that we developed for Takeda pharmaceuticals and Takeda wanted to create a game that helped their staff build empathy with sufferers of IBD. And the game itself is a augmented reality game or a role-playing game where you take on different challenges that look to simulate some of the trials and tribulations of living with IBD. So you might get a notification saying, you know, run to the bathroom right now in the next five seconds or else you fail the game or you need to drive home in order to pick up a new pair of change of clothes. You need to take a photo of it, then you need to drive back. You're in the middle of a meeting, but it has to happen right now. You don't have any other options. So Takeda did an evaluation study of this project. There's information down the bottom if you wanted to look up how they approached that. But yeah, it showed that there were statistically significant increases in IBD understanding and connection to patients, empathy and pro-social job prospects. And it encouraged patient perspective taking and a strong desire to promote patient advocacy and reduce stigma around chronic illness. So a fairly simple premise for a game, but really good at helping build empathy and open people's eyes and broaden their perspectives. So from just those few examples, we can see that games can help inspire us and get us interested in new subject matter. They can potentially change our career path. They can help us see a different perspective or help build empathy with people in a different situation for ourselves. The field of games is very, very broad and that's just a few examples of some games that are solving real world problems. But how do we know that they're effective? So I thought it'd be good to quickly run through some different examples of scientific studies that have been done into different areas of games being used to solve real world problems. The industry of games is very broad and the transformational games industry or field is only about 20 years old. So I've never seen any overarching evidence of games are good in all use cases or scenarios. All of the evidence I've seen is quite niche and focused on a particular area. So I'd say if you're interested in looking into this more, Google the specific use case that you have in mind and look for evidence in that space. But games have been shown to improve teamwork and cooperation with others, improve management skills, simulation and learning physical world skills. There's lots of research into how simulation games have been used in the workplace and how effective they are. Games can improve problem solving and logic skills, planning, resource management and logistic skills, inductive reasoning and hypothesis testing, perseverance. I think perseverance is an area where they've done a lot of studies not necessarily into transformational games, but just gamers in general. And it's been shown that gamers have much higher perseverance or ability to continue in the despite of adversity. And a lot of that is because when you play a game, you have an agreement with the game that if I try hard enough, if I keep trying, I will eventually be able to overcome this challenge. And that isn't necessarily the case in the physical world. Sometimes life perseverance doesn't actually get you any need to be. But it is a valuable skill or behavior to exhibit and does help in a lot of scenarios. And games can really help emphasize that or train that. Games have also been shown to improve memory, hand-eye coordination and multitasking. So how do we measure the efficacy in games? Ideally, when we're working on a game, when we're working on, let's say an education game, measuring the efficacy is fairly straightforward. We can look at direct data capture and direct observation and we can use that to evaluate whether or not the students had again the knowledge. I thought it'd be useful to go through some typical data points that we look at in some of our projects, just to give you an idea about what sort of data we're gathering and how that's been used by different parties. So the completion rate of content, how far are they getting through the different levels? What score did they get on different levels? The usage data, so when did they sign in, how frequently, how long did they stick around for? Demographics can be very useful for cross-ferencing against some of these other data points, where the user is looking. So in a recent project, we tracked the position of the mouse on the screen and used that to record where the user was interested and what they were considering. They were considering options A or options B and this was used in a study where they were looking at if changing the text prompts in the questions had an impact on people's decision-making and then they surveyed people afterwards and they essentially identified that even though people said that it didn't impact the decision-making, in reality, the data showed that everybody's decision-making was impacted by some small changes in the language that was being used. This could also be applied to virtual reality. So in the latest virtual reality headsets, they have eye-tracking in them so we can literally see where people are looking at all times. So if you had a particular use case where you wanted to track where people are looking in an environment or where they're looking in a room, that could be something that you track in these simulated spaces. Whether players stopped interacting with the experience or where they left the experience, this is often used to improve the content or we see that a large portion of players were leaving in the middle of the level. What is it about that spot? Maybe we confused them or maybe they didn't know what to do and we can use that to improve the experience. The value of different variables. So this is something we've done for different research projects where we made it very easy to configure variables and run different experiments and then those variables were tracked in the analytics platform and could be cross-referenced against all of the other data. So making that sort of A-B testing a lot more straightforward and somewhat automated on the reporting side. External changes in behavior. So if we can monitor whether or not people are acting in a particular way or surveying them to see if there's changes in behavior, external changes in performance. So if we have a game that's being used in the workplace, it's good to potentially we're monitoring their job performance or how they're doing in a particular task. And it could also be that we look at group A, like played the game for this long. How do they compare against another group that didn't play the game? And then pre and post launch tests commonly use those for educational projects. Very easy to run in a classroom environment and just get an early indication of whether or not we're achieving learning outcomes. Nick, how am I doing for time? I think you're okay. Why don't you keep going for a bit? Well, maybe we can stop in 10, two and folks, if you have questions for Niko, please drop them in the chat or have them ready and we can get up there. But yeah, please keep going. Cool. Yeah, 10-2 sounds perfect. So yeah, next slide. I kind of just wanted to put up a few examples of some interesting data sets that have been produced from games that might give you some ideas. As I mentioned, evaluation isn't my specialty, but my hope is that by showing you all these different data sets, it can spark some ideas and some creativity of different use cases. So on screen at the moment, up in the top right is World of Warcraft. And a while ago, they had a bug in the game where an effect sort of got out of containment that acted a lot like a virus. It got passed from person to person and damaged their health until they eventually died. And what ended up happening was it got out of control, the sort of capital cities or the areas of dense population emptied out. Players themselves set up quarantine zones like healers went around trying to save people and stop them from dying. And this all just organically happened and evolved. They also had just bad actors that were going around intentionally trying to spread it. And it's just this really interesting scenario, but researchers actually contacted the developers, Blizzard, and they got access to the dataset. And that was actually used in a few different research studies to understand how do viruses spread? So this is a way where because so much data is being attract in these games, we can get this really interesting insight into areas of data that we wouldn't feasibly be able to track in the real world. And that can be done on a huge scale and then that can be sort of automatically processed and analyzed. And in the middle of the top of the screen is a game called Fold It. So that's a citizen science project where it's essentially a puzzle game that players try and solve these protein folding puzzles. And the solutions to those puzzles were used by researchers in order to develop new drugs and cancer treatments. But in a matter of months, the players of this game were able to solve some problems that have been stumping scientists for years. So that's not necessarily tracking data within the game, but it's getting the players to produce some sort of end product that can be used in research and training. I'll jump over that one. Theory of change, I'll assume that everybody here is familiar with the theory of change. And it's not my area of expertise, so I'm not gonna try and explain to everybody, but it is a useful tool that we've used in some of our projects where we weren't able to directly measure the end outcome that we were looking to get. We've worked on a few environmental sustainability thing projects, and the end results we wanted were sort of behavior change and reduction in CO2 emissions. And we didn't have the capacity in order to do a huge evaluation study. So we used the theory of change to get a basic model for how much impact we were having and whether or not the actions or the activities that we were taking were heading in the right direction. So that's definitely a good tool to consider if you're working on a game project. And very final point, the transformational framework. So if you're interested in working on a game in the space or working on a project in the space or you just wanna learn more, you're just curious, I would highly recommend the transformational framework. It's a free online e-book that can be downloaded. There's a little URL down in the bottom right of the screen. But essentially it's eight different components that are critical to the success of a transformational game. It's where we picked up to transformational games over serious games and was developed by Shell Labs over in the States. And it also just does a good job of sort of laying out the game development process, the different steps that are involved, what different parties are there and how they all interact with one another. So the transformational framework focuses on the following eight points. So high level purpose. So what are you looking to accomplish? This is mainly for your internal team to keep everybody aligned and to make decision making easier and more straightforward. We can say, yes, this is aligned with the high level purpose or this isn't. Audience and context. You really need to know who your audience is and it's very common that we get brought on for a project and the target audience is everybody. If we're designing a game for everybody, we're actually designing a game for nobody because we can't listen to a particular group's feedback and design a game to appeal to them. We end up having to make something that's bland and unappealing. So when you do have a broad audience, it's always a good idea to try and start with a small subset of that and then expand out because it'll allow you to get to an effective end result a lot quicker. Player transformations, specifically, what transformation are you looking to have within the player or maybe multiple transformations? What barriers exist to the player having those transformations? Is access to the game going to be a barrier is underlying pre-existing beliefs going to be a barrier? Domain concepts, so what information is contained within the domain concepts that you're dealing with? What's going to be in the game itself? Like what knowledge is in the game? What's the prior knowledge that we're assuming the player is coming to this experience with? And what concepts have we acknowledged but said we're not going to deal with those at this stage? We're just going to focus on these ones. Expert resources or subject matter experts are critical for projects in this space where they need to be able to advise and say when something is right or wrong or that is right enough or maybe we should just not deal with this concept right now and we can leave that one out. Prior works, are there other games or movies or documentaries or websites or books that have tackled a similar challenge to what you're looking to tackle? Can we look at how they have succeeded or failed? Can we rely on them to be like a component in what we're doing where we're referring to them or referencing them? And then finally, assessment plan. I'm sure that everybody here would remember to do an assessment plan but it sometimes gets forgotten in other contexts and it's definitely one of the most critical components of understanding whether or not you're having the impact that you're actually looking to have and using that to inform whether or not you should spend your resources on going down the current path or maybe change your approach and try something new or different. So that is the end of the presentation section. I'd love to hear some questions and if we run out of time and anybody wanted to connect with me on LinkedIn, that's a place to connect with me and I'd be happy to answer any questions that you have. Thanks Nikoi, that was fantastic. Got a couple of questions in the chat. Danielle's asking, I think it was in relation to the capturing of information in the tracking. I think you're talking about the, I forget what game it was, but tracking kind of people's eye movements and where they were looking. Danielle wrote, I guess there are ethics considerations around that. So how do players give permission for that sharing of data? Danielle, if I've misrepresented that, let me know. Yeah, particularly, I think it was the one where you said there was a virus and people acting as healers and then suddenly researchers contact the game makers. So it wasn't necessarily the initial intent that other people came in, which is a little bit different, I think the one where the intent is to find that out. Inflating that. Definitely, there are a lot of ethical considerations with a lot of these projects and I think all of the projects that we work on, that's a consideration and we make sure that we get consent and make sure that the player understands what we're using the data for. And usually for the projects that are used in that capacity, it's very clear that it's either done in a research environment or it's a very clear, potentially it's in the workplace or a similar sort of environment. Yeah, I'm not sure what the ethical considerations were for the World of Warcraft Corrupted Blood example, but there are a few research papers that have been done on it and I believe that they wouldn't have got any identifiable data. I think that they would have been looking at like where people originally got it, how far they traveled, looking at how many contacts they had and how many people they passed on to, data points such as those. Thanks. Thank you. Another question was from Samantha, talking a bit more about the medical adherence game with people coming back every day for several years. That's the question, could you talk a bit more about it? It is still in development. We're hoping to have it launched early next year. So I'll be posting about it when I can. LinkedIn's a good place to stay up to date on that. What can I talk about with it? It is, yeah, it's for chronic illness. It's for creating and maintaining good, healthy habits that help support their chronic illness and they need to essentially do these activities and a lot of people fall off on doing the activities because it can seem futile and we want to create essentially a support network and a social network focused on the specific adherence in order to help people live healthier and longer term lives. If you've got specific questions, I can probably try and talk my way around it but yeah, I can't say too much at this stage, unfortunately. Yeah, no, just thanks, Niko. Yeah, just interested in finding out more about how these games are used and even that's useful information in terms of looking at the potential for evaluation and evaluation capacity building because even what you're saying in terms of building social networks through those games, in terms of for motivation and connection. Yeah, this game is more similar to animal crossing than it is to an action game or an adventure game where it's more of a social space and I think this goes back to the earlier point of games are incredibly diverse and there's just so many types of experiences that whenever we get brought onto a project, we look at all of the different styles of games and oh, we need a social experience that has this cosy, inviting atmosphere that you want to exist within that space. That is a good tool for the job or for the problem that we were looking to address with that one. Thanks, Niko. Another practical question was from Alison on what kind of cost is associated with developing games like these? It depends a lot but in terms of some ballpark figures, it might start from sort of... If you know what you want and it's very simple and straightforward, maybe in the tens of thousands, maybe 50,000 is a good initial jumping off point but those would be very simple, straightforward experiences and as soon as you start getting into, I don't know, 3D or immersive gameplay or especially social features, so any sort of online or interactive features, you're probably looking at hundreds of thousands. As soon as you're talking about like a massively multiplayer online game where everybody connects and it's just in one environment together, what a lot of people think of when they think about video games, so things like Fortnite, you're talking about millions of dollars for an experience like that. But yeah, if you were looking at the Financial Times, I'll see if I can jump back to this one. If you're looking at the Financial Times climate game, then you might be this one. Then it takes back to Narrow Adventure, there's not a whole lot of asset production that's involved. There is a lot of design and whether or not you wanted to be doing a lot of, I guess, iteration of that design, testing with people, looking at the impact goals, that one could become quite expensive. Yeah, I guess there are also multiple points where you'd say, maybe it's 50,000 to develop the prototype for it, but then is there an ongoing maintenance and support component to this where you need to expand or keep on updating it? That's very common for games where you create the proof concept or the initial validation and then you continue to expand on it. So yeah, a lot of it depends on where you draw that line, but definitely tens of thousands, ideally hundreds of thousands, potentially millions, if you're doing something really, really complex. But I mean, usually for most use cases, we advise against those like million-dollar ideas and we just say, there's a far simpler way of actually achieving this where we just scope it down and focus on the core of the experience. And in terms of timelines, yeah, four to 12 months is pretty typical for the projects that we work on. I know whenever we work with government or larger organizations, it's usually a decent amount longer, worked on some projects that are sort of two years long, so it can certainly take a little bit longer. That's great. Thank you. Yes, Alison, I think I'm hearing it depends as well. How big is your game and where does it go to? Thank you, Nico. Thanks from all of us for sharing these insights. I think it's really interesting to hear from some different perspectives and it's definitely got my brain going around and how we might even begin to evaluate some of this stuff. So thank you. As Nico mentioned, please reach out to him on LinkedIn or connect with him to find out more about these games. Do join us for our session tomorrow if you've got time and please do fill out the survey that I've put in there because we're always trying to think of what do we do next in these sessions as well. So thank you for joining us all and we'll see you soon. Thanks everybody. Nice to see you all and yeah, thanks for having me.