 Okay, I can hear myself. So a very good afternoon to everyone who's present over here. Thank you for attending PyCon and for choosing to attend my talk. So this talk, as he said, is titled Entering the World of Series Gipset Python and I hope that at the end of this talk all of you learn something new. So before we get started I would like to introduce myself briefly. I'm Harshini Sriram and I'm currently a final year interning student in SRM University, I worked with AI for the past year and a half and my area of research is artificial intelligence for mental health and well-being. I'm also the author of the books The Horses of Happiness and I couldn't be more excited to share with all of you what I've learned in my journey so far. But before we get started I would like to introduce two fundamental topics which are the basis of this talk today. So firstly what is mental health? So mental health I would like to define it as a state of your mind. It is something that can affect how you think, how you feel, how you react to certain situations and even people. It is something that is important at every state of your life be it infancy, teenage or adulthood. So what is mental illness? Mental illness is or our disorders that affect the way you think, feel or react. The three major kinds of mental illnesses that are prevalent in today's world are an anxiety disorder, depression and clinical bipolar disorder. So with that what will we be learning today? I'll be covering what are serious and therapeutic games, stages involved in serious and therapeutic game development, current examples of therapeutic games. Then the second part of the talk is why should we use AI with serious games and a few intersection topics which includes decision making, decision trees, fuzzy logic and virtual effective agents. The third part is the conclusion. So where I'll be introducing how Python can help you know incorporate all these modules into your project if you want to work on this intersection and what is the future whole obviously. So what are serious and therapeutic games? Serious games are basically games that serve a purpose that is beyond pure entertainment. So they combine gamification techniques with a real life objective in order to make the experience more enjoyable as well as effective. So serious games are further divided into games for education, games for health and so on. For example, think of a pilot simulator game. I mean, although it is very interesting to play, it is used in the aviation industry for training purposes. So that is an example of a serious game. So serious games can be classified on the basis of two models, the two-parameter model and the four-parameter model. So the two-parameters in the two-parameter model are the function and public. So function is the, you could say, the basic utility of the game and public is the kind of target audience or audience in general that you intend the game to be for. And the four-parameter model, which is more frequently used, it defines the gameplay, function, purpose and public. With that, what are therapeutic games? These are subcategory of serious games which deal with a mental illness. So they can assist with symptoms and they can even potentially eradicate the presence of the mental illness altogether. The approaches they incorporate could be either directly therapeutic or indirectly therapeutic. And they're basically of three kinds, which I'll be explaining in the next few slides. So in order to understand the stages in serious game development, you need to understand three different approaches to therapeutic game development. These may overlap based on the overall application of your game. So the three game players are informative game playing, therapeutic game playing and therapeutic game designing. So informative game playing is when the game designer creates the game and while you're playing the game, you're given all these visual and textual information about a particular mental illness. So by the time you finish playing the game, like not only have you finished the game, you also have, you know, increased knowledge about this mental illness in question. This can help you, you know, find out, you know, better treatment options for patients who are suffering from this illness. So apart from that, therapeutic game designing is when the game designer himself is suffering from the mental illness and then they design a game as a form of their treatment, as a form, you know, to maybe even eradicate the symptoms that are a result of this illness. And so if another player is playing such a game, they get to see, you know, how the condition is from a first person perspective, because they get to see, you know, how a person who's suffering from this illness would think of it as a treatment option. So this gives you a more in-depth review into what could potentially be, you know, a treatment option that can solve this illness for a majority of population and patients. And thirdly is therapeutic game playing. So this is like the most common area. So in this, the game designers, the researcher mental illness, and out of all the treatment options, they may include two or three many treatment options in that game. The target audience for such games are people who are suffering from that mental illness. So while they're playing the game, they may directly or indirectly receive treatment. So with that, let me introduce a few current examples of therapeutic games that you can actually play if you do want to. And if you're interested, firstly is Papu and Yo. So Papu and Yo revolves around this young Brazilian boy called Quico. And it's basically a puzzle based game. And Quico needs to find a cure for his friend, that is the monster who is addicted to frogs. And in the game, if the monster comes across a frog and eats it, it turns into his addiction of sorts, and then it keeps eating more and more frogs, until it becomes this beast that can harm everything around him. Like even though Quico is his friend, he can harm Quico if Quico doesn't run fast enough. So in the end, we are shown that Quico alone is not capable of solving the monster's addiction issue. The game designer himself had an alcoholic father. So this game is like a symbolism of how his childhood was. Next is a game that I'm sure all of us have played here. And it's called Tetris. Now Tetris initially wasn't created as a serious game per se. However, it has been evolved to accommodate certain case scenarios, which makes it a serious game. So it can help reduce flashbacks for people who are suffering from PTSD, which is post traumatic stress disorder. And it also assists with an anxiety disorder by calming down the people who might be undergoing some unexcited and used events. Thirdly is the game called Social Clues. Social Clues was developed for students with autism and developmental disabilities. So the character in the game, they lose a toy or a belonging and in order to retrieve the lost toy, they have to converse with the people around them, make meaningful conversations and infer clues from these conversations. And that's how they find the lost belonging. Also the performance of each child is recorded and it can be sent to the child's therapist or parent so that they can make the future levels more effective for the child. Lastly is my favorite. It is the award winning game, That Dragon Cancer. This is an autobiographical game that was based on a true story of two parents and their baby Joel, who is diagnosed with cancer when when he was too young and then he shortly passed away. So this game shows us how the events occurred. For example, the first scene is you know in the hospital where the doctor tells them about the diagnosis and you know the short life expectancy of the child and then it shows on how they took care of the child and how his last moments were spent. In fact, I do have a short clip of the game in order to give you an essence of how the gameplay works in That Dragon Cancer. So I would definitely recommend playing this game to everyone here. It is definitely an experience worth remembering. With that, every time I tell someone why my research like the area of research is AI for mental health, they ask me why. So to this I have four major reasons. Firstly, machines and intelligent robots are not susceptible to tiredness, burnout, forgetfulness. So they can be efficient during long shifts. Secondly, intelligent systems are immune to personal biases that human psychotherapists or therapists may have. Also, a range of studies concluded that usually people are more comfortable in speaking with a virtual agent or virtual humans during clinical interviews because not only are these beings always friendly, but they're always there for the patient whenever they need them and not they don't resort to appointment based in interactions. And also in the economic side, I mean, if you do have more intelligent systems, not only does it make mental health facilities cheaper, but it also makes them prevalent in areas that were devoted to them before. So there are four things that I want to share with you today. For that, let's I'll be introducing artificial intelligent techniques in serious games. Firstly, we have decision making. So an expert system is a computer system that is created in order to mimic a human's decision making capabilities. So what we essentially have is an outline that you can see. So I have three components, a knowledge base and state inference engine and a user interface. So the knowledge base has, you know, ground truths and rules that are determined by a domain expert. And then these rules are used to determine your present state, your future state, your past states in the state inference engine. And then these states can be used to operate a UI. So in its core, it's predominantly if else statements, if then statements with associated probabilities, like if I tell the system that I have a symptom and then it can predict that I have this disease with the surety of some percentage. So that is exactly what an expert system does. So let's look at some two examples of expert systems. Firstly is Diagno, which was developed by the Columbia University for psychiatric diagnosis. It was based on a logical decision tree model. So the decision tree, it comprised of a series of questions, each of which were of the true or false kind. And based on your response in one question, it ruled out certain diagnoses and also formulated the next data was from the psychiatric status schedule, which is an instrument that contains standardized interview questions associated inventory for psychotherapists and questionnaires. And the output was one of the 25 standard American psychiatric association diagnoses, as well as two unofficial diagnoses. So next we have Oncusknight. Oncusknight is a decision-making system that utilizes bioinformatics, artificial intelligence and statistical learning to help identify and characterize patients who are resistant to cancer treatments. So not only does it help provide guidance to therapeutic specialists on how to deal with such patients, it also helps drug companies and firms on how to develop new drugs for these kind of patients. And they're still at a very early stage of R&D. So decision-making systems and expert systems lead to decision trees based gaming. A decision tree-based gameplay takes into account the errors or decisions that have been undertaken by the player in order to find the particular state in order to rule out future states and create a personalized gameplay for the player. So one of the reasons I have been wanting to play Detroit Become Human is because of its core component, which is different possibilities and outcomes based on the choices you make. So the game also shows you major decision points and the branches that lead up to different conclusions. Also, one instance of a decision tree-based series game is the SimSensei, which was developed by the University of Southern California Institute of Creative Technologies. This is a simulation-based game where you converse with a virtual agent that acts as a therapist and based on your expressions and responses, the agent conveys statements or information. So let's look at a case scenario where this tries to converse with a human and possibly give out diagnosis. My kids keep me going. What advice would you have given yourself 10 or 20 years ago? Do not be so gullible. Do not be so gullible. I told recently that I have a tendency to believe too readily without them having to prove anything. So as a result recently I've really I've gotten myself in a lot of trouble and so I think that looking back, if I could have, I would call myself to not believe to not be so stupid and to think to not be so gullible. Is it uncomfortable for you to talk about this? Yeah, it is. I haven't really thought about that. So another area in the intersection is fuzzy logic. Okay, so fuzzy logic the word itself, fuzzy, means something that's unclear or unsure. So that is more or less the essence of a fuzzy logic system. Our brains or rather, you know, our reasoning and our thinking processes, we use inference that is very similar to fuzzy logic. For example, if I ask you how is the weather today, you could respond with it's kind of good or it was better than yesterday or I mean, I mean, I've seen better weather. So, you know, as you can see, these are not straight 0-1-1 responses, but they're like the in-betweens. So in fuzzy system, there's no logic for an absolute truth or an absolute false value. But there's an intermediate value that's also present, which is represented by the degree of trueness or the degree of falseness. So if you look at the general architecture, there are like four major components of a fuzzy logic system. So we have the fuzzy fire, the defuzzy fire, the rule base and the inference engine. The rule base is similar to the decision-making components. So you have ground truths and rules determined by a domain expert. Fuzzification, what it does is it takes into input distinct or definite numerical values and classifies them into fuzzy sets. Now the inference engine takes into account the user input and tries to determine which of these fuzzy sets is most appropriate for the user given response. And the defuzzy fire, it takes that fuzzy set and converts that into quantifiable values that you can work on. Also, there are four different kinds of fuzzy fires that are currently present, singleton, Gaussian, trapezoidal or triangular. So let's look at an example of how fuzzy logic is used in a serious game. So in this game application, fuzzy logic is used to provide information about the patient or the person playing the game and all related configurations. And then based on the input data, the expert system can provide a set of games that the patient or the child has to play. So there are six games, three of which are 2D and three of which are 3D. So the 2D games are far from time, find the food and color it. In these games, the patient only uses their upper extremities and also in these games they recognize the movements of the user's hand in order to navigate the cursor on the screen. The three 3D games are dodge it, catch the balloon and imitate it. And in these games, the user has to use his entire body and to, you know, in order to fulfill the game dynamics. And the movements of the user are recognized by the control and avatar that's on the screen. So as you can see in the leftmost table, there are two more, like the two major classes of values that are used to determine the difficulty level of the games. In this case, we have the case scenario for Down syndrome or a child that's suffering from Down syndrome. So based on their motor skills, MS, and their high activity or their health condition, we can determine how difficult or how easy you want the game to be. For example, if a child has very good motor skills, superior motor skills, and mild high activity, then the game would be the most difficult for them. And I mean, it's the complete opposite for a child who's suffering from dysperaxia and has severe health condition. So to evaluate the configurations that were created by the fuzzy system 25 cases of children with disabilities were formed. Of these five are characteristics of your patients, as you can see in the table to the right. Whereas the remaining 20 were created by a panel of experts, which includes psychologists, educators, doctors, therapists, etc. Also, as you can see in the third picture, we have the award the obstacles game. So this is the 3D avatar controlled game where they have to use the entire body in order to award the obstacles. Lastly, the last topic is virtual effective agents in games. So intelligent virtual agents are basically computer control systems that interact with human beings. So, I mean, if you remember, there were a lot of funny videos made a few years ago about EV Bot. EV is a rudimentary virtual effective agent. Now, how the virtual agent looks and reacts is completely malleable. I mean, it can be made unique for every user. Also, the way you interact with your effective agent is also malleable. So it could be text based input, text based character output. It could be voice recognition, synthetic speech. There are a lot of possibilities. Also, the extent to which your virtual agent reacts to the user's responses depends on how intricate or how complex you want your system to be. So virtual effective agents are used in a lot of games, be it entertainment or serious based games. So let's look at a few examples of virtual effective agents. So we have Lola by the effective social computing lab at Florida International University. Then the virtual agent Mary playing a game, card game and the emotional range of 3D agent Max by the University of Bielefeld, Germany. So with these examples in mind, there are two major characteristics of a virtual effective agent, emotion recognition and expression as well as effective and social realism. So the recognition of emotions and the display of appropriate emotions by agents or computer characters are the two most primary areas of effective computing. So they both rely on identifying features, facilitating data analysis and rendering the expressions. So the filter data is provided by pattern recognition techniques and also the recognized emotions are usually limited to the basic emotions like joy, sadness, anger, fear but then research is being done to identify more complex emotions such as shame and guilt. Secondly we have effective and social realism as well as emotional intelligence. So the concept of believability of an agent is dependent on factors such as its appearance and effective social realism. Effective realism is the agent's ability to display, to show the appropriate emotions in the appropriate time frame and emotional intelligence is you could say a cloud form or an umbrella term that takes into account why these emotions were shown by the user, what are the consequences of these emotions and so on. So in games usually virtual effective agents are used to form NPCs or non-playable characters. As the name signifies, NPCs are agents in a game that can be indirectly controlled by the playable character which is the user of their game and so the NPCs are predefined by action state mechanisms and the exhibit most NPCs exhibit features such as visual perception social believability as an understanding the kind of situation and reacting accordingly which is not the case of the Elder Scrolls 4 which in my opinion had the worst NPCs or the funniest NPCs ever. In fact here is a virtual reality game where you have an NPC that helps direct you with the fear of heights. It's helped to conquer your fear of heights and let's have a look at how this project works. This is a virtual reality simulator designed not only to induce a fear of heights but also to help conquer it. The view from here is amazing isn't it? I'm now right up the top of the building and I can see the roof just above me and it is very high up. How is your anxiety now? I still feel okay. Quite high up. To be this close to the edge is very little barrier. But okay. What I'd like you to do is pick eight apples and to reach them, use the platform. You push the lever forwards and it'll take you out. Let's go there. Okay I'm moving forward. That was one. I need to get eight right? Yeah. That's fine. This one is quite far away. I've got to lean quite far over to get this one. I don't think I'd be able to do this for a long time. I would not have let that far forward if this were real. I can tell you that. Alright so with this comes the third part. Now the reason why I shifted to Python for most of these tasks is because of its flexibility and fluidity. So with that into account, I mean there are also a wide range of tools and libraries that you should look out for if you want to work in this intersection. For example, we have Pygame for game development in Python. And then scikit-learn in fact has a module called fuzzy that can be used to formulate fuzzy logic systems. Then for agent-based modeling for your virtual effective agent you have Mesa. And then obviously you have image recognition, voice recognition, all these modules that you can use to build your own virtual therapist or your virtual effective agent for your project. So yeah. And then finally, what does the future hold? So firstly I would say better emotion recognition because the thing is even if you do identify complex emotions different people react differently to the same situation. So you need to take into account certain aberrations or certain different kinds of emotions and different kinds of possibilities that could be determined from a certain situation. So that is something that we should work on. Also, there is a lot of stigma about video games, serious games. There's a lot of people think that video game addiction could interfere with your treatment. It could formulate another mental illness. So we need to address that as well because controlled usage of serious games is definitely more effective. And I mean there's no case of addiction that could interfere with your diagnosis. Also, I mean I would urge everyone to do research in this intersection because you know mental health is something that can affect anyone irrespective of their socioeconomic standards. And in fact do research because collective help, collective research works because none of us is as strong as all of us. And I mean with that thought this talk comes to an end and I hope you learned something new today and have a great year ahead. Thank you. So I mean would anyone want to ask any questions? Hi, I wanted to ask you for the 3D libraries do you use open source libraries or do you create any 3D models in Maya or something? I mean I've only used open source but then I mean you're also willing to create your own. I mean I'm pretty like I'm currently working on dual diagnosis so there wasn't anything specific that could help with it. So I'm in the process of creating one so I mean if it helps with your current project it's good enough or I mean you could always create so. Hey great topic. I just had one question so you have been like doing a lot of research on this topic right? Yeah, yeah, yeah. So do you have any factual numbers of how effective games have been on actually curing the mental illnesses? Yeah so well the problem is the examples that I showed there would be around like this is 80% of all the games that are out there and you know even if there are games because you know hospitals they don't really believe that a game could help with this so I mean a lot of health organizations don't like to adopt it but I guess with that you know even after you're creating education should be spread about the effectiveness of such games so I mean more circulation there will be more rewards. In fact in the one with fuzzy logic so out of the 25 cases 23 of them worked because I mean you know even in one mental illness if you have a set of symptoms there will always be a patient that has a different symptom. It's not like you know a straight cut out part so I mean it really depends on how comprehensive and how many case scenarios the designer works on. Hi. Amazing topic. Really well defined. Thank you so much. So my question is emotion mapping essentially I think Sim Sensei got the closest to kind of really doing you know a good decision tree. Do you know of any other roots where emotion mapping is considered at least from a neuro biological perspective? I'm not really delved into neuro biology that much but then I mean maybe for yeah I mean like if the diagnosis is dependent on the physical behavior of the patient you could always use that as input so I mean in that case scenarios also it could help even not like a therapist of sorts even like you know if I can just motor the activity and help diagnosis certain it like if it's something that you know affects by extremities then it could always help with that but deep down into neuro biology I'm not sure I haven't delved into that part yet so Hey over here. Yeah okay. Great talk Harshini. One of the things that I was concerned about serious games was it is used with some in some insurance companies where they use some questionnaire based serious games some of the struggles that they tackle is that there is some noise associated by people who are filling those questions as they don't respond it correctly or don't want to speak out them so what are the tackles that some of these industry does for this like Yeah I mean even while playing I mean if it's someone that's not suffering but they're trying to like figure out the profile of someone suffering from this so they might be associated noise with that because they might not get the full picture that will also affect the performance of the games I mean it really depends on like how good your target audience is you know how many case scenarios it works on because with serious games you have to test it out before deploying because I mean else you can't really find out the accuracy parameters so I mean even in insurance companies I mean I totally understand that point of view so I guess before deploying they should test it out if it works with all case scenarios maybe even potentially understand which of these are noise values to help filter them out so I mean that's a lot of work that could be done over there Thank you for the talk Thank you so much How much of this is applicable to text based games? Text based I mean yeah have you heard of Vobot? I did yeah I mean you could always have something like that or like there's a website called Seven Cups where you talk to your psychologist so maybe like a simulation of that I mean text based also that would deal heavily with NLP but then again the struggles of NLP you know you can't identify sarcasm you can't identify the persons if the problem is deeper than what they're seeing and out to be so you can have text based applications but I'm not sure how effective that would be really expressive textually in order for that to work so So we've run out of time but we have space for one question one last question So it's really amazing So my question is like like it's with the concern of doctor right like the patient should play such kind of games so is there any time limit like for this amount of time a patient should pay this particular game like because generally like over addiction can happen over this so Yeah exactly so I mean like even though you can play these games for treatment it should be made sure that you know I can't say like it's a 30 minute slot or a 100 or a one hour slot I mean that would depend on the entire gameplay experience and how many times you want to repeat it but then obviously it shouldn't be played continuously because the patient you know that would be a case of dual diagnosis because then the patient would be suffering from comorbidity issues but they'll have two concurrent diagnosis so that can be prevented the amount of time the patient plays so I mean that should also be taken into account definitely So we come to the end of this session and please give a huge round of applause for Harshani