 Hello everyone and welcome to the 12 30 p.m. to 1 o'clock p.m. session of the 2019 open simulator community conference in this session We are happy to introduce a presentation called a VR tool to improve your public speaking skills Our speaker is Maryam Eliamri. She is currently completing her PhD at cumplea tense University in Madrid her main research area is the use of VR to train the ability of public speaking And exploring how to make computers understand human communication She is also the co-founder of crotech a software and hardware development studio based in madrid Please check out the website found at conference dot open simulator dot org First speaker bios details of the sessions and the full schedule of events The session is being live streamed and recorded So if you have questions or comments during the session, you may send tweets to at open sim cc with the hashtag pound 0 c s cc 19 Welcome everyone. Let's begin the session over to you. Maryam Hi, everyone, and my name is Maryam and I'm here To tell you about the tool that we've been working on i'm cumplea tense university and it's a tool a VR tool that we try to Use it to improve public speaking skills First I would like to introduce myself a little bit more I'm actually doing my phd in cumplea tense university in madrid and my main research focus is About how can we make machines understand human communication? Like how can we make them? I know if a human communication is effective or not? And this project is part of of of my research I'm also the the co-founder of of crotech a slier set and In crotech we Basically do all kind of technology projects and some of them are related to virtual reality and augmented reality as well So one thing I I always like to to ask when when I'm talking about this this work is Who in this room or who in this auditorium is not afraid of public speaking not afraid at all? And I'd say raise your hand But as we are in this virtual environment I'm guessing that not a lot of you will will raise their hands and this is because Public speaking is something that is very common that we we do in our daily life in in daily situations, but We all experiment some kind of nerves and some some we are afraid sometimes when we go in front of an audience However, oratory or the art of public speaking with eloquence has been cultivated since ancient times I mean the greeks were pioneers in in this and there was There were other characters throughout history that That use this discipline to even change the course of history. I'm sure you can think of some great ones But nowadays the the internet has multiplied the importance of of these disciplines because today audiences can be counted in millions and however this fear, which is a disproportionate reaction to To the threatening situation of facing an audience affects a very important part of the population actually, um, there are There is a statistic that says that 57 percent of young people between 12 and 17 years old consider the fear of public speaking the second most feared social situation And this fear affects a very very large large part of the population so, um this fear, um can sometimes Turn into a phobia when it's very extreme and and the technical term for this is glossophobia However, uh, this does not affect a lot of people but, uh, the the just the nerves or the fear is there and There's another statistic that says that it affects 75 percent of the population Uh, so when we think about communicating or when we communicate not only public speaking But any type of communication there are like two types of, uh, of things that happen or Types of communication or underlying communication. We have verbal, uh, which is the one where we use linguistic signs to transmit a message such as Sentences or words or things like that and then we have a whole part Which is the non-verbal the part that also happens when we communicate and it's also very important um The process of sending communication with signs and cues and things that are not words is this non-verbal part And actually there are animals that show certain types of non-verbal communication um So in this non-verbal we can also distinguish between two Types there is the para language Which is everything that goes with the voice like how the voice changes how the volume changes and everything that has to do with voice and then and then there is this other A part that it's the body language our gestures our posture and Everything related to to how we move our body when we speak or when we communicate And when we speak, um We feel some emotions and some of those emotions are are transmitted and Coupled with with with what we say. So verbal communication Has this also this emotion affect affect our our way of talking or our way of speaking So in the in the non-verbal part When we speak our voice or the emotions that we are feeling affect our respiratory system And then they produce our voice our voice tone to be modified Obviously not all the emotion that that the speaker feels Are transmitted or reflected in the voice but for the purposes of the work that we that we are trying to do What interests us is those emotions that the audience can see and in consequence react to And also when we speak the emotions That we are feeling in that moment Make us use some words or some sentences differently depending on on those emotions So they also affect our verbal communication. So bearing that in mind we looked at some previous work that was Already there from people that were attacking also this How can we improve public speaking skills with virtual reality? and a lot of The the tools that are already there Are limited to offering just a safe space where to practice and the feedback is very limited And this is because Saying if the speech is Right or wrong. It's a very very complex task It is something that is very subtle and it's a human thing We do it Instantly and very easy but for a machine to understand and say if a speech is An effective speech or not It's not so it's not so easy And that's why we decided to focus on on just the emotions so our objective for this work is to create a A virtual environment where Where the speaker can improve public speaking skills and it is in the form of a video game and in this video game the speaker is in front of an audience and There is a virtual audience that will react and bring real-time feedback to the speaker So that the speaker can can see how effective the speech is So in any technological tool that pretends to teach oral communication skills The feedback of the listener must be crucial for the speaker to know if the message is being effective or not because We can understand Oral communication as a continuous phenomenon of adaptation of the speaker to the feedback received by the listener So in this project on our first prototype, we decided to focus on three main things We built an emotions extractor which is based on the voice and the content of the speech So we extract emotions from how the voice changes and how the speech Content is we also have a gaze tracker which Checks where is the speaker looking at because it's important to know if the speaker is looking all the time at the screen Or mostly at the audience and we have this reactive virtual audience That brings real-time feedback to the speaker so it can see if if the speech is good or not This is the architecture of our system and we have Two main main parts So the project has been designed in such a way that it's divided into decoupled environments that work together We have the virtual environment where the speaker's action takes place and where the speaker features are collected And we have the analysis environment where these features are processed to generate reactions in the virtual audience And then draw conclusions about if the speech is effective or not In our system the speakers features can be analyzed and these features can be internal or external So we have internal features that refer to the parameters of the speaker that the audience does not see So they are more like related to Biometrical parameters like sweat or heart rate, for example And these features do not have to affect the reaction of the audience since the audience is not aware of them But they are interesting to know if uh, how is the internal state of the speaker and see if this internal state affects the speech But for for this first prototype we focused on the external ones, which are externals to the speaker Those are the features that are clearly visible For the audience and therefore will have an impact on the degree of of the attention And we can distinguish between different types of features according to the parameter that is being analyzed at each moment For this we we've used uh voice content and gaze so If you look at the if you look at the figure, uh, you can see that in the virtual environment, we just, uh put the player to to to uh practice the speech and then We collect the voice and collect the gaze the voice is sent to the analysis environment where we do two things with it We first, uh extract the speech the content to text and then, uh, we send Both the text and the audio to to the emotions extractor And this emotions extractor will generate, uh, or will, uh decide which is the predominant emotion in the speech And then we create a reaction for the virtual audience and we send it back to the virtual environment and in the virtual environment, um the the the audience the reactive audience Which is virtual will react with that with that reaction and will also take into account the gaze So here's a picture of how, uh, this looks like And as you can see this is from the point of view of the speaker And there's this auditorium and the audience and some stats on on the top As you can see the audience is spherical. So why is this? Well, um, the spherical shape of the of the audience has been, uh, designed in this way Uh, for two main reasons mainly because 3d modeling is an expensive process As you know, and we wanted to create a prototype, uh as fast as possible to test it and also because, uh There is some literature that says that the appearance of the characters in a virtual world does not need to be realistic as long as the behavior is consistent So what is important is that the behavior is consistent And in this case the behavior is very simple the sphere's turn from red to green depending on how effective is the speech So, uh, imagine that you are talking and then at a simple glance you can see if your speech is good or not because If if it's not like the spheres will just turn red all of them Um So after creating the prototype We did a little experiment with with the voice because we wanted to test if this if this made sense or not so, um We wanted to test how do the emotions in voice affect the effectiveness of the speech and for that we gathered a real audience of people and we gathered some actors that performed In front of this real audience speech And uh the speeches that the actors performed were speeches with certain emotions Sadness or anger or happiness, etc So we made the audience write down, uh, the emotion that they detected and also How effective they they thought the speech was and we gathered some all these results After that we passed the audio of of the actors to our system and then we compared results and the results were very interesting because, uh We had some weights for the emotion at the beginning As you can see there is a five kind of uh of emotions and we had some weight based on common sense Basically on how how they could affect the speech, but after we we did the experiment it was interesting because One one interesting thing is that we saw that the stress emotion or the anger emotion Which one could think that it's something bad for a speech The audience reacted very well to it when it was uh in a very short time It means that uh when the speaker was doing a boring or let's say a happy Speech and then it turned back to to stress emotion for the audience it was engaging It was good. So we had to refine our algorithm and and try to to make it better to to Take into account all these things like how long was the emotion projected and also How good was it? Uh, how how is this weight affecting the the emotion and this experiment helped us a lot to to the show so um To to wrap up a little bit, uh, as I said, uh, our communication is inherent to to human beings and allows us to transmit emotions and desires and facts and ideas And there is two types of underlying communication that we need to to take into account uh non verbal and verbal but uh when we communicate we also need to to see how the listener is is Is giving us feedback and take it that into account in order to To to improve our speech or to make it more effective to communicate better And that's why we created this tool the first prototype and we are still working on it our current work is uh Doing more things adding uh more features to the to the To the system and uh improving our feedback algorithm because now we have three features But we would like to add some biometrical parameters to try to see if They are effective or not of the or if they influence However, the the speech and also, uh, we want to make a more realistic audience I know the the spheres audiences Actually, it works for us very good But we need to make a more realistic scenario for people to really practice in a safe space that feels realistic and uh We want to also include uh an automatic question generator That it's based on the speech of the speaker so that the audience can make it even more realistic by asking questions about the speech So that would be all I hope you enjoyed the the presentation if you have any questions Uh, I don't know if we have much time, but you can find me This is my info or you can find me in my in my booth in them in the expo region three And that would be all Thank you Miriam for that fantastic session. Are there any questions for Miriam? There were some comments in the chat chat log and of course over the terminology and how it can be paralyzing to to for public speaking And Miriana says thank you. It's much appreciated. Thank you You As a reminder to our audience You can see what's coming up on the conference schedule at conference dot open simulator dot org Following this session the next session will begin at 1 p.m. In this keynote region And it is entitled the open source high fidelity panel Also, we encourage you to visit the oscc 19 poster expo in The region to find accompanying information on presentations and to explore the hyper grid tour resources in oscc expo two region along with the sponsor and crowd funder booths Located throughout all of the oscc expo regions Miriam has a booth over there also so we recommend that you go over and check out her content Thank you again to our to Miriam and to the audience