 welcome to the 1230 BZ3 breakout welcome to the conference everybody today we have a our guest speaker will be Kenny Alwein he's five years of extensive experience in virtual worlds with Scorum learning management systems including integrated Scorum into multiple virtual world platforms Kenny was developed multiple virtual world training scenarios and some simulations utilizing the latest technology of open simulator he has a BS in computer science and an MS in management information systems from the University of Mary Washington he's also a senior software engineer at Tech Wizards and an adjutant professor at the University of Mary Washington please give a warm welcome to Kenny Alwein thank you Dave good afternoon everybody and welcome to using open source frameworks to create an intelligent bot in open simulator so to start off here's our agenda for this afternoon go over a brief introduction Dave do a great job of telling you who I am so I probably skipped some of those slides and then I'll give you a brief overview of a bot probably sure most of you know what a bot is but just in case we'll go over that a little bit and then we'll go over the technology that we're using for these bots and then discuss some of the future possibilities that bots allow us to have and then finally any questions and hopefully there's a lot of questions and we can we can get into some technical discussions if you'd like and I'll try to leave a little bit extra time for discussion at the end so as Dave said I've been doing virtual worlds for a while now and the reason that we have been using bots is because primarily what I'm doing is is learning technology using virtual worlds and using virtual worlds to teach so bots help me help me do that with with the company that I work for and that that is tech wizards and the reason I'm going into this is because primarily what tech wizards is focused on doing and what I've been doing for tech wizards with the team is using virtual worlds doing training development and and really using open source solutions so that we can provide a great benefit to both virtual worlds and to the end user so we've done some virtual worlds using these bots for nav air nav c and the US Air Force and probably not going to go into much detail about those builds during this talk but if you have any questions please see me afterwards but this kind of just a background as to why why this was developed and why we're using bots so now to the important part of the talk what's a bot and then as well as what's a bot what what's intelligence because in the beginning we said that we're gonna we're gonna create an intelligent bot but what does that mean and then finally why do I need a bot so first off what's a bot so when we're talking bot we typically mean that it's automated so it's something that can act on its own doesn't require interaction from us and can can do what it wants and what it's programmed to do also in our case we want our bot to look like an avatar we don't want the user to realize that they're interacting with the bot so we want them to look just like you or I and then that allows the user to be able to suspend their their their disbelief useful so we probably want the bot to be useful sometimes you might not want a bot to be useful there's plenty of reasons for that to occur maybe you're trying to implement something that causes noise but most of the time we want our bot to be useful we don't want to just take up space serving a purpose that's goes along with useful so we want we usually have a purpose in mind for a bot to be serving and that that's the reason for creating this bot and finally a bot can mean a lot of things in our case a bot is an automated intelligent agent that we're using in the virtual world to help with our our either our scenario or our virtual training situation so intelligence intelligence is very hard to define I'm sure most of you would agree with that so one of the things that we're constantly striving for and one of the things that we're getting closer and closer to is is the Turing test so for those of you that aren't familiar with the Turing test it's it's based off of a test that was introduced by Alan Turing in the 1950s and it's basically trying to understand the question of can machines think and thinking and intelligence is very hard to define as we know so is it possible for us to create a bot that passes the Turing test and by passing the Turing test we mean can it act like a user look like a user as we talked about earlier but can it act like a user and and communicate with everybody so that the users aren't sure whether it's a bot or if it's an actual user and that's what we're trying to to achieve with this bot that we're we're talking about today another part of intelligence is is it giving you the information you need so it's one thing to just be able to fool people into thinking that you're a real person but are you getting the information from the bot that is pertinent that's relevant and isn't just some canned response that could still trick the user but at least it you know it's just not sure so that's part of intelligence teaching so we want to be able to teach the bot either through us as the administrators or from the actual users that they're interacting with so that's that's something that you know if if you're intelligent you can you can you can teach things and you can also learn and that's the next point so the bot needs to be able to teach things and he needs to be able to learn things and that is kind of going a little bit past the turning test but to us that is what intelligence is so why do I need a bot I guess my first my first question to that is do you there's plenty of examples and I'll show you some screenshots in a second of of when we've used a bot primarily we've used them to teach asynchronous courseware and classes so we've created a classroom instruction and we'll have a bot teach that class so the bot knows everything it needs to know it may have answers to some of the questions that students will ask and it will walk around and behave like a classroom instructor and it can teach an asynchronous class and that allows us to do training that students can come in and get 24-7 365 and we don't need an instructor present this is a little bit more realistic and it provides a little bit better access to certain functions that we can go into later that that you might not be possible with just scripted prems and then also we use the bot to demonstrate future possibilities so we might integrate something that you know might not be fully bot like we'll say but at least it demonstrates a future capability and that will allow that's that's something that you might not be able to do with without a bot so I guess the question is you all are probably familiar with the NPC functionality in open sim if you're not I highly recommend that you look it up and with that there's a lot of stuff that you can do and it's great so sometimes you might not need a bot if you don't need intelligence if you don't need some of the features that we're discussing now then maybe an NPC is the way to go it's easier on the server and there's there's a lot of good reasons not to use a bot but I will say there's a lot of good reasons to use a bot and one of the things is it's ultimately extensible so you can do pretty much anything you can dream up with a bot and I think that's that's an important point to make is that with the NPC you're limited to what open sim is implemented as opposed to if you're using a bot like we're discussing there's a lot that you can do so these are some of the examples that we've used a bot for and so some of these were the top left picture is we use the bot to you know kind of discuss with students where they were at what they were doing we use them as a tour guide in the top right similar functionality we use them to you know kind of talk about what ship they were on where they were going what their objectives were and had them move around in the bottom left we actually used the bot as an instructor so the students could you know ask what their objectives were they'd be told what their objectives were and if they got things wrong the bot can tell them you know hey maybe you should do this and then finally on the bottom right this was where we actually did asynchronous teaching for the students and so the bots actually ran the classrooms the students walked in they would start a classroom session and the bot would teach the entire class so some bot capabilities we've been talking about bots and what they what they are and what they're not but really what can the bot do so one of the things I don't know if you've noticed or not but next to me has been Alan he's been very patient and quiet the whole time so if you if you direct your attention to local chat and just to remind you to please stay off of that for the time being we can communicate with Alan and he can respond back so hopefully everybody seeing his responses to us so not only can Alan obviously log in and log out from open simulator he can respond by name so he knew who I was we're not going to show following an avatar because I don't want to run around the stage right now and have him run around but he can also just articulate so we can have him go ahead and wave to everybody hopefully you guys saw that so let me do it again just in case so just articulating is important obviously we use gestures when we're talking with one another and having the bot be able to respond and just articulate is important part of passing that attiring test and making people believe that he's real so I guess I'm gonna chat with Alan so Alan where are we and he knows where we're at so he says we're at the open simulator community conference so Alan why are you at the conference so he's here to demonstrate his capabilities okay well Alan what are your capabilities and you can see there hopefully you saw that while he's also answering the questions and carrying on a conversation with us he can just articulate at the same time there's a lot of questions that we can ask that are you know not necessarily pertinent to our our our conversation at hand so Alan what is what is your favorite color his favorite colors blue that's interesting so you can see just you know simple things like what's your favorite color some of the things that seem kind of innocuous or are not important those really add to him being a a intelligent bot will say but also one that can kind of carry on a conversation somebody might just see somebody standing around and just start a conversation and so you kind of have to have that pre-programmed knowledge in there one of the other things that that we're talking about here on the slide is to provide relevant real-time data so just being able to respond to what's your favorite color obviously anybody can program that but when we start to do things like Alan what's in the news and you see he he throws back a lot of information there and and that's really just to demonstrate what that is and it's really the Google news feed so what we can do and say what's in the news we can actually have Alan go out find out something that's actually in the news and he could then go in and kind of drill down and give you more relevant information so that was kind of you know all of it at one point but the goal with that is to kind of you know show some of the capabilities of using real-time data one of the other things is Alan what is the weather like in Fredericksburg Virginia where I'm at right now so he's saying currently it's it's cloudy and 78 degrees and that's right so we can we can pick any city and he can figure out what the weather is like so you know if you give him a home location let's say and say you know Alan you're in Fredericksburg now and then somebody says hey what's the weather like where you're at then he can respond with the real data and that kind of lends to you know making the bots seem more realistic one of the other things that we have on the slide there is is obtaining knowledge so what we can do is we can actually train Alan so if he responds that he doesn't know something we can actually tell him the correct answer and another feature is the admin feature so what I can do is say Alan show me the admins and there gives me a list of of the admins that we have so what that allows us to do is to to tell Alan to go away and you can see he logged out and said goodbye so with the admin functionality what we're trying to do is allow us to do some of the obviously admin functionality and so a general user can come up he can he can talk to Alan and he can ask all these questions interact with them but certain functionality we have reserved for admins such as teaching so that somebody can't teach him you know something profane or inappropriate or teach him something that's wrong so we have admin functionality for that we can add admins so if we want an instructor to be an admin then we can do that and another thing that we can do is we can have Alan log in and log out using the admin functionality so that's that's pretty much most of the functionality that we have right now and you know if you want to afterwards or maybe during discussion we can show you a little more of that and I'm going to get Alan to log back in now so that he can be here for this conversation so feel free to feel free to to talk if you want in in local chat now if you need to but Alan might respond and we're not sure how how Alan will respond so now I'm going to kind of discuss some of the technologies and libraries that we're using and try to give you more of an understanding of how this spot works and how he is is programmed so here's some of the technologies and libraries we're using Alan's mostly written in C sharp he's using .NET Frameworks we typically are using mono to run him so he's usually on a Linux server so we're using mono to interact or have him run on a Linux server and Dalia Alan only responds to his name in a capital A so you'll have to change that to a capital A if you want to talk to him we're using lib open metaverse which I'll go into in a minute amel which I'll also go into yes he is very formal and and then finally we're using Apache for for a part of the bot control that we're we're going to talk about so let's get into some of these technologies so lib open metaverse is actually a collection of .NET libraries that are written in C sharp and that allows us to interact with virtual world so lib open metaverse is sort of a what you could use to make a viewer and so Alan is is acting like he's logged in via viewer like your eye would that provides us with an API that lets us interact with the virtual world we can log in we can log out and most of the actions that you as an avatar can perform and so that's very powerful that that kind of contrasts with the npc functionality where you're sort of limited to us to to lend in scripting language but with this API Alan is is basically a real a real user at this point as far as open sim is concerned and that allows not only him to behave like a real avatar but we can program his behavior and we can also interact with other programs that are that are not in the virtual world and that's how we get our relevant real time data so amel i've probably mentioned this a few times so what is amel so amel if you're not familiar with it is an artificial intelligence markup language that's very similar to xml it's an xml based language and it allows for customized responses for from a program so what that allows us to do is to program alan and allows me to you know say hey what's in the news but i can also say alan show me the news and he'll do the same response so we can actually program both a same the same response for multiple questions and we can also do multiple responses to the same in question because when somebody asks you a question and you don't always answer the same way and then we're also using a program from sharp and this is an amel interpreter that's used to interact with the lib open metaverse and so that and somebody found that earlier actually and pasted it into group chat but the program sharp is is basically an amel interpreter that's written in dot net which allows us to use that in our dot net program for alan this is a high level architecture and hopefully my um pointer is showing up on the yep okay um so at the top here we have the web interface um and i actually have a screenshot of that um and um yeah alan's saying some weird things now um so the web interface allows us to log alan in and out and it also allows us to send messages as alan uh and and so we've we've kind of started to build more on the on the web interface to provide more functionality um but but mostly it's for logging in and out um the program sharp is what i talked about earlier that's how we interpret the amel files and that's how we get alan's responses so the bot program here in the middle actually uses program sharp uh when the bot receives a message um he sends it through the program sharp and um then that allows him to um figure out his answers to it we also have real-time data here on the right and that is stuff like the weather so we're getting that from yahoo weather and uh we also have the news from google news and we've also been able to pull some other real-time data and potentially that's a uh a future area of interest that we're looking at and we'll talk about that some in the in the future um and finally at the bottom is the lib open metaverse and that allows us all of the the apis to interact with open simulator so logging in logging out uh interacting with objects um allows us to access inventory and interact with uh and receive messages okay uh so this slide hopefully everybody can read that you may have to zoom in a little on the slide um but this is sort of the bot control code and i'm not going to go over all of it um really detailed if you have questions feel free to to message me my contact information will be on the on the last slide um and we can have a discussion um but uh basically the the login uh function here is is how we're logging in to open sim um and and you can see we have a seat location and that's where we're going to log into typically what we do is we have alan log in and sit on an actual prim um and that allows us to do a little more uh interaction with him and certain things within the environment because part of our challenges is uh when we log in as a as an avatar we have to have alan be able to see what what we see and do what we do as humans when we are in uh an open sim so typically you log in as a viewer and you're driving you're driving your experience but alan's logging in just like you or i would uh and we don't have a way of doing that so um usually we're interacting with him with prims um and as you can tell uh you can see client network login right here um that is uh basically where we we have a um configuration file and we uh put in the first name and last name just like just like you would have um so alan is just like a real avatar would be there's some interesting chat going on with alan i'm glad everybody is is getting to enjoy to chat with him so the login is pretty standard um the next part is is the part of the web interface i was telling you about so you can see at the bottom we have a little screenshot and we've been we've had close to six or seven bots uh in one actual um virtual world before for one scenario so you know alan is is just maybe one of those seven bots that we would have and you can see usually we have a web interface we have a login at the top right um and then we have alan and or the name of the other bots and we have their um their status so either they're online they're actually in the virtual world or they're offline or they're logging in and then we have you know the ability to log them out or log them in depending on on what their status is and the the code that you see here um is just how we um how we see the um how we send the message to the php page that allows us to see alan's current status or the bots current status um and so basically what we're doing is we're writing to a file and sending a message to the php page that that gives that alan is giving to the php page so um he is intelligent in the fact that he is the one that is sending messages uh to the web page to say his current status uh we don't rely on on listening for a process id okay um so now we're going to talk about detecting users that was one of the um features that we discussed earlier um and i i didn't really show that and the reason is because that we turned that off for alan for today because as users get close to him and as users are in the area he actually responds to you by name um and that's that is an important part of making him more realistic so uh we typically have two four two parts to um having alan uh detect users and part of that is because as i as i mentioned earlier when he logs in as a as a typical avatar would um you or i would be able to see an avatar approaching us and we would respond appropriately to that person um but alan doesn't have eyes obviously so uh we have to tell him that people are nearby so the code you see here is part of um a alan scripting language object so a prim with lsl script on it and um what we're trying to do is is use the uh ll sensor to figure out what's in the area and so when something's in the area and an avatar's in the area we actually send a message um to alan and then he sends it through his code and that's how we figure out who's around so what this uh gives us is the ability for alan to initiate a conversation so a lot of times when you you know are talking to an intelligent agent uh you have to initialize the conversation but with this uh when you approach alan he can say hello my name's alan or whatever our response is at the at the time and he can actually engage you first and i think that's an important part of making this experience more realistic because a bot an unintelligent bot i would i would say um only relies on um um you know being being pinged um or or receiving stimuli so to speak whereas with alan he can actually be engaging and he can be the person that starts the conversation so uh this is one of the features that we worked hard on just to to ensure that and the next part is actually how we're detecting the users um in alan's code and in alan's code we're receiving an im from ourselves uh or from the the object that we just created and so now alan can get to the name of the person and can respond appropriately um so that i think that's important uh the next part is detecting chat so obviously uh we've been using that extensively right now and alan has been responding to all of your chat messages hopefully correct it's been hard to try to keep up with both um but um this part is alan receives a message uh from the local chat and then he goes through and parses it and figures out what he wants to do with it so um if there's any uh chat that comes from the bot or the detection object uh and the detection object is what we use to uh figure out who's around alan um then he's not going to use that information here in this in this function and we we've seen that prims can actually start to spam alan so we don't want um alan to respond to prim messages unless they're from us so we instant message them to him we just want alan to respond to local chat and that's what this is doing um and one of the other things that we've noticed is um if you don't um make it so that alan only responds to his name um and that's down here um so if alan doesn't respond uh if alan just responds to everything then you can get to a point where bots can actually start talking to bots and they constantly continue to talk um so what we try to do is is eliminate that possibility and we we require the bot to be addressed by name um so that is a little bit formal um and that is a limitation right now and that's something that we can you know potentially look at in the future but for right now we require people to address them and that's sort of how you would do it in the real world anyway um so that's not really something we see as a downside so this is some of um the processing of amel um and what what we're doing here is actually showing how we're getting the google news feed so um first thing we do is we allow we we program the responses into the amel files that we talked about earlier uh and and if it's something if it's if it's a special case um so if it's the news case like we when we talked about when we said alan was in the news um we actually call up a function in the amel file and then we go here to the discode and so what this allows us to do is allow multiple questions um yep just like that justin um so what that allows us to do is um have multiple questions lead to the same response so if you say alan what's in the news he does that and if you do alan show me news um he does the same thing um so and that's sort of how a real person would be would respond they understand that there's multiple ways to get the same answer um and that allows us to just program you know in amel the different responses uh are the same response for different questions um and the code that you see on the powerpoint is actually how we're going and um getting that information from the web um and and as i discussed earlier part of part of the the benefit of using the system like we have now is the ability for him to get relevant real-time data um you know if if you ask a bot what's it like there and then they say you know i'm in a computer or whatever that's not as engaging as saying the weather is nice out or the weather is cloudy and stormy so when a user interacts with the bot and they get you know the actual results for the weather where they're at that's really powerful and it's it it helps to uh to to to engage the person and the bot more um same with the news um so what's in the future so some of the things that we've discussed um at at tech wizards is is doing facebook integration um so a lot of people might not have facebook um but i think the majority of people uh at some point or another have some sort of facebook um and so why facebook integration everybody's integrating with facebook um but for us um facebook really gives us um a lot of good information to be more engaging um so imagine if you will you're you're talking um with alan and he says oh how's your brother um so that would be very very difficult to do um um the normal way i would say or or the the i guess the old way of doing things um where we would have to store information um alan's brother is fair to partly cloudy um where we would have to store information for each user but if we could store a user's facebook information one time um then we could go and get all the publicly available uh information on facebook about that person and then that allows us to um use that information and be more engaging so um he could say hey uh you know how's your brother uh happy birthday so he'd know your birthday uh he would know a lot of information about you from your facebook if you put that information on there um but i think that would be very powerful in giving a more engaging experience um and and he'd be able to you know know your friends so he'd say hey have you seen so and so lately um and and and he'd also be able to talk to you about you know your interests so if you like certain things or you have favorite movies or whatever um he can use all of that information and be a lot more engaging and potentially uh you know i guess uh show the user that he's he's more than just uh uh responses that are canned um similar to that is storing experiences with users so if you're um you know if you come and see alan uh maybe he stores how long uh or when you when you last talked to him and then when you see him again he says hey i haven't talked to you in a while if it's been a certain amount of time or maybe he says hey i just saw you yesterday um so maybe his responses change based on past experiences he's had with you um and that sort of goes into emotions and and so he you know if you're rude to alan constantly nobody be rude to alan but if you're rude to alan constantly um maybe he gives you shorter answers maybe he's not as nice not as forthcoming uh maybe he doesn't greet you when you first come up because you're rude to him um or maybe he's very nice to you if you've been nice to him in the past um so that's that's things that you know also provide a more engaging experience obviously um intelligent actions so um you know having alan respond more intelligently um is is obviously something we're always working forward uh working toward but also having actions that are more intelligent so maybe he goes and sits down if he's been standing for a while um maybe he goes and sits down if you're talking for a while and he suggests hey why don't we go sit down at this table um there's a lot of different actions that people do in the virtual world um and i i think that's important um i'm getting okay um and finally uh more real-time data combined with responses we've done news and we've done weather and there's a lot more that we can do with that and finally ignoring in input errors so if you spell alan's name with capital a he responds if you if you call him alan with the lower case a he doesn't respond so uh we can be smarter about that um so hopefully we have a little bit of time for questions and discussions okay we do have uh we have about 15 minutes now or a little bit less than that actually uh about 10 minutes maybe for questions and answers actually redundant about five minutes so it's going to be short folks if you can go ahead and ask your questions i'm reading case give me two seconds and hey uh can we can we pause on asking alan questions real quick while we take real questions please um so k asked for the purpose of training have you integrated outcomes from online tests or quizzes in other words rather than ask alan the weather does alan go into a test database discuss wrong answers on quizzes or tests um that's not something we're doing right now most of what we've been doing as far as um training is more objective based so um you know did you did you meet the subjective did you go here did you do this um so we haven't done that yet that's definitely something that would be um relatively easy to do um and it's definitely something where alan could actually be the quiz um quiz giver um so alan could actually ask questions you could respond to alan and he could immediately give you feedback and give you say you know either it's right or wrong and give you the right answer um so that's that's definitely something that's possible with this framework um and it's just not something we've explored yet for our purposes and he did answer your question i see that now are there any other questions i think everybody else likes asking alan questions more than me which is fine all right and at this time then we'll we'll go ahead and uh and take a break and uh i believe kenny will probably be here with alan to have any more questions afterwards and uh thank you for all being patient and being such a good audience and uh not typing in the first section of the presentation um we hope you've enjoyed this this session i sure have and uh don't forget that uh you can log on for more uh sessions and times to get the answers you need at uh http http colon slash slash conference dot open simulator dot org and please when you do leave don't all stand up at once that's a little bit harder than the servers and thank you for being such a good audience thank you everybody and actually can i answer doug's real quick uh doug had a question everybody still hear me uh okay um so doug we have not looked into alternate ai or behavior engines yet um mostly we've been using amel because it's worked for us um and we're we're not we're not trying to um i guess we're not trying to be uh more intelligent than just trying to pass the turing test and be engaging for the moment um but that's definitely something that that probably needs to be looked at in the future um and if we want to go uh you know down the emotion path and and learning we'll probably have to um do that yes definitely yeah we are um we are definitely interested in that yeah okay um so uh if you have any other questions um feel free to uh contact me my contact information is on the slide uh it's just kenneth dot all wine two ls at tech dash wizards dot com um you can also google my name and and find me on social media but we'll be more than happy to to answer any questions that you might have or you can give me a phone call