 question and answer at the end as well. So in the next 15 minutes, we're going to guide you through the process of taking a scholarly approach to your virtual world educational activities. And as we all know, virtual worlds are continuing to be an expanding field and many of the studies on virtual environments that have been done over the past 15 years have been fairly basic and exploratory. So we all have a great opportunity to contribute to the field of virtual world education by evaluating our virtual projects thoroughly and disseminating that knowledge through publishing and presenting. And so we're going to cover various project types, creating learning objectives, the potential types of settings you might want to use, the study design approaches, including a little bit about IRB approval and data collection as well as some tips for publication. And so in my next slide, I'm showing some data that I'm sure you're all fairly familiar with, but it just illustrates that there are huge opportunities for networking, collaboration, and scholarship with a worldwide network of learners and educators using virtual environments. Likewise, there are opportunities for publication and a recent review of virtual world related healthcare studies show that there are publications in the literature from educators around the world. And potentially those could be our collaborators. So stating the obvious, open simulator community conference and other virtual conferences are great opportunities to disseminate this work. And many of us are doing that, but as educators who want to not only share what we've done, but help others succeed, we need to take that next step to put more of the information out about what we are doing and how we're doing it so that others can replicate that work. And to do that, we really need to publish it. So first of all, I would like to talk about the different project types and take a quick poll from the audience. First of all, as you think about your project and what you might be working on right now, take a look at the various categories that I have up on my slide. And I have practice and simulation. I have games, distance learning, collaboration. I also have role play, informal learning, visualization, dramatization. Those are all various categories of work that virtual projects tend to fit into. So what category does your project fit into? And if you could type that in the local chat. And if you'd like, you could say briefly what kinds of learners you're working with and what you're doing so that we can tailor our discussion, but you don't have to. You could just type in one of those categories as something that you're interested in doing or have thought about doing. Okay, I have some distance learning from David and practice and simulation from Max. Hi, Max. Good to have you here in the audience. I believe we have one slide that shows some of your work down the line here or two. Nova visualization. I have good representation from the Vibe group today. All right, very good. So, and at the end, we do have a short handouts on a note card that will help to guide you as you start to think about project development in any of these areas. And maybe you're planning to start a new project, maybe or try to venture into an area that you haven't ventured into in the past. This will guide you step by step into how to storyboard and conceptualize the things that you might need for your new project. Because we wish we had time to go into that today, but we have so many other things to cover. So we have here a collaboration from 12th Night, Ellie Brewster, Gender Studies. Yes, hi, Ellie. Role play for literacy. Everything but role play and distance learning, okay? A bit of all of those altogether. Okay, excellent. So we have a little bit of everything. Very good. Well, that's wonderful. And so next, Carolyn will talk more about formulating good learning objectives that will help you to measure outcomes in each of these areas. I imagine most of you know about learning outcomes, but I want to go over it a little bit more. And sometimes in education, we almost forget that. Just imagine that you're going on a trip somewhere. Where are you going to go? How are you going to get there? And how will you know when you're there? So in education, where you're going, that's your outcome. How are you going to get there? That's your lesson plan or your project design. And how will you know when you arrive? That's your assessment. Now, many of you are in universities perhaps, and you may know that universities now require, accreditation requires that you have accessible outcomes. And so this has become, while in the past, that wasn't something in a lot of higher education situations. Now it's become top of the list of what we need to do. I'm missing a slide. I'm missing a couple of slides somehow. This is not good. Okay, there's two different kinds of assessment. There's formative assessment, which is what you do while they're learning. So this could be quizzes or... I'll just pass you... Okay, yeah. I don't know what happened to my slides, they were in there. That's not working either, Rachel. Okay, well, so what else? Maybe go off of the Google Doc, just to talk through, and I'll get your slides up here in just a second. Sure, thank you. Okay, to make a learning outcome, first you need to decide what you really want them to know. A lot of times what we assess is not what we really wanted them to know. Also, most of you are probably somewhat familiar with Loom's taxonomy. Is anybody not familiar? Just type no or in in local chat and see, so I can see. If you're not familiar, type an in in there. I'm just curious. Okay, so everyone is familiar. One of the things about virtual worlds is it's a great way to work your way up the ladder of higher level thinking. Thank you, JJ. So this is a modified Loom's taxonomy, and a lot of times what we assess is remembering. Those definitions, just memorizing those. That is the lowest level of learning. Virtual worlds are a great way to get up to the higher levels of learning. Analyzing, evaluating, creating. And so here is a great list. If you can click next for me. Okay. Thank you. Terms, should I read those to you? Probably not. But you can see that there are a lot of different ways to, different kinds of learning objectives that you can have. Students can formulate. They can calculate. They can categorize. They don't have to just memorize or sort or quote. Okay, so here's some examples. Next slide. Of some learning objectives that Rachel and I have used. And the first one I actually used this past week with my students who are going to be elementary teachers. The student will be able to create a tour through the cell explaining the organelles and their functions. And since my students are going to be K8 educators, their requirement was they had to be able to explain it to a third grader. So they couldn't use any language, a third grader and an 80 year old. So they couldn't use any language they wouldn't understand. The student will analyze a scenario and evaluate whether the environment presented is a healthy one. The student will design and create a virtual habitat demonstrating the requirements needed to maintain life in a closed environment. Okay, next slide. So here's some possible outcome categories. There's concept acquisition, which is just the content. What do you want them to learn of this unit that you're teaching? Learner satisfaction and motivation. Learner behavior in the virtual world or outside the virtual world, which one of my grad students is finding quite interesting. Performance in the workplace and on work outcomes following the virtual world experiences. So did they learn in the virtual world what they needed to know? And so here we have some learning strategies, some things you could do. And this kind of goes back to the slide that Rachel showed. Active learning strategies, the problem-based, case-based learning, collaborative learning, experiential learning, skill acquisition activities, active discussion, discussion during a lecture, back chat, which is really, I find, very powerful technology. And then there's that passive learning strategies that we often rely on, the lecture and the passive reading. Virtual worlds are perfect for those active learning strategies. They don't set themselves up quite so well for the passive. Rachel, I'll pass this on to you to talk about designing modules. OK. Well, now that we have learning objectives, we can move on to the next step of defining what kind of environment we'll use. And I'm going to show some environments that I've used for our learners. But certainly one thing to keep in mind is that functionality is really important in virtual environments. It can be really beautiful and very detailed, but if it's not functional and if the learners can't learn from it, then it's not achieved the purpose. So this slide shows the south side neighborhood of a bill that we call SPH Places. And it's part of a public health simulation for medical and public health students on the social economic determinants of health. And it's being used by our students for that purpose. But a very different bill is the East Africa Traveler Simulation, which many of you have visited, and I've been so thrilled to host a few tours on that, on the Vibe Grid, where travelers, people that are interested in global health, can learn about working with limited resources, basics of cross-cultural interaction, traveler safety. And we've created this more for medical students and residents who are traveling to global health sites, but certainly anyone who has an interest in East Africa could certainly visit it. And we're currently building a simulation for the West Africa region, which includes cultural factors like burial practices that have been responsible for Ebola transmission, as well as personal protective equipment for healthcare workers. And that's currently on the Moses Grid, and it's nearly ready for beta testing. But if you're already on Moses or have access to Moses, you can certainly hop in at any time. And of course, the environment really needs to be tailored to what the learners are doing, like I said before. So let's talk a little bit about what learners do in these environments. So the first example I'm going to give is the Biochemistry and Genetics Labs on Genome Island, UMACS, on Vibe. And this environment is specifically tailored to this purpose, and it's been hugely successful for students that are learning these concepts, and they can go and practice at any time. The next slide shows a giant cell that's also on Genome, and learners here can understand cell physiology by walking inside a cell. So the environment itself is the cell itself. So you don't have to have a large spread out simulation. All you need is just this one cell that the learners can use. The next slide shows an activity on Biome, which Chloe has created that allows students to carry out a forest diversity study. And this is patterned after what they would do in the physical world, but it's been recreated in Biome. And then I'm going to switch back to the medical field again and say that health professional students, and this does not just mean medical students, it can be nurses or occupational therapists or pharmacists, but they can practice patient interview skills. And there's a lot of data, a fairly significant amount of data that shows that you can teach empathy, you can teach how to deliver bad news, you can teach motivational interviewing to try to get people to see how certain health habits should be adopted, as well as cross-cultural communication. And the military has done a fair amount on cross-cultural communication, because they kind of had to. And the next slide, again, switches back to public health. And we've used the STH Places Southside neighborhood to allow public health students to do a community needs assessment. And so they go around the neighborhood chatting with community members and learn a little bit more about the community so that they can think about the kinds of interventions they might want to develop to make it a healthy community. So I borrowed this slide on gamification from the VWBPE talk by Hughes, because I really think it demonstrates the elements needed to make virtual world activities engaging. You have to think about the mechanics, you have to think about the reward and what behaviors you want your learners to exhibit and how you're going to measure those behaviors. And so some of the things that we've done have tried to get at some of those elements. So in the East Africa Traveler Activity, we have a student-square health tracker that shows their exposure to diseases. And one of the motivating factors of the activity is to stay healthy. And if you do get exposed to a disease-causing agent to try to get rid of it by treating yourself as quickly as possible, look through your manual, figure out what you need to take and use it so that you can stay as healthy as possible and your scores tracked in that way. And we did do kind of an adaptation of that for the public health bill as well. It was modified to include a stress tracker. And so I want to give a shout-out to Kira Komarov and Wizardry and Steamworks, because she did a lot of the scripting behind this. But these kinds of trackers can be used for assessment. And while this is one way to assess learners by tracking their progress and looking at a score, there are other ways to do that as well. And I'll let Carolyn share those with you. Okay, now we're in the right spot. Assessment is how you know that you got there. And in traditional classes, probably your classes, that was homework in the test. But now we know there's a lot of ways to assess, a lot of better ways to assess. What's important to know for our presentation is you have to know, in order to know that your students learn, you have to assess. For scholarship, which we're going to talk about, you have to have assessment. So assessment is essential in virtual world activities, just like it is in any learning activity. Okay, next slide. Formative assessment is what you do when it happens. So how do you know that the student is learning while they're learning? In the virtual world, that could be the observation of student behavior. It could be questioning the students. It could be practice quizzes. It could be a health meter that's working as they go. Summative assessment is the kind of assessment you do at the end, the test. But in virtual worlds, that could be a post-test. It could be a concept map. It could be a presentation or a project. Or it could be done in-world, an in-world presentation or an in-world project. Okay, next slide. Okay. That's yours. Yes. So I'm going to go ahead and put it all together just to give you the big picture of what we've all been talking about. One of our modules is designed in this way. It's the module on the impact of environmental factors on health. And our learning objective was to have the students discuss the distinguishing qualities of healthy neighborhoods. And so we selected an environment that had quality low-cost housing, clean parks, safe streets, sidewalks that allowed lots of social interaction, a new clinic, a new public library and community center. All sorts of services like employment services and childcare and community gardens. Spaces that allowed social interaction. And that we consider to be a healthy environment. So we let loose our students into this healthy neighborhood. And we had them do a scavenger hunt to find all these healthy places. How did we assess that? Well, we had the students take a picture of their avatar next to a healthy place and upload it into the course management system for their instructor to review. But they also had to annotate that photograph and say why it was a healthy neighborhood. The other thing that we had them do was wear the health tracker and so their health score increased with the healthy spots that they found. And that was recorded on a scoreboard in the environment. So we had two measures. One was qualitative and the other was quantitative. And as you might have observed, this can all be done without the instructor actually being in the environment with the students, which was a really key part, I think, to the success of this activity. The instructor of this course, I don't think he ever logged in, to be honest. But he was content with getting these materials back from the students as part of an assessment, uploaded into on course, which is our course management system that he's familiar with and he was able to assign them a grade. And this model has been used successfully with other activities as well. And I do think that one way of getting more courses done in virtual environments and more instructors to buy in to the use of virtual environments is not necessarily mandating that the instructor has to get in there and get their hands dirty because they may not have time to do that. So moving on, now we're going to discuss the different approaches to study design. And I want you to hold on tight here because we're going to go on a wild ride, but it's going to be fun, I promise. So my first question for the audience is how many of you or has anyone done a qualitative study before? And feel free to type yes in the chat. Anyone done a qualitative study? Focus groups, interviews, open-ended questions that students answer. Okay. Thank you, Chloe. One yes. Okay. So there are several basic approaches to qualitative studies. Okay, one more. Hi, Connie. The list is not exhaustive, but the focus groups, interviews, and open-ended questions, observations would be another. But there are certain types of approaches to qualitative analysis as well. And phenomenology is really that lived experience. Basically what we do with that is we ask the subjects certain questions to try to understand what their lives are like, and then you try to conceptualize that lived experience in your analysis. Grounded theory is another approach, which actually tries to develop a theory behind whatever phenomenon you're observing. It's kind of interesting that I've done some grounded theory studies. And then ethnography is actually one of the approaches that's very common in virtual spaces where there's a lot of observation. So you're trying to understand the life, the culture, what people do just by watching them and interacting with them in a sort of removed fashion. So the good news for qualitative researchers is that chat logs collect your data. And so if you use chat, which a lot of us do, then you kind of have an automated data collection process that you can just go through and review. Okay. Alrighty. So let's move on to observational studies. And I've kind of just done a cross-sectional study. Right now I asked you a question, how many folks have done a qualitative project in the past and some people answered yes. And so I could assume that anyone that didn't answer yes has answered no. And so that's my cross-sectional study. And I've just performed research. That is fabulous. However, there's a downside to the cross-sectional study. What if you decided to do a qualitative project tomorrow maybe as a result of what you learned today? Then I have not captured that. But a cohort study would because basically with a cohort study I'm following you over time and I'm watching to see what you do or if you develop a disease or whatever it is and from the medical side it would be disease development but from the educational side it may be some outcome or the other that I want to understand a little bit better. Or a case control study actually compares individuals that may have been exposed to a condition and intervention, whatever it is, or that have a disease in the medical field with those that have not. And so I'm trying to understand the differences between these two groups. Cohort studies are generally prospective meaning that I'm going forward in time. Case control studies are generally retrospective meaning that I had to look back to see oh you had this disease and or this intervention. Actually I should not be using disease because we're talking about education here. But for example we had some students in the global health activity they went to Kenya and that was their intervention. And now we're following, we're going back to them about 10 years later to say how are you different from a group of control students that did not go to Kenya. That's a case control study. So that would be an educational example. And observational studies are good because they're fairly easy to do. They don't require a lot of funding. You can study things that kind of happened in the past. But where you really start to get serious with research is with your experimental study designs. And I only mentioned this first one to say that a lot of virtual world articles use this approach but it's definitely not recommended. The one shot case study where there's no control group and it's really difficult to justify the results because they might have really just happened by chance. So this is the what not to do part of research. Don't just say, well A, got this treatment and then we did this measure and that means it's great because it really might not be. All right, so let's move on to slightly stronger study designs. The two group post-test comparison is a design in which you have this a group that gets the intervention and a group that doesn't and then you do your measure after that. Now the main advantage of this is if you randomly assign the participants to each group that randomization controls for your demographics and for any changes over time. And because you didn't have a pre-test there's no... the pre-test doesn't influence your results. So this is a pretty decent design. I mean it does have its downsides but I think it's a good one to try. The next one, again, is a... another design but it's not... it just has one group and it has the pre-test and the post-test and a lot of people will do this if you don't have a large group of students and you just want to do a quick study with a pre-test, you do your intervention and you do the post-test. It's really difficult to generalize these results without a control group though and so it's a fairly weak study design even though it's very popular. If you can do something a little bit stronger than this then it will be better received. So here we have two groups. We have an experimental group and we have a control group and again we have a pre-test and we have a treatment or an intervention and then we have a post-test. So great that we have a control group poor that we have no randomization. So while the groups may be different at the end of the study it may be that they were different to begin with. So a lot of times when we're doing virtual world studies we'll have a group of students that are fairly motivated maybe they'll volunteer for the study and so they may be a little different from the controls that didn't volunteer and so that intrinsic difference may account for your study outcome and there's no way to really tease that out unless you randomize. So our next example is a randomly selected experimental group and that's compared with a randomly selected control and basically you have your pre-test your treatment and your post-test and the key advantage here again is randomization so any differences that appear in the post-test will be really the result of whatever you did educationally with them rather than a possible difference between the two groups to start with. This is a classic type of experimental design but it has one downfall and that's the possible effect of the pre-test so the pre-test might in some way influence what the subjects do on the post-test because they've already been sensitized to the things that you're interested in. So the next design which looks like we may have lost one slide I'll talk you through it is basically a design in which you have exactly what you have on this slide but you have another group that does not have the pre-test so you have an un-pre-tested experimental group and an un-pre-tested control group and with the help of your statistician you can sort out the effect of the pre-test. Now you have to have quite a few students to do this though so you may not be able to do that with your first foray into collecting data but you should be able to do that as you get more and more students. So next we're going to talk just a little bit about IRBs and so IRBs are not our favorite group of people to work with but any data collection on human subjects that is going to be used for generalizable knowledge is considered research and so if you're working with human subjects which our students are human you do need to generally go through an IRB and so institutional review boards may small attempts at humor may not really have much experience dealing with virtual projects so they do have questions often on identity like how do you know that this person is actually who he or she thinks she is how do you know that they are the age that they say they are what about privacy on the internet well we know that doesn't exist and how are you going to incentivize so Connie might actually be able to tell us a little bit about his experience or his or her experience about passing through the IRB process but here are some good research practices to consider and if you include some of these elements in your IRB application it's not a guarantee that they will like it but there's a good chance that they might look upon you favorably and so identifying yourself as a researcher is always a good practice you don't just land in an environment and start collecting data without telling people who you are and what you're doing next setting up a consent room to provide an opportunity to identify yourself and allow others to potential subjects to ask you questions is a good idea and that can be clearly demarcated in your region marking areas with active surveillance and chat logging with the fact that that's going on is good practice and just be aware that there might be the risk of duplicate participation especially depending on your incentive try to tailor your compensation to the specific context that you're in so if you're in second life use Linden's don't say we're going to send you a check it's kind of silly but some people have written IRB applications like that and then read the terms of service I know we hate to do this but try to figure out which ones apply to the research and be sure that the relevant information privacy risks are included in your research materials that you're distributing to your subjects and be aware that any changes in terms of support may affect the participants and you might have to go back to them and update the material so very good alright thank you Cody so next we're going to very briefly touch on the after data piece so Heroin has had some experience with this as well and so we can both speak on the challenges of publication but there are two approaches traditional and non-traditional publications journals so peer reviewed journals with an impact factor and searchable databases like ESCO are kind of the gold standard a lot of virtual world educators have written books which have been hugely helpful for others coming up through the ranks and there are non-traditional approaches like creating web based educational information as well as educational software or video and YouTube is just filled with those but a short list of potential journals that you might publish in is listed on this slide and many educational journals now include research on innovative technologies for teaching and learning so it's not all inclusive but these are some of your best bets and I am going to go ahead and put a little plug in for the journal virtual studies and full disclosure I'm one of the section editors for that journal but it is the primary publication for the virtual best practices and education proceedings and what I'm going to talk about on this next slide is actually from my experience with this group and so just for us to know that journals are looking for papers that speak of the ability of virtual spaces to really transform learning or transformed individual it might be one or the other and any studies using an environment where the medium is not face-to-face can be considered virtual studies or virtual or acceptable for a journal like this so and they accept lots of different kinds of papers and papers from really any field of study and I know there's been a call for papers out for a little bit so if you have a paper you've been sitting on or maybe you're smarting from rejection from other venues and you want to give us a try feel free to do that right Caroline I'm going to pass this back to you because I know a lot of our audience must be thinking about well what kinds of resources do you need to actually be able to do all of this I don't know how much time we have so I'll go through this very quickly but to do all of this you're going to need some resources and it may or may not cost money but you need the basic technology you need computers that run the viewers which today is much easier than it was when I started in 2007 you may need help with scripting, building, curriculum design or basic skills you may need money we'll talk about quickly and server support which vibe virtual islands for a better education does provide some of that and you're going to need time and I can't help you with that but it's something you're going to have to find the time for learning, building, designing okay so where do you get this help a mentor find someone who's done it before my mentor happens to be sitting in the audience it's Max I found her right away and she helped me a lot as I got started experienced builders people that have done it before get them either to teach you or to build for you or for your students we Rachel and I can provide assistance and advice we can provide we have already built web page tutorials how to get on virtual worlds how to get your students on and we can provide workshops in a number of different aspects I think we're kind of running out of time money you're going to need money for a server or server space to pay for scripting or building services if you need that to pay for consulting to pay for basic yes and I've used student help which sometimes is free projects in their class or inexpensive so I think we're out of time actually I think we are actually done this is just about the last slide and so we'll throw the floor open for questions at this point and we have our emails up there in case you would like to contact us later time yes please do thank you Carolyn and Dr. Umoran for a terrific presentation this concludes the first day of the second annual open simulator community conference 2014 though the conference programming has ended for the day little field grid is hosting a social event this evening at 8 p.m. PST at the speakeasy dance club we encourage our attendees in world and on the web to visit little field for fun and post conference discussions you can hyper grid teleport to the event with your conference avatar or for more information see the conference program at conference.opensimulator.org we'd like to thank our audience in world and on the web and our speaker staff and volunteers for a terrific first day of the conference we'll begin tomorrow at 7 a.m. pacific time keynote regions with an exciting keynote address from Steve LaValley from Oculus Rift we'll talk about the race to bring virtual reality to a mainstream audience we hope to see you there