 All right, everybody, it's about a quarter to four or so well, I won't tear the podium up. I'll just get started So this is user experience for online education putting UX to the test Just out of curiosity how many for how many people here is this your first Drupal con? Wow, that's awesome. Okay. Well, welcome. That's so great to see so many new people here really shows how the community is growing Awesome. All right, so that's me. My name is Becky Gessler and I've been involved with the Drupal world for four or five ish years I Guess around when Drupal 5 was coming out I used to do kind of independent web consulting Drupal web development that kind of stuff I worked for Google for a little bit mostly on the search team but also had a little bit of exposure to user experience projects there and most recently I worked for a company called University now, which is the Edtech piece of the talk today that we'll talk a lot about that's me making coffee in my office with cats I'm really happy that Drupal con is in Portland this year a lot of really good coffee being roasted out here So that's really great for me. Oh and About these slides there on like they're inside of a Google Doc So it'll be you know open for everyone to look and I'll show the Billy link at the end And that's my Twitter handle and all that stuff Okay, so here's what I want to talk about today first I really want to kind of give you guys an idea of what it's like to be doing user experience For online education. I mean very specifically from for within my company University now what the unique challenges are that we face I'm gonna talk about a couple of different I guess recurring themes lessons kind of things that has been pounded over and over again into my head From from doing work with you now I also want to share with you guys a couple like very three maybe Tools that I've kind of used a bit and found really useful Okay, so let's get started About University now, it's really important that I get this out so that way you kind of everyone understands where I'm coming from So the mission of our company is to ensure that high quality post-secondary higher education Well, that was kind of repeating myself to basically it's to ensure that high quality post-secondary education is available to people everywhere So this means basically that a college degree is within reach of everyone that wants it So it's a very mission driven organization and we operate These two universities we own them one of them is called new charter University It's a hundred percent online offers like associates bachelors MBA degrees. The other one is called Patton University That one has a campus in Oakland, California and also does online degree programs So that's sort of the structure of the organization So has anyone ever heard of any of these guys up here on the slide? Okay, a couple hands cool So this is sort of the world of ed tech That you might read about in the New York Times every day or any other You know news technology kind of website and what these guys are doing is is really cool Anyone familiar with the acronym MOOC? Kind of cringe when I say it but MOOC. Yeah, so MOOC stands for a massively open online course and What places like Coursera, Udacity, Udemy, edX what they're doing is essentially taking this really high quality Amazing course material packaging it up from some of the best universities around the world There's just from the from some of the best content authors and kind of making it available and accessible to people And it's really awesome. I've taken some Coursera classes Really great thing that they're doing and so where does university now my company where do we fit into this ed tech thing? So we we kind of sort of fit that's what I like to say I mean we we certainly embody the combination of technology and education however, we're really different from the other guys because we actually own and Operate to real universities In the sense that you can go to one of our schools and get a full degree program or you know get a degree Engagement for us and success is incredibly important So Daphne Kohler who's the I guess founder of Coursera actually came to our offices And we had a nice conversation with her and I asked her I said what is the role of Coursera in? Keeping people engaged keeping your students engaged and and I thought her answer was really really interesting She said you know We're making that information available for people everywhere and that's that's the point Like we don't care if they complete the course all the way to the end The point is that it's out there and for the people that want to do it to the end They can for the people that want to just access it. It's available to them So for us in terms of student retention, which is something very important to any university Keeping our students engaged and coming back and completing courses is absolutely critical to what we're doing So that's sort of a from a design standpoint. That's a very big difference for university now Okay, but online university right like I'm sure you know some of you guys are familiar with with these names up here Online universities are not a brand new concept. This is not some something we invented There are many working adults that pursue higher education online and these are some of the institutions they attend But our schools are even you know, okay, we're fine We're not the whole ed tech thing exactly, but we're also not the whole online university thing exactly And so here are some of the reasons why we're not like them And I'm and I'm going through these to give you guys an understanding of the Cognitive sort of overload this presents to our users when they try and start to understand what exactly it is that our schools offer So to begin with our coursework is self-paced You can go as fast or as slow as you want as a student You know if you have something going on one week You don't really need to do anything if you don't want to you can come back the next week and work harder We don't have any financial aid or any government-based loans So you pay out of pocket to go to school the school is incredibly affordable We have a monthly or an annual tuition. So we don't do the we don't do financial aid Classes start every Monday. So Whenever you want to go to classes really whenever what you want to begin schools up to you We have this thing called a disaggregated faculty model, which is just essentially means the person grading your exam is not the same Person that's coaching you through the course material and that's also not the same person who's kind of advising you throughout your degree process Which is unique to many schools and also we're something that's called competency based Which essentially means that we are really focused on that you actually know the material not exactly how you learned it So if you come in already having a lot of knowledge about a subject you could theoretically test your way out of a class So for all of these reasons I kind of want to move a little bit, but I'm afraid to this microphone So for all of those reasons and for many others that I didn't talk about We basically have created tons of mismatched Expectations for our users for our students for prospective students when they come to explore our school websites and try and wrap their heads Around what it is that new charter or patent offers them So this is an example of some of the quotes that in user interviews, I'll often times hear with prospective students stuff like well when the classes start because University classes always start at a certain date, right? There's a semester, but well a new charter classes start every Monday And they say sometimes wow this is really expensive and so we're talking like for a bachelor's degree like $200 a month But the idea is that if you're used to this I if this model of paying later Where you're taking out loans and you're not worrying about it until after you graduate with thousands of dollars of student loan debt Having to pay $200 right now seems like a lot of money. It's a totally different model You know when's my final well you schedule your final whenever you'd like and take it at home in your pajamas I mean, it's kind of new We don't have to take in your pajamas, but you can it's proctored via webcam But it's you know, we're not going to judge your clothes or anything Okay, the question is like how long does the course last well with the competency-based stuff If you already know the majority the course you might be able to just test your way out of it You know, how will they grade me the whole things online? And we the person that's coaching you through your course doesn't grade your stuff So these are all a lot of like these are some of the things that we hear routinely from prospective students from new students even And as user experience people It's very much our job to Kind of fix this or at least to deliberately try and help People understand what it is that that makes us different and how our model works And so I kind of think of design as sort of this layer between What our users want to achieve which is being successful in getting a college degree? and the actual Work and doing it so yeah I've I've been with the University now for I guess about about a year and a half and it's it's a hard problem And I've seen some stuff Some stuff that's worked well some stuff that hasn't worked well But as a user experience person, it's been a really wonderful opportunity to kind of Think about problems that do not That are not my own like I'm our team is very consciously designing for people that are not like ourselves And that's a very you know UX thing like don't design for yourself design for others Or rather design for your user So yeah, it's it's been a very it's been an adventure Okay, so these are the three things I want to talk about As kind of items that I've learned sort of time and time again from from working at you now and Doing user experience work So we're going to drill into each of these the first one is just about the importance of talking with people even when they're not considered your Target customer or your target user The second one is sort of stressing the importance of quantitative analytics and quantitative usability results and the last one is about The word customer and the word user and why they're really kind of the same Yet they have different homes inside organizations, which is weird Okay, so talking to people this is the first website that our school new charter launched with It launched right after I joined the organization And one of the things I was scratching my head over was like have we shown this to people before we just going off and making this Website I mean it looks nice and everything but have we shown it to anyone and no we hadn't really shown it to many people Okay, so I was scrambling. I'm like, oh my god We got to show this to as many people as we can but like right before launch I think I was there maybe a week before it's supposed to launch And so I just went around talking to as many people as I possibly could like the crazy guerrilla UX type stuff You know like can you give me five minutes to look at this, please? What do you think this is like that kind of stuff and a lot of the feedback I got was like I Don't know what is this? I mean I'm talking about asking questions like what kind of a website is this? What can you do here? You know like how would you proceed if you wanted to find out more very basic vague? Questions just to try and pick apart. What is it that people are getting from from here? And what I found is that people were like no idea Oftentimes it like the initial thing was that this is not a school It's a bank or financial services thing when people finally got it was a school. It was very quickly dismissed Oh, this is a University of Phoenix type school. They're gonna take my money, but I was like, oh, okay Very not the right impressions. We were trying to get and when I brought this back to my team At that time the feedback was kind of like well. These aren't our target users. These are not our real customers And and really we went we ended up going down this path of weeks and weeks and weeks before it was sort of formally acknowledged Yeah, we're doing something really different and we need to take our time to better explain What it is that that makes us as a school unique? And really I guess I'm using this as an example because yeah Like there are some things that of course are very specific to the to the customer that you're trying to sell to But there's some general kind of Do people understand what we're trying to do? That you should feel confident going out and getting some feedback on and bringing it back to your team Because you know even if you're not the youth like even if this person you're talking to is not your exact user You should be able to at least get across some from very broad things to them So that's one point about you know not feeling Married to talking only to target users because as a startup target users are can be very hard to find and very costly as well To recruit and schedule and stuff for interviews So another thing about not always having access or working with target customers It's the idea of kind of using the ten usability heuristics that Jacob Nielsen wrote about a while ago Jacob Nielsen is a god of usability with the Nielsen Norman group They do conferences and write books and stuff very insightful findings and articles and so There this is linked to at the end of my presentation But the idea of the heuristics are that basically this is like a list of ten things that In most all cases make for a better user experience We're talking about these kind of things like oh do people know where they're at Do they know how to go back to where they came from do they understand? What is currently happening when happening in the interim of them doing an action and so? Performing usability studies task-based usability studies where you're just sitting with someone and asking them to do a thing and Observing them it will very clearly give you feedback about these kind of just interface level interactions And you can almost consider this is sort of objectively does the design we've created Allow our user allow a user to meet their goal So just in this discussion of well, we can always find like the 33 year old single mother working two jobs You know well, okay fine, but this Super tech savvy web developer couldn't even find our tuition page So it's like you know Those types of things getting that type of data can be very useful to broaden your discussion Okay, so this is a piece of raw meat and a bunch of different numbers And I put them next to each other because I think that numbers by themselves are really cold and don't earn like not really that appetizing I Mean beef tartare like say tartare whatever it's pretty good not gonna lie But I mean just like by themselves without any context without any story without any appeal to pathos, you know numbers kind of fall short kind of lame, but Like I'm going back and forth here Numbers don't usually lie You can tell data Points in a way that sort of construes them towards a certain point of view But I think one thing I've really seen is that I do a lot of qualitative Usability testing like I talked to people I listen I write quotes down I get really nice quotes that I put on slides and in presentations and it's like wow people so oh my god That's a really great quote. It really summarizes what our users are feeling. Yeah, but having numbers to sort of back up what Percentage of users feel that or what percentage of users took an action can be really helpful in Distributing the results of the usability work that you do throughout the rest of your organization These are three tools that we use at different points at university now optimizely for a B testing mixed panel for sort of I guess granular granularly measuring actions that users are taking on pages as well as Google analytics, of course And now I think it's unreasonable to say well all usability You know people and researchers and designers should understand analytics and go out and be measuring stuff that way I mean you can't do everything right, but at the same time if you can align yourself with someone in your organization That has data and cares about data and knows how to work with it It'll make what you're doing all the more powerful and allow it to kind of seep into to people that may not be interested in Fluff which unfortunately is sometimes how qualitative feedback is looked at So it's not rocket science, but I just think as a user experience like as a community It would be helpful. I think for for all of us to take some responsibility for actually working with with people that work with data Okay, here's the the last point in the thing. I've learned section Which is that and I believe this very strongly that your customer is your user There are many different depending on how big your organization is but oftentimes there's different parts of an organization that touch a Customer or a user at different points in the life cycle of their involvement with your organization For us. This is something like The person that's chatting with them on the website the person that answers the phone in the enrollment Center if they call in the person that is responding to their technical issue All the way to their student advisor their instructor These are people that do different jobs, but all touch You know our students and then you even have me who's sitting there in the room You know in the office in San Francisco calling you up in In New York you meaning our student and having a conversation with you And I'm learning things about you, but so is everyone else along the entire you know chain there and at least in my experience I have seen that when these groups of people do not Communicate properly or at least as frequently as they should very different kind of ideas and expectations of our of who our students are Form pretty readily and it creates a lot of problems because it turns out that the marketing team is actually They actually have their own set of personas than the UX team And they're not the same people at all and number like so we're selling to these people But we're designing for these people and it just doesn't make sense So this is a this whole thing is a quote from from an article that also is cited at the end from this website 52 weeks of UX and I love this idea That user experience is really just good marketing It's about knowing who your market is knowing what is important to them knowing why it's important to them and designing accordingly So I started off at you now as a user experience designer and I was working Pretty exclusively inside our actual learning management system the actual course platform that that students interact with as it became more and more Obvious to me through the usability research I was doing with our actual school websites that we just were not explaining ourselves properly I started getting more and more involved with that world and so at this point like I'm on the lead of our UX team But I'm doing a crapload of marketing I'm doing a lot of like how do we Display something on a website that helps people understand how online courses work. It's a totally different like Departmental type world, but at the same time it is so fundamentally a part of the role of a user experience designer of an information architect Helping, you know, your your your customers understand what you're all about so If you guys have any marketing related people, oh wait, that's the next slide Right so You know if you have any marketing related people that you work with or in general people that are out like a sales team Interfacing with customers go out and talk to them say hi to them Maybe work with them. Maybe just sit in with them at lunch or something like that And I and I think that type of collaboration is really what allows companies to have a more holistic Vision of who their users are and better design and market to them all at the same time and another argument Taking this even one step further would be to say that this is like my Google hat now would be saying that SEO like search engine optimization is also really just about good user experience and about Creating semantically appropriate metadata rich websites and I and I really do think that's true so Broad and the even further diluting what UX is right Okay, so Any questions on that stuff so far? We can we can like do that at the end too, but What is the thing with I just me personally You know, that's a good question. I just love cats. I love every kind of cat Yeah, I do. I was semi quoting a YouTube video right now, but I don't know if anyone else the e-harm anyone But yeah, no, I just I have I have four cats in New York I'm in California now. I don't have them with me anymore. So it's like a sad parents have Yeah Yeah, that's a great question. So the question was about if we do any user testing with faculty in addition to students so Unfortunately, we do not do enough. I think it's very known that the faculty have their own experience instead of tools that they use And they are not as well Sort of maintained as the student-facing ones, but it's a very, you know, that's like a very good thing to highlight that You know, your user is not necessarily just your student What we do do is talk to students as well as prospective students people that have kind of considered our school or in that decision process. So Yeah Okay, so let's go on to the small bag of tricks section I love today at the keynote mental models, right? I was like, well, she's totally saying exactly what I wanted to talk about today. So thank you Karen. That was awesome Yeah, so we're gonna talk about that a little bit also paper prototyping and Last but not least creating a continuous feedback Mechanism which kind of gets into how we actually talk to people. So here's Jacob Nielsen again same picture and this is his kind of definition of what a mental model is and You know this morning was was great Like I said that that it was talked about and it's really important here to think about a mental model not as How someone not as how a thing actually works but really as how someone imagines it to work And the fact of how it actually works is kind of irrelevant, right? Like it really only matters what how this user experiences the thing to work what they think happens when they will do an action Or when they do an action So, okay, we're gonna talk about why this is important So this is a screenshot of a course dashboard inside one of our courses a lot going on there So I'm gonna zoom in to a portion of the dashboard called the readiness meter Okay, that's this right here There's there's multiple readiness meters actually So the idea the original idea of the readiness meter was that it would be a way for students to be aware of How far they've come in a course and sort of help them understand How much longer they have to go before being able to Schedule their final because at our schools you actually schedule your final exam What you know when you're sort of ready when you've when you've hit a threshold So the readiness meter was both a measure of progress like how much you have actually done In terms of how many practice quizzes have you taken have you taken your initial assessment like those types of things but also a Predictor a predictor of how far you have to go that was the original intent So in some of the original Interviews I was doing with with our first set of new students feedback was overwhelmingly positive It's like oh my god. I like watching the green bar go up the green bar. It's so great. I you know I always lets me know where I am. I like watching my progress helps me feel like I'm getting ahead And so this was great feedback, right? I didn't really ever ask what is the readiness meter or like what is it telling you it's just This is my error kind of just asking what people thought about it What you know how they experienced it? What was their favorite part of the of the platform and this is something that would frequently come up a couple months later it was like Readiness meter like people students were really not happy with the readiness meter. They were really pretty pissed off These were some of the the pissed off quotes They weren't necessarily all in caps. I'm kind of just playing it up But it was like it doesn't get greener. I just I'm looking at I'm reading things and it doesn't get greener Or I go to Internet Explorer and it's here and I go in Chrome and it's there I've just I've given up We've had students like on the verge of leaving the university over the readiness meter and Student just it makes no sense Okay, like when I when I read 20 pages it goes nowhere and I take one quiz it goes up there So just like total infuriation and I really sympathize with them because being an online student if anyone has Has taken any online courses. It is really really hard. It is very difficult work and If you are trying to trust in this little metric idea of like oh I have a readiness meter and it doesn't do things when you're putting in all that work Imagine what that feels like So really what it came down to is that our our students didn't Didn't really know at all how the readiness meter work They just sometimes it went green and in the beginning it would go green more frequently and we had this Formula around it where you know based on the activities you did and how well you did on them The thing would go up and it was overall the logic on it like the idea was good But the logic wasn't well implemented and so our students thought that well every time I do an action the thing will go up But in reality every time you read something it you never got any readiness like if you read 20 pages and spent four hours Your readiness just wouldn't go up like it just wasn't built that way And we did also have browser compatibility issues. That was the whole browser thing But it was this idea like I could trust in this signal to tell me where I am in my course How far I am away from graduating we actually the readiness meter was bubbled all the way up to how far you have to go in your degree program so it became this core metric that Users had placed trust in and just didn't deliver because it you know it it wasn't just about progress It was about different ratios of how well they had done on things and really a measurement of more Assessments rather than readiness So this is also from Nielsen's article about well, okay, what do you do when you become aware of mental models? How do you fix problems related to them? One option is to make the system conform which basically means okay If users think that when they do actions the thing will go up. We change it so it does that we meet what their expectations are The other option here and what Nielsen would say is improve users mental models I Don't know if improve is the right word But I think or maybe just taken out of context it sounds weird But really what he's getting at here is the idea of educating users and showing them How your interface is supposed to work and kind of giving them contextual clues along the way of this is You know how the feature works So these are generally the two kind of pass forward when you have a mismatched mental model situation so What a number of my colleagues are working on right now is completely changing how readiness works And my current understanding of how that's gonna happen Is that we're kind of doing away with readiness because the concept just falls short of what users expect? They expect that when they do a thing it goes green and they can feel good about it So we're really kind of moving towards a model that's about progress about track your progress not your readiness We're not gonna tell you your readiness. We're gonna tell you how far you've come We might also break that out have readiness be a separate metric But really that bar is gonna become about progress. We're kind of meeting users where where they're at where they're thinking about it At the same time, this is something that has been added to the platform Where we've got this nice little hover tooltip when you hover over the readiness meter. It actually tells you This is how that thing fills up So this is kind of the idea of contextually helping users along the way to understand how your model works Okay, so now I'm gonna talk a little bit about paper prototyping which is a really nice way of getting feedback from users in Person so the idea is you know, if you're doing a traditional usability study Task-based in front of a computer, you know, you have someone sitting next to you and you're saying Find out how much this course You know would cost you to take and you kind of just sit there and you watch them do it You basically are giving them things to do contact a student advisor and kind of see how they go about doing the task But you know on the screen, you need to have something designed or at least you need to have something on the screen With paper prototyping, it's great because you can essentially just draw with pen and paper Or you can make something up in fireworks, whatever and print it out cut it up and do a usability study based off that so this is an example of This is actually inside of a PDF, but so This is an example of a mock-up that we made to illustrate an early prototype of explaining how to courses work online courses at our university and What we did was we brought in I think four people and we printed this thing out and we cut up all the little boxes And it was basically this large container element with no with empty and we gave them the put the puzzle pieces We were like, how do you think this would fit in here? Like how does this make sense to you? And we just kind of watch them arrange it sometimes they didn't even arrange it linear What is this called? Vertical they arranged it like horizontally or diagonally all these different directions And it was kind of this very fluid way of getting inside of the user's head And it was paper and the actual tangibility of it that allowed us to do that We also were able to iterate really quickly because of that We found out this page is only half of what the original design was and we realized very quickly This is like information overload and we can't we cannot put this all in one page So we ended up just cutting the page in half and kind of splitting up the usability study for the following users Which ended up being Really successful for us So why does paper prototyping rock well because it's really fast and it's cheap You can make changes on the fly like I said and granted That was like that was a legit mock-up done in fireworks, but you could do this with pen and paper as well And the other nice thing about it is not involving any development resources So I did a presentation at the Bay Area Drupal camp in 2012 about prototyping And I'll link to that at the end of this slide because it's I'll talk about click-through prototypes in HTML There's many cool ways of getting feedback But I think paper prototyping is is really great because pretty much anyone can can do it The other thing about paper prototyping is that what it allows you to do is get a larger group of people inside of a room witnessing usability, you know in practice and actually Kind of being a part of that at an early level oftentimes sort of the key stakeholder person will not get involved in You know a project inside of it from the usability standpoint until the end and then they're like well, you know We actually this feature, you know doesn't make business sense for us to build it So we can't even you know have it in the place you're thinking of so what paper prototyping does is it's since it feels so low Fidelity since it feels so generative and like we're all working together kind of thing It encourages people to kind of come in and speak up and be there at the early part of the process And so I find it to be a very successful method for introducing non usability people to the world of usability And to how fast and iterative it can be Okay, so the last point I want to talk about here is continuous engagement and how to Kind of foster that inside your organization So for us as a school, it's sort of just You know like I was saying in the beginning we we have students who are invested in our Schools because they are trying to get a full degree from us But when I first started working at university now that didn't I didn't really get that I kind of felt the entire time like well Why would they want to talk to me if I'm gonna just say hey? I'll give you 20 bucks if you talk to me for 30 minutes like why would they be invested in doing that? But if you're solving a problem for someone with whatever it is you're building a website a product You know an app like whatever it is if you're solving something I'm pretty sure you know you will have some users that will intrinsically be motivated to make that product better If you're solving something people want to help you solve it because it's a real problem And I was very heartened to see that there are tons of students that we have that want to go out of their way To help us can continue to make the experience that they have better So this is an example email that I received from Coursera actually Yesterday maybe if anyone else is a part of Coursera you probably also got this email So I'm like making this presentation. I'm like oh look at that So this is just an example of how Coursera is trying to reach out to its super huge student base And kind of asked like hey, you know Tell us what you're thinking kind of thing so on our scale This is an example email that I would used to be sending out pretty frequently to a Group of students who at one point or another had sort of opted in the idea of being contacted It was this very vague thing that kind of popped up in the bottom and said hey Would you be interested in talking about your new charter experience? Yes or no So they'd say anyone that said yes, I would hear about it And I'd send them this really kind of simple somewhat personal email Like hey first name like But I'd love to talk with you about how your experience is going. Would you be interested? Compensating I highly recommend people have lots of things going on and time is valuable So depending on how long you're asking to speak with them, you know amount makes sense But yeah, there's actually a sentence in there which is sounds kind of stupid It says in like you're not dreaming or something oftentimes when I would send these emails out the response I got was like is this real is this really the school contacting me and you really want to talk with me and hear what I'm saying I Find that people oftentimes are really kind of happy to be given a place to air their feedback and talk with you about it so doing small things like this to kind of Get engaged with your users oftentimes can can be successful for you So this is another example of a way that at our school. We we try and solicit feedback from our students This is a service called user voice Which is one service among a bunch of others like Zen desk that we kind of have on on our dashboard It's a site suggestions Students click on that and it's a way for them to sort of post ideas Share, you know other people's ideas and kind of have a discussion about it And so this is another sort of more background Channel that we have that allows us to get feedback from users and you know user voices is just one service. There's tons of others and really having sort of continuous feedback cycles where Sometimes we're hitting people before they enroll right when they enroll a little bit down the line After their first semester and kind of keeping tabs on that It it helps to you know give you a better perspective insight into the life cycle of You know your users and also just make them feel more invested. Is that a question? Yeah, let me see Like I was zooming in with my eyes to the screen shot And yeah, you can see it says like 47 votes number of comments So that's actually a really nice thing about it because it makes it gives students an opportunity to kind of vote on things within themselves and comments, so We do have that as well and oftentimes really maybe not in user voice I think it might be more in Zendesk, which is sort of a support Network we have students complain about certain things and get upset and it's kind of like we let the community see that and Kind of pick it apart ourselves and with them. So it's good to have open feedback. We found So yeah in summary These are the things that we talked about or I guess I talked about Just the importance of showing your stuff to people even if it's not your target user And really actually doing that and getting feedback Working with analytics, even if it's not you yourself Finding people in your organization who work with analytics and figuring out a way to get them involved in your work Or you get more involved in their work Customers or users the idea of you know talking with people all throughout your organization that are talking with With your customers and being all in sync and then the three methods mental models Really trying to understand how do people approach your your system and how can you work for that around it? prototyping on paper and lastly, you know Setting up continuous mechanisms for feedback. So yeah, that's all I've got any questions Sure. So for face-to-face usability studies, I so I live in San Francisco And if anyone here has heard of a service called task rabbit, it's a really great way I don't know. I believe they're in most metropolitan areas, but it's a really great way to sort of post a chore And have people go out and bid for it Like normally it's like I'll pay you money to get my groceries or to pick up something from the store So what I do is I'm like, hey you come to downtown San Francisco for a usability study and people bid on How much they'd be willing to do that for and I actually pre-screen them with a with a survey and I'm like to be eligible Please fill out this five-question thing. So I have some general background of where they're coming from Craigslist at least in San Francisco works really great also Well for universities, I used to go to University campuses and literally put up flyers said hey like come call this number if you want to make $50 and talk to me and Also another service I use is called user testing comm. That's for remote usability studies But those Craigslist just in-person ads and task rabbit are the three main ways. I've gotten people so pretty minimally at least for With task rabbit the questions I was asking for screening were highest level of education age What they are currently doing and I think like last institution attended so really kind of big blocks there With task rabbit you can get people you can get people that are just kind of maybe not what you're looking for for Very cheap prices or you can get maybe someone you are looking for for much more expensive So it really depends on the project. I think I'm just using us because it's easier. Yeah, no, it's that's the smart thing. Yeah, okay I'm curious to know a little bit more about the marketing personas and the UX personas We're kind of in the same situation and So, yeah, maybe you want to I'd be interested to learn a little bit more about that and maybe what the difference is between market research and UX research Yeah, okay, that's a great question. I guess everyone could hear it because it was on a microphone. So I won't repeat it So in my experience at university now what ended up happening was that we We were putting out ads on Google like Google search ads with words like you know Almost almost that felt cheap that felt like we were selling this idea of being cheap And then we were talking to users who would never use the word cheap would never use the word like Inexpensive they were using affordable or they were using like well-priced or stuff stuff something that just felt better and kind of Then like looking at the ads and hearing what people were saying sort of what woke us up to this like huge disparity In between just at a language level how we are talking about ourselves and to other people The marketing world at least on the internet can be very run by search engine stuff Like how do we best use words in a way that search engines will pick up our pages and all this type of stuff? Which is not people speak so for us at least that was a very big Like how are we gonna get around this? We ended up, you know really kind of I Guess using usability studies as a way to put some more color to what you know The marketing team was working on where I was like well You know that these keywords will do well, but this is what our customers actually say and it was just a lot of conversations over and over again Honestly at this point we don't really have a dedicated marketing team. It is the UX team that handles That handles marketing I think that as as user experience people It's really our role to facilitate these discussions coming from like marketing like I think market research oftentimes Can be more about From a larger scale like how an industry is doing like what are people buying And not as granular about how this individual product that you're doing will be received by people so kind of countering this kind of The industry says with this is what our customers say Can be useful, but yeah, it's a it's a tough line And I just really think that communication is the absolute communication and collaboration is the key and if As the UX person to come in and be like I really value what you do And I want to work with you so we can like find those people together The other thing now that I'm thinking of it was that I think the marketing team at this point was trying to sell to Someone who wasn't really the user that was going to be successful at our school If you are a student that's looking for the cheap fastest way to your degree Like our schools are kind of hard like you're not gonna that's not gonna be the answer for you So making sure it from a business perspective everyone's aligned on what they're trying to do It's it's hard But I think being a facilitator and getting people in the room to talk about these things is kind of how you have to start And that article that I it's in the end of this. Let me home That's the that's the bitly link to the slides and I'll put some more resources at the end of it too But that article I mentioned I checked that one out Sure Pushback from developers on UX recommendations for sure and it's someone that also does a lot of front-end development implementation time and Cost and tearing hair out nests is certainly something that needs to be grappled with I think oftentimes I'm trying to think that there was a specific example once where We were doing a course tour for inside of our courses and the desired Functionality was that there was an X button to it's like a modal window like let's have an X to get rid of it Or you can click outside Or you can hit escape and the developer was essentially saying having the X button will be really complicated And there are already two ways for them to to get rid of it They can click anywhere or they can hit escape and so the conversation just came about like well, okay What if you don't know those two things? Let's think about what kind of user actually Knows to hit escape or click out anywhere How would that feel to be kind of trapped inside this modal window and not be able to escape? I think as UX people appealing to like empathy and Emotions and trying to help developers who oftentimes are very receptive sometimes not get inside the head of what users feel like Is what I do also I try and invite people to all the usability studies I do whether it be in person or remotely or just publishing all usability reports in a public way For the company and trying to get people involved sometimes if there's a developer that's particularly kind of Not responding to UX results. I will invite them personally to a study and say hey like Sarah, please come and I want you to see what what happens here. Just just come and observe I think getting people it's all about getting people involved getting people to feel like You're not making it up and the numbers help Yeah Any other questions? Question about assessment and stuff. So that readiness meter. Is there anything that's replacing that? Yeah, so the readiness meter is still going to be a meter But it's no longer going to be the idea of helping you predict when you're ready It's going to be literally just about progress like how far have you come and do you have to justify to the users how that? Score yeah, that's that's a great question. I think that's something that My colleagues are trying to figure out right now. We there definitely needs to be so It's my view that we definitely need to communicate that there's a change in the way that the bar is being calculated And sort of that that tooltip that I that I showed is kind of one example of how we're trying to communicate that But yeah, it's it's a question of how do we Kind of contextualize that the thing has changed in a way. That's not I Guess totally in your face and makes you freak out and is that like the kind of the metrics like of how that is all Determined do the teachers get input on that at all or is that pretty much like is it like an auto? Autograded system or it is a summative or formative. Yeah, so the assessment meter itself is actually just it's an algorithm It's really just based off how many things you have completed at this point It'll be like you've you've done your initial assessment check you've done quizzes one through four check check check like that kind of a thing Instructors and advisors hear a lot of feedback from students who who are having those frustrated quotes I assume you get pushback Yeah, the put the pushback and so that kind of bubbles up to us and we can you know work from it from there But it is supposed to be this objective measure of your progress Which doesn't link to do with the subjective opinion of your instructor or any human and one last quick question Oh, which is generally more of the more advanced user the students or the teachers? That's a great question. Okay. Who do you think the students? Exactly. Yeah, I mean we have so many students who are software developers who are people who like run the internet because these are people who are self starters super motivated know how to manage their time and To be honest to be successful in the tech world You don't always need it. You don't usually need a degree, right? Like you can teach yourself stuff So we have a lot of students who are super tech savvy who are literally like I've had students be like This is why it's not working in IE 7. Okay the box model. Let's get on it here Like literally like people who totally know their stuff Our teachers I'd say are more from traditional academia or in general from from online instruction models as well But are not the type of web savvy-ness that our students are many I take it that requires a much more kind of a greatly more simplified You use UX experience for the teachers more so than the students. Well This in a perfect world. I mean in a perfect world teachers would feel like They wouldn't have to even think of the fact that they're using an interface, right? It would just be like a grade book and they can just do their stuff and it's all good, but Yeah, teachers have a lot of responsibility And I think we have not figured we have not optimized like the question before we haven't optimized the teacher experience Yet because we were small and we're still trying to figure everything out for the student first But they are very different experiences and they but they are just as much users as our students So yeah, thank you You said that retention was important and we all know that taking higher ed is difficult Whether it's online or offline you said it's difficult Can you just like maybe talk for a brief moment on usability things you did to Help people get through that challenge besides just it's easy to use but were you able to Give people a little boost. Yeah, that's not it's a great point and I really think that the the majority of that Of the success there is really about our academic staff about our student advisors and our instructors who are Interacting with students on a very personal level and helping them and coaching them through their degree progress They are the first line of defense for technical issues for academic issues For billing issues. I mean we have an incredible staff of people that you know really Guide students along the way I think so in our case. I really think that's where the where the credit is due in terms of the UX side of things Yeah, I mean I think in and if anything this Establishing those sort of continuous feedback cycles where we're learning something is a really big pain point And then working to fix it like the readiness meter. I think that's kind of the You know that's sort of that is our role to to fix things when we really hear about people struggling through them but it's definitely our academic staff that Kind of helps students get to where they need to be So this isn't a UX question, but I thought it was kind of interesting that your pricing model is a time Based but courses can be done as quickly as you're ready. Do you have anything to add to yeah So well, okay. I guess I'll say the slightly more extended version of that So semesters at let's pick new charter. So semesters are 16 weeks long And basically when you begin a course you have you can go as slow or fast as you need within this Semester period within 16 weeks if you you go one after the other maybe you can complete five courses maybe you only complete one but It is a 16 week kind of time-based semester chunk So we have like a bachelor's degree which you know can take four years or if you coming in with a lot of transfer credits And you're working incredibly hard. I don't know maybe you get it done in two years Like in that sense, it's very up to you, but we do have this delimited 16 week block That makes sense And the tuition is monthly or it's annually or no, sorry monthly or I guess by term paying up front with that So yeah, the website's you now calm for the company A new dot edu for a new charter and patent that need you for patent My question sure So I'm you said like with user voice that you hash things out with students and staff And I'm just curious if you Like when you are addressing the students if it's like The UX team says this or do you get like personal and say like your first name or like that? In that process. Sure. That's a really great question So I wanted to mention that in the context of this email So this is an email that I wrote on behalf of like me the UX person to a current student and In my experience, I've found it very helpful to be very specific about who you are and what your intentions are and having a Conversation I have nothing to do with your grade I am not a teacher and I can't like help you or hurt you or anything All I'm concerned about is the experience that you have with our school online So being very upfront about that and kind of saying my role and and who I am has for me Been the most successful when I started out in the beginning I don't think I mentioned outright about user experience and I did have some students saying well Do you know my teacher? Do you know like are you gonna be able to like how? What are the communication channels like so in I think as you know if you're doing this on behalf of user experience research It's really helpful to just say that and oftentimes it makes your users potentially for at least in our case for students It makes them feel a bit more relaxed and willing to open up Okay, well, thanks for coming