 I work for Delta, which is distance education and learning technology applications, and I'm on the learning technology application side. And so my role is to coordinate between the various stakeholders for our learning technologies and help make sure that everybody's pushing in the same direction, and I also run the governance for the LMS. So what is Wolfware? It's our learning technology portal, and I know everybody hates that portal word. It was, you know, vastly overused over time. But our goal or our intention is to have a collection of best-of-breed technologies and tools in one spot for people to use so that our faculty can easily access a variety of different technologies. Some of the things we have in there include a bootle, a blackboard collaborate, media site, which is a lecture capture system. We have a syllabus tool. We're adding a WordPress tool for differently accessed materials of the web and an email tool to create your own lists and send out email to interact with students. And we want to try to integrate the tools as much as possible. So one of the things that we've done is we've pushed as much of the user configuration out of Moodle and into our portal. So for example, if you're an instructor and you're using a few different tools, you do not want to add your TA to each individual tool. If you have a student doing it incomplete, you'd like to be able to add them in one place. You want to set availability date, those types of things. You want to be able to do it all at once. So we have Wolfwares where they configure their tools and then they go off and use them. And in theory that sounds wonderful. So we've been going along and I think, you know, first I have to have a disclaimer. I'm going to whine and complain a little bit, but I actually work with a fantastic group of people. And, you know, I think all of our institutions or, you know, wherever we work, there's always things that we can do better, but we're always overwhelmed with a variety of different things. So things like usability, support, training all have a way of kind of falling off as we try to keep up with the train of progress. But I noticed that... I think I missed one slide here. This thing is slow. So anyway, hopefully that'll load, but between help calls, my own anecdotal evidence, you know, me being the coordinator of what we're doing here. Oh, there we go. Okay. So I felt like our website needed a lot of work. Our faculty generally were kind of happy with it, but I noticed that people were having a very difficult time doing things. There seemed to be some issues and problems. And so I felt like we really needed to kind of step up our game and do a better job. So I, you know, couched it in the most dramatic terms that I possibly could, you know, that the website sucks and it's having problems and it's not doing this and that. And I learned a valuable lesson. Be careful what you complain about because I knew immediately I jumped right to the answer. We need to redesign our website and we need to get some usability people in and go for it. And everyone else on the team looked at me and said, well, we don't know that there's a problem yet. Prove it. So now I added on the role that I had to go out and do a usability study. And so I said, okay, I embraced the opportunity to get my foot back out of my mouth. And, you know, so I got to do the study and I got the assistance of a postdoc to give me a hand whose area of research was actually usability. So it's great to have her. So some of the goals that we had were to try to get an understanding for how faculty feel, think, and use the site to try to come up with a set of problem areas and recommendations for what needs to be fixed, not just say, okay, this is a problem, that's a problem, but kind of look for the general themes of what we're issues. And to also get some practice conducting usability so that in the future, as we do new development projects, we can try to change the way that we work, get the interface design away from the developers, get it more into the support, the training, and the usability realm, because while the developers create very functionally accurate interfaces, they're not always done in the way that faculty or whatever might want to try to get the test done. So, you know, we looked at what the potential problem areas were in order to kind of put this study together. We first grabbed 13 faculty members out of a random sample. We had to send an invite out to 1,000 faculty members at random, and we got 37 people who said yes and grabbed 13 of those and kept the other 20 something for later. And we came up with eight representative tasks. The way that we did that, and I'll show you what they are in just a second, was that we mined some of it with anecdotal, the things that we felt we had a problem with, and then a lot of it was based on help calls. The director of our help desk went through a three-month period and looked for all the help calls and we picked out what the most common themes were. Here are the eight tasks. They range from logging in and creating a Moodle space or a section so we require you to actually create one. I know some schools automatically create one for every section, but we have a bunch of sections that don't use, so our model requires you to request one. Adding their picture to a profile so it would appear in there in Moodle and what have you. Adding a TA, so availability date. We have a course copier that will copy their content from a previous semester to a current one. If we made a mistake and we copied the wrong section, how would we do that? Cross-listing section so when a faculty member is teaching more than one section, you know, can they get the one Moodle space for both sections and if they wanted to schedule a collaborate session to use for their site or something else. So those were the things that we looked at. And we did both qualitative and quantitative stuff. I'm not going to talk about the qualitative, like, you know, how they felt and whatever generally they were very positive about the site even when they had trouble, which was interesting. But the quantitative, we have a lot of data and you can take a look at all the qualitative as well. We also tried to stay away from aesthetics because that's a matter of opinion. For example, we have animated clouds and six people thought the clouds were wonderful and beautiful and seven people thought that they were just annoying and distracting. So, you know, those could be challenging issues. So just to give you a quick little preview, task completion rate, how many people actually of the 13 actually successfully completed for something as simple as creating a Moodle space or requesting a space? Seven out of 13 were able to do it successfully. That doesn't sound very good, does it? Another three thought they completed it, but actually hadn't. And then three just gave up. Now, I'm not going to talk about errors, you know, we counted errors, being anything where you did not make progress towards your goal. If you were, you know, performing actions, you can take a look in the report to see the full extent of that. You can see that there were several tasks like task four, seven and eight that they also had a good bit of difficulty trying to do. You know, and so people really had a very different sense of what Moodle was, you know, what Wolfware was rather. So we tried to ask them, you know, what do you see it as? What does it work for you? Some of it saw, some of them saw it as a container for their materials. Some people saw it as a gateway to get to the tools. Some saw it just as a unifying brand, like we were just slapping Wolf on everything. And some folks confused Moodle and Wolfware, they thought Wolfware was just a synonym for Moodle. So we had a great deal of difficulty with that. The reason why this is important to us is because we're adding tools now and as we add these tools, things are only going to get worse. We're only going to get an additional layer of complexity. So we have two new tools coming online, which means we have two new tabs, two new bars that we're adding. We also found that some of the ways the metaphors, the things, the ways in which we're doing things do not match with the way a faculty member wants to do things. So for example, when we talked about cross-listing classes, when we talked about combining two sections, we see it from a computer science point, the easiest way was you have a course and you add another course to it. And what faculty members were trying to do is they were trying to smush two classes together. And while that doesn't seem like a big difference, how you might go about to do that at an interface is very, very different. We also had a huge problem with our... We have a current tab for courses and a future tab in Wolfware. And so you have to know or have to be able to see, get the feedback of where you are in time. And many people were acting on current, were they meant to be acting on future. So there were a variety of little confusing things like that. I don't think of what else. So we've corrected some obvious issues, the easier ones, things like inconsistent feedback or things that are different color or a message that's supposed to appear but has not. We've also made the course creation a little easier because that was kind of an urgent thing given how many people were having trouble with it. So we've kind of put everything in one place to make it easier for them to find it. And we are actually looking at evaluating or going over the entire design and considering next year when we retire another tool and we've added some other tools to the suite of coming out with a completely redesigned site and starting from scratch. This time with a lot of usability involved in it. So I would urge you to go take a look to that URL. The slides are loaded and so they're available. Please take a look at that. It's 65 pages of all the gory details. Anything you might possibly want to see or find out about our study. My email address is in there as well. So if you have any questions, once you've read it, then you're more than welcome to get in touch. We'll be happy to help you anyway. I also have all of the instruments, the questions we ask, what the responses were. So feel free to grab, you know, if you want to do a quick discount usability study, grab that stuff, print it out and give it a try. So that's all I got. And I would also encourage you to go to wolfwear.ncsu. You can only unfortunately see the unauthenticated portion of the site, but at least you can see the clouds that people love or hate. So with that, I will throw this open to questions. And... Any questions or requests to go back to the previous slide? Yeah, I think also if you go to delta.ncsu.edu slash news and just click on search on usability, you should be able to find it that way too. Other questions or comments? So well, in that case, let me ask you folks a question. So why are you here in this presentation? Are any of you having trouble with usability of your site? I know you're lying. Okay. So what are you working on? Or what site are you having a problem with? Our faculty basically uses our site as a repository. So they don't use the higher level tools. So I'm learning those tools myself and I'm finding them a little difficult. Okay. Anybody else? I guess that's it. Okay, thank you. All right, thank you.