 say, hey, do questions as we're going, which if you feel fairly strongly about a question you have, do feel free to interrupt me. But otherwise, we'll try and save all the Q&A towards the end. A little bit about myself, I won't spend a lot of time on me because that's not why you're here, but really, I'm Kelly Tetherton. I'm the director of web experience at Clarity Partners. I've been doing this sort of thing for over 20 years now. Lots of different platforms, lots of different roles. I used to be a hardcore developer, and now I'm more of a pointy-haired boss. Lots of different clients. I've done private sector clients. But now with Clarity Partners, we're very much focused on public sector clients, largely in the Chicago state of Illinois region. And I mentioned that only because, again, what we're going to focus on with this case study is a public sector client. And if you've worked with government and private sector, you know they move at very different rhythms. There's a lot of times in this talk where I'm going to say your mileage may vary, and that's my first one, is your mileage may vary, depending on the type of client you have. I do have information about Clarity and lots of cool stickers and my business cards up front. So if we do run out of time and there's more questions that you have, feel free to reach out. But here's what I wanted to focus on is Chicago Park District. So I'm not going to dive deep into the technical details behind what we did for Park District, but I do have the link to the Drupal.org case study that we did about Park District so you can find out a little bit more about what's going on under the hood. But just so you understand a little bit about Park Districts, because I think people hear the term Park District and they think it's a very cozy mom and pop kind of thing. But Chicago has tons of green acre space, over 600 parks, lots and lots like thousands of events and programs that they do every year. And they are genuinely trying to serve all of Chicago, Northside, Southside, all different income levels, the whole kit and caboodle. So when you talk about dealing with know your audience, they have a real problem because they have such a diverse audience that they're working with. In terms of the website itself, they get a range from between 100,000 to 500,000 views per day. It's very spiky and very seasonal. Back in February, the ice rinks pages were getting tons of traffic in a few weeks, because even though the weather doesn't tell us so, summer is right around the corner, their beaches pages are going to be very, very popular. Having said that, there are a few consistent pages that over and over and over again were clearly the places that people were going to. The programs offerings, the parks, the events, the jobs. Jobs were always a really big one. So why did we move? We started the project about a year ago, and we launched just before the end of 2017. So this is still relatively recent. But the reason that they wanted to move was they were on a very old customized platform. And this was a case of nobody did anything wrong, it was just this platform had been slowly customized over the course of five or six years to the point where nobody really owned the code anymore, and nobody really understood what was going on. And it made it very, very difficult for them to add new functionality, because it was a classic case of nobody really understands the code anymore, so you'd make what you thought was a really simple change over here, and it would totally break something over here. So they needed something that was more stable and really more future-proof than what they had. They had also, again, when they initially launched, they said it was about five or six years previously, and about, I want to say, three and a half or four years ago, they did refactor the design to be responsive. I have it in quotes there, because they really couldn't deal with truly making it responsive through and through, and so they made the major pages responsive. So if you went to the home page, it worked well on your mobile device, but you'll see later on that there were some pages where they just sort of said, we're not dealing with this right now. But as with just about anybody who's got a website out these days, they saw their mobile traffic going up and up and up, and they realized that they really needed to have a truly responsive site. So what did we know when we started? We actually knew a heck of a lot. Park District had analytics, has analytics going back well over 10 years. They know where people are going seasonally, month to month. They have done very, very quick Google user surveys, so they had some feedback from users, and we did a fairly extensive benchmark in comparison analysis for them, which is what that pretty Excel spreadsheet graphic is there. We had compared the Park District site to, I believe, six other very large Park Districts as well as something like, I think it was a dozen or 15 other sites that Park Districts had pointed to as featuring designs that they really, really liked. So we knew what people had done over the course of time. We knew where people were going. We knew what other Park Districts were doing in terms of vocabulary and features that they were highlighting. So then the question is, well, if we have all of that data, why would you bother doing a user focus group? And really the problem is, is that data only takes you so far. So this was a classic, I wouldn't say argument that we had, but certainly a series of discussions that we had with Park District was, they have a series of photo galleries on the site, lots and lots and lots of photos from events that they've done at various parks, and consistently it didn't matter when you looked at the data, that was like at the bottom of the list of pages that people were going to. And so we, Clarity Partners, was saying to Chicago Park District, okay, well, when you're focusing on the new version of the site, you really wanna focus on the pages that people wanna go to, and they would come back to us and say, well, people aren't going to the photo gallery pages just because we haven't highlighted it enough. If anything, we needed to make it prominent on the homepage. So we don't know why people are doing what they're doing, and that's really why you wanna use a focus group, is find out what people are actually thinking when they're going places, especially since the Park District folks that I've worked with and have worked with have been at the Park District for 10, 15, 20 years, some of them. I've worked with them for over 15 years at this point. Guess what, we're all thinking inside our own heads. If you say facility, I can guarantee you it means something different to you than what it means to the folks at Park Districts. So the other thing to do is to get really external validation. You think that people respond to a certain term or a certain navigation label in a certain way. How do you know? It's just because you've been saying to each other for 20 years. In Park District's case, again, because they were a public sector entity, they also had the bonus of publicity from using a focus group. That's not minor for them because for a government entity, if you were gonna spend big bucks on something like a new website, you better be able to go back to your constituents and explain why you spent the big bucks on this new website. And for them to be able to say, oh, and we were using feedback from Chicago City residents was huge. But bottom line, we really needed to know how people were using the site. So this is the point at which I will break and I will confess we had a secret weapon that not everybody has. We were able to use a group known as Smart Chicago. They're now known as City Tech. They did a reorganization at the end of 2017. They are really focused. This is a group that's focused on improving lives in Chicago through technology. They've been doing this for several years and they have a built-in audience of testers. They started out with just focusing on the city of Chicago. They've expanded to Cook County, which encompasses more than the city of Chicago. They have a very specific methodology, which is that's really the URL that I've got there for the cutgroupbook.org, if you wanna find out more about that. They were very invested in working with park districts to figure out their goals. And I will talk about alternative options later. So even if you don't have a secret weapon like an organization like Smart Chicago, there's still some useful things to pull out of this. But for us, it was huge that they already had a built-in audience of testers that we were able to tap into. So the first place to start, in terms of, again, our experience, but I think that would be useful for you folks too, is in terms of where to start, is think about your demographics and who do you want to do the testing. This sounds like a simple and obvious place to start from, but it's not necessarily, and it's worth reminding yourselves why you would be doing a focus group in the first place. For instance, I don't live in the city of Chicago. I live in Evanston. I don't want our effort to be put in a situation where people who are outside of the city of Chicago are answering these questions, and then if management within Park District reviews these results, they could then dismiss it by saying, well, half your testers weren't living in the city of Chicago. We're not gonna take their responses all that seriously. So think about who you want to do the testing. Now, in Park District's case, they wanted people from Chicago, duh. But they also wanted people, a mix of people who had and hadn't used the website and who hadn't hadn't used their programs, because they really wanted to know if they were getting their messages across. They also wanted a mix of families and singles, again, to see how people were responding to finding the information they had. The next step that we had was developing the questions. What are the pain points of the site? Can people find the information that they want? We knew fairly early on that we needed to make our questions task oriented, but that was actually a little bit harder in the execution than we initially thought, because the next step after we started developing questions was then having to eliminate questions. So the way that we started was we just had a shared Google spreadsheet between Clarity and Chicago Park District, and it was a classic case of it took us forever to get the first two, three, four questions, and then it was like the floodgates had opened, because this was the first focus group they'd done, and so they realized, oh my gosh, all these things I've wanted to ask users for decades, I finally have a chance to ask. And so we started out with over 50 questions. At this point, our main contact at Smart Chicago, who's a very smart, very diplomatic woman, came back to us and said, that's wonderful, that you have so many questions. You need to keep it under 10. 50 is not gonna cut it. You can't ask people to go through 50 questions. So then it was, okay, well, now that we have these questions that all of us really care about one way or another, how do we get rid of them? So what makes up bad questions? Well, bad question rather, well, some of them were too vague. Why would you visit ChicagoParkDistrict.com? What are you looking for? How are people supposed to answer that? What is your opinion of the photo usage on the site? Which pages are informative? Or which images are informative? Which images are really decorative? You can smell the internal argument coming off of that question. You can't ask focus group users to solve your internal arguments like that. We also had questions that were too narrow. Again, if the goal is to solve the most problems for the most people, asking about incentive and reward programs, that's really pretty narrow and is answering a question for one marketing person within Park District. How would you go about contacting a local park advisory council? Most people don't know that parks have local advisory councils. Why are we wasting our time asking these types of questions? And then we had questions that were ultimately not solving a problem. How would you find and apply for a job? Well, as I mentioned in an earlier slide, jobs is one of the top places that people go. People are already finding that information. Likewise, how would you register as a vendor if you wanted to do business with CPD? I can guarantee you that if you are a vendor in the city of Chicago and you want to work with the city of Chicago, I could bury those pages 10 levels deep. I could take them away from links and navigation and people would still find them. If you want to find them, if you're dedicated, you'll find them. So what made a good question? Good questions were questions were really, really focused and narrow, but almost kind of like a scavenger hunt. Give people specific information to find. We did try to make it as you know, as ginned up as these questions are as real to life as possible. So search for a park that's closest to this library. We ended up using a local library to do the focus group, which is where that question came from. Find the hours for an indoor pool closest to your home. Then there was a trivia question, fun trivia question, which turned out to be one of our toughest actually, which is find out the year the Buckingham Fountain was installed. Now, one of the things I should have said earlier, which is interesting about Smart Chicago is they will work with civic organizations to conduct things like these focus groups, but their flip side is they are required as an organization to be completely transparent. So the URL that I have there lists the full set of questions that we asked, the full set of raw responses that we got from users, the initial summary report that Smart Chicago provided to us and the final full report. So again, if you wanna dig in further, it's definitely the information's available for you. So then the experience, and again, this is part of why I think I'm hoping it will be somewhat useful for you guys is what was it really like once we got down to it? Scheduling. So again, caveat, I'm working with a government organization so they're moving at their own speed and their own pace. That said, one of the collateral damage elements of working with Smart Chicago is they needed to do their own scheduling. It took about six weeks from the first discussions to the actual testing itself. We needed to figure out who's available, physical space, equipment, all of that kind of thing. I'm mentioning this because I think it's important to do user focus group testing if you wanna find out why people are doing certain things, but you also need to do it in a timely fashion so that it's useful and it's gonna matter to your project. So unless you have all the time in the world and all the money in the world, which almost nobody does, you have to get these results back when they're gonna matter before you've really set your design in stone, your information architecture in stone before you've made all those decisions and started developing. So again, I think it's useful to remind you about the scheduling element just so you can scale appropriately. Getting the people. So obviously we had Smart Chicago who was really getting the participants, but Smart Chicago, no fools they, although they had staff that were also doing the proctoring, they wanted us to do the proctoring as well. So Clarity staff and Park District staff to do the test proctoring. I cannot emphasize enough getting a client to be part of that experience is invaluable. So there's one thing to be able to produce a report at the end that's very tidy and wrapped up and easy to digest. And there is another to have your client sitting with you with a visually impaired woman trying to navigate her site for an hour and a half and absorbing that and then understanding, you know what, we do need to take into account ADA compliance because it's really affecting people. It's huge if you can get a client to even sit in and experience that from the sidelines if nothing else. So then we had all our pieces in place. We were ready to do this. We had 28 testers who were confirmed. I think we had something like 23 or 24 who ultimately showed up. We had 30 to 45 minute testing slots over three and a half hours. I think it was a Tuesday afternoon. We had scheduled it, I believe, from four to 7.30 in the evening. We had our staff identified from both the Smart Chicago and the Clarity Park District side. And then, Sonia, again, our contact at Smart Chicago, said, okay, you, Clarity and Park District folks, you need to come in and we'll have an hour, hour and a half meeting. We'll run through the procedures. Now at this point, I kind of, I'm not gonna lie to you, I kind of did an eye roll because I'm like, we know what we're doing. Why do we have to do an hour and a half meeting ahead of time? Hugely, hugely useful. I mean, Sonia had gone through the basics of you don't wanna be leading in your questions and you really wanna get the responses from your users. She also went through just the basics of we're gonna usher people in, we're gonna have them sign up, we're gonna have them sit down. Here's where you're gonna do things. But then she actually had us run through a process of running through the testing with each other, which was hugely useful because we're getting, we're capturing the user's feedback as they're doing it, but we're also sort of backgrounding and recording a script in our head as proctors because after we've done the test, we're also separately writing up what our experience was observing the users. The reason for doing this is you can ask a user, well, how easy did you think it was to find information XYZ? And most of the time people are really overly kind and they'll say it was really easy to find information XYZ. And then you as a proctor can say she spent seven minutes trying to find Buckingham Fountain. This wasn't easy at all. And you have to keep that in mind as you're doing this. So it was super useful to do this initial run through because we realized that it was really difficult to do this sort of double sync at the time and also to get the reassurance that if you don't have to write full proper English sentences and paragraphs, just write things that are understandable enough so that they're bullet points and when you come back to those bullet points they'll make sense to you. So the test itself had some surprises. The first one, physical space matters. So the room that we were in is probably but was probably maybe two thirds or three quarters of the size of this room, which we thought was tons of space. We had tables, we had chairs, whatever. And that wasn't quite right. So the test, as I mentioned, was run from four to seven, 30 in the afternoon, which was great, except that that was basically right at Chicago's rush hour. So we had all of these carefully slotted 30 to 45 minute slots and people weren't able to get there right when they meant to get there. And so people were stacking up and we were running out of room as people were coming in. Physical space matters. And that's also why the actual test experience took long as people were coming in later than we were expected, but they were also really, really invested in the test. So even though we had stripped down to I think it was roughly 10 questions, people were really invested in doing the right things and we were trying to take all these notes and most of these tests took roughly an hour even though we had scheduled 30 to 45 minutes. I've already mentioned we had a very, very painful experience with one particular user who was visually challenged, but it was incredibly useful to see her experience. And then one other thing that we weren't expecting that I only mentioned, because I think a lot of larger sites will have this, is there's the main park districts, www.chicagoparkdistrict.com site, and they have tons and tons of ancillary sites as well. There's a site that's entirely separate that takes care of registration. There's a site that's entirely separate that takes care of their loyalty program. Users would get lost in these external sites as they were going through the tests and even though we didn't really care about their experience on these other sites, it wasn't really in the spirit of the test to say, don't worry about that, we want you back on this site. So there was some time wasted over that. It was really, really hard not to help the people who were doing the testing. Because we had scheduled or constructed our questions really as scavenger hunt items, when was Buckingham Fountain installed? What are the pool hours? We already knew the answers. And so it was very hard not to say, click here, you'll find the answer here, especially since really wasn't expecting this last point, which is the users were really invested in this. And even though we had told them up front, if you don't find the information, it's fine. We just want to find out what your experience is. They really didn't want to be wrong. They were super invested in getting the right answer and it didn't matter if it took them 10 minutes to get there, they were going to get that right answer. And that was again, something that we weren't quite prepared for. So what were the actual outcomes of this? We did have some fairly specific outcomes that came out of it. So these might be a little bit hard to see but these are before and after pictures of the homepage. The before picture isn't that bad really, but there's a lot of information on there. And we had been from the beginning sort of advising Park District, you really want to have a cleaner homepage. And as with all organizations, even though it doesn't matter as much as people think it does, there was always a battle royale for real estate on the homepage. It was huge to get the focus group results back because again, you can see this on the Smart Chicago page. One of the initial questions that we had asked users was what action do you want to do when you come to the homepage? And Sonia again, very gently and diplomatically as she was presenting the summary report to us basically said, I have never seen as fractured a set of results for people just didn't know what to do on the homepage. They didn't know if they were supposed to click that big green banner or the main hero image or anything. So we had a very strong case to say it's not just us saying you need to clean up your homepage or your users literally don't know what to do. So we got a cleaner homepage out of it. We also got a very clear focus on mobile usage. We had a set of users that came in that only tested on mobile devices either their smartphones or tablets that we provided for them. So even though Park District was already leaning that way if there were design elements where they were like, I'm not sure if we want this or not. And we would say it has to look good on mobile. Remember, it has to look good on mobile. That drove a lot of our design decisions. We streamlined searching. Again, it's a little bit difficult to see but the image on the left, there's that column on the far left. Those were all different and they dropped down too. Those were all different search options and it became very, very clear when we're asking users to search and find things. They got completely lost in so many options. So we simplified the searching quite a bit. Highlighting the mapping options. This was interesting because this was definitely one of the surprises that we just weren't expecting. It's a little bit hard to see on the image on the left but where that little blue line is in the middle, there was a tabbing system across the page. The second tab on that page is a map. No one, no single user knew that there was a map there. So it was definitely one of the design outcomes was we need to put our maps absolutely in everybody's faces. So a few things as I'm trying to auctioneer through this to wrap up. What if you don't have something like Cut Group or Smart Chicago available to you? There's still things you can do. On your own, clarify your goals, develop your questions, find neutral testers. Neutral doesn't have to mean I don't know the person. It could just be it's not a person associated with this project or it's not a person who's already deeply invested in the outcome of this project. And an oldie but a goodie, we had the advantage of a couple dozen users to test. Five is usually enough. Five is gonna get most of your feedback that you need. And finally, just remember that you do need to give them some incentive but the incentive can vary. It could be a Starbucks card. Smart Chicago gives their users Visa cards. Other people I know have used just a raffle ticket. You don't even have to give everything to everybody. And finally prep for the test for sure. If you wanna use other resources, just know that there are online resources of various kinds and you can do paper sorting, paper prototypes, card sorting, A-B testing with Google, you've got options. Although again, none of those are gonna get you quite the why that you're looking for from a user group. Quick wrap ups. We are hoping to do a second test with Park District because we'd really like to close the loop and find out did we in fact solve our pain points and also surprise, surprise. We're still fighting over real estate on the homepage. So already a couple of times people said we need to do another focus group to find out if this would work. I know, finally, yep, finally wrapping it up. Last three things just to say, if you don't remember anything else, focus on what you wanna get out of your user testing because that really drives why you're doing it. Prepare, prepare, prepare. And be ready to be surprised. Your users are gonna find things that you weren't expecting for sure. So having said that, obviously I have to wrap up. I'll be wrapping up some things here, but again, I was afraid this might happen. So do grab my contact information if you have any questions later. Thank you. Thank you.