 Well, for joining today, the last talk of the conference, you're troopers. The title of my talk has explained it to me like your five learnings from user research with kids. I gave this talk at DrupalCon in Nashville. And I had a really nice guy come up to me afterward and say, oh, cool talk. I thought it was great. But I got to say, me and my friends only went to this because we thought your title was a metaphor for something drooply, something development. I don't know, maybe testing with kids is like hacking the core. It's super dangerous. You shouldn't do it. So I hope you wasn't too disappointed. There's no development. There's no Drupal in this talk. It is literally about doing user research with kids. So if you thought this was going to be a development talk and you'd rather not spend the next, like, 45 minutes listening to this, I get it if you want to go. Your time is precious. I'm the only person standing between you and the after party. So no hard feelings if you go. But if you are interested in learning about user research, I'd love to tell you more about it. So a little bit of background about myself and my company. So I work at MyPlanet, where a software studio here in Toronto, and we make smarter interfaces that focus on the workplace. So what that means is a lot of our clients are kind of B2B-facing people. They might be trying to build internal tools for their own employees or portals for their customers to come in and manage all the servers that they buy. So really a strong B2B focus. And we're trying to leverage some of the newer technologies like AR and VR and personalization to just try to make the experience a little bit easier and faster for the folks using the tool so that they can get in and out more quickly and do their work more easily. So yeah, we're about, I think, a little over 100 people now worldwide. Mostly in Toronto, but we've got some on the West Coast and some in Europe as well. And that's all of our social media, et cetera, stuff if you wanna at us, get at us. And about myself, it's still not working. It seemed to work like if I was literally almost eating it. Okay, that's good. I'll just get real close to it. We're gonna lip gloss all over this. It's cool. So about myself, my name is Cara. I like to call myself a recovering academic. So I started my research journey during my PhD in cognitive psychology and research psychology. And when I graduated, I decided that I wanted to focus on some more real world problems. So I joined my planet about five and a half years ago as a researcher and interaction designer. And that's all my stuff if you wanna get at me. All right, so what are we gonna talk about today? We're gonna spend a bit of time just going over user research as a concept, the different types of research that we're all on the same page. I'll spend a bit of time talking about getting set up before you even go into your first testing sessions, all the prep work that you need to do. Then I'll dig into some sort of tips and tricks and strategies for what you can do inside the session to make it go more smoothly. And then finally, I'll end off with just some logistics. So like little bits that we all tend to forget, but if we do take care of it in advance, it tends to make things go a lot more smoothly. All right, so whoa. There I am. Thank you. It's turning to like a Foo Fighters concert. Now I'm like afraid to be too close. Is that okay? It's not. All right, thank you. Thanks at the back for helping out. So right, user research. What is it and what is it good for? So some of you are already probably familiar with the different stages of research early in a project, but just to get us all on the same page, I tend to think of the research in two stages. First, there's the discovery research. So that's where you're trying to understand the goals and the needs and the pain points and attitudes of people who might be using your product. You're trying to figure out if the features that you're thinking about are gonna offer utility for the users of your product and also maybe discover some new needs that they have that you hadn't thought about before. And then once you're a little bit further along in your design process, you'll also wanna do some research around the experience of actually using your product. So can people understand how to use it? Do things make sense? Do they know how to get from A to B? This is where usability testing comes in. So let's dig a little bit deeper into what each of these kinds of research entails. So for discovery research, let's say you're trying to build an e-learning app so people come to your site or your app to try to find training and content related to whatever they're interested in. Before you even start designing, you're gonna wanna find some people who would wanna use this tool, so people who wanna consume content in this way. And you might bring them in for some sort of in-depth one-on-one interviews and observations to try to understand the context and the needs that they're coming to your product with. So as you're doing this research, you might find that different clusters or types of users come out. So you might have some users fall into like a busy professional category while others are more learning just for hobbies, like based on their own interests. And these two groups, even though they're probably gonna do similar things on your site, they're gonna search and look for content, they're gonna come to it with very different mindsets. So the professional person might be, under a bit of a time crunch, they need to know exactly what marketable skills you're gonna get out of this content they're consuming. Maybe they've been put on the dreaded performance improvement plan at work, so they're kind of stressed out about it all and they really need to catch up. So there might be some urgency there. But for the person who's a bit more of a hobbyist, they're just kind of learning in their spare time, they might be a bit more open to exploring. They've got more time to check things out to kind of poke around. But on the flip side, they might need more guidance to tell them what types of content are available and more inspiration about what they might want to look at. They might also need a little bit more motivation to stay engaged because they don't actually have to do the content, they're just kind of doing it for fun. So doing the research can reveal these different types of needs and mindsets and tell you the different goals and workflows that your users are likely to have. But beyond that, it can also help you understand what we call the experience criteria. So this is how users want to feel when they're using your product. And of course, like everyone's gonna say, oh, I want it to be easy and intuitive and really seamless. Everyone says that about every product, so that's all well and good. But there's like more aesthetic things to think about. Like, do they want it to feel playful or serious? Do they want it to be really pressure-free and really supportive? Or do they want that little bit of competition in there? So doing some user research is gonna help you get some input to help inform those design decisions that you're doing. So once you've done your discovery research, you kind of have a good idea about what product you wanna build, what features you wanna have in there, you're gonna start your design process. And as you do that, it's equally important to test that what you're making is actually understandable and easy to use for your users. So when it comes to usability testing, you present your users with some form of interface. And this can be as low-finality as like pen and paper sketches that you're putting in front of people and getting their feedback on. To wireframes, to clickable prototypes and like envision or action. Or it can be like a fully coded functioning site that you wanna do easily to check up on. So any level of fidelity you can and should do testing. The goal here is to present users with the interface and then ask them to go through some common or frequent tasks that you might have in your product. And then you just kind of sit back and watch and do it and ask some probing questions along the way. And as you do this observation, it's gonna reveal really important UX issues like maybe it's not clear in the workflow how to get to the next step or it's not clear how to navigate to key parts of the system. Maybe the terminology is not clear and it's really obscure for users. Or calls to action aren't where they expect them to be so that they get missed or they're not obvious enough. Or sometimes we realize that there are use cases and sort of edge cases that we didn't even think about that users try to do. And you're like, oh crap, I totally forgot to design how you get out of this section of the site. So doing that testing early and often is gonna help you catch those mistakes before you move into the code because it's always gonna be more costly, more time sucking and more soul draining, I think, to try to fix it in code versus when you're still in the design phase. Yeah, so by doing this early and often, you sort of fix the problem that you find, present the next version to users in your next round of testing and just validate that you actually have solved that problem and they can now do the task successfully. And by doing this, you kind of stack the cards in your own favor for building a product in the end that's actually usable and robust in the way that you need. All right, so why should you care about usability and user testing? Why is this important to you? Well, if you're a designer, hopefully the reasons I presented were compelling enough or you've already been convinced and you don't need me to talk about it. And maybe that's also true for product owners and developers, but for product owners, it's really helpful for you to see the user testing sessions and understand the context and the pain points that these people have because it's gonna help you make those tough decisions when it comes to prioritizing. So you can say, you know, well, we know stakeholder that you really love feature Y, but based on the research that we've done, like that's a nice stuff for people, but what they really need to do is task X. So you can have some, I don't wanna say ammunition because it sounds like you're fighting with your client, which is never good, but some, I guess context or background to help you inform those decisions to rationalize those decisions. So it can help you with product scope and release planning. And for developers, it's gonna help with all those like little front end decisions that you inevitably end up making. I think the first few times I worked with developers, I was really surprised at the amount of design decisions they had to do because working in like a fast paced agile environment, designers don't always have time to compile like every single state, every break point, every format, every edge case. So it falls on the developer's shoulders to make those decisions. And sure, you can chat with the designers about it, but if you have a really solid foundation and understanding of what your users are gonna do in your product, it's gonna be a lot easier for you to just say, oh, you know what, I think it should look like this. And then, you know, if the designer doesn't like it, we can deal with it, but it's gonna be a lot faster for you. It's also gonna give you some context for deciding on or suggesting solutions for problems that might be out of scope. So I'm sure I've never done this to any of my developers, but I've heard that sometimes devs get designs handed to them and they just start laughing because it's like, okay, this one component is gonna take us half the project to build. Like there's no way we can do this. We have to simplify it. And of course, you can always work with the designers to figure out another way to do it, but if you've been in these sessions, you see the behaviors of these people, you know their goals. It's more likely that a solution will jump to your mind for, well, you know what, they can still accomplish this goal, but in this way, that's like way less effort and less time for me to build out. So that just like elevates the team as a whole. And then finally, it gives you a leg up on UAT. So we find that when devs and product people sit in on our testing sessions, they notice a lot of these little bugs, whether it's just like, oh, this needs more padding or this filter isn't like showing all the terms that we needed to show. And some of them are really quick fixes that you can just note and kind of fix in line in the sprint instead of waiting until the UAT period. And then you have like a backlog of a million things which is never fun. So it can help you sort of keep those things under control. And then finally, I'd say that for everyone, one really great effect of user testing and observing user research is that it keeps us really empathizing with their users. And I know personally when I work on a project and it's like month four and I'm deep in the weeds, I get kind of like, you know, stuck on a problem and I forget the big picture of what we're actually doing this for. And presenting your work in front of users and seeing them say like, oh my God, this is awesome. Like when is it gonna be real? When can I use it? Or even if it's critical feedback, it can be really re-energizing, I guess, to keep us all aware of the big picture and aware of where we're actually, who we're actually building this thing for. So I think that's one kind of side benefit that's really important. All right, so hopefully I've made a case for how the research process can help all of us in our work. But why test with kids specifically? Well, obviously, if your product is designed for children, you should be testing with children because that's your demographic. You wanna understand their motivations and goals. And as you can see with this poor child trying to drink water, things designed for adults don't always translate well to kids. And beyond the fact that the kids are just physically smaller, they also think differently about a lot of stuff. So they expect their digital interactions to be wild and zany and colorful and fun. And I guess we've just been trained to accept boring interfaces. But kids want that rich experience. And a lot of things that adults do very naturally online prove to be very challenging for kids. So for instance, we had one older child, I think she was about 10 or 11, she had trouble with the concept and the label of filtering very common web behavior. But for her, she hadn't had a lot of experience with websites to do that. So in that instance, it was a web experience-based thing. But sometimes kids just don't have like the mental construct even built up for to represent like the thing that the interface is supposed to represent. So the maturity and the development of the child can throw kind of a wrench into the works. So that kind of stuff is really good to catch early on. And even if what you're building is intended for adults to use, doing a bit of research with younger children can help your product integrate better into people's lives and homes. So one kind of product that springs to mind for this are things like smart home speakers, like Google Home or Amazon Echo. These certainly aren't products that are marketed at children, they're not toys or not for kids, but they're in home environments where children live and play and they say all kinds of weird things. And sometimes you get like a little customer delight where like, I don't know, the kid is really happy when they ask Alexa to make a dinosaur fart noise and then she does it and they're like, yay. But sometimes it goes a little haywire. I'm sure you guys probably remember this video that went around I think like a couple years ago where a little boy was trying to get Alexa to play his favorite song, but because he wasn't enunciating in the way in adulthood, Alexa kind of misheard and then started returning some results that were like very not safe for kids, not safe for work. And the parents were like, Alexa stop, no. So like if you test with children you might catch those moments of delight and capitalize on that, but you might also catch those red flags, those issues that might crop up, that wouldn't have cropped up if you'd only talked to adults. And then finally, when you simplify and streamline an interface to make it easy enough for a kid to use, you inadvertently make it easier for everybody to use. So whether it's a typical adult who's just like not focusing on the interface because they're multitasking or tired or whatever, or someone who doesn't fluently read or speak the language that your app is in, someone with a physical impairment, a cognitive constraint, making something so easy a kid can use it also makes it easy enough for everyone else to use it. And I'm not saying, if your product is meant to be used by a special population, you should always test with that population because that's going to give you the best data. But while my talk today is focused on techniques and strategies to use when you're testing children, I'm hoping that a lot of these can form the sort of foundation or inspire ideas for those of you who do need to test with other populations in the future as things to keep in mind or things that you can leverage. And then finally, just to give you a little bit of context about some of the examples that I'll be talking about today, a lot of this is going to come out of work that we're doing with the Girl Guides of Canada. I'm sure you're familiar with it, Canadian nonprofit for girls. You probably know them best for their cookies, which are, that was like a workplace hazard for me, being so close to them for so long. But they also do a lot of cool work to help young girls, girls and young women find different interests, foster their self-confidence and just build real life skills that are really relevant for the challenges that they're facing today and some of the challenges that we didn't have to face when we were younger. So my plan has been working with them to move all of their programming related content and activities into a digital format because previously it was all like literal paper books, which was not very scalable or adaptable. So our challenge was that girls as young as five years old all the way up to 17 could be girl members so they could be guides. Plus there are also adult members, the guiders who facilitate the meetings. So we needed to make something that was fun and easy enough that potentially a five year old might even use it, but wouldn't be alienating or seem too babyish for like a teenager and would also offer the robust functionality that an adult guider would need to like organize meetings and take attendance and all that stuff. So with that in mind, we carried out both discovery and usability research and some of the anecdotes that I'll be sharing will just come out of that experience. So setting it up. Before anything else, preparation is the key to success, quote from Alexander Graham Bell, who we know is an amateur, but I also found it recently, he got into sheep breeding later in life. I don't know why he was really into that. So there's a tidbit for your next trivia night. So once you've decided that you wanna do some user research, you're gonna need to decide whether you wanna do in-person testing or remote testing and both have their pros and cons. So in-person testing is almost always gonna give you richer data. You can see participants' facial expressions and physical reactions to things, which can tell you a lot more than what they're just saying on the phone or even in a video chat. This is gonna be especially true for younger children who might be shy or unable to express their thoughts for whatever reason. And certainly if you're testing with a population like with visual impairments, with whatever, you definitely wanna be in their environment as much as possible. Yeah, and you can also see the environment where they're gonna use this app. So maybe using your iPad game in an office is easy and fun, but you realize the kid using it is actually sitting in the playroom in the basement that doesn't have good lighting and then their sister's trying to grab the thing away from them so you get to see that in motion. Or maybe they're gonna be using your site on a school library computer that's running IE8 and won't let you download anything and times out up to 20 minutes. So seeing that real-life context can really highlight some considerations that you might have been aware of. But remote testing also has its benefits. You can be a lot more inclusive and talk to kids from a much wider range of demographics and backgrounds, both in terms of geography and in terms of socioeconomic status, which can be important for a lot of products if you wanna be inclusive. And it's just a lot faster to test remotely. People don't have to worry about inviting you into their homes, you don't have to travel to them, you just have to set up the screen share or whatever, and then off you go. So for our work, we did a mix of the two for the younger kids about under the ages of nine or 10. We stuck to in-person sessions for the most part. Although we did do some remote sessions, which were a bit wobbly, so I would say for kids under 10, stick with in-person if you have, at all possible. But for the older kids and for the adults, we tested them remotely using Skype and Hangouts and that worked well. Okay, so once you've decided that you wanna test in-person or remotely or both, you're gonna need to find some volunteers. You'll need to find some participants. But not like that, a Hunger Games volunteering scenario is not what we want. So what can you do instead if you're not gonna like conscript users? If you're like us, you might luck out and work with a really great client that has access, like direct access to a big pool of participants and will do the recruiting for you. In that case, we provided the Girl Guides of Canada with some email templates, just describing what would happen in the task that they could send out to the parents. We also gave them our criteria for like, we want girls from this age and these types of branches from these types of environments. And they worked really hard to set that up for us. You definitely wanna factor in a lot of lead time. So even with this kind of like blue sky scenario, it still took us about two to three weeks to get all of our participants for the first round set up. And especially if you're working with an organization that has like, like marketing has to review every communication that goes out or God forbid legal has, if they have to review it like double that. So give yourself lots of lead time. But if you don't have a client that has access to users, there are other ways to recruit. If you have any connection at all with an after school group like swimming classes, sports, music, whatever, try to capitalize on that. So you can ask your friend if they'd feel comfortable or if it's you, ask yourself if you'd feel comfortable talking to other parents about their research. You can give your friends a handout that just lists who you are, what you're gonna be doing, what's expected of their kid. And importantly, any incentives, that's always a big draw for people. So we've done cash incentives in the past or gift cards for the parents. There's just sort of incentivize them to volunteer their kid for our activities. You can also try recruiting online, blast your own social media networks, posting in a topic focus group. So I know that there's like Facebook groups for parents based on neighborhoods. So if you've got friends in those groups, you can see if they would feel comfortable posting. And there's also some boards that regularly post marketing or research opportunities. Like we've had success on Buns actually posted to someone there. You can take this into the real world as well, put up posters where you think young families are likely to hang out. The library, for example, swimming pools, day cares, restaurants, wherever you think families are gonna be. And of course, you'll have to get permission first. If you are really lucky, you might try the same thing with the school. If you've got an in with someone on the school board or principal or something, although there's probably gonna be a lot more red tape there. And then worse comes to worse or if you just have lots of money, you can use a recruitment agency. That'll be your most expensive option because you have to pay them to do the work, but it can be a lot faster and also help you out if you don't have a lot of resources to tackle that. So if all of this seems like a lot of work, that's because it is. Every project I do, I can't believe how much time we sank into setting this up. But once you get the ball rolling, once you have your template set up, your pool kind of, it's a warm introduction the next time. The next rounds of recruitment will go a lot more quickly. I will say that you should recruit way more people than you think you need. Like if you want 10 people, reach out to at least 50 people. Like a 20% return is already really good, especially for a cold call, spam everybody. You can always decline and say, oh, we've filled up all our slots this round, but can we keep your info on hand for the next rounds that we do? And people will always say yes. And it'll give you a backup pool too for when people cancel and they will cancel, especially with kids because schedules get unpredictable sometimes. All right. So you've reached out to participants, you're trickling in, you're starting to schedule interviews. The next thing you need to do is to write your questions, your testing guide. So when it comes to prepping the questions, you're gonna ask, for younger kids especially, we found that they handle concrete questions that are based on their own personal experience more easily. Some kids might have trouble with questions that require them to imagine what other people feel. So this can be true for neurotypical kids as well, especially younger ones. But for instance, we had one parent asked to review the questions in advance because her daughter was on the autism spectrum and she didn't want the child to feel put on the spot if we were gonna ask her like, oh, what do you think your friends would think about this or what do you think another person would think about this? So that's something to be aware of. I'm not saying you can't ever ask about hypotheticals or ask kids to project what they think other people will think. It's just that be aware of it and have maybe a question in your back pocket that kind of gets at the same thing, but maybe couched in more concrete personal terms just in case that the person you talked to is struggling with that. Another example of this came up when we were drafting some introductory questions to sort of just like warn people up. So I thought foolishly that a question like, what's the best thing about being a girl would be like a casual, low ball, easy thing to like girl talk, let's just jam. Terrible idea in retrospect. Like if someone asked me, what's the best thing about being a woman or a Canadian or a designer? Like I don't know, I have to think about it. So asking a six year old to come up with that on the spot was not the smartest idea. We had much better luck with questions like, oh, what's your favorite subject in school? What games or TV shows do you like? Kids were much more enthusiastic about that. They would tell us about, oh, I like science because I like explosions or I like math because I'm good at it. And people don't think girls are good at math, but I am and that's cool. And that kind of stuff helps give us those experienced criterias like how kids wanna feel. Like we wanna inspire feelings of pride and excitement. So that can be really valuable beyond just being a warm up question. And this early stumbling block with that bad question, that brings me to my next point, which is that you should pilot test your session if at all possible. Another way of saying this is like shit test your stuff. You know, find a kid even if they don't belong to the exact demographic you're looking at or recruit an extra person and test them early, but give yourself enough time to iterate on the actual test script. So this will help you avoid these issues. So when we pilot tested, we found the first question was bad, so we were able to drop it. Even if you can't find an extra person who fits your demographic, like ask a coworker to just role play for you. Obviously, it's not gonna be perfect or any go close to what a kid will actually say, but it'll give you as a researcher just a chance to run through the motions of your script. If you're testing with a special population and there's like special terms that you need to use, it'll just get you used to saying those terms of phrases so you don't like stumble over your words and inadvertently say something that, you know, is offensive or problematic in any way. As well, we generally recommend that you have at least one other team member with you in in person sessions taking notes, because it's a super weird dynamic to like have this in-depth conversation with someone and then be frantically typing and jotting down notes on the side. It kind of makes you seem like a weirdo and it really throws off that session, that rapport. So having that trial run can help you and your teammate figure out how to best situate yourself if you've got recording equipment, how to figure out how to best position that so that when it comes to your real testing sessions, you don't have to waste time setting that up and figuring that up. All right, so you've scheduled your people, you've recruited them, you've written your questions, you're ready to go. So let's talk about some things that you can do to help your sessions go smoothly. All right, so when you start your sessions, when you explain to people what's going to happen, framing is always important for any participant, but it's especially important for kids. Let them know what to expect because they've never heard of a focus group, they don't know what this is. Adults at least have some conceptualization of it, but a kid is just like, there's a weird grownup in my house now, okay? So explain to them what's going to happen. Let them know that they can ask questions, stop if they get tired. If you're going to ask about any potentially sensitive topics or even just if you're asking them about things they didn't like about a particular experience, it's important to let them know that no one's going to get in trouble, no one's going to have hurt feelings, like nothing bad is going to happen based on what they do in that session. Kids can be susceptible to the social desirability effect, which is the desire to say what they think is a good answer or the socially acceptable answer, even if it's not what they really think. And that can be especially true if there's authority figures present. So you just want to try to build a safe space where kids feel free to be open with you. But then on the flip side, you'll get kids who just have like no filter at all. I showed one design to, I think she was about seven, a seven year old kid. And I was like, okay, what do you think you're seeing? What do you think this is for? And she's like, boring stuff. And it was like, okay, well, I guess that's boring. I worked hard on that, but that's cool. And I mean, that's fine. That's honest feedback. She gave the right answer. It looked boring to her. And that's totally fine. And if that happens to you, you can just pat yourself on the back and like look at what a safe space I created that this child fell free to tell me that my work was boring. So you did a good job, if that happens. Maybe not on the design, but anyway. Yeah, so tell them it's okay if they don't know how to do something. If they get confused, they can tell you so. Just like when you're testing with adults, with anybody, you really wanna convey the idea that it's not the participant that's being tested. We're not interested in their ability. It's really testing how well or how poorly the site has been built. So we tell them things like, if you find a mistake or something is hard, it's really helpful when you tell us that because that's a mistake that we make and it's gonna help us figure out how to make it better for other girls in the future, other kids in the future. So just try to really drive that point home. And yeah, once you're about to start your questions, creating rapport with your participant is always important. But again, with younger kids, they might need a little extra time to warm up to you. So one thing that we found that was helpful was to break the ice by offering the child a choice of activity. More energetic and quieter, just based on the personality type what they like, but also there might be like a parent sleeping in the other room so you wanna let them not be too disruptive. For the more energetic option, we had a lot of success with a freeze dance game. So apparently like all five to nine year old girls are really into Katy Perry or Justin Timberlake. So have some songs downloaded, your Spotify history is gonna get kind of messy, but it's for the sake of research. And yeah, you might feel a little weird dancing with a second grader, but they're not judging you and they're just there to have fun. So you can channel your inner Elaine Benes and just let her out. Having a parent present might also help and in fact might be unavoidable because most parents are not okay with just leaving their kid alone with some internet stranger that's come into their home. So if that's the case, just make sure to let them know ahead of time that they should be silent observers and try not to like give the answer to the child or sway them. You can tell them in an email beforehand. And you can always save a few minutes at the end to ask for the parent's feedback and impressions just so that they feel included as well. When you're asking more of these discovery or exploratory questions about preferences, often with adults will say, okay, so on a scale of one to five, how would you rate this? How easy or hard would you rate this? For a kid though, you know, it's less straightforward for an adult. They do the stuff all the time, but for a kid, they might have a harder time conceptualizing that. So we made use of face scale so that we kind of illustrated every decision point or every choice option along the way from very hard to very easy. And then the second question we had asked them that I'm showing you here is who do you think should plan your meetings? Your Girl Guide meeting should it be just the girls, just the guideers or girls and guideers. And that just gives us all more confidence that the girls themselves understand what we're asking and understand what their options for answering are. And it gives us more confidence that the answer they're giving us is actually legit instead of them just saying the last thing we said because they don't understand what we're trying to ask them. When it comes to introducing the usability test, so the part where you're actually testing the interface, you'll wanna keep the language as simple as possible too. So in our work, we say the word prototype and feedback and usable and these things like very easily but with a child, you're gonna wanna keep it simple. So maybe say pretend or fake website instead of a prototype or what you think and what you feel instead of the word feedback just to make sure that they can understand that. We also found that children were very, very literal when they looked at websites. So an adult, if you show them a wireframe, you can just say, oh, this is just like an early mockup but they'll probably get it. If they see more MIPS, they'll get it. But kids might not. So you can show them an example of text and say you might see some words that don't look real. That's just cause we don't have the real story written yet but eventually that would go there. Or if you see this gray box, that's just because normally we'd put a picture there but we don't have it ready yet. So we're just putting a picture there or this box there for now just to give them a little bit of heads up about what to expect. And when it comes to testing the prototype itself, it's much easier on you as a researcher if you make it as realistic as you can. We tested the site when it was almost ready to be released and that was literally like, I can just say try to do this and I sit back and I watch them. But when it's a clickable comp or a wireframe because we don't have all of the states mocked up, it takes a lot more finagling and creativity to get them to go where you want them to. And one thing that can make it easier is just to link up all as many states, as many options or different screens as you can because kids are a lot more willing than adults to just kind of go in and click around and click on things that you don't expect. So make sure you do have a quick way to reset the prototype so hide some links in the photo that can bring you to certain key screens or something just so that you can get your testing back on track. And yeah, use realistic icons and imagery. That's a big one, especially for younger kids who are still learning to read. That was a big stumbling block we came across because it was a content heavy site. There were lots of words, but kids couldn't always read yet. So they relied a lot on the icons and the imagery, which also makes it look less boring as well. Not that I'm stuck on that point. Get ready for things to go off rail. So kids might see a piece of content about camping and then they go off on this long story about how they went camping with a best friend and the dog fell on the river and then they ate fish. It's a cool story, but you kind of have to bring them back to it. It's totally okay to be like, oh, that's really interesting. Let's talk about that later. But for now, how about you try to find a video on here or whatever. It's totally okay to do that. I think kids are much more used to that type of direction than adults might be. If they really do get lost and they start to get frustrated like they don't know how to do something, you can prompt them by saying, oh, well, what do you think would happen if you clicked that button? Or what do you think this thing is for? That doesn't work. You'd say, oh, let's just try it together and see what happens. And then ask them afterwards, is that what you thought would happen and try to probe for that that way? I'd also say that you should give positive feedback to the children. Some schools of thinking say that you shouldn't give any feedback at all to a user about whether they've done a task correctly or incorrectly. And I personally, I find that that feels very robotic because you're asking a person to do a thing. They do it and you're just like, mm-hmm, next. It's just like a very strange dynamic. So you don't have to tell them that they did something right or wrong, but you can just say, oh, great, good work or nice, let's keep going. So you don't have to explicitly say you did that wrong or you did that right, but just a little bit of feedback to make the session go a little bit more smoothly and make the kids not feel like they're failing miserably or something at a task. Yeah, so logistics, as Sun Tzu said, the line between disorder and order lies in logistics. Sun Tzu, the military strategist in Shenzhen, don't know how he felt about breeding sheep. I was not able to find out his relationship with sheep. So here's a photo of me and my colleague, Jesse, after one of our in-home sessions. They had this pet pug, this dog, that became obsessed with us. Like he was at one point, like he's still watching us after we're outside waiting for the uber. At one point, he was like sitting on my lap and I'm like reaching over him to try to type notes on my keyboard and interview this child at the same time. It was probably the funniest testing session I've ever sat in on, but all that's to say, dress really informally when you do these home visits and not just because there might be dogs or you have to sit on the floor or the kid might spill something, but you don't wanna give the sense of like, I'm this business person, this authority figure who's come into your room to examine you. I always say dress like a camp counselor, like dress like an adult who they might be used to taking direction from but also just having fun with. So you just wanna have that openness and that casualness in your interactions with the kids. For in-home testing, if you have allergies to pets, bring medication, if you have allergies to siblings, I don't know how to help you. If you can't be around certain pets, ask ahead of time, maybe ask them to be put in a bedroom or maybe just have someone else do that session for you. When it comes to siblings, sometimes parents, if there's only the one of them at home, they can't be with you and the person you're testing and manage the other child. So it's nice to bring like a coloring book or a small toy or something just so that the sibling can sit at the same place as you and not be bored. Download your prototypes. Oh, we found that out the hard way. You're not gonna have Wi-Fi at these people's homes and even if you do, it might not be very good Wi-Fi. So make sure all your sites, your prototypes are all on your local machine. Bring a mouse as well. So mostly like younger kids are very used to using touch pads for most of their digital interactions. If you can test on iPad or iPhone, great. But sometimes as you're developing or designing, you just do the desktop and you're not ready with the responsive yet. So it's totally fine to test the desktop, but it's a good idea to bring a mouse because like a Mac trackpad is really, really tricky for kids to navigate. Leave yourself tons of buffer time. People come home late. The kid has to have a snack before they can talk to you. They run off for 15 minutes. You can never predict things as easily as if you're doing a remote session or you're bringing people into your lab. So just make sure you're not stressed out about getting to your next appointment in time and leave lots of room there. And then finally, it's always nice to bring a little prize for the kid at the end. So this is in addition to any incentive that you might be offering the parent, something like a sticker or like stickers that they can choose from or a small toy, like the treasure chest at Swiss Chalet. I don't know if you guys remember that when you were kids, but that's always really popular. We do try to stay away from any food because kids have all sorts of dietary restrictions that we may not know about. And then finally, really at the end of the day, your main goal is to get data, but I'd also argue that you should have fun with it. Like, you know, if we were bankers or lawyers or whatever, like we would not get to sit with kids and watch them work with the tools that we're trying to build. So I think it's a pretty unique and interesting aspect of our work. If a kid is really getting frustrated or bored or tired, it's okay to end early. Like you'll always need to be flexible and think on your feet in any testing scenario. And that's gonna be doubly and triply true when kids or any special populations involved. And ultimately, I think our job as product creators is to treat, our goal is to treat users well. And whether that's, you know, through the product that we build or through our interactions with them in a testing session, I think ultimately you want that person to leave the session with a really, really positive feeling about it. And then one of the best parts about doing this type of research is just seeing the kids like unbridled enthusiasm for things both positively and negatively. So it's really inspiring to hear kids talk about, you know, the ways they like to learn, be motivated to do certain things. And then as a bonus, sometimes they give you really amazing responses that are honestly hilarious and provide great soundbites that you can show your clients like our friend, Kiana here, who said for a fake website, you guys did pretty good. So thanks, Kiana. Yeah, so open yourselves up to being inspired by the energy of these kids that you speak with and the users in general that you speak with. And hopefully that can carry over into the work that you're doing with the product that you're building. Thank you. That's all of our, oh yeah. So we're hiring. So if you're interested, you can come find us on the last door in LinkedIn. Thank you. Any questions or comments from anyone? Yeah. I'm just gonna repeat the question because this is being recorded. So I have a question with how long are the sessions and what's the incentive involved? We try to keep it less than an hour for kids. And really the testing itself is probably only half an hour with all of the setup involved. Just because anything over an hour, they just can't focus, like adults can't focus after an hour. And for the incentives, we typically do between 25 to 50 per hour. For the Girl Guides, they actually just did it for free because they were invested in the program and excited about it. But when we do these things with people in the world, we range from 25 to 50. The more specific your target demographic is, the higher up you'll go. So for families, we might do 50 just because they have to invite us into their houses and that's a bit of a constraint. For if you're just testing adults on like a regular consumer app, we get enough people with 25. And then we've gone up to like a hundred for very specific people. Like we were talking to, I think, CTOs at some point and you know their time. You kind of like scale it to how much their salary might be, which is kind of a gross way to do it. It works. So yeah, it ranges. But I'd say for kids, you're looking at about 25 to 50 for the one hour. So how do you aggregate all of the intel and then transition it into actionable intelligence again? Yeah, great question. So how do we take all the data that we get from the sessions and kind of make it into something we can use? So that's the synthesis process, which follows the research. And essentially there's lots of methods you can do it. You can do affinity mapping where you're kind of finding similar points of feedback between kids and sort of like clustering them together and say, okay, so this seems like it's all about, I don't know, the exploring flow isn't easy to use or they want more visuals in this or that. So it's really kind of looking at all of the data and pulling out common themes from different types of users and then creating sort of an output of that. We also create a user scenario map, which is kind of mapping through the user's needs in a roughly linear way across the product, which is meant to be like a living document for us to make sure that when we are building out the product, are we hitting all of these goals? Are we getting all of those goals? Or are there ones that we've deprioritized that as a team collectively decided that's going to be a future release or that's not for this product? So it is that process probably takes, say we leave like a week for every two weeks of research to do that. I mean, at some point you can do it faster, but yeah, so it's a lot of, it is very qualitative in that way. It's not like a quantitative check check. When it comes to usability testing, it is a bit more quantitative because it's kind of just like a pass-fail. So if they are able to do the task, if most people are able to do it, then you're probably okay. But if you're finding like 30% of people are failing at the task, then that's something that you can highlight. Does that answer your question? That was perfect. Great, cool, yeah. So correct me if I'm wrong, but I think this kind of the purpose of this is to try and apply these tactics to also user testing with adults, correct? Yeah. So is there anything that you find that works particularly well, that transfers well between children and adults and anything that doesn't work at all? Yeah. Like identifying with stickers? Yeah, good question. The question was how, which one of these methods translate well for adults and which ones don't? Yeah, stickers, adults are like less into that, although I haven't tried it, maybe I'd be surprised. Adults are obviously much more motivated by the cash incentive. They need a lot less of the warm-up stuff because you can just kind of explain, you still want to explain to them what's happening and what the goals are, but you can pretty much just dive right into the questions. Generally, it's always good to stick to sort of simpler language regardless, but with adults, obviously you have a lot more leeway, especially for the types of tools we build because they're so context specific, like if it's the data admin person trying to manage data centers, there's going to be a lot of lingo that you obviously wouldn't use for a kid. So I think with an adult, you still want to do a lot of these, the steps will be there. I think you're just going to have an easier time with it because you have to spend less energy to think, like, will a six-year-old understand this? But generally, the recruiting, the framing, the explaining that it's about the interface and not about them, the debrief, I think the general workflow is going to be the same. To, I guess, make a typical interaction as close to a real-life interaction as possible. For instance, I work for TVO, but a large portion of our kid's audience actually visits the site using public computers, be it at a library or at a public school. Typically, the processing power of those computers is quite low. Is there anything that you do, if you were to conduct this kind of research at their homes to simulate that kind of experience? Yeah, so the question was, in cases where the technology constraints might be kind of like limiting, is that right, where kids are accessing your tools in a way that it's not ideal, like the bandwidth is bad? Well, even in a greater context, let's say they typically don't access this product via their home, and they typically have to sit through other means. Is there any way to kind of simulate accessing it through other means? Right, so simulate the actual context that they're using it in without actually being in that context. So we haven't had to do that for this project just because it's kind of where people would use it. We were aware of the guidance would be using this app or the site in the meetings, but they may not always have internet, like data or wifi, where they're meeting. So we had to make sure that there was enough functionality that they could do offline. So that was one thing we had to think about. But in terms of creating distractions, you can always kind of give them a false load task. So if it's kind of a mental load, you just want to have a bit of distraction in the space, you could set things up like, bring a couch into your lab and have them sit, but then also have a TV on that's not related to what they're doing or play some loud music or you can even do, like this is not very realistic, but you can ask them to do a cognitive load task concurrently, so like count backwards from 100 by three. And they have to do that while they're doing your task. So it's not the same as, oh, you're in a busy library or you're in the street, but it can sort of add a little bit of stress, I guess, or a load to the person as they're doing it. So it's not as clean as, you know, just doing it in a quiet room with no distractions. That being said, it's always going to be better to do it. Like if you're building an app or you're going to use it outside, take it outside. Cause like we found that some screens, like it looks great inside, but once you're out in sunlight, if it's like a smartwatch, the glare is so much or the contrast is an issue. So as much as we can try to mimic it, I think it is important ultimately to at least take it out. Even if you're not testing it with users, take it out yourself and try it out in the context that you would be in. Yeah. I'm curious to know if you've ever been in a session in which so you've got the child over there and then you are she always like mommy or daddy and then and pretty much looks to the parent for the answer when what really needs to happen is for a good child to respond to your problems. Yeah, great question. So what happens when you want the kid to answer but they just look to their parent and say, mom, what do I do? What am I supposed to do? That's when prepping the parent beforehand can really help. So they can be your ally in this case. If you say, you know, because we're building this for the four year old or the five year old to use, we need to know what they're doing. So if they ask you about it, help us out by redirecting it back to them to say, I don't know, what do you think? You know, just ask them to sort of be your ally in that testing session. And then sometimes it's, you know, the parent just won't listen to you or maybe you didn't remember to remind them. So they'll just answer for the kid and they'll just like lead the kid and that can be frustrating but it's also not completely unrealistic because if a kid doesn't know what they're doing, they're going to ask their parent anyway. So you can still get good data out of that. It's just, if you set it up in advance to tell the parent to try to like redirect it back to the kid, that's the ideal scenario if they can help you out. But even if not, you can still get worthwhile data out of that because, you know, it's not unlikely that a parent and a child are going to work on the same interface together. Okay, great. Well, thank you all very much.