 I'm Kenny, I'm a research scientist at BBC, especially as I'm a music expert into research. So today I'm going to talk to you a bit about my role, my backgrounds, how I got through where I am, a bit about real world UX research and how perhaps, so I'm just going to tell you what I'm doing. So research scientists doesn't really mean anything, most people don't know what I do. So research, I'm actually a behavioural research scientist when people don't understand what I do, otherwise I'm a UX researcher but being at a UCR and doing that to make it sound important to the business scientist. So my background is in psychology and then I did a master's degree on teacher experience and focused on music experience, use those in that good interaction style and approach studies to make them easier for the kind of things that are bad at stuff. So my background is in music experience, I'm not a designer but I did use to be a designer but that's not kind of the principle you find out now, it's more research based. And so in research development these are the kind of things that I do and those are evasives. So literature reviews, so basically you have to become an expert in a field very quickly. According to a new project with a new device, new system, new experience, we want to know exactly what that is straight away in order to start informing the design. Things like starting a participant, I'll talk about that a bit later, I've got a few projects that I've been working on, I'll start. The main part of my job is designing experiments, whether that's field-based or lab-based, sometimes we go to the plane, sometimes we go to work, sometimes we bring them into our lab. Analyzing data, we need to use what we're expected to do, qualitative and quantitative analysis. Writing reports and recommendations and lo-fi prototypes as well, we kind of are expected to do things like wireframes and suggestions for the sun. So this is where I work, this is our lab. So this is where we want to do quite a control experiment, we can bring people into the lab. It's meant to look like a living room, it's meant to look like a bachelor pass. Basically, New York City is quite new and over-designed, therefore our lab had to fit in that as well. This is our control room, so we've got a one-way mirror. That's in pitch black when we're sat in there, we have notetakers in there, people just writing down everything that's happening. All maps of cameras can't really see that white bubble, we can move those around, we can zoom in with them. We've also got lights on the ceiling, can't really see that either. So we can listen in the control room to everything that they're saying and to write down everything that they do. So on the screens there, we'll also see what they can see on the screen, whether that's computer or TV. We've got a base at the back here that's for computer research, so anything on a PC. Recently we've had children in, looking at a CDC website. We'll do that at the back, much more focused than the corner, but a lot of the time it's been watching the TV. So, generally what I do now, it's all about understanding new technology, but within the media, is there anything to do with the services or new technologies that can be production-based stuff, or it can be consumer-facing stuff. So you could be helping the people in the studios with new technologies, or ultimately understanding the experiences people are having at home consuming our content. So, just to give you an idea of the range of different types of projects I do and therefore the kind of skills that you need to be a UX researcher, this is very much the start of the project. We wanted to build a voice interviewer interface for TV, so initially, so visually impaired people could search for programmes, plan their listings, what they want to watch, things like that. We don't really know anything about them, so we brought them in and interviewed them, but very basic stuff. We want to know what they do now, what problems they have in terms of their TV now, what would they like, what do they think of a voice interface, and how would they expect that to work. We haven't built anything yet. We've basically gathered some requirements to start to feed into how it would start to build that system or prototype that system. So I've got a video of one of the sessions. You can see how we're going to interview those. I've got a lot of data. Thank you. It's a good example of my life, part of an interview with a visually impaired participant. You might be able to see her guide on the TV as well. I was sitting through the TV. Do you use the guide on the TV to bring out the guide and how we look through that? I know it's definitely not very clear on the virtual box. I don't know what's on the internet. So you use that as your programme guide, and then go back to the TV to watch it. That's interesting. What do you go to to look for the programme guide? Is that planning to watch stuff later on, or could that be to watch something now? If there's nobody in the community to look for the programme guide, then that's what it is. So if there are people with you, they read out some of this channel. They just skip all the things they know what you want to watch. You don't want to skip all the ones that you think about. They're relying on their opinion. So you don't know where to look? Mae ydych chi'n ddim yn dweud o'r ddweud i gael yma, ac mae'n gwybod yn gwneud bod drwswn i'r gael yma, sydd wedi bod yn ymweld ddweud yma, ac rwy'n meddwl i'r gael yma, ac rwy'n meddwl i'r gael yma, ac mae'n gweithio i'r gael yma, ac rwy'n meddwl i'r gael yma, ac rwy'n meddwl i'r gael yma. Yn mynd i'w neud, rydym yn gwneud rhai o'r cael ei aud oherwydd eenderwydd y nifer. Yn ten mae'n fideo yw'n ddechrau, mae'n rai o'n obaeth iddo, mae'n edrychwch yn cael ffynuedd yn mynd ym gerthu. Dyma sy'n cael ei ddechrau, mae'n taeth i'r diwrnod yn ei ddweud. Dyma rhaid i'r cyflwyso rydym o bwyloedd bwyddiol, ychydig gwyddiol sydd. Algrwp yn cael ei fod yn ystafell i gyda, gallwn peir. Mae'n gweithio bod yn yr unig, ond mae'n mynd i gael. Rwy'n cael ei fod yn eistedd, byddai, a dwi'n gweithio bodonod yn cyd-айт, lle ar y cyflen! Rwy'n cael eu rai fod mae'n ceisio. so this is the kind of output you get that you can read through and... Basically when we are analyzing the data you won't be looking for anything in particular you will just go through and pick out the really interesting points that they say I highlighted there some of the points that I would pick out to have impacts on what we ante to achieve with their voice system So, mae'n gorabed dda sayi here. Basically, trying to understand what she does at that moment and do we want to fit into her character behaviour or do we want to give her an opportunity to do something new? So, instead of having to go to the computer and look up tv listings, The voice interface to the TV is ultimately going to take that stuff out so she doesn't have to dedicate half an hour of her day going and looking what someone's moving. Mae'r blwysgol hefyd yn ei fod yn ddod i'r cwmwy, yn ddod i'r cwmwy, ond hefyd yn ddod i'r cwmwy, mae'r ddod i'r cyfrifiadau a'r ddod i'r cyfrifiadau. Mae'r ddod i'r cwmwy. Mae'r ddod i'r cwmwy ffordd fel wahanol, oherwydd i'r ddod i'r ddod i'r ddod i'r cyfrifiadau? Mae'r ddod i'r cwmwy. Mae'r cynnwys yma, mae'n gweithio, mae'n gweithio ar ei trwy, ond mae'n gallu ei ddweud. Mae'r gweithio ar gyfer y tiff, mae'n gweithio ar y dyfodol, ond, wrth gwrs, mae'n gweithio'n gwisio'n gweithio eu cynnwys. Mae'n gwisio i ddim yn gwneud o'r systemau o'r cynnwys, mae'n gweithio'n gweithio i gael'r cynnwys. Si'n gweithio'n gweithio, mae'n gweithio'n gwneud. Ond ydw i'w ddim yn ddweud rydw i'r cyffredinol, felly rydyn ni'n bwysig ironiolicheu casur maen nhw'n dweud i'r zypadaeth og. Rydyn ni'n ddweud i'r ddweud i gemyn ar y cyffredinol yn dweud i'r ddiweddol? Rwy'n meddwl fod yn hi ddweud, ac i'n meddwl gwmpodd gan y gyffredinol, fe wnes ond iddyn nhw'n ddweud i'r ddweud, efallai rhwng yn ymddi o'u gweld, A am meddwl y gweld y tîm yn ddwy i oedd y tîm mwy o'r amser ac yn y plwy mwy o'r amser. Byddwn yn dnif yn unrhyw brydion ar gyfer hynny, hefyd yn ammysgol i'n blynyddion y lleol honno, ac yn ychydig fel ei bod mwy o'r amser. Mae wedi'n eisiau ychydig, mae'n ddiwedig sy'n mynd i'n addysgau. Penthyl y gallwch chi'n dweud i'n ddifig sydd dros yw, mae'n ei ni'n meddwl i. Mae eisiau mawr ei fawn. Mae eisiau mawr ei fawn iddi'n meddweud. gysylltu'r myfyrdd o bwysigol cyntaf gydag y kryddai hynny yn cael ofael y byddai yn ymweld fel y cyfrifodol mae gan gydag gwirionedd. Felly mae'r cyfrifodol yn gallu hynny o ddifud i'r bobl yn y stadau rydym ni, mae hi'n gweithio chwinogion, ond gydag rhag rydyn ni i fyny i'ch o ganfer gydag y thrathau cyfwil, cyfan. Mae'n rhaid o phrojectau ar y dda, yn gydag ei wneud, ac mae adeilisiadau yma ychynig. ...a'n dod o'r tyfu, o'r teimlo wedi'i'r cyffredin... ...yna'r gennyddu nifer o gwaith o'r rhaglen... ...yna'r gwaith o'r llwyddiad yn y pwysig... ...yna'r gwybod yn cynhyrchu i chi... ...yna'r gwaith o'r rhoi gwirioneddau... ...a hefyd yn gwneud y pwysig... ...a hynny'n gwneud gwirioneddau sy'n ei wneud... ...ol ydych chi'n gwybod yn gwybod ar y gweithio... ...oherwydd os ydych chi'n gwneud... roedd yn gweithio'r cyllidol sy'n gweithio'r cyllidol yn oed. Mae'r cyllidol yn cael ei fideo a phoblau ymlaen nhw, oed yn fwy o fwy o'r cyllidol. Roeddwn ni'n ddiddordeb, oherwydd y dyfodol yn y bryd i'r cyllidol, oherwydd dyfodol yn y bwysig. Felly, mae'r ysgol agen nhw yn amlwg, a'r cyllidol yn ddifu'r cyllidol, roedd yn meddwl, ei bod yn dweud am gyffredig, ond mae'n meddwl, o phosaneidd, gweld i, a newydd dalau y tîm wneud. Felly, mae'n ddweud i'n gyd deinwys ar y cyfnodd hwnnw chi yw'r onaidd yng Nghymru. Rydyn ni'n fawr'r tîm ar ddod, efallai os yn gynllun i'r tîm gyffredigau, llawerr yn ddflwyio. A oes yn canhwch yn y tîm, mae'n ddflwyio, yw'r ysgrifennu i'w ddlwnnui, a'i ei fod yn cofymu. Ar ddwl unigid gwybod eich anffreddol byddwn i'siwch iawn. Dbylch, a dwi'n med hyn fydd drws. Mae'r neud am gwybod eich anime'r anffreddol, mae'r neud i gwybod eich anffreddol byddwn i hisniadau, mae'r adeithi dod o'r cyflwyno o'r gwahyd. Rym ni'n lle newsyddiaeth oherwydd nesaf oherwydd oed i oedd o'r teimlo'ch sefydlaraethchildren で different sizes which was their preferred size, different colour combinations. And when I joined the team I wanted to put together some guidelines of size of the text. So I would put in the isolate sized variable in the experiment. And so that's kind of where this one goes on. It's not a bit of a crano, it's just a bit cheesy. But I'll show you so it gives you an idea of what we did. Participants was put into 2 groups. Felly, eich gweithio, mae wedi gwneud o'i serifon o'i serifon yma, y serifon wasg yzelfol o ddimensiwn, ac maen nhw'n gweithio serifon. Y cyfnod o'r tex yw'n gwneud o'r try, y cyfnod ddechrau, y cyfnod yw'r cyffredin, y cyfnod yw'n cyffredin o'r gweithio o'u gweithio. Mae ydych chi'n mynd i gweithio'r ysgrifennu, yn gwybod i gael eu gwirioneddau i ddwyng, yna'r rath o'n gweld o'r ysgrifennu. My first question we learned is what they're drawn onto on the screen. That gives us an idea of a tension or focus when these screens come up. We would ask them something about what they think about the size of the text, on the screen and we would ask them to read part of the text on the screen to assess the litigability in its current form. Then we would ask them if they wanted to see any of the text parts any bigger or smaller for them to set them to their preferable reading level. The text was made up of different parts. valuable to see it. We have body text, the subheader and the main header. And I would go through each part and ask them whether they wanted to see it any bit or smaller. Three of the screens look at the ratio between video to text, to understand the interaction between the text and the video and whether that has an impact on the legibility. One of the screens was the scores table with a video playing behind and we looked at the different text elements and I reduced the size of those to find their smallest for each part of the text, so the header, the sub-header and the body. The idea for the sub-header and the header text was to find the threshold of the point where it started to lose its distinction from the other text. One significant finding was the difference in the older users' smallest acceptable font size and the younger users' smallest acceptable font size. The impact that has on the guidelines font size, it means that we have to concentrate on the older users' smallest size cwm ydych chi'n meddwl o'r cyffredinan? RwydICH iawn, dwi'n deillad o'r llwyll iawn ac yn ddweud. Mae'r cwm ydych chi'n meddwl, nid ddim yn rhan o'r llwyll iawn. Mae'n ddiwethaf cyffredinan o'r llwyll iawn, iechyd bod yn ni, mae'n ddweud, dyma i ddod ei dda o'r cwm yn gwneud.chine allan o'r llwyll iawn, a wnaeth i chi wedi eu pethau'r cwm yn ddweud? that no video is no nothing in the screen, just a block of text so then I can measure any adjustments they made in any of the other screens to see if those screens have environments on their reading level. With this so design wise kind of because I was designing how I can do these things, but we do work with designers as well so we're not expected to know how to design these screens ond hyd yn ymgyrch arno ar y gynllun i ddau. Mae'r profiwyr yn ymddych chi'n gwirioneddol yn gyffredinol ac mae'r profiwyr wedi ddod y gallan ei bod cynnid i ddinkarol am y cyflawn. Felly mae'r profiwyr yn ymddych chi'n gwirioneddol a gyd yn ymddych chi'n gwirioneddol yma i gyrtiadu busgu. Mae'r profiwyr yn ymddych chi'n gwirioneddol a ddyddwn i gyd yn ymddych chi'n gwirioneddol yn ymddych chi i gyd, ac ydych yn ymwneud i fy hun i'w bwysig i'w cyvelygu armug yr ysgol, sy'n cyflan â'r cymoedd, yn hyn i gan gwasgau. Felly wnaeth youn yn chef i Bodi Gael, o'i gael yr ysgol ymlaen i'r llyfr, rydyn ni'n gwybod o'r anodd ac y dryf ar olwyr o'r cyflwyll. I bwysig i ni'n chweillio'r ymchwil, ac yn y teimlo, o'n gwneud i gael yr un o'r treicoli. Yn y cwrnod, mae gennym o'r cyfwyr i'r cyfwyr oedd yn ddweud yn fawr. Cwrnod o'r cyfwyr i'r cyfwyr i ddaeth am y cwrnod yn y pethau, mae'n gweithio'n ddaeth. Mae'r cyfwyr yn y pethau, mae'n ddweud yn gweithio'n gweithio'n ddweud. Mae'r cyfwyr yn ei ddweud hynny, mae'n ddweud hynny'n gweithio'n ddweud. O'r f gleichbwladau yn rhaid i'w prif, gallwchныnau'n rhaid i y newid i'w unrhyw y shapedelion. Mae'r fudd i'w cydnod i'r pwylfaen, fel gydwch chi'n ddylch i y styrm, mae'n ffordd i'n ffordd i'r pwylfaen o'r eu pwylfaen, mae'n ffordd i ddwy'u gatio'r pwylfaen yn eu rhwng. Mae'r rhaid i'w pwylfaen o'r pwylfaen roedd y chyfrnod, Ond yna hiwn, rhoda, ond felly rôl ac roedd o'i ddweud y cwmal functionality yn eich cyfrif ymogol i ddiwedd i'r ddweud, rydych chi'n credu hefyd i'r hunain a'r cyfrif yn cael ddwybod… Toru'r cyfrif yn reisio, roedd unrhyw gwaith yn rhaid o'r cwail sy'n rhaid o erbyn mwyn ddim eich bydd yn ddysgu'r iawn. Yn rhaid o'r hyn o'i gweithio, mae'n ei gweithio'r llyfr yma'n granteis. Mae'r ferwedd diwedd ar y honned, mae'n myfyrwch mewn method Everyfyrwch yn myfyrwch, a'r embygwyr hefyd. Rydy'r maes ychydig ar y maes i gweld, yn ei gweithio'r ddigon, a fyddwch yn ei gymwymiadau, dewch eich bod yn oed. Gweithio'n rhannu cael ei hynny iawn i gwybod gyda yn cysylltu i blynyddiadau disagreeoedd. Mae'r Plynedd wedi yw beth ydi chi i ni ddarllewch ac yn fwygof, a chi'n rai wneud o phobl arferwch gweithio femeidol. a phobl yn ymwneud i'r ddigon o'r tŷ ar gyfer y ffordd o'r ffordd o'r tyffan, i ffawr i'r cyfan. Wrth gwrs, mae'n rhaid i'r cyfan o'r cyfan. Wrth gwrs, mae'n gwybod fel ymddangos i ei ffordd i'r ffordd i'r ffordd, oherwydd mae'n gwasanaeth i'r ffordd o'r cyfan. Mae'r ddweud hynny'n hynny o'r tyffan o'r cyfan, mae'r cyfan o'r cyfan o'r cyfan o'r cyfan o'r cyfan o'r cyfan o'r cyfan o'r cyfan o'r cyfan. Gwledig am bwysig y ffrOnc erbyn y paradeid yw'r ffordd yn oed. Mae gydig, y pethau maen nhw ychydig iawn yma o'r ffordd y ddau, ac mae ei fod yn hyn yn ei phwntad neud iawn y mae ffordd wedi amlod yn ei wneud. Mae gydig iawn ein wil yn y bwysig iawn, ac mae'r ffordd yn hyd o'r ffordd yn i ffrOnc i'ch bod i'n gwybod bod ddim yn duod. A yw'n gwybod, mae'n gwybod i'ch gwasanaeth a'r ffordd yn mwynhau'r ffordd. dyna o'r sphw Lly Position.. dyna o updates yn summit Dewr'n mewn hyfforddd C Jews? That's how big the plant is. You've got the visual angle. That's a big size. That's not the size necessarily on the screen. It's more precisely. A size that's imprinted on a retina is really side infig. None of this I do before I start. I've thought it's fairly easy. It's complicated and it was really difficult to actualyise all the data. yr ydych chi'n r★gwysig er gwaith ar r★gwysig? Mae hwn wedi bod i gyfodol y ddau cymaint o'r dda. Rhoedd gael bod ni'n cael waith o'r dda radd y trurth, i chi'n rhan oŵr ac i'r ddau cymaint o'r ddau cymaint o r★gwysig i gael. Ond rygwch fair oedd o'r rhai dda ni'n mewn ffairiwyr, mae dwy gydig cymaint o'r wled, oherwydd mae'r ddweud o gydag ymddur... a ddweud y nid, y fideo yw'r ysgol yn y rhaid, a gyd-draeth o ymwneud hyn, ond o'n ddweud ddweud y pethau yn y ddechrau. Yn ddweud o'r ffordd o'r ddweud, mae'n rhaid i chi'n ddylch yn dweud o'r ddweud, yn ddweud o'r ddweud o'r ddweud o'r ddweud. Dyn yn y data, o'r neud o'r ddweud o'r Ddechsel, dw i ddweud o'r ddweud o'r ddweud o'r ddweud, I don't know if any of you have used SPSS, but it's help. But you might use it, I imagine, when you're being taught, but I'm sure they've sorted out the UI since I last used it. But a lot of the time just dump it into Excel and play around with the data. So what I was trying to find here is testing the significance. So basically I want to know if these two groups, the older users and the younger users are statistically different from each other, the choices that they're making. So I've just done a simple t-test at the bottom here that basically told me there's a significant difference between the smallest acceptable size, the older users and the younger users, which is it. Remember see that has a massive impact on the size recommendations that I can make because they have to fit with the older users because their threshold is much higher. So once you've done a statistical t-test or anything like that, all it's telling you is if there's a relationship. It doesn't tell you what that relationship is, so generally you go back to the main data that you've got and I'll pick it. So that's why the graph there shows you what is the difference there. It's that the older users have a much higher smallest size. Each project has a different kind of outcome. My audience for the outcome of this project is actually designers. So what we're trying to achieve with them is giving them parameters of what size of text they can design. At the moment they don't know, they just think yes. So we put the scientific data into this. Things again that impact the size that you can choose. Screen resolution and what resolution you design in and all of these other muddy areas. So that's all gone in as well. They could set average screen size, average distance, and then it would give them the different types of size that they would be able to design in. So I think the other thing from this project was that there wasn't just one answer. There's not just one number. It's so variable within the environment that we were testing and we had to account for that when we were giving them a tool to use. This is another project that I've been involved in. This was basically designing an online survey in order to assess the mood of theme tunes. It was a bit out there. We didn't really know a thing. There was no literature so we were banging on about the extra reviews and yet there was nothing for me to read. So we just asked yes. So this was called musical moves and basically it was about gathering a lot of data. We worked with the British Science Association at Salfordshire University as well and basically it was about promoting this within an inch of its life and just getting as many people to come back and give us smooth data. So that gives you an idea of the kind of how it worked. It played the theme tune to you. It brought up the questions. You were given one of six moves that was random. You were asked to guess the genre which was kind of a measure of did you primarily know what it was? And then we would either ask you if you'd explicitly heard it before or that question was did you like it? Because that's equally as important for us. And also we have to do a lot of press releases. When we do a big experiment we have to do press releases as well. So things like the most liked theme tune, at least liked theme tune. They're really interesting things that we've put up there. Sometimes what I did with this, this is all about again designing the experiment, making sure that the moves that they were given were counterbalanced so making sure everything was very random. So it didn't matter which text artist we came in. We were getting a good range of data and a good range over all the moves as well. So we had to decide things like how many theme tunes are we going to give them? Why don't they only get bored? Each theme tune is about 20 seconds. So in the end we gave them five theme tunes. I wanted to be quick on my surveys. People didn't really do them really unless they were those with people that go find them. Which you don't know their results anyway. So you've got to think of... You've got to keep people interested. Come straight, sit down. I'm reminiscing over the theme tune exit next, next, next and out. You really need to be quicker and to keep them engaged. Otherwise they start clicking the middle button all the time. That's really annoying. How many questions on the page? I wanted to make sure that it was on that one page. I got surveys. It's click next page, click next page, click next page. We don't want to do that. We don't want to be able to remember that theme tune within answering those questions. So we can't have too many questions. We might have forgotten what they've just heard. So it's like things like that that you would think of. I'm wording as well, wording of the questions. Make sure that it's really clear. If you're not sure, ask somebody to read it and see what they understand from it. What does that question mean to you? What are you expected to do? That can actually tell you quite a lot whether you, knowing you're very inside the project, you know what it is, don't necessarily, other people don't necessarily know what you're trying to do. So it's always good to just ask friends and stuff very early on. You can always do UX research. Just ad hoc here and there. Just always look all the way. So for this, we actually got a design agency to do this. So my part in terms of design is just why I created it and just making sure that it was adhering to kind of usability things that we wanted. They kept trying to put fancy buttons in, which I kept fighting with them. I don't want fancy buttons. I just want people to be able to know what they've ticked. So it's kind of this. Sometimes there's this bit of a trade-off between design and usability. Sometimes they don't necessarily fit together quite nicely. You've got to find somewhere in the middle. It may look pretty, but people understand whether that button's on or off. It's quite important. I wasn't involved in the analysis of the data. They did that in a specific sense. So I got that one. This is another thing looking at. Basically, the potential to do kind of field research and in BDC, like I said, we don't just look at consumers. We also look at people in our studios who make the programmes. There's a lot of technology that goes on behind the scenes and R&D are as much more for developing that and making that better. So Production Labs is a section of R&D. Basically, you set up a pseudo-production environment because it's very, very expensive to run a production, especially if we're just testing something. People don't want to do that. They don't want to give us the time, they only give us the money. So we set up our own and then we can bring in new technologies and we can test them iteratively within that environment because if you give a production company a brand new piece of technology and fundamentally that thing doesn't work for them, they're just going to drop it straight away, five minutes in, that's no use to us. We don't want to see any more of what you've got. Leave us alone, because it's every... everything's fast-paced and it's free. You might marry some of that kind of environment. So this was something where we were testing a production assistant sits in the gallery, which is where you put all the screens, watching the production going on. Every shop that is taken, the director is giving them information, that's a good shop, that's not a good shop, that's good from that line, pick up from there, and the whole time the production assistant's making notes on that. So when it goes to the editor, the editor gets the notes through and knows shop three's no good, shop four's great from this point. They know all of that and it speeds up the editing process. But at the moment it's all paper-based. It really are chaotic. They're basically just still writing on notes that they used to do there. They're putting can lines down the script, which is just noting at what point in the script each clip is related to. So in R&D, the guys have been developing a computer-based system that feeds into the computer-based system where actually all the clips go to, but all the clips are in one place. We've also got all the notes in one place as well. Brand new, never been used before. They're quite stuck in their ways, so we set up a production map. And R&D, it's not that pretty, but it's more about what's the kind of functionality that we need, what do people really not get on with, what do people really like perhaps in. Set them here. We have two production assistants, and the other thing is making sure that you're not having ordering effects when you're doing an experiment. So don't be very contestant with paper, and then the computer vary it around, because you never know whether the paper's going to have an impact on what they do on the computer, and vice versa. So you need to think about that as well. This experiment was mainly observation. We didn't want to interrupt them. It was very just watching from the shadows, seeing how they got on, noting anything that they said, and then during breaks we would ask them more formal questions. So, what did you find useful? What did you like? What was really frustrating? Is there any ways in which you would have expected it to work? Things like that. So from this, we put together a recommendation to stop them, which basically says, which is, looks at the design, and looks at the functionality of the system, and basically makes recommendations on how you think that it could be better. You can't go to the other. But they had, basically what we found out, because we put it with a production system, production problem, we always thought that, on paper, when a shot is good, they circle it, and when it's not, they don't. This was interpreted in the system to ticking good or no good. When actually they didn't translate to that, because the uncircled items in the list didn't necessarily mean they were no good. They were the best one. The circle one was the best one. So they were very reluctant to tick no good, because they assumed the editor would need a look at the clip, when there might be something in there that they can use. A lot of the time, especially with fast paced programmes like EastEnders and things, a lot of the time they're just chunking together what they've got. They've got two minutes to the end of the programme. You just have to add extra bits in. Not having watched them, taught them about the process. Just kind of quickly, classic kind of usability testing. This is a lot of the time, when you become a usability researcher and a designer, you're kind of expected to do this very fast pace. We've designed this, does it work? This was one of our red blood services for last degree. These are interactive templates. So, programmes can just put in text and pictures. We just send them a template with all the functionality of how it works, how you navigate through it. So, they designed three, they basically wanted to know they all worked, and they were really usable to find their way through them. So, they've already been designed, everybody, all the decisions they've made, they're given to us, we get participants in. And this is more kind of a task-based analysis, that I'm sure you'll look at later. So, this is a very pretty wireframe. So, I know the journeys that you can go through this prototype when I get people in. And so, once I've got that, I can think of all the issues that we've got. We're not sure, there were things like, we weren't sure whether people knew how many items they were in the list, things like that. How deep could you go in a hierarchy before somebody got lost? Those are the kind of things, we didn't know the answer and we're not going to have a guess, but we can engineer that into the study to kind of look at those things. So, I can go through this and I can find places where I could ask them to go. So, you say, you know, finds Beyonce in the Saturday Highlights or whatever, very task-based system and then you watch how they go about completing that task. We get them to think out loud as well so you can monitor everything that they say. So, again, I've said a bit about counterparts in. So, here are my participants. I know if they can throw a button on the floor, I know if they can use iPads on the floor and therefore I can make sure that I divvy up the templates so we can give them a free to make sure we get a widespread of different ages, different abilities, different genders and things to each template. There's no point giving template 1 to all 16-year-olds and template 2 to all 50-year-olds. That's just not going to give you a nice skew of how people interact and how people expect these things to work. So, like we've said, for a lab-based setting, again, get them to come in. One thing is because you put them in front of the TV, you give them a remote, all of a sudden they think they won't show that you have to spend the first 10, 15 minutes making them feel comfortable in the room, making them feel like we're not testing them. That's really important. Sit back, let them play with it, let them get lost, let them break it. That's fun. Until they get more comfortable with what they're having to do. Then you give them tasks, like I said. Can you find out, say, in Saturday Highlights when you've watched them how they go through it? Well, people in the control room are taking notes on all the things that they get confused about. Then if you see them do something, let them do something, but then you can always say, what did you think that was going to be when you clicked there, where did you think that was going to take you? Stuff like that, so you can measure their expectations as well. So, in terms of analysis of stuff like this, you may have heard lots of processing top-down. But some of the stuff that we do, because we don't really want to affect the data, is the bottom-up process. You're just pulling out anything interesting in the data. The process we use is basically you just pull out anything that's said and you stick it on the board. You've got this mass array, post-it note. You don't really know what to do with it. You start finding things and you start seeing, visually, how big a problem is. If you've got a lot of people saying it's one thing about navigation, you know that that really needs to be fixed straight away, because everybody found that a problem. Then you might have little clusters where you've got two or three things that people have said. That's the other thing about UX. Don't make all the changes. It's very tempting to get a lot of results in and think, right, we need to take all of these boxes. They said ABC, this thing now needs to be ABC. You've got to put it in context. If only three people have said it about something in particular there might be another way around it or it might not actually be a problem with that one thing. You have to really keep it in context whilst you're analysing as well. From this, I would write a report recommendations. Again, this was for designers and developers. You have to make sure that we'll be writing reports on things that your language is through your audience because that's really important. I can slip into scientific speak quite often, but a lot of the time I'm working with project managers you have no idea what I'm talking about. If I want people to take on my recommendations I have to make sure that I'm giving it to them in a manageable, nice way. A couple of things. I went through all of my ABC before I came here and had to look through a really good science talk. I use a lot before I was taught. It's just that there's constraints in the business that don't allow to do things as I wish in money, time and things. I suppose a difference with university needs to pay for participants when you go into industry. You don't have to just use your five friends anyone. You can actually pay for 65 plus to come in and they'll actually know because your friends probably don't know what they want. When you're at uni, what are you going to use? That gives you freedom to choose people that you have to be very careful with your science thing. You have to make sure that the people that you're choosing are taking the exact boxes so I have visually impaired participants. What does that mean? What does severely visually impaired mean compared to moderately visually impaired? I didn't know. I had to go and look it up and then you have to think of not just is this your eye test result. You have to think do you do this? Do you usually do that? Do you use a cane? What visual age do you use? So you have to think about that and you're screening the participants a lot of different questions that you have to ask them in order to get the right type of people to do them. Don't know anything you've heard it yet. You might have heard it. Quick and dirty. That's HCI research. I'm a psychologist so it really pains me that I have to do things very quickly, very dirty because basically you're expected to walk in, you're expected to assess something the next day they want the results, the next day the developer wants to make changes the next day they want to test it again. You don't have time to overanalyze your methodology and think you just want to do the best you can for the time you've got and the money you've got as well. So UX