 Okay, hi everybody. Thanks very much for letting me do this today and thanks very much for coming along and listening. Just to give you a bit of an overview then of what we're going to be talking about today, I'm going to give a bit of a background on the labor force survey. I imagine a few of you know it, but probably some of you don't. Talk a little bit about our design principles that kind of governs the way we work. Give you a few kind of tangible examples of questions that we've redesigned from the labor force survey. And then follow that up with a little bit of quantitative validation as well, but there's not much of that. And then we're going to have time at the end for some questions. So without further ado, to give you a bit background on the labor force survey, it's the UK's largest household survey that we run. It was started in the 70s. And because it's so big, we sample around 200,000 people a year. It meant that every data user has wanted to try and get their questions on the labor force survey so that they can get the low geographic granularity. Well, that does mean, however, is that the labor force survey has become something that's quite different from just the labor force survey. It covers health and qualifications and a raft of other topics as well. That does mean, however, that it's long, really long. If you include all the variables and the derived variables to something like 900, obviously not every respondent would get every single one of those. But when we're talking about moving online, it's obviously impossible to shift and lift the entire thing. So I don't know if you've ever played this game where you've got the folded up bits of paper and you draw one part of a monster and you kind of pass it on to somebody else to kind of complete the next bit and then they pass it on to somebody else. That's kind of how the labor force survey has been put together up until now with odd bits of testing around particular topic areas done, but then those topic areas more or less just slotted into the existing process that we've got. So what I'm trying to do is kind of make one coherent monster is basically the gist of my task. So to put this in context, the reason why we're moving online is because data collection should be online in this day and age. There are public expectations to do it, but there's this wider government drive to move everything online so that it's digital by default when it comes to the public interacting with government. So in the future, we're moving to mixed-mode data collection. We're going to keep our existing loads of data collection on the LFS survey, which is interview face-to-face and interview telephone, but we'll be adding an online collection to it. Some work that went on before I actually joined the O&S was a program called the Electronic Data Collection Program. And I imagine if you're dialing in from another NSI, you might very well be in this space or if you're having conversations with stakeholders and they want you to do their survey and the survey's been running for a long time, you might be in this space. Basically, we had a hard fight ahead of us in order to get this idea across that simply lifting and shifting what we've got in the existing interviewer modes and placing them unchanged online doesn't work. Our questions are far too complicated and convoluted and there's too much jargon in there that's been written from the perspective and the definitions of us as data users and hasn't really taken into consideration kind of the public's understanding of these terms and ideas and these concepts. And when you're moving online and you're losing the interviewer and people are going to have to do this in a self-complete context, all of that makes a big difference and the questions we've got now basically don't work. Which is to say that our interviewers are basically doing a really great job of making sure that we're getting quality data from the respondents for the questions as difficult as they can be for some people. So our approach now is completely different. It's about transformation and not translation. So what that really means is going back to the output concept. What are we trying to measure? Forget about the question as we've already always asked it. What are we actually trying to measure and we'll start from there. In terms of moving online, that means counting the survey content. Like I said earlier, the LFS is far too long to move the whole thing online so we concentrate on a set of priority outputs. We can make changes to the questionnaire flow. Like I said, it kind of looks like a monster that's all higgledy-piggledy at the moment. What you want is a nice, succinct, easy-to-understand journey through the questions and not ask respondents anything that's irrelevant. And then of course making changes to the questions themselves to strip out that jargon, to strip out the definition and all the extra guidance that perhaps isn't needed to simplify and use plain English so that people completing in a self-complete context can actually do so. So we've got a set of design principles which I'm going to be talking you through shortly. But just to say as an overview, what we do is we design for the online space first. More specifically, we design for mobiles. This constrains how much space we've got to work with and make sure that what we're doing is the simplest. It's easy to transform a mobile question to the other modes of data collection than it is to do it the other way around. And we're being respondent focused in the way that we're doing it. We're also being agile. So for those of you that don't know what that means, it's about taking small chunks, working on those, finding out what's wrong, reiterating, retesting them, and then bringing in extra complexity. Even though what we're moving online is a stripped-down of the LFS version of the LFS survey, it's not like we're working of all of that content all at once. Rather, we do much more manageable chunks and then kind of work our complexity up from there. We do focus groups with the interviewers. It's really important actually to get their expertise and the understanding. They'll work with the questions day in, day out. They know where the pain points are. And actually, I can get them to be quite candid about when they go off script in order to get respondents to give accurate data, and that can be quite illuminating. And we do some designing with data as well, which I'll explain what that means in a little bit. But the office is moving to a sort of non-survey administrative data model first with survey supplementing that then. And one quick thing I'd like to say in case there's any uncertainty about this in the room, definitely if you're doing something, building a service or doing a survey that you want to host online and you want general people to access it and they're going to access it any which way that they want, you should allow for that. So this is why we design for the mobiles in the first space. If you follow this graceful degradation path and what you end up with is a cat that doesn't fit on the screen, you can't see the image properly, none of it works. Whereas if you design it for the mobiles screen first, everything's well positioned, well centered, and it looks appropriate. About 17% of households in the UK only have access online via a smartphone. So if you don't make, optimize whatever service or survey you're building for smartphones, then you're going to lose them. If you're expecting them to break off and pick up another device and go and do your survey that way, they won't, you'll probably just lose them. Better instead to design for the mobile, allow people to respond in the way that they want. Test it in that way as well and you end up with a service that's much more accessible for everybody. Okay, there is a UK government cabinet body called the Government Digital Service, GDS, and they've got a set of principles. They don't work with us on this project or anything, but their principles for how you design online services are really useful. All of their stuff is open source. They do a lot of coding that's open source. So even if you're listening in from another country, I would definitely recommend visiting their pages. They've got a lot of good advice there about how to do testing and make sure that what you're building is suitable for users. Okay, so we use these principles. We find them very useful in making sure that we're building something that's going to work for users while making sure that we're meeting the output needs. So I'm going to talk to you a little bit about designing with data. If you're listening in from an NSI and you've got a survey with a history, then this is something you can do if you're an academic or something and you're just starting up a survey, then perhaps this is not applicable, but I'll try and get through it fairly quickly. Hopefully you've all taken a deep, sharp breath because this looks so horrendous and there's a lot going on. Good, it should. It's meant to scare you. This basically is actually a route map for one of our derived variables on the LFS, the most important one, which is about your economic activity. So the circles are kind of outcomes. You can be on a government training scheme. You can be employed, self-employed. UWF stands for an Unpaid Family Worker. You can be economically inactive. That's the dark green. Or you can be unemployed. That is, you're looking for work. So right out of the household section, this is what people are faced with on the LFS. What I'm going to do is talk you through one-quarters worth of data as we get them through these questions. So there's our sample of about 73,000 odd people. And the first thing we ask is, are you doing one of these government training schemes? Now these government training schemes are very few people are on them. Virtually nobody knows what they are. Even the people that are on the government training schemes don't necessarily have the ability to tell you what their name of their scheme is or to recognize that they are on the scheme. Nonetheless, straight out the gate, we ask our full sample, were you on one of these government training schemes, for the sake of capturing about 200 people per quarter. We still got 73,000 odd people in our sample that we've burdened with this question that meant next to nothing. Next we ask them if they're doing any work in the reference week and if they're not doing work in the reference week, is that because they're away from a job? Okay, that's good. We deal with about half our sample there where we capture all of our employees and self-employed people. Still got about 28,000 people. We don't know what they are. The next series of questions we ask them is, are you doing any unpaid work for your own business, which between you and me, I'm not sure if that makes any sense, or if you're doing any unpaid work for a relatives business? And again, this captures, at least in this country's context, only about 200 people that do unpaid family work. Then we have a series of questions about if you're looking for work and if you are looking for work whether you're able to start. If you're looking but you're not able to start, that's one kind of economically inactive. If you're looking and you are able to start, that's the International Labor Organization's definition of an unemployed person. So it's about 2,500 of them per quarter. But as you can see, we've still got 25,000 odd people that we haven't managed to code. These are people that are not looking for work, not working, and not able to start. Basically, you're long-term unemployed. These are your retirees. You're long-term sick and disabled. You'll stay at home parents and your students. People that could have told us that that's exactly their economic situation right out the gate. Instead, what we do is we walk them through this rather horrendous process of asking irrelevant questions about 11 to 13 of them just to code that. When you rationalize that and think about it and just strip it all out, you can end up with something that's a much simpler picture. And in fact, we can be doing things in the background to kind of disqualify people. I'll say what I mean by that in a minute. So just to talk you through the same quarter's worth of data, just ask straight up, are you doing any paid work? That deals with the largest chunk of our sample, which are your employees and your self-employed people. Now, if I were to ask you, did you do any work in the Reference Week? And you said no. I would still have to ask, OK, well, maybe that's because you do have a job, but you're away from it. So even though you weren't working, do you have a job that you're away from? If you had, say, even the day before an interview you do or you've got this survey, had gone to a job interview and been accepted and you're starting in two weeks. In terms of how the public think about themselves, you would say, hey, I've got a job. You'd phone up your mom or your friends and say, hey, I passed my interview. I've got a job. By our definitions, if you haven't started working yet but you're going to be starting in two weeks, we actually count you as an unemployed person. There's about 100 of those people that we can kind of disqualify through that process. But the point is that as a respondent, your experience is never the wiser. You say, I didn't do any paid work, but the reason I didn't is because I'm waiting to start a job that I haven't actually started yet. Fine, we'll put you in our unemployment statistics. In terms of your experience, you've just said that I do have a job and that's a much more comfortable ride. Then let's deal with the largest chunk of the sample. So we'll ask them if they're looking for work and if they're able to start. And if not to either of those, then why not? Then we can code out that 30,000 people that are unemployed and you're economically inactive. Then there's only the 300 people left that are unpaid family workers and are on government training scheme that we kind of give this sort of worst experience to. It's only the minority then that we burden with the least comfortable experience. The majority we've thought about and we've considered and we deal with them first. So these are more GDS principles. In fact, because we've stripped out so much of the complexity and the burden and the rigmarole from that process that I've just showed you, we can actually build more complexity back in. So what we do now in the labor force survey is we're interested in people's economic activity and they're working and they're working hours, stuff like that. If you have two jobs, what we do at the moment is the interviewer says decide which is your main job and which is your second job. I'm going to ask you all the questions about your main job now and just hold all that stuff about your second job for later when we'll ask about it down the road. That's not how people think. As soon as you start asking people about their paid job they're going to be questioning in their head which one are we talking about. So why not bring that behavior and that kind of build something that fits with their mental model then? Oops, sorry. So ask early how many jobs, as soon as you've established how many, that they've got a job, well, how many? And then you can kind of leapfrog, well, what's the status of your main job? Is that full-time, part-time? What's the status of your second job? Is that full-time, part-time? How many hours in your main job? How many hours in your second job? And so on. In terms of a respondent experience that's a lot more natural and it's a lot easier for them to answer. They also then keep in mind which job they're talking about at any one time. So that's kind of a bit of work that we did to sort of improve the respondent experience just by moving the questions around and cutting out the ones that didn't really seem to be doing anything. What about the actual question wording then when it comes to move online? One thing I cannot stress enough, in fact, it's probably the most important thing you take away from this talk today, is to do the testing. Get out there and iterate and iterate again. The best version of a question or a service or content for a page that you're going to write at desk is not good enough. You need to go out there. You need to meet the weird and wonderful people that are your public and you need to find the odd people that really turn what you've designed on its head because you have to understand that they're representative of a much larger group of people actually in your country. So get out of the building. Do the testing. It's incredibly important to do this and not skip this step. If you're an NSI, I would suggest you budget for this quite largely if you're small academic researchers then at least try and do one or two rounds of testing because you're going to find things out that you wouldn't otherwise know. I've highlighted usability and cognitive testing together because we actually do these things at once. So cognitive testing for those don't know is about trying to understand what are the mental processes that people are going through when they're comprehending your question, making judgments about how to answer and then mapping that response given and usability testing is can they use the online tool. Now the thing is these things can't be disentangled. How I interact with the tool definitely affects the way I comprehend the question. If there are response options that are off screen then I don't immediately have access to them. So doing these all at once is the best thing to do. What we do is we and you don't have to be any kind of development expert to do this. We build our surveys on a wireframe tool called XURP, A-X-U-R-E and basically what you can do with that is build it to look like a real survey, like a real website. You can host the online via URL so what we do when we do testing is we usually go to the person's house. We get them to get out whatever device they're comfortable with. We don't force anything on them so if they want to get out their mobile phone and access our survey then that's what we test them on. We let them run through while we just observe and then we do retrospective cognitive probing afterwards. I'm going to talk you through a few examples now. The first ones are kind of more specific to those of you that are interested in labour market analysis but the later ones hopefully will be of interest to people that are doing online survey design more generally. So this is kind of the most important questions about trying to establish whether or not somebody's in our employment statistics or not. Well, it's part of the process anyway. The original question is did you do any paid work in the weekend, Sunday, or whatever, either as an employee or self-employed. The interviewers complain about this question in terms of people hear the word paper work or they quite often say, oh no, I wasn't doing any additional work, just my normal job, which is exactly what we're trying to capture them on so it's not working ideally. The next version of the question that basically I designed and then went out and tested was sort of more aligned with the national account definition of doing work for payment or profit. Unfortunately in the self-complete context we focus people on the payment or the profit. Now we're not trying to capture unpaid work, payment is definitely a part of it or we're not trying to capture either as whether or not you've been paid, just whether or not you were doing the work. So we find basically when you do that people who have just started a job and not yet been paid will say no to this question even though that they were working and people who were working sorry, people who weren't working but did get paid will say yes to the question because the payment was still an ongoing thing. So we moved to a new version that actually tried to talk about a more stable concept, which is that of having a paid job. It doesn't matter whether or not I'm sick in the week, it doesn't matter whether or not I'm on holiday. It's more of a stable, ongoing thing, doesn't matter whether I got, received my salary for this month or not. And it seems to work largely quite well. Job is definitely an easier concept for people. However, we had a few issues with the word business for people who are sole traders. These are people who employ themselves as an employee to their business which they then can control so they can see themselves as either the business or as the employee in which case they're in work if they're an employee even when the business is not doing any work and that's what we're trying to capture. So we moved to this other option about did you have a paid job either as an employee or a self-employed and that seems to be working much better for now, but we're still constantly testing things and we don't subscribe to any of our kind of designs. Everything's up for change. So that seems to be working much better now. But it's not just about the question that you're looking at. It's important to understand that what you're doing also has follow on effects and implications for subsequent pages and questions. So it's definitely the case whenever you're designing either a survey or a service that you're looking at your pages holistically that you don't break down the work and give it to people so that they're only looking at small windows in isolation you really want to consider the whole thing. So when we're using the question about did you do any work for payment or profit, you've obviously got to have this follow up question to capture whether or not people do have jobs, but they were just away from their work in that week. There are problems with that process so did you have a paid job or business that you're away from in that week? We had a teacher during the summer holidays so obviously hadn't been in work they said no to were you in work the first question which was correct but then they, at this question, were you away from a job or business that you've been paid from? They also said no because they're still being paid through the summer months and so in their mind they weren't away from the paid aspect of the job. This in no way was a question that I would expect to go wrong. It seems fairly straightforward to me so this really, again, I can't impress upon you enough how important it is to go and get the testing done and find out what does and doesn't work. I mean she's a teacher relatively well educated person can still go really wrong. When we switched to the idea of asking about a paid job as a concept then we found that there were sort of follow up effects and issues on a different question. So as I mentioned to you earlier we try and ask early do you have more than one job so that we can root people through a better experience through the survey. Tried to keep the tone casual tried to keep it in plain English very light. Did you have more than one paid job or business in that week and actually the casual tone made it too easy to misinterpret this. So I spoke to somebody he had heavy dyslexia very low digital literacy skills and he was a self-employed person and essentially he ran a bike shop he fixed bikes and so what he was his answer to this question was well every job that is every bike I fix I get paid for as individual so that's lots of jobs and I have a business so what do I press both. The simple to answer for this question for me would be yes. Two pages later he's getting asked what is the status of your second job are you employed or self-employed is that full-time part-time and instantly he knew he had gone wrong. What second job I'm confused where did I say I had a second job even though I've just quoted to you in the light above that's exactly where he said he had two jobs. So what that highlighted to me was that there's something special about the word second that he saw that made him know that he was he had gone wrong. There's something separate and secondary about this extra business whereas if you're just trying to be casual about did you have more them and paid job or business then really he's that's a lot more open to interpretation. So our next iteration of this question will probably use the word second or separate in the question just to make sure that people understand that it's about separate work streams but again this is not a question that we found was going wrong in multiple rounds of testing. It's important actually we used to do our testing and our recruitment ourselves via gumtree and after a while we were saturated and we kept seeing the same people we since started using a recruitment company and if you've got the budget for it it's well worth it because they can find people in the public that you might have trouble recruiting yourself and if I hadn't spoken to this guy I probably wouldn't have thought there was any problem with this question but actually there is. So here maybe are some examples that if you're not interested in labor market analysis you still might be able to take something away because it's really more about the process of getting it right. On the labor full survey we're interested in capturing people's usual hours here is a pattern that I designed thinking oh well you know what some people do shift work and then maybe they don't work the same hours every day maybe maybe if I was to ask them how many hours they work in a total week they've got to do all this mental arithmetic to work it out perhaps I can build something that when they put the hours in for each day it'll calculate the total at the bottom. Absolutely nobody thanked me for this I was trying to be helpful and to break the question down to make it easier steps for some people but even if you're a shift worker you've recently completed a time sheet you know what you're going to put in for your hours for the recent week. So let me impress again you are not your user go out and do the testing what you design at desk and you think is brilliant or what your stakeholders or data users think is brilliant is probably not actually ideal from what a respondent needs in a self-complete context so do the testing. Don't design at desk. Following on from that so the labor full survey they want to capture usual hours and the actual hours that you worked in a specific reference week this is about tying it in to see whether or not people are over or under working or to explain why hours are down based on holiday that kind of thing. So as I mentioned to you earlier we kind of went under this pattern of breaking things down to here let me ask that question about your main job then let me ask it again for your second job what we were finding was that if you separate the usual and actual hours questions people don't differentiate between them some of them think that it's the exact same question again even if they don't they tend to be lazy and just input their usual or their interacted hours again the only way we could find to actually get quality data for this question was to put the two on the same page where you're asking about what is the usual number of hours you work in job one or you know your main job and then in the week specifically this week how many hours did you actually work and this is the only way we've managed to get people to actually give us discerning actual hours times for a week so context effects are important how you place the questions next to each other affect their comprehension and how carefully people read them so this kind of you know pulls on things like crisis maxims people won't give you if you are if you ask somebody a question again they're going to assume they're after something qualitatively different but that only seems to work when you make the context really obvious by putting them on the same page be consistent but not uniform so this is about we've got all these principles that I've been telling you about this generally about trying to make it easier trying to simplify things trying to reduce the number of words that you've got on a page however sometimes you need to abandon your own principles particularly if you're not just building a service but do care about statistical robustness so for the labour force survey somebody is unemployed if they've been looking for work for in the last four weeks and if they're able to start in the next two following my own principles of trying to simplify and keep the term casual my first iteration of this question was did you look for paid work at any time between whatever the dates what we were finding was that this is the first question you get when the reference week has previously been talking about the week before the survey now stretches to a four week period what we were finding when we wrote the question this way was that nobody was noticing that people were rushing through it just far too quickly and they weren't taking heed of the dates so what we had to do was actually make something that's clunkier and much heavier much more heavy handed so we put the dates at the front of the question it slows people down it's not as nice an experience but what it does get people to do is actually pay attention to the dates so that the quality of the data that you're collecting is much better the same goes for the question about are you able to start in two weeks so this first iteration if a job had been available in the week whatever would you have been able to start in two weeks we then move to a question that's if a job had been offered on a specific date would you have been able to start before a specific date nobody's confused everybody knows when the job was supposed to be offered when they're supposed to start by everybody can then provide a quality and accurate answer even though we're heavy handed with the dates on those questions to get people to slow down and consider them properly it's not like you've got to consistently do that all the way through so for example when you say no I haven't looked for work in the four weeks the next question is this one and I can simply refer to this period or that week or whatever it might be that's appropriate and at least one page along people can follow that if you're not familiar there's some great articles by the Nielsen Norman group if you do a quick google search for Norman and F shaped reading or internet reading they've done a lot of great work about the way people digest web pages they call it this F shaped reading pattern which is you read along the header probably not even all the way across before you start scanning down the page and looking for stuff that's relevant if people going to engage in those behaviors then work with it rather than against it so we can actually if people are going to jump from the question stem straight to the response options then fine let them use the response options to actually finish answering the question if you take the example of that first top left question otherwise I'd have to duplicate are you an employee or self-employed and then options an employee self-employed why why replicate why would be repetitious this saves you the effort another nice thing you can do online is even though do you ever do work which would you consider as overtime is technically a yes no question we sort of skip the step and allow them to say no which I do not work overtime or specify what kind of overtime they do and that seems to be working well even though it's kind of two tasks all at once okay I promise I would talk a little bit about the quantitative data stuff so just move on to that now and then we'll have some times for questions so we've run three quantitative tests so far I have to say that the main purpose of these tests to date is really looking at engagement and take up how can we get people online incentive strategies and that kind of thing we have in October a contest planned which is going to be our first mix mode test so that's going to be the first proper evaluation of how well these new question designs are working for outputting labor market data so please do get in touch later later in the year if you're interested in that so just to explain then the first three tests reasonably decent sample sizes as I say it was mostly about looking at the materials that we've designed in order to encourage people to come on that's a entirely separate talk if you're interested in that stuff please do email me and I'll put you in touch with people in my team that can better answer these questions but in so far as what we have gathered in terms of validating the questionnaire that we've done we had a rather heavy handed you know quite an explicit how easy or difficult did you find the survey at the end as you can see from the responses in the table there below generally people found what we had put together very easy completion times were short usually no longer than 20 minutes in the longest version that we ran and when we look at the pattern of kind of complete households so of the people that accessed our survey, 86 and 85% of people actually went on and completed it there was a small percentage of households that partially completed very few of them actually only accessed the survey and then didn't do anything I think a few households were put off by the household groups it's probably at that point that they realized if I've got to put in all seven of us then this is going to take a while maybe I won't bother and very few partial response and we're not quite sure if that's to do with passing the buck to the next household member and then you know it kind of falling out at that point but what we do know as well is that of the people that went and completed our survey in test one it was 98% and then test two it was 96% of people did it in one sitting so obviously it's not so difficult or horrible experience that you know you need to take a break maybe need a short holiday or something most people tend to do it all at once and so that's quite encouraging so brief summary then if you've got the time and the budget and the appetite from the people that own the questionnaire if we're talking about redesigning a hysteric one then try to convince them that what they have in their mind about the existing questions as they've always asked abandoned you'll go back to the concept that you're trying to measure but you might have to reword the question in line with the way that the public or your respondents think about it particularly if you're designing for a self-complete context so then design for the respondent don't design only for desktops and laptops as I mentioned earlier some people only can access the internet via their smartphones if you don't design from them they're not going to switch mode you're just going to lose them not only that but there's something probably quite specific about that population that only have access from smartphones maybe their low socioeconomic status for instance and they're probably people that are hard to count anyway so you definitely don't want to be losing any of those if you do design for the respondent then what we found is that you can get much better and much more accurate correct self-complete data whereas if you don't think about them when you're designing your questions or you just think about the data user and their jargon you're going to lose that one thing actually I didn't mention in any of my other slides but it might be pertinent and somebody might have a question about it I don't know if you noticed but we don't build guidance all of our questions basically don't have any help text we try and write the question to be as self-explanatory and as simple as possible if people don't get it then we bring in extra complexity test again if people still don't get it we can make it more complicated again or maybe it's about stripping it down into several questions that's more comfortable but what we don't do is put a ream of definitions and hidden guidance behind a click to expand type button people don't even if they're unsure people won't read guidance so don't expect them to don't build it in a way that you expect them to and if you're building for a voluntary survey context then at least being a little considerate of the people that are giving up their time in order to give you the data I think is well just a nice thing to do as well as giving you better quality data be holistic questions don't work in isolation their flow through the survey doesn't work in isolation from the questions the suitability of the response options also impacts how you word the questions so what I'm saying is think of this altogether got to think how is this question what is it following on from where is it going next how does that fit with the respondents mental model of their expectations through the journey how does the question relate to the answer options are people going to be jumping about is there something I can work with there rather than work against and this is also why we do the cognitive and the usability testing how you interact with the tool itself affects your comprehension of what you're doing so consider all of these aspects at once if you've got a team of researchers looking to modernize either a service or a survey don't give them too small a chunk of the work because then they'll be blinkered and they won't see how it impacts on the things that go around that rather combine it as one big team sure they can work on it all together but they need to look at the bigger picture and then do the testing even if you're just an academic PhD student listening to me and you're wondering oh what is what is there is a takeaway do at least one round of testing build your survey or your service then go out even if you just find sort of 10-15 people ask do a little read one of Steve Krug's books what I'd recommend for the usability testing don't make me think is really good it's very short very simple to follow get some idea about cognitive research principles and then do at least one round because if you don't talk to the the public will always surprise you basically is what I'm saying there are always in the most endearing way those odd people out there that you need to take account of and build for them it's not going to make it any harder for the majority of people that wouldn't otherwise have had a problem and you need to make sure that you're measuring what you think you're measuring basically so that's me I'm going to pass it back over to Sarah and then I think she's going to field me some of your questions if you have any thanks okay perfect thank you Alex okay then so now we have a question from Terry I hope I said your name right and the question is to what extent is the shift to online data collection meant to supplement rather than replace the current approach to data collection I might we expect to sample boost from online collection while this is being trialed that's that's a good question so to answer in stages firstly we're not changing the existing modes of data collection that we've got will always be using online in conjunction with the other one sorry can you hear me so there are people that you're not going to encourage to do online or encourage to do a survey at all until you start knocking on their door and then there are other people that would much rather do an interview over the phone or face-to-face in terms of then how you actually the sort of more operational issues about how you manage your field for allocation we're still working some of that out I think perhaps the role of our field interviewers might change a little bit in the future interestingly from our initial contest what we're finding is that kind of contrary to what we're expecting the sample composition that we're getting online isn't actually much different than what you get in the traditional collection mode so it's not like introducing online only leaves like the worst and hardest to count cases left for our interviewers not any more than that's already happening anyway I'm not sure if I fully answered that question okay thank you I have a question actually I don't know if you answered it already but at the moment this is testing I gather what's the time frame for trying to include this actually in the other other surveys so as I mentioned in October we've got our third contest that'll be our first mixed mode test where we're actually looking at the data where the data matters so we're going to be able to from that have some idea of what the introduction of the new online mode is doing to the estimates if anything or whether that's just can be explained by sample compositions and then later down the line before this actually goes live what needs to happen is what we call a parallel run so what that means is that the labor force survey would carry on whilst a comparable sample was then drawn up for the transformed labor force survey you run them for the same period whether that's three months or 12 months I think it's got to be discussed depending on how expensive that's going to be and when you're running it over the same period with a comparable sample then you get an idea of what the changes are doing to the estimates and then once we can explain how much the estimates are going to change once something like this goes live then we can do it so are you thinking next year or the year after I mean we're thinking probably more 2021 at the earliest okay so we have a few other questions from Lucy is there ever a good way to design a questionnaire for multiple people to complete such as different people oops sorry it's just moved sorry sorry sorry it's lost that question there we go sorry is there ever a good way to design a questionnaire for multiple people to complete such as different people within one business so if we're talking about business surveys then yeah I would actually probably the best thing you could do is do precisely figure out what those user needs are if you're sending your surveys out to businesses and you know that they're not completed by one person like maybe somebody fields the email in the first place then collects all the people from the organization that she knows is going to need to do the survey like maybe there's the accountant needs to put some financial stuff in someone from HR needs to put employee numbers in something like that if you know that basically go along if you can to their business I know they're busy so they're not always willing to let you in find out exactly what their process would be for going out and filling out the questionnaire and then design that process so it's exactly easy for them to do that that would be my suggestion but absolutely try and design it so that it does work for all of them rather than one size fits all unless that works go and find the user needs from the horse's mouth is what I would suggest if you can okay thank you so Sandra asked you mentioned that you do not include guidelines when asking your questions why people don't read them even if even if even when we've seen that they answer the question incorrectly even when they're going wrong even when they think that they're going to go wrong they still don't read it the other aspect is that we want to move away so the current survey I mentioned is like 900 variables long sitting alongside that is a thousand page document of interviewer notes explaining what all the questions mean all the definitions you know what to do with edge cases and stuff like that we want to move away from that design your questions to be as simple as possible and keep simplifying them until they work for everybody if you design the question right you don't need the guidance if it's a complicated thing try splitting it out into several questions but if you're relying on people to open guidance in order to get the question right particularly if it's hidden guidance then forget it they don't they just don't do it I mean people don't even read the please select all that apply that tells you that whether or not you can pick one option or multiple options people don't even read that very very rarely so yeah anything other than the question and then jumping straight to the answers they ignore so you've got to play into their reading behaviors online great alright next question from Jerry great work on going back to the concept redesigning questions no hidden definitions etc for many of us this seems to be out of our control any advice on how to educate our clients particularly when we did not have the budgets that you have for the LFS transformation so for example with hidden definitions hey Jerry not it's yeah I mean as you know Laura and I had those fights a long time ago trying to get people to accept that you can't just use your existing questions and expect the same data in a self-complete context maybe you could ask them to budget for a pilot study or something allow you to collect some initial evidence that you can immediately go to and say told you so maybe we should be you know taking a bit more of a back to the drawing board kind of stab at this but I'm not exactly sure because I can appreciate you've got to sort of do it on the cheap and do it quite quickly because it's almost before any of the project gets going or if you have any like leftover money from previous projects maybe you can save yourself a bank of this evidence just to whip out for these for this kind of occasion I'm not sure okay so from Steven will move online eventually enable the ONS to boost the LFS sample or will it remain broadly the same it's that's a good question and it's not my work to sort it out and I think there's a little bit of an open question there because we're also possibly thinking about an integrated social statistics model which might actually include increasing the sample by having a kind of master wave that's almost like a one percent population coverage survey that then has surveys that follow on from that which you get sub-sampled into so you can get sampled into the waves two to five of the LFS or you could get sampled into I don't know a household financial survey so that's one of the things that we're thinking about and planning but I don't think any of those plans have been firmed up yet okay right then so from John will the things that you're learning through cognitive testing feed back into design of the face-to-face or phone questions it seems that you'll end up with a twin track questionnaire here with the web questionnaire potentially being optimized to a greater extent than the face-to-face phone questionnaire how are you dealing with that that's a really great question so we've actually been engaging the field and the telephone interviewing staff with us the whole way through this process I mean they're ultimately an end user of the survey in terms of the questions that they've seen that have designed for the labor market stuff they're actually like really eager to take them on because I've managed to remove a lot of the kind of more awkward aspects and the more irrelevant questions through the flow so they have to ask people fewer questions they like that but I'm definitely taking on board their considerations and their insights when I'm building the question however we are designing for the online mode first and we've actually recently started then redeveloping for the for the face-to-face mode so we've got the online questions as a starting point but we're not beholden to keeping them like that in the same way that we're not beholden now to keeping the existing questions where we're redesigning for online if we need a slightly different process for face-to-face because it works for the interviewers then we're perfectly comfortable with doing that as long as we do a certain amount of testing to make sure that conceptually they're going to be measuring the same thing in terms of the output and the definitions of those outputs they're going to be the same thing so we are talking with our interviewers and taking their thoughts on board and basically we'll build them something specific for them that might be different from how it looks for the online respondents so it's about improving their way of working as well as just in doing it for the public okay then so from Jennifer when you send pre-survey mail-outs do you include an opt-out if you do does this reduce how far the mail-out increases response rates no we don't so we sample based on addresses so we do a randomised or maybe it's a randomised sample of addresses drawn from the royal mail mail's postal address file sorry could you repeat the question Sarah when you send pre-survey mail-outs do you include an opt-out right and the survey itself is voluntary so basically your opt-out is to just not do anything we don't otherwise you're not mandatory involved whether and you have to opt out to get out so if we don't hear from you what we do is a pre-notification soon you've been sampled soon we're going to be sending you an online survey link then we send them the online survey links they go online and do it then we send them a reminder and then we send an interviewer to follow up to try and chase them up so if they don't want to respond at any of those stops they just have to not do anything okay so from Fay if you can only afford or have time for very limited testing is it not worth following with an online survey is there anywhere I can find a body of questions that work e.g. employment status if you if you need a really quick fix then maybe but probably not it might be specific to your situation your survey and your respondents in terms of the labor market questions that I'm developing it'll be a little while I guess before you can pick them up we're not absolutely done with the testing and we keep reiterating so I wouldn't want to say anything is in its final form just yet although we're pretty comfortable with where they are but what I would do is if you can just do even just get 10 people do a brief cognitive interview with them yourself about you know what do you understand this question to be asking you how would you explain this term in your own words that'll make sure that what you're writing in the question stem people understand in the way that you think that they've got it if it's just pure economic stuff then maybe send me an email after and I can give you a hand and you can take those questions as a starting point okay then so we've got another question from Faye which is do you have an aim for the number of questions you wish to reduce the LFS by? no and there's no hard and fast rule so as part of kind of pushing back against stakeholders so that they didn't try and put everything on the survey and turn it back into a really long thing we had this sort of a little bit arbitrary rule if we didn't want to make the survey longer than 20 or 30 minutes fact of the matter is though when you do this kind of work and you've put the time in in order to make something that's respondent centric that is easy to get through easy to do we're starting to maybe find out again that you can probably if you do this make surveys a little bit longer because they're not so horrible to get through so there's no fixed survey length it definitely depends on how interesting the topic is how motivated your sample is to do it that kind of thing but you could you know if you've got the budget keep testing and when you start seeing the massive drop-off then considering you know that's the point to stop okay so we have a couple of questions from Karen and Oliver so yeah how do you intend to cut down the amount of questions do you split up into more waves how do you cut down the amount yeah right so a lot of the questions because the survey has been going for a long time I and if you've got the budget to do this it's quite resource intensive but I spend a lot of time basically understanding where everything there came from which isn't to say that I always found out what the answer was because there are some of those paper trails are kind of rabbit holes with no end in that you couldn't find out who the data user was for the question why it ended up there in the first place and kind of what it's still doing there now basically what we've done is got labor market division here at ONS so that the primary users of the data to prioritize what it is that they output from the labor force survey and we're designing their priority outputs first if you can meet the same output in fewer questions then we've kind of got the remit to do that so I don't know if you remember back at the beginning I mentioned this idea about having this response option for why were you away from a job to say because I was waiting to start a job that I've already taken now that as it exists now is actually its own question are you waiting to start a new job it doesn't need to be it can be a response option to that question so without losing any data points without losing any outputs I have dropped a question that way so you can be a little bit clever about it if you think about the journey and the flow whether or not you can combine questions into response options on other questions if you're a little bit smarter about it but other than that get rid of anything that's trimming and prioritize the outputs build in the ones that are the highest priority and stick in the ones that are lower priority if you have room okay there's another question from Karen again by Oliver you said we should apply more sorry you should apply we should more apply the mental models of respondents but those are different from our definitions ILO concepts etc so we will lose comparability of data comparability of data so it's not that I've completely thrown out the output needs definitely what I'm designing is meant to meet the output needs as well it's just doing it in a way that fits with the way that the respondent thinks about it so they're mental models about talking about two jobs all at once rather than one first and one second or you know being able to oh this is a survey about economic activity so I want to tell you that I'm retired straight away I don't want to have to wait that kind of thing actually I've not found that it does impact too much on the actual definitional issues and so this is kind of why it's important to go back to the concept even though you can abandon the old question as long as you're measuring the same concept the comparability should be maintained okay and we have one last question again also from Karen and Oliver which is the transformation process takes long how long will it last what is the time schedule hmm yeah so we've got the contest in October we're trying to get ready for that I'm not exact I think 2019 is where the parallel run is planned for and then the parallel run itself if we decide to run it for 12 months takes 12 months so this this is why I'm thinking probably tentatively 2021 before there's a live transformed labour market version that replaces the existing lab before survey okay we've had about according to me about 2 minutes left we've got a last minute question from DAG I think did you try the hours worked grid for actual work towers as well am I still sharing my screen yes you are yeah so what we tried was was this so we used to have it what was your usual hours kind of like this on its own on a page what was your actual hours on its own on a page if you had two jobs however it was doubled up so it was main job second job on the same page but the only way we could get it to work was to have usual and actual hours contextualized next to each other so when it came to have sort of the usual hours version of this page where it was what's in your main job what's your usual hours people don't have a problem with usual hours they're just default to their contracted usually and when you ask them what does usual mean to you that's what they say about my contracted or typical hours but it's about getting them to think about did I work more or less than this week that's the bit where people tend to be a little bit lazy except where you really heavily handily juxtapose it one from the other okay I have a last second question I think you may have answered this already it's Steven will the LFS eventually completely online or is the plan to run online alongside bone and face to face yep so the plan will always be to keep the mix mode model so I don't think there's even like plans about shrinking field force or interviewer capacity or anything like that so I don't think it's going to affect kind of what we do in the field space it'll just provide us this extra opportunity in this extra mode of data collection okay great well thank you very much then I think we're out of time so if anyone has any further questions they can email you that's right is it yes please do yeah please do be happy to take an emails okay fantastic well thank you very much that was very very interesting and the session was recorded so assuming the recording has worked we'll be putting that and a PDF of the slides up on the UK data service website and this will be in the news events and if you just follow that through to events and past events you should find the slides and recording up hopefully in the next day or two okay so that's fantastic thank you all for coming and thanks again Alex for a fantastic presentation okay bye bye thanks everybody bye