 Hello and good afternoon everyone and welcome to this session introducing and the transformed labour force survey. My name is Jen and I'm from the UK data service user support and training team and so to get started. I'm really pleased to open this session on the transformed labour force survey. The LFS is a really important survey and it's fantastic that the ONS have joined us today to talk through some of the changes that are happening with the survey during the next year. So today's speakers are all of Fraser and she's head of the statistical transformation for social surveys at the Office for National Statistics and has been responsible for the design of the transformed labour force survey. And we'll also have James Harris, so he's head of the labour force survey at the Office for National Statistics and has been responsible for running the current labour force survey and annual population survey and leading on engagements with users around the transformation. So as Jen introduced so hello James Harris head of the current labour force survey so working with Martina to get out the numbers that you're all currently using and working in partnership with all designing the new thing you're going to be getting. So we are now in our 50th year so collecting the labour force survey for 50 years on the labour market demographics various aspects of people's everyday lives. But right now we're having to think about the digital age, although the survey has been evolving over many years, big massive change to the structure of the population, the way in which surveys are conducted, the new technologies that we have available. So this transformation is bringing us very much fully into the digital age and in our 50th year as well. Thinking about all the various uses of the data so some of you have given some very good answers but a whole bunch of publications and statistics and outputs make use of our information all across government. You have various outputs by the ONS itself, things like the beyond GDP measures, breakdowns of families and households, analysis of disability, analysis of adult smoking habits, analysis of aging, all sorts of publications that ONS produces on a regular basis based upon the LFS data. We obviously make our information available through the secure research service, the UK data service, known as all sorts of mechanisms for yourselves, other users, academics, think tanks, local authorities, whomever, to get access to the data should they want to. And part of that analysis includes feeding into things like the health and safety executive, the institution for occupational safety and health, all sorts of analyses of various topics like health and safety at work, like technology with tech UK, like the sustainable development goals, analysis of social mobility, analyses by the race disparity unit, which is in the cabinet office, looking at potential inequalities in both the population and the labour market, looking at different characteristics of people, breakdowns by age and sex and location and everything else to investigate the current status of the economy. And of course, an awful lot of analyses by local authorities, local users, especially in the light of the levelling up agenda. So breakdowns of the analysis by local authority, by age, by sex, by ethnicity, by employment status, analyses of specific disability groups, health groups and everything else, feeding into things like the local industrial strategies and the local policies and local deliveries in different policy areas. So an awful lot of use of this data, and that's hopefully why you're all here today, because you do some of these uses of the data and then you're engaged in some of this analysis as well. But what this means for the survey itself, well, we've been continually adjusting the design of the survey over the course of many years for the changing population and the environment, but we haven't done a full branch, a root branch review of the questionnaire, the structure, the flow of the content, everything from the bottom up. We have unfortunately seen a steady decline in the response rates over the last 25 years or possibly longer for a whole variety of reasons. Concerns about the confidentiality of how we handle the data, a growing sense of mistrust with the government, changes to the structure of the population. So an increase in gated communities, an increase in both single person and temporary households, and of course single person temporary households as well. A bunch of different changes to the population and our target audience for conducting this survey. And of course some industry-wide complexities, all sorts of things affecting the survey industry. So difficulty in recruiting interviewers, public fatigue with taking surveys and calls. The global move to online interactions and the introduction of smartphones and tablets, which change the way in which people interact with surveys going forwards. And I've put on here as well, public fatigue, you go to a shop, get a receipt, you go to a shop, you get a review of how well did we serve you today. You go to a restaurant, how well did we serve you today? There are only so many surveys that people can handle. So unfortunately we have seen a drop in the number of interviews, but that's largely why we're going into this transformational process. So thinking about the transformation where we're going, so the objectives of us transforming the survey, we are taking an online first approach. So rather than the old survey used to be face-to-face first, now it's online first with a new adaptive and responsive design. So that we are targeting our activities, our resources, our fieldwork capacity and everything else in the most efficient way to maximise the number of responses we get. But not just the number of responses we get, the representativeness of those responses, the quality of the responses that we're getting, the overall improvements to the survey. Which include a larger overall sample size, so the Wave 1 of the transformed survey is far larger than the current Wave 1 of the current LFS. I think it's seven times the size of Wave 1. More robust processing systems, so making sure that we're able to produce the data as accurately, consistently, error-free as possible and ideally as quickly and easily as possible as well. Adapting the design to reduce the bias, so introducing options, alternatives, changes to the survey design so that we're reducing the bias and improving the representativeness. Making the survey hopefully more flexible and adaptable to respond to change. Things like when COVID get to impact the population, making sure that we're able to make immediate changes as much as possible so that we can then collect the best information for the latest economic, environmental and policy outcomes. And updating and upgrading the questions, as I said, this has been going for 50 years, so making sure that we're asking the most relevant questions that we possibly can in the best possible way. And of course we do have future aims, so a bunch of those changes have already happened. Some of them are being implemented over the coming few months as well. But we're planning to add administrative data, trying to increase the timeliness, potentially adding new requirements as we found out through our engagements with users, with other government departments, with policy makers, trying to improve the survey with all of these different resources at our disposal. Thinking of the way that we're designing the survey, so we are trying to put quality first. So make sure that we're not just producing a new survey, but focusing on continuous improvement across a whole range of quality measures, both the statistical and processing approach and the production outputting and analytical approach as well. The whole suite of activities going on all across ONS and trying to make this respondent centric as well. So improving the experience for the person answering the survey, the person typing in their responses, tapping the boxes, filling in the online questionnaire. And by improving that respondent experience, it then improves the quality of the data that we get and hopefully increases the inclusivity, whether there were any language barriers or comprehension barriers, whatever it might be. Trying to make it as easy as possible and as inclusive as possible for people to actually answer the survey. And trying to make the design of the survey as responsive and data driven as possible. So all the information we're getting back from the respondents, from the way in which we're collecting the data, all the management information, making sure that we're integrating that continuously improving, changing our approach so that we are targeting the resources in the right way to the people who need them the most and then hopefully driving up both the quality and the efficiency of the survey. Things like whether you live in an area of deprivation, whether you have particular language skills or disabilities or anything that you're facing, making sure that we are giving you the best possible way of answering the survey. So where the transformation has been going over the last few years. So this all started in 2016. Then in 2017 we ran our first couple of tests. So tests one and two testing whether it could work online, what the response rates might look like online, what the engagement strategy should be at that time. And then in 2018 running yet another test trying it mixed mode. So both online and face to face, analysing the effectiveness of that, analysing the different outcomes, analysing the difference in modes, whether it made a change to the responses that we were getting. And then heading into 2019 and 2020, the final tests leading up to having an online survey active, especially since as a response to the pandemic when the current LFS was struggling because it was based face to face first introducing the online mode as a supplementary measure as an alternative measure of collecting the data and information so that we were collecting as much as we could in the best possible way. And that 2020 introduction of the survey that is still live. It is still running. Obviously things have been changing over the course of time, but back in March and April 2020 was when technically the TLFS first went live to the public and has been going ever since. But more recently since 2022. So back in February 2022 we introduced telephone mode as well. And then in April we reached the target sample size so going out to all the people that we were expecting to go out to giving them the online option and the telephone option and then over the coming few months adding more content into the survey so all the key labour market content was included from September 2022. And then in November and April and since then implementing and expanding our field work capacity. So a process called Noctenage, which all of will come on to, but going out to people's houses and making sure that we're getting responses from them through all the different modes that we have available to us. Leading towards July when we've been conducting peer review processes, so presenting the data as it currently is to a bunch of reviewers, both in departments in devolved administrations and a few economic experts as well. And the reason this is coloured orange rather than green, this is still ongoing. So this peer reviewing has been going on since July, analysing investigating the data, finding any problems or issues or any questions that people have so that we are refining and improving and correcting any problems. That we find in the survey. Leading towards where we are now in November. So we've implemented a couple of final changes, one change in October, one change coming up in a couple of weeks time. These will be the final substantive changes to the design and the content of the survey going forward. There will obviously be an ongoing process 2024, 25, 26. We will continue with improvements and updates going forwards, but for this iteration of the survey for this current change for the TLFS right now, these will be the final changes that we're making at this stage. Leading towards January when the aim is that we decommission the current LFS because the transformed survey has met the quality criteria that we've set for it. It can then become the primary source of information for labour market statistics and of course given all the breadth of other questions and variables in the survey, the primary source of information for a bunch of those as well. Leading towards in March and in May, the first key publications based upon this new TLFS data. So the labour market statistics produced in March and then in May based upon TLFS data and the first release of formal TLFS micro data currently aiming for May for that to happen. So in May that will be the January 2024 to March 2024 quarter of TLFS data when it is the official source of statistics from that point. At which stage you've heard plenty enough from me. So handing over to Ola, Ola, over to you. Thank you, James. So I'm going to take you through a little bit more about some of the detail of the design and the development process we've been through over the last few years and let you know a little bit more about what the new survey really looks like and what that means for you. On to the next slide. Thank you. So this is a little bit about the design of the survey. So as James mentioned, this is quite a different survey from the LFS. It's an online first survey and that means it comes with multiple other changes as well. We've got a much larger household sample size rather. So we've got 140,000 households that were invited to take part on the Transform Play before Survey at Wave 1 each quarter. And then from there it changes again a little bit where we have half of that sample receives a shorter questionnaire which contains all the core labour market content and socio demographic content, all the education questions, etc. But then we've also got half the population, half of the sample rather, that received this slightly extended questionnaire. So our TLFS plus questionnaire, which has a few more additional questions which enables us to ask a few more questions of those people at that point in time. One of the other differences with the current labour force survey is that we then don't rotate all of that entirely very large sample through to Wave 2, 3, 4 and 5. We choose to take just a portion of those. So it's 40,000 of the original 140,000 that are then rotated into our Wave 2 sample. And that has then carried forward that full sample all the way through Wave 2, 3, 4 and 5 in a similar way to the current labour force survey. But that is the way in which we've got this much larger Wave 1 sample enables us to look at more things to be able to provide that more detailed data to enable us to be able to provide more data to support essentially the more detailed analysis that you might like to conduct. Can we go to the next slide please? Thank you. So to go into a bit more detail about the sample design itself. So rather than using half the current frame for the current labour force survey, the transformed survey uses a new sample frame. So that's the dress based premium, which is based on a series of data including an ordinary survey, geoplace product complied of kind of local authority data. It includes council tax data, Royal Mail data and other things as well. And this is updated very regularly. So it enables us to get the most up to date data that we can on the quality of those addresses as well. This includes private households only, no communal establishments. And this is a place where actually the address days data is quite effective enabling us to be able to identify the different types of addresses in the sample. So enable us to be more accurate at excluding those communal establishments at the beginning of the survey and therefore reducing the number of ineligible addresses that we get across the survey as well. It's a systematic random sample within England, Wales and Scotland. So all households in the frame have an equal probability of being selected and that's within each of those countries. So within Wales and Scotland we have a boosted sample and that means that they have a larger relatively greater kind of sample overall. But within those countries there's still an equal probability of being being sampled relative to the population size. So rather than the LFS we've got different local authority boosts at different areas. On the transformed survey you've got essentially an equal kind of representative sample, at least within England, Scotland and Wales. And that sample is also representative for every single week. So we issue samples on a weekly basis and each of those weeks is representative across the geographic areas as well. And those we can determine essentially the size of those samples, the relative size across each of the regions and the areas according to the media population estimates for each country and within the English regions as well. Now I've talked about England, Wales and Scotland. Nisra are actually conducting their own transformation of the survey in Northern Ireland. So that's done entirely separately. So what I'm going to focus on here is really what we're doing to England, Wales and Scotland, although we will be producing UK level estimates with the Nisra data as well. They're conducting their own transformation on a slightly different timeline. So I talked about the initial sample size being 140,000 households being invited to take part at each quarter, which represents a much larger way one issued sample compared to the current survey. If we then translate that to look at what that means for the achieved sample sizes, how much data you actually going to get relative to what you're getting at the moment. This just gives an example and there's more detail here alongside this in our user guide. It's a link at the end of the slide pack as well that will enable you to see what this is. Look at this in more detail. And so here we've got the economically active individuals in a quarterly and annual data sets and we tried to provide something kind of similar. So the transformed survey covers both the equivalent of the labour force survey and the annual population survey. We will be producing both quarterly and annual data sets from this one transformed survey. And you get a much larger kind of data set size. And these are conservative estimates here because for example we've included the data brought forward on the LFS, which we currently don't do on the transformed survey. So here you can see that it's kind of roughly double the size of some of those data sets and in some cases more than that on your data sets. And again we've provided some figures I think in the user guides as well which provides some of the issues sampled by local authority and things like that. So I urge you to go and look at that if you're interested in more detail. So what does this actually mean for the respondents who are taking part in terms of the survey design for those people? So we start at the beginning, we send them a pre-notification letter. The difference with this being an online survey is that we have to do everything by email. We have to invite people to take part and we have to ensure that they actually open that mail and take part. So our pre-notification letter is there to encourage people to make them aware of what the survey is all about. So they're more likely to then open the letter when they finally receive their first invitation letter a week later. So their invitation letter includes an access code that enables them to go online and complete the survey online. It also includes a telephone number so that they can phone up and make an appointment to complete via telephone should they wish to do it that way as well. The following week we send a further reminder letter that again has their access code to encourage them to respond at that point. So what we're trying to do is to encourage as many people as possible to respond without us having to intervene, I suppose, for them to be able to just get their letter, open it and choose to take part and to do so online if they can do so or telephone if not. But we know that not everybody will take part if we just invite them to do so. So we need to have some way of following up those people. And this is where we try to use our resources to try and focus it on those people who haven't responded at that point. So this is where we start a knock-to-nudge visit. So these are the times when we go out onto the doorstep and actually engage in a face-to-face manner. We don't complete the interview on the doorstep, but we do provide new access codes. We collect telephone numbers to enable us to be able to phone them up and complete that interview via telephone if needed or to encourage them to respond online if they can. And week four is then the end of the collection period. So over that period of time we try to encourage people to respond by themselves at the beginning with increasing the level of intervention through those reminder letters. And then knock-to-nudge visits and follow-up telephone calls where we have their phone number throughout that period to encourage maximum level of response. So one of the other ways in which we have to encourage people to respond is by the use of incentives. So this is important not just to get people to respond but also to look at the types of people who are likely to take part. So there are always some people who are likely who will complete a survey no matter what you ask them to do. But we found that incentives, particularly physical incentives, something that they can feel in an envelope, makes it more likely for people just to open the envelope when they get their invitation to take part. And we've done a fair bit of work on this recently. We used to send out some tote bags which had a real squishability factor where actually you get something through the door and you think, oh, what is this? I'm more likely to rip it open and not just chuck it in the mail with your dominoes leaflet. But actually we've looked at the relative value of these things and now we've looked at these notepads which we're sending out at the moment which provide an almost equal impact to the tote bags but is much more environmentally friendly. But it's still this kind of hard thing that you want to feel, you want to know what's inside the envelope hopefully and open it and use that. So we've done some tests to see the impact of that and that was a lot more effective than, for example, just providing more money to encourage people to take part. So we have offered vouchers at the beginning as well but that was less effective than just a notepad and that's key for driving up that level of response. But we also provide that £10 e-voucher for everybody, every household rather, that takes part at the end once they've completed the survey. Once everybody in their household has taken part in that survey to encourage them to get all the way through the questionnaire, to an online questionnaire. Again, we have to have slightly different techniques to encourage people to get all the way through and not give up part of the way through as well. Next slide please, thank you. So we want to focus on quality and it's not just about increasing the overall level of response. We want to make sure that we're getting as representative data as possible and we do that in a number of ways such as by looking at the impact of the various incentives that we provide and what impact that has on different segments of the population and encouraging them to take part. But we also look at then the whole design of the survey and I'll talk a little bit more about how we introduce some of that later on. But our priority really, first of all, is to reduce the bias in the data. The overall level of response is probably less important than actually managing to reduce that level of bias, making sure that we've got the data that's as representative as possible at both national and local level. We want to be able to maximise inclusivity. We need to make sure that people are able to take part and that we're willing to take part as well. And we look at that by looking at the variability in response rather than just an overall level of response, but actually how much, how variable is that one figure that's often quoted in terms of a response rate. Our variable is across the different regions, across the index of multiple deprivation and across output area classification. It's kind of our three metrics that we're looking at the moment, so the ratio of response between the highest and lowest performing areas across those measures so that we can then reduce that variability and target our design to focus on those most underrepresented groups, drive up response in those areas and therefore drive up the output data quality. We want to make sure that our design is kind of as proportional as possible by age, sex, disability, tenure, ethnicity and other factors as well, but those are the key ones that we're looking at. We're looking at the impact of any of the design on the quality of the data that we receive. We need to make sure that we're reducing attrition, so we want to make sure that when we've got this five wave survey that we're maintaining those people through the survey is not enough for people just to take part at wave one. We can go actually, can't quite be bothered when it comes to kind of the next wave. We want to do what we can to try and keep them through those waves and that means making sure that we've got sufficient sample size at wave five. Enough people are taking part to make an able kind of meaningful analysis and reducing the impact of attrition on bias. So not just getting as many people as possible, but actually are we still maintaining that representative element as we get through the waves and that's a particular challenge for us. And I think we've probably still got some work to do, but is one of our areas that we really want to focus on. And only thirdly is our focus on improving response. Response rates are often quoted as the kind of the key metric in terms of the quality of the data. And we believe that actually it's more important probably to focus on the bias and the attrition with the response is still important in order to ensure that you have kind of sufficient achieved samples. But it mustn't come at a cost of the bias. We must make sure that we've got as representative data as we possibly can. But clearly increasing response has many other benefits as well as increasing the, you know, your achieved sample. It obviously improves the operational complexity and reduces respondent burden. You're not having to invite all these people who then aren't going to take part and obviously decrease the overall cost if you can get as many people to take part as possible. You have a greater proportion of people taking part compared to those who've been invited to take part at the beginning. And one of the ways in which we're implementing this is through being more adaptive and responsive, being able to use all the tools that we have at our disposal to look at how we can manage that design to deliver something that enables us to provide that more responsive data to ensure that we're using the best, the most expensive resources in the right places as well. We're targeting them where we can really achieve the best value. So what we're looking at, you know, our simple kind of plan of the process is essentially you draw the sample, then you invite people to take part, you follow up the non-responders and you process the data. And any of these stages we can look at focusing on kind of reducing bias and doing various things. But the bits that I want to focus on today is this middle bit. So the inviting people to take part in the following up non-responders, how we do that, how we make sure that we maximise response and reduce bias in that time in order to optimise this response, improve inclusivity and drive up the data quality. And one of the ways you do this is by being flexible in a way in which we implement our knock-to-nudge visit, which I've already mentioned. Are these visits where we visit or field interviewers kind of visit addresses on the doorstep and encourage response? They're not trying to get the people to take part in an interview then there as you would traditionally for the labour force survey. But we're trying to gather telephone numbers so that we can then phone them up later. We're trying to get that engagement with the survey. We're trying to get them to understand why we want to take them to take part and we're providing the making it as easy as possible essentially, but providing new access codes, etc. Building that report and being essentially less expensive enables us to make a lot more visits over the same period of time than if you were going to conduct kind of face-to-face interviews with all these people. It enables people to go away and complete the survey at their leisure as well as the time that suits them most while making sure that we're getting that engagement. We're explaining what it's all about, particularly for those addresses where we find it really difficult to engage with them just by sending them a letter in shared households, for example, where younger people are less likely to open their mail, but when we knock on the door at that point, we're able to engage with them, able to explain, able to provide all the information they need at that point in time so that they can go away and complete the survey. How do we do this? Well, we've done this by implementing an adaptive survey design. An adaptive survey design is where you look at the population as a whole and the characteristics of the different people within that population. Then we segment the sample into smaller groups of similar characteristics so that you can apply alternative survey features to each group. What that means in practice is that basically you're treating different groups slightly differently. You don't have the exact same design for everyone in the population. The way in which we're doing that is by looking at where we can focus on not-to-nudge resource. If you follow up the entire group, the entire sample, and you then have an equal probability of everybody responding at that point, you're likely still to then get those people in those easier areas and more likely to respond to those areas where you've already got overrepresented people. You don't really want to get more response in those areas. What you want to do is to be able to focus your resource on getting those underrepresented areas to take part. What we've done is run a logistic regression model, which looks at historical values, transformed labour force survey data to split that into various strata, which essentially represent the propensity of the population group to respond. We know that the characteristics that are most likely to influence the response of households within a particular geographic area, so lower super output areas that we're working with at the moment, is actually the age group, so the younger they are, the less likely they are to take part. Those are LSA ways with a median age group of under 45. The urban and rural classification, so we know that those in the urban areas are also less likely to take part. The index of multiple deprivation, so those most deprived areas, the IMD, DSLs, 1-4, are significantly less likely to take part than the least deprived areas. We've used this to then create these different strata, which enables us to target our resources most effectively. We're using these knock-to-nudge visits with geographic targeting for the underrepresented sample to make sure that we're focusing our resources on improving response in those particular areas. The way ASD works, our adaptive survey design, is that you can look at the options for investigating different modes of collection, different materials, incentives or timings, etc. Based on these different strata to see then, well how can we encourage response or different levels of response in different groups depending on their propensity to respond? What this is key in doing essentially is to understand more about how to improve that representivity. So it's not just about increasing the overall level of response, which you get a higher level of response if you've followed up all households, but you'd also get a much higher level of bias in that data as well. So by targeting it, by reducing the variability and response across areas, you're improving that representivity, reducing the bias in the data and reducing the challenges that we've got in some of the data quality. So I think in the next slide I'll be able to tell you a little bit about the different strata that we're using. So I mentioned the urban, the less deprived and the, or rather the more deprived and the age, the younger age groups has been those that are less likely to take part. So from those categories and from those variables, we're able to then assign every lower super output area across Great Britain into one of these strata and effectively then determine which our priority areas are. So this first iteration of this adaptive survey design is relatively simple. That those in the strata 2, 3, 4, 5, that's those urban areas that are more deprived and that are younger or urban more deprived and slightly older as well. Basically this is the combination of those variables which enables us to optimally use our resources to target those underrepresented groups and reduce that the bias in the data improve its representivity. So those are the ones that we're going out to. So essentially we have, we split the sample into two effectively so you've got those that are in those strata 2, 3, 4, 5. Those are the priority cases. Those are the ones that are getting field visits versus the other ones that we determine essentially that they won't be followed up. So you can see this chart here. It looks at the response, the return rate here by day during the operations across the 28 days. Now in the first two weeks, if you remember, we send out the invitation letter. We send out a reminder letter. We don't do it make any kind of knock-to-nudge, any doorstep visits at that point in time. So at day 14, you can see here this dark blue line. They're the cases that will be getting field visits. They will be getting those knock-to-nudge visits. And you can see that at that point in time there's a significant difference in the response rate for those areas compared to those which we have deemed to be not needing essentially that. Those knock-to-nudge visits. Those are the cases. They're essentially the less deprived areas, the areas with the older groups and the more rural areas as well. But you can see that once we implement that knock-to-nudge visits, once those dark blue lines actually start getting the visits from day 14 on, that brings that value up and reduces the difference in response across those following two weeks to the point where you've almost joined those together and reduce the disparity between your two groups. So, again, you're reducing the difference between your two groups and reducing the variability in response and therefore increasing the representivity of your data. And you can see that compared to essentially our grey line, which is actually before we introduced knock-to-nudge at all at that point. It was a very flat line. Nothing much was happening for the final two weeks. So you can see that overall we've increased response, but we've also reduced variability since the introduction of those knock-to-nudge visits. Now move on to the next slide, please, which shows some of the work that we've done on the questionnaire redevelopment. So it's not just about the overall design and the sample, but actually what the questions that we ask. So as James alluded to at the beginning, the LFS is very long. A lot of those questions have been on there for a very, very long time. What we want to do when we're looking at transforming the survey is about looking at all of those questions. But also if we're transforming to an online first survey, we need to make sure that it works from a respondent-centred approach. We need to make sure that the people who are asking to take part understand the concepts that we're asking them. And that that is then meeting that end data user meet. We want to make sure that you're getting the data that you need by ensuring that the respondents also understand the concepts that they're being asked to improve the quality of the data. And when you have a face-to-face survey or even a telephone survey, you've got an interviewer there to help explain, to help expand, to help provide a little bit more guidance around that. We need to do extra work when it comes to transforming to an online survey to make sure that that's a self-explanatory as possible to ensure that the respondents provide as consistent responses as possible as well. They're interpreting the questions in the way that we expect them to do so. So we do that through this respondent-centred approach by looking first at understanding and meeting the user need, not just the definition. But understanding then what that means for those respondents, ensuring that both the collection is both respondent-centred and inclusive. So we're not excluding particular groups of the population and that they understand what is required from then and taking part. We're reducing the burden very much so that we're making it as easy as possible for them to take part. We need to use the data to design to actually show what we need to be able to be doing and to take this opto-mode approach. So while it is online first, it's not online only. We do have by 8% of people of households at the moment which are completing our transformed survey via telephone as well. We want to make sure that we're focusing on both how we can ask the same questions via two different modes and ensure that we're not introducing any additional mode effects there. And we're looking at the impact of that and in when we're designing these questions as well. We're looking then at how we do this through the different flow and cognitive testing. So we're actually understanding the mental models behind some of this work. So the respondents understand the impact, both the concept that we're asking them and the particular information that we're asking for. And maintaining consistency with standards and time series where possible is also really important. We're also looking at what's currently being asked on the labour force survey and what we're asking on the transformed survey. And I think we've got an example next exactly. So one of these examples of actually when some of the work that we've done here is in looking at kind of employment in the reference week. This sounds quite a simple concept where it says on the LFS, did you do any paid work in the week ending Sunday at 8? Either as an employee or a self-employed. And when we actually tested that with some users, we looked as a self-response survey. People said, well, no, no, I didn't do any pay. I just did my normal job. This concept of work isn't always something that we think might be quite straightforward. But actually in terms of what these respondents from all sorts of different backgrounds might be interpreting a slightly different. Some people said, well, I did paperwork. Does that count? And they're kind of uncertain how to respond to this. So then we looked at, well, how can we rephrase that question? How can we go back out and test it again with the users to see how they would interpret it? So then we tried, well, did you do any work for payment or profit, including self-employment in the week ending Sunday? And they said, I still suggest because I still get holiday pay, but do I do anything to answer this or not? And they're a little uncertain they were on holiday, so I was paid does this count or not? And so then we were like, well, okay, what about if you've got, you know, if you're self-employed as well? What happens? What do we say then? So then we tried, do you have a paid job or business in the week Monday to Sunday? The job was a fairly easy concept. I had a job. Doesn't matter whether you're away from it or not at that point in time. It was quite easy to enable us to ask that question for people to understand it. But then we had that issue with, well, they were paying the silver wage, but nobody was paying the company. It's slightly complicated. The question you're talking about is if it's a business or not. So we didn't quite meet the needs for all of the population. So this soul trade has in particular found this slightly more challenging to respond to. So then we went with, did you have a paid job either as an employee or self-employed in the week Monday to Sunday? And this we found better captured the different types of self-employment. And it made it a little bit easier for people to take part. And we've gone a step further from this as well and where we said, even if people said no at this point later on, we've then asked them, well, did you own or operate a business in any of that period of time to make sure that we were still capturing some of those self-employed people who might have said no because they didn't have a paid job necessarily, but they still owned or operated a business in that period of time. So that kind of gives an example of the sort of iterative process that we go through to make sure that these questions are actually fulfilling the user needs that is required. And there are a number of ways and places in which we've implemented kind of some of these new approaches and done some kind of different things to try and optimize the data quality. One of those is industry and occupation. So this is a particular challenge when you're when you're asking these questions online compared to where you've got an interviewer who can probe many times. And one of the ways in which we've enabled us to be able to use the information that respondents provide when we ask them about their industry and occupation is by using this automated coding system that we've adapted from some of the sensors technology that we use there where we were trying to do a very similar kind of process. So what we've done is we've got this new machine learning automated coding process. That tells us about the confidence of the matching. So it lets us know actually how confident we are in the output of that coding tool. But then we've got some kind of clerical coding as well. So we've got people looking at these codes and manually coding some of this stuff. And we're able to then make those direct comparisons, which not only then tells us what's different from the clerical coders, but helps us to then use that information to feed it straight back into that machine learning automated coding. And then improve those tools as iteratively as we go on as well and reduce that need for clerical matching. We've got methodological tailoring. So we talked about samples differences between the labour force survey and the transformed survey. So we did a full review of that, that sampling and the stratification to come up with something that best met kind of user needs to deliver a representative sample. We've learned from some of the alterations during the pandemic, particularly around the waiting, the introduction of different variables and the different data sources that we used to look at those population totals as well. By integrating the latest census and population figures as well to update some of that estimation methodology and using an approved approach to imputation and editing as well. Reviewing the current process on the LFS and looking at how we can make those better and better meet kind of user needs as well. Quality of work is another topic where we've done some more work to look at actually what the impact of that is for users. And we've actually gone out and there's been full advisory panel to look at actually what the real requirement is there and how we can better fulfil the user need for that data. Including adding new questions, improved analysis and outputs and enabling us then to have that more targeted policy interventions as a result of the improved data that you'll get on the survey. It's not just about the respondents and the data users, it's about actually the production system as well, which enables us to provide this data to process it through a much more secure system on the latest platforms. We use Python, our open source and modular software to enable us to be able to build that to be able to process the data and using the latest lookups and standards. But making sure we've got this end-to-end flow and that it's optimised to work sufficiently and effectively as possible, reducing that requirement for manual intervention as well. So I think that's the end of my overview of the overall design. I think I'll hand back to James to talk about actually where we are now. So that's the development we've done so far, but we haven't finished yet. Thank you very much, Aula, and hopefully I'm back on. Hopefully you can hear me again. I'll leave the camera. I think it's crossed. So where are we now? Well, the current performance. So I've stolen this from Aula. Sorry, I know this is your favourite topic. So the current response rates to the TLFS, the Transform Survey, are looking around about the 39% mark. So 30% full response and 9% partial response, leading to an overall response return rate of about 39%. And you can see that in the blue bars, both on the left and the right. So on the left-hand diagram, it's week by week. So the different cohorts of people. And you can see the lighter-coloured bars. They're the cohorts still out in the field. So they're still being collected. So that's why they're lower. And you can see over the course of time as we get more and more responses, they come up to where the darker blue bars are. So that's us monitoring the process on a week by week basis, making sure we're getting all the responses we can. On the right-hand side, it's probably more familiar to you, thinking about quarters. So you can see there the response rate quarter by quarter by quarter. And you can see a big jump happening towards the right-hand side. So January to March, April to June, July to September. That's when we had implemented knock-to-nudge processes that all I was talking through. That's when we started to see even increased response rates by a good 5% points or so compared to where we were before. And still working on that, hopefully that'll increase over the course of time as well. Thinking of some of the measures that all are mentioned. So on the left-hand side, looking by indices of multiple deprivation. So I forgot to add the label, sorry. Category 1 on the left-hand side are the most deprived areas. Category 10 on the right-hand side are the least deprived areas. And you can see the difference in response rates from each different index of deprivation there. But the key target here is that the response rate from the least deprived areas is within double the response rate of the most deprived areas. And we are around about that at the moment. So just shy of 30% compared to just shy of 50%. We are within that tolerance limit, that quality standard. Of course we're still working to improve this, still trying to change things as best as we can. But this is an improved representative to representivity in comparison to the current LFS. On the right-hand side, I've also put there a diagram of the response rates by local area, excluding Northern Ireland, because we haven't got the responses from there just yet. But for the rest of Great Britain, you can see there the responses by local area. I won't pick out anywhere in particular. It's quite a big, detailed, complicated diagram. But you can see there that the more rural areas getting slightly better responses and the more urban areas. But this is still work in progress, watch this space. And if you want to, you can zoom into that diagram in your own time to find your own local area. So some changes that are being made now and early next year. So I mentioned earlier the final transition states happening last month and in the coming couple of weeks. So we've been improving some of the translations to the survey materials, especially the survey is available in English and in Welsh in a printed so to speak form. So improving the translations that we have in the materials that are available to people. Changing some of the questions. So all it was going through about the employment status a minute ago. We've also made some changes to the disability questions, the health and safety questions. We've added a question about trade union membership, removed questions about internet usage, and we've upgraded the well-being questions. They were previously in the half sample. They're now in the full sample. So everybody is now getting asked the well-being questions. So those changes are fundamental changes to the content. We've also had a few mechanical adjustments. Some behind the scenes things that you may not notice about the routing and the locking in the handling of individual questions. So some mechanical technical things in the actual design of the survey. Part of this is also feeding into the user guidance. So all of those changes, all that we've been doing to this point are feeding into the next iteration of user guidance. So there's already a web page available online with the background methodological information, with the sample sizes, with the metadata about the variables in the dataset, with an example of what the dataset looks like and some other information as well. All of that's available on the web page. We're expecting to update that in January. So watch out for version three of all that information coming out. And as we touched on earlier, the other changes that are coming up, we are incorporating the latest released population figures. So they came out a few days ago. We're trying to incorporate them into the system in both LFS and TLFS. And that's likely to happen January, February time. And we're also building finally the production processing systems. A lot of the focus has been on the quarterly person files, but we're now adding in the annual files, the longitudinal files, the household files, integrating all the GB and all the Irish figures all into the processing system over the coming three, four, five months. So hopefully it'll all be there ready for when we formally transition in March and May when those labour market publications go out. We've also been doing an awful lot of work as far as analysis, evaluation and review. So not just the day-to-day monitoring of the levels of bias and response rates and so forth, but also with our methodological colleagues and our analytical colleagues investigating any discontinuities or indeed just the time series of data that is available, trying to see where the LFS and the TLFS are the same, where they're different, where they compare with each other, any differences, whether we can explain them, whether it's a reasonable difference, whether we're actually, this was a design difference that we intended to do, or maybe this is a potential problem in the survey that possibly we need to fix. And part of that is also looking at the publications that are out there. So not just our own publications, other departments, other devolved administrations, any published bulletins out there. So trying to track the trend lines that we can see, making sure that they're moving in the right direction and that the TLFS and the LFS are giving explicable results in comparison with each other. And not just those two surveys, we're trying to look at other indicators as well. So other economic indicators, things like workforce jobs, things like VAT returns, things like PAYE tax returns, all sorts of business surveys and administrative data, whatever we can, whatever we have the time to try to compare the TLFS against these results and see whether they're in the same ballpark, whether they're moving in the same direction, whether there's anything we can learn that maybe needs a tweak or an amendment somewhere. And of course, part of this is the peer-reviewing process. So we're working with teams all across ONS, with the devolved administrations of Wales, Scotland and Northern Ireland, with a few other departments like the Bank, the Treasury, the Department of Business and Trade, a few others, and a bunch of economic experts. So specific expertise brought in and our fellows programme, so the ONS fellows brought in to peer-review the data, look at the user guidance, extract some data, extract some tables, compare it to their own knowledge, their own information, and then provide feedback and information of what they think is or isn't working and if so, how. So what you can expect to see over the coming few months is the continued delivery of micro data, so still releasing the files wherever we can, making sure that you still have access to the information, the continued delivery of labour market tables and the releases through NOMIS, the NOMIS website. So all of that should still continue to be published on a regular basis. And continued support and guidance and ongoing development. So not just giving you the information but being here to answer the questions that you have, give you the documentation, the information you need, if needs be producing some ad hoc tables or some information that you need extracted from the data. And as I said earlier, this is an ongoing development. So this is the first cut of information, the first massive change now and over the course of time adding a few more tweaks and changes and improvements over the coming months and years. This is a live survey that will continue to change and develop. There will be some changes to the micro data, specifically moving from SPSS format in the past to CSV format now, but there will also be changes to the variables, the variable names, the descriptions, the response options, some of these variables as well. Standard variables like age and sex are very unlikely to change. They've been set in stone for arguably thousands of years, but some of the variables will have changed your employment status or your family structure or whatever else from the analysis and the research that we've been doing. Please be aware of the user guidance and have a look at the mapping documents that we're producing to see what may have changed that may be of interest to you. And there will be some changes to the time series. We're doing what we can to revise the time series, make things as continuous as possible, employ the latest methodologies to both the LFS and the TLFS and the LFS results going back all the way to 2011. So at the top headline level, the employment rate and the breakdowns at the UK level should be as continuous and consistent and non-discontinuous as possible. But when you go to lower level breakdowns, if you use two, three or four variables, if you go down by age, by sex, by local authority, by disability status for example, you may start to see some differences between the two surveys as a necessary result of the transformation and the change. So finally from me and probably us as far as the presentation, we're taking this journey together with you, with all of you and with our own staff as well. So making sure that the people collecting the data, the people processing the data, the people analysing the data and yourselves, the users who do your own analysis, your own policy investigations, publications, whatever it might be, giving you as much information as we can. As I said, the user guidance, the next iteration is due in January. We're producing quarterly project delivery updates. The last one was last month. We're expecting the next one to be in January for where the project is going and what to expect next. Still providing all the updates we can through the newsletters, the outreach events, walk-through sessions that we have booked and organised. This being an example of one of those. So trying to book in outreach events with people wherever possible and giving presentations and updates at annual conferences and events and some user groups, for example. And just finally, if you have questions, things that you want updated, things you want to be added to a newsletter list or whatever it is, if you get in touch through labour market transformation with any questions, problems or issues or thoughts that you have, get in touch with that email address, let us know and we'll give you a response or bring you into whatever information we can. And that's it from me. So thank you very much everybody.