 So today we are going to talk about weights in understanding society, and I'm Olena Kaminska. I've been working on weights for understanding society for the last 10 years. And all the weights you see on the dataset of UKHLS and IP innovation panel have been created by me. Okay, so today I'm going to talk about four main topics that a lot of users are interested in. First of all, should we use weights? And what happens if you don't? Second, how do you select a correct weight for your analysis? Then I'm going to talk a little bit about advanced topics. So we have some users who are worried about zero weights, and they select the dataset, they run the analysis and realize that some cases are dropped because of zero weights. So you're going to talk about it whether it's a problem, whether you need to be worried about it, and what you can do about it. And finally, I'm going to talk about how you can create your own tailored weight for your analysis. Okay, first, should you use weights? The answer is yes, definitely. This is the easiest way to represent the population with our dataset. So in addition to weights, to any statistical software you use, you need to specify a cluster variable in the situation at PSU, a weight variable, this one example, do in stator it needs to be p-weight and straight a variable for stratification. This is the three things you need to specify to a software, and based on that, as long as you use a 3-byte column and you run your analysis, the software will know everything it needs to know about how we collected data, about the selection probabilities, about non-response. You don't need to worry about it, you can concentrate on your topic of research just with telling clustering weights and stratification to the software. You will be able to concentrate on your research and not worry about anything else. Now, what happens if you don't use weights? So this is an example, let's say you want to look at the proportion of the population, this is 0+, and you want to estimate it from wave 8 of other dataset. If you don't use weights, this is the proportions you get. Basically, especially in England and Northern Ireland, the proportion is quite off. So let's see at the weighted results, the weighted results are much closer to the population estimates. So basically our data is designed to be used with weights. Now, let's have a look at another question, and this is an example of an opinion question. Generally, this is a question about how people voted in the general election of 2017 in the UK, and the data comes from wave 8, but it's not the full wave. It's asked of people who participated in July and December 2017, and the results also exclude Northern Ireland. So generally, when we look at the different questions, we often can compare the factual questions to the population statistics, but it's extremely rare that we have opinion questions from the population statistics, and election questions whom you have voted from the last election is one of very, very few questions we can actually have a population statistics for. So because people go and vote, express their opinion, we have the record of that, we know the election results. So if you use unweighted data, you see that we would think that the UK has quite a lot of labour favourites, they like labour, the difference between Conservatives and labour is very big. Let's have a look at the weighted results. They are much, much closer to what happened in the general election. So you would say there are some analysis published without weights, which you should think about it, and actually what happens if you run your analysis without weights? Well, my response to that is it may be close to the truth, it may be not close to the truth, it may be a little bit off, it may be quite a lot off, but you will not know until you run your results with weights, you will not know. And going back to this general election question whom you voted for, imagine you didn't have population results. Imagine all you had is answers in our dataset. And this is what most of the questions that our users study, this is our situation, we don't have population statistics, that's why we use survey data, because that's the means to estimate them. And what happens is that if you only look at unweighted results, you will be writing a paper telling that look UK is favours labour quite a lot, and conservatives is much further away in favouring, so the difference is around 12 percentage points, it's very big, and so this is the way UK is, and if we didn't know the true results, you would be writing this paper, and what the risk in writing such a paper with unweighted results is that you will have a colleague looking at your paper thinking let me re-analyze this data using weights, finding that the actual results are very different to how you describe them, and saying actually the paper you published with unweighted results is misleading, the actual results are such and such and the actual situation is different, and your conclusions are wrong. This is what you don't want to do. So if you actually ever considered publishing unweighted results, my suggestion is at least run weighted results for yourself, at least know your risk, and I would at least put weighted results in a footnote, but ideally you want to have results that represent the population and you would want to use both weights, and in clustering and stratification. Okay, so our data set is designed for multiple purpose, it's complex, and we have a lot of weights, so I'm going to talk in this section about how to select a weight for your analysis. If you open a data set like InDress, and if you scroll all the way down, in the very end of the data set you will see weights, and in this situation we have a lot of weights, these are longitudinal, these are cross-sectional, there are lots of them, so how do you know which weight to choose? You need to answer a few questions, and these answers to these questions will fill in the name of the weight, which consists of five different parts. So in the following section I'm going to go through these questions and show you how you can fill this better than the name of the weight. First, think about your analysis, are you using one wave? So is it one time point? Are you using multiple ways? And are you looking at the change over time? And so if it's change over time, you are looking at longitudinal data, and you will be using underscore LW weight, and if it's cross-sectional, it's going to be underscore XW. Next, think about the waves you are using in your analysis. If you are looking at cross-sectional analysis from wave 9, for example, your weight will be corresponding to I, and it will be I underscore. Now, if you are looking at longitudinal analysis, think about the last wave of your analysis, and if you are looking, for example, at all waves, starting at 1, and ending at 9, the last wave of your analysis will be wave 9, and it will be I underscore. Next, think about whom you study. So if your analysis is at a household level, for example, you are looking at the proportion of households in the UK with a particular characteristic, you are going to use HHD. If you are looking at people, but zero plus, or you are looking at adults or any aged people, but you are looking at... All your questions come from household questionnaire, and they can be found in Indole. It's PSN here. If you are studying youth, and your questions come from youth questionnaire, it's YTH, and if it's for adults, 16 plus, and your questions come from in grasp, it's IND. Next, we have a number of different instruments for adults, and so if you are looking, if you are studying adults, think about where these questions come from. If all your questions are asked in main questionnaire as well as to proxies, it's BX. If all your questions are asked in main questionnaire, but are not asked to proxies, it's IN. If questions are asked in self-completion questionnaire, it's SC. If your questions come from nurse wizard, it's NS. And if your questions come from blood samples, then it's BD. And we have also five extra minutes questionnaire. These questions are asked to minority groups, but also representative small example of general population. This is fine. Okay, but you will tell me, in my analysis, I'm using a number of different instruments, for example. I'm using questions from household level, but I'm also looking at questions that are asked to proxy and main interview, and some questions come from self-completion interview. That's fine. So use this table and then have a mental take on the levels that you use and always use the weight from the lowest number. So if you use the lowest in this example, your weight will come from self-completion interview. Another example is, for example, you will be using information for household level, main interview from nurse wizard, but also information from blood. Then your weight will be from blood, will be tailored to blood and will be BD. So always use the lowest level of analysis. Okay, next, this is also related to waves you are using, but this is slightly different. So think about the waves you are using and if it's any wave six or onwards, you will be using your eye weight. For example, if your analysis is using wave nine, so you can use your eye cross-sectional weight. If you are in your analysis, you are starting a change between wave eight and nine. This will be also your eye weight. But if your analysis starts at wave two or any time afterwards, you can use your B-weight. If it starts at wave one, it's your S-weight. If it starts in 2001, it's zero-one weight and if it starts in 1991, it's 91 weight. Now, there is a little bit of a hint I have for you. If you are interested in the largest number we have, it's best to start at wave six. If you are interested in ethnic minorities, so wave six or wave two would be a good start, or wave one. If you are interested in starting on an island but going as far away, as far ago as possible, zero-one weight is good. If you are interested in starting Scotland and Wales separately as subgroups, you would also use zero-one weight. If you want to look at Great Britain and you want to go as far back as possible, you would go for 1991. And if you want high sample size for Great Britain or for UK, you may want to start at wave one or wave two. Now, we are going to slide some more advanced topics and I met users who told me they want high sample size but they noticed that when they looked at our data, some cases are dropped because of zero weight. There are some reasons we have zero weight. Some are by design and some are because of how we calculate other weights. So to start with, we have temporary sample members. These are people who move into other households where we have original sample members who we selected at wave one and they are there, so we ask them questions, but they are not part of the longitudinal sample by design. So their longitudinal weights are zero. Note that the cross-sectional weights are positive because we use weight-share method, but a lot of the reason why we interview them is that they provide very valuable information about household and context for other longitudinal sample members. What's unusual about understanding the society and this is related to its complex sample design is that we have TSMs from wave one. This doesn't happen normally in other household panels. This is slightly unusual and in other household panels you will see that wave one weight everyone gets and it's positive for everyone, but not in understanding society. So who are these? These are the people who lived in EMB or IMB households, but they are not eligible for these samples. So EMB is ethnic minority boost and IMB is immigration and ethnic minority boost. They happened in wave one and six and as part of them, we were interested in boosting specifically ethnic minorities. But of course, ethnic minorities, some of them live with people who are not ethnic minorities or who are not eligible for this sample and they were still interviewed to provide the context of the household, but because the way they were selected they will always have a zero weight. Basically the selection probability into other panel is zero and that's what weights reflect. So if you ever get a weight, if someone creates a weight for you that doesn't have zero values at all, be very cautious because most likely it's not correct. Then we have longitudinal weights and they assume participation in all waves. So we do have zero weights for anyone who missed at least one wave and we have cross-sectional weights that currently require household participation in all waves, although this requirement is not very strict. For example, your eye weight doesn't require participation in waves three, four and five. Now, do you need to worry about zero weights? And so here I'm providing an example what happens and which effect zero weights have on your analysis. In this situation, imagine you want to estimate a proportion of natural adoptive step methods of a child under 16 and I'm using wave 8 here. What I find is that there are 33,818 people who have positive weight, but total number in the data set is 39,000. So technically you're using over 5,000 people should you worry about it. So to demonstrate the effect of this, what I did, I used these 33,818 people. They have positive weights and I randomly select different numbers from out of this sample and estimate this proportion and showed you the estimate and standard error and confidence interval. So imagine you selected 20 people. What happens? Your estimate is 19%, but the confidence interval is really big and so it changes from 0 to 40%. This is big, but then the sample size is very small. Let's imagine you have 50 people. What happens? So the estimate goes down to 13.9 and confidence interval is much smaller but still rather wide. It's somewhere between 4% to 24%. Okay, let's move to 100 people. Your estimate is now 10.6 and confidence interval is somewhere between 4 and 17.2. Let's add another 400 people. So now we have 500. Your estimate is now 13.6. Your confidence interval is narrower. Let's go to 1,000. Your estimate is now 13.8. So it stabilizes and confidence interval is now narrower somewhere between 11.3 and 16.4%. Well, we have now 1,000 people. Let's see what happens if you have 5,000 people. The estimate doesn't change much. The confidence interval goes down. So it's narrower. Okay, another 5,000 people. Your estimate doesn't change and the confidence interval is narrower but actually the advantage of 5,000 people is not that big. Well, let's go and add 10,000 people more. Like the estimate almost didn't change and confidence interval, if you compare, it's 12.5 here, 12.6 here, and 13.6 here. So now it is 1 percentage point wide and it didn't change here. What happens if you add another 10,000? Well, it doesn't change and here the confidence interval barely changed as well. So if you imagine you add another 10,000, you will be at around 40,000 here. What do you think will happen to the estimate? It will not change much probably. And imagine what will happen to the confidence interval? It will get narrower but by how much? Not that much. What happens if a user says, well, actually I'm worried about this 5,000 people. Now I'm going to use everyone I have because I'm going to get maybe better estimate. What happens then? So he uses all the people, even though those who have zero weight uses unweighted estimates. He gets 15% as an estimate and he gets this confidence interval. What I see as a problem is that his confidence interval doesn't include the weighted estimate at all. So it's likely that his estimate is biased and the confidence interval, it's so biased that even confidence interval doesn't include the weighted estimate. So what do you gain from unweighted estimate? Well, you gain a bias and a few extra people. Let's have a look at the same, so this is the same table but expressed in a graph. And so these are the sample size that we were talking about. This is 20, 30, and so forth. Going to 33,818. And what happens, this is the estimate of the proportion of mothers. This is the confidence interval. So what I want to show you is that the estimate is very volatile and fluctuates a lot up until 500 probably with 1,000 it stabilizes and really doesn't change anything from 5,000 onwards or maybe from 1,000 onwards. The confidence interval is really big with small numbers but goes down quite quickly and maybe at 500 is already quite narrow at 1,000, it's very narrow and after 5,000 really you don't gain much if you gain more people. You don't gain much in your estimate and in confidence interval. So my suggestion to you if you are anywhere here in your sample size 1,000 plus, 5,000 plus getting another 10,000 won't change your results and definitely not worth not using weights just for the sake of having more people in your analysis. Now if you are anywhere here between under 300 yes, you may want to consider some alternative options. So some people would say I still want more in terms of sample size and my suggestion to you is first analyze our data with our weights as a first step and have a look at it. If you have, doesn't matter which sample size you have if your results are significant just go and publish that. If you have small sample size in your subgroup and your result is still significant it's perfect. You probably have an important bake effect and it's worth publishing it. Don't worry about anything else use weights publish. Now if your results with other weights are not significant but the p-value is large let's say it's 0.6 it's unlikely that adding 10-20% of a sample will make this p-value below 0.5. It's, you know, you can add a lot of sample size but it's probably going to stay not significant. So I wouldn't try to just not use weights or even do anything with weights I would actually try to think about why is my effect not as big as I was expecting. Now if your p-value is marginal so let's say your p-value is 0.08 this is a situation where it's worth considering tailored weighting which can help you with sample size it can bring us a 10-20% sample back but keep in mind that it's a big exercise and in the end you may still get the same p-value of 0.08 that may be what it is maybe not, you know, an effect an important effect but not that big but before you complain about sample size what about sample size? Just do these steps first around our data with weights and this brings me to tailored weights there are three reasons why you would want to do tailored weights and the first I already mentioned so this is sample size you may gain a little bit in sample size second, there are situations where users use very unusual combination of other instruments in this situation we won't have a weight for you and this may be a very good reason why you can create your own tailored weight for your analysis and there are some times the third reason to use tailored weights is when and there are some users who want specific predictors in their weighting model or they don't want a specific predictor in the weighting model for example they don't want gender in any of the weighting model to appear so in this situation you can create your own weights the positive point about it is that we provide everything to enable you to create your own weights so it is possible on the other hand it is complex and suddenly all of this magic of the weights that you don't need to worry about the data how the data was collected all the selection probabilities all the little things that influence weights and probabilities and non-response you didn't need to worry before when you used weights now you need to worry about it you need to take everything into account correctly and you need to be very careful that you don't miss any bit part so if you actually want to create your own tailored weight my suggestion and the easiest way to do it is to start with one of our weights for example you may be studying a longitudinal analysis of the change between wave 1 and 9 so you don't care about anything that happened between these weights you just are interested in wave 1 and wave 9 so what you can do is you can start with wave 1 weight and you can model the non-response conditional on wave 1 another example is you may want to study a change between wave 8 and 9 so I would suggest in this situation to start with our wave 6 issue weight and model a joint wave 8 and 9 response conditional on wave 6 positive weight and third there's an example with different instruments for example you are studying with aspirations at wave 1 and what actually happened to them with their adults at wave 9 so in this situation and there are different opportunities here but you will be using use questions at wave 1 and main questionnaire at wave 9 and we don't have a specific weight for this unusual combination of the instruments but there are a few different weights you can start with and for example you can start with enumerated weight at wave 9 and then conditional on that you would model a joint response to use questionnaire at wave 1 and main questionnaire at wave 9 conditional on this weight okay in general if you want to create your own weight I suggest you do your own attrition adjustment it's much easier and you again start with our weight so if you want to do your own attrition adjustment you can start at wave 1 using our cross-sectional weight you can start at wave 2 or 6 using the respect the issue weights that we created for you in this situation you would use predicts that are already available in the data set from wave 1 to 6 respectively and you don't need to remember to also take into account new bonds death moving out of the country becoming 16 mortality adjustment among others remember you can also create your own cross-sectional weight through a weight share okay let's say you are ambitious and you want to create your own weights from scratch this is possible and we provide all the information you would need for this this is not necessarily easy and I'm going to tell you to talk to you through the steps we take to create our weights to give you a little introduction how you can do that so first of all the weight is a reciprocal of the probability and the probability that goes into weight consists of three probabilities the selection probability probability of response in wave 1 and probability of response from wave 1 onwards with the opposite of attrition so the product of these three creates a total probability and if you divide one over this probability you get the total weight so this is a simple situation and it would be true for our GPS the general population sample also for BHPS 1991 sample it becomes a little bit more complex if you look at all the examples in our analysis so this graph represents all the examples and let's say if you are looking at wave 9 these are all the examples that feed into it and so from statistical point of view and from sampling selection point of view this is how I think about it we have three countries selected so an example in 1991 was selected in three countries we had a boost of Scotland and Wales in 99 Northern Ireland started in 2001 we got general population so an example in four countries that's UK HLS start and ethnic minority boost in three countries and then we got immigrant and ethnic minority boost in the three countries and these are the immigrants who moved into the household with our original sample members so all of these samples are combined so why is it important for each person that is in the sample we need to figure their probability to be selected through each single sample and to have participated up until point that your study in the situation will be wave 9 so for example imagine an Indian who moved into this country in 1990 and he moved to the Northern Ireland so let's say he moved to London he lived in London until 2000 moved to Northern Ireland in 2000 moved back in 2005 let's say to London again for such a person he could have been selected in let's say he was selected in IMB here but actually he could have been selected in 1991 in London he couldn't be selected through Scottish and Welsh boost but he could have been selected through Northern Ireland example in 2001 because he was in Northern Ireland at that time he moved back to London remember in 2005 so let's say he could have been selected in 2009 in London but he also could have been selected through ethnic minority boost in London as well separately then he could have been selected through IMB in 2014-15 sample as well now all of these probabilities need to be combined because he was could have been sampled through each of them but also it's not only the probability what we need to think about is his chance to not only be sampled here but also to respond through all of these years up until this point and if he was selected in 1991 the attrition is so much bigger over this year than if he was selected in 2014-15 the attrition is smaller over the three first years of the sample so non-response probability is separate calculated for each of these samples as well now moving forward we have basically for each person 17 selection probabilities and they are combined by adding them because these are chain probabilities so the design weight consists of 17 selection probabilities they are currently released and you can use them if you want to construct your own weights for example now the actual weight that is released is a combination of 34 parts so 17 probabilities multiplied each by a respective response probability and this probability of response consists of two parts which is weight 1 a response and attrition correction so if you want to start the selection probabilities in addition to the previous slide you need to remember also that we have weight 1 non-response and it's not negligible some people say I'm going to start with design weight because design weights are provided it's easy and I'm going to correct for non-response but what they think about is attrition attrition is easy to correct because you started weight 1 it's rich in information and then you model it and you model your response based on weight 1 but don't forget about non-response at weight 1 what's the issue with it the problem that if you want to correct for non-response you need to have predictors for both respondents and non-respondents you need to have predictors for all the sampled households both the ones that responded and also those that didn't respond so what we did is we I had access to postcards and small area indicators and I linked them to external geographical data sets sensors but also a number of other ones I obtained these predictors for our sample and I used them to control for this non-responsive weight 1 so my suggestion is if you are going this route and you want to create your own weights do remember about weight 1 non-response this will be additional step for you to link our data to external sources and how these predictors and remember that correcting just for attrition using design weights is not enough because basically you'll be correcting only for around 20% of non-response or total non-response and that's not sufficient the main point from this presentation is to remember that weights are like magic wand you tell a software statistical software what the weight is and you don't need to worry about how the data was collected about selection probabilities and germ selection probabilities you don't need to worry about non-response you don't need to worry about linking our data to external sources to correct for non-response you also don't need to worry about things like eligibility we correct for non-eligibility for ethnic minority boost we correct for mortality we give correct weights for new entrances such as newborns and raisin 16s all of these you don't need to worry about this job is done for you but what I hope you do is spend your time on your topic of research just tell when you use our data you tell a software this is a classroom variable this is the weight, this is the stratification give me the representative results of our population so remember weights are magic wand and your only task is to select the correct weight now if you have other questions on the weights this is our user guide the way we calculated weights is described in detail there we also have user support forum if you have a question check the support forum first a lot of people have asked questions before you and we have answered them so it's very useful it's a place to look at you can also ask a question there and we will be able to answer it you can also email to us or you can ask to have a help desk over where we talk to you and we answer your question via video conference finally there is a separate video on how to select a correct weight you can also watch it