 in the CLSA unpacking sampling weights in their use. And I'm really delighted to introduce our speaker, Dr. Lauren Griffith. Lauren is an associate scientific director and the Hamilton site lead of the Canadian Longitudinal Study on Aging. She's an associate professor in the Department of Health Research Methods, Evidence and Impact at McMaster University. And her research interests include physical functioning, multi-morbidity and frailty, as well as the harmonization of longitudinal data. And I think that for many of us, this is a long-awaited webinar to discuss the weights and the new weights that we're gonna be using in the CLSA. So I invite Lauren now to begin the webinar. Thanks, Lauren. Thank you very much, Tina. I'm very happy to be here today to give this presentation. Let me just see if I can get started here. So as Tina suggested, I'm going to be talking about the new weights in the CLSA, but I'm gonna talk a little bit about the development of weights and how we use them in the CLSA as well. And I would be remiss if I didn't start this presentation without acknowledging the incredible contributions made by Dr. Mary Thompson and Dr. Ching Ba-Wu from the University of Waterloo and Dr. Harry Shen, and who is now emeritus at McMaster in the original creation of the CLSA weights, as well as Dr. Nazmul So-Hel, Dr. Arun Erbaz-Auz and Dr. Henry So in their work in also creating the weights, creating the documentation for the weights. And as you will see, there's some new documentation that Dr. So-Hel has created that I think will be of interest to many people in terms of how the weight should be used in analyses. So I wanted to start by giving a brief outline of what will be covered in the talk. So I'm going to give a fairly high-level description of why do we use sampling weights at all, talk a little bit about the CLSA sampling and the use of sampling weights in the CLSA, what we provide, and then about why we need new sampling weights and finally how the original sampling weights and the new sampling weights differ and when they will be available and then a little bit about what's coming next. So I wanted to be upfront so as not to disappoint anyone during the presentation. What I won't be doing is giving technical guidance on the use of sampling weights during this presentation. But as I said, at the end of the presentation, there'll be some additional information that maybe a bit of a cliffhanger right now in terms of what will be available on the CLSA website that will be useful in this aspect. So without any further ado, I'll start the talk with why do we use sampling weights? And essentially what we do when we do any sort of research projects is we collect data from a sample of people, usually participants in our situation, but what we want to really do with this sample is we want to make generalizations about the population that the sample represents. But the issue is that the sample is almost never fully representative of the population. And so what we do is we create sampling weights. So let's assume in a very simple case that we have our population, and in this population there's 50% females and 50% males. But when we draw our samples for whatever reason, our sample has 70% females and 30% males. But we still want to make some inferences or draw conclusions about the population. So what we do is we create sampling weights. And so as I said, essentially the sampling weights are used to make the statistics that we compute from our data, from our sample data, more representative of the population. It's a standard practice in surveys to use sampling weights. And then each participant in the study, in the sample is assigned a weight that is constructed based on their inclusion probability. And I'll give a little bit of an example of what we mean by inclusion probability in the next slides. But essentially, sampling weights are always positive and nonzero. So in our example, when we have our population of 50-50 in terms of sex and in our sample of 70-30, we know that we have an underrepresentation in this sample of males and an overrepresentation of females. So if you could see the red text, you could see the formula that we could use to create our basic inflation weights. So here, this formula you can see has something to do with the population of the females in the target population, the proportion of females in the target population and the proportion in our sample, as well as the total number in our target population and the number in our sample. So how does this actually work? So let's say we have our target population is 200 people, 100 females, 100 males, and we draw our sample of 30. And as I said before, it was 70% females and 30% males. So in our sample, we have 21 females and 9 males. So what we can do then is we can figure out how many people in our target population, each person in our sample represents. So in this case, each female in our sample represents 100 divided by 21 or 4.76 people. And for males, each male is going to represent 11.11 people. And again, intuitively, it makes sense because we have males underrepresented in our sample that they're going to have a larger weight when we want to try and generalize back to the population. So this is a very kind of high-level example. But essentially, it really is what we do in CLSA and creating the weights. It's a little bit more complicated because our sampling is more complicated, but that is essentially what the weights are doing. So I wanted to start and just make sure that everyone who's attending the webinar, I know many people have used CLSA data and I'll probably be telling you something that you already are very well aware of, but just to make sure that everyone is aware of the design of the CLSA, I just want to take a minute to review this slide. So in the CLSA, the target was to recruit 50,000 participants aged 45 to 85 at baseline. And we started recruitment, as you can see on the bottom of the figure in 2010. We finished recruitment in 2015. And of that 50,000, our target was to recruit 20,000. We actually created or recruited 21,241 participants who would be randomly selected within the 10 provinces. And these people, these participants provided their questionnaire data by computer-assisted telephone interviewing. And we generally refer to these people as tracking participants. And I'll probably use that as I'm going through the slides. The other 30,000, as a 50,000, and again, the actual number was 30,097, were to be randomly selected from a catchment area of 10 to 50 kilometers around 11 data collection sites located throughout Canada. And the reason that these people were in a geographically restricted area is that we were going to collect the data through in-person, in-home interviews. But these people also came to one of the data collection sites across Canada and provided additional clinical and physical tests as well as blood and urine if they consented to do so. So as you can see as we move down in this figure that at baseline, we completed baseline recruitment by 2015, and then every three years we have another wave of data collection. So we completed follow-up one in 2018 and we are in the middle of our follow-up two even with our COVID restrictions. We're still collecting data via telephone interviews from all of our participants and we'll be finishing follow-up two in the spring or summer of 2021. So just to give you an idea again what this looks like, and I should have said the latter group that we collect the data in-person and at the data collection site, we usually refer to them as the comprehensive cohort. So this slide gives you an idea of what we think of in terms of our sampling for the tracking people. They're represented by the purple dots in the map. So they're essentially a stratified random sample of the ten provinces. And then the green dots represent the data collection sites. And you can see that they're, although they're not clearly a random sample, they are distributed across Canada, but they're not in every province. So we have three in British Columbia and Victoria has a data collection site and then Vancouver and Surrey actually share one data collection site in Calgary, Alberta, Winnipeg, Manitoba. We have them in Hamilton and Ottawa in Ontario and in Montreal and Sherbrooke in Quebec as well as Halifax and in St. John. So if you think about in terms of our sampling, the sample was chosen randomly through the ten provinces, but at the data collection sites there was a geographically limited sample for the comprehensive. So one thing that can be quite challenging when you're trying to recruit 50,000 people is to be able to do it with one sampling frame. That would have been nice, but we had to use multiple sampling frames to actually have multiple methods to recruit our sample. So a sampling frame is just a list of all the possible sampling units for, in our case, participants, people. And so our first sampling frame that we used, and this was in the tracking only, was the Canadian Community Health Survey on Healthy Aging. And this was actually a survey that was done by Statistics Canada and for the first time, Statistics Canada showed the participants of CCHS Healthy Aging to be asked if they would be consent to provide their contact information to CLSA researchers for the purpose of recruitment into the CLSA. And so this was our first sampling frame and essentially our inclusion and exclusion criteria were adapted from this as our first sampling frame. But we also recruited people using provincial health registries as well as telephone sampling. And for comprehensive only, we did recruit some participants through the New Age study, which is the Quebec Longitudinal Study on Nutrition and Aging. So you could see in terms of our example of picking that nice little sample it gets more complex as you have multiple sampling frames. The other thing that we did in CLSA is we used stratified random sampling. And stratified random sampling, it's just a matter of a population. So the full population is subdivided into mutually excluded subpopulations. And simple random sampling is used to draw the sample from each of these subpopulations. And why do you do stratified random sampling? It can be done for a number of reasons. It can be done for convenience. But it can also be done to provide more precise estimates under many circumstances. And when I mean precise estimates, I mean with less variance or a confidence interval would be tighter. And also to obtain estimates for the subpopulations. And here in the CLSA it was really mostly done for these last two reasons. And we knew that the CLSA, clearly it's a study of Canada, but it would be really important for us to be able to at least look at province level estimates because we want to look for differences or similarities across provinces. And you can imagine if we just did simple random sampling we'd have very few participants from provinces that were smaller like PEI and we'd have a lot of participants in larger provinces like Ontario and Quebec and BC. So we used a number of strata in CLSA and we had the 10 provinces for tracking. Seven from comprehensive as you remember they're not in the data collection sites and not in every province. We use mail and CML as a strata. We balanced the sample between those two as best we could. We also used age group as strata and we had four essentially 10 year age groups. And what we did here is our goal was to have about 60% of the sample in the two younger age categories in two younger age groups and about 40% of the sample in the older two age groups because as it is a CLSA as a study of aging we wanted to make sure that we had enough population in the younger age groups so we could look at transitions and trajectories over time. We also had a strata of the data collection site catchment area or non-catchment area and this was to create the overall CLSA weights because you can imagine that for each data collection site we had about 3,000 participants so we over sampled in those areas if we wanted to actually combine the tracking and the comprehensive groups together to have an overall CLSA sample so we had to take that into account as well. And so that was our early plan and what we found was early on we did analyses and we realized like many cohort studies or studies in general there was an underrepresentation of people with lower SCS with respect to education and income and we wanted to try and figure out how best to rectify this. We knew that we potentially could not be completely representative with respect to this but the other part was that which was just as important was that we needed to have enough people enough heterogeneity across SCS so that we could have well-powered statistical tests to look at this as a factor because we knew that SCS clearly is a factor in health and aging. So what we chose to do and we had to take into account in terms of our sample weights is we chose to over sample from dissemination areas so these are just geographical areas that we identified using census data that had a higher percent of people with lower levels of education. So in the end we ended up with the geographic education strata as well so lower levels of education versus higher levels of education. So we had to use all of this information then to create our weights. So we have two types of weights. The first weight is what we call an inflation weight and this is in we've constructed this for the tracking sample for the comprehensive sample and for CLSA overall but essentially what this is we start by creating the basic design weight which is the proportion which are proportional to the reciprocal of the inclusion probabilities that we computed those. So that's kind of similar to what we did in that first example when each male represented 11.11 people in the population. So we created these design weights and then we recalibrated them to the overall targeted eligible Canadian population. So again similar to that other sample if we would have added up the weights for all of the females in our sample of 30 in all of the males in our sample we would have gotten back to the actual target the actual number of people in the target population in that simple example but here again we wanted to then recalibrate them to the CLSA or to the our target sample and as you remember because the CCHS was our first sampling frame and we used very similar inclusion exclusion criteria as the CCHS healthy aging we thought that would be a good way to potentially a good source to use to recalibrate our weights. So we have inflation weights again and inflation weights are used for the estimation of descriptive parameters so they reflect the estimated parameters in the target population so a descriptive parameter would be say the mean grip strength so we want to say something about strength in Canadians between 45 to 85 that meet all of our inclusion and exclusion criteria or you could estimate something like the prevalence of coronary heart disease so you can't have a webinar on sampling without at least one formula but again just to give you more of an intuitive feel of how the weights are used and again a very simple example of estimating the prevalence of coronary heart disease so here in the numerator we're just summing over adding all of the values of the weight times this yi so each person has their own weight and yi is one if the participant had coronary heart disease and zero otherwise so here oops sorry here if we sum all of these together this numerator is going to represent the number of people in the target population with coronary heart disease and then the denominator is just the sum of the weights and that's going to be the number of people in the target population so that's going to give the prevalence of CHD in the target population and you can see as well if all the weights were one so that each individual just had a weight of one that numerator it would be the number of people in the CLSA with CHD and the denominator would be the number of people in the CLSA so that would just be the prevalence of CHD in our sample with the weights were able to take our sample estimates and provide estimates that are generalizable to our target population so there's also another type of weights that is called analytic weights and these are proportional to the inflation weights but they're rescaled to sum to the size the sample size within each province so their mean value is one within each province and these weights are intended to be used for modeling so for example in regression analyses where the weighting variables are included in these models and so why do we need the two kinds of weights you can imagine that the weights even if we take a relatively large sample the weights in a place a province like PEI is going to the weights will be relatively small because they just have to weight up to the population in PEI whereas if we take maybe even a little bit of a bigger sample in Ontario the weights for that sample have to weight up to the population of Ontario so provinces with larger populations tend to have much higher inflation weights compared to smaller provinces and so the observations from those strata would tend to dominate statistical analyses but with the analytic weights the point estimates will be fairly similar should be the same but they are more efficient if the model is correctly specified so what we mean by efficient again that is a statistical term that means if you have a number of unbiased estimates of a parameter the one that has the smallest variance is going to be the most efficient so again we are trying to get the most precise estimates of our of our parameters that we are estimating so we have our weight for the pool of data for all of the CLSA then and the inflation weights were provided for the two sub cohorts but also for the full CLSA and that was based on the tracking and comprehensive inclusion probabilities for participants within the DCS areas because clearly there are no comprehensive participants outside of the DCS and the tracking and inclusion probabilities for participants in the non DCS areas and so when we got these again then we recalibrated them to the population and so what we have in CLSA or what we still have but what we will be updating soon are three types of inflation weights and three types of analytic weights so if you are doing analyses where you are only using the tracking participants there is a set of weights for those if you are only using comprehensive participants there is a set of weights for those and if you are using all 50,000 some odd participants then there is a set of weights for those and we have both inflation and analytic weights available so we did all that work the question why do we need new weights and with many studies as we move along I think you would always want to try and improve what you have done but our original anticipation was that most of the analyses would be conducted at the province level or at the level of Canada and so we used the CCHA healthy aging to calibrate both the tracking and comprehensive weights but CCHS healthy aging weights they were very good for estimating parameters at the province level and at the level of the health region so not really at the level of the data collection site catchment area as we were using and so as they worked very well for the tracking cohort they worked a little bit less well for the DCS because again if you think about the data collection sites it really is it's not even like in terms of the data collection site in Hamilton it's not really 25 to 50 kilometers around the center of Hamilton it's around where the data collection site is located in Hamilton so in Hamilton we include Hamilton but there's also people from Burlington from Oakville and for some of the more rural areas like Flamborough and Hamilton so it doesn't really line up necessarily with one health region and as well there was a project that kind of brought this to our or made us aware of this where they were interested at using CLSA data at the sub-provincial level and what they were finding is it was not necessarily reflecting well in terms of what we knew from census data and the other thing that as you remember what we wanted to do is we knew that we wanted to increase the variability or our power to look at analyses with respect to income and education and other SES factors we knew that that the sample was not it was representative in many in many respects if we look at say some many chronic conditions in Canada we look at them with CLSA it will match pretty well but it was matched less well when we looked at some of these SES factors so we thought that maybe we could go back and do a little bit more. One of the other issues is although CCHS healthy aging was ideal for many reasons especially that it really was the population of CCHS was essentially our target population in CLSA they actually used the 2006 census to help with to construct their sampling weights. So what we wanted to do was we wanted to use something that was a little bit more in line with when we actually did our recruitment in CLSA so we're now using the 2011 national household survey to do our to do our calibration and again the additional refinement which I think is important is that we used the individual level rather than the geographic level variable for education for our weight calibration. So what is that going to look like? This is a little bit of a busy slide but if you focus on the blue column so if we look at CLSA overall the 51,338 you could see the age distribution of our unweighted sample but if you look at our old weights and our new weights it's fairly similar as you would expect because we're really not changing the way that we're calibrating that much in terms of age. So it's fairly similar in terms of age but where it really differs is when you start looking at education. So here we have education, the first row in education is post-secondary degree or diploma and then the last row is less than secondary school education graduation. So again if we look at that last three columns you could see that there's quite a difference between the old weights and the new weights and that the old weights are fairly similar to the unweighted analyses where the new weights are quite different. And you can see here where they differ the most. So here the light blue column is the unweighted estimate in terms of the proportion of people in each of the education categories. The darker blue is the old weights and the medium blue is the new weights. And you could see where the big differences is in the two extremes. So now when we're estimating the education in our target population it's looking a lot more like what we know it to be through our census data. So before there was fewer people the estimate would be in the lower levels of education and now it is much higher in the lower levels. So we have this SCF distribution that is more reflective of our target population. So what should a researcher expect in most cases the point estimates of prevalence or associations will be similar but it should better reflect the target population especially in the DCS catchment areas. We can also say that the underestimates of low SCF status will be lessened with the new weights and that parameter estimates for variables strongly associated with SCF are likely to be more affected. So it may be that if you are looking at associations that are not strongly associated with SCF you'll get very, very similar results. You'll get less similar results potentially for associations where things are very for estimates where that are very highly associated with SCF. But again the overall estimates will better reflect our target population. What you should not expect from our new weights although we kind of started this because people we were recognizing that at levels smaller than the geographic areas smaller than the province that are weighted data we're not necessarily reflecting the census data we cannot even using the population in a specific DCS area we will not provide estimates at the city level. Again the data collection sites are only at 11 locations across Canada and the catchment areas include that geographical region 20 to 50 kilometers around the data collection site. So it's not necessarily a specific city it depends on where your data collection site is located within the city. So we just need to be clear that we still cannot do city level estimates with the CLSA data. The data sorry the weights will be available soon it'll be this fall and they will be provided to people that are currently holding data for projects using CLSA you'll get an email and they will offer you the opportunity to get the new weights. For new projects if you've not yet received your data from CLSA the new weights will be the only weights that will be provided except under a special request. What's coming next in terms of the weights we know that this is something that's come up a fair bit now that people are thinking about using doing longitudinal analyses with the baseline and follow-up one. The baseline analytic weights can be used for longitudinal analyses using baseline and follow-up one but we will be creating new follow-up one inflation weights for estimates of descriptive parameters at follow-up one and these weights will be we need to create these new weights because we know that both the CLSA population is a bit different at follow-up one due to attrition in the sample but it also we are going to be using new census data that's going to reflect how the target population has changed between baseline and follow-up one and this is critical because often many of the parameters we may be interested in is looking at estimates of prevalence at follow-up one so we may be interested in looking at chronic conditions for example at baseline and follow-up one as well there's some modules in the CLSA that are unique to follow-up one so we didn't even have them at baseline so to estimate those descriptive parameters we'll need these new weights so we are working on those that's going to be the next thing in the queue and we will keep you in the loop as to when they will be available the other thing that will be available soon that I kind of alluded to at the beginning what is this new technical document that was led by doctor Henry so it's actually quite useful in that it provides a number of situations estimates for different types of parameters so for descriptive parameters for looking at relationships so looking at logistic regression estimating relative risks etc. using different statistical software so he's gone through how you would do this with the CLSA data including the weights and including the strata in RSS SPSS and STATA and shows you how you might get slightly different estimates depending on which statistical software that you use so I think I will end it there I'd like to acknowledge our funders and as well as doctor Wilson said at the beginning if there are questions our contact information is here in terms of data inquiries and general inquiries but I think I'd like to stop and take questions now thank you okay well thank you and I really do appreciate and acknowledge how much thought and how much work went into not only developing the new weights but thinking about how these needed to present to be presented to the approved applicants and the potential applicants for the data so I really really appreciate that I'm going to open it up I say I'm opening it up to questions but basically I'm going to read some of the questions that were put into the chat and copying and pasting them as they came in so I'll just start with the first one that came in and I'm just quoting here what if the sample of males and this is an early question that came in what if the sample of males are not representative of males studies show that lower the response rates the less likely they are to be representative of their group in your case males so weighting in this case may magnify their competitiveness so if you have any comment about that this refers to your specific example sure sure and that's I mean that's clearly a worry in any of the large studies now because many of the response rates are not as high as some of the studies by statistics Canada and even statistics Canada now is having slightly lower response rates than they used to that is one of the reasons why we created the new weights because we did acknowledge that while we looked at well let me take a step back we actually were we did want to see we cannot say that the CLSA is completely representative of the target population in Canada but what we can say is we've used a number of different sources including census data other health Canada surveys that had and sorry stats Canada surveys that had very high response rates and we had very similar estimates based on those compared to the studies that that using our weighted data compared to what we know are close estimates to what are what is going to be in our target population. What we were less good at was some of these SES factors so while we worked really hard to make sure that we had heterogeneity in SES so that we could still do estimate our associations it wasn't as representative so we are hoping in these new weights that actually is a bit more representative and that does seem to be the case just by comparing CLSA weighted data to these other sources. Okay thank you thank you so the second question we had is a little bit not so much about the weights but about the sampling so I'll just read it to you how would your sampling by a phone and in person affect the ability to sample other underrepresented groups i.e. the homeless or the transient. I mean that's it's a very good question and one of the criteria for inclusion criteria for CLSA is that people had to be able to respond in English or in French so there may be some populations even thinking about refugees or or other groups that will not be represented in the CLSA for that reason. I think in terms of the homeless and for some of these other groups I absolutely think that it's very challenging to try and include them in studies of this sort because you know there's in terms of the way that we are recruiting that is definitely a challenge and that me will likely be very underrepresented in CLSA. It's interesting as well in terms of it's a challenge to recruit people like this but they tend to be more mobile so it's also a challenge to keep people in the CLSA. Yeah I think going back to the planning of the study a number of decisions had to be made which does limit the overall generalizability of the results from the CLSA because there were practical implications about recruitment of certain individuals that would have required a whole new strategy but I think it's a very important thing to keep in mind when we're interpreting the results. No absolutely. So back to the weight a little bit so the next question is since inflation and analytic weights are proportional are there any concerns with using analytic weights instead of inflation weights for descriptive estimates of a population like sex, age, education categories for example we might not want the weight a number of males and females to be calibrated up to the Canadian population. I'm not sure I 100% understand that question. I think in terms of the estimates you should be able to get the estimates to the target population if you're thinking about in terms of estimates of variance the estimates of variance are actually relative to the sample size of the CLSA so it's not acting as if you have 15 million people in your sample but I don't know maybe if there could be a clarification I'm not sure if I answered that. Okay but maybe we can take this one offline and think about a little bit more and perhaps even get back to the person who asked the question. Next question so with these new weights how does this change affect the modeling results when an education variable is used as a covariate in the model? So it depends again I think on how strongly education is associated with your if you're just using it as a covariate in a model if it's not strongly associated with the association that you are estimating then it probably still won't have a big effect but we do recommend that you include education then in the models that you run and that's I should also mention that there will be a new technical document that is available when the new weights are made available we'll have a new technical document on the website that will go through some of these issues. Okay so the next question is a very practical question so it says I'm currently on a project using the Wave 1 I'm assumed that that's baseline being the first manuscript already accepted for a peer-reviewed journal and we're moving on to the second manuscript would you recommend to apply the new weights for the second manuscript also what would I do with the already accepted manuscript very practical question Yes and this is this is challenging for sure and we know that there's a number of manuscripts that have already been submitted and published and one of the things that we do I think each first of all for the question I think each of the researchers are going to have to decide what they're going to do in a situation like this what you can do is get the new weights and if it really doesn't change things much then you're good to go and you're fine if it does change things then you have to decide what is the most appropriate what is the most appropriate thing to report if it's already been published clearly you can't change that one thing that we do in CLSA that's part of the data sharing agreement is that in every in the acknowledgement it's clear what data set was used so at least we have a record of the data set that was used in to to create the analyses that were reported so it should be at least to if someone really wanted to look to see which weights were used you could clearly see which ones were included in the data set right right and I think also of course consult the statistician that you're working with about how to address this as well I think it's important no absolutely so one more question and I know the answer but I'm going to let you answer it Lauren is the technical report you mentioned in your last slide that compares estimates from different statistical software available on the CLSA website or is it only accessible to project holders in CLSA it will be available it will be not right yet but it will be available on the CLSA website because it actually will likely become a manuscript that will be much more widely distributed I think it's a very very useful document and goes through a number of different types of analyses I think most of which would be the way that people have approached using CLSA data are included in this document so I think it will be a very very useful document I think I will just add to this that at the CLSA we do our best to provide as many what we call data support documents as possible on the website because we know that often people who are considering using the data want to find out a little bit more about how things work before they make the application so we do try to get as much as possible up on the website so I don't see any more questions but I have a question that I sort of come across even with the old weights and the new weights so a number of approved projects and a lot of people who are looking at CLSA data are focusing on a particular subgroup and it's not always a subgroup that's defined by age or geography or by sex but sometimes it's a characteristic so for instance caregivers or immigrants or people with a particular condition and I've always struggled with whether if the analysis is limited to that subgroup whether we should or should not apply sampling weights or whether there's even a clear answer to that. No, that is a very good question and it's clearly a lot of people are interested in not necessarily looking at everyone in the CLSA but just a subgroup and that's actually the weights should still be used or at least that is our position and it's that type of analysis is actually covered in the new technical report that we'll be putting out so even examples of how to do it are going to be included in that report. Oh, well that's great so I will be looking eagerly for that as well but I think we're going to wrap it up now I want to thank Lauren of course for giving the presentation and the team as you acknowledged in the beginning who worked on this with you I want everyone to know that I did paste all of the questions so Lauren and I will be going through them and determining whether there are some nuggets in there that we want to make sure that they go into the frequently asked questions section on the website. I'd like to remind everyone because I'm sure there are some of you that are interested in accessing the data at some points the next deadline for applications is January the 27th, I'll say it again January the 27th 2021 and if you visit the CLSA website under data access you'll find what data will be available and additional information and as well the data support documents I just mentioned. I'd also like to remind you to complete the survey this is very very helpful for us and of course stay tuned to hear about the CLSA Webinar in November which is going to have an update of the COVID-19 studies and you'll get details to follow and you can check the website so thanks everyone again for attending and again thank you Lauren for the presentation.