 And colleagues today to hear about the latest research in COVID-19 dynamics from an epidemiological and demographic perspective. And we're delighted to have the World Pop team at the University of Southampton join us along with colleagues from Facebook as well as discussant from statistics South Africa. So what we would like to do today is to do two presentations and then have a short intervention from our discussant before turning it over to Q&A. I would ask please if you could put your questions in the chat box as we're going through at any time those will be fed through to me and then during the question and answer session we'll be able to get a bit of a discussion going about COVID-19 migration and mobility and in particular about the use of newest forms of mobility data to understand some of the emerging dynamics around the pandemic. So please let me just first pass over to the World Pop team I'll do a very very brief introduction so we don't take up too much time. The World Pop team at Southampton University is led by Professor Andrew Tatum he's a Professor of Spatial Demography and Epidemiology. He will be joined by a number of different presenters from the World Pop team that we're delighted to host today. Not all of them will be able to join us at the moment but I will let you know of all of the researchers who are involved in this particular piece of work. We have lecturer in infectious disease modeling Nick Runananchai. He will be joining us Shenje Lai who is a Senior Research Fellow at Southampton University working as part of the team. Alessandra Carioli also a Research Fellow at the Southampton University World Pop team. Corrine Rucananchai who is a Senior Research Assistant with the team and Jessica Floyd a postgraduate research student so we're delighted to have them join us and take us through the latest analysis that they have been undertaking using COVID-19 mobility data. We're also joined by Cecilia Zapala she is the Public Policy Manager in the Data for Good team at Facebook based in Brussels and we're delighted to have Cecilia presenting on Facebook's data capabilities and I'm sure we'll get into some interesting discussions a bit later on. And lastly we're really delighted to have Diego Iteraldi who is normally a participant and always poses the most interesting question so I'm really delighted that this time Diego has accepted the invitation to be a discussant. Diego is of course the Chief Director of Demography and Population Statistics at Statistics South Africa so thank you again Diego for accepting the invitation to join us. I'll hand over to Andy first he'll be sharing the PowerPoint presentation with the rest of the team members joining in throughout the presentation. Thanks Andy over to you. I think you're muted Andy, sorry. Thanks Marie, yes sorry thanks for joining us and thanks for the introduction. So yes I'm going to start off this presentation and then hand over to other members of the World Pop team who've been leading a lot of this research. So just to start off we are a applied research and implementation group, about 30 of us at Southampton University, a big focus on mapping population demographics, counts, characteristics and dynamics and trying to make everything as open as possible in terms of data peer reviewed methods and we have a lot of collaborations with UN agencies including IOM so it's great to be here to present this work and we're going to firstly a bit of background on pandemics and human mobility, how we measure mobility and then I'll hand over to Lai to talk about his work in China and then Nick to talk about his work in regional connectivity in Europe and then Alessandro will finish off on what's coming next. So what we're experiencing now is actually throughout human history nothing new. There's been some pandemics regularly throughout human history, some of them actually are bigger and far more deadly than what we're experiencing now but there are some components that are new and if we look back we think about a disease like smallpox and the speed it took to spread across the continent, we're looking at actually centuries to make that spread from endemic areas in India to China to Europe. We fast forward to the Black Death, that's then emerged in China around 1333 and actually took just a decade or more to spread across to Europe and the UK. Then we look back to 1918 and what was called the Spanish Flu but it's a major influenza pandemic. There we're looking at global spread in a matter of months so we start to see this acceleration of spread of pandemics and then finally looking here at H1N1 swine flu from back in 2009 starting in Mexico and spread across the world here we see in a matter of weeks and of course what we're experiencing now is a disease that arose in China and is now in all countries of the world and actually made it there in more like a matter of days and weeks so seeing this acceleration and question that I get all the time from my kids every day why why why why one component is changing in human mobility. So as malaria rheologists David Bradley decided to just do a little experiment or not even experiment asking his relatives to map out their entire life history so this is great grandfather who lived in Kettering mapped out his entire life history he didn't make it even to the neighboring town of Corby and his lifetime. His grandfather had access to steam trains public transport made it to to London and his lifetime. His father as the cheaper personal transport became becoming much more widely available and was able to travel across Europe all the way down to Corsica from the UK and in himself his travel pattern looks much like I'm sure many of us on the calls and global travel he's a malaria rheologist a lot of travel to Africa but that expansion in just a matter of four generations of mobility is is huge and one component that's driving this is the the global air network so air travel in 1933 this was the the network you could travel a few places and it took a long time about 10 days to go from London to Cape Town and a lot of stops and a lot of chances of crashing along the way nowadays we have a picture like this where we can pretty much go anywhere and get from one side of the world to the other in in 24 hours so how can we actually start to measure and map this growing mobility importantly there is what we call here traditional data sets and these remain vital household travel history surveys cross border and traffic surveys census data all providing information on different spatial scales and temporal frequencies and I'm going to talk about oh I'm sorry just these these have been vital data sets across the last century or so in terms of understanding drivers of migration mapping out flows and so we have some epidemiological work from Mansell prodo back in the 50s and 60s that did some really interesting work on understanding the migration patterns what we're going to talk about today are these other types of data set that are coming in really to complement those traditional data sources so they come from smartphone apps from personal gps from satellite nightlights and the major one that we're going to hear about most today is those coming from mobile phone smartphones and where these data sets come about firstly we're working with a mobile network operator and I'm a phone user and I make a call or receive a text at 948 in the morning that gets rooted through the nearest mobile tower and recorded anonymously for for billing purposes I then move to a different location that's and make a call receive a communication receive a text that's at 1048 that's rooted through a different tower so there's evidence that I've moved from one location to another location dropping these kind of digital breadcrumbs along the way I want to aggregate all these anonymous tracks of movement across an entire country we get firstly beautiful artwork like this this is millions of users across Bangladesh in three months just mapping out the the the most popular movement rates but it's also richness of data that we've never had before importantly these are biased data not everybody has a phone not everyone's using a phone regularly and there are geographical differences as well but in terms of richness of data that we've never had before and there's also now advances as we all move to smartphones and we hear a lot about this from Cecilia with Facebook data but as we we look at our phones and see traffic statistics of roads going red that's essentially smartphones slowing down smartphones with GPS in them and Nick who's on it on the going to speak next led a study where we got some volunteers to give over their their location histories which most of us will have on our phones and compare that to other mobility sources and show that it's a really detailed and valuable source of information importantly again we have to be careful with the biases and the sensitivities of these data hopefully we're here how they can be valuable in terms of modeling diseases and these are becoming more widely available across the entire world so again we'll hear from Cecilia later on this so how can we actually use these data for estimating disease movements and designing conformages so here as well I'll hand over to my colleague Lai who led some work on that yeah sense Andy yeah sense Andy um in the very early date of this pandemic um the warp up group has already um done some uh analysis on the spreading risk of the COVID-19 we're in China and beyond China by using the mobility mobility data and for example this one please um next slide yeah please um for example we uh we we call the data from Baidu the one of the biggest search engine Chinese search engine engine in China and also the the map location his uh have lots of map location history data and we use by those users locations to look at the connectivity between Wuhan and other cities in China and this video shows the incoming and outcoming flow of Wuhan and such as the yellow bubble means the outflow and the green column is the inflow and you can see that after Wuhan slowed down they have a very small number of people coming and leaving Wuhan okay next one please yeah and and we can see that based on this mobility data we can estimate estimate that a lot of people already travel leaving Wuhan before the Chinese New Year and such as this shows the the because the population movement pattern has a very significant signal phenomenon signal patterns in China due to the holidays such as the Chinese New Year and national days and you can see that the the the cities surrounding Wuhan have received more uh travelers from Wuhan and also some other big cities like Beijing, Shanghai and Guangzhou in China have a higher risk of uh imported this virus from Wuhan and yes we also use the L chopper data to look at the uh the risk of other countries or cities outside of China and you can see that the Southeast Asia and Europe and America looks that have a higher risk to have this virus imported from China yeah and this one please and and further we we use the multi patch epidemiological model to simulate the spread of this virus and this model is is a classical epidemiology local try to satisfy the population into the subgroups such as the susceptible population exposure population exposure to this virus and the population becomes infectious and then the population recovered or isolated to interrupt the transmission and use this model this model and combine with the morbid data we can we can define the how many people infected with this virus might be travel from one country to another country or from one city to another city and also use using this model we can we can estimate or to assess the uh the impact of different control barriers this one please uh use uh here's one example we're using this model and international population morbid data to estimate how many travelers in fact with this virus and potentially transmit the uh the virus between countries and we can see the two pigs here in the uh in the early stage of of this pandemic and the first pig is um might be the mod traveler from China and spread this virus and the second and bigger pig is uh is is this the virus uh might be exported from other continents so next one please yeah and uh because the it looks like the outbreak in China had been successfully contained in China uh increasing the uh travel registrations and the case isolations and early detection and and the third one is the social dispensation and contemporary distortion and we tried to use the model I mentioned before to and combine with the morbid data to um to assess the effectiveness of these three kinds of interventions next one please and here's just uh some preliminary result and the early stage of uh or the outbreak in China it shows it looks like the early case detection and isolation have a highest impact and followed by the uh intercity social distension and and contact reduction and it looks like the like the lockdown of cities in Wuhan especially in Wuhan and other cities in Hubei might not have a very high impact because I guess it might be happened in the and the late stage of the population uh movement before the Chinese New Year because already have a lot of people move out of the the city Wuhan or other cities and uh yeah and how can we um uh estimate the intervention and in other regions I would like to hand over to Nick to these sections thank you so you might be mute Nick is that better okay yeah okay yeah sorry um yeah so after the work in China um we wanted to explore what would happen based on different scenarios of mobility um in countries around the world so we use location history data from google um to supplant sort of the my due data in China which we had from around the world so next slide please um in this study essentially we wanted to see you know at this stage countries had put in stated lockdowns um they brought cases down and they were thinking about what would happen when they lifted those uh lockdowns and what might happen during future lockdown so we wanted to model the costs and benefits of international coordination and we did this using the same uh epidemiological model as what why presented but here we simulated two different types of scenarios so in the first case we looked at what happened if um countries synchronized they're lifting an implementation of future lockdowns at the same time versus countries doing it in an uncoordinated asynchronous way um so putting them in at the different times and the other uh scenario we tested was early lifting so what happened if certain countries lifted their um lockdowns before others we had three main pieces of information that we use to inform these hypothetical scenarios so we had the number of cases for throughout europe from the european cdc using those cases at early stages of the pandemic we estimated r0 for each country so you can see that on the right and then we also had movement over time on the bottom so here we jointly used some data from google the international country level data that we had and also some data from vodafone as well one really important thing that we could do with that google data and the vodafone data is we could infer how effective lockdowns actually were so what we did was we used the mobility patterns we saw in late march when all the lockdowns were in place across europe for the most part we compared that against january and february 2020 to infer how much movement actually reduced in other words trying to infer how effective their lockdowns were so one powerful thing i think about this method is that we didn't assume how power how strongly um lockdowns would actually influence mobility and exposure we had sort of this real world picture of how heterogeneous that manifestation of lockdown actually mattered across countries and you can see this here on the right where you see how much mobility reduced for example it moved reduced a lot by late march in italy and france less so in germany and poland so we use those four pieces of information to simulate what happened firstly when lockdowns were synchronized over time so these were on off lockdown so you can think about the uk where we came out of lockdown and now we went back into it over the course of december what happens if countries actually synchronized those across the entirety of europe and we found really huge differences so on the right you can see the cases across europe over time in blue if the uh lockdowns are actually synchronized and then in red you have if they're not synchronized and importantly if they are coordinated if they're synchronized 90 of our simulations went to zero sort of community transmission by the end of the simulation whereas only five percent of simulations went to that case if the lockdowns are asynchronous on the other end of it so we looked at the lifting so what happens if one country lifts their npi's early and what we found was that it had a really dramatic impact on continental resurgence so if certain countries lifted their lockdowns early it could cause resurgence across the entire continent up to five weeks early and this is obviously valuable time it could be used to expand test and treating strategies um as well as uh net vaccine rollout it's critical for that um there was a lot of variance in terms of which countries were most important so you can see on the right um the countries in red if those countries lifted their lockdowns early that led to a much earlier resurgence of the pandemic so sort of resurgence depended most upon what France Germany Italy UK and Poland did whereas it depended a little bit less on what say countries like Austria Switzerland um Norway Finland did so and just to show you what that movement looked like over time um this is the google data over the months so you can see now we're in March where movement dropped really dramatically across the entire continent and now you can see as you get into the summer there's actually um as lockdowns are lifted a recovery of a fair bit of that mobility um not just within country but also some international travel gets restored as well so essentially the take home message I think from this story is that we found really substantial variation um spatially and temporarily in terms of connectivity in terms of transmission in terms of r0 and in terms of lockdown effectiveness some countries when they place lockdowns it caused a much more dramatic decrease in mobility than others so because of that heterogeneous picture and because that means there's risk of spread to other countries what that depends on what each country does coordination and synchronization of strategies is really important to prevent resurgence um you know obviously we were sort of simulating different hypothetical scenarios and these predictions um are different realizations of what could have happened and they informed the difference of what happens when different countries synchronize rather when they don't um in reality we might find that coordination across the entire entirety of europe is infeasible but you know we can also look at this connectivity network to see if there's specifically especially highly connected sub-communities that form travel corridors where maybe certain countries could coordinate and you'll actually need every country in europe to work together so here on the right you can see some of that community detection analysis from our google data um and you can see countries in the same color essentially more strongly connected to others so it's more important that they coordinate rather than countries in different colors so for example um france spain and italy form one group so would make for one really natural uh coordination group within europe so that's it for me i'll pass on to alessandra he'll talk about what's next for our work thanks nick so um we've established that alessandra we're actually experiencing quite a lot of sorry aless alessandra alessandra sorry to interrupt but we're experiencing a lot of breakup i can i can take over i guess i think um uh adrian because i i just can't hear it's breaking up quite a bit andy that would be great sorry about that yeah that's fine so um yeah i think we're we're losing you alessandra um connectivity okay it sounds good now but okay i can um finish off um yeah some some next next steps i think on our on our work here is to focus a lot on the seasonal mobility and we saw that in the the start of this pandemic that coincided with chinese new year and this is the it's the biggest annual seasonal movement of people on the planet and and it's those kinds of movements that can really change the course of these outbreaks and pandemics substantially so we're looking a lot at these kinds of mobility data sets linking up multiple different data sets we're working with facebook in the the uk to look at these these changes seasonally um but also um as part of our wider world pop work are gathering together a range of different geospatial data sets on population vulnerabilities so this is some of the work that alessandra has led um modeling and assembling data on population age and sex structures we see here this is um a map of the uh the young and and whole also showing the much younger populations across sub-saharan africa than in europe but great sub-national variations as well in many countries that are affecting why some countries are seeing greater rates of hospitalization and death and also thinking about the delivery of vaccines and access to to health care to be able to to achieve that so um mapping out the locations of health facilities where particular vulnerable groups are in relation to those facilities and distribution mechanisms for that delivery of the vaccine in some of the poorest countries so i think we'll finish off here but to summarize that hopefully we've seen now that these disease outbreaks are becoming more rapid uh and rapid spread than ever before um and one of the key drivers is this change in mobility that we've seen over the last century of huge growth in reach and volumes um these new forms of data are aiding our abilities to map model and respond to the outbreaks they're not perfect there's biases and we need to validate and compare against more traditional data sets um but integrating them and accounting for the the geography and demographics is going to be vital to understand what's happening and what what happened in this pandemic i think we'll finish out and there's more information on our website great thank you very much um indeed andi and i'm sorry we lost you there alessandra um because your work on on demography was quite critical and i'm sure many people like including myself was thinking about the intersections with um some of the demographic profiles and what that means going forward so sorry that we uh that we dropped out there but we i'm sure we will get questions um in the q and a discussion and thank you very much andi for picking that up and for the whole team presenting a very rich um and and fast moving uh body of research and analysis that is is also looking forward to uh the next big issues especially in regards to vaccination roller and i'm sure we'll get into a discussion a bit later on that i will now hand over to Cecilia uh Cecilia as i mentioned earlier Cecilia Zappala is the public policy manager for data for good within the data for good team at facebook in brussels i'll hand over to Cecilia as she sets up her powerpoint presentation and we look forward to to hearing from you Cecilia and then we'll go on to Diego thanks over to you yes thank you so much Marie and the old iom team for inviting me today um my name is Cecilia Zappala i'm public policy manager in the data for good team at facebook and i'm gonna walk you through today through um some of our mobility data sets but before i just wanted to quickly introduce what the data for good team does at facebook and what's the history of that so this team was born around three years ago um under the consideration that facebook had uh quite some data that could have been helpful in disaster response so we really realized that um we could we were actually able to help some disaster response organization to maximize to to be more effective in the disaster response because we could detect uh where population was living uh how it was moving during a disaster and so on and so forth so everything started there and then as you can imagine in the last year um with covid the work really grew and focused very much on on on this pandemic um the team also grew a lot it's a cross functional team um there are some people in the public policy area like myself there are engineers data scientists economists geographies and and you name them so um i wanted to um give you just a very quick overview of the products we call them products of course they are mainly data sets that we have and that we share either publicly or with our partners so NGOs and academics to help them doing work in the humanitarian and societal space so the first family of data sets we have it's called maps for good and those are the data sets i'm going to talk about more more in details today then we have a second family which is called surveys for good and it's pretty much surveys uh that we um that we ask to facebook users of course within this um this family we always cooperate with universities so it's them really um processing the data collecting the data and providing analysis we did a bunch of surveys on covid um and we will continue to do so in the future and then the third family it's called insights for impact and with this um number of data sets we generally cooperate with uh um non-profit in order to monitor public costs around a specific issue one example uh was zika in brazil so we try to detect uh people's well facebook users feeling um around a certain issue to help those organizations to better target specific campaigns so vaccination would be another um another theme that we address uh with insights for impact um another aspect i wanted to mention is that some of our data sets are fully public so available to everyone who wants to see them we upload them um in the humanitarian data exchange or amazon web services data exchange and those would be uh typically the data set a little bit less privacy sensitive that use maybe data um which are not facebook data and we are gonna see that more in detail later on in the presentation uh while other uh data sets are under controlled access so as i said before uh we share them only with with our partner organizations who sign a data licensing agreement uh and we do that because those data sets are typically more privacy um uh sensitive and so this is a a better way to um to to to deal with this kind of data set and this approach of course we have it for the surveys and the insights for impact but i'm not going to go into the details so i'm i'm going to go through a number of examples in case studies now um i wanted to say that you can see all of them all the case studies that have been developed uh using our data sets are available on our website which is dataforgood.fb.com so i would really invite you to go there and uh and have a look if you are interested in in things that i'm gonna tell you if i'm gonna tell you today but let's start with the first example in the first data set which is called high resolution settlement layer what is what is it this is basically a very high resolution population density map and another thing i wanted to to say is that this is not um typically a mobility data set it's a population density map that shows where uh people live so you can see here um a little bit more of an of an explanation so what we do is that uh we start from satellite images which are publicly available and we apply to them our um machine learning our artificial intelligence to detect on those um satellite images where buildings are and this is a very similar exercise that that everyone could do looking outside the window of a plane and and and trying to detect where are human buildings and where are not to infer where population lives and so you can see an example here there's a small village in Namibia which has a few buildings which are made out of concrete others the majority actually are made out of more traditional material and the machine detects uh where those buildings are and therefore where population lives to this population um kind of location overlay data about sensors to see where specific kind of population live so you can see here an example of Nigeria that shows where the population over 60 years old mainly live and as you can imagine this has been quite helpful to a number of um of organizations to uh help COVID-19 support measures because it could really illustrate where vulnerable population was living. Let's go to the dataset which is movement range maps this dataset is also openly available to everyone who wants to consult it differently from the other one this dataset uses Facebook data and namely it uses location data of people who use Facebook on their smartphone as Andy was mentioning before and who have opted into sharing their location data with Facebook so this is an opt-in feature that we have and we developed this this maps to answer the question with COVID-19 whether people are moving more or less since the beginning of the pandemic and how are lockdown measures being effective are they really uh you know able to contain movement movement of people so you can see here how the dataset works so we detect mobility across specific area mobility of Facebook users as I mentioned before and we define mobility as traveling across level 16 being tiles this is a bit of a specific term it is not other than a square 0.6 kilometer per 0.6 kilometer and so we detect how people move across those those little squares so before the pandemic I could travel every day across maybe 10 or 15 or those tiles because I would go to work then I would go to have a drink with my friends then I would go to pick up my daughter at school so I would travel those number of tiles now with lockdown measures working from home I would probably travel one I would probably stay put in my tile or I would travel maybe two of them and so what we did is that we detected the movement of people across those tiles we aggregated the numbers and then we surfaced the numbers on a county level so what you can see in this chart is counties in the US of course we do that in in every country's globally almost and and so we would surface the data for the equivalent of county in another country it would be the province in Italy the department in France and so this really shows how mobility dropped after the the beginning of the pandemic and you can see also here another interesting visualization because we created this metric of percentage of population which in a specific time would basically stay put not move stay within their own tile and so you can see that at the beginning of the pandemic people were I mean there was a low percentage of people staying put with the spread of COVID-19 more and more people were actually not moving and then with the relaxing of measures a little bit later people started to move again so it's a pretty interesting position over time next data set it's the oh my god I'm sorry I went back so next data set is called the connectedness index so what this what this is is if you take the the the map on the right side we take we start from a specific area in this case is the province of Lodi in Italy which was the one of the most hardly hit by COVID and we actually assess how people in this province are connected with people in other regions in Italy and when we say connected we mean what are the number of of Facebook friendship ties from people in this province with people in other in other region and of course we can do it starting from any place in the world to any place in the world so you can see on the right map how people are sorry in the left map how people are connected to each other across Italy and this is very interesting because if you check them up on the right in fact you can see how COVID-19 cases was spreading you can see that this really mirrors the social connection ties within Italy so it's it kind of rejoins a little bit what Nick was mentioning before the level of connection really is a proxy to analyze and to foresee how the pandemic could spread in a specific region so I forgot to say that this data set social connectedness index is also available openly in our website on in the humanitarian data exchange now let's go to the data sets that are under license agreement so the first one is the facebook disease prevention map so which is a little bit more complex because it's cross magic and this once again responds to the to the need to detect whether people move during a disease and how do they move so once again we have we use as a as a basis a movement across styles so we use actually three sub data sets to create those disease prevention map the first one is the facebook population so in every tile we take a baseline that in this case would be before the the start of the COVID-19 pandemic and then we assess whether in a specific moment the population in is in an area is more or less compared to the baseline then we take a movement data set so for two pair of places we detect again we compute a baseline and then we detect whether people move more or less between those two points compared to the baseline and then the third data is connectivity connectivity which is less important in the in a COVID in a COVID space it's more important for instance for disaster response and what we understand from this disease prevention prevention maps and you can see that quite clearly in the map which is San Francisco during and after the beginning of the pandemic of COVID-19 is that the tiles in in blue are the tiles where there are less people sorry more people than compared to the baseline and the parts in red are parts where there are less people compared to the baseline so you can see that the area San Francisco the areas of the of the university Berkeley Stanford have less people compared to the baseline to be for the the the spread of the pandemic the beginning of the pandemic and then the vectors define this the movement of people as I said before so what you see is a change in movement compared to the baseline so you can see how inbound movement towards San Francisco was dramatically reduced during the first phases of the pandemic once again in order to create this these maps we use data of Facebook users that have enabled location information on their mobile phone of course this the disease prevention maps that been used in a range in a variety of ways and countries I wanted to just show very quickly an example from India that actually showed how after the the beginning of COVID-19 there was a population reduction in the cities so movement in towards the cities was was reducing but there were other trends of movement in other parts of the countries and of the country and this was helpful to actually foresee as well potential new COVID-19 hotspots if you are interested as I said you can go to our website and read the study next map is called collocation maps map and this tries to address the problem of whether what's the what's the probability that people from two geographic regions were in contact where a meaningful enough time to actually transmit the the coronavirus so these data and you can see an example from the university of Taiwan on the screen so these data take a pair of places it's in let's say Brussels and Geneva and then it kind of assess the probability that two random people in Brussels one in Brussels and the other one in Geneva can be in contact so within the same 600 meter per 600 meter tile I was mentioning before in a given time for five minutes over the period of one week so what's the probability of collocation between the two people one in Brussels and one in in Geneva and so this these data are then aggregated and once again surfaced at country level county level so you can see them for the you know the the administrative region and they provide information on how again how could the the spread of the pandemic be foreseen because population in a country that is particularly are in contact with population in another region that could be either in the same space or or or much broader much farther and then finally I wanted to touch upon the travel patterns data set and in this case the question was what are the movement patterns between countries compared to the pre pre-covid levels so you can see here the travel patterns that basically shows comparisons of the number of Facebook users that move large distances so across countries by plane or by train and so this comparison again is based on people who are using Facebook on their mobile phone we have shared location information and so for a given time we can see how people travel across the national boundaries and so this is also useful to to kind of help epidemiologists to find regions which are at bigger streets of exposure of COVID-19 because there are people traveling from regions that actually have a higher rate of virus infection so I can stop here for the time being as I said please go and check our website if you're interested and also if you have very specific questions please email me there's my email on the screen and thank you so much thank you very much indeed Cecilia that was a very very very rich and very interesting presentation of course I'm sure there'll be lots of questions on that and and certainly we have a fantastic discussant too to lead into the Q&A you mentioned a lot during your presentation the collaborations with academics and with NGOs and of course with the humanitarian data exchange particularly around crisis events and so forth but we thought it would be interesting to bring in Diego because Diego is of course as I mentioned the chief director of demography and population statistics at statistics South Africa so from a state perspective and from a government perspective it is an interesting sort of dynamic in regards to the mobility and other data that is being accumulated by Facebook by Google as mentioned earlier in terms of supporting member states and their responses to something like COVID-19 so with that let me hand over to Diego to offer his remarks on both presentations and then we'll head into the Q&A just a reminder too for participants please feel free to put your questions comments in the chats and then we can go into the Q&A after Diego. Thanks Diego. Thank you very much Marie and thank you very much to the IOM research for giving me the opportunity to be part of this webinar it actually brings together two of the things that drive me the most and which attract most of my interest and that obviously being COVID and the issue of mobility or migration of people over time. I'd like to first of all start off by thanking the presenters both from the WorldPop program as well as Cecilia from Data for Good. Besides being very interesting presentations I think I learned a lot about what the potential opportunities are with regards to this kind of data. I think we are living in a time now where the use of shall I call them alternative or new data sources particularly around mobility in tracking COVID-19 indicators can no longer be sidelined. It's no longer a site activity that a niche group of academics do but it really is something that is becoming very much mainstream. The use of mobile devices or the use of tools like Facebook advertising are sources that need a lot more exploration for us to glean more knowledge with regards to what it what it can tell us not only with regards to COVID-19 but with regards to various dynamics that are in in our society. It's also what I think is really important and I think that Andy touched on this is that it's critical to integrate traditional sources with these new data sources. We are not saying that we want to replace one source of data with the other but we really want to integrate what new technology is able to offer us with what we already have. A very simple example if we take aerial photography with remote sensing we can tell almost in real time where new communities are developing but we won't know anything about the people who are living in them and that's where sensors and surveys and other forms of data collection come in handy so that we know what the characteristics of those people are and we don't leave them behind as we move to our 2030 goals of the SDGs. I think it's also important to look at relationships between geolocations in order to be able to build any predictive element into any index or model so in other words what are the key roots between big cities and major or smaller towns what is the impact of roads or airports or harbours and how does that impact the movement of people. Of course many nations around the world have been through lockdowns where this movement has been curtailed and that kind of movement has really come to a full stop so I think that in terms of COVID-19 how the movement has taken place after lockdowns have been relaxed or lifted and how that impacts on the spread of the pandemic is an important issue to take into consideration. I think one thing that we probably haven't spoken about is that it's important to discern between migration and mobility whereas the former is a well-defined demographic phenomenon which I've been part of with the UN expert group on international migration statistics in defining migration and a whole bunch of other related terms. Mobility is a much broader concept but far more relevant towards the handling and the management of COVID-19 so as much as mobile devices are able to indicate to us how people are moving or where they're moving to or from which we should not confuse this with the actual act of migration which has got a separate element attached to it. What's noticeable is the relationship between mobility and the spread of the pandemic in many ways. I think that the two presentations today have indicated various ways in which movement of people have resulted in the spread of the disease over time from its origins, from its genesis, but also as resurgence are occurring and we can't really avoid or ignore the issue of mobility in this regard. One thing that I feel very strongly about though is capacity building around these new sources is really important. I don't think that it's enough for us to say what is available and what we can do with various tools but in order to get various countries around the world looking at this data, using data and maximizing on it, I think it's necessary to capacitate countries around the world on how to explore such tools, how do they access them, what does it tell you, what are the limitations and how can governments around the world make use of that? We have seen, I think Nick was speaking about the impact of lockdowns in Europe, but many countries at this point in time are going through second waves or the beginning or right in the middle of second waves which are anachronists and are not occurring simultaneously or not occurring at the same time and that's exactly what we see in South Africa where two of the provinces are finding that they are experiencing the beginning stages of this second wave and using this type of mobility data and looking at other epidemiological indicators I think is the way to go in trying to manage this second wave without going into a hard lockdown which I don't think anybody wants or that we can afford after a winter with the hard lockdown that we had. This kind of leads me into the experience of South Africa and very briefly I mean the outcome of COVID-19 in South Africa has been not as bad as what we thought it may be in March. I think that the young age structure which has been alluded to in sub-Saharan Africa has been partly responsible for that. Urban centers where there is a far greater diversity of age structures have probably been hit harder not only because of the age structure but because of the bigger population and intermenting between people in larger cities. The government of South Africa set up what was called a national coronavirus command center and feeding into this command center was a it was called the NAT joints the national committees which advised this command center. Various work streams were identified in this NAT joints and one of these work streams was the data and statistics work stream which statistics South Africa and other partners were involved in. So in this work stream we looked at issues around mobile data actuarial models, epidemiological models, pollution maps from weather services and any other source of data which would look at the three main components of COVID-19 as being the health impact, the social impact and the economic impact including issues now of the of the resurgence. Speaking of the economic impact statistics South Africa yesterday just released quarterly GDP figures just to give you an indication of the impact of the lockdown. In quarter two there was a 16.6% drop in GDP and in the third quarter once lockdowns were relaxed a 13.5% recovery was was experienced. These are all quarter on quarter seasonally adjusted and analyzed indicators. Then whilst many sources such as the ones showcased today need to be explored I think there are key indicators have been identified to measure the impact COVID-19 has had on South African society and a dashboard to this extent has been developed which is being shared on Google Data Studio. But I think that there's still a lot of the type of of work and the type of indicators that we've seen today in this webinar which need to feed into this particular dashboard. In this dashboard we also include data on educational institutions as well as a municipal barometer to see what the impact has been at municipal levels particularly economic impacts in that regard. The in the case of well there's there's also something to consider is the case of mobility before during and after lockdown in in in South Africa and that when lockdown was introduced on the 26th of March we found that leading up to the 26th of March many people moved to where their family homes were away from the big cities and would spend the first what what was thought will just be three weeks of lockdown but which ended up being a lot longer movement into smaller more remote villages where their family homes are and as it was realized that lockdown is going to be a lot longer than three weeks a window of opportunity was given for people to return and this kind of movement certainly had an impact as we were moving towards the peak. There's also this of the culture of funerals in South Africa to take into account funerals are a big cultural and social event with large groups of people usually moving from from region to region to attend funerals. Funerals were limited to only 50 people during lockdown and the so-called after-tears reception which is usually a social reception which takes place after the funeral also needed to be managed because they were seen as epicenters of of superspreader events if I can call it that. Just this week the president of the republic has announced that no after-tears parties or receptions after funerals will will be permitted in certain regions where the resurgence appears to be stronger. Then just two final comments in closing is that it is evident that the demands for data are more and that they're needed in real-time particularly to manage a health crisis such as COVID-19 and to do this we need to invest in infrastructure of new data sources and in their application and interpretation as well and the skills that are required in order to take us forward. I think also in this regard that resources such as the data innovation directory on the migration data portal hosted by IOM Jindak are an invaluable resource in this regard. So thank you very much to the presenters. Apologies if I overstretched my five minutes and thank you very much to you Marie for this opportunity. Thank you very much indeed Diego and great to bring in some of the experiences of South Africa more recently and including in relation to the public health but also the economic kind of impacts that we're certainly looking at going forward. We've got a few questions in the chat. I also have a couple of questions of my own really kind of to start the conversation. We often hear about some of the risks and some of the real challenges in terms of the new data sources that we certainly have emerged in the last decade as being very powerful forces both from a public policy perspective but also from a privacy perspective because they raise a whole range of new issues that haven't really existed with population survey, census data and other academic type of outputs related to behavioral economics for example and decision making in terms of movements. What we have been talking about today a lot is looking at monitoring, trying to assess behavioral impacts and the the nexus with public policy determinations. What are we talking about in terms of responses, their effectiveness in regards to this particular pandemic but also as pointed to by Andy and the World Pop team in terms of vaccination sort of rollouts. So a couple of kind of key issues keep coming up in that regard and I'm very interested to hear from the World Pop team and Cecilia especially in regards to privacy. We're hearing more and more over time of the so-called gold standard of privacy being differential privacy and so I'd like to hear about their experiences with differential privacy both from a kind of a policy perspective but also from a methodological perspective and what sort of limitations that puts on kind of some of the data analysis and in terms of integrating it with more traditional if I can call it that more traditional underlying data sets around population. There's also of course the aspect that has come up again and again throughout all of the presentations including Diego's remarks and intervention at the end in relation to biases within the data. It was interesting to hear Cecilia talk about the Facebook population. You know we're so used to talking about you know the population of a country or a subpopulation within a particular community or region geographic region but what we've started to talk about is the Facebook population which of course is global but it's highly segmented and of course when we're talking about something like COVID-19 which as Diego and all of the presenters pointed to in terms of having very significant impacts on particular age demographics. There's also a lot of research to talk about you know minorities including migrants within those populations but we know for example that age is a really big factor in terms of impact. What that might look like going forward as we move into other coronaviruses unfortunately this is our kind of new reality there's a lot of research on on the fact that they're they're increasing the potential for big coronaviruses going forward is not going to abate and we've seen in previous pandemics that there has been a very very different impact in regards to age population groups you know the the so-called Spanish flu in 1918 had a very different impact in terms of age and other demographic variables. So with those I might hand over to the world pop team I'm not sure if Andy if you wanted to take those we'll pass those off into other members of your team around privacy differential privacy in particular and and how that works and also then in terms of some of the challenges and risks that come with a very biased kind of data set and a view of the world that is changing dramatically because of the data that we have at our fingertips. So over to you first Andy and then we'll then we'll go through the other presenters. So I can I can start off I guess so yeah a lot a lot a lot to cover there but firstly I think I would say a lot of the sensitivities with these kinds of data sets come from it's a lot of analysis that are looking at more individual level movements and very small area patterns I think firstly it's when using these data sets it's under it's an understanding of what is actually going to be useful to end users and policy decisions and often that's that said and at quite an aggregate level there is actually no need to go down to individual levels in tackling some of the challenges that are being faced by policy makers and also understanding the dynamics of this pandemic and and that then involves very aggregate population level flows with like traffic statistics where these concerns about privacy become I think a lot less than if we are dealing with individual level data and also yeah just to I think to echo what Diego said that these these data are biased and on their own they are perhaps not not so useful it's when the power comes in the integration these can add more more spatial detail more temporal frequency these kinds of new data sets but only really when we understand those biases and integrate them even in an epitome logical model to understand the try and get a handle on flows of infections or with ideally with household surveys on phone ownership with other forms of mobility data sets and when we're talking about the Facebook population there's also the Google population and then there's the Vodafone population there's and then there's household surveys asking about phone ownership and then so the more that we can bring together and the more they actually although we won't get we probably won't get a definitive answer the more these data sets all line up and show similar patterns gives us confidence what we're seeing is not not so biased and the more we can also understand what are we missing and where are the uncertainties and that can be very important if we are talking about a disease that affects the elderly and we're actually looking at movement patterns of the younger population which in some policy relevant questions can be still useful in terms of the spread but in other cases in terms of vulnerabilities we may be completely missing and providing advice that's actually misleading so yeah I think it's as important here for for being as open as possible about these biases any kind of analysis is done and trying to look together different data sets to understand those uncertainties um yeah I don't know if anyone else from our team or I can send over to to see if there's more technical insights on the differential privacy and things like that yeah if I can add just a little remark on the demographic concept Celia you're breaking up again I think we're losing you again yeah maybe you can you can type in the chat and it could be a solution in some of the books or turn off your camera because that when you turned your camera off we were able to hear you very clearly so yeah try again Alessandra doesn't seem like it Adrian yes we can hear you with your cameras off thanks Alessandra go ahead you're on mute yeah but I don't know is it working now yeah never mind yes can you hear me oh okay I will just add very briefly that the direct consequences of COVID is the death of males of over 70 years old but of course there's a direct consequence of especially in developing countries of not providing health care to those that are most vulnerable for instance if we think about pregnant women's antenatal and natal care for deliveries vaccination campaigns have been put to an old HIV and all these sorts of health issues that have been put aside to face for the COVID emergency that was just luckily I'll be on mute again thanks Alessandra I'm glad we could get you your audio at least I'll hand over to Celia now please Celia over to you yes so um first thing and privacy so just as a caveat we have actually an academic paper that goes very much in detail on our differential privacy approach so I'm not going to go too much in detail today and I would encourage everyone to either find this paper it's available on our website or email me if you want more details that said obviously privacy it's at the core of of what we do of our data sharing we always I mean the slogan of our team is really make available our data set in a privacy-producing way it's super super important and I think it what I can say is that our privacy approach is really on different layer so the first layer is this different level of sharing different approaches regarding sharing so as I said before some data sets are available publicly updated on human humanitarian exchange other data sets are available only to our partners of obviously to minimize privacy risks then there is a second layer which is a focus to users consent so as I said before most of our mobility data sets actually all our mobility data sets are based on people who actually have consent and so actively went into their location settings and Facebook and say yes I want to have location settings on so they are also informed about the you know the implication is to to enable location and we assumed that they're absolutely agreeing with the fact of using the location data for scientific research purposes and then there is a third layer which is the kind of more technical layer in a way so there are techniques that we use that we apply to the data sets to make sure that they are I mean the privacy is protected so there can be less sophisticated technique like aggregation as Andy was mentioning before it's not I mean for the researchers the NGO it's not important to know who exactly is in which location in which time aggregated data are sufficient to build modeling so aggregation is definitely something that we do in all data sets we also add a small amount of random noise to to the tiles to make the true counts really impossible we use another technique which is called smoothing which basically is a weighted average of the different tiles between tiles in a space and the the immediate detail which is immediately next to it we also drop the the tiles which have two small counts and that could you know entail some privacy risk for the user so in the different data sets we drop specific pounds when the counts is not high enough we totally drop it so all of this to say and obviously differential privacy so all of this to say that privacy it's really at the core and and it's something that really drives our activities since the beginning of of of data for good of the building of the team just a quick word on the bias and I wanted to very much subscribe to what Andy said our ambition is not to drive to kind of influence a specific decision or a modeling with only our data we are very conscious that our data are limited because they reflect the facebook population which is especially in certain areas it's definitely not the majority of the population it's not the particularly vulnerable population in some areas so it's very important to be extremely mindful of that and to combine as much as possible our data with other data sources to create a less bias model but but just to say that that for us it's absolutely obvious that there are bias obvious bias in in the data that we provide great thank you very much to see there I just wanted to see if Diego wanted to add anything otherwise I've got further questions in the chat that have come through yeah I can just add that obviously confidentiality for a national statistics office is is critically important all of the data that a statistics office releases needs to take that into account and we would not release from any of our census or surveys information that would enable somebody to be identified in this regard as as we were navigating through how to manage the coronavirus pandemic in South Africa there were a lot of calls from the public indicating that they were weary of being followed or being tracked by by the state and and that they were not that there was a high level of reluctance in this regard so the the national department of health put together a free app which provided your bluetooth and your mobility settings on your smart device are switched on it it would actually indicate using bluetooth whether one bluetooth user was in close proximity with another bluetooth user who had registered and and and had indicated that they had tested positive for for COVID-19 so in in this way data is collected without any personal identifier without finding out where you live or where you work or what your movements are but simply by identifying that one person with the bluetooth switched on was in contact with another one where one of the two had had been had been positive so I think that this this this has been a very successful innovation in in terms of being an additional tool to break the the chain of the of the virus amongst communities and that's a really good point Diego because we have seen that there have been say some misinformation or disinformation campaigns about you know the collection of data about even vaccines coming through now as we are as we are seeing you know some kind of quite outrageous conspiracy theories being put forward in different on different platforms and through different users and so forth and it leads to questions related to the aspects in concerning data for good and public policy kind of interconnections but also messaging for good and I'd be particularly interested particularly Cecilia from your sort of perspective and and you know moving a little bit away from the epidemiology and demography kind of aspects but more into the public policy space what's Facebook's experience in regards to say if I can characterise it as messaging for good and countering some of the misinformation in relation to you know migration and migrants we've got an interesting question in there about vulnerabilities which I think we've talked about in the previous kind of round of of responses but we're very conscious of the impact increasing impact of these kind of conspiracy theories and the pains that as Diego has pointed out that some governments will go to to reassure populations but nevertheless some of this misinformation and disinformation does persist thanks this is actually a great question that I I'm not the best person to be able to answer because as you as you probably imagine we have a big amount of people in the company who work specifically on misinformation on vaccines on ads and so on and so forth so what I can say from a very kind of a general perspective is that of course we consider in within the data for good team we consider our activity as part of the overall broader COVID-19 response set of instruments so we did have a put in place specific policies on misinformation on COVID and on vaccines actually there was this week or last week there was actually an announcement on on some changes in our misinformation policies specifically on vaccines we do use ads to help governments to target specific users on campaigns again around COVID-19 misinformation vaccines we have a number of product we have a COVID-19 info center so we kind of provide those quick screen you know prompt on the screen to tell people you know COVID is this and that but you know drinking I don't know it's called in English but anyway drinking whatever it's not going to be helpful to to prevent the spread of the disease to to protect you from from having COVID so all of these measures are put in place of course we can always do more we can always do better but again we we kind of we didn't shed away from knowing that we do have a responsibility on on misinformation around COVID-19 vaccines. Great thank you very much indeed and and you quite right I mean IOM is like a large organization as well so I'm sorry to put you in that spot in terms of misinformation if it's not your area but that is a really big and growing concern of course being highlighted you know very clearly by the pandemic and certainly something that partnerships going forward need to be working on much more closely across a number of different sectors I've got a really interesting follow-up to some of the discussion in regards to kind of digital literacy for example and in the context of platforms and apps and different providers having permission or requiring permission say for the geolocation opt-in type of location history features and what this means in different communities where there might not be high levels of digital literacy and understanding the implications of that particularly for some of our vulnerable groups who may be beneficiaries in humanitarian crises in pandemics for example and I was particularly interested in finding out really from both from Andy from a research perspective and a scholarly perspective his thoughts on that but also of course from Cecilia as well in terms of the the bigger issue of digital literacy and informed consent and really understanding those permissions thanks yeah it's a it's a big question and it's one that has been evolving for sure over over time we've started working with mobile operator data back in 2007 I think then there was a situation where and the malaria control program could simply go to the offices of the the mobile provider and come out with a hard disk and and there wasn't there were no protocols in place there was no consideration of these these kind of challenges so thankfully things have evolved a lot over time now there are strong recommendations about how these kinds of data should should be used to overcome some what you've mentioned particularly in terms of stronger much stronger data protection laws being put in place across many countries that have I mean they've caused us many challenges in terms of projects that have had to be restarted two or three times because governments have adopted strong data protection laws and that's a that's a good thing it's a frustration sometimes for people trying to work with the data but it's I think it's great that these protections are coming in now we're in a situation where data actually never and when we're working with a mobile operator we're not and we're not touching data at all it never leaves the operator and there are much more I guess protections put in place and it's only again that that aggregate and anonymized data that ever leaves essentially broad strapped traffic statistics where no individuals can be identified but yeah these these problems still do exist and I think this is a active research area of ensuring that the most vulnerable are are protected and that they're aware of that and these kinds of data are being analyzed all the time in commercial for commercial purposes and have you sign up for it for a contract or users data that's there's the small printers ahead that these these data are being used by the operators for for commercial purposes for understanding their customer database and often that's not not well understood so it's something that does need to be addressed for sure how indeed thank you very much Andy and that's I mean that I think that's really important especially to see the changes over time that you've that you've spoken about and how things really have shifted and will shift I think as you've pointed to even further Cecilia I'd love to get your perspective on this too thanks yeah and I think it's a great question and I wish I had an answer which I unfortunately don't have so I can I can actually take my casket so before I joined the data frequent team I was working on privacy for a number of years in Facebook and in fact it is an issue it is an issue trying to inform Facebook users about what it means to select certain settings to share the data or not share the data at Facebook again there are a number of teams that work in on digital literacy from a privacy perspective from a safety perspective so again not just on data for good now specifically on data for good obviously digital literacy can be an issue as connectivity can also be an issue so once again I go back to to my point on on data bias we know that we can't reach all the population probably the most vulnerable population it's difficult to reach probably the most vulnerable population it's not able to kind of understand what are the implications of specific choices and all of these make our activity of course not 100 percent perfect but as I said before we do with what we have we try to work on different sides on the misinformation side and the digital literacy side on the connectivity side on the on the kind of on on many sides in order to make things better and try to do our work as as best as we can but yeah I wish I wish this was not a problem but it is actually great thank you very much and and I think it is it is kind of increasingly recognized as a real challenge we often talk about capacity building for data and often we're talking about you know the people who collect the organizations who analyze data but really it's also in terms of the people who provide data and the big digital digital literacy issue is a very significant one especially for populations who who may be very vulnerable because of their circumstances in particular. Diego I'm sure you've got I'll give you the last word we are a little bit over time but I know that this will be something that will be particularly close to your heart and part of your kind of role so give you the last word and then I'll really just ask for final comments if anything that you wanted to kind of offer into the discussion or that has come to mind I'll I'll let the final round of present presenters offer their final remarks so Diego over to you. Thank you well I think that misinformation has a huge role to play and a huge reason why digital literacy should be something that is in cons there's often a lot of misinformation about what is this data being used for who is checking up on me the reason for for having to to share this kind of information and I think it's it's it's it has the potential to be a stumbling block to the exploration of this kind of information so I think that communication is really really key and assuring people that there is there is no individual level surveillance which is ongoing is really important for for us to to be able to develop this and to capitalize on this more. One thing which I also wanted to add on was the issue of remittances I think remittances is one aspect of migration which COVID-19 has impacted the most I think the World World Bank had indicated that remittances are likely to drop by 20% through lockdowns and I know that there were other webinars that dealt with this but from a demographic perspective I think that one of the the main ramifications of lockdown and of COVID-19 has been around the slowdown in remittances which which has ultimately picked up we we have already spoken about reproductive health services and services to pregnant women which had been highlighted and of course from a mortality point of view the whole concept of excess deaths has certainly highlighted what what the chasm is between reported deaths and actual deaths so I think that literature is is awash with a lot of information I think that social media although it although it has its downside has has most certainly brought the the findings and the research and the knowledge being generated over this this year much closer to home to many of us this kind of webinar that we that we are having right now we probably have probably have been a face-to-face meeting over over some other theme other than COVID-19 had there not been COVID-19 but it has brought the expertise of people around the world into our own living rooms on a weekly basis I don't think there's been a week over the last five six months that I've not been logged into a webinar of one sort or the other and although you say that I ask a lot of questions I sometimes wonder if I'm not the nosy guy who asks too many questions so yeah I think that this this whole experience this morning has been very much eye-opening I think there's still a lot for us to learn a lot for us to try and implement and once again I just want to thank everyone involved for for the opportunities to be part of this webinar this morning great thank you very much Diego and I was really referring to to quality not quantity of your questions you always ask good questions so thank you very much indeed any final kind of thoughts from Andy or Cecilia no just thank you very much and yeah great discussions thank you very much Cecilia anything from you no I thank you as well and I really enjoyed this this chat this morning yeah thank you so much and thank you for the excellent presentations and for Diego's intervention as well it is the final in our series for the year but we will be resuming next year and I think Diego is exactly right it has been a bit of a silver kind of lining to a very dark cloud in regards to pandemic the ability to bring people together from various parts of the world with different perspectives I think has only been enriching and we've certainly been heavily involved in not just hosting but also participating in some and I would encourage people who are interested in remittances as Diego brought in this is something that IOM is very much focused on and we do a series of analytical snapshots to try and make sense in very short spaces of time some of the new emerging findings research and data and certainly as it comes to remittances we are seeing a very substantial shift and the world bank's revision of its projections of decline of international remittances is a case in point going from 20% to 14% a few weeks ago as we see some of the data emerging from central banks showing that informal ruins channels are being formalized out of desperate need you know we're no longer seeing people traveling across borders and taking money back to family and friends and extended families and so forth that they are having to to move into the digital world and I think this is one really big unifying kind of aspect in regards to COVID-19 it has squarely placed us into a much more significant digital world going forward whether that relates to social care whether that relates to epidemiology whether that relates to mobility whether that relates to remittances and also how we communicate socially we're heading into a very significant digital age with some of the challenges and the risks that we talked about this morning so thank you very much for a very engaging discussion thanks to the participants special thanks to the IOM research team especially Adrian, Josiane and Celine who've helped put together the webinar today but also the webinar series over the last six months or so we look forward to engaging further on the topic this is not the end it's only in really many regards the beginning and we're wishing everybody a very safe and happy new year and we're all I think looking forward to saying goodbye to 2020 in many respects so thank you again and thanks for joining us