 Fantastic. Michelle, have you run into the head of psychiatry there, Laura Roberts? No, I certainly heard that name, but I don't think I've met her. She was a graduate of the University of Chicago and is on the board of the Bucksbaum Institute. Yeah, if you go to meet her, give him my regards. Okay, I'm sure I'll run into her at some point, but even after six years, I haven't gotten around to the major units of the med school yet, you know how it is. Oh my gosh, you've been there. You left Harvard in 2014. Yes, I did. Right. And you went back to where you did your undergraduate work. Yeah, a lot of people do. You know, it's funny, we find our way back to California eventually, even though I know it seems like a hellscape from what you read in the papers. It's actually a wonderful place to live. I love it. We're going to do medical regular time, which is usually three or four units after the hour. Yes, I've noticed that that has persisted even without the sandwiches that normally explain people slightly. And maybe the lack of sandwiches has actually worsened the problem. That was maybe the one incentive to get there at times if you under order the sandwiches. I think I will start. Good afternoon. My name is Mark Siegler and I just wanted to welcome you to the lecture series on ethics and the COVID-19 pandemic, medical, social and political issues. Today's lecture will be the fourth in the series of 27 lectures, all on Wednesdays from roughly from noon to one with a few Wednesdays off around Christmas time. Let me introduce our fabulous speaker from today, Michelle Mello. Professor Mello is Professor of Law at Stanford Law School and also the Professor of Medicine at the Center for Health Policy and Primary Care and Outcomes Research in the Department of Medicine at Stanford. Professor Mello conducts empirical research into issues at the intersection of law, ethics and health policy. She's the author of more than 200 articles on a wide range of topics, medical liability, public health law, pharmaceuticals and vaccines, biomedical research, ethics and governance. And her work on COVID-19 as published in New England Journal and JAMA, Annals and Science Magazine has been extraordinary over the last eight or nine months. The recipient of many awards, Professor Mello was elected to the National Academy of Medicine at a very early age. From 2000 to 2014, she was a professor at the Harvard School of Public Health, where she directed the school's program in law and public health. She then returned to Stanford where she had done her undergraduate degree. Dr. Mello holds a JD from the Yale Law School, a PhD in health policy and administration from the University of North Carolina, and an M. Phil from Oxford where she was a Marshall scholar. During the COVID-19 pandemic, as I've said, Professor Mello has written papers on a wide range of issues relating to COVID-19. Triage standards of care, federalism in the U.S. response, stay at home orders, and interstate travel restrictions, changes on how scientific research is being done, and even on school reopenings. Today at our lecture today, Professor Mello's talk is entitled Ethics and Legality of Disease Control and Surveillance Measures. Now let me say a few words about the specifics of the talk. Professor Mello would like to organize her talk in the following way. She would like to speak for 15 minutes on the first topic, ensuring the uptake of COVID vaccines, and then answer about 10 minutes of questions from the faculty and from the attendees at the meeting. Second, she'll speak for 15 minutes on the topic research practices during COVID, especially the difference between non-randomized trials and randomized controlled trials. And again, we'll open it to 10 minutes of questions following that. And finally, Professor Mello will speak for 15 minutes on digital contact tracing apps and also have questions following the third part of her talk. I ask you to please list your questions and submit them in the section of the Zoom entitled Q&A rather than in the section entitled SHAPT. So in Q&A, the questions will be directed by Dr. Brian Callender, and we encourage all of you to feel free to send questions on any of the three parts of the lecture. Please join me in warmly welcoming Professor Michelle Mello. Michelle. Thank you so much, and I want to thank you in advance for tolerating the somewhat unusual format of the talk today. As Dr. Siegler mentioned, my work has been all over the place for the last few months. So rather than having one deep dive to present, I have three shallow dives that I'm hoping will provoke some good discussion. I've also learned in the last seven months that the number of people who want to hear me talk on Zoom for 45 minutes at a stretch is zero. So I hope this will be a little bit more engaging. I also want to begin with an apology because, as it turns out, this talk coincides with my fourth grade son's music class, which is going on in the next room. There needs to be a special new German word invented for the hell of trying to work at home when a nine-year-old is learning a new instrument. I'm very sorry that you have to be a part of it, and I hopefully it won't be too disruptive. Let me begin with a disclosure, which is that although I'm not going to talk about this product, I've been an advisor to Verily, which is the health arm of Google on a product that relates to COVID return to work. It's not a digital contact tracing app, but I did want to disclose that. So as Dr. Siegler mentioned, I was going to talk about three topics. I'm going to tackle them in a slightly different order than he mentioned, and I'm really hoping to get some good discussion going. I know we're hamstrung a little bit by the format here, but I'll offer a couple of discussion questions relating to each, and of course, I'm happy to answer any other questions that you may have about each of them. Let me begin with the first topic, which I've been working on since very early days in the pandemic, and that is what we do about the fact that it appears that manual contact tracing is simply not able to be successful as a strategy in this pandemic, and there's now a lot of companies stepping forward with digital alternatives. So I want to talk about some concerns that I raised in the paper published in Science earlier this year with Jason Wang, who has been instrumental in helping the Taiwanese get their digital contact tracing approach off the ground, and that's an interesting experience. What we're talking about when I say digital contact tracing is the use of smartphone apps and low-energy Bluetooth technology to track people's geolocation, and in some manner to notify somebody when there is a diagnosed case of COVID that comes into contact with a phone that is not a diagnosed case of COVID. There are two versions of this technology. The centralized version, and the version that most people probably think about, has a centralized data storage system such that when an infected person reports his or her status to the app, their anonymized ID, it's a phone ID, it's not their name, and all the key codes that the phone has received from people nearby are uploaded to a centralized database, and at that point the anonymized data are linked with other data that would allow public health officials to do outreach to the contacts. So again the data are put in one central location and there is contact tracing that happens via that notification, but at the initial user level what's happening is just that phones are exchanging Bluetooth signals with one another. The decentralized version of this technology, and that's the version that Apple and Google have been pursuing in their exposure notification system, works like this. When an infected person reports her status to the app, only her anonymized IDs are added to a central list of infected persons. Other users' phones download that list on a regular basis to check for previously seen IDs, so they're looking for matches between the anonymous IDs that their phone has come into contact with and the anonymized IDs that have tested positive, and that contact matching is done locally on users' phones, and that's really the end of it. There isn't any automatic reporting out to a centralized database or to health officials, so this really is an exposure notification service that's done privately. So that's the technology. As I mentioned, interested it has grown because of the just the mathematics involved in having manual contact tracing succeed, which are the key math figures here are just the number of cases that would have to be tracked, which has proven to outpace even the efforts to scale up the number of human contact tracers that some cities and states have undertaken. The fact that 40 to 50 percent of cases will never develop symptoms and the fact that up to 50 percent of transmissions occur from asymptomatic individuals, those three facts together make it highly probable that manual contact tracing just can never succeed in stemming the epidemic here. We've also been buoyed by the experience of some foreign countries who have deployed digital contact tracing apps quite successfully. These are countries with different systems of government from our own and that made these systems either mandatory or opt out. They have consequently had high levels of use of the apps. They've also actually enjoyed high levels of public acceptance of the apps. Again, these are East Asian countries and others where a different sort of role of government in citizens lives is the norm. The apps have been very successful in enabling public health officials to identify cases to work with them to make sure that they are supported during isolation and are complying with isolation orders and to notify contacts and make sure that they are self-isolating. In the U.S., our experience has been less successful. A number of states have tried to launch apps through their health departments. Some have been more successful than others but a key fact is that Google and Apple, although they've been developing a platform, have until very recently required states to develop their own apps and that's not something states excel at. There's been a lot of companies in the market but they've had uneven levels of success. As you can imagine, the pressure to deploy an app very quickly has led to some shortcuts and development with some embarrassing outcomes including a major privacy breach in one state. There also has been very low uptake of these systems which have all been deployed on an opt-in basis. Even those with lavish privacy protections have typically enjoyed in-cell rates of 10 to 25% by Americans. As a consequence, Apple and Google have tweaked their exposure notification system recently instead of making it purely opt-in. This is now something that, as of September 1st, is going to reside on users' phones. It's just going to appear there in places that the city or state or university has decided to use this express system. They call it Apple Exposure Notification Express. They've had a couple places in California at service pilots for this system and although there hasn't been a lot of pushback, there also hasn't been a lot of use of the systems. There's still a lot of agency required by individual users. They still have to take steps to use the system because it doesn't liaise. It doesn't report to any centralized database. We've had a slow, fitful and unsuccessful experiment with digital contact tracing here in the U.S. There really remains a lot of questions at this point, particularly as we move into a new phase of the pandemic, hopefully in 2021 where we are now moving into a chronic phase of the epidemic as vaccination rolls out as to whether we can do better with digital contact tracing than we have, whether it's worth pursuing or worth pursuing in a different way. It clearly raises a lot of ethical issues that I think will be very readily apparent to this crowd. It's not unprecedented to use people's smartphone geolocation data for government purposes, but it's unusual. It's mostly done in the law enforcement context and then only with a warrant. This is a pretty big departure from Americans' expectations about the use of their data if government is going to be involved. It's also a disruption of our normal expectation that when people take our health data, we have the autonomy to consent to those uses. It raises for some people some equity concerns because not everybody has a smartphone, about 80% of Americans do, but of course the 20% who don't are not evenly distributed throughout the population. On the other hand, even traditional ways of doing public health surveillance can have disparate effects on particular communities that are unwelcome. Of course, this pandemic, the burdens of the pandemic itself fall heavily on disadvantaged communities, so there are also potential benefits associated with more effective contact tracing. Minimizing the risk of error is obviously a huge concern when you have a hastily developed technology that has never been used before, particularly because during pandemics courts are willing to brook shortcuts when it comes to deprivations of citizen liberties in isolation and quarantine, the normal panoply of procedural due process rights that we would have when we're being confined by the government don't necessarily attach. We have an abridged set of procedural protections, so the potential consequences of mistakes can be amplified in that context. And then finally, and obviously there are big concerns about government and technology companies' accountability for the uses of data in these apps after all the same location data that are useful for contact tracing are also useful for tracking where your political opponents are and it's not always clear what the exit strategy with these apps and this information is going to be once it's collected and how do we cabinet into the uses that people thought their data was going to be used for. So I think there are three ethical principles that are useful load stars for considering how to grapple with these problems and they should be familiar, I think to this crowd. One is that with any public health intervention, we want to evaluate it not in the abstract, but in reference to the counterfactual, what would we be doing instead? If we didn't do this, what kind of public health interventions would we need to do or what would happen if we didn't do it? And you know, I think in this context, the counterfactual manual contact tracing is not a viable one. So that really becomes a question of what else would be happening. And I think that alternatives are probably failure to contain the epidemic as we have had so far, or a much more restrictive set of community mitigation measures, like social distancing, business closure and school closure order. So this counterfactual situation is quite serious and I think points in the direction of using DCT. A related principle is the least restrictive alternatives is a core principle of public health law as well as public health ethics. And it really asks whether there is anything that we could do to minimize the burdens on individual liberty, privacy, and other interests as an alternative to the intervention that is being proposed. And then I think the final load star to keep in mind is that there has to be a reasonable tradeoff of benefits and burdens. We might, for example, be very drawn to technologies or designs for these apps that are privacy maximizing, like the Apple Google setup. However, if it doesn't yield any public health benefits because the information doesn't ever reach the people who are in a position to do the actual contact tracing to support people in isolation and quarantine and so forth, there's an argument that you're just actually taking from people without returning any benefit. So the questions about do we want to take more and if we're able to give more, I think, a really important one to grapple with. I think it's really salient to consider what Americans have to say about their willingness to use these apps on a voluntary basis. There have been actually quite a few surveys, albeit most of them done in the spring when, you know, people were in a pretty different state of mind, I think, than they are now. But they point to a majority of people saying they're willing or somewhat willing or likely or somewhat likely to download these apps. But it doesn't quite tip the point where we would say with confidence this looks like people are absolutely ready and willing to go to their phones and downloading use these apps in droves. And more recently, when researchers surveyed people just about the general acceptability of this idea, there was less than 50% of Americans saying that they supported somewhat or strongly the idea of government doing digital contact tracing. A further difficulty is it, of course, not everybody who says they are likely or somewhat likely to download and use an app would actually do that. More than likely we would see much lower numbers as we have here in the US of people actually taking those steps. And it's difficult to tease out in these surveys, people's genuine resistance to apps versus just inertia where, you know, we might be willing principal to do something, but in practice, we don't do it. Finally, there's emerging evidence that out levels of uptake would have to be pretty high in order to get big public health benefits from digital contact tracing. We get modest benefits even at low levels of uptake, it seems. There is a modeling study done by researchers at the University of Oxford using data from three counties in Washington state. And that modeling study made some very rosy assumptions, I must say, about a number of surrounding supports for digital contact tracing, including that there would be a really robustly staffed manual contact tracing program to partner with the digital side that everybody would immediately self isolate if notified that everybody would immediately get rapidly tested. Assuming those things, this is, you know, these are the data that at 15% uptake you would have a modest reduction in the number of infections and deaths. You'd need to have upwards of 75% uptake in order to get major reductions on a scale that we're talking about reversing the spread of the epidemic. So, you know, I know the folks who did that study and the folks who paid for it really like to focus on the 15% figure as having some benefit. But when I look at those data, they seem pretty marginal and pretty optimistic at low levels of uptake. We would really need high levels to get somewhere. So I think this raises the couple of important questions that we might focus our discussion on today. One is the equity question, which is of obvious concern, even though the survey data actually don't show big differences in the willingness of African American and Latinx white communities to use these data, much more difference by age. You know, there is concern about what it might mean for vulnerable communities who've already suffered. So, you know, what are your thoughts about whether those communities stand to benefit more than be harmed or vice versa? And then what is the optimal consent regime for this kind of tracing program? I'll offer some thoughts of my own, but I'm going to pause here and just open it up for folks who might like to contribute some comments or questions. So there is a question from the Q&A that refers to the tracing aspect. So, wouldn't the fact that many infections are asymptomatic also be a problem for digital contact tracing, not just manual contact tracing? That is, how do either the apps or a human at a public health agency know that someone is positive and is it just through testing? Yeah, so it's a great question and you're absolutely right. In order to get the maximum benefit here, there has to be screening testing, not just symptom-based testing. And as you know, different places around the country are different places with regard to who they're testing and how often. You know, most places I think are somewhere in between purely symptomatic individuals, you know, high index of suspicion individuals and pure screening. There are parts of the country, including where I live, that where screening testing is widely available and used, for example, for our kids to go back to school, they've got to be tested every couple of weeks. So that's going to be a driver of the overall effectiveness of this strategy. But as with all other things in this pandemic, testing is critical. And just a quick follow-up. Is that self-reporting then? So that's the idea. Yeah, I think all of the systems that I'm aware of in the U.S. have been designed around self-reporting. Of course, it doesn't have to be that way, you know, and the product that Verily has worked on, for example, is a product that integrates a screening testing program with clearance to return to work or school, and it reports out directly from the testing program. So in theory, of course, this technology could be linked to labs, but that's not the way it's proposed to work. So it really relies on users both to install the app and then use it. And, you know, question mark, is somebody who's newly diagnosed with COVID likely to, you know, rush to their app and report it, or are they going to have other things on their mind? Certainly. There's another question, then, about sort of personal data, that how are personal data of patients, you know, such as birth dates, addresses, test results, protected against leaking? So, you know, different apps have different firewalls, but the idea is that with the Apple Google approach, everything is local, and that's how it's protected. It doesn't get uploaded to a big server somewhere, so you share what you want to share, and everything that is going on outside of your phone is anonymous. Your phone has a number, and it's only when another, a number meets another number that it recognizes, because there's a match that more information is revealed. The ones that use a centralized server, of course, there's more information that's going to be external, and the privacy protections are going to vary, you know, technologically with the design. But part of those privacy protections would have to be at the level of the public health agency, clearly. And sort of along those lines, do you know if the sort of the user agreements have anything about using this information in public health emergencies? And if not, do you foresee updated user agreements actually having clauses related to public, the use of data in public health emergencies? So, I assume you're asking whether, contrary to the general understanding that these user agreements would convey, there would be some clause saying, well, in the event of emergency, we might just take data that otherwise we're saying that. Not aware of that being part of the agreements, you know, I think they do their best to be pretty transparent about how data are going to be used and not leave big escape clauses like that. But they're complicated, and there are all kinds of problems with user agreements, quite outside this context of whether people ever read them, whether they understand them, and whether they have an opportunity to tailor them to uses that are acceptable to them. So, even if the user agreement is completely transparent and doesn't have an escape provision like that, a lot of people would question whether that constitutes meaningful consent. Great. Just for the sake of time, another question or two. So, are tracing apps better than use of data routinely collected by telecommunication service providers? Well, they're different, you know, they are collecting a different type of data, and they're also doing it in a very different way. This use of the low energy Bluetooth technology is supposed to be privacy enhancing, and it's also supposed to enable a much more granular capture of geolocation. Normally, what's done is that signals bounce off the cell phone towers, and you can see where people move around according to the cell phone towers, but only, you know, at a pretty macro level. It's not fine-grained. Is the Bluetooth technology can capture people who are moving within six feet of one another? So, from a public health perspective, that's much more useful than knowing who was in the same five mile radius as who else. Great. Another question then. Would the response to a digitally generated report that I have had a contact be voluntary, or would the assumption be that if I had a known contact, I would undergo mandated quarantine, and if the latter, I'd be much less likely to sign up? Yeah, great question. So, I don't think anyone in the U.S. is talking about a system where exposure notification triggers application of some standing public health order to you. I mean, you could do that, right? A local health department could write an order saying, there's this app, anyone who uses this app and gets this message immediately must self-quarantine for 14 days. That's what Israel did. They had this mandatory use of this app, and as soon as you got reported as a case or a contact, you got sent into a 14-day either isolation or quarantine as appropriate to your clinical circumstances. That's, you know, that's a pretty significant and swift deprivation of liberty. It's going to be more effective than relying on people to self-isolate, especially in the absence of any offers of support services to people who are being asked to immediately self-quarantine or isolate, and who are not prepared to do so in terms of their jobs, childcare, family living situation, and so forth. But that's not really what's on the table here in the U.S. right now. And one more question. What are your thoughts and sort of just playing devil's advocate a little bit, that given that we have this data or that it's sort of available out there, is it unethical not to use that data and information in a public health emergency? Yeah, that's a great question then. And I segue into my set of recommendations. I believe it is, you know, because of the severity of the counterfactuals involved here. And I think that that is a core principle of data ethics, that sometimes it is unethical not to use the information that we have. And given the death toll here and the utter inability of the government to do anything effective about it, you know, I think we're there. And so for me, really the question's become, you know, how do you do this in a way that is ethically defensible? And I think these three recommendations are the most important. One is that, you know, as I sort of alluded to before, if you're going to do it, do it in a way that actually has some public health benefit. And, you know, that means number two here that it has to be, in my view, installed on an opt out basis rather than opt in, because otherwise it's simply not going to be useful. And so you're going to be taking information and bridging privacy from a bunch of individuals who've stepped forward as volunteers and who have no reasonable prospect of receiving a commensurate benefit. So I think if we're in for a penny, we're in for a pound with this one. And that makes it all more important to have this third recommendation, which is that this be done through some kind of transparent and public facing process. The way it has been done so far is pretty closed door. You know, there were early meetings, well, at least an early meeting between White House officials and representatives from technology companies about this. There have been smaller versions of that going on with state and local governments since then. But I'm not aware of any public facing process, any multi stakeholder process, particularly that would bring representatives from highly affected communities affected both by COVID and by concerns, equity related concerns about smartphone tracing together to talk about these design issues. Nor am I aware of any oversight process that would reassure the public that the data in fact were used in the way that they were intended and not in other ways. Other countries have managed to do this process. I won't say in a in a deliberative democratic way the way we would think about it, but in a way that has engendered very high levels of public trust. And part of that has been very candid with the public about what's going on and what the access strategy is. So that's clearly going to be critical. Great. For the sake of time, we'll plan to move on into the keep you moving along. But there are definitely some still outstanding questions on the Q&A board that we'll try to get back to towards the end. Great. So the second issue that's really perked my interest over the last couple of months has been thinking about the way that science is changing during COVID. And we've obviously heard a lot about some of these issues in the ongoing public debates about vaccine developments, concerns about transparency. But I think there's a broader set of sea changes and research practices that has gone on that is worthy of our attention. So I'm going to just pick up on some themes from a forthcoming article on Scientific American. The broader context for these concerns is just the unbelievable scale of the mobilization of scientists around the COVID-19 pandemic. And many of us have gotten involved, myself included, because all our regular research got shut down. We're looking for something to do. There's also just an unbelievable need due to the novelty of the virus as well as the novelty of the many public health interventions that have been implemented here in the U.S., which really are unprecedented. So those of us who normally spend our time thinking about law or public policy have also been drawn into research in this area. And then I think many scientists have really felt a call to service here, particularly given the mishandling of the pandemic by the people in charge and the repudiation of expertise and formal policymaking in Washington. I think there's a felt need to contribute expertise and put it out there to whoever might be available to receive it at the state and local level. So yeah, I find this scientific search really, really inspiring and really astounding. Over 65,000 articles related to SARS-CoV-2 have been published just in the academic journals alone in the last seven months. And that's incredible. I think there are some contextual factors though that maybe give rise to a few concerns about how all of this research is being conducted. One is just the speed with which the research itself is being conducted. Everything is going fast. And the openness around that, which is generally a great thing to have more openness in science. I think we saw, even during the Ebola virus disease epidemic of 2014-15, a greater commitment to sharing research results quickly than we've ever seen before in public health. And that has just been magnified 100-fold during COVID. But there's a dark side to that, as I'll talk about. And then third, just the extraordinary high pressure to get answers to decision makers quickly has also created a lot of pressures that would potentially lead to shortcuts both in the conduct of research and how we analyze it, how we report it, how we interpret research findings. And I think in three areas, there have been some real challenges that have arisen. One that I'm familiar with from, you know, from Stanford, but I hear about going on at other institutions as well is the challenge of research prioritization. You know, in the context of a pandemic, it's surprising to think that there are not enough participants in research studies. You might not have to go around, but that is the fact for a lot of research that involves hospitalized COVID patients. Those patients are really unevenly distributed. And there hasn't been enough to go around at the time that a lot of the clinical trials and therapies were launched in the spring. All of these research teams coming to hospitals and to universities to launch their studies overwhelmed both the hospitals and the IRBs and left a lot of universities in a situation where you know, there were more teams than there were patients. And so they had to make difficult decisions without a lot of guidance about who gets access to patients and how to prioritize among potentially competing studies. And this is not the first time we face that problem. It's a perennial problem for cancer clinical trials. That's why, you know, a lot of universities have cancer centers set up that have cancer research committees that deal with research prioritization. But it's also been a problem during prior pandemics during the 2014-15 Ebola virus outbreak. It was a huge concern as a crush of Western teams jockeyed to get access to patients in Ebola affected countries in West Africa. There was not a robust and centralized system in place for prioritizing among those teams and among those studies to decide who was best able and positioned to carry out research and who had the best study questions and the best designs. And so these African countries that were already overwhelmed by the epidemic were now overwhelmed by all of these researchers asking to come in and set up shop. And they had tremendous resentment about that. And it also was a wasted opportunity because the folks that got in were not necessarily those with the best designs. It was often people who were first to propose or people who had existing contacts in the government. And the result with that is that one study of a therapeutic modality for Ebola that was randomized was carried out in these West African countries during the pandemic. So there's a real potential for lower quality studies or studies of maybe ancillary questions that are not central to stemming a pandemic to get prioritized inadvertently over more important or better quality studies. And because there are not enough patients and research sites to go around, there's a potential for all studies to suffer under enrollment for patients and researchers time to be diverted and for results to be delayed as a result to these problems. So I think this is a major challenge that has not been confronted or even talked about much for COVID, but may really have hampered what we were able to stand up and get underway during the pandemic. A second issue, again, a perennial one in pandemic response is how do you design studies as between the gold standard randomized control trials and non-randomized designs that can be stood up quicker but have obvious limitations. As you're all familiar with because you read the paper, non-randomized studies have been common for COVID-19. Many of them have taken place in the context of FDA expanded access programs and they have led to some extremely suboptimal outcomes. The two cautionary tales with which most of you will be familiar are a convalescent plasma and hydroxychloroquine. With the CVP, this was part of the FDA's expanded access program through which about 78,000 Americans received convalescent plasma with no control group. The studies evaluated hires versus lower doses of CVP. And so we're not designed to answer the question of whether it works relative to not receiving CVP or how well it works, only to answer questions about dose response. Nevertheless, even admits criticism that the evidence base was insufficient. The FDA approved an emergency use authorization for CVP in August based on the finding that patients who received earlier in higher doses did better than patients that received lower doses. And as we now all know, both the president and the FDA chief initially dramatically overstated the results of those trials. And then of course, there's the hydroxychloroquine example where early on in the pandemic, the president urged doctors to prescribe the drug based on an observational study of 20 patients. There was an emergency use authorization and later randomized control trials showed that the treatment was not effective and was associated with serious safety events in higher mortality and the FDA was forced to rescind its EUA. So there are serious potential implications associated with preferring these non randomized studies. As these examples show, really low quality evidence can guide very important decisions. And as we again learned during the Ebola epidemic, one consequences that patients and researchers time and funding can get diverted from higher quality studies that could bring more definitive answers to questions. There was intense debate about this issue during the Ebola epidemic with all agencies of the government at that time really pushing hard for randomized studies. There was a great deal of pushback on the ground against those designs. And so as a result, only the vaccine trials were randomized as I mentioned before. And we learned very little, there was a very thin scientific harvest. Last challenge that I want to mention is the use of preprints. Even prior to the COVID epidemic, there was an increasing movement in science towards pre printing results before they underwent formal peer review. And the creation in 2019 of Met Archive and Bio Archive to pre print databases operated by Yale and BMJ and Cold Spring Harbor Laboratory really facilitated the growth of pre printing in part because they were able to gain the permission of about 30 prominent journals to allow pre printing. But during COVID we have seen just an explosion in the use of pre prints, the dissemination of research results prior to peer review and often at a very preliminary stage. Just Met Archive and Bio Archive alone now host a staggering 10,000 papers about SARS-CoV-2. One study found across all pre print servers over 16,000 papers just through the end of April. And interestingly among pre print authors, 83% of them were posting a pre print for the first time if the paper was about COVID much lower proportion for non COVID papers. Additionally, we know COVID pre prints are downloaded at a vastly higher rate than non COVID pre prints. They're immediately picked up by news reporters who troll these archives looking for cutting edge research. They're widely circulated on social media. And there's no doubt that making these study results available quickly has had an important influence on both the evolution of science and on public policy in positive ways. There are also I think some reasons for concern. COVID pre prints are in one study 44% shorter on average than non COVID pre prints suggesting that authors may be using these platforms to share works in progress rather than fully baked research findings. And while scientists understand the limitations of foregoing peer review others including reporters and people on social media may not. Of course peer review is no guarantee scientific quality and there have been some high profile instances even of peer reviewed studies being discredited during COVID. And the journals are also going very, very fast cutting some corners in some cases on peer review in order to get results out quickly. But it nevertheless plays in my opinion a pretty important role in quality assurance. One study of COVID pre prints that later went on to be published research articles found that more than a quarter underwent changes to the wording or numbers in their abstracts that affected the study's conclusions. It's also the case that about 30% of pre prints in bio archive never go on to be published in peer reviewed journals. And the most tweeted pre print from January to April 2020 was a study of antibody seroprevalence in California that about which major concerns were later raised. The second most tweeted pre print was a widely criticized and later withdrawn study linking the SARS-CoV-2 spike protein to HIV one glycoproteins. So there are some documented concerns about the use of pre prints that I think ought to lead us to really consider whether these evolutions and scientific practices are good or bad. So here are a few discussion questions that I'll throw out there with respect to each of these three topics research prioritization and then concerns about inadequate study designs and early dissemination. I think there's a view that because science is inherently self-correcting we don't need to be too worried about these and I think you know my own view is there's some reason for concern about whether that's empirically true. So let me pause there and take some questions. Great so yes participants please feel free to submit questions to the question and answer. Drawing from the question and answer there's one that many things could lead to resistance to randomized trials. Patients may not want to be randomized doctors may not want to randomize their patients IRBs and regulatory agencies place administrative barriers to scientists who want to study new treatments that don't exist for clinicians who just want to use non-validated therapies. So which do you think is most important? How can we get a world in which more RCTs are done and done quickly? Yeah I don't think I have an empirically defensible answer to the question of you know which is more important as a like a descriptive matter which tends to come into play more and I suspect it really varies according to the context. I do know during the Ebola pandemic I know this because I was on the National Academy's committee that looked at the the research harvest during that pandemic. The leading concern by far was that the humanitarian concerns essentially by the frontline providers who were taking care of Ebola patients who were not researchers it was it was folks from Nezans en Frontières who could not justify denying access to a therapy or a vaccine given the high mortality rate of the of the disease could not and so didn't want to be a participant in a randomized study. You know that's not really an issue in this this pandemic in part because the death rates are much much lower and randomization seems more ethically defensible but you know all of the other things that the the writer mentioned are clearly in play. I also just think there's an element of ease involved in standing up a non-randomized study that that isn't the case for a randomized controlled trial and so you know how do you how do you address that? You know obviously one key element has to do with the funder and what they are willing to fund but it it's not always the case that they that they can exert the kind of control that they want that was that was clear during Ebola even though NIH was adamantly opposed to non-randomized studies they just couldn't get them couldn't get local ethics committees to approve randomized trials so it has to be more than that and you know I think fundamentally what it comes down to is how much longer are we willing to wait for a stronger answer and you know again I think that's going to be contextual but I think it's hard to justify in the context of this pandemic not being willing to wait a little bit longer for strong answers about therapies as we are now doing for vaccines you know there was a push to to close down those studies earlier and accept the answer that we had without a long follow up period and that was rebuffed. Great and and there's sort of a question a little bit along those lines in terms of like how long we we wait and how do we study or how do we validate sort of the the quality of the research is how do you value the hope someone feels when they receive a not yet researched enough treatment? Well I don't value it very highly especially if that treatment can hurt them. Yeah I think that's that's my answer and that can't be a good enough reason to subject them to potential harm and to delay answers that might benefit others in the same position a short time later. One thing that's been interesting to me throughout all of this as it relates to sort of the rush to research and how it gets to the public I mean you've mentioned sort of the media several times right that how you know these how they make it into the paper what are the ethics or do you know what the ethics are in which certain studies end up making it into the paper and being promoted in the papers and then a follow-up question to that in terms of you mentioning science being self-correcting do you feel that the media is at all self-correcting when they sort of promote something that ends up not being being you know safe safe and efficacious? Yeah interesting questions you know I think you know for good reasons there aren't any controls over what the media picks up and and considers new as worthy and publicizes it's it's tricky to do that and who's in charge of doing that but what you hope is that there are journalists who take seriously the obligation to get expert opinion a bit before just publishing the results of the study and my observation is that journalism has been like everything else during the pandemic you know it's been even faster and higher volume than ordinarily and that there's a lot of variation in the extent to which the coverage really reflects deep reporting so I would be a little hesitant to interpose some kind of body that regulates what the media picks up but I do think it's it's more important than it ever has been before for reporters to vet story ideas that the way they're going to characterize particular studies with experts in the field um I kind of what was the second part of your question? I think that was the question um but but I guess a following up on that though then should either medical journals or institutions that are that are you know funding and supporting this research should they have some obligation to think about the relationship that they have with the media in terms of what research and what stories they're going to promote I mean I know a lot of institutions big institutions just sort of push out certain certain studies to get it into the media to generate generate clicks like what responsibility do journals and or institutions have to putting safeguards on their relationship with the media? Yeah you know it's actually interesting to think a little bit about how much the communications offices have been involved in pushing out pre-printed research I'm actually not sure you know my sense is more that it it gets up there and people start talking about it on twitter and that and then reporters kind of pick it up but that's a more important mechanism than actually the kind of press releases that our officers would do when we've got an article coming out and in many ways that's it's more hazardous right because it only takes one scientist who thinks some pre-print is interesting and tweets about it and perks up the ears of a cub reporter for things to start taking off so I think actually I'm recalling now the second part of your question was about you know the self-correcting nature of journalism and my observation during this pandemic is it's been pretty good actually that the journalists have been very attentive to the evolution of the evidence base at least the journalists in the finer outfits that are really doing high quality science reporting so you know hydroxychloroquine of course was critically important for them to continue to report on and I think they did and did a very nice job but the problem is that news reports are sticky in people's minds and I especially worry that that clinicians will not update their their views about therapies along with the science great and there's one more question from the from the q&a before we move on so it's easy enough to insist on stringent controls for safety but since this is a pandemic and we're racing against the clock how do we balance the checks and balances versus getting something out in a timely manner for example russia was criticized for putting out a vaccine before stage three clinical trial but if the vaccine has a history of being safe is this a reasonable move I mean that's the million dollar question right and I would actually answer it differently for vaccines than for therapies because I think that we have a very fragile edifice of vaccine acceptance to to build on when we're talking about pushing out the vaccines I'll talk about in just a minute so any mistakes that we make along the way in pushing out products early are going to have very serious and long-term consequences for our vaccination efforts I believe with therapies I think there's maybe a little bit more latitude to err on the side of less evidence what concerns me is that the search for answers might stop too soon or that because non-randomized studies have occupied time and funding and attention that the better quality studies you know can't find participants can't find sites can't find teams or don't ever get done for other reasons and that to me is a more serious problem because it suggests that where we might ordinarily think about a life cycle of evidence in the time critical situation that the life cycle gets cut short in adolescence so I think that the the prospect that that might happen would be a critical determinant of how we should think about this and and Ebola is not COVID in this respect so the Ebola epidemic burned itself out very very quickly before randomized trials could get launched by the time they were launched even for the vaccines there just weren't enough patients left and that's great that it was over quickly but it really suggests that for something like that where we expect to see it burn through a population and flame out pretty quickly we have to have a different approach than to a pandemic that is likely to be with us for a longer period of time and or is primarily centered in countries that have better research infrastructure and where we can get things going more quickly great and now we'll move on to the sort of third topic and then come back to questions after that yeah so um the last thing I want to close on is talking about vaccination which of course everybody is doing right now um I published my two cents along with Sada Omar who's an epidemiologist and Ross Silverman who's another public health law type in Negem uh earlier this month um the the problem will be familiar to all of you of course which is that now that we've got a vaccine on the near-term horizon there's lots of focus on how we're going to get people to get it um public health experts think that we need somewhere around 70 percent of coverage and that's likely to be with a two-dose vaccine so not an easy task and most surveys find that willingness to receive a hypothetical vaccine is hovering somewhere around the 50 percent mark sometimes a bit lower sometimes a bit higher of course these surveys are difficult to contextualize because it is a hypothetical vaccine we don't have any information about the actual vaccine yet um we believe there's a number of contributors to the problem here um one is that the anti vaccination movement which has always been incredibly effective given its small size has been greatly bolstered during COVID um it is merged with a general libertarian sentiment arising from other COVID intervention measures like masking um and it is has pivoted its attention almost entirely from childhood vaccines to focusing on a prospective COVID vaccine um and it's used social media very effectively the second problem here is that the vaccines are being developed at warp speed it's very likely that the initial approval mechanism will not be a vla application to the FDA but an EUA where the standard for approval is that the product is safe and may be effective it's a pretty low bar FDA as you probably know is pre-committed to some benchmarks for approval of the vaccine but they're pretty low um so whereas we might historically characterize vaccine resistance as scientifically irrational it's actually not the case in you all right there's a there is actually a rational case to be made that a person ought to be concerned about the risk benefit trade-off from receiving these prospective hypothetical vaccines in this case and and and that concern is compounded of course by the extreme politicization and partisanship around the issue and the known efforts to influence both CDC recommendations and FDA approval processes bipartisan within the White House so again we've got a situation where a like a longstanding hardcore resistance movement is intersecting with a much broader set of people who have reasonable concerns about how these vaccines are being brought to market and who they can believe about the messages about their safety and efficacy this study I think provides some useful data for thinking about the problem um this was a online panel study done earlier this month or sorry published earlier this month but reporting data from July and trying to measure differences in people's hypothetical willingness to receive a vaccine according to different vaccine characteristics um and there's some really notable findings here one is that overall again that the willingness is not super high it's hovering around 60 percent in this survey here on average and then second is that the you know the vaccinate a vaccine scenario that is probably pretty likely which is that it comes through EUA and it has efficacy that's proven at a pretty low threshold around 50 percent that's a situation where we're likely to see substantially lower levels of willingness than a general hypothetical people are also very concerned about vaccines that are developed outside the US which is less likely to be a problem here although some of the leaders are working in Europe and they're very unwilling to trust vaccines that are advocated by the president as opposed to the CDC and the WHO so again a lot of these conditions that are associated with lower acceptance are happening they're in play right now so there is a lot of reason for concern here in the paper and as I've continued to think more about the issue I think several recommendations emerge and the most obvious one and I think there's a little controversy about this is that there is a need for absolute transparency around FDA's approval process as well as the underlying clinical trials that the all of the things that we have started to demand for clinical trials generally ought to take place here including not only sharing of the results but the clinical study report with the study protocols statistical analysis plan and arguably I would say even the participant level data ought to be shared and ought to be shared at a much earlier stage than would ordinarily be the case ordinarily these things get shared if at all six months 12 months after FDA approval I think there's a strong argument for requiring vaccine developers to share them now I think it will also be critical secondly to have independent opinions after FDA's opinion comes down and and state officials are not what I have in mind I live in a state where they've already announced that they're not going to accept FDA's decision on this front they're going to conduct their own review but I'm much more interested in what ACIP has to say and I think they're very wisely withholding judgment until they have a particular candidate approved but I think it's also important to have this information these data shared so that other independent experts that the public trust can review and offer their own opinion on what's been done here I think a third critical message that comes out of some of the polling around vaccine acceptance is the need for messaging that's tailored to particular communities there are different concerns across communities according to partisanship, age profile, race and ethnicity but we haven't really developed a messaging strategy that takes account of those differences and then finally in the paper we argue that I think there's an argument for considering a vaccination mandate but only if several trigger conditions are in place and these are the triggers that we lay out in the article one is that there's evidence that other community mitigation measures have failed to contain COVID in the state and we are probably talking about a state mandate that's generally the legal mechanism through which vaccination mandates get handed down not federal the second is that ACIP has recommended vaccine vaccinations for the groups for which a mandate is being considered we suggest in the paper a variety of high priority groups that accord pretty well with what the national academy committee on vaccine allocation just came out with I think ACIP is really a critical body here because it is still independent and does still have a great deal of legitimacy among trusted elites and one thing we stress in the paper is that the fact of FDA approval cannot be sufficient to support an mandate ethically because it is based on a very different set of judgments than we would normally make when considering whether to burden people with a coerced intervention I think the third recommendation a lesson from prior epidemics is that you don't mandate something until you sure you have an adequate supply to carry it out this is a mistake that New York state made during the H1N1 epidemic where they mandated vaccine the vaccine for healthcare workers and then couldn't cough up enough doses to effectuate the vaccine so they ended up stoking an enormous amount of anger from nurses and other healthcare providers that soured relations and really not only didn't achieve higher rates of vaccination but also made it more difficult to achieve the government's other aims during the pandemic the fourth recommendation it has to do with transparent transparent communication around safety and efficacy evidence you might ask what difference does it make if people have to get the vaccine whether they know the evidence or not but I think that is a critical element towards building a sense of trust a fifth and very important recommendation we think is that the state has set up certain supports for people to receive the vaccination I think three are most important one is that you're bringing the vaccine to people where they are in making it available with near zero logistical and financial barriers the second is that there is a compensation program for workers who are required to receive the vaccine and who are injured by the vaccine that too is a critical sticking point during H1N1 and the third is that you have a robust system of real-time surveillance in place for vaccine side effects and then finally I think and really critically that we should only be thinking about a mandate if a time limited trial of voluntary uptake has failed to achieve sufficient levels of coverage and when I say time limited you know probably a matter of weeks or a couple of months here that least restrictive alternative principle I think caution is very strongly against starting with a mandate but in this case the cost of failure is sufficiently high that we shouldn't wait a long time to conclude that that voluntary action is failing and so in order to figure that out it's going to require that states have a system in place for measuring vaccine uptake among critical high priority groups that's been an essential component of the national academy's advice and it's it was again in the paper today with with the military talking about the need to carefully track every dose of vaccine so that we can measure that and I think ensuring that the economic and logistical supports that I've already described are in place will help maximize the chances for a success with the voluntary program and I'll close by just saying I think it's really critical for all the stuff to be happening right now you know they can't wait until a product is approved some of this will have to wait until then like evaluation of of who's getting it and what the evidence is for them but these broader systems of delivering the vaccine and making sure that people have the support that they need are things that should be architected now so I'll end with a couple of discussion questions about that and I'm going to just open the floor this time great so taking questions again from from the from the Q&A the first is why do you think sort of the WHO is regarded as a better source of vaccine for a vaccine recommendation than either presidential candidate well I mean I think it's clear that partisanship in the U.S. has has risen to such a point where people are simply unwilling to accept any message however rational if it comes from a particular candidate and they aren't inherent of the other candidate so regardless of whether I might think one candidate is a more trustworthy source of information than the other with fairly objective standards that message is just not going to get through if it's sent by particular communities now is WHO better in this case probably not because it has also been bound up in the partisanship of this policy area because the president repudiating it as a useful organization and a trustworthy organization so you know it's actually surprising to me to see survey data saying that more Americans trust their recommendations from the WHO than from the president because I would have guessed that those would be pretty closely aligned levels of trust in other bodies you know are also pretty compromised because of the way in which they become entangled with this administration and you know data about trust in the CDC is also really dispiriting it's hovering under 60% now FDA only a little bit higher so we've got a real problem here and that again points to the need to have a number of different messengers at work speaking to different segments of the population I mean what while we're on the sort of topic of politics a little bit when you mentioned sort of sort of ASIP as an important sort of group that should make recommendations I wasn't completely aware where that sat but it looks like it sits under the CDC and its members are selected by the secretary of the department of health and human services who I think many probably regard as a partisan actor so how do we again how are we going to sort of reconcile that when everything is already so politicized and you put a bunch of eggs in a basket in a committee that I think people may still look like look at some degree of mistrust yeah I mean I don't think it's a committee that's widely known by the average man on the street but I think among you know health experts it is pretty widely known and one of the things that's known is that this is a standing committee those people have been doing this for a long time they've been doing it together for a long time they're not political appointees and their expertise is very hard to argue with when you look at their backgrounds so even though they're nested within the CDC I think you know like many lower level scientists who are also nested within the CDC they are they are true experts who are continuing to do work that hues pretty closely to science-based principles again second they convince the average man on the street no but if you have someone that they trust saying look this this committee is trustworthy and I believe what they say that I think would work and speaking of trust there's another question about it so giving that trust around such a vaccine is already so delicate what would you recommend for the dispersal of the vaccine in terms of determining who gets the vaccine first or in other words what should the mechanisms be for identifying those high priority groups yeah well this is one area where I think the the preparation has been actually very very good that committee appointed by the national economies was requested by the lead federal agencies their you know I think it was a very well composed committee and their recommendations have very high face validity and have been largely echoed by a number of other independent analyses you know that have have sought to arrive at a vaccine allocation scheme the trans their criteria are very transparent and they they also have very high face validity that we have to be concerned about you know preventing transmissions and preventing deaths and that and each of those goals points towards a certain set of groups and they're not always the same group you know there are people who get COVID and don't have serious problems I think there have been a number of really sensitive treatments or some tough issues like should vaccines get directed to disadvantaged communities there was an article in JAMA by Zika manual and Govind Prasad and really the third author was but I think that you dealt with that issue in a in a really effective way I think it was Monica who is still on the call here that although that idea had you know seems very appealing from an equity standpoint it can actually depending on the dynamics operate to the detriment of those communities so I do think there is really good thinking I don't think there's a lot of of controversy actually about who ought to get it first and and what helps is that we haven't tested the vaccine yet in kids so we don't have to worry about rolling it out to them right away we're talking about a relatively confined set of individuals and we have a because it's taken so long to get here have a good sense of who's at risk both for transmission and for death and and so as a result I think there's there's a good sense of how we ought to allocate right and for detailed questions to your answers to that question fortunately we have at least two if not three lectures later in the series that will directly deal with vaccine allocation so in terms of sort of other groups that potentially should get it would be should it be mandatory for certain groups like travelers business people service members to get it so who would you potentially sort of mandate it to well again I wouldn't think of mandating it for anybody until someone's proving to me that we can't rely on voluntary action because again I do think it's a very fragile edifice of public trust here that we have to work with but I think there's pretty broad agreement that if there were to be mandates just as in prior respiratory illness outbreaks they would probably be centered heavily on on first responders health care workers and others who are in frequent contact and at high risk situations you know for travelers and stuff I think that that's a much harder road to hoe because we're talking about a vast group of people with very different risk factors and also I think some evolving evidence about the risks of travel that if travel safely handled it's it's actually not clear that it presents as great a risk as we might have thought early in the epidemic and you had mentioned like that now is the time to be sort of thinking about some of these policies but that it happens on a sort of state to state basis what it what is your your sense or your assessment of how good states are doing and how is politics playing into what states are doing I think it's really unclear you know I think that this process is going on in a lot of states at least you know states that are relatively specific but it's not public facing at all you know it's going on within health departments in partnership with their their procurement organizations and that's unfortunate because it would be it would be very helpful to know kind of get a progress report on how far along their plans are and also transparency just sharing their thinking you know when the social distancing orders were coming down in the spring until late spring a number of health officers started blogging about their experiences about their thought processes and I thought this was a tremendous way to communicate with with the public in a you know a non-mortally very relatable way that these were incredibly wrenching decisions that they were getting advice from a lot of different places and taking it seriously that they were struggling with those decisions and that you know they had adopted these principles or these load stars to guide their decision-making and I would love to see more communication like that around vaccination planning. A question from from the Q&A then do you think we might be on a hybrid path to herd immunity partly through people just getting COVID and partly through people getting vaccines? I'm not a believer in the let's get COVID and hope for herd immunity plan and I think the evolving evidence is all in the direction of being pessimistic about that strategy so I don't think we're on a two-track place to herd immunity I think we're on a one track and that track is long and likely to involve a lot of other interventions over the medium term as well. So should the time-limited evaluation period include post-market effectiveness data and if so should that data sort of feedback into the prioritization framework for allocation? Good question you know I think the timeline that we're talking about is going to make difficult to have robust analyses of post-market safety and effectiveness data when I'm talking about you know a trial of maybe two months duration of voluntary uptake it's going to be difficult as a practical matter you can measure side effects but it's going to be difficult to measure efficacy in that time frame or a real-world effectiveness rather I think it's going to be a much longer time frame so it's super important to do those analyses and it should affect our longer-term decision-making about how we exhort or require people to get vaccinated if we learn after the fact that effectiveness is much lower than we thought um or much higher than we thought we may need to adopt our strategy but I don't see that being among the criteria that we would have within this you know two-month trial of voluntary effectiveness or voluntary uptake. Now what are your thoughts on shortening quarantines to persuade the public to take them more seriously? So I assume they're talking here not about shelter at home but actually people who have been exposed and are at a higher risk of developing an infection you know I think that the survey evidence at least from prior pandemics is that people are more likely to adhere to an isolation or quarantine period that is shorter so that's encouraging. It also seems that the evidence is pointing out towards a little bit shorter period than 14 days being viable to catch most infections I think the median time to showing symptoms is five days but you know it seems to me that we could obviate the need to grapple with those difficulties if we could just make testing more widely available rather than relying on a strategy where everybody who's come within six feet of a case has to stay home for 14 days we can test them after three days and release based on that. Well speaking of test there's a question about what are the ethical lessons to take away when a viable test is available through the WHO and a country refuses to deploy it for commercial or financial benefit? So I've written about this I you know I agree with the Atlantic reporter who characterized this as the original sin of the government with respect to this pandemic I don't think there's actually evidence that the refusal to use that test was due to proprietary concerns. I think it you know my understanding is it's just more a sense that well CDC has the best scientists in the world so why wouldn't we use their test and that's usually the way we do things and it's worked in the past and it was just an unfortunate confluence situation that led to this being just a complete and unmitigated disaster this time around but you know it is hard to provide a strong ethical defense of not using a test that seems to be robustly developed validated and being used effectively around the world simply because you think your scientists are probably better. I think probably towards the last question we were running up on time but what do you think about challenge trials? Yeah it's about a great deal about it but I don't feel an intuitive resistance to the idea of challenge trials as long as their eligibility are appropriately tailored. It seems like a reasonable way to get around the dilemma of having to wait a very long time for an answer in a randomized trial and I do think that we're getting good enough at managing the disease clinically at least in relatively young and robust patients that it could be justified for people to be in those trials. Of course the consent process has to be unbelievably robust but I don't feel strongly opposed to it in fact I think it is promising. All right Mark or Laney do you have any closing questions you would like to ask? So I'll take one question you know when you talk about knowing which groups are priority those are some of the groups that are going to be the most resistant to getting the vaccine. Maybe I'm going to switch where you're thinking about in particular. Health care workers so our nurses here at the University of Chicago were on strike about a decade ago in order not to be required to get flu vaccines to be incredibly safe. Yeah I talked a little bit about that experience with H1N1 but you know at least in New York where this really boiled over into a huge debacle. Their position was not I won't get it their position was I won't get it unless you compensate me if I get sick and have to miss work and there was no provision for that. So you know I think it's true there will be pushback but I think you can lessen that in the same way that you lessen pushback for other kinds of occupational exposures to an infectious disease which is making people feel as safe as they can and then being there to support them in the event that they are stricken and and again I haven't heard a great deal of attention to that but it was critically important during H1N1 for them to have a compensation program. Our nurses are still not required to get flu shots. I think it's a lot more long-standing but I also would wonder whether the elderly are going to be willing because a lot of the COVID fatigue that I'm seeing is in the elderly groups who are just lonely and isolated. Yeah well I mean I think you're right to raise the question and I think there's only one way to find out right and so to me like the ethics question is simply do we do we do this and wait and find out if they're willing or just plow ahead on the assumption that they're not going to be willing and I'm reluctant to do that second thing. Well that is a new one on Zoom for me. A lot of echo there Mark. He's on twice. He might want to write it in. On behalf of Mark I'm going to say thank you. He might have a question but I'm going to say thank you and we'll see if they can get it. This was an incredibly interesting three sets of questions. Thanks very much for having me and thank you everyone for sharing part of your day. Thank you very much Michelle. Bye. Be safe.