 So, convincing that we're at the end of the day, end of two days, a few bits of background. And I think what I'll try to do is just a little description, but then we can talk about how you might help us inform the recommendations going forward. So this is the concrete next steps part. So I, my name is, for those of you I haven't actually met, Jeff Khan, KAHN. I'm on the faculty at Johns Hopkins in the Berman Institute of Bioethics, which is a large bioethics institute here in the United States, of course. That's relevant in that there is a group of the directors of bioethics programs across North America, so not including Mexico, Canada, and the United States. There are about 65 to 70 members depending on who pays their dues in any given year. I was the founding president of that organization when I was still working at the University of Minnesota. I moved to Hopkins in 2011. During the course of the years of the organization's existence, we have been able, by their real generosity, to meet with the leadership of the NIH to talk about the ways that the community, at least represented by this organization, thinks that ELSI funding might be expanded, better focused. The topic here is it seemed to be worth pursuing, preferably not instead of, but in addition to the things that NIH has traditionally done. That began with meetings that included not the current director of NIH, but the previous. We go back quite a ways and have had a series of meetings annually, sometimes more than once a year, with senior leadership at the NIH. Included in the current group that meets to talk about this is Elaine Collier. I don't think anybody else in the room is part of that group. But we meet with, oh yeah, Charlize, thank you. We meet with Elaine. We meet with Chris Grady, who's the director of the clinical ethics program at the NIH. And we meet with Francis Collins and Kathy Hudson, so very senior leadership. And we talk with them about what we think as a field are the areas we ought to be working on, and if we think it's important to work on them, then we hope the NIH would see it as appropriate to fund those areas. So that's an ongoing conversation that includes a meeting coming up, I think the 19th of February is the next one. Is that right? So I just add that in response to these conversations, Dr. Collins set up a coordinating committee for bioethics research and training that has representatives from every institute. And we discuss together what's going on across each institute and how we can move this field forward. So I think that was one area, and that group is going to meet in February again from across the campus with the representatives from the bioethics community. So it's, first of all, very generous on the part of the leadership of the NIH to entertain these conversations, but more important and impressive is that there has been real sort of impact as part of these discussions. So all by way of saying there's an opportunity to really kind of help shape what areas might be funded in this area as we talk more about the future of bioethics funding at the NIH. So just to give you some sense of the kinds of things we've been talking about. So you probably know that there's a long history of funding from, LC funding from the Genome Institute. That's where things began. And over the course of time, we have seen funding in areas like ethics of research on human subjects. The community at large through this effort I've been talking about has suggested that we would really like to see funding for training of the next generation of bioethics researchers and scholars so that a T program, T32 is the one everybody knows about, and just now as of a recent release of RFA, the Genome Institute has signed on and there'll be at least consideration of LC T32 proposals. So that's a huge change in the way we think about available funding from the NIH for bioethics research. So LC in the broadest sense of LC. Am I forgetting to say anything else? I want to link a very helpful addendum. Okay. So I've been sitting here the last couple of days partly with this hat on because I'm going to be, I am part of this discussion. I think I'm going to miss unfortunately this February 19th meeting, but I will pass on to my colleagues and there are very few of us who have, who attend these meetings what we've learned in the last couple of days. And I have a substantive interest in the topic area as well. So he probably heard me say a few times articulate the kinds of things I think are appropriate areas for funding. One thing we haven't really dug in much on except to name it is what we term generally as informed consent. So there has been historically a fair amount of funding in how to improve the informed consent process in research on human subjects. One, I think important question is what's the appropriate way to even think about the concept of informed consent in this area we've been talking about. It will require some descriptive work prior to getting to the more prescriptive work of what it ought to look like, but I think that's a really important area and one that would be relatively easy to translate from what's in the current portfolio into what might go forward. This is my just riffing here before I let you talk. I want to say one thing about what was on the previous slide. I won't try to put it up, but it's straightforward enough that the notion of protection of human subjects I think is not just about making sure bad things don't happen to people, although that's part of the history of course. It's really I think a paradigm to protect the interests and rights of those who participate in research. It's a much more positive way to say it. We have the same interests and values that I presume would go forward in anything like what we call citizen science. How do we translate that value, that commitment in this area? That's the question and that's an important research question which again easily translated into the way the NIH thinks about funding LC research now currently. The harder questions for me and this is where I think if people have any energy at all left after two hard days of work is how do we translate some of the newer issues into potentially fundable research areas? That's the rubber meeting the road, the concrete stuff. So if I'm sitting in front of Francis Collins and Elaine Collier among others, what do I say to them about what this group who got together in brainstorm for two days thinks would be an appropriate, important, necessary area of LC research funding? Literally that's the question that is on my mind at the moment. So an opportunity, not an opportunity for us to write an RFA, but an opportunity for us to talk about what the research questions to be addressed might be. Yeah, your name is? Yeah, Deb? Yeah, Deborah, or Deb, either one. Okay, so this may be totally inappropriate to what you're saying, but I'm going to use it as an example from a patient advocate viewpoint working with medical researchers now for over 20 years and an issue that has come to light that actually took me many years to figure out was an issue is non-inferiority trials versus superiority trials. And it sounds like it may be esoteric, but there's actually some real foundational issues and it's similar to some of the things that we were discussing with the environmental health issues and areas as well. And what do people actually know and what do they think about between those different types of research? Because when we talk to, like in cancers, we have talked to the patient communities about clinical trials and why they're important to participate in and that this is something that everybody needs to think about. The thought is that everybody can participate in some form of research, whether it's a treatment trial or behavioral or whatever it is. But then we start to learn some of the, well, not all research is the same. There are different levels, different qualities, that goes beyond phases and all of that. But the real issue here is with the non-inferiority trials, what we're learning is researchers don't even understand the difference between non-inferiority and superiority. IRBs don't always understand the difference between that. And when they're reviewing these, and I'm going to use a particular field that's not involved in cancers, but antibacterial resistance, this is a major issue and it's for an impending health crisis that they've been really good about hyping and making sure everybody knows that we're going to have resistance to antibiotics. The problem is the way that we're studying and developing new drugs isn't necessarily going to answer the needs that we're going to have when that happens. So let me stop you, only because let me play the role here. So if I'm Dr. Collins, I'm going to say, that sounds like a science issue to me, a clinical trials issue to me, and it doesn't sound like a particularly LC-oriented issue to me. So if I'm there to pitch my, the reason I'm sitting there is to talk about, help us do the right thing by this particular area of research, what's the LC issue there? Okay. And like I said, this may not be appropriate for this discussion, but I brought it up because I think there is an LC issue here. One is, what do people and the research community actually know about this? And how would we be able to elicit information on what they feel are the ethical issues involved in some of these types of research and the way that that research is implemented? So what, Sima? This might get onto the controversial comments list, but I myself would feel you might want to be a little more open-ended if you're talking to Francis Collins about citizen science and not restricted to the LC issues, that it's the whole issue of citizen science and how NIH is going to grapple with it. The LC issues are incredibly important, they're part of it, but the LC issues may not cover everything about citizen science that NIH needs to consider. But this is an LC conversation that Jeff's going to be having. Where's that voice coming from? Okay. In here, yeah. Okay. And I'm being a little rhetorical in my, I'm doing this as a means to try to get the issues out because I'm not going to be there on the 19th of February, unfortunately, so it won't even be me having this conversation, and I don't think it will probably make it to that particular agenda, frankly. We're, in my mind, at two amorphous stage in this conversation to be talking about this quite yet, but if we were to help craft a research agenda in the LC area related to citizen science, okay, yeah, you get the point. For me, I'll just keep talking until someone else raises their hand because that's what people who are faculty do. We're self-selected for that skill. I'm really wanting a kind of case study analysis, frankly, to help get my head around what the issues are. And a few select really good cases that are different from each other, obviously in important and distinctive ways, I think is a really, would be a really helpful starting point. And if we can, maybe if you think that's right, I see some nodding, I see some people just looking at me blankly. But if you think that's right, then what would be some good concrete versions of those case studies? That, I think, would be a really helpful concrete suggestion. I see Dave and I see Karen. Jeff, can I just ask a quick question, do you think, are the case studies both descriptions of exemplary cases of citizen science, but, and also the ways that they've addressed, you know, LC in human subjects issues? I would want an example of a really kind of rich citizen science example where we could follow it through and look at the way it does all of the things that are relevant. Yeah. No. Well, I think you'd have to understand what other things are going on to really get your head around the LC. So that, yes, I think is the answer. Karen. We actually talked about that in our little last group, the federal government group, because I think that we could learn a lot contextually by looking at that from the perspective of the different institutes, perhaps, and is there a difference? So is a citizen science initiative in the context of clinical trials and cancer a very different kind of mechanism than it might be in mental health or something? So, and what are the similarities and what are the differences? I don't like to silo, but I mean, you could take neuroscience that might involve a lot of institutes together, but going from beginning to end, as I said, I don't think we only want to know the successes. We also want to know the pitfalls and what we could learn from those. Yeah. And that might actually generate increasing interest in the institutes in being interested in funding LC, because there's been a lot of disparity among the institutes at NIH with respect to their interest. I like that. I'm supportive of that. I like it. We have to nuance it a little bit more. Yeah. Jason, I would love to hear really concrete suggestions like this, I mean, so you offered cancer versus something else, but I mean, are there ideas for a good, robust case to look at, you know, name this? Not one that just studies cancer, but are there actual examples that one could point to? Is that a finished? What's that? Is that a finished? Well, or that are an ongoing. That are ongoing. Yeah. Or that are a network. Or I mean, there's so many different flavors here that's being part of the challenge. In the way that this back and forth between the two of us, I think points out that you may be thinking about it a little differently than I am, which is not a bad thing. Jason? And then Seema. I would, I would, I think I'm more experienced in citizen science around sort of general health than in severe disease and cancer and whatnot. And I think there's a really, for looking for case studies, we can have a side conversation about a bunch of the people who are doing, who are sort of transitioning from a quantified self in equals one to organizing a bunch of in equals ones to actually do more large scale organized formal research, everybody's doing the same thing sort of independently. That's, and there's going to be plenty of examples of places to look for that. I think I just have to voice a potentially annoying thing in terms of funding is in my experience a lot of citizen science, which is happening today. In terms of organized health research, there's a sort of funding that happens, which is shoestring, philanthropic, pioneering foundations, and the NIH is last. And I think it would be annoying, this goes to the comment of, you kind of have to have a conversation about citizen science funding in general and not just the study of citizen science, LC first, because I think it would be annoying to those people who are putting their own dollars into operating citizen driven health research enterprises and sort of paying their own way. And then somebody shows up that says, oh, I got a $250,000 grant to come study you folks who are sort of paying your own way to do this research on a bootstrap. And that's going to be really annoying if it sort of happens that way. So I think it has to be a packet in my perspective of how this works. It LC has to be sort of a component of a larger citizen science package. If you actually want real, if you want to be studying it on the inside, if that makes sense, I'll just read out that. I think you get it. Well, I'd be interested to hear from the folks from the federal community about that. Is there a way to think about this as a broader brush look at the way citizen science works? Soup to nuts is kind of the way I'm hearing you describe it. So don't just say, we're going to focus on LC and ignore all these other things that are important but aren't part of the narrow way you're seeing the world. Is that even something that's feasible to think about? I'm looking at. So it certainly is feasible, and we do think about it that way in this broader context, which is that one of the things that's intriguing here exactly is that it isn't a one size fits all. There's lots of different methodologies here. It's not quite clear which methods work for which kinds of questions. And all of that is very interesting. And I think those of us in the room here at least would certainly like to see it all move forward in a much more sort of concerted organized fashion. I think we do need to think about approaching some of these subtopics, though it is a vast, vast space and there's a lot of diversity here. And so it's why we've sort of in a way artificially cordoned off LC here and we recognize that that's a problem. But yeah, we are thinking about it in a more cohesive way here. But it's sort of, I mean, there's almost a way that you can draw the circle that's a little bit bigger and say, how does it work? Citizen science, how does it work? And then within that are the LC issues, but also some of the other more practical. Yeah, but it doesn't take on everything. So that makes, feels a little bit better. All right, I see, I see, I want to make sure Sima, oh, we got some of the fed folks. Why don't we do the feds Sima and Amy? What's your name? Carol, sorry, Carol and then Dave. Oh, it's actually getting back to the case studies. And I think we heard some excellent ones in this meeting. Denise Dillard, somebody mentioned the NARCH program. Certainly, what a surprise, the environmental projects that are driven by community are just really excellent examples of when citizen science works. And, and I particularly want to point to some of the research that tribal communities are taking on and have their own tribal advisory boards as being truly excellent examples of citizen science. Carol, and you need to come to a mic. Yeah. I'm not sure how this is going to fly, but I think there's a real cross cutting across NIH Institute's need to understand how communities view privacy and how individuals within those communities view privacy. And I think if you could pitch that as a sort of a case study of how citizen science modalities can be brought to bear on the concept of privacy, do people in different communities view privacy differently and how does that impact how we can fund research, the balance of privacy versus putting research dollars into research ends, and, you know, whether you're willing to abandon privacy to a certain extent in order to see research goals met. And does that differ across different communities? All of those issues, you know, I think citizen science modalities could be brought to bear on that sort of case, a case study on privacy. So that's sort of a plea for lots of citizen science, but focusing on one particular conceptual issue, right? So that's one, I don't know, whatever way you want to think about across, and then versus a very deep dive in a kind of case study approach. Maybe both of those things are worth trying to pursue. It still feels like, I mean, I think it's really important descriptive kind of work. It's very hard to think about prescriptive work before you understand what it is you're looking at and talking about. OK, it's funny for people who know me would laugh, because that's not the kind of work that I do, but all right, it's my own self-interest. I've got Dave and then Pearl. I was just going to say that, you know, at Genome, we do have, you know, the examples and structure of studies that are going out and doing, you know, new things in the case I'm thinking of, you know, sort of trying out genomic medicine and, you know, writing right along with, in each of those studies is an LC component. And, you know, and it's sort of a, I wouldn't say, naval gazing sounds negative, but it's a very sort of introspective, you know, look at how these studies are doing what they do and whether the methods are good and respectful. And so, I mean, I think it's a similar kind of descriptive sort of thing. And so there are models for that. OK, Pearl. This is probably more of a question than a suggestion. But as I'm sitting here, is ARC or CDC doing anything? Are these sort of almost institutes that have, I mean, what to learn from those institutions and that maybe this is very simplistic thinking, but that's closer to stargazing than a cancer therapeutic trial looking at, you know, how flu epidemic things and how to either collaborate with or learn from what they know. Certainly, CDC and PCORI and others are thinking in this space as well. So we're trying, yeah, as you say, to learn from them and also not reinvent the wheel and kind of move forward together, right? Yeah, well, there's a big ethics component in PCORI, of course. But this sounds like, I mean, it's sort of a different, I think, a different question. I don't know, is there stuff going on at CDC that's citizen science related? I mean, I'm on the PCORI regulatory committees. And the focus there is really just patient involvement, much more than the term citizen science I've yet to hear in a PCORI regulatory discussion. So I think we are talking a little bit further down the spectrum than at least where the PCORI... It's quite different. No, I think that's right. If PCORI represents one flavor, it's a very particular kind of flavor and there's all this other stuff that we haven't captured at all. So I think that's right. Does someone else have a hand? Sarah. So I wanted to maybe build on the point that Sima made. I agree that the examples that were provided about South Central Foundation and NARCH are two very relevant case studies. But I wanted a point that we could maybe tease out from those examples. I think something that would be valuable to do with case studies is to examine the different ways in which the federal government and federal funding is relevant to the conduct of citizen science. With those two examples, it strikes me. I don't know all of the details, but one is a much more... Sounds like a more homegrown, really self-sufficient entity that maybe isn't dependent on federal funds, South Central, that is. Whereas NARCH is a federally funded entity that's designed in a way to place the funds within the community, but it's very much a bridge and an explicitly collaborative model. And I think it would be helpful for us, for Francis, and for the NIH to understand where our involvement is most constructive and where maybe we have a role in stepping out of the way and letting communities do what they do best. And I think identifying some of those kinds of case examples can help us learn where our collaborations can be most effective at facilitating research. I think that example is so important because it shows how citizen science is helping to... There's a lack of representation across science of certain communities and those are examples of how citizen science can work to remedy that. But I think the NIH's role in that is an important thing to understand. So I'm trying to identify ways in which we can use these case studies to better understand our role as a way of advancing goals of justice and LC principles that I think are supported by citizen science. Did that... I rambled a little bit too long now. Yeah. Thank you. Anyway, yeah. No, I think we're all kind of trying to figure out how to articulate this in a way that's going to make sense. We're not... Obviously, we've worked to do to get there. Did I see Sima before and then I see Kelly's hand now. Oh, okay. Kelly. I think in the broadest buckets, of course, I think of just the ethics of citizen science itself. And so I think... And then I think some of us have talked about the ethical obligations to engage in citizen science and that that'd be just sort of a whole line of inquiry to push forward to say, you know, should we... Should the whole research enterprise really shift to be more participatory and do we have a social contract that really requires that of us? So to me, that is a series of ethics questions by Elizabeth. Thank you. Safe travels. So I mean, that is why I'm on the side that sort of... I guess I'd want to be careful even as we're identifying discrete issues and projects to work on that we haven't... I don't think it is not the case that everyone already buys that citizen science is either worthwhile or productive or there's a moral imperative to do citizen science. It's a threat. And yeah, it could be very threatening to a way of life. And one final little comment is we're setting a research agenda right now and we've shifted away from yesterday's thought about are there policies and NIH practices even within the NIH that would make a real difference here. For example, when some of the institutes required a community advisory board as part of every RFA, that suddenly became something that people got. And I think we could say did they do it in a good way or an effective way or a meaningful way? That's for the study sections and the community partners to say. But I think simply changing some of the requirements of what goes into our proposal and how study sections review proposals can go a long way towards changing practice. Yeah, so Coby, I see you and let me just say one thing and that way you don't have to hold your hand up. I think your points are really... The point about obligations to participate are really interesting in light of what's going on around so-called learning healthcare, right, and a kind of change in the way we're thinking about what happens to your information when you interact with the healthcare system. And the bright line between research and clinical care is where the argument is being laid out ought to be much less bright. And so this is part of that discussion it feels like which would be a nice way of including discussions of citizen science in what's already important in ongoing policy discussion. Thank you for the thumbs up. Okay, so Coby. Oh yeah, just following up on a couple points. With some of our work with CCPH, we talked a lot about big P policy and small P policy. And so a lot of the practice piece, like the institutional practices, we talked a lot about that. So the points about sustainability, dissemination, translation, having it as part of the RFAs, community advisory boards. There was a big push a few years ago to make sure that all, at least the Children's Environment Health Center grants, the Supreme Center grants, had a community outreach engagement core that used to be optional. And I was on the panel and I kept raising my hand, why is this optional? And so when I critiqued the centers, if they had a really weak COEC, I gave the overall center a bad score. Because to me, why are you doing the work? It's the why I gave. It's not having the impact and translating the action then. And now those COECs are no longer optional because of the pushback of people, experts doing the work saying this needs to be really represent what the community needs are and been responsive to those needs. Now, again, to the point of examples, the EPA has a RFA on citizen science sensors, sensors to do some of this exposure monitoring work. That's a good example. Some of the work of NSF and not learning in non-traditional settings, that citizen engagement piece, that's another example. Prevention of research centers. I think some of that work is an example. And the justice component of ELSI, I think the place to go is really with all the environmental justice groups who've been doing work for 20, 30 years. That's all community driven, community led work and a lot of examples of how that's impacted change in policy. So, minding that information and a lot of that information is already available at NIHS or also through the PEPH program and also through EPA's program. So that's data we wouldn't have to collect. It's already a secondary data analysis and maybe doing some interviews and looking at the best practices of those groups. I think that'll be a good place to start. And also for CIMMA, not to put CIMMA on the spot, but I'm gonna put you on spot, CIMMA. With the recent RFA that they had on centers of excellence in environmental health disparities which falls again up on the points about there's a lot of citizen science, CBPR within that RFA. Having smaller RFA's, not to have to be center level. R1's, you know, R13's, R21's, R03's. And that will be a place to start just to build that out more and bring LC and more into that sort of overarching mechanism. Helpful. Elaine, I'm watching the clock too. Right. So I just wanted Sharon Terry had to leave but to go back to our first session today and some of the thoughts she shared about experiences of playing different roles. And as both a mother of individuals with a disease, advocating for it as an advocate, as a leader of clinical trials. And so I think there's many roles that citizens can play and researchers can be citizens as well and play different roles. So I think trying to think about the issues as how we don't think about people in boxes but we think about the whole person and how they can serve different roles in our biomedical research enterprise and some of the issues around that would be important for us at NIH. So I'm blumping here. Motivations, relationships, roles, just as a catch-all for that. I think that's really helpful. Dave, and you guys need to decide when you want to switch gears, okay. I mean, just building off that Elaine, I mean, it was part of the same conversation depending on what roles people are, both traditional researchers and citizens are playing in the study, what rules may be applied to them. And where we don't have good answers and where there's gray areas, probably where the next generation of LC work might be done to sort of, okay, this might be a place where we need we need a little bit more thinking about policy. Policy will, existing policy will work in lots of places but that might help us figure out where we are. I see Kelly at the back of the room. It seems to be usually a sign that you want to do something different. I'm not, I'm a slow learner but not that slow. Right, well, I'm just sensing, if you want to close this out, I'm just sensing that we're, we have generated so much and I think the organizers, I'm like, I mean, I think our, I know my mind is full. And so I would love to give, I want to close, I just wanted to shift us into closing out as and close out relatively quickly. So we won't, we have a half an hour on the agenda. We, it is not going to be that, are you good, Jeff? Oh yeah, yeah, that's great. Thank you all and let me just add my, my thanks and kudos to Dave and Charlize and all the folks who work so hard to pull this together. Kelly and Kelly, actually, I don't think we thank, we didn't thank Kelly sufficiently. And Katrina. So we do separate applauses for Kelly and all the hard work she's been doing the last two days. All right.