 That was that was great. Thank you to everyone before as I as I still think about and try to formulate some of my own Thoughts on the the great ideas that were just shared I think I'll begin just by taking your questions while they while the ideas are still fresh and People can raise their hands. I'll try to keep some logical order to this. Well, you're up first Just muted myself This question is to pearl Sorry, thanks. Yeah, I make a luci a human computation Institute, so the So he says only that that Interested me it's something I probably should have known already, but I may be forgotten about over the years and that is the evaluation The different aspects of the IRB evaluation that is the the notion that that it's good science and the competencies of The individuals in terms of their role in the project and that was the part that really caught my attention because This this falls in the category of what's new And I think what's new is that as we consider citizen science Where there's human computation going on that is the interplay of humans and machines That can be thought of as a single organism with a single competency then it creates a Kind of a new potential paradigm for thinking about these things so as an example There was the the malaria spot where the idea is that you outsource to citizens who have minimal training the diagnosis of malaria from from slide imagery and The idea is that they conducted studies That demonstrated that if you have enough Different people evaluating these slides and then you aggregate those evaluations or combine them in some way that on average The result is more reliable than the same or better than a trained physician so Wondering what you think about the evolution of standards on this dimension I mean, I think there's a lot of room for evolution and taking the malaria as an example the malaria study as an example I think the question from an IRB back to the study staff or the PI would be you know And how do you validate this as a measurement and you would come back and say well? We have looked at you know nine out of ten is better than your pathologists So it's me it would be the same questioning of any Expertise that if you just say oh, I think it's a better that probably is not going to hold a lot of water But I I mean I think you know IRBs have to and I think they Do already look at new ways of doing things But we need is the justification for why you're making that statement And it can't just be we have to involve citizens therefore. This is it tough Which oftentimes is the response we get Other questions. Oh, yeah So Nancy Jones the National Institute of Minority Health and Health Disparities and I'm still struggling with it doesn't sound like citizen science is very much different than Community engagement in the traditional ways and Although the one part that seems a little bit different is maybe more a virtual relationship a relationship through the web Maybe you don't see people in person. So I guess with that too. I guess Is there anything different? I know we do online surveys or different things that we don't really have You know real connect, you know like human to human connection. It's more virtual connection But is there anything different about a virtual environment and then also? How do you control for people who don't? Present themselves as truly who they are Yeah I'll take a swing at it So to the first question if you a lot of it has to do with the taste you exercise as the person designing the study So it's it's fairly easy to build these kind of app-based studies that are that treat the participants like oil wells And you drill the oil and get the data out and you ignore them, right? And that's actually where a lot of the industrial work is going in my opinion a lot of the wellness and device apps That's what they do and so that's me. That's not citizen science. That's just large-scale enrollment And you can make choices throughout there that that make it more citizen e or less citizen e You know one of the things that that we thought was really important in both of our studies was the ability for participants to journal in ways that automatically the journal information was syndicated to the PIs Because that's a way for them to say things that we didn't design into the into the study Because by definition there's going to be things that we forgot to ask or forgot to talk about and so coming up with Short easy ways to sort of send those pieces of text out Makes it more citizen e because it allows us to surface questions anecdotes Interventions right things that might not have been controlled for in the you know traditional survey instruments And so I think to your first question is it's it's a close line right now But a lot of it has to do with design And making sure that there are channels both ways not just data polls, but communications channels On the on the ensuring that someone is who they say they are that's actually fairly hard to do and The the standardized tools for validating identity NIST has a level one two and three identity validation They are such that they could chill a lot of people from participating They involve photographs of your government-issued ID bank account validation Not a lot of people are going to want to do that To say the least and so our our response was to not worry about it and to just go at scale Because the motivation for trolling us is relatively low but as these scale as these Projects get more popular the likelihood that that someone has a motivation to troll you goes becomes a virtual Guarantee and so we don't have a good answer for that yet The hope is that that's where working with communities can help you because communities Both have the ability to respect Pseudonymous behavior, but also communities are going to be better at detecting someone who's faking it than the clinicians are in my opinion Clinician communities are much better at noticing someone who is not playing the role correctly Than the clinicians are and so I think that's that's going to scale better than Demanding that everyone take a picture of their driver's license and enter their bank account information to validate their identity Let's work our way down the road Pearl Yeah, and the second question regarding people faking who they are. I didn't think is sort of nefarious as you were thinking But I was thinking more on the self diagnosis self, you know, I have fill-in-the-blank disease I think this is where the Importance of truly having a validated diagnosis or type of person if the study really is Requires that that has to be an in-person situation where someone is making the diagnosis So I think it varies with study. There's somewhere. That's less important You know assuming their driver's license and bank accounts are all in order, but I think again not one size is not fit all Perhaps the other Another issue in the online environment where you cannot verify people or cannot have a face-to-face Is these sort of peer pressure we were talking about, you know You can't know if the people are truly wanted to do what they were doing Maybe they were forced to do that somehow in some sort of dynamics that there were in some sort of background that you cannot assess so I think would be very important to see how Again going back to the different kind of activities for some activities It would be fine to do it this way for some kind of research It would be fine to do it this way for some others. It won't be the ideal way to do it Well, and I think You know part of it is I think some of the research that as the type that John was mentioning is what we're seeing the most of Using technology and in that way the way I've kind of thought of it is that's similar to someone who Wants to participate in a walk or a run. It's an event. It's a tangible short-term Process now some of the some of the studies may have long periods of time, but it's still But kind of a task-oriented thing rather than full integration and engagement So I think there are different levels here that we definitely would want to tease out and have parameters around those different levels And just a couple quick final thoughts on that May I think to your first question? What I've seen is that engagement has tended to be a little bit more Reactive and I pick up on Deb's point that what I've been seeing at least is that there's a lot more front-end involvement in citizen science at the Design and the co-creation and you know in picornet. They really do sit shoulder-to-shoulder Designing just selecting outcomes together And then to your second question. It's interesting because I was reminded that's always kind of been a risk when I used to do a lot of Mailed interviews out on studies. You were never really sure if it wasn't the wife filling something out for her husband So I think it's just it's a different mode, but the same Potential risk that still exists. I've been tempted to fill out my husband's health questionnaires before he's gotten them in the mail I'll confess, you know So I think that you know and I was able to sometimes see that the handwriting in a mailed survey That was a couple years of follow-up changed over time and you you know So I think we it's kind of been out there ambiently and now there's just kind of a different ways of maybe kind of faking Who's actually completing providing data or participating. It's a great question. I see lots of hands up all of a sudden I'm gonna I usually start on this side So I'm gonna force myself to start now on the right and we'll kind of work our way around this way towards rich I'm Bob Gellman. I'm a privacy consultant I'm doing a project for the Woodrow Wilson Center on legal issues for cities outsource crowd sourcing for federal agencies My question is in an IRB context. When is a citizen scientist a data subject? If you're reporting stuff anonymously, you're all the way on one end I wouldn't think so, but if you've given your name or you're identifying information Even if you're counting birds on a beach Are you are you a data subject? And if you're giving information about yourself and your medical condition? Are you are you a data subject? That needs to be considered by an IRB I Would say if you're counting birds on a beach, it's not human subjects research and go and count. We don't want to hear about you I think if you are reporting health information under the auspices of research Yeah, you're a data subject and I think At least at current You know you would fall under all of the protections that You know we apply to that We we decided to just decide that we were Covered and that we had to go get consent and work with with an IRB for hours There've been all pleasant hours. No, no, it wasn't bad hours in the sense of not not with an H us hours us right We for our studies, but the we had a we had a wonderful IRB experience Like it's I think it's nice when you're at an unaffiliated regional IRB It's a little different than when you're at an institutional one But the one of the main reasons for that is is that we wanted to differentiate from a lot of the other organizations we're gathering human data and You know Gleefully not being part of an IRB like part of what we think we can do is demonstrate that the IRB community is actually not that difficult to work with But you've got to design your protocols in a way that you know is that recognizes them as a community of practice in the space You know if you sort of show up and say approve this It's going to go badly But if you show up with some humility and treat them as a community that needs to be a reviewer It actually went very smoothly for us in both cases I do think it is worthwhile also for maximizing the use of whatever data you get He's maybe for your current project to fall outside the regs But it's an incredibly valuable database that then someone who is under the regs would want to use So I think you know with any research you want to maximize use I'm Julie Brody from Silent Spring Institute and to kind of pursue this idea I wonder what your thoughts are about ethics review when the Lead investigator is outside an academic institution and I gather you went to a private IRB and we've done that we've also dealt with like a dozen academic IRB's and state IRB's But I wonder what the panel thinks about whether NGOs should form their own IRB's whether Universities should provide this as a service to their communities the way they let them use the gym sometimes Or or what kind of Arrangements you think Makes sense like the private IRB's are quite expensive for an NGO that's a really interesting question and I'd like to mention one example From the European context one group Where again an online platform people can upload the road genetic data And there's no profit to be made etc. And they wanted to do research and then they wanted to have it reviewed They wanted to be reviewed. It's not that they were trying to cut corners Which you know some people might think it's implied they went to a an IRB And university IRB that one of the people who was involved with funding this project was associated with as a student So the IRB said well, that's not our business. You're just a student as a student. You're doing something different You're not collecting data of people. So it's not our responsibility So the group was thinking so where am I going to go if I want to publish something the journal was asking Well, where is your IRB approval or in Europe as well? Where's your ethics review approval? So they were trying so we had a situation where they wanted to have review. They really wanted to There was no review to be Provided to them And they were not in a position either to go to some private IRB somewhere because they didn't have the funding But also at that point, they didn't know whether they should themselves, you know Create an IRB in that platform and who would be there how they were going to do that So it was a total vacuum and I think they've documented that on a paper I'm talking about open-snip that they shared that experience And so I guess, you know for some place for some of These communities to create They can't have what they want to have because the institutions that offer these Services either are not feeling their responsibility for them and they shouldn't or are they very expensive if they had to be You know going the private way So the possibility of having some internal body in those groups again, it would be probably one solution but Somehow we have to decide how these are going to be developed. Just say create your own IRB is again, you know Not very clear guidance. I think so it's Vacuum sorry I think key to the question is to understand how IRBs are Established within an institution We work through the federal wide assurance, which is an institutional agreement. It's not the IRB But it says that I promise as you know mass General Hospital that Any hospital that you know any human subjects research the fallows on the regs, you know We'll make sure it's reviewed by an IRB and it has all these reporting responsibilities which are not IRB but Institutional so for us we cannot we cannot review a protocol unless that person is somehow integrated into the institution If you're an external totally free of the institution Our first suggestion would be if it's appropriate get a koai who is institutional and then we can pull you in as a koai there or You know, we have not done this We would explore though the idea of bringing you in but you'd have to go through the credentialing all those things I talked about because ultimately it's not just the review of the protocol It's the responsibility of the conduct of the research and if we don't quote have any Tether to you, you know, here's your I'll be approval and you go when you go rogue We hold the water on that one water not being good in this case Yes, so something we found with with our institutional partners is the liability concerns affect Institutional IRB is in a way that it doesn't affect the the private IRB's This is a case where the government could do something there at the NIH could set up an IRB for these kinds of protocols and a fee waiver process for certain kinds of NGOs and you could even imagine developing a set of IRB filing processes that were more Acceptable or or recognizable to NGOs and one of the biggest problems that we have had was building the internal capacity to write a protocol and Figuring out all the different pieces of paper that needed to be filed to demonstrate that a protocol was sufficient If you say you just give us a consent form like man the consent form is the easiest part of the protocol Because the protocol has a semantic and a syntactic structure that's expected and so Even the vocabulary in the lexical aspect of getting a clinical protocol written can block a lot of innovative Community-based research from getting designed and so I mean this is the sort of thing that the government could actually step in and take care of it Could create the capacity for that Eliminate the funding requirements and create them and create sort of a fast-track filing process that was actually relevant to the community Actually brings up another topic that I thought was important to bring up and it gets to one of the points that Pearl made when she was up at the podium at which is Being Being able to be a part of the research is important I think there's actually something else that Could be done from an NIH standpoint and that's As a funder, there's more power there To help change the system and some of the ways that we need to do it and I've come to believe that a lot of our issues lie at an Institutional level so if there is a way one of the things that Pearl brought up, I'm sorry. I'm having to fade here Was that you know, this is not this is not a politically correct thing This is not why it should be done It's not why people want to be involved in the research itself We want to make the research make progress for people and so rather than it being politically correct or a checkbox Which it is being used as in some cases There could be some Reinforcement to the institutions who receive NIH funding to help us Incent the system in the right ways instead of some of the wrong ways that are being done now I forgot to put my gun Just behind you there. Yeah a couple of things that a couple of threads that I heard in this panel that I wanted to Reinforce is first was that the comment about what's in it for me and then just before that about vulnerability of researchers and I think that What's important there? Oh, by the way, my name is Eric Brown. I'm from the University of Maryland Baltimore County What's important there from my perspective is About cultural exchange and when I'm talking when I when I say cultural exchange I mean between the community of researchers and the community of lay persons for lack of a better word and you wonder whether or not citizen science isn't in a classical sense a way of saying lay person Science people who don't have scientific background as opposed to maybe a larger definition of citizen as we would conceive it in civics and so I think that there are deep disconnects between the community of researchers and the community of lay persons because there The one doesn't know how the others world works They don't understand that there are deep vulnerabilities that researchers have that researchers Have Can feel as powerless as someone said earlier as Kelly said to me earlier as they might feel in their relationship with the scientists that researchers feel as powerless in their relationship with the funder and Sharing those kinds of stories are great ways to build relationships because you know at the bottom at the heart of it The researchers doing a job and they have some of the same kinds of concerns that the community members have in their jobs and I think that that that deeper understanding of science working that way and not I think some of the the the Idealized views of science and scientists that researcher that I'm lay persons have it could be very helpful in developing and building relationships and Then the other thing that I keep hearing sort of on and off I keep hearing biases and privilege in Comments and it sort of reaffirmed to me the difficulty of being able to check biases Biases and privilege at the door as was mentioned By Elizabeth earlier in the day and if you don't think that community members can't read that right away They see that right away And it doesn't matter how objective you seem to be See that right away and and trying to suppress them and and act as if they that That the objectivity to which scientists Aspire is the reality as opposed to an aspiration I think again can result in difficulties and some of the problems that That Deborah mentioned about the larger world Outside of this outside of this room and then just lastly I Wanted to mention that sort of checking biases and privileges is neither fun nor easy because it so often requires the seeding of power to groups who generally haven't had any power in relationships and Nobody wants to do that. I live in the District of Columbia. We will never have congressional representation because no one wants to seed power and Recognizing that up front. I think are ways to support these kinds of relationships and I'm sorry. I'm going on just one last thing and that's about this idea Of sort of what's new and For me what's new and I don't like that. I don't like that formulation I think what's new is that what's been old hasn't worked extraordinarily well so yeah, we need to keep doing what we have or Iterating so that what we have done can be done better as Opposed to sort of looking for novelty, which is what researchers do Which is what researchers are incentivized to do which is to look for novelty So a bunch of comments less of a question But some things that I've been chewing on and maybe hopefully some things we can continue to chew on for the rest of the days Thank you Do you want to jump in on that point or great? If you can introduce yourself to I'm Simifin I'm from the National Institute of Environmental Health Sciences I just want to make the point that you we should not assume that citizen scientists do not have a scientific background You have a highly educated public. You have people who may have an Expertise in one type of science and get interested in a different type of citizen science You have a lot of retired people who are living longer who want to get involved And so please don't make that assumption that citizen scientists do not have scientific background. Oh Oh Could I just go I know just quickly you respond and then we'll have a response up here Yes, the only thing that I would say to that is given how specialized science is Even if you could be a wonderful Geophysicist, but you're a lay person as a geneticist So that you're you might understand how science works, which I think is powerful But in many ways you are a lay person in another area given how specialized the scientific enterprises Yes, and and even if they don't have a scientific background There's an expertise that community members have through their life skills through their professional skills through their Experiences that needs to be a part of the equation and I would just say while I don't think citizen science is right Lay person is one of those terms that we try to stay away from as well because there's lots of connotations of lay I won't go into but But it sets up the elitist barrier Even further between science and the communities that that hopefully will be involved So I can see there Pearl Sarah and then Sarah. I'll want to respond, but go ahead Pearl We'll start with Sarah then All of those were were really terrific and really provocative points that I'll probably think about on the long flight home to Seattle And continue to think about because I you what's really important And this kind of goes back to something Pearl said about how IRBs are part of a system The researchers are also part of a system. There's a culture of you know promotion and tenure is dependent on novelty Rapid discovery publishing in high tier journals all of that is going to put food on the table and to your point about you know If you if you decide to be vulnerable when you're working with citizen scientists or participants in your research projects It does mean laying that bear and trying to do so in a way that doesn't sound elitist or Again kind of reinforces the power dynamic or the asymmetry that might exist between two people But I think it's a really important starting point is so that the everybody understands Where each other comes from so thank you for those comments I don't know if this is on target, but since I'm sitting up here. I'll go anyway But as you were making those comments What came to my mind is we have historically done a very poor job of being inclusive We being like from the researcher standpoint and almost feels like I mean our lunch table We said the pendulum has gone all the way over when it probably should be in the middle I think one thing that we really have not spent a lot of time on her which studies are Really the low-hanging fruit on where citizen scientists can have a meaningful involvement. I think looking for a new You know biochemical dye for a cancer cell is probably not the place So I think we need to look at this in a very rational way and you know not make it a One you know one size fits all well, and I've I've been thinking throughout this panel about how I Appreciated the comments too on sort of laying out our Assumptions I think it's very important to pay attention to terminology and the different way that words have meaning to different people and How you emphasize even the syllables of a word make turn them into different meanings and I'm thinking about how we relate that back to the challenge of Governance and oversight of the research and and how we can either adapt existing structures like IRBs to the task do we do that by including members On our boards who haven't been represented before do we actually invite citizen scientists to become board members? And there's provisions for that in the regulations But I think we should really force push ourselves to think about what what it means to represent the public or any one particular Area of research and so that's one form of expertise. I'm also aware of lots of other Forms of governance and oversight Tribal advisory boards community advisory boards the creation of data access committees or other mechanisms especially to plug in in places where I'm going back to Pearl's comments to think that a lot of the research that We're talking about might not be Subject under the federal regulations to formal IRB review and oversight if it's not federally funded Or if it's involving the collection of de-identified data it sort of maybe Should be a cause for concern, but also an opportunity for some more flexible forms of governance So I'd really be interested as we're going around the room and hearing from those of you in the audience and at the table to Think about how do we actually translate some of these very important issues that have been identified into very concrete ways of thinking Innovatively to use Effie's term about governance and oversight So wanted to kind of put that out there to to redirect some of the conversation Towards that goal because we're open here and we're poised to thinking about we heard a charge to the NIH But I think there's a lot of opportunities here to think about concrete things we can do now I'm we have 10 minutes left. I want to balance I'm going to take the responses from my enthusiastic panel here But then I'm going to go quickly and make sure we get at least part of the way around the table to some people who have been patiently waiting so Quick responses from the panel so I just want to make a point that that governance isn't just The burden of the participant when they enroll in consent, right? It's the burden on the investigators and it's the burden on the data users as well And so a lot of times we use consent to sort of wash The data so that it can be transacted sort of seamlessly and if you de-identify it you can do that even without consent We do a couple things we first tell people that we're going to de-identify and transmit their data Even though we don't have to we obtain consent for what is already legal because it's a way of telling people What's going to happen which is important? And the other is that we have the ability to put constraints on the data users as well And so this is something that intermediaries and technological communities can do fairly effectively So we you know we let people upload data and share it in one of our Technical systems and we don't want them uploading identifiable information So if you want that key to the API to be able to do that you have to pass a 15 question test That's fairly complicated and inspired by the personal genome projects consent process, but it's on the data use side and So it's entirely possible to restrict access to this to some of the data to people who have passed an ethics test or an empathy test Right. There's an enormous number of controls you can do this that make a better deal back with the original participant And the funny thing is that it actually works, right? It's worked really well from a from a governance perspective on our on our data uploads Having the confidence that we didn't have to audit them. So when we think about governance Let's think about the way that we Apportion data to people to use not just how we get it out of folks, you know, because I would love to see You know every data user have to take a like a three or four sentence Oath like in a video on YouTube that was on their government funded webpage that said I agree not to be a jerk with someone's data Right, it would be incredibly powerful if every researcher who was using this kind of data had to take a video oath Where they said in plain language that they weren't going to attempt to hurt people They were going to take care of the data and they were going to be good stewards and they were going to share their insights back And that's what people want to hear But we and when we talk about governance, we focus so much on the burden on the participant of understanding this complicated process We have a fairly highly trained group of people in the research community who have incentives to behave the way the government Asks them to Right, we could require a Hippocratic oath for data relatively easily and that would change a lot of the system Just to work out that I think the the governance They just hope we don't focus as John said just on quote fixing the IRB the IRB is one stop and a long journey I think quality of research design You know funders holding people's feet to the fire. I think as I mentioned over here I think all of these in concert Perhaps may be something more provocative I'm wondering whether we can think also of oversight mechanisms that are different from IRB They're not if you're not in an institutional kind of setting and I'm not I'm not meaning only federally funded I'm just talking in general outside of institutions and we can see this kind of research occurring outside of institutions Can we think of a different model? Can we think of the a type of IRB that is not the IRB we know it is of the members of that community Now people often raise the question. Oh, can you have independent review if you know the same people who are having the project are also The reviews of the project perhaps we should remember that we came to that independent review Because we had a different way of doing somebody was doing research on somebody else And now we're my having a context where people are themselves doing research on themselves So we have to be sensitive to these differences and the second point I think we might want to reflect on is whether We have perhaps an opportunity here to develop tools that that those citizen science can use to Evaluate the ethical soundness of the projects. They are going to get involved with so consent is one model So we're giving the consent people will read it and they will decide whether they like this or not But do we have other tools like in science? We're saying we can give people tools to do science We're gonna give them apps to do science All those things can we think creatively about tools that can help people understand whether something It's you know done ethically or whether you know at the end of the day They have their certain obligations to men or certain duties that have to be fulfilled So that kind of brings me to my point, which was to talk about Training and and we've discussed that a little bit about how to orient not only the citizen science piece of it but the research piece of it because there is learning that needs to take place on both sides and Being involved in some of the areas that I've been including the cooperative group structure within NCI There are it's different between a general kind of education thing and there there is a level of Ethics that I think would be important on a general level, but there's also Specific applications right and that's where we get down to we can't have a one-size-fits-all There are specific things so it's not an either or it's a both and I think Being able to tease that out would be important and it's not only for the patient representatives or Scientists citizen scientists or whatever you want to call them it is also for the researchers involved and one of the things that I've done as a covert move is Constantly talk about and propose doing training for Patients and patient advocates when in fact knowing and people later coming back and saying gosh This is really good for the researcher involved or the you know Clinician involved or whoever it is this kind of information is good for everyone in the system So I'm gonna now return to the audience with Holly rich Sally Let's let's hear from each of you first to share your your comments and thoughts and questions And then we'll regroup and see if we have responses or more comments from the audience at that point Holly you're on I'm Holly PA from Dishon Connect We've heard a little bit about this from John and Effie just in the last discussion But one of the things I'm curious about is there's the gatekeeping that happens in many of these systems and Really trying to balance this idea of paternalism versus non maleficence in these systems where for example You may have individuals within a citizen scientist suggesting something that might be harmful expected to be harmful to the population and also the idea of Individuals who are very upfront about wanting to do data fishing and being concerned about type one error And those are just two examples that we've experienced in the past month So I wonder if you all have thoughts about this sort of gatekeeping role and how to do this appropriately and Still make the most of the citizen scientist opportunity Well, we're gonna hold that thought and make a note of it I'm gonna come to rich next to capture your thoughts very briefly. I'm rich sharp from Mayo Clinic I'm sort of pushing this issue of accountability a little bit further still There's also post-publication accountability as another element of research integrity and so forth that it's good That's particularly relevant here if I publish data that's been misrepresented as an employee of a place like Mayo Clinic My employer has all kinds of recourse available. I lose my job. I'll lose NIH funding I'll lose the ability to apply for NIH funding my career is over if I engage in that kind of behavior There aren't comparable levels of accountability that would be placed on a citizen scientist in the current Regulatory structure that we have that's a very different issue than simply thinking about IRB related regulation And I think if we're going to embrace greater forms of Increasing numbers of citizen science Activities we really have to be mindful of how we're going to create that system of accountability more generally Thank you Sally Actually a great follow-up to my comment and that is that There are those of us who are not subject to the regulatory environments and the constraints that many Institutions are who choose to be accountable Such as patients like me So I think it's important to recognize that there are those among us who are actually operating at a very high level of Ethical practice and legal practice and following HIPAA guidelines without necessarily needing to And so I think you could learn a lot from engaging with those of us who have been really trying to walk that line with you In order to be able to do work with you So we actually have a federal wide assurance because we actually have a contract with the University of Maryland We couldn't get it unless we actually went for that process We have external review IRB Relationships so we've taken that quite seriously and I think it came up earlier this morning about you know Establishing trust without that kind of attention that we've paid to it We wouldn't have the trust of the research world and nor would we deserve the trust of the patients and the citizen So I think you can learn a lot. I think from from our experience over the last 10 years We've had to go through all of these processes Naive to many of them and sort of saying well this doesn't apply to us does it and we'll be like well Maybe we ought to have it apply to us even if it doesn't because we want to be sure that we can play in the same field here But I think it begs innovation it begs for innovation. So This is great. Is there any there's been a real richness to the last three comments? I want to see if there's anybody that I missed who'd like to jump in and make a comment On the I just want to flag something which is connecting two comments the last comment Which is a desire to ensure accountability with the previous comment Which was something related to equitable access to oversight and a lack of it so I think those two things are connected and on the one hand demanding accountability on the other not providing avenues for oversight and in some cases if you're well resourced More than others you can go voluntarily find and pay for Oversight But I think there's a tension there Which and then to Sally's point to begs for innovation that addresses broader audiences great We're at the one one last comment, and then we'll be at the end of our discussion One of the things I just wanted to bring up is I'm Denise Dillard I work for self-centered foundation which is a tribal health organization and I wanted to bring up kind of the tribal oversight of Research as kind of one model that I think certainly could be applied You know I've often thought about like the African-American population or other populations where there isn't a There isn't a nation to nation kind of relationship And who's kind of overseeing the possibility of community harm and that's a lot of what a tribal review does is look at In my institution how Alaska Native people are Portrayed and depicted and a lot of it is like Deborah said words matter and how you describe a population and So much of Western medicine is very pathology focused And if you have a you know populations that are already Disenfranchised or stigmatized, you know the way that we write about them can be very stigmatizing So I just wanted to kind of bring up that point one other question that I had to and maybe somebody can help me with This is where this whole term of citizen science came from because honestly for me A couple of things come to mind thinking about You know ethnicity and other issues related to races, you know, how does this apply to like the immigrant population? It's just an interesting term that we would choose that and then you have American Indian Alaska Native tribes that are sovereign nations and so you know, I think the terms that we use Sometimes can open the door to conversation or close the door to conversation And I guess I was just curious about that terminology. It's a great question Okay, well, I think we may have an answer to that last question But I also see our moderator in the back who's reminding me that we're at the end of our time Can can we hear that answer real quick? It's an interesting history the term emerged in two different places to refer to almost the same thing But not quite at about the same time the more popularized definition is Came out of the Cornell lab of ornithology and in retrospect Rick Bonney who was the one who popularized it said I would have called it civilian science If I knew that this was where people were going to take it and the kind of controversy it would stir up So it's not like a term that anyone has been married to Accept that it's gotten so much traction that it's being applied really broadly and is has kind of eclipsed other labels So everybody who is in the citizen who's like central in the citizens lives world knows that the terminology is Really problematic and flawed, but we're almost stuck with it at this point. Oh Yeah, so the alternate Version was from Alan Irwin who is in science and technology studies And he was thinking of it much more from the ground up community involved in the whole research process as opposed to some part of the research Process so it's a slightly different take on it and at the upcoming citizen science Association He's actually gonna be speaking about it, which should be fascinating to hear Oh, great. Thank you I want to thank our panelist as well as all the people in the audience who contributed to this very rich discussion. Thank you. I Think I also want to add my two cents on the the terminology because our colleagues at NSF Just three months ago hosted a meeting on civic science and they were very specifically speaking against citizen science And so I think it's a it's a really important debate about what are we engaging and what are we taking up with that? So we're actually we're at we have a moment to synthesize and process what we've heard before our break So we knew we're not done with the day But we wanted to ask you guys to start thinking about what you're taking out of this because you've already heard you've heard Two incredibly rich panels and discussions. We had a rich morning. And so just to start thinking through the first two questions on this half sheet of paper and just take Two minutes to think to to jot some notes down for yourself We will be collecting these so informed consent that anything that goes on the page I guess if you voluntarily hand it to us, we will be collecting them. You don't have to put your name on it So what's so far? We have one more panel to go. So there's more to learn There's more to know but but so far. What is surprising you today? What's really standing out for you as as surprising and perhaps unexpected today? And then second what questions are you still chewing on? What questions are you chewing on? I'm gonna give you two minutes to to work by yourselves to kind of pull that together And then I am going to ask you to turn to a neighbor Preferably a neighbor you haven't really you don't really know and talk about though Just talk about one question that you're still chewing on With your neighbor for about five minutes and then you can break yourselves and we'll we'll come back at quarter tail quarter two So you can take your neighbor up to coffee if you if you wish for you to but share your questions