 So, we tried to synthesize and summarize first the stuff that we heard in round one, you know, just some of the major LC issues that we've seen in citizen science. And the things you'll see on this list, you'll see things you didn't raise or you won't – sorry – you'll see things you didn't raise and you won't see things you did raise. You know, we looked for things that kept coming up. So things that seem to sort of rise to the top are the new roles and the boundaries of those roles that come up as citizens become scientists and the idea that maybe they're defined by – oops, sorry – by what you do in the study, not who you are. The management of different sort of power differentials and how those differentials are of course shifting that goes to the roles. A lot of mention about the different sorts of resources that are needed for non-traditional groups these include but are not limited to access to IRBs, access possibly to legal and ethical consultations, of course financial resources, methodologies, technical expertise, all sorts of possible solutions that can be incorporated into citizen science studies. The issue of the digital divide and access to citizen science and to some of the technologies and equity and inclusion of understudied, underserved and minority populations is extremely important. We want to make sure that we are not exacerbating divides here and are in fact working in the other direction. Transparency of course came up lots and lots as well as it should. See about the goals and the roles of traditional scientists, citizens, participants, funders, those who are getting access to data and sort of how to encourage this, sort of culturing and maintaining relationships and all the communication that needs to go on to do that. Dissemination and other forms of dissemination and heavy emphasis on other forms of dissemination for both scientific and lay communities including sort of use of clear language, which you can see I'm expert at. And then a few others, sorry I didn't quite squeeze this on one slide, but citizen participation in grant review, so new roles for citizens not just in the scientific study but in grant review and participation in IRBs, training and education and this could be of, I mean there's so many different possible people who could be trained and educated about different aspects of citizen science, so that's a broad one. And again I apologize these aren't in much of a logical order, but data stewardship obviously very important and a subset of roles and power and things like that. NIH, I think it came up in several groups, favoring the idea of NIH establishing an IRB for minimal sort of, minimal risk citizen science and doesn't have to be minimal risk. I guess that was in our group, so I copied and pasted that from my notes. I like this one, just sort of the interface of formal and informal systems, which I think is going to increase in many ways and places and how to make those interfaces smooth and functioning on both sides and of course funding, which isn't necessarily an LC issue, although one can argue about that. So those were I think some of the main things that we, that you all came up with, we came up with as major LC issues in citizen science, biomedical citizen science. And my, before I turn it back to Kelly, are we missing, do we miss any biggies? Do you stun silence, Jean? I think maybe this will come out in the research gas, but I think the whole issue of conflict of interest I know came up in our group and I think that's, yeah, if I didn't put that I definitely should have, that came up in our group as well, what are the motivations? I think that goes along with transparency, but it is a, you know, a distinct point. And another one is intellectual property issue. Oh yes, I meant to put, I just forgot to type that. Oh yeah, absolutely. No. The other first slide. Your other left. I think that's it, right? Yeah. I mean one thing maybe to, for the first bullet, new roles, boundaries and responsibilities. Yes. Agreed. Yeah, and I think that's kind of the goal is to start putting finer points on these things, Jeff. So one of the things in our group, I don't remember this going by in your slides, but we talked a lot about peer review, which lives underneath publication, but also just peer review of the science, not just peer review of the results of the science. So sort of grant, grant review-ish kind of things? We talked about it as peer review in, yeah, across the spectrum, but part of it would be the process by which there might be peer review, or it's an issue really in the context of citizen science, more of the point. Right. I mean the idea that these, you know, no one, not no one, but these things aren't looked at with, we need some fair and good criteria for evaluating these. The same issues that we think it addressed by peer review in traditional science. Yep. Okay. So it doesn't seem to have a process within this space, so however you want to articulate that, but it's really an issue to be captured rather than, you know, what we do about it at this point. Sorry? Who are the peer reviewers? Right. Who are the peer reviewers? I mean, it's all the things that would be unpacked or spin out of that general category. I mean, another one on the goal, maybe it goes under transparency with the goals and roles is also clear expectations. Someone tell me where I put that. That was on the first one. Expectations. Yeah. Yeah. Being clear about them. Yeah. Kelly. I think we had, we had an issue that was redefining impact factor. So being, and I don't think that's captured yet. Yeah. No, you're right. It counts as a benefit and for whom, but I think just that tagline of impact, impact, impact factor, which impacts matter for whom and can we re, can we reorganize the current impact of the impact of the research, the other outcomes besides publications. Yeah. Is that, is that sort of, I'm so glad you're watching me type. It's my nightmare. Let up. Pearl. Oh, that was your, that's your water. I was just commenting on your typing ability. Oh yeah. I saw your hand with the water and I was like, that's other, other, other things and we can, you know, sub bullets, refinements, perhaps, David, if we go, if we look at the gap, I know for us, the identifying issues and gaps and research questions were all kind of intersecting areas. So perhaps if, if, do you have the second round, that's why it's not ready yet. I was going to make you talk for a little bit while I made that sign up, but no, we, we, we have some of it. Yeah. Here, let me just grab my, because I think if we get out what you have and then I would also love to ask people just to notice, what do you notice about the kinds of issues that we're surfacing here? Are these, I think, are these what kind of, what is unique about this, this domain of LC research? Or are there particular issues that just get re-centered because we're talking about citizen science? I mean, are there, you guys are all kind of observant people. What, are there things you're noticing, we've just made a list of themes and issues. What's, what do you notice about these, these themes and issues? Are these different ones that I want to show you? No. This and this. Yep. Oh yeah. Those are, those are, those are me typing my stuff. I just put, I just put the summaries up at, at the top. So sorry. Here. We'll. No. This is, I know that, it's that we a lot of have access to all the data. I know we do. Yeah. Yeah. I won't de-eat it. We only get access to the aggregate data. Exactly. Kelly, to respond to your question, the thing that strikes me is it's still definition issues. We're still trying to find who is a citizen scientist, who's a citizen, who are the rules, what is it? So it's still an awful lot of just definition or clarification. I mean, I think that kind of runs through it for me. The other thing is, and we talked about this in some of the groups and I don't know how it, where it goes, but infrastructure is critical. Yeah. Yeah. So do you want me to try and, try and summarize round two? I think one, one other point I might make is that I do, and I think this is a question to the group about priority setting, is that I, I think often, especially empirical ethics research is sometimes is very descriptive and, and clarifying, we're clarifying concepts, we're clarifying definitions, we're describing how things are. Is that to do the def, if what, what Pearl just pointed out, this is, we've got a lot of definitional work to do, is that sufficient? You know, it's, it might be necessary, but is it sufficient? And, you know, would, if we, we could reverse priorities and say, okay, we, we agree we have some definitional work to do, but, and perhaps we need to start, do we need to start, is that where we need to start or do we, is there some other action place we need to be, and can we do it as a both and as well? So I would just ask, as we think about priority setting, I would just ask those questions. Yeah, I mean, I think in terms of definitional, one of, one of the suggested research priorities in round two is, you know, sort of ethnographies and, and typographies of biomedical citizen science, and, and, you know, and digging a little deeper, maybe what resources and issues each, each of those requires and incurs. So, I mean, that is very much definitional, but would clearly benefit a lot of our thinking, I think. I think one other observation I would just make is that this, I think a lot of us are talking about culture change efforts and how this, this really isn't kind of tweaking around the edges or, I mean, if, if this is real, and if the real change that we're talking about happens, we're not, I mean, it could happen as a fringe effort on the side, or is this really transformative work? And if it is really transformative work, then it, we're talking about culture change effort, meaning cultural change, structural change, process change, policy change, practice change, and that all of that has a science around it of how to do that kind of work. And so I feel like I might, that might be in our, that might, I don't know which round that shows up in, but that's, just want to flag that as a theme that, as, I think what I'm noticing about what's surfacing here. So Kelly, I mean, you say there's, I mean, not you say, but you said there's a methodology to that culture change. I mean, that to me sounds like round two in a way. I mean, I don't, it's, I mean, if I, I don't know what the methodology is, but it seems like it would be somewhat gradual, not all at once, or maybe you try everything at once and see what works or many flowers bloom kind of thing. What do you, I mean, I'm, I'm asking, not telling. Well, I just wanted to add to that, I agree with that. I think maybe one place where the LC contribution might be, be able to add to that is that I think there's some work that I don't know if you'd call it descriptive, but I would, I would say it's, there's some normative work that has to be done as well. And, and I think in parallel to the more empirical what works, what doesn't work, because I think there's normative issues to be addressed that would help sort of set the stage for defining what does it mean to work, what is successful, what is good, that has to also go on. And I wanted to add to that the part of what I think you're referring to is behavioral change research and social scientific research. There is an S in LC. I would also remind us about Jen Fishman's talk yesterday, that their push is here, that this is not just within institutions deciding how they're going to change cultures, that there are other cultures developing outside of institutional frameworks, and that they're going to push what institutions do and the speed at which institutions have to do them. So this is not just our ability to decide how we wish to change our culture. Yeah. I think David, as you're struggling to summarize what I was saying, I think I was intending it as, you know, what, what are we noticing about the issues that are coming up? So I was in, I wasn't intending to add it as an issue. Oh. It was, but I would leave it there because I do think it is an issue to say, are we going to do this for real? Are we going to act like we're doing it? Made a question going on, too. It's like, and who is, who are the, who's the we, right? Because we've been talking about citizen science, but is there citizen science around the LC issues, too? That, I don't think has come up in the course of either of the two days we've been in the room. Jeff, that came up in our group where someone suggested that, you know, asked, well, could the, could citizens be the ones to tell IRBs and study designers what's ethically acceptable in, in research? I don't, I don't know if that's exactly what you're, yeah. Somehow we need to capture the Gestalt, right? The, whatever's going on around how LC research can be done in the same kinds of way as we're talking about citizen science or whether that's even an appropriate thing to say. I would say yes, but that's, that's kind of a question about a question. I mean, do you think like empirical like surveys and things like that where you're asking the citizenry, what they think of, is that, is that some form of that? I'm not sure, I'm not saying. Surveys, qualitative interviews. Yeah, I mean, it can be either both, Leah. NSF's SBE program is funding some research, or trying to look at how citizen science approaches could be used in the social sciences. So that is being discussed. I think it might help us if you can, if you can go on to summarize just the other, because I think the rounds were all integrated. So if there's anything you have of themes from the others that, because then I think we can have a full discussion. Sounds good. Thank you. Yep. This is incredibly hard. You guys worked really fast. So I really appreciate what you did. So round two, this is more, this is more of a bit of a laundry list, maybe a little harder to summarize in big bullet points, but issues about how we scale some of the LC work, how do we scale building and maintaining trust, returning information, thinking of research about how the current regulatory system facilitates or gets in the way of citizen science, work on thinking about, I think sort of what Jacobi raised about sort of colonial and imperial nature of research. I know I'm missing the third one. Yes, you're always briefed, Rob. Yeah, thank you. Sorry, I forgot the racism. And sort of how incorporating new stakeholders can transform some of that. I think there's a lot of interest in methods of dissemination and implementation and how to work about building capacity to understand data privacy and stewardship. So some how to work, I think. Comparing transparency in citizen science and other types of science to see if citizen science is really walking that walk. You could apply that to other things as well. Studies comparing different governance structures, evaluating different governance structures, evaluating structures other than the IRB that could be used to vet studies. Other forms of gathering input on, I can't, I'm not sure what that says. Sorry. A host of possible questions about data stewardship and ownership and the implications of broadening or changing those rules. What would different, what would be evaluating different models of peer review? What happens when you introduce different sorts of stakeholders? And then some others evaluating reporting dissemination of results in dispersed communities, that's I think scale. Can you tailor dissemination based on personal preferences? Questions about the effectiveness of different new models of consent? Would it be interesting and useful to involve IRB managers and members upfront in study design, sort of interacting with citizen scientists to help everyone understand one another? What sort of long-term, research sort of on long-term considerations and sustainability? Conducting actual empirical research on whether citizen science increases or changes privacy threats? We could develop clear legal guidelines and checklists. Questions about representativeness, that's not very clear. I wrote it and I'm not. What can we do to increase access to emerging tools, sensors and technologies? Again, creating equitable access so the communities can do their own monitoring. Do we need newer different privacy standards? I mean, there's lots of sort of possible evaluation of process work to be done. I think that's a lot of it. I wouldn't say that's all of it, but it's a lot of it. Entrepreneurship, lots of thoughts about dissemination, models, tools, study results, methodologies. Sorry, it's a bit of a muddle in my head. I'm sure it is in yours too and I'm not sure that was clarifying at all. Yes, yes.