 So, thanks very much, everyone, and I have actually a lot of questions, but I want to first turn over to the audience any questions that you might have, Sarah. Great. Thanks. This was terrific, and I loved how just kind of spontaneously the presentations built off of one another. Maybe you planned it, but I sense there was a little bit of spontaneity going on there. I had three comments. The first is, I think, relates to something Laura said, the notion of publishing all of the results of a study being an obligation. Principally, I agree with that, but I think one piece of the dialogue that might be missing here, and this goes back to something that was said yesterday, is that the citizen scientists, people who do not do research for a living may not understand the problems with getting published sometimes, like the lag time and the, I don't know if there's any journal editors in the room, but that might be a dialogue to open up about how can we do more, if you will, free the data, free the research, free the results. So, that was my first point. The second one is about this notion of learning from mistakes, which Laura, I think, might have also been a point you made. It struck me as really interesting as scientists were, you know, there's this, you know, inherently they learn from their mistakes in the research context, but in terms of the process context, I think there's still that tendency to keep doing things the way that we've always done them, so there's some humility that's required to say, we didn't do this right the first time we had, you know, patient advocates joining us, or citizen scientists, and I don't know, we don't have a better label yet, Sharon, so I know we're trying to figure out the right word, but you know, that humility to say, we got to do this better, and we want to roll up our sleeves with you, so, you know, I hope we'll get there at some point. And the third point goes to something Denise said about this notion of presenting things in language that everyone can understand. Work I did at Group Health several years ago showed that writing in plain language is not second nature for scientists, it's like you have to use a different part of your brain to write in a way that is more of an accessible summary. So I thought that there's an opportunity where the citizen scientists could contribute on that piece of writing up studies in a way that is more accessible and isn't, you know, kind of the science speak, and so I would love thoughts on any or all of those points, the plain language, the learning from mistakes, and how do we kind of surmount this problem of getting all of the results out into practice in a timely way, so thanks again. So I'll just make one sort of frame-setting comment, Sarah, and then it'd be great to hear from everybody on those points. What if we shift the center even in our conversation today, and that is, instead of saying, you know, I wonder if citizen scientists know that journals lag time is long, who cares? Let's not have a long lag time. Let's have new journals. Let's have different ways of, you know, being that part of the system. So we've spent a lot of time, and I think even citizen scientists have spent a lot of time on the research part. Let's also change the after research part. And similarly, you know, that scientists would learn about plain language from citizen scientists. Let's shift the center again and say that isn't the goal entirely, and figure out the way to bring that together. And the other is how do we have them join us, meaning join traditional scientists. And I'm not picking on you. I know that this is fine for you because I know you. But let's shift the center, like let's bring the center over to the center and see if we can have the conversation. OK, yeah, thanks. I didn't hold it. OK, so I'm also, thanks for those comments. I'm also on the board of community campus partnerships for health, CCPH. And so a lot of what you're talking about, you know, we focus a lot, and part of reason I'm on the board because the folks I'm on the C, the community part of the C, and then just moving in more like you were just saying, to really think about, you know, the different ways of knowing the different community knowledge systems. And it's like just to say they're out there, they're there. And so it's not like it's what they can do for us. It's like what we need to do in a more collaborative fashion to recognize and value that those things are already out there. And as part of CCPH, what we do, you know, we've done a lot with our mission statements to talk about, you know, authentic, transformative and equitable partnerships. And what does that mean? And you see all this citizen science out there and bringing those principles into play. And then we actually have what we call CES for health to allow for these other products. So you don't have to lag time. You can publish your protocols. You can publish your photo voice. I do a lot of photo voice too. I didn't mention that. A lot of photo voice, a lot of these other types of, you know, promoters work can be published in CES for health. And we're trying to make that another dissemination outlet, but it's kind of worked. So these new journals, we can do it because they're models out there. And another journal is Progress Community Health Partnerships. They actually have a segment in their articles where it has to be something that's translated. So you have like your regular article and then the community partner will write their own article to go along with the regular article. So that's another mechanism that provides sort of a structure and a template that we can use, a model that we can use for this. That's great. I mean, I think the challenge I think is changing the currency and academia to value CES for health on the same plane with New England Journal. You know, that would be a win. I think that last point you just made is really important because if we don't, I mean, Silence Spring Institute has made a decision to publish in really high impact journals. And we've done that because not only of the impact in the scientific community, but because that is the way you get on the front page of the LA Times. So I think to say, well, citizen science is going to go this other route can be devaluing. And we have to be, it may be that, yeah, we need some other channels, but we also need to really push ourselves and push our skills to get into the high impact channel. Is someone going up a tenure promotion? Yeah. OK. OK. Is someone going up a tenure promotion? The point you just made is very important. And the point you made is very important. Most of my work is not published in New England Journal. I'm not a medical doctor. I'm not an epidemiologist. So for general public health and environmental health, our basic impact factor is between one and three. EHP is 7.92 or 7. Wherever NIHS, y'all know what it is. But you know, that's the highest for environmental health. So you have to have these other channels because the work is not being counted. As Thomas Ruder doesn't invite you into the Good Old Boy Network with your journal, then you're not going to get an impact factor. Progress Community Health Partnerships is the best journal for, I think, for CBPR. It has no impact factor. Environmental justice. I'm on the board of the editorial board for environmental justice. We're a subset within the subset, within the subset of environmental health. It has no impact factor. Most of my papers and journals that either have low impact factor or no impact factor does not mean that that work is not valuable, right? Does that mean that work is not valid? It means that the system that I'm in does not value the work, right? It only values New England Journal. It only values JAMA. It only values science. And if you don't get into that, then you're not a good scientist. No, that means you're not taking account of the scientific inquiry, remember, the five dimensions, the commodification, the first dimension, you're not, that's where all the value is. It's not in the other dimensions of engagement. It's not in translation. It's not in teaching. It's not been transdisciplinary, right? So we have to push the whole system to be better. I think that's a big point. We have to push the system. And I think citizen science is a paradigm that's pushing the system. And we need to provide those outlets so that work can be seen in ways that, even with popular media, that we can do more using social media. And citizen science is disruptive technology. And so we need to see it as that and basically support it in that way, right? So I'm a rubber meets the road kind of girl. And I'm Chicago, it seems like the real value is how the people in that community is impacted. Bottom line. As far as plain language and simple language, one of the things that I did for several years is train new docs how to write their first clinical trial protocol. So, and this happened in a group situation over a summer, it was like protocol camp. So there's lots of tools out there. There's lots of tools online, even in your Word document, you can go in and do that to get a reading level. So there's simple tools. There's lots of tools. It's just getting it down to the level where everyone can use them. Yeah, no, actually I was bringing that up because actually not shameless plug time, but I did actually work on a readability toolkit that's freely available. So I agree with you, but I think it was just, we had so much trouble getting the traction for it. Right. The people didn't think they needed it, they knew how to write. So I have Pearl, Sally, and then Mildred. Hi, Pearl O'Rourke. I think that, I'd love to hear comments from the panel and the audience on this dissemination of results in that I think this is all research. This is not a citizen science issue. And I think there's this issue of responsible dissemination. And yeah, there's obviously, there's concerns or there's initiatives going on, the EU now requiring or promising they're gonna require individual level data to be publicly available. Huge issues about privacy. What are people gonna wanna be in research because of that? Issues about having summaries that are not promotional. Who is gonna write that? Concerns about the context, it's one study. How does that compare to the other 10 studies on the same thing? Is this the Hallmark study or is this the one rogue study that no one should look at? I think the power of blogging is a little bit scary at times. Some of the websites that we have been asked to look at that are both resources for advocacy groups as well as recruitment tools. Very difficult, which is which. So we've had to like paw through an awful lot of what the research pieces or the resource pieces. And it's a little scary that it's a study totally out of context. So my concern is this that generally, I mean, if Elsie wants to take this on as an issue, this is not a citizen science issue. I think responsible, timely, and contextual dissemination is something that we are lacking in across the board. Just a comment and a wine, I'm sorry. I'm sorry, I'm taking over your moderator role. But my friend Deb has spent quite a bit of time on writing study result summaries. And so she's raising her hand too, so. Well, I was just gonna say there are some efforts in that. And it could definitely be an Elsie issue and something that the Elsie community could get more involved in. But for example, the Harvard multi-regional clinical trial group MRCT had this last year as one of their projects return of results. And I was one of the co-chairs of that group. And we've, so there's a guidance document and toolkit that's going to be publicly available I think next month. We're in the final draft of it right now. So there are efforts going on and they address the EMA issues and requirements and there was a multi-stakeholder group involved in that. So that's something, and I agree, it's not for this meeting but I just wanted to bring that up because it's an incredibly important part of what we're discussing now. So there are efforts that I think the Elsie community could join and that would be important. About plain language study results on this, getting that on all studies and all phases from all sponsors. And as far as context, I can envision this as part of say the informed consent document for the NCI has a required section on what is the usual care and why is this study being done? So there are contextual pieces in there that help to, yeah. No, I mean, I agree we do it in pieces. When I was thinking context, I'm more, it's updating. It's hard enough when you're even very interested in a very discreet type of research to keep track of how it is evolving over time. I see. And it's that context that I think is difficult. I don't know, maybe impossible. And it needs huge resources. This is something that's really central to my interests and work. So I'm thinking about it in the context of environmental chemicals and body burden and what's in your house. And this issue that the knowledge will evolve is actually one of the reasons that we felt it was important to report to people on their personal exposures. And you also asked about context. So I'm gonna say that first and then come back to the evolving knowledge. So I think it's a profound responsibility of the collaborative science team, which may in citizen science contexts be kind of large to grapple with those questions of context and provide that knowledge when they report results. When you give people their results, that enables them in some way to keep up as the knowledge evolves in the case of environmental chemicals. So in the past, it was common ethical practice to report to people only if their blood level exceeded a clinical guideline. But there are no clinical guidelines for any of the chemicals that we study. And in fact, the clinical guidelines have changed for lead. They just keep going down. So one of the reasons that we decided to report results fully is so that people would have access to this information in the future as knowledge about the implications evolved. We're trying to figure out now what is the responsibility of the team to keep in touch with people. We did have a situation in one of our studies where there was no guideline and then a guideline came in. It's not like the lead guideline, which is a clinical guideline. It's an environmental health guideline. But we did end up reporting to those people a second time when the guideline came in. Okay, I have Sally Mildred. I'm sorry, I forgot your name. Nancy, Carol, Eric, and then Sandra. So Sally. Wow, I feel fortunate. You know, one of the things that concerns me, and I really appreciate the conversation. It's been very enriching. But one of the things that I'm concerned about, and maybe NIH needs to take this into consideration, is just how little funding is focused on that phase of the research. So, and I'm gonna add a couple more here. And I think maybe we need to be calling for, and it isn't just about citizen science, obviously, but I think when you start to engage the public, then there's certain expectations that rise. And I think those go to justice. I think there's sort of sentences of social and ethical components of this that are really important. So it goes beyond just disseminating results. It goes to implementation of those, what the next steps are, but also two more that I really think are important when you start engaging the public more robustly. And that's scalability and sustainability. I am not a research scientist by background. I'm a clinician, mainly in the community. And I can tell you how frustrating it is to see research applied within a community setting, only to find it fall short because you can't sustain it. So, you know, there are demonstration projects that look great in their first three years, they go robustly well, and then suddenly you've got everybody kind of excited in the community about it, and then you can't keep it going. So I think there's a responsibility to start thinking about, you know, how can we maintain the robustness of the project in some way in action steps that actually the community can look at and say, that's what we did, that's how it worked. And this is what we're doing now as a result. That evolution of knowledge I think is really important. So I think NIH should step that up. I think we need to have some more dedicated understanding of what does it mean and what's the cost, who does it? Where in some of it may be people in the public who are doing it and they may need to get something in return for that action. So that would be my soapbox. I just wanted to say, you know, I totally agree, and I think the other thing that funders often don't pay for, although they may even have it in a funding announcement is the community engagement process and even developing the research question or the design. You know, you get this funding announcement, you're supposed to respond in 60 days with something that you've used a CBPR approach with. I mean, that just doesn't work. Yeah, so I agree. And it's really interesting, so I mentioned some of the work that we've done with wear and with LAMPSE. So the output of LAMPSE was C-CRAB. So the sustainability part was this organization that will move forward with a lot of the work that we've done in the project. And another part of that was, you know, we wanted to, we really wanted to provide more funding. So it's all the panelists talking about this whole capacity issue. And so we've had these discussions with NIH and also with NHS over the years about, you know, having different types of grants that, and it's not just on NIH, of course, but as a board members of CCPA, we talk about our peeps, the people who are our customers. And NIH is a customer for us because it's a partner. And so any federal agency, any foundation that's provided in funding, they have to be at the table to understand the whole structure and what's the, you know, what's sort of the continuum. Right? And so as it relates to the continuum, there's no funding for some of the continuum. And so what happens is, as you said, you have this excitement and then you're left with limited resources that have been extracted out of the income support from the partners. And it ends up, could end up messing with the trust, particularly when you use a CBPR paradigm. And so I think more needs to be done. And I know NIH had a dissemination, was it a dissemination conference or institute a few years ago, or was it a year ago or two years ago? So I think that's, we need to go more into that as it relates to dissemination, but again, the translation, the action, the sustainability. And you ask for it in the grant, but then you want, we put it in the grant and then there's nothing there to go, you know. And then to get to your point, Denise, about community engagement, one good thing that the EPA has done recently for all of their star grants, there has to be a community engagement plan that goes with their star grants now. And they also in the process of trying to modify the peer review process to make sure more community members actually under peer review process. So CCPH was involved in helping the EPA do some of this. And so you have again, these examples of how we can move forward with changing our structure and our processes. But I think that sustainability funding, different types of grants are needed. And we have to make sure there's just not the short-term R03 or R21, and it goes away. We got some results. We pile it, hey, hip-hop parade, you know, we were happy about it, and it's going again. And then five years later, we get an institute director change and we go back to it. It's like, you know, I'm seeing my tail again. Why am I in a circle? We have to get out of that circle or, you know, a process. So. Can I just one follow-up thing on that? My expectation would not be that NIH takes that on completely. I think that there's a requirement of creative collaboration in the assessment and planning of the grant itself that says, who are our future collaborators that will help us keep this going? And it may be that it's just other funding sources. It may be that it's community resources and things like that. But just so that no one thinks that I think NIH should take all that on, but they should have expectations about it. Yeah, my comment was actually just in complete agreement with all of those things about dissemination because I actually think that NIH has a lot of leverage here. I mean, as you say, they always ask for that little section in the grant at the end about dissemination and there isn't any funding and nobody ever does it. But in Canadian grants that I review, they have a much more robust section on knowledge translation, which includes the translation to action that you talk about and so forth. And it is much more part of the budget. It's more part of the actual part of the, it is part of the research. It's not looking at it as after the research. And I think NIH not only can do a lot more to actually sort of put the money where the mouth is there, but also that it is absolutely an ELSI issue that is not unique to any particular research. I mean, it is actually unconscionable that all this money is being spent and more than half of clinical trials are never published. And in fact, more than half of clinical trials never accrue a single patient. And how that gets perpetuated, why are we still funding the same people to not publish their research and not, I don't know how that's happening, but I don't see why NIH can't do something about that. And maybe funding needs to change as well. Nancy. Nancy Jones, National Institute of Minority Health and Health Disparities. I wanted to go back to some of the models of Mr. Wilson, where you had like the different types of what you saw science doing in the last part, you know, like really linked to advocacy or using data to rule, right? Getting to the point that you move it, but bringing it back to the LC2 is that there's also conflicts and roles. And, you know, one of the things that we do, you know, like in the epistemology of science or how we know what we know, how we believe that we know, I think Pearl sort of talked about the difference between one study and a body of knowledge and, you know, when is it actionable? But then there's also the concern of objectivity, you know, that when you really want to use data to rule to make decisions and to be actionable, then you are biased and so then, you know, that puts science intention. But there's a difference between, you know, like hypothesis or making links versus when it's more diagnostic or, you know, the body of knowledge can support now, you know, like what is it at the local community? So it's at a later stage. So there's, so I guess what I'd like to do too is that when you think about those different models, I think then there are different constraints where, you know, like earlier in the process, it's really even more important to have, you know, actually, you know, constraints in there to keep it more objective, like peer review and where you get later in the process and it's consistent with the body of knowledge and then you can have a different role and putting it more in action. You know, there you're more concerned with the technical precision. Was it done in consistent with the science or the techniques that were there? But like for me though, I like how you tried to come up with different phases of where you see it being used. But I think, you know, understanding with the constraints on, you know, like the citizen is in those different places, you know, is important to be able to use the data in such a way that you don't get yourself in a bind because, you know, all of us know in science it's really easy to get blinded to your own theories. And so I just wanted to put that out there. You know, those are good points. I mean, I think when I talk to my students about objectivity and sciences there's no objective scientists. You know, we all have biases. I mean, you know, I grew up in a community and there is sewage treatment plant. My dad worked at the Grand Gulf Nuclear Power Station. He worked at coal far plants. You know, that's why I do what I do. I mean, that's not, that drives me. You know, that's my lived experience. Your lived experiences really inform what you do. And then, you know, but I think your point about peer review is important but then who are the peer reviewers? Right? Because I mentioned in my previous comment, we need to have more community members who are the contextual experts. It's peer reviewers. I do not live in your community. I do research. I'm an outsider who goes into research. I still go back home to my home in Bowie where there's none of those concerns, right? Community members don't want it every day. So as a contextual expert, they need to be peer reviewing this research to make sure that it's valid, it's representative, that it's actually gonna be useful. So I think the utility component is important. And so, but I do think, you know, there are things that can be done as we go through the different phases also across the continuum of community engagement where we have, you know, roles and responsibilities to define, but again, there has to be sort of this give and take process of it. So, because your roles responsibility can change as assistant scientist and also as a collaborator on those projects. Some projects I facilitate, some I'm just there for technical assistance, right? And so one of the models that I didn't mention that we use. So we have sort of this commerce one model is sort of the whole data research ownership process. It's another model that we use called collaborative problem solving, which is really about the translation of actions. So we end up having work groups. So we have a health work group. We have a sort of revitalization work group. We have a policy work group. And so these work groups take the data and they use it, right? And they work with other partners in the community to figure out how the data can be used for action. So we use that structure in helping us think about solutions and implement solutions. Can you move to a microphone? So just an observation. NCI currently has a clinical trial just focusing on exceptional responders. Those are people who had an exceptional response in a study drug that otherwise didn't make it. And so tracking down these people has been horrendous because basically the clinical trials are never published when the results are negative. And so I think we've hit on a really kernel issue here where even though it's not unique to citizen science, citizen scientists could play a huge role in getting these results out. Maybe not in the New England Journal, but that's the beauty of, you know, citizen scientists aren't looking for promotions and tenure. So I think there's a real role here for citizen scientists in conjunction with NIH to see that these kinds of trials get published because we're using taxpayer dollars to fund the same study over and over and over again. Yeah, the only comment I'll make there is I think we try to invite Sarah Green, different Sarah Green from Rapid Science and she's making an attempt to do that. Again, way underfunded efforts. The other piece that will be very interesting and actually I'm gonna leave it to Kelly Edwards after one o'clock since this is being webcast to give you the summary of the IOM report. But these are exactly the issues we deliberated around sharing, publishing negative results, publishing clinical trial data, et cetera, for a year and listen to everybody and then wrote recommendations that I think people will be really excited about and those are exactly the kinds of things that we're looking for there. So I have next, Eric. Yes, Eric Brown, University of Maryland, Baltimore County. I can't help but feel the, from lots of the comments being made, I just hear the inherent conservatism of science and sort of recognizing that, whether it's in reward structure, whether it's in publication, whatever it is, I mean, and this to me speaks more to the fact that we're all biased agents than anything else, right? So we do the kinds of studies that get us the kinds of results that results in us furthering our career. But again, not to impugn the altruistic goals that come from, that are deep in the psyche of many scientists, but still, this is what folks do and what don't you publish? You don't have some great results for a drug that you don't publish. No, that stuff gets published. So it makes sense that these things wouldn't happen. And part of what I hear, and I thought this yesterday, and I don't remember who it was, maybe it was Holly who's the, is she the genetic counselor? There's someone, yes, and I immediately made the connection between her as a genetic counselor and from the other workshop I was at on Civic Science where they had lots of folks who were cooperative extension agents. And that in some ways they operate in this same sort of organizing translatory space and that what seems to be coming to me or what I keep hearing is that we need more of these kinds of folks who aren't researchers in the traditional sense, who aren't community members in the traditional sense, but folks who operate in that in-between space and who manage those relationships, understanding that there is a continuum and that you might inhabit different roles, but that there's a richness to having people who can maintain the continuity of effort in those in-between spaces and that perhaps this is something that funding agents should understand is important, that we need folks who can do that and that they should be paid for. And similarly, that universities should understand that these kinds of people are extraordinarily important and that we should educate them to understand that this is a real role in the science of the 21st century, not just citizen science I would argue, but science more broadly, that getting the results to the regular folk, even if you're in a traditional kind of research setup, is extraordinarily important because as someone said, we do pay billions of dollars for this sort of stuff. So citizen science and the other civic science efforts I think really are pushing the natural innate conservatism of science and the mechanisms of science and we need to listen to those kinds of pushes and figure out if there are really innovative ways that we can or to give more support to the innovative ways of expanding that and looking at the models that have worked and are working like genetic counseling, like the cooperative extension agents and seeing what lessons we can learn from those translators. Thank you for being an advocate to advocate. Those are part of the relationships that I was talking about, that we do have a place at the table where we're not worried about what's gonna be published, where we're not worried about whose toes we're gonna step on because I may have to work with him or her sometime down the road on any given project or I'm gonna blow your tenure structure or whatever. So yeah, thank you for those comments. Can I ask you just one quick follow up? How do I, as an advisor to lots of students and a teacher of lots of students though, convince a student that your role is a role that they can inhabit and make a career of for themselves because that's not how you came to the role that you inhabit but I can see many of my students for whom that role would be an awesome one. It's a perfect place for them to live and marry their different interests but then figuring out where the place for them is in the larger sort of edifice of science is harder to figure out how to instruct them how to make a life for themselves there. Well, the roads inward are many. I do wanna correct one thing that this is not my career. This is an avocation. This is something I do from the heart and 99.9% of my time is volunteer. So there are many ways to get involved directly talking to somebody in the field that they are interested in and the other members of the panel can speak to that but I got involved with an online chat group or a listserv that were comparing research studies just in the specific type of leukemia that I have and so out of that grew, I thought I was gonna have a backyard barbecue with people from, there are 3,000 members in this group but I thought in a smaller community in the central Ohio area, well that grew to 100 people and then two years later that grew to 200 people. So it's amazing to me how people come into this but I think even a conversation with somebody who might be a leader in the field is a great enroute, even just asking questions. This is one way. Yeah, so I think you make some really important points and it's interesting and you're asking in what field, I mean you do have for me one of the disciplines departments would be, at least in public health would be health behavior, health education because that's where you get the MPH and that's what people have been trained. That would be the first place where students could go to get a degree at least from a public health perspective but then the cooperative extension, a point you make, I think it was really important because we have University of Maryland, we have a cooperative extension and they do a lot of work, you have co-op extension agents in every county. I think part of the problem I've seen with the structure at Maryland is really still that traditional model and so they haven't taken, and you had to think about it, George Washington Carver played a major role in the cooperative extension, what it is today and so that's infrastructure that could be leveraged and expanded upon that we need to use more of and I think NIH has a science communication fellowship program, right? So those types of, you're talking about science communication, science translation, being that glue when the project ends and being the person who helps the next project begin, right? I think there's, again, those could be fellowship programs that get funded, you could have a training grant, an NIH training grant that would go to institutions just to do that type of thing. So again, being creative thinking about the box and saying what are the models we already have and just providing more resources for those models. So I think Sima wants to jump in here then I have Sandra and Effie next. Thank you Sharon. Sima Finn, NIHS, I wanted to address this incredibly important issue about the sustainability of projects and that the environmental health scientists in the room know that we do include language on that and it's a review criteria in our programs about community engagement. I would encourage the other NIH people to consider making that a review criteria, making that a requirement in any project that involves human populations. But I'm gonna throw it back in the researcher's court and this goes to your comment about conservatism. This speaks to that research with communities are not equal partnerships if the budget is not given to the parts of the project that serve the community and that that is something that is the scientist in this atmosphere where their careers are dependent on their research and sometimes their tenure at the school depends on their level of funding. It's a sacrifice to make a part of the budget for the community for sustainability but if you require it in funding announcements you require it be a budgetary commitment. You require that key personnel include community members and they get paid. That begins to equalize this and I think that is something that we can do at NIH and it is something we do at NIHS very often and I would like to see it spread a little more and that is one way to ensure sustainability is you have people propose what is their plan for sustainability which also includes identifying other sources of funding. Thank you. Comments and I was struck Julia by your comment that your NIHS grant was a social justice grant was pivotal in ceding a lot of the activity and you said that they didn't require a hypothesis they didn't require preliminary data. I wonder if you could share a little bit about what lessons are learned about thinking about out of the box in terms of funding mechanisms and also what different ideas around metrics of success in terms of the renewal. So for example, I'm just thinking about Sokobi's comments about building a research infrastructure within a community as potentially a metric of success or this translation into policy action even at the local context. Could that be a metric for success in terms of sustaining funding and also whether you needed a traditional institutional partner can this happen simply by the community themselves? That's a really great set of questions I hope I remember them all. Don't let me forget that last one in particular. So I was thinking last night actually that maybe citizen science and community questions tend to be descriptive that this like that the idea that we didn't have a hypothesis to start with it's not like we didn't have any scientific ideas or any theories that we were working from. We were working from the idea that endocrine disruptors might affect breast cancer risk but that was 20 years ago and that was a very new idea and in California we were working from the idea that we might or might not be able to document chemicals from the refinery inside homes in a way that would show that the outdoor air monitors that were being used in the regulatory process were inadequate. So you wouldn't call any of those things a scientific hypothesis but they were descriptive questions that were science based and the whole human genome project and a lot of astronomy is descriptive. So I think citizen science maybe inherently opens the view to this kind of research in interesting ways. And okay so then your next question was sort of related to that. But where, so then what are the metrics? Well I think NIHS actually has a really cool manual on metrics for community based participatory research that includes a range of outcomes. I think it's really important to include policy change and I also think that dissemination through the news media and social media we need metrics on that to be counted and important along with scientific publication. My own view as I said earlier is that scientific publication is a foundation for news media outreach and it's important to remember that. And in a way the capacity building then proves itself. So for us as an institute it would be kind of maybe it would seem silly to report as an outcome of your grant. Wow we learned how to manage the finances of a federal grant. But that gave us courage to apply for another one. And it gave us courage to support other organizations to apply for them too. So then we won the grant and then that becomes the metric. We did have partners. But one of the things that was transformative was that Shobha Sridivas and the program officer by encouraging us to be the principal investigator at that time the money wasn't split it all went through us. And that meant everybody was accountable to us. And that grant include researchers who were then at Brown University, UC Berkeley and communities for a better environment. It's not that easy to make academics accountable to you. I have a grant right now where that's an issue. But it's very different from when the money gets split down the middle and their grants to each entity. We issued subcontracts that described our expectations. We chaired all the meetings. Although now we've got like a rotating more collaborative perhaps structure people had to come to our space. And a lot of those things turned out to just make a lot of difference in how the project ran. So then your last question was could we do it without our partners? And I think that is something that would be as an executive director and thinking about the strategic plan. I am thinking about whether we should try to become the lead on a center. And like right now NIH isn't prepared to let us win. But maybe it should be. And I think that could be very powerful. I'm trained, I have a fancy education. Some of my staff does, some doesn't. But I have learned in my experience that where you sit really does matter and the social scientists in the room shouldn't have any trouble believing that. So even though I have a fancy education the fact that I sit outside the university makes the work that I do different. I'm mission accountable to my board of directors and to my constituency. And I think it could be really, really great to have some NIH centers that are led outside of university. I just wanted to build on that a little bit. One mechanism that maybe people may or may not know about in this room is there's a mechanism in NIH called the Native American Research Centers for Health and this is perhaps a model that could be used with other funding announcements where the requirement is that a tribal entity or a tribe is the lead applicant and that they may partner with an academic institution. But the requirement actually is that most of the money goes to the tribe or the tribal health organization. So that's kind of one thought. The other thought that I... Oh, I totally forgot my other thought. Maybe I'll pick up on it. Go ahead, go ahead, you got it. Yeah, I forgot. So no, those are really good comments and I appreciate your question because with these pH we've been pushing, it's been advocates for funding mechanisms that go to communities. So when you say the word community we all come from a community. Why is this sort of a stereotype where the community doesn't know what they're doing? They have many people, professionals who live in communities. We all live in a community. So why is this stereotype about, I mean, it's like this natural, y'all can't even... I mean, I don't understand that. So that's been, that's like one sort of point and one sort of quick sort of funny situation was in with my, with what Omega-Wilson was president and we're not related of WERA. We submitted an R01 grant while I was a grad student and we got comments back. There's no way the community group wrote this grant. Right, Steve, was this 2002? I was a PhD student. Chris was a master's student. We wrote a grant and we got, how rude and disrespectful that they assumed that these professors were faking like they were community. And that was the review. It was an NHS grant we submitted in 2002. It was a great grant. And so this is the kind of things that you have to overcome when you have a community who takes the lead because it's just bias, it's assumptions that community cannot, and we know that's not true. But I think there's been this ongoing discussion about changing the mechanisms. Again, it's not all on NIH as one of the attendees has said, but that NIH, federal agencies, foundations like the Northwest Foundation, I think Funds Community, or Kresge Foundation, or Boyd Johnson, they have to be the ones to change the way things are done. Taxpayers' money is being used and we're not seeing value out of the outputs of a lot of these projects. That has to be more scientific accountability. That has to be more science democracy. Citizen science is a part of that. And NIH has the opportunity with other funders to be the lead, to change how things are done. So people like me, folks behind me, who are doing great work when it comes but when it comes to team promotion, a third of what I do is not count it because even though the university said it's valuable, schools of public health say we love social justice and addressing health disparities. In the end, it's not counted. It is not counted. So funders have to change that so we can have more students like I think the one that Eric mentioned who want to do this work but their career opportunities are not out there because it's not valued. And the university is not funding those positions. So there's ways for that to happen but I think these larger players have to be a major driver to make institutions change the way they do things. I did get permission to go over a little so we're going into a lightning round. So we'll ask the questioner to make the question short and people to make their answers short. I'm not a winner, sorry. And if people could turn off their microphones because only three can be on at once. So Effie and then Elizabeth and then we're done. Okay, that's a quick comment. I just wanted to mention one initiative which seems to me an interesting evolution in some of the stuff we're talking about. The BMJ, which is a high impact medical journal has an initiative in place since last year where anything you send there, research, analysis or whatever is also reviewed by patients. So it has a parallel review from the scientific experts or whoever, medical experts and patients. I thought that's an interesting idea. We'll see how it goes, it's new. And maybe there are other journals that I'm not aware of but in that sort of high impact level, this is the one. But my question maybe to the panel more is I do like the idea of that sort of evolution, this integration of citizens and patients in what gets recognized as scientific knowledge that we need to, we can use. Do you think there is a risk in having knowledge produced by citizen science being considered as not up there if it has its own separate channel of dissemination, of assessment and all this? And where would you see, because that's something I would say we have to avoid. But do you see any ways, do you have any suggestions of how we can actually integrate those systems and having one way of assessing what constitutes good science that we can all share, no matter whether you're in the scientific establishment or outside of this? Fast answers, anybody? What are you doing? We need to, I think we've heard a little bit about impact from sick hobby and also Julia I think talked about how to create metrics and I guess NIEHS has some metrics so I think you're right that that's a piece we're gonna have to put together. Elizabeth's gonna have the last word and then we're going to adjourn for break. That's a lot of pressure. So there is, I think we need some guiding principles. We in the climate justice movement use the HEMIS principles for democratic organizing. I would suggest that everybody read them, believe them, use them, incorporate them because what I'm feeling is that there needs to be a cultural shift and as we're trying to sort of take apart years and years and years of a structure that guarantees certain results and the challenge for those of us that are in the community is that we don't just deal with science. We're dealing with law, we're dealing with agencies. We deal with a lot of different issues because our communities exist in an interdisciplinary frame and they are dealing with so many kinds of challenges that science is just one piece of everything else. That's why most of us have never taken an environmental policy class even though we change environmental policy every day because we are these generalists that are there to facilitate community power and transformation and so that cultural shift has to happen everywhere and it's happening in a lot of different spaces and it has to happen in this space too so you need to have those guiding principles. I hear communities talked about a lot and for me it always is what does that mean for communities of color and leadership of color specifically? The reality is that foundations don't fund leadership of color at the same levels as they fund white leadership. If you're white and you were doing a study on the work that I have done, you're going to get a grant that is three times the size of the grant that I receive and that's true. That's been well documented that there is discrimination in funding and so as we're looking at the change in demographics and we're looking at urban areas that are gonna become bigger where there are going to be more public health disparities there has to be more an investment in leadership of color that is addressing those issues that affect their community because it is going to be really serious. Of course I always talk about climate change. Somebody call me the climate change alleluia, all right? I said everywhere I could handle that. I'll take it, I'll take it. So I say that we've had conversations with Kresge. We've talked to Kresge about the fact that the big greens get all of the funding and that climate adaptation and resilience has to happen on the ground, that has to be... And they're looking at that and we're meeting with funders who are looking at thinking about funding and supporting organizations on a grassroots in a very different way. So there's a lot of folks that are thinking about this and we know that it's hard and we know that it's challenging but I think that given what is ahead of us that we have to start thinking about these issues in a very different way. And so anyway, I'm gonna stop right there because I feel like I've lost my way and I don't want you to call me an alleluia. I mean, I've had enough. Elizabeth, that was fabulous. And I'll just end this panel by saying, I think my recommendation for LC work at NIH would be for LC to turn the LC tools on the NIH and the way that it works in the world is a very practical thing. Thank you so much. I just wanna set up the next session because when we... Thank you for that awesome panel and comments. I just wanna set us up for our next session when we come back from break because we're all gonna move and it's gonna be super disruptive because you guys are all comfortable and spread out and everything else. So pack up your stuff now because when we come back, we're gonna sit down in our new assigned small groups and here's what the tables look like. So you all have your grid, your sheet of where you've been assigned. So we're just gonna go in a big U like this where that front table where Pearl and Elaine are right now. That's table number, that's A. My table back here and we're all gonna move. So the facilitators are tagged with their own tables. These are just for letters. So that up front table is table A. Back table is B. Middle back table is C. Side table here with Deborah and Rich is D. And front table where Sarah is is E. Okay, E, D, C, B, A. And so find your table when you come back. So that means we're all moving. So have a good, and please come back at 10, we got a 20 minute break instead of half hour. They do have coffee upstairs. Thank you.