 We have almost an hour for discussion. I wonder if I might just take my prerogative as moderator to try to follow in your footsteps, Kelly, and just throw out a series of questions for each of the panelists to try to answer, to try to get at some of the issues you raised at the beginning of this session. Would that be all right? Okay, and then throw it open to questions from everybody. But I thought what would be really interesting to hear from each of the panelists is given that this is a panel about relationships, if you could talk about at least one way that you know that a relationship is successful, and one way that you think would be useful to help build a relationship, any relationship. So maybe we can start at the end and go down unless somebody else wants to jump in. All right, having to go first here is tough. So relationships being successful. I think one of the things that's tricky in this relationship building is it often takes a while for the relationships to develop. And so any metric for success has to be timed appropriately. So I'm not answering your question exactly. I think for me, the primary way that we have determined whether our relationships were successful is whether we conducted research that the community felt was meaningful and important to them. And then we're able to disseminate that research in a way that was accessible to the community and had direct, we could tie a direct implication to either changes in care or changes in policy. And that was all mediated through stakeholder relationships. I'm gonna think a little bit more. I might be able to come up with some more nuance there, but really that's kind of the gestalt of how we determine success in that relationship building. And of course, if people are willing to stay engaged, that always is a good sign of relationship building. And then you had a negative, a second part of that that you're gonna have to remind me of. If there was one way you could, something that you could name that you think would help build or maintain a relationship. Right, I mean, so I think primarily for us, the ways that we've been most successful in retaining those relationships is making sure that it's a meaningful engagement so that people really do have some power and some control. Because the way I've seen it fail is when you invite people in and they have a very preliminary role very early and it's quite minimal and people are unwilling to really make serious change. And so I think that that is really the key in our engagement for success is that we truly are willing to change things very significantly, change our approaches, our methods, our priorities based on that engagement. I think, you know, I mentioned before that we have patients representing over 2,300 different conditions, but we probably excel in about 25 to 30 of them. And the reason for that is that in order to be successful with building relationships is that you need to take the time to understand the condition, understand the trajectory of that condition, understand the kinds of questions you need to ask someone when they start looking at the diagnostic process versus someone who's been living with that condition for many years. So it actually takes us about 12 months to decide that we're going to fully, deeply dive into a particular condition from the point of making that decision to the point of having what we believe is a successful relationship. And a lot of that is built upon ethnographic interviewing that's done with patients in the field via Skype, with clinicians, with researchers, really trying to understand what are the tensions and the problems that we actually on our platform could begin to address. So that's actually a big undertaking on our part and actually requires a lot of investment on our part. So we oftentimes are looking for collaborators to help support some of that work. We will never get to all 2,300 conditions, but we believe as we chip away and we deeply dive into these, we will begin to start to create a framework within which others can start to think about using that as well. A few things that we've learned along the way, I think that are just absolutely critical to success in any relationship. And that is, as I think we've already heard, engaging people in something meaningful. But really, and we heard this earlier as well, that research questions in and of themselves aren't necessarily the questions patients are asking. So we need to be thinking about what are the kinds of things that matter to them that we can reframe into a research question and then find ways of being able to measure that and then learn from it and then collectively share that. So the other piece would be that we have learned one key measure of engagement for our patients is making a commitment to them that if you give us something, we're gonna give you something back. That can be as simple as telling us your age and then we give you an infographic that tells you all the other people within that community that have that same age or a co-morbidity that you might have in conjunction with your primary condition and give you an instant infographic that shows you the other people like that. So I think it's an important piece that we have to keep remembering. And then finally, the dissemination of research results. We actually don't wait until the publication. We will share with those who participated in a survey the preliminary results so that they can get some insights right away from that. And then after that, the publication might come out and be more specific about the details. So I think there's still a bit of an open question what is successful in patient contact. PCORI had at one point a must measure enrollment or engagement and everyone in the country scratched their heads on that really meant in terms of is it for now, is it sustainable? So I think right now, what I would see is being successful is from purely from a clinical research recruitment perspective is sustained retention and over time. We've got lots of experiences of patients and participants who do maintain participation in these research studies for no really good reason other than their loyal. And we need to determine how that can be really more methodologized, made methodological such that we can understand how to retain and ensure those relationships. I think it's very much echoing everything else here. How we seek to do that, I think there's two parts of the CENA project that I think are trying to address that. One is that during the patient reported outcomes which is where the participants sign up and start answering questions about the disease efficacy, we give them immediate feedback of how everyone else has answered those questions like sub-second response time. If you've got this condition, if you've got that, we tell you how everyone else has answered it. So it's very engaging to learn dynamically while you're engaging how everything else is doing. It's a very real time, active and actually fantastic environment. But as part of that, we're also establishing conditions by which we can communicate with them to give them value ultimately over time and how do they best respond? Do they want to participate actively? Do they want to just get outcomes? Do they want to merely lurk? Do what they want to know more about their communities? So we're trying to establish mechanisms by which we can supply those values back but during the process of them actually engaging themselves, we're also simultaneously telling them how their community is also being engaged. That's, I think, our two chunks. I'm really happy to follow you because one of the ways that we also measure successes by retaining engagement, it's so difficult to do with working class families that are working two or three jobs and then going home after work to take care of their children. So we do a few things to make sure, first, the organization's intergenerational, which means no one ever ages out because we believe community is defined by having everybody at the table and that we learn across generations from each other. We listen to body language. There's a lot of ways that people communicate without saying a word. We provide childcare, translation, and food. We meet when the community is accessible. And we also, one of the ways that we have been able to get people involved over and over again is by having and documenting and celebrating short-term, mid-term, and long-term successes. When people are involved in a campaign, whether it's to address a public health disparity or to gather data that shows that their condition is directly related to that condition, to that situation, it's important that they have some short-term successes because that's what keeps them coming back. It's something they can celebrate. It's something that comes out in the news. And then they stay in for the midterm and then they stay in for the long term. And so that, I think, is really important. People who are working really hard, who've got a lot of other stresses to deal with, need to know that the time that they're committing is actually manifesting in something positive that the community is benefiting from. And once they've seen that, they keep coming back. Okay, I'm gonna open it up to anybody, including the panelists, if they have questions. Yeah, thank you so much to all the speakers. Following up on sustainability question, I think we have some models out there, such as Framingham Heart Study, the Nurses Health Study, where there has been real ownership by all of the participants, which has required an awful lot of infrastructure of communication, et cetera. And I think those are successes maybe we can learn from. But a question I have, and Elizabeth, I appreciate your comments on having short-term, mid-term, keeping people engaged, but so much of research is multiple years and incremental. There is no big, oh, wow. In fact, it may be a negative wow, if that's a possibility. But how do we not get citizen scientist fatigue or citizen fatigue when the outcome is, oh, we didn't prove the null hypothesis, we got nowhere, onto the next study? I mean, it's hard to keep professional researchers engaged in those as well. So I think it's, or will only big, oh, wow research be appropriate for this. Can I just give you a brief answer? I think the problem with how you're framing it is that you're thinking in terms of outcomes and we think in terms of process. So the process of bringing people together builds trust, builds relationships, has people sharing information across the table. And those things are worth celebrating. Things change when you bring people together to work on a particular project and there are successes and those things have to be, so if you think about the outcome, and this is a country that always thinks about what's at the end, but when you have a justice framework, we know that a lot of the change happens in the process, those are the things that you have to lift. And we have a group of cancer survivors, these women who came to us because they lost their bus and we got their bus back and then they came back to us because they wanted to expand the median on 4th Avenue because there were 88 fatalities and deaths in five years and we got the thing expanded. And now they're involved in working with recovery folks and they know that's gonna take longer but because they've gotten things along the way and we've been able to succeed, it's made it, but then we're also multi-issue which makes it a little easier for us also. I'd like to just follow up because I think it is a really important question and one of the things that I think we need to be recognizing is that there's a sense of urgency amongst people who are living with illness that you just as researchers cannot appreciate and understand and that we need to find ways of being able to build in incremental opportunities for learning, one of the things that we have learned over time is that patients are actually learning and benefiting just from the process itself, I think we've just heard, but also that we can have interim outcomes that actually help to benefit whether it's a positive or negative one. The lithium study was a negative outcome but yet people learned a lot as a result of that. We also learned a lot as a company about what that takes to engage patients and understand them and listen to them carefully. And then lastly, I would say, sometimes as I mentioned before the research questions don't need to take so many years because the patient's kinds of questions may be different than the kinds of questions you're asking. So think about ways of reframing it so that we can maybe incrementally understand it better. I'll give you one example. We had one of our ALS patients who was scoring zero on the clinical endpoint for ALS, the functional rating score, so which meant she could no longer participate in clinical trial activities, said I'm still functional, I'm assisted in every possible way with ventilators and every other kind of device but I'm still a functional member of society. She actually worked with us and we actually published with her the extension items for extending the ALS FRS score so that you have three additional questions when you are using assisted devices. It did not take forever to get that done but the positive impact of having that patient participate in that and have her be a co-author on that paper was incredibly valuable. Can I just get people to raise their hands again and then I'll see if I can keep track of you. Let's see, okay. Okay, let's start here and then go this way. My question is about the process part of building trust and so on. Whether you're a patient and dealing with a for-profit medical services system or a community member who is exposed to toxins or who's denied services, would it make a difference if the trust-building process is with say a for-profit company or with a corporation that actually is funding the research that produces the pollution versus, so historically, there are interests of people in building trust in the authorities or researchers in building trust for purposes of controlling people and maybe even profiting as well as interests of people who seek democracy and economic and social justice. And I'm wondering how you parse the trust issue if trust could come from both a motivation, either a motivation of benefit or exploitation. Well, of course, in our prison projects, we are not directly channeling farmer trying to recruit patients, though indirectly we are, ultimately. We're trying to provide the environment search that there could be a trust relationship that the participant would recognize that entity as something they could contribute with, but it's an educational process. So right now our trust relationships are again, all led by advocates or of nonprofits or which are patients, but prominently, and all which are defining directly within their community is what they consider effective trust-based relationships. But, and then some of them, of course, have already given some serious thought to the more the for-profit or other conditions, but we haven't actually made that a priority for developing the trust upfront. The first part is just to go from the advocacy, go from the community, and then determine disease community by community, what their aspect, what their interests in some of those other relationships are. And some of our communities don't seem to have any direct interest in the for-profit relationships, so others, of course, might. So we're still testing it, but right now we're trying to still lead from the advocate, patient, participant as a representative of trust first and letting that community self-develop such that it can be in a position to answer the questions of secondary use or secondary engagement. So I think it's different based on who you're talking about, whether it's a for-profit or a non-profit and you can't always assume that the non-profit is always on the up and up either, right? Because that does happen. We've got a business in our community that has reached out to every non-profit with funds so that they can support and bring their members into supporting a plan that would end up in their displacement. And our organization is literally feels like, you know, like we're fighting all of them. So those are non-profits, but they get resources and they provide services. And the community they serve will be displaced by this corporation coming in and doing this. So the approach is different depending on who it is. We had a power plant company that wanted to expand its business and knew they couldn't get past us to expand and we put together a coalition because they wanted to reach out to the community. We were sort of like the first stop, you couldn't reach out to them without us getting in the way. Put together a community benefit agreement where they had to reduce net emissions. They could expand, but they had to use equipment that would reduce net emissions. That had never been done before. It was a peaker right next to our community that was dropping tons of knocks and socks on us. We meet with businesses all the time to try to get them to engage in practices where they can continue their business because we're EJ, so we have to work somewhere, but that they work in such a way that it's not killing our community. And so before they even get to the point where they need to, so we get funding so they can hire engineers so that we can look at their facility. But the trust thing doesn't come without the work being done. We can't trust unless we've had an engineer look at your plan. We can't trust unless we've gotten all of your information. And then the trust happens and we have been able to develop really good relationships with some businesses on the waterfront that have been open and willing to work with us in a way that's different. So we could support them and celebrate them as a business and the community can feel confident that this particular business is operating in a way that's cleaner and better. But it takes a lot of different efforts. Like me, there we go. I'll just say quickly. I mean, we do have financial relationships with biopharmaceutical companies and with us it's all about transparency. It's in the consent we have on our homepage. We have how your data is being used so we make it very transparent when and how we're sharing data. And we periodically survey our membership about their perceptions of the privacy and confidentiality and data usage. And we make sure that when it's necessary we alter our processes. So it's a rare disease community that's highly motivated. So I think I'll go back to this idea that it's very dependent on the community and what the community motivations are. As the one for profit company sitting here at the table, I feel like I do need to respond. I think one of the areas that we believe very firmly is we would not have a site at all if we haven't established the trust of our patients to participate with us. So we take that very, very seriously. We have an approach that really we adhere to which is patients first. So we do engage with pharmaceutical companies because actually they have a lot of money to be able to start exploring questions that actually matter to patients and especially patients who are looking for solutions to problems that we're not finding answers to. So, but we do that very intentionally and very carefully. And as I mentioned before, when we work with external researchers, whether it's an academic, clinical, non-profit environment or a for-profit pharmaceutical company, we actually vet that project very carefully. It goes through an entire team and some of our patient advisors to determine does this seem to matter to you and are we putting patients first in this perspective? And then ultimately, if we can answer yes to those questions then we'll continue to move on and move forward with it. We actually reject many proposals that come our way because they are actually trying to answer questions of interest to the researcher and not necessarily to the community. So I think it's an important way. I think it does, you do need to parse that out some. We actually think of ourselves as not just for profit and in many ways we don't believe we could have built what we have today had we not had the investment of people who really believed in what we were doing. So, a non-profit can only go so far. I think we've been able to innovate more rapidly and to be able to grow more rapidly because of the investments that we've been able to generate. But without the trust of our patients we would not be able to grow at all regardless of how much money we had. I might just add something in here which is I think that your question raises this issue of conflict of interest which we brought up before because I think this is one place where the regulations kind of clash with sort of common sense and what we think we need to build relationships with trust because the regulations really focus a lot on for example financial conflicts of interest which might help with transparency in certain areas but not in other areas where the issues are not primarily financial. So for example, if you're thinking about maybe a patient advocacy group that is recruiting its own members to be in a clinical trial and is also involved in the design of that clinical trial there might be a conflict of interest there because if a pharmaceutical company were playing that same role you would say oh no you can't do that. That's you can't recruit and design the study and fund the study and do all that stuff. So I think the regulations and our thinking needs to kind of adjust to well what makes it okay and what makes it not okay is it oversight by a third party? Are there what, how do we negotiate that? Because it seems like you can't just sort of have one set of rules and have it apply in both the financial context and the non-financial. I know we had a hand up over here but that was you Pietro and then in the back. So I found the question about, can you hear me? The about relationship success interesting in particular your answers, the spectrum of answers was interesting in that it seemed to cover different interpretations of relationship success and who the stakeholder might be in that success. So in some cases it seemed like success, a successful relationship was about the success of the individual, in other cases it was more focused on the success of the community and in some cases it was literally the success of the relationship, how long the relationship would endure. And so for me that sort of raises the question of whether or not there's some sense of obligation to study these relationships, to formally assess the impact of the relationships on the various stakeholders and then be transparent about that so folks could make informed choices. So the first part of my question is to what extent do you already do that sort of thing and or believe that you should or shouldn't do that and then the other part I think relates back to Leah's point which is that for some people if you give them enough rope they'll hang themselves. So I think sometimes because I think everything you've described is motivated by positive intent that we sort of tend to assume that this is gonna be beneficial and so are we obligated to determine circumstances and when it wouldn't be beneficial and also to report that where do we draw the line of what we let people decide for themselves in a transparent environment versus what we would never ask them to do even if they wanted to do it. So is, so to what extent do you believe that there's an obligation maybe a moral ethical obligation to assess the impact of the relationship on the various stakeholders and then report that to all the potential stakeholders so they can make informed decisions versus just assuming that because the ultimate goal is a positive one that it's okay. Take a crack at it. I think one of the things that we've done early from our earliest days is to sort of gauge what are the perceptions of the members of our communities what do they perceive as benefits or maybe not benefits of participating in a site like patients like me. The interesting thing about it is that the results are generally always positive. They're rarely negative and I think when they are negative some of it is partly because the community isn't well developed and so it may be a little bit lonely yet. There might not be enough people there yet for them to feel part of some of the same experiences that others have. But I think overall the perceptions of the benefits of participating on the site actually lead us to better understand how we can then engage them better. So many of the perceptions are the more connectedness people have they perceive their outcomes are better. So the more people they're connecting with on the site those patients actually report improved outcomes. So it's like okay well then let's start thinking about how do we then identify how to engage patients a little differently. And I'm gonna give you a little bit of data science as well as the design science that we use. So one of the things that we did a few years ago is do something called persona-based design that comes out of the design world. And so we try to understand who are our users and then build some of our environment around who those people might be. Whether they're people who are there to lurk in the forums and not ever participate or they're there to track and monitor their disease or there's some combination of the two. So we actually understand that a little bit better. And then the other is we have our engagement team and our community team who are constantly looking at the level of engagement people are having from the moment they come on all the way through and who drops off and who stays on. Our data science team then takes that same information and starts to create predictive modeling. So we actually now over seven years of time can now begin to think look at a person who joins the site today, understand some of the characteristics about them and predict whether they will be on the site three months from now. And then better understand what are the things that keep them engaged. And some of it is just the connectedness. Some of it is helping them improve their outcomes. For some in epilepsy it has to do with helping them better understand the side effects of their treatment so that they get better understand when they make a decision to go to the ER or not. That a lot of different things that come into play but that starts to build a trust and it also builds the relationship. So I think studying that is actually really important and so we continually try to evolve our user survey to kind of get a better sense and we just haven't found a lot of negative as yet. Okay, not to press the issue too much but and you know I think the world of patients like me I actually use patients like me. I'm a patient like me. And but it's conceivable to me that someone could in a site like patients like me have the perception of having a positive outcome. Yet relative to people who aren't engaged in the site have a negative outcome. So I believe my outcome was great but actually my life expectancy was less than someone who wouldn't have joined the site. And so that's just an example but to say do we have an obligation to actually measure and study those things and report those risks to people to say maybe even to say we haven't studied this and you might actually not be better off measured in these ways. I actually don't think about it in terms of studying it I think in terms of evaluating and that throughout the entire process you're evaluating whether or not people are meaningfully engaged whether you're meeting all of your outcomes and engaging people in the evaluation process constantly having conversations with them and getting your direction you have to be adaptable and getting your direction from listening to people and having those conversations. So that evaluation has to be an integral part of anything that we do. So I think from the perspective of NIH these issues of the role of pharma in creating the infrastructure that allows for engagement and collects the data and makes the data usable is really interesting to reflect about what kinds of questions can't get answered if that's the funding model and I'm thinking in particular from my own perspective about environmental factors in illness and prevention. There isn't really a comparable industry like pharma that can fund prevention and there's like if you're preventing disease by avoiding environmental toxics there's not a profit generating opportunity in the same kind of way. So NIH needs to think about who can make the investment in the infrastructure to answer those questions. I guess the comment extends beyond pharma to the device and social industry out there that is the front line to frankly interacting with virtually everybody in this world and potentially our use of that data. At some point we've got the Apples, the Googles, the Facebooks who are in the position to build partnerships for broad citizen science engagement and there's no clear profit or model for that or obligation to them there but they are there and so I think even though pharma would be therapeutic based other partners are I think equally important and NIH would be great to have a policy that would extend to better understanding how we can develop those models and designs. I would also add, I think it's a great comment and I would also add though that in our experience over the last couple of years with pharma for those that don't have a specific therapeutic molecule that they're studying but they still want to get some patient insight into a variety of different things they're actually starting to ask questions that are actually quite interesting around quality of care and understanding what does empowerment mean and starting to really try to get a better sense of what is it that motivates patients to do anything in particular around their health which actually I think is a real shift in thinking. When we first started most of the pharma money that was being thrown our way was from the marketing department and we weren't all that interested in that. We wanted to talk with the epidemiologists and the scientists and really think about what's our data start to answer questions about that really matter. Not necessarily whether the packaging is right although we did do a study on packaging and we informed them a lot about it but really the marketing piece of it wasn't as interesting to us as the research piece of it. So I think there's actually ways of being able to push that envelope a little bit by answering certain questions that matter to most people around the environment and finding ways of pushing the buttons of big companies like Google's and Facebook's and things like that for which this matters frankly so. Thanks. I want to maybe talk at a little higher level for a minute. So the title of the session is building the relationship and I think what we've heard and probably what we knew coming in is that there are multiple relationships and the way that the structure of how we think about research was about people who participated in research and people who carried it out but it's quite different than that and so I wonder if to hear from you all we could hear from you all about how you think about taking Kelly's direction to heart about generative work. What we need to know about that we don't already in terms of the relationships that are different in citizen science between those who participate, those who carry out research, whether they're researchers, whether they're private companies, pharmaceutical or PLM or the institutions where people work. I work at Johns Hopkins which has had a kind of checkered history frankly of doing research involving communities. So how is this different than what we've been doing in the past and what do we need to know from the perspective of generative ideas that we don't already as we go forward, we being the royal we here, NIH, us in the room, society, everybody. Just off the cuff I think we need to know if it's worth it to maintain these, I mean we need to determine if this sort of engagement can actually truly generate outcomes and at least moderately generate outcomes for the investment in the engagement that we're trying to address. We've seen it in small scales but very few of the citizen science projects we saw earlier are really, I mean they generate research data but are they generating positive benefit or economic benefit I guess. So I don't know, what don't we know? Is it worth it yet? It might not be. I'm thinking though from an LC perspective, right? Since that's why we're sitting in the room today among the reasons. So the kinds of ethics, law, policy, social implications questions that ought to be thought about and pursued. And maybe you don't have a thought about that but I wanna try to keep us, to me that's where the action ought to be and a lot of this, the general conversation is very interesting too but I wanna sort of dig in and ask, is there anything different going on here in the context of this notion of relationship? So I'm gonna echo something that was discussed previously which is this idea of access. So we can, from our side, we can educate which is really required, that idea of research imagination but we are still, I'll speak for myself but I suspect that applies to many people, we are still trying to engage a small subset of the patient caregiver population within our net. We're trying to engage abroad, we are engaging a small subset. So we still have really overriding issues of access to this whole process and representatives in our engagement. And I think that those issues, while it's not a new issue, is a really compelling issue that is going to have a major impact on how successful this whole enterprise is. So I sort of half answered your question but I do think that this is something that is an area, so does it work? So does it generate new ideas that are novel and important to communities but when we say important communities, are we talking about the top 5% of the community or are we talking about the community? Okay, I just wanna add, we work with RAND and Lifelines and we meet with them every Monday around 10 a.m. for about two hours. And I can say that maybe the recommendations and the things that we're saying just seem obvious but they're not obviously being operationalized. That maybe some of the input that we have may seem like well anybody can figure that out but in fact people are not doing these things. These relationships are not working in a lot of spaces partly because of the conventional training that people have because there's old school thinking about community. And so it's my hope being here that people can get out of that space and think about these things differently. It takes a lot of work. If you're someone who has a particular discipline, your discipline is to study, I don't know, epigenetics or something and that's your thing, right? That's your passion. It's very difficult for you to get out of that space and start thinking okay I gotta build these relationships. This is really important to collect the data so that we can get these outcomes. It's not the way that a lot of folks in this room have been trained but it is a process that is going to contribute not only to a body of work but to changing the landscape in a way that has to be changed. And so while some of it may not be, I don't know, higher order thinking, it is the kind of stuff that results in collecting data, getting policies changed as a result of data collection on the ground with our elected officials to getting people in our community access to resources that they may not typically have if we hadn't been able to prove that there's a connection between these different things statistically. I just wanna add something about sort of what's new in the Elsie part of what Elizabeth and Holly said and I think it's just a modification maybe of the old justice issues and so we have been talking about who benefits from research and we've moved towards, oh, now we study women and now we have to study minorities and we have to report that and be accountable. I know, but that's how the NIH, that's how it's discussed. But now we have to move that discussion into now what is the just way of engaging communities at other levels, the other levels that we've been talking about and perhaps maybe not just patients because we have been sort of focused on patients but I think there's a lot of other stakeholders in communities that might be considered in this citizen science space. So Sally, and then I know Effie had her hand up for a long time and Jason and then. Just real quick, I think just to follow on that I think the key piece here that NIH needs to really think about and this is not news to them is that we need to modernize the clinical trial and the research paradigms such that we are actually starting to address and get a sense of why people would wanna participate in research today under the current governance and legal issues and ethical ways that they have to deal with all of the constraints of getting into a research agenda. I think we need to really push that envelope because people that want to participate it's just that the barriers to participation have been relatively high. And so I think modernizing it, understanding what do we need to do to get to this action point does involve creating relationships with people to better understand what's keeping them from participating. But when we ask would you participate if you were given the opportunity to do so, they say yes. They tell us why they haven't been participating and a lot of times it's process and structure issues. So let's start to think about what those look like and I think we'll start to break down some of the barriers. My question follows closely to Jeff's and one of the new things that we're hearing and I think Sandra mentioned that earlier today and I think sadly you mentioned the work democratization of research. And so I'm wondering if the way we're using democratization is clear to all of us or we have criteria that need to be met in order to say yes we did democratize science or we did democratize research. Do you have a sense of what those criteria would be? Is it simply just involving more people? Is it simply increasing numbers? Is democracy or democratization requires a different degree of engagement more meaningful or more advanced or augmented or I don't know what. So I'd be interested in hearing your thoughts on that because I think this is one of the new things we're hearing in comparison to our standard research that maybe wasn't democratized. So take a shot at it. I think one of the things that just information access has done obviously is given a democratization to everyone to have a better access to something that they might not have had access to before in terms of just knowledge about things. Now when I say that I think in the context of trials and research studies that are being done in a place like NIH I think what we are offering is a heterogeneity that you just have never seen before and that the homogenization of research actually has not really given us the kind of insights we need for people to live as well as possible with the illnesses or conditions or situations or environments that they live with every day because we're not heterogeneous enough. So I think if there's one thing I think that democratization could start to provide us is access to people who would not typically have participated in something. And then I do think however that there does need to be some structure and process around that. That it's not just a free for all though crowdsourcing and the other ways of doing cell slide, I love cell slide. I'll go on that, identify cancer cells all the time. But I think the idea of doing what we are trying to do is not necessarily just corraling a lot of people and then suggesting that's going to answer all the questions. I think there does need to be some sort of science behind what that looks like. And hopefully we're starting in some of the examples we've talked about to create some of that. But I think the environment is there, the milieu is there. Now we have to figure out how to apply some of the traditional and yet contemporize them approaches to thinking about how we get the best answers possible with a more heterogeneous population. So I think there's the need for democratization and then there's the now. Right now, for example, the CENA project has, it finally recognizes that there's a very broad spectrum of how anyone defines their expectations of privacy, their nuance, what they wish, what they may know, what they wish, some participate actively in everything, give away their 23 music sequences, others do not. And this moves, this moves based on the social media, this moves on based on family and influences, moves on a thousand things. So we're trying to, we've established a mechanism by which that can be defined, characterized, that hopefully will allow anyone who wants to at least take the step of defining that ability to define that. And they can change that. Right now though, we're only testing that in a limited number of actual, much more focused, less democratic, I guess, disease communities because that's where our funding is right now. But the goal is, again, to identify and evaluate that as a method for recognizing that we all have certain expectations for respect, for representation, for privacy, and we need to define that in some way, and they will change, and they will change, and we need the mechanism for that to change, and then of course test that. Hi, it's a great panel. I'm having a lot of fun, and I'm trying to, trying my hardest to get my thoughts together, but it really touches on circles around a little bit. Many of the comments have already been made. But part of it is sort of painting a scenario that we are in such early days of citizen science in that we should anticipate in many ways that there are sort of infinite combinations of the ways a research study and its governance, its communication style, its presentation, its features could be set up, and that we've talked a little bit about sort of research literacy too, and that a lot of people are coming to this with very little to know, no research literacy. What is a good scientific question? What are high priority research goals and how do we design appropriate research studies to achieve those goals? And as people, hopefully, if this movement is successful, we will go from sort of a rate of participation and research that is very low to one that is very high, and that participation and research is a cultural activity that's very much like people go to museums and concerts and all sorts of the zoo, all sorts of things, and the same thing people should consider sort of participating in research as one of those things. And as, so I guess the point that I'm struggling with, and I wonder how much each of you, or if whether or not you've thought about this, is you can sort of imagine a future where they're on the one hand, there are like a couple big platforms where they provide all of the possible combinations of features and governance and whatever so that they can be applicable to the most people possible. And then on the other hand, there are sort of small communities of practice who have sort of developed their own styles of research that totally transparent, you can call into the IRB meeting, data is shared with each research participant or not and all the different combinations. And so in that sense, since we're sort of so early on this, everybody should anticipate failures of relationships that whatever you are building for people may not be ultimately what they want, but they don't know what they want. And so how much are you planning in advance for failure and the ability of people to change their mind and take their data and donate it to some other research enterprise, which actually meets their new expectations? Maybe it's something that needs to get on the agenda. I think one of the things that I've learned, I'm a clinician first and foremost, and I think one of the things I've learned from my software engineer and technology experts that I work with is that seeking perfection is probably folly, that we need to get to a point where we're constantly iterating and learning and we're aligning ourselves against some measure that we're reaching for and then we're evaluating how we've done to get to that point. So I think while we've certainly had some failures, there's no doubt about that in terms of how we've developed the taxonomy or the infrastructure within the company. Each one of those has been an iterative learning experience to say, well, we're not gonna do it that way again. Or gosh, some part of that actually worked pretty well, but maybe we need to think about a different approach the next time we try to deploy something similar within the technology. So I'm not exactly sure how to answer that question other than to say we have a hard enough time sometimes just as a company celebrating our successes. You know, we're really hard on ourselves. So I think that we don't oftentimes sort of think about ahead to say, okay, well, this could be a failure so we better prepare for that. I actually think that we have a more optimistic view and we go forward thinking we can accommodate and adapt as we find things that actually haven't worked but not necessarily call it a failure but a learning opportunity. And I think that's actually sort of embedded in the continuous learning system. Sort of really thinking about what did we learn from that even though it didn't work as we expected it to work, there was something that we actually got out of that. So I don't know if that's the answer to your question but I think we don't necessarily say, gosh, what will we do if this doesn't work? We're sort of thinking, how can we iterate this to the point where it's good enough for us to feel like we've got something really, you know, satisfactory for what we were trying to accomplish. Elizabeth. Yeah, I think even our responses tell you how complicated it is because we come from very different places. So you say citizen science is early and I think that indigenous people and people of African ancestry have been doing it for a minute for hundreds of years and so we have our own ways of thinking about citizen science and I think what's new is us coming together to try to figure it out together and that's exciting. But I think about, I agree that you have to adapt and that you can't have a cookie cutter approach. That the approach is gonna be different depending on whether you're an Indian country or whether you're in Brooklyn, New York or whether you're, you know, for us, it's easy. We're very localized, we're in one particular community. We know that community. The organization's been there for 50 years so we have deep roots and so those relationships are much easier. But trying to come up with a framework that provides guidelines for the entire country is impossible. It has to be something that lives, that changes depending on who you're approaching, what their motivation is, what their interests are, what the leadership looks like on the ground and that flexibility has to be built in to the approach. The thing that should motivate you is this desire, I think, if I could say that, to address something that needs to be addressed in a way that brings all these great thinkers together. We meet, I had mentioned we met with Rand and Lifeline's, we learn an enormous amount from them and they've learned an enormous amount from us and they've changed their approach and how they communicate with our community and we've changed our approach in terms of how we gather information and how we document it. It's been learning across the table and I think it's been a really positive experience but it's unique to us and it's unique to our community. Okay, we're getting to the end of our time here so I know you had your hand up for a really, really long time and so you might have the last word. Did you have something to, Dave, I'll let you end it so go to you first since you had your hand up for a long time. Okay, Leah Shanley and I co-chair the Federal Community of Practice for crowdsourcing and citizen science. So when I was a young grad student, I was sent up to work with the Menominee Nation for three years on a project and every single community member I met asked me three questions. Who are you? What are you getting out of it and what are you giving back? And there was, we talked about conflict of interests, corporations, first of what we jumped to but they perceived that I was gonna be making a lot of money as an academic in publishing their knowledge. So we have to be very sensitive to that. Also time scales, I don't think, we've talked a little bit about it, 50 years. So I'm an academic, I'm thinking three years, wow, I got a three-year grant, that's a long time and the community's like, drop in the bucket. You're gonna be here and gone. What do we do after you leave? Also the pace, the academic pace, we expect, you know, you gotta take exams, you gotta get your papers out and the community needs to build trust and get to know you before they make decisions about what they wanna share. So I think there's a disconnect between the academic and local communities. So I wanted to ask you, Elizabeth, we talked a lot about IRBs and institutional IRBs but what about community decision-making about what research they invite in and what research they don't? So the tribe I worked with had a board of elders, leaders that made that decisions about who they wanted to invite and how did they make a sustainable relationship with these researchers? Our organization doesn't work on any issue that the community doesn't bring to our attention. So all of our campaigns, all of our initiatives grew out of a request from the community. So for example, when we built the Climate Justice and Community Resiliency Center, that was in response to the community saying that after Sandy, they wanted to know more than changing the light bulb. So we started a block-to-block organizing effort on adaptation and resilience. So it's the same thing for the research. We basically have people come in and say, you know, we're really concerned. People went out during Sandy. They were engaged in recovery. We're just wondering, we have all these brownfields. What does that mean? Did we breathe something? Did we touch something that was unhealthy? And what we do is we play this role of facilitator. We facilitate community, meaningful community engagement and we bring together partners that make it possible for them to get answers to their questions and use that information. I don't know if that answers your question, but that's how we do it. We train our young people to facilitate community meetings. We think that young people are not at risk that they're at potential and that they need to be engaged in meaningful way early on in science. We start as young as third, fourth and fifth grade where our young people are learning how to organize and they're learning about environmental justice. And so we have our elders working really closely with our young people, but a lot of times the request, the demand is coming from our elders. Okay, I'm gonna let you finish it out, Dave. Thanks, Miller. I appreciate it. First of all, great panel, great moderator. Thanks guys. And I won't hold us up on lunch too long. Just maybe one thing that you all could think about thinking about if it registers. I mean, I think about four of the five of you, not so much you, Mildred. The presentations you gave show that you're all sort of moderators and mediators between communities and research interests, corporate interests, you very much serve this mediating role. And I mean, I guess I've sort of trying to get back to what Jeff was asking. I mean, I would ask you guys to think about whether, as in these mediating roles, are you helping to cut down or nip in the bud potential ethical, legal and social issues? And if so, is that something that are obviously one size does not fit all, but do we need to consider having in citizen science, mediators go between professional mediators of some sort. And the flip side of that coin is as mediators, I think you guys can sit in tight positions where issues come up. And you also sit in positions of power and influence on both sides. And so maybe something else to think about is, as mediators are not, are there new ethical, legal, social issues that you all have to address or that being mediators bring up? So lunch goes from now until 1.30, you've got 45 minutes. The only, and the food upstairs is not bad. There's some decent sushi all the way in the back of that place. And the only thing I would ask, as we go around and ask questions and speak, maybe we can just quickly say our name so that, I mean, I think a lot of us know one another, but by the end of the day, maybe we'll pretty much all know who each other is, would be good. I'm Dave Kaufman, NHGRI, sorry.