 Thank you to that panel, fantastic discussion, thank you to all the presenters today and thank you everybody for spending the day with us, this thing needs to work. In 2011 my spouse Bethany got a fellowship at the Center for Advanced Study at Stanford and they gave me an office and it was really an opportunity for the year to sort of spend the time thinking about all these issues that we had been talking among colleagues in psychology, how is this playing out across other disciplines and so just getting to read the literature very widely about all these issues that we've been talking about today. And that year Bethany's best friend Sarah was diagnosed with a rare form of ovarian cancer and it was rare enough that her doctors didn't really know a lot about it and so she said I need to get my head around what this is. Her husband, Patrick, her husband started searching the literature and would call us and say I found this title, it sort of seems relevant, I can read the abstract, I'm not sure if it's relevant but it costs 40 bucks and I keep finding this ovarian, ovarian, I don't know how to even start to advocate for my own care, they're not academics, they don't have subscriptions. So we just said they'll just send us all the links, we'll get you the papers from the Stanford service and we'll just send you papers. And of course now I'm thinking, God, what will all the people do that don't have friends that are at an academic university, what are they doing to try to get access to advocate for their care in this context. And as I continued to get my head around what's happening in other parts of the literature, she's starting to get into clinical trials for different experimental therapies and I'm reading at the same time about the rate of non-reporting of clinical trials. Less than half of clinical trials, any outputs are reported after 24 months at that point in time. And then finding out that not only are the outputs not being reported at all, that outcomes are getting switched and those are getting reported. And so we don't even know if we can trust the literature itself. For Sarah, being in the trial has all kinds of impact, losing organs, incredibly painful side effects and she went into it because she wants some hope and she has some knowledge that if it doesn't help her, someone's going to get helped from that knowledge. Someone will benefit. But she died in 2017, the day before one of our SIFS meetings and just the presence of that group together put in sharper leaf the implications of pushing for open science. And I don't know if open science would have saved her, but I know that closed science didn't help her, provided no benefit to her that it was hard to access the literature, that outcomes were getting switched in trials that the research that she participated in as a participant with her body wasn't even getting reported in literature. She didn't benefit from that at all. So we may not be able to save her, but her kids, Ben and James, my kids, Haven and Joni, they might get cancer 40 years from now. We can save them. That's what open science can do. They start to close that OODA loop, make it tighter so that we can iterate faster and faster to get to the solutions that we need to solve the problems that we confront. And it's easy to feel the palpability of it when we talk about something like cancer or another disease. But those same considerations apply for any basic fundamental research that we do. We might be studying the thing that leads to the insight that is the most important idea of the next generation. We just don't know it yet. And so if it's worth spending our time doing it, we should be doing it well. And part of doing it well is a constant attention to how can we do it better. And if we're not trying to do it better, then we're not earning the trust that the public is investing in to try to help get the most out of science for what it's here for. We can't control whether the community trusts us, but we can control whether we are worthy of that trust. And that's what the broader project is, not just of open science, but the broader movement of how is it that we improve and embrace the ways that we can be more rigorous, more transparent, more reproducible. And what we've discussed today is that that engages the entire life cycle of the research. It's not just at one point in that process that we can say, if we open that up, it's all going to be fine. Because the elements of what makes research trustworthy is different depending on where it is in the life cycle. So for example, has the research been assessed is an element of trustworthiness. Has it gone through peer review? Has it been reproduced? Has it been robust? Has it been replicated? Most of that is in the latter stages to date of the research life cycle. Likewise, do the claims match the evidence? Do the researchers represent the uncertainty, identify their limitations, look for alternative explanations, advertise the constraints on what they know generalizes or doesn't? Does their conclusions align with evidence? That happens mostly at the end, but also in the process of actually producing and interpreting those findings. Is the research, can it be assessed at the openness of the data, the materials and the code? Just makes it possible to do that assessment in the first place. There's only three components of trustworthiness. There's also, does the research take into account relevant knowledge and perspectives? Is it grounded in the existing theory and evidence about that domain? Is it informed by the communities that are affected by it? Does it control bias with validated measures, winding, randomization, pre-registration? Both of these things are things that happen at the earlier stages of the research life cycle. And does it reduce error? Increasing sample size, providing reliable measures, power analysis, et cetera. And then, of course, the last is that our researchers accountable to the evidence as an element of trustworthiness. And that's relevant across the entire life cycle. So if we're not embracing the whole life cycle for how is it that we can improve openness, integrity and rigor, we're not going to solve those broader problems. But there's a lot that we can take stock of as going well and having moved in productive directions. So, for example, the open access movement has the most mature and most successful part of the open science movement. Our paper's really just available at all. This is showing the growth of open access papers with the only time that we can use a y-axis of the proportion of the entire literature because it's gotten so far ahead of nearly half of the literature is now openly accessible. Where 10 years ago, 12 years ago, when Sarah got her diagnosis, it was 20% of that literature. So that's a remarkable 30-year effort and success of advancing that. Likewise, preprints is moving to scale with that dramatic growth that Lisa opened with, change and diversifying across silos and expanding dramatically, 600,000 on the y-axis scale there. There are lots of ways we can look at acceleration of data, sharing data, materials and code. One simple way is just to look how many papers mention a generalist's repository in the paper, a very molar type of analysis. You can see the incredible growth at about one quarter of the scale of preprints by comparison. Papers mentioning preregistrations are a subset of that, a generalist's repository or registry, but also mentioning a preregistration at, also enjoying lots of growth over the last 10 years, but while it's scaling, the scale there, 10,000. Oh, it's the peak. And if we zoom in on that, we can see that the growth is happening, but also is diversified in each of these different behavioral domains or each of these types of open science actions. Some what's more popular in some communities than others. So this is just breaking out growth and preregistration and also mentioning OSF by different disciplinary areas. All of them are growing, but you can see social behavioral sciences wipes out the rest when you look at it at scale. But if we take that same data and look at it for when did it kind of start in that disciplinary area? Because when were there 40 papers in that field? Make that year one, all of them show more visible growth compared to where they began. It's just we're further along in the social behavioral sciences. That growth started in 2014. Life science is a couple of years later, and the other one's even more recently than that. So growth is happening, just takes time to spread. And of course those disciplines are themselves very different sizes overall. So what it means to scale in one discipline will peak out differently than another that is 10 times the size. And then finally, papers mentioning registered reports. Same trend. So with all of these, it can feel like the half dome hike in Yosemite. If you haven't done this, you have to do this at some point in your life because it's an amazing hike. But there's a point where you get up to vernal falls and you just feel like I've accomplished everything. Now you turn around and look down to the valley floor and it is so far away. I can't believe I got all the way up here. And then you look up to half dome and you say, oh my God, there's a long way to go. But we're on the trail and we are heading to the top of half dome. And that's really where we are with the advancement of open science. And the key challenge that's come up again and again today is that it's not just about the life cycle, it's also about the coordination problem. That we're not going to fix the research culture without aligning the various actors in the space to all be helping to push in the same direction. Because if we can't align those actors, there's no way to scale and sustain that type of change. And so the solution to that coordination problem is you. It is all of us individually and collectively aligning on what it is we're going to try to solve and then from the various vantage points that we have in this research ecosystem, figuring out how we can move our efforts together. Because if we don't align as a reform change agent community, there's no way that the mainstream is going to align to change without us. So we have to keep doing that work. So the biggest gratitude for us is just you doing the work that you're doing. And being part of this conversation and dialogue to figure out what is the change that we're trying to make and how can we try to make it. So let me just close with explicit thanks to those that are here, that are part of culture change organizations, that are making these efforts in all different parts of the research community every day, that societies, institutions, publisher representatives that are doing the same to inculcate some of these practices in how it is they decide to publish or reward or promote people from their institutions. Thank you for the federal agencies and those that are pushing on public policy reform in so many different ways that are trying to help promote the adoption of these at scale. There are many research champions that are the exemplars within their disciplinary community here today that are pointing the way to others to be able to do the same. Science journalists have played such an important role, Christie represented them in speaking for surfacing this and not just informing the general public, but science media also is a big part of the conversation among the science public, especially across our disciplinary silos. And they've played a huge role in that. The funders have been doing so many different things, so thank you to funders for starting to experiment with new models and new approaches, a particular call out to those that have likewise supported us in trying to advance our mission. And thanks, John and Laura, for making a big bet on us. We did not deserve it, but we're trying to earn it every day, so we appreciate that. It's like the second best thing that's ever happened to me. You know, right in between my two kids. So that was great. Our board members, thank you for your volunteerism, your advocacy for open science in your communities and then the advice and support for us in our work. And the team itself, I get credit for what everyone else on the team does. We have 50 people on our team that come from many different types of backgrounds, of work histories, of areas of training, of areas of interest, and we are all collectively aligned on the mission to increase openness, integrity and reproducibility of research. And you saw some of them appear. You saw the evidence of that effectiveness from Whitney and Ron and Doreen just running the meeting. And then there's everyone else that's in the audience and at home just continuing to do their work. So thank you to them and thank you for them for your support of us. So we're done. Thank you for the chat.