 Hello everyone, we are now cutting into the drinking time. So the sooner you sit down, the faster we get to that part. I would like to invite my panel to please come up and maybe sit in the order, skip at the far end and then Adam and then Tara and then Alondra at this end. Yeah, exactly. Well, welcome to our wrap up session. My panel had the hardest job of anyone here because we have to give the big picture from everything we've heard today, from everything about making it possible, making it normative, rewarding it and then requiring it and trying to come up with some ideas for moving us forward. For those of you who don't know me, I'm Marcia McNutt. I'm president of the National Academy of Sciences. It's a pleasure to have you all here for this important meeting. By way of background, how I got involved in this whole topic goes back to the year 2013 when I became editor in chief of science. That was the year after Glenn Begley published his paper that was like a shot heard around the world on the fact that so much research in cancer biology was irreproducible at his organization, which was Amgen. So as soon as I got to science, all the editors at science were very concerned about that and very much wanted to focus on things that they could do in order to improve the situation for the benefit of the reputation of science. And so I put in a short proposal to the Arnold Foundation and they were so wonderful to fund me to hold a series of three, or I think it was four workshops on different aspects of it, open access on things like data availability, code availability and materials. One of the biggest outcomes to come out of that was the top standards because at our second workshop I teamed up with Brian and he held that workshop down at COS in Charlottesville and that was probably the most lasting impact from that series. So I started working with Brian at that time and then joined the board of COS. So my distinguished panel here on the far end is Dr. Skip Lupea. Skip is the Gerald R. Ford Distinguished Professor, University Professor of Political Science at University of Michigan. Dr. Adam Russell is next to him and he's the director of the AI division in the Information Sciences Institute at USC. Then we have Tara Schwetz, who's the Acting Principal Deputy Director at the National Institutes of Health, and then of course Alondra Nelson here, who's the Harold F. Linder Professor Institute, Harold F. Linder Professor at the Institute for Advanced Studies, which is associated with Princeton. No. No. All right. In Princeton, but not at Princeton. Oh, okay. I'm sorry, in Princeton, in Princeton. And of course Alondra is well known around here because she was the former Acting Director at OSTP. So Skip, the floor is yours. More than we know. More than we know. So we're here to celebrate the incredible accomplishments of the Center for Open Science. The Center for Open Science has changed the world. It has changed what is possible. It has changed what is feasible. And it has changed what is done. And it's supported by all of you. It's a great moment. Ten years ago, the idea of open science was still seen as much of a fringe in most parts of academia. But it was an important fringe because what has always been at stake in the quest for more open science is the public value of science itself. The public value of science. We know that science has great potential to create value for the public. It has the potential to foster the next revolutionary technological advance. It has the potential to plant the seeds for the next life-saving vaccine. It has the potential to help people all over the world who depend on science for the next policy intervention or the next medical intervention that can improve quality of life for people and save lives. But the extent to which the potential of science to do these things, the extent of that potential, is still an open question. And it exists in an era of uncertainty and there will be questions about science. In an era of multiple truths, there are questions about who can be trusted. And in many places, those questions extend to science, but very credibility and legitimacy are under question. And it's not just politics because there are varying levels of awareness that scholarly incentives have created ecosystems that mass-produced false positive claims and sold them to other researchers and to the public as being valid scientific claims. The Center for Open Science led a vanguard of people in communities in 10 years they built infrastructure, created focal points, and empowered communities through an approach to open science that's accessible, scalable, and incentive combat. Thanks to this work, researchers all over the planet can engage in more credible and legitimate research practices. It's an incredible thing, and it's thanks to all of you. But the success matters more than we might know. And to get a sense of the scale of what you have accomplished, I'd like to raise a question. Who are the beneficiaries of open science? So some of the beneficiaries of open science are easy to see. They're in this room. But most of them are invisible to us. Most of them are invisible to us. Think about it for a second. Think about all the people who've never been to the National Academy of Sciences. Randomly select any number you choose. What do they look like? What are their days like? What are their concerns like? Think about the university or institution at which you work. Think about all the people who've never stepped foot there. What do they look like? What are their lives like? What are their concerns? You can think of people who've never stepped on any university campus or have never met a researcher. Think about that population What are they concerned about? What do they need to get them through the day? While these people are invisible to many parts of the scientific establishment, they're not invisible. They're part of communities and they have families. And every single seemingly invisible person deserves the very best that the scientific community can give them. And the heroes in this room are a big part of giving it to them. They're making open science accessible. They're making open science scalable. And they're making it meaningful. And how have they done it? You've seen it. You're part of it. How do you bring something from the fringe of the center? It's through perseverance. It's through evidence-based strategy. But I think the key ingredient that we can all see but no one's talked about is that it's fundamentally other-regarding. It's fundamentally other-regarding. In science, it can be so easy to make it about us. But at the Center for Open Science, it is so clearly about the people we serve and how to do it better. It's an incredible accomplishment. So it doesn't matter. Because what matters is today and what we do tomorrow and how we build on this. And that's something everybody we connect to. The world, I would argue, needs more reliable science more now than ever before. Particularly for all the people who can benefit it we can't see. So let's get to that exercise because the value of it is more than we know. Man, come on. I said I wasn't going to cry. Okay. You ever have deja vu? No. I'm having deja vu right now because every time Brian asked me to give a talk I end up having surgery just before that. So in 2019 in meta science I had my knee done and now here I am with my bionic arm. Not DARPA approved. I know, right? It wouldn't be great as I could just fly off. Not yet. And I can no longer speak for my age. More on that in a moment. But I also bring that up when you know that my injury is my arm and not that I'm having a stroke when I say this word UDA. UDA, right? So Department of Defense they use this term UDA which stands for observe, orient, decide, and act and it's called an UDA loop. And Colonel John Boyd from the Air Force proposes as basically the person with the tightest UDA loop wins because they can adjust as fast as they need to. A good UDA loop allows you to amplify what's working and allows you to inhibit what's not. And I think we are on too slow to take a key mechanism to speeding up that UDA loop is open science. One example, in 2005 Nature had an amazing article about how oxytocin enhances trust in humans. And that is big if true to quote Tom Kalil. So when I was at DARPA we stood up a program on this question of can we measure trust? Given that it came out of nature it was a perfect example of everything I would argue that's essentially wrong. It's a novel claim there was no open data there was no ability to actually verify this stuff so I knew by 2010 that that was probably not true and it took until 2014 for my program to have a null result essentially arguing that published. Nine years is too long as an UDA loop period. So if I end up talking to Brian here briefly y'all can just leave or listen in if you want for Brian. And the reason I'm mentioning this is because we sort of have been on this journey you know Christy mentioned today writing about the reproducibility problem since you know since it started I was living it right in 2008-2009. So early on even before open science stood up you know you were an inspiration you allowed me to say I wasn't crazy that I should force my performance it forces the operative term by the way to pre-register they had to make predictions they had to share data we ran replications people were like will you pre-register what happens if we find stuff we didn't think we'd find? Well that's why we're going to replicate because then it's a real fact and then when I got to DARPA you know you really provided some amazing capabilities I would argue in sense of consulting you really helped sort of I think it form decisions we could make you helped us do better I think in our program designs and then ultimately you had the infrastructure when you were up there that allowed me to use pre-registration allowed me to do data sharing and ultimately hopefully even engage on some of the more crazy stuff which I'll talk about in just a moment as well then you proved your ability to equip us so when we were worried about computational reproducibility as we all should be in increasing age of machine learning and AI analyses you actually developed a tool called the Retro Rubrik that we were able to use in our program and it's still there I think so you equipped us and then you strategically positioned us as well by using workshops to help think about how do we get social scientists to actually make predictions how do we bring forecasting as a way to improve our ability, our OODA loop and then how do we even think about AI and reproducibility where there's going to be this challenge and by the way it's not obvious that you want everything open in this area of machine learning and AI it's not clear to me that just making it freely available is the answer you also helped with innovation so we tried to register reports on one of our AI exploration we moved too fast it didn't work but we were willing to take that risk you ran a prediction market on our experiments turns out social scientists are very bad at predicting complex experiments that's fine now we know our OODA loop is tight and then finally the score program I did my best to break the center for open science with this program and instead I ended up broken just incredible effort that teaches me right but if you don't remember the score program it was really an ability to run an unprecedented number of replications to build tools to help us understand because there are decision makers who need this stuff today and the stuff that's published cannot be trusted for the most part certainly when it comes to making life or death decisions so when I got to ARPA H in May courtesy of Dr. Tara Schwetz by the way who actually is the godmother of ARPA H stood this thing up almost by herself I was excited to have that opportunity as Phil was talking about to build an ARPA from the ground up to bake that meta science in you know we had everything someone mentioned Paul Smaldino's study on the natural selection of bad science we ran simulations on ways to counter that we were going to do modified funding arteries we had an opportunity to bring forecasting markets to peer review we registered reports to the very beginning why aren't we doing that more often even an ARPA can do that and then ultimately I'm really interested in the sort of collective intelligence let's not presume that even in a hyper empowered place like the program manager at ARPA knows everything and there's lots of signal out there to be tapped I am no longer at ARPA H I don't know where that is at this point to be honest but I was very very excited and you and Center for Open Science moving forward so all of this is in pursuit of this tighter oodaloop I think we can get there keep on going on I'm going to end with very briefly it's a hall of science so why not end with a prayer a prayer of relief that you were there you had people who were smart enough to fund you in advance so you could be there a prayer of gratitude that you are still here and then also a prayer of hope that you remain here and the thing that you have started all of you continues forward and I will be there to help just let me know after you I mean the two of those but it's true actually when thinking left out of that was Adam and I sort of geeked out and bonded and I think half the reason I was able to convince him to come help me stand up ARPA H was because we talked about a lot of those things anyway that he just mentioned which is a potential opportunity to leverage hopefully not just at ARPA H and maybe we could adopt some of those things at NIH as well and we've been having this conversation but Brian and I were actually talking earlier today too about just 10 years ago and when I at least I was first engaged with the Center for Open Science and this was back in 2013-2014 NIH was also launching a bunch of regular reproducibility efforts we had an initiative that we were launching and we were working closely with the Center for Open Science as they were getting up and running and sort of seeing what they were doing and I know I went out to Charlottesville a couple times to sort of just to get some new ideas and inspiration and some of the things that we did at the time have I think have been able to stand the test of time and hopefully can continue to iterate and to develop more and so that includes things like clarifying what we mean by rigor and reproducibility and transparency in both our grant application process as well as review criteria we partnered with publishers and we were talking about that today on developing some principles and guidelines for publishing preclinical research we developed a bunch of training but fast forward to something that kind of said that kind of harkened back to one of the things I was going to mention which is during the pandemic which occupied a ton of our time at NIH for good reason we had a lot of work to do to try to move us beyond this public health emergency but it reminded us I think in a really unfortunate, unique way of the importance that we had to share information and data in real time and this is where that sort of potential I think that Skip mentioned helped to shift into reality. Having those real time data helped inform decision making, guide scientific progress just to run through some facts and figures that at least we have on our side and I'm sure it's much bigger overall but the scientific community shared immediate access to over 2.5 million genomic sequences of SARS-CoV-2 thousands of protein structures for SARS-CoV-2 hundreds of reagents across the biomedical research community billions at least this is only in one of the data sets that we have the national COVID cohort collaborative or N3C almost 10 billion rows of clinical data that we have through there and of course the final outputs of all the things are over 150,000 papers that were published and all of that helped to enable the overall response to the COVID vaccine pandemic but I also wanted to touch on a few things just because they've come up in the various some of the various sessions that I've seen today since I'm representing the government perspective I guess on this panel filled by former government folks but you know they're sort of thinking about two main components of open science and that's sharing public access and I think as probably most of you know 2023 is the year of open science according to OSTP and it's a you kick your plastic bag but you know it's been something that I think has been a priority for NIH at least for quite a while and we've been hard at work refining some of our policies that we are now in the process of putting into place so first and foremost I think scientific data sharing I think probably we're all in this room for a reason so I think we can probably safely say that we all agree that data sharing accelerates discovery enhances our group reproducibility and helps support that transparency that's so important and NIH we undertook a multi-year process it was five years actually we took to really work with the community engage the community develop and implement our data management and sharing a policy that just went into place in January and this requires researchers to do three things first they got to develop a plan and presumably a budget for how to actually share their data and preserve it to submit that plan when they are applying for funding and then to comply with the plan so hopefully fairly simple although we all know the devils in the details and you know this is again just only been a few months old that it's been an implemented policy and so we're continuing to work with the community to get extensive feedback and engagement and would appreciate that from all of you for anyone of you who have had experience with that and I will say we have a really great website that's called sharing.nih.gov that has a ton of information on all these things the other thing is public access as I mentioned before and in August of 2022 Dr. Nelson here and OSTP issued a memo on ensuring free immediate and equitable access to federally funded research now NIH has sort of long champion this and we're really supportive of this important step and it's really focused on three major pillars so the access to the data which in our case the data management sharing policy that we just put out covers that access to publications having that and for NIH funded stuff that means removing the 12 month embargo that we currently have in place for NIH funded publications and by 2025 having that shift to zero months so it would be immediate and continue of course to monitor all the trends and publication fees and policies that go along with that and then also the metadata and persistent identifiers and that's something that we're still working in development I mean we obviously have we have a lot of we have PubMed right so we have some experience in this and we have been working with ORCID ID and others to implement policies to put in place but I think that there's many more things that we can do there and we've had a lot of listening sessions over the last well the last several years but in particular April was a busy month because we had an RFI that closed a couple weeks ago we had a listening session gathered feedback hopefully from many of you and I think the take home message that I just wanted to iterate is that we are really hoping to continue to push forward some of the policies that we have in place or are planning always looking to think about what the unintended consequences are of those policies and we operate on such you know big scale that what we do we understand can have a huge influence so with that we always want to hear from the community who are actually living and breathing this every single day so we hopefully will be able to continue to partner with this in our for open science and helping to meet the expectations that we are putting forward it was not boring and also Tara was my colleague at OSU for a time so it's great to be back so good afternoon everyone congratulations to the center for open science for a decade of leadership and this year that is also the year of open science that has helped to move scholarly communities closer toward our public benefit mission and I think that it is important to underscore what skip was saying and has helped to provide transparency accountability and rigor and mostly quantitative research in particular so I left government service about two months ago I no longer feel any obligation to speak on capacity on behalf of an official capacity on behalf of the Biden Harris administration but I think on a panel that's about change agents I would be remiss to not actually talk what we were able to do in the administration is because the president of the united states is a believer in open science particularly from the perspective of science of cancer research and when given the opportunity to talk about it talks about it so I'm going to give you just a couple of examples because I think it's really important that people appreciate how important his role was and allowing us as former federal employees and Tara to do the work that we do so this is a quote from the president from 2016 when he was vice president speaking at the American association for cancer research so he identified the contradictions of a publicly funded research publishing ecosystem that had impact as one of its aims but was uneven in the way that it pursued it and so this is quoting the president right now you work for years to come up with a significant breakthrough and if you do you get to publish a paper in one of the top journals for anyone to get access to that publication they have to pay hundreds sometimes thousands of dollars to subscribe and here's the kicker that journal owns the data for a year the taxpayers fund five billion dollars a year in cancer research every year but once it's published nearly all of that taxpayer funded research sits behind walls tell me how this is moving the process along more rapidly then again in September of 2020 on September 12 2022 speaking on the 60th anniversary of president Kennedy's moonshot speech and a few weeks following the OSTP updated public access guide into federal agencies and on the occasion he would introduce Renee Wiesergen to the world as his appointee for to lead ARPA H the president said this so this was just a few not long ago we don't share enough data and knowledge to bring the urgency we need to find new answers he talked about how in the past he had talked about how federally funded cancer researchers were not sharing the results with their peers and with the public and then the president continued we made federally and he's here invoking the OSTP 2022 memo we made federally funded cancer research were available to any patient any doctor anywhere for free and today as president we're making sure that transparency applies to all federally funded science beyond just cancer so this has been you know so you have a president who's been a change agent here who's actually been on a journey on this journey with some of you I think coming to an awareness of how important it is for the public service mission of many of our institutions and for much of the research that we do that it be shared in ways with the public so while I'm honored to be on a panel with true change agents colleagues I admire I wanted to take a moment to register the president in that space as well so if my colleagues have done before me this afternoon I want to offer a few reflections mine are a bit more provocative some of you know me so you won't be surprised to hear that about what we might do for the future which is to really celebrate and spur on the Center for Open Science but to also think about what this incredible now infrastructure and networking community might spur in the future so I offer kind of three things three sort of categories one something that's obviously to most of you but I would what I'm trying to do here is to challenge this community to really I think do more of the narrative change work that got you going around the theme of open science not being a publishing model transformations that promote often open software privacy preserved open data to the extent possible with a you know asterisk for Adams really important remark about machine learning and AI and how that changes the dynamic a bit open code these transformations are critical to the culture of inquiry to research to discovery and while these implicate business models they are themselves not business models and indeed have the benefit of being being able to you be used for an array of business of uses of uses commercial not for profit and others so the revised 2022 public access guidance really highlights the importance of the American's public access to taxpayer supported research and this is a criteria that can be accomplished in several ways you don't you can use an agent designated repository you don't have to pay APC fees and you know I think the hope and we'll have to see how it's implemented by Tara and others will be that it helps to bring more kind of dynamism and flexibility and the larger ecosystem that you all help to catalyze and create so what I'm offering here asking here is a kind of reassertion of and a commitment and a recommitment from the open science community to the public mission of open science to improving the craft of research and its transparency and accountability while also strengthening scientific integrity to do it truly in a way that offers public benefit and does not double down on business models again going to skip really important observations at the beginning that really constrain access to these broader communities that are invisible to some of us but not all of us second open doesn't mean accessible or inclusive and I think that you know open science communities open software communities often just assume that because something is open anyone can come and anyone feels welcome to come and you know I hope that we know particularly those of us who lead organizations those of us who work in the social sciences actually just know that's not true and so while the kind of founding mission and vision of this organization and this community around openness was important it is not sufficient and that you know that work needs to be done to expand the institutionalist institutionalization and successive open science in its next decade is going to have to deal with the fact that all researchers don't feel welcome in the space of open science and the scale that Adam is talking about is going to require more participation from more people in the open science ecosystem for example you know a real commitment to providing data in ways that are recommended in the 2022 OSTP menu memo that enhance accessibility to assistive devices by using machine readable research findings and data not just research findings data but research findings and data that we can make available to people who might not have access to them otherwise. Also in the same vein a deeper commitment to finding ways to include those who may not always be able to bear the work of experimenting by publishing in PLOS so some of my you know I've published in PLOS a few times and but that was because I was teaching at Yale and I could publish you know there are a lot of people who can't take what seems to be the risk of working in experimental journals and so how do we sort of change that sort of calculus for people and how do we help to support that work. I wanted to highlight the work of Dr. Denae Ford Robinson at Microsoft Research so I hope is a name that's familiar to you if I'm not getting any looks of okay it should be a name that's that's really familiar to you she publishes as Denae Ford. I've learned a great deal from her about this very topic so her work identifies the cognitive and social barriers to participation and online kind of socio-technical ecosystems in particular she looks at github communities she looks at open science and programming communities open software communities and she seeks to understand empirically often with experimental research and sometimes with behavioral research why people don't participate in those communities. You know why people comment on or you know thumbs up to some people's codes and not others people's codes and these sorts of things so this is the kind of research you know the meta research the meta science I think that will give us observations that will help us to be more sort of equitable and inclusive open science communities. Lastly be careful what you wish for great success so successful in fact that the transformation of science has been well beyond the field of experimental social science and natural sciences and physical sciences you have succeeded in establishing open science but also created new norms for the research enterprise more generally so I would remind you those that you don't know that the ten year anniversary of the center for open science is also the ten year anniversary of the national endowment of humanities open book grant program so you have had fellow travelers in this mission for a very long time and my last kind of challenge or provocation would be to work outside of the space of experimental science RCTs and the like and to think about how the scale that Adam really reminded us that we you seek and need will require us to work across research communities that are doing working in different methods and spaces the rise and fall of the replicability crisis has rightly returned our attention to other methods maybe replication is one goal but there are other critical goals including transparency, accountability and the broad benefit and dissemination of research and knowledge to a broad set of audiences and the caution there is just not to have replicability be our kind of north star for this work so just to close what lies ahead I think is a future for the research ecosystem that has been much more transformative than I think you all even imagine because you've changed the whole ecosystem not the kind of particular sector that I think you imagine that you are working with and many of the kind of tenets, bylaws charters of the organizations we're creating of the organizations we work in and do our work in ask us to do our work in the spirit of the public good and I would just challenge all of us to continue to think about open science as one of the vehicles to live out that mission thank you