 Hi folks, we're going to go ahead and get started. My name is Martin Halbert. I'm the NSF program director for public access, and it's my pleasure to introduce this panel. We have four presenters for you, two of whom are remote. I'm joined here on the panel dais or whatever with Louis Barbier of NASA, and virtually we have presentations from Miriam Zarin Halem from the White House Office of Science and Technology Policy, and Andrea Medina-Smith from the National Institute of Standards and Technology. They send their regrets, they were not able to attend in person, but have recorded their presentations for you. We are going to try and reserve a little time at the end for Q&A as well, and we've got a couple of mics that will work. So with that, I wanted to start with Miriam's presentation to set the stage from the White House, especially on the topics of the Nelson Memorandum. Brian, can you go ahead and start the video? Good afternoon. My name is Miriam Zarin Halem, and I am the assistant director for public access and research policy at the White House Office of Science and Technology Policy, or OSPP. And here, my work centers on promoting open, equitable, and secure research practices, advancing the Biden-Harris administration's commitment to providing public access to data, publications, and the other important products of the nation's taxpayer-supported research and innovation enterprise. And I'd like to start by thanking the organizers of the Coalition for Network Information for hosting this meeting, and thank you to Martin Halbert for organizing this session. I'm really grateful for this opportunity to highlight the really exciting work that is happening across the United States government in this year of open science and really beyond. So the year kicked off with the unveiling of a U.S. government-wide definition for open science, and this definition was developed through the National Science and Technology Council subcommittee on open science, which is made up of representatives from research agencies across the U.S. government. And collectively, we've defined open science as the principle and practice of making research products and processes available to all, while respecting diverse cultures, maintaining security and privacy, and fostering collaborations, reproducibility, and equity. Now this definition brings together a number of the Biden-Harris administration's work to strengthen, advance, and use American science and technology to achieve our nation's great aspirations. It leads with a commitment to enhancing access to the products and processes of research, which has been really foundational to all definitions of open science that we've come across. It's inclusive, noting that these research outputs must be made equitably available to all, whether they are researchers, students, policymakers, community advocates, professors, small business owners, or members of the broader public. It notes the need for members of this open science enterprise to respect diverse cultures in their pursuit, instilling a sense of curiosity and humility while reinforcing that inclusive spirit of open science. It also reinforces the need for consideration around security and privacy when making decisions about what tenants should be shared broadly, and it ends by looking towards desired outcomes of open science, namely opening up more opportunities for collaborations with diverse stakeholders across society, enhancing reproducibility by increasing access to the data and tools underlying research findings, and supporting equitable access to those findings and outcomes. Now at OFTP, we've been really inspired by the push for openness, and this slide is intentionally blank, so do not worry. We've been inspired by the push for openness in research by community activists, researchers, publishers, higher education leaders, policymakers, patient advocates, scholarly associations, libraries, philanthropic organizations, and the broader public. And we've been inspired both by the ideals of this movement and their ability to deliver impactful results. For example, practices like open source software and code have helped facilitate transparency while enabling greater cooperation and fostering replication in science. For example, the Intergovernmental Panel on Climate Change published all of the source code underlying their figures with these goals in mind. And we've also seen open science enable greater awareness, participation in, and understanding of research. For example, data journalists and artists have leveraged open climate data from NOAA, NASA, and other institutions around the globe to illustrate and describe a change in climate and share the evidence that carries the stories of the communities impacted by the effects of climate change. Now, if we zoom out to look at the administration's aspirations from curbing greenhouse gas emissions to reducing social inequalities to ending cancer as we know it, all while bolstering public trust, strengthening our decision making, capacity, catalyzing discovery and innovation, and driving equitable outcomes for all across our nation, these are really complex and multi-faceted challenges that require a diverse and collaborative knowledge base. And so advancing open science policies, building on past successes to pave the way for more open, equitable, and secure scientific research is really critical to delivering on those aspirations for the American people. And to advance that vision for open science, the White House declared 2023 a year of open science, announcing a slate of actions from federal agencies to advance open and equitable research. Of course, this announcement comes on the heels of OSTP's memorandum titled, Ensuring Free Immediate and Equitable Access to Federally Funded Research, issued by Dr. Alondra Nelson to agency and department heads on August 25th, 2022. Now this memo builds on and strengthens public access guidance issued by OSTP in 2013. And the driving principle motivating this updated policy guidance can be summed up in the memo itself, where Dr. Nelson writes that American investment in research is essential to the health, economic prosperity, and well-being of the nation. There should be no delay between taxpayers and the returns on their investments in research. And since that memo was issued, we've been hard at work coordinating and collaborating with our agency partners around its implementation. And I'm sure my colleagues with you at CNI today can share more about their plan development efforts. And as of February, all agencies with over $100 million in annual research and development expenditures have submitted their plans for public access policy development to OSTP, which marks a major implementation milestone. And agencies with less than $100 million in research and development expenditures are working hard to submit their draft plans to us by August 20th of this year. Of course, the celebration of the year of open science goes beyond public access, with agencies moving forward with a slate of commitments to advance a vision for open research that truly benefits every corner of research and every sector of society. From NASA announcing its Transform to Open Science program with opportunities for students, researchers, and the broader public to engage in open science, really placing an emphasis on engaging with people and groups who have been historically underrepresented in science. To the national endowment for humanities issuing a funding opportunity for its 2023 Digital Humanities Advancement Grants program, advancing humanity scholarship by enhancing or designing digital infrastructure that is open, accessible, equitable, replicable, replicable, and sustainable. And these commitments also include funding and training opportunities and open data and open government developed by colleagues at NSS, NOAA, and the US Digital Core, and improvements in infrastructure that better enable open science and enhance public access led by NIST, the Department of Energy, and the USDA. And these programs together engage various actors across the open research ecosystem, from those who are newer to the open science space, whose participation helps us realize that inclusive, equitable, and sustainable vision to those who have been working for quite some time at the leading edge of open science. So as we continue into 2023, I wanted to direct you to this fantastic new resource, open.science.gov, developing a collaboration between SENDY, an interagency group of 10 federal agencies working to improve productivity of US federal research and development efforts, and the National Science and Technology Council. And here you can keep track of what agencies are doing around the year of open science and look for opportunities to get involved. And so with that, I thank you for your attention and turn it over to my great colleagues on this panel. Thank you. Just making sure it's working. Good afternoon, everybody. I'm Louis Barbier and nobody knows me, so I'll say a few quick notes about myself. I'm a particle physicist by training, been at NASA for 36 years. I joined the Chief Scientist Office back in 2014, about nine years ago. And when I did, the first thing I was asked to do was to develop an open access plan for NASA. That was right after the Holdren Memo had come out under the Obama administration, and the Chief Scientist was charged with developing that plan. And so I led a team to develop NASA's first plan. But let me say from the very beginning that open access has always been one of NASA's goals. NASA data, almost from the very beginning, has always been available. We have data centers and archives all over the country. That data is freely available, always has been, always will be. But what you'll see when I'm going to talk about today is how we're going beyond that. We're extending the reach of our open access commitment. So just a little highlight, you just heard from Merriam, you know about the August 2022 memo to increase access for the public to scientific research. NASA has this as one of its core values. I'm going to skip ahead just to make things a little bit more interesting. Here are our core values. Openness is fundamental, it's essential, security is essential, freedom and integrity are crucial. We have a lot of problems in the science community today with reproducibility, with things not being as, let's say, open as they should be. NASA is committed to going forward in a way to make these things better. We want to increase accessibility. I'll talk about that in a minute. We want to increase inclusion. And then there's a burden that people, the researchers face when they want to do this. You have to pay somehow, somebody's got to pay somewhere, so there's a burden. So we want to do our part to try to relieve that burden wherever we can. So I'm currently leading the NASA team that's responding to the August 2022 memo. We're working, as you just heard also, we submitted our revised plan to OSTP in February. So I can't say anything about the details of the plan because it hasn't gone through all the vetting yet. But I think when you do see that plan, you'll be hopefully happy. We want to change the way science is done. And science needs to be transparent. Sometimes, you know, scientists can be very close-minded. They get in their little shells and they like to do things in very obtuse ways. I have chastised many of my colleagues over the years for being too obtuse and not saying things in simple words when a simple word will do. So I think people need to learn to be transparent and not cover things up with gobbledygook, as I'm probably doing right now. And science needs to be inclusive. So in the plan this morning, I was very happy to hear one of the speakers, and I don't remember which one, mentioned citizen science. So NASA, and this is really separate from our public access plan, but I wanted to just say that NASA has a strong citizen science program. We have hundreds of citizen scientists organized working on NASA-specific problems, and they're doing real science. They're discovering exoplanets. If you have time, you want to look at something interesting. You can see something called Zooniverse. That's Z-O-O universe. Fantastic work that our citizen science community is doing. So we're moving to be more inclusive. You can't do anything without data. You see there data. Data is very important. We're swimming in data. We're drowning in data. The James Webb Space Telescope, which was launched just back in last December, it sends down 20 gigabytes of data a day. We have Earth science missions that are sending down terabytes of data every month. We have petabytes upon petabytes of data in our archives. We clearly do not have the resources to analyze all that data. We have only finite number of people and finite dollars. So it's critical to get the rest of the world behind us to help us to analyze all this data, rather than just have it sit there and stagnate. And I always like to say this, and I was also very pleased in the previous session I was just saying to hear somebody talk about humanities. One thing that people need to realize is data is data is data. It's all ones and zeros. And the tools you use to develop something for application X can be applied to application Y and can be applied to the humanities. And now this is no longer NASA, but I'm just trying to say that data can be handled with the same tools if we all work together. And then, of course, reproducibility. We want to be able to be sure all of our stuff can be reproduced. That's critical to the scientific method. So I'll probably skip this in the interest of time. And this is sort of what we've been doing. We updated our plan. We send it to OSTP. The significant changes. Now I can't say specifics, but these are the significant changes. And we're going beyond what the August 2022 memo wanted. We're going to go beyond that, because I think that's the NASA way. So we're going to remove the requirement for the one-year embargo on publications. That's what's been called for. We're also asking all of our researchers to release all of their software. Not just some of their software, but all of their software should be made public in a forum in a format that people can understand it and use it. That's, I think, really critical. That was not a part, really, of what was called for in the OSTP memo. And then sharing of data, not just the data that supports publications, but, again, all of our data. So these are the things where we're trying to sort of push the envelope maybe even a little bit further than what's just required. Because I put this in, probably shouldn't have. This is the history, and history's not always my forte. It's not really so important. Because what's really important is what happens at the end of that timeline. And that's what's on the next page. Transformation to open science. So the federal government has declared just a year of open science. NASA is doing transformation to open science. It's a five-year program. We're spending $25 million to support our researchers, support the community to make everything open access. And this is something that I don't think any other agency has done. It's certainly a first for NASA. So it's a commitment from NASA to open science for the next five years. And as you can see there, we're partnering. We're not just alone. Now we've got 12 other agencies who've joined in with us. So I think this is going to be really transformational. And I'm probably out of time, so I'll just say that this, of course, will have a huge impact on the sociology of how we do science in this country. And we actually need sociologists. There are any sociologists out there. We need you. We need sociologists to help us. And I'm going to skip to the end. Here's the big thing. More data and more people equals more discoveries. Thank you very much. Thank you, Lewis. Okay. We have Andrea Medina-Smith's presentation is prerecorded, and we're going to make it go. Hi. My name is Andrea Medina-Smith, and I am a data librarian at the National Institute of Standards and Technologies Library and Museum. Today I'm here to talk about data in the wild, open data at NIST. And through my presentation, I'd like to describe some of the successes we've had and some of the challenges we still face with publishing data at NIST. Specifically, I'm going to be looking at what happens to our data once we've published it and how we can analyze and track that reuse. So first off, I would like to read. So where are we? Well, in the fall of 2015, our public access policies began, the mandate began, and in them we recommend and strongly encourage publication of data sets that back up our other research outputs. So papers, technical reports, conference proceedings, anything like that. We want you to publish your data in our data repository. We also strongly encourage publication of standalone data sets if there is value to those data sets, even if we are not going to write a paper about them. So in those, let's say, seven years, more or less, that we've been doing this, we have published over 800 data sets. There are nearly 60,000 downloads, various files or entire data sets. We also have had almost 30,000 unique users make those downloads. We publish multiple new data sets every week. And all of this is great. It's wonderful to see that we're publishing, that more and more of our researchers are thinking about data as a first class research output, and they're working hard to make their data as fair as possible, as findable, accessible, interoperable, and reusable. And one of the ways they do that is through publishing in our data repository. However, we could do better. We publish approximately 1,500 publications every year that are journal articles, books, technical reports, all that, and we would love to see the data immediately being published at the same time. Anything that's backing up a paper, we want to see that. So we're encouraging this, and as we develop our new revised public access plan in response to the OSTP 2022 memo, those partitions will be updated and changed as well. So one thing that the users and my researchers constantly are asking for is, well, you know, what happens to my data? Once I put it out there, I know it's for the good of science. It gives me warm fuzzies, well at least some of them say that. But others just, you know, they don't necessarily want to do this just as a compliance thing. They want to know that it's going to have some benefit to them. And not just for the good of science and the like. So what we want to look for is evidence of the data reuse. And you know, I can talk all day about how there's a citation advantage to papers that have data cited in them, their citation advantage to open access, and other open science features. But that only goes so far, especially when we start talking above the researchers and looking at the policymakers and others in admin. They want to know the stories of what this data, the impact this data has had. So currently we don't have a great way to track that usage. Right now we have a manual process to search for reuse. Basically I do multiple searches using both the DOI assigned to the data set and the title. We search web of science, Google scholar, plain old Google. We do a lot with these and often we can't find that much evidence of reuse for a few reasons. Data citation or a data availability statement is not necessarily standard in all publications at this point. It's starting to be, but it's not there yet. Another reason is, oh, some of these data sets have 50 different names that they're known by. And sometimes, instead of citing the data set itself, especially when you're able to just pull it down and use it for reuse, often papers that describe the data set are still being used as the citation. So this could be either to a journal that specializes in data papers, which are just descriptions of data, or excuse me, this could be a case where an experiment is run, they describe the experiment. This is a regular general article where they say, here's the data that I used, that data gets reused and the paper is still cited. And that's not necessarily the appropriate thing to be doing, but that is still what happens. So in a kind of perverse interest, I took a data set of my own that I published within our data repository and I looked at the metrics that were available for that and the citations that I could find. So it is a SCOS version of data vocabulary and we can see here that it's been downloaded approximately, well, depending on if we look at just the file downloads or data set downloads, let's just say it's been downloaded 500 times, which is amazing. Our metrics tell me how many bytes total have been downloaded and that there's almost 200 different users that have looked at this. And it was most recently downloaded just a few days ago, which is fascinating. And we've been keeping these metrics and counting these metrics since May of 2018. So we've got about five, almost five years' worth of data on this. All right. And, okay, so we can see some sort of use here. We can't really say what a download means, it might be just someone looking through, they might not do anything with it. So we don't really know what's happening with those downloads, but people are looking at it, which is good and exciting. All right. So what about citations? What about that level of impact? We can see that it has some use and on the left-hand side of this slide you can see a screenshot of our landing page within the data repository here. So what I've been able to find in Google Scholar, there were two citations to the data set itself. There were two mentions of this that I found on Google that aren't actually citations and that came through with searching for the title and those were both in conference presentations. And then finally two citations to the paper we wrote about this program that should have actually been to the data set itself. So that's somewhere between four and six citations that show the type of people that have been using it and where it's been folded into other projects. But that's about all that I can get right now and I know that we'd like to see more. So where can we go with all of this? Well what we need is an efficient way to understand data reuse, both data reuse and the real world and scientific impacts of a data publication. Now we're not going to be able to say that because I published this vocabulary on materials science that it led directly to the creation of a new material for auto-lightweighting. We know that drawing those direct lines are going to be nearly impossible if we ever did anything like that, but there are other things that we would like to be able to see. We'd like to be able to see how industry is using our data. We'd like to be able to look and say, oh, this data set was also used in a teaching laboratory or this data set is backing up software developed by XYZ for usage in materials manufacturing for automobiles again. But it's really hard to find those things out because they're not published in the traditional manner. But there are some things that we can do to get there. First of all we want to continue developing policies that promote open data. We need publishers of research journals to require data citation or data availability statement, something that says this is where the data is I used in this paper so that we can track reuse when it happens. We need to work with industry to understand how they're using data produced by federal agencies and once we get all of that we need to continue to have open metadata to harvest and analyze reuse. So those are the things we'd like to see from the perspective of a federal librarian. We want to see these new public access policies continue to build on where we came and help us get to a place where we can be talking with our researchers and be able to say, hey, you know that publication or that data set that you published in 2025? Well this year we can tell you that not only have six other researchers used it to back up research in their papers but it also led to these improvements within industry. So thank you very much. If you have any further questions or comments you feel free to let me see if I can pull mine up. And in the interest of time I'm going to curtail a number of my slides because I'm determined to keep about 10 minutes for Q&A. So and my colleagues have anticipated a lot of the things that I was going to cover. So again Martin Halbert I'm the NSF program director for public access. Delighted to be here. I did want to say just a one word, one slide on to amplify what Merriam said on 2023 as the year of open science. This is a effort of more than a dozen U.S. federal agencies that we are trying to align our practices, policies and do synergistic activities in the year to foster greater engagement with open science throughout the country. And if you are interested in more details and this site will have regularly updated announcements open.science.gov take a look at it. Again if I had more time I think I'd go through a lot more detail but again most of these points have been anticipated by my colleagues. One thing I did want to say just that's particular to the NSF going back to the foundational document that created the NSF that led to the creation of the NSF and largely in many ways the contemporary landscape of federally funded research on this report from 1945 by Vannevar Bush yes that is the way you pronounce the name of our patron saint here at NSF. There was a lot of people don't realize that right in that report right at the end and the concluding last bit of the appendix it was entirely devoted to what we would now consider concepts of public access and this is coming directly out of World War II when there was a great you know more concern than there is now about keeping secrets. So I would direct everybody to take a look at that document at the endless front here it's still every bit as relevant today as it was then you've heard about and many of you know about the Holdren memorandum salient feature there was the one year embargo a period articles after publication at NSF it was implemented under France Cordova with a both just a linger on that for a minute we've always accepted either author accepted manuscripts or version of record in compliance with the policy 2022 memorandum had it's a very dense memorandum if you haven't taken a look at it I encourage you to take a look at it and all the different stipulations of it the things that have gotten a lot of attention are the move to zero embargo a period articles and underlying data I do want to highlight some recent initiatives at NSF that we've undertaken first time ever solicitation in 22 the pharaohs RC N solicitation which led it's a first agency wide solicitation that was directly trying to advance efforts of open science in the country it led to 10 collaborative national projects which are comprised of 28 distinct awards total of 12.5 million and awards most of them are three year projects some of whom are presenting at this conference so I encourage you to go take a look at them we've been issuing a number of reports recently there are a number of NSF things that we've surfaced heard from our community in sessions again I'm going to kind of zip through these because I do want to get to the Q&A this emphasis on equity is a very big topic both within the memorandum and has received a lot of attention within the agency lot that we have thought about in terms of principles and values that I won't go through all of these slides heard a lot also about costs and concerns from the field and different stakeholders in the field that we are taking into account and trying very hard to represent and balance all the interests of the different communities that we serve finally just the timeline just to echo what what Lewis put up August 25th was the issuance of the memorandum we did all turn in our our new plans February 21st I had hoped that maybe by the time of this event maybe that our plan would be public we still have not gotten clearance from the White House to release our plan but in lieu of that I will try and endeavor and speak to whatever I can in Q&A about it note that there are two subs at least two actually as much timeline goes on after this additional points in that the memorandum calls for first it calls for the primary implementation or the the implementation of the primary requirements of the memorandum by the end of 2025 NSF intends to implement them by the beginning of 2025 just right now that's our plan anyway and then there is a second set of stipulations in section four of the memorandum that get implemented by 2027 so with that sorry for skipping through the slides but I did want to reserve some time for Q&A so with that do we have any questions or comments from the audience that that Lewis and I can try to respond to and if you if you want to come on up to the the microphones and Pedro is going to activate them for us back there go ahead hi Lisa Hincheliff at the University of Illinois at Urbana Champaign it strikes me as I look around the room there's a lot of librarians here and so one of the things I'm wondering if you're able to you've been hearing from a lot of different communities as you've done these stakeholder engagements and I'm wondering if you can share back to us particularly what you are hearing from the researcher community as well as from sort of the academic leadership at the institutions like the University of Illinois at Urbana the research associate chancellor associate dean whoever it is that sort of is overseeing the research enterprise from campus and particularly how they are grappling with this as you know a coming compliance mandate I mean that I'll start and I'll invite Lewis to many of these in gate we have conducted thank you for acknowledging that we tried very hard to set up a number of different engagements with the community we will be doing many more especially this year in 2023 we have heard a lot of concerns of various kinds around compliance just from the researchers the kind of the typical concern of additional researcher burden and we're very sensitive to that there is concern about you know complying with the zero embargo requirements from the memorandum both in terms of peer reviewed publications and the underlying data let me highlight it's not the memorandum is not calling for all data to be you know immediately made available it does not supersede any existing laws in terms of you know restricting access to personally identifiable information national security secrets any of that stuff so we've heard certainly from offices of research about okay what are we going to be obligated to comply with in terms of you know being as as institutions that are the basically the ones that get these awards and that the the researchers are obligated to you know be the ones that execute the projects but what are the the institutions the university is going to be on the hook for again we're trying very hard to take all of that into account in how we respond to the memorandum let me just say one last thing while you know the our member our new public access plans from all the major agencies are going to be coming out within weeks let's say this is not the full policy development it's not the full policies not the full practices and requirements that will be worked out between now and at NSF the beginning of 2025 so that's one of the reasons why we're really interested in doing these engagements to hear from as many different stakeholder groups as possible what you're going to see largely in many of these agency plans is going down the list of all the stuff that was in that memorandum saying yep we're going to do that yep we're going to do that yep we're going to do that you know without going into a great deal of detail in most in many or most cases about how it's going to be implemented yes and Lewis there we go yeah I just wanted to say yeah that so so at NASA we have we will be going out with a broad RFI as soon as the plans are allowed to be made public we're going to have a broad RFI released and in the next I would assume between now and the end of the year we hope to have several session listening sessions with community members to get their input and as Martin just made very clear this is the first step this is a plan not a policy hi Kyle Remkes also from University of Illinois wondering what the implications of this might be for institutions that run their own institutional data repositories part of what I've heard people saying is that what's going to happen is that each federal agency is going to have its own data repository that will only be used for data that they funded and that might change the nature of what institutional data repositories collect because right now they're the ones hosting a lot of that data so is that where things are going and what do you see the implications of this could be for just broader data repositories great comments from our colleagues at Illinois today that is a great comment indeed the I can attest that not all agencies are going to stand up their own data repositories NSF is not we are we already implemented our mechanism for reporting and making publicly accessible data sets in 2021 while it was manned I mean not mandatory it was optional it was voluntary when we implemented it we already have had some I think 600 or more voluntary data set deposits in the system we are taking as probably you might expect from NSF we're very bottom up distributed kind of organization we allow data and we encourage data to be reposited in any of the many long-established disciplinary and in institutional repositories that NSF has often funded over the years and made major investments in we do not want to undercut any of those investments the only thing we do require is that the data set be assigned a DOI persistent identifier that represents long-term a long-term commitment on the part of the repository that stores it to make it publicly accessible without cost or delay so and I can't speak for other agencies I know I'm sure some are probably creating centralized repositories but that is not a universal requirement and we actually at NSF fully support institutional repositories as an option do you want to add anything from the NASA perspective I don't I don't think I have anything to add I mean that NASA we of course have our own repository archives which we've had for decades focused on our five you know sort of core scientific areas excuse me but the idea of making data more widely available and sort of interlinking data sets is of course I think very important but I don't think we have thought about how to do that yet hopefully that will come out in future discussions and hopefully you can help us excuse me sorry just just a final point on that that was mentioned in your programs we in the interagency space we did spend a lot of time on this document with his very long name desirable characteristics of data repositories for federally-funded research and while that is a very short document it represented a lot of work to develop a consensus among the all the federal agencies that participated in it to capture the core characteristics of data repositories that we that we wanted to see implemented for data arising from federal awards yes this is very high point down there okay monoramnete Stony Brook University and this question is actually for you Lewis and you may have actually started to answer it you had mentioned that you would like all data to be accessible and openly available are you talking about curated sets are you talking about raw data can you clarify exactly what you have in mind for that yeah yes this would be curated data you know from from sort of approved projects you know and missions you know our mission data is already curated and archived and available now we want to go just beyond that to some of the smaller projects the more focused projects let's say but it would have to be curated data sets not just not just you know some sort of off the cuff kind of thing but curated data sets that we can validate and we can properly document and we are confident that this data is what we think it is what we say it is and that we can be trusted we're gonna go just a minute or two into the break we've got time for just a couple more questions is that Mackenzie I can't you're back with sorry Mackenzie Smith UC Davis and I have a follow-up question for you Lewis which is I happened to meet an early career scientist who works at NASA recently and she said that outside the mission data archives you know she's on board philosophically but when she asked for help within NASA she got nothing no guidance no standards no training no advice nothing so she's really paralyzed and I just wonder how you're gonna address that even within your own agency yeah yeah thanks that's great first of all tell her to call me give her give her my phone number in my email that would be I'd love to speak with that person but no she's right we don't have training right now and that's one of the things we are doing that you gonna it's gonna be coming next is training we just haven't had the resources up till now to develop that but we're gonna be doing training and we're gonna be not and not just formal training for the researchers but training for the managers the branch chiefs and division director so that they know what open data what public access means and they know how to support those people I admit it's not there yet I wish it was but it's coming I'll echo Lewis's point we I know we at NSF need to do a better job of giving a making training available for both individual researchers and cyber infrastructure professionals that support them Heather you're gonna close us out go ahead first of all thank you so much for a great session and also for everything that you're doing in the agencies my question is actually maybe a follow-up to Mackenzie's and it's with NASA's fantastic tops initiative it's actually a funded mission of tens of millions of dollars million dollars right to transform the agency over a five-year period to organize around open science practices being embedded throughout the agency so first of all freaking awesome thank you amazing and second of all any other agencies thinking about maybe following that blueprint a great question I don't know that anybody I haven't heard any plans to directly emulate tops I think she'll gentlemen if you have followed all of her great discussion of the initiative is doing an absolutely fabulous job yes kudos to our NASA colleagues you have really blazed a path with it yeah yeah I we would like to see other agencies do that when they can and yeah kudos not to me but to our colleague shell who couldn't be here she is like the John the Baptist of public access and open access and I think if she can corral other agencies she will because she's a dynamo while they are not the same exact kind of things the pharaohs RCN projects if you've seen them are exactly research coordination networks that's what this long acronym stands for the fair data guiding principles OS open science and research coordination networks that are based in different disciplinary context and across disciplinary settings to advance the adoption and development of open science practices nationally and I commend all of them I think there's a session by Matthew Mayernick for example tomorrow who is I don't know if he's in the audience but they are probably going to be talking about some of they are one of the awardees well with that thank you all for your attention and great discussion