 Hello, my name is Daniel Evans and I'm a Programme Scientist in NASA's Science Mission Directorate. This is a re-recording of a virtual community town hall that we held on March 3rd, 2020 to discuss NASA's Science Mission Directorate's implementation of dual anonymous peer review. On March 3rd, I was joined by two SMD leaders, Dr. Thomas Debucham, the Associate Administrator for NASA's Science Mission Directorate and Dr. Michael New, the Deputy Associate Administrator for Research. During the town hall, we asked anybody to submit any questions using the following link and later in this presentation, I will actually go back and review some of the most frequently submitted questions. Let me start by saying that NASA's Science Mission Directorate is strongly committed to ensuring that the review of proposals is performed in an equitable and fair manner that reduces the impacts of any unconscious biases. And so to this end, and motivated by an extremely successful study conducted for the Hubble Space Telescope, the Science Mission Directorate is conducting a pilot program in Rosers 2020 to evaluate proposals using dual anonymous peer review. Now under the system, not only it proposes unaware of the identities of the members of the review panel, but those reviewers themselves do not have explicit knowledge of the identities of the proposing team during the scientific evaluation of the proposal. This webinar covers four main subjects. Number one, we'll begin by introducing exactly what is dual anonymous peer review before moving on to discuss which programs are converting to dual anonymous peer review. Then we'll be discussing how exactly the proposing community can go about making their proposals compliant with this new method. And then finally discuss how proposals will be reviewed under the auspices of dual anonymous peer review. Let's start off with the motivation. What exactly is dual anonymous peer review? SMD is strongly committed to developing an excellent workforce, both within NASA and also within our extra mirror community. We have started a whole host of measures to develop the workforce. And one important technique that we're doing to do just that is dual anonymous peer review. I wanted to pay credit to the pioneering work performed by the Hubble Space Telescope team in implementing and standing up dual anonymous peer review. And indeed it's from Hubble that I'm showing the following plot. This is data from the Hubble Space Telescope for each of its annual call for proposals over the past dozen or so years. And you can see the inferred male PI success rate exceed slightly the inferred female PI success rate year after year after year. It's like flipping a coin and coming up with the heads 12 times in a row. And so that motivated the Hubble team to switch to dual anonymous peer review in cycle 26 and cycle 27 that you can see there on the right. The implementation of dual anonymous peer review first resulted in a slightly higher success rate in inferred female PI proposals. In 27 that train was actually flipped, but their data from their most recent mid cycle review shows that the parity exists between genders. In other words, it was a 43% success rate for inferred male PI and a 43% success rate for inferred female PI. But gender as course is one of the many, many parameters of diversity that we wish to think about. And it may mainly represent the tip of the iceberg. So what else are we missing to move further? This is a part of the success rate by institution type for roses programs in the National Science Mission Directorate pilot. You can see that government contractors, federal government, research one universities all have in excess of a 20% success rate for their proposals. But take a look at research to universities or minority serving institutions, the success rate declines dramatically. But I wanted to point out that one of the main motivations for moving to dual anonymous peer review is to the level the playing field for everyone. I should point out that dual anonymous peer review is not completely a blind process, however. And in that sentimentation of dual anonymous peer review, we require that our proposing community submit two documents. The first one is an anonymized proposal. The second one is a not anonymized expertise and resources document. And here that latter document allows the proposing team to demonstrate why exactly it has the qualifications experience resources that will make the execution of the proposed investigation successful. I'm showing now some feedback from the Hubble panelists. I just wanted to note Rachel Somerville's remarks. She was the chair of the cycle 27 Hubble Time Allocation Committee. And Somerville noticed that several time allocation members themselves noted that they felt the discussions at both the panel level and at the time allocation committee level to more collegial. That's emotionally charged and previous tax, perhaps because either positive or negative feelings about the people involved in the proposal were largely removed. Let's move on. This is a program to converting to dual anonymous peer review in NASA's raises 2020 pilot. We have selected for programs to be performed under dual anonymous peer review. One from each SMD division. We have the astrophysics data analysis program, otherwise known as Ada. We have the science US principal investigator. We have habitable worlds. And then finally we have the heliophysics guest investigator open program. In addition to that, or astrophysics guest observer guest investigator programs are permanently converting to dual anonymous peer review. We held a separate town hall for that on February 27. So how does the proposing community go about making their proposals compliant with dual anonymous peer review. I wanted to say that NASA has provided very detailed guidance to assist the proposing community with this major change in the submission of proposals. The first one is that every single program element, which will be performed under the auspices of dual anonymous has very specific program level guidance. And the inspires page of each program element contains a document, which we turn the guidelines for anonymous proposal, which describes in detail the specific requirements for anonymous proposals. Number two SMD has a whole host of resources, including a quick start tutorial, as well as frequently asked questions, both of which may be found at the URL provided on that slide. The five main points to remember when submitting anonymous proposals are these. Number one, exclude the names and affiliations of the proposing team, including in the figures and references to personal websites. Number two, this is an important change. Please do not claim ownership of past work. So don't say my previously funded work or our analysis shown. Instead, we need you to write, including references in the passive third person. So prior analysis, we need you to use square brackets. Number four, of course, please do describe the work proposed. So absolutely, you must say we propose to do the following or we will measure the effects of. And then finally, you'll need to include a separate not anonymized expertise and resources document. And we'll be providing more details for that later on. A very common question is how do you reference unpublished work or how do you reference priority results? So our advice is as follows, we understand that it may be occasionally important to cite exclusive access data sets or non public software or unpublished data or findings that have been presented elsewhere in public, but not yet sightable. And each of these may reveal or strongly imply the investigators on the proposal. So in these instances, we recommend that proposers use language such as obtained in private communication or from private consultation, when referring to such potentially identifying work. Now I can recall that the goal of dual anonymous is to create a shift in the tenor of discussion, not to make it absolutely impossible to guess who the team. Another common question is in proposals where a team member has institutional accesses to unique facilities such an observatory or a laboratory, both of which would be required to complete the proposed work. So in italics here I'm sharing an example in the anonymized section of proposal. You can say that the team has access to telescope time on the cake observatory. This will enable spectroscopic follow up. So that doesn't really give clues to the identities of the proposing team. But we recommend that you validate that claim in a not anonymized expertise and resources document. Here's an example of anonymization. The paragraph above has not been anonymized. The paragraph below has been anonymized and thanks to the Hubble team for allowing us to include this. In yellow. You can change in Roger set out 2014 we concluded to prior work square brackets 12 concluded in green if our model from Roger set out 2014 is correct. You would simply change to if the model from 12 is correct. And our first epoch you can change to a first epoch to again not signify ownership. A very common question, of course, is how is the capability of the team to execute the proposed investigation taken into account. And here this is where we provide details for what we term the expertise and resources not doc not anonymized document. This document will get distributed to panelists for a subset of proposals, typically the top third according to the distribution of assigned grades, projected selection rates and so forth. It contains many elements. A key summary is as follows, you need to include a list of team members together with their roles. Brief descriptions of the scientific and technical expertise that each team member brings a discussion of the contribution that each team member will make to the proposed investigation. And discussing the specific resources that are required. A summary of work effort to include a not anonymized table of work effort by a sketches if required, statements of current and pending if required, and that is a resource support again if required by the solicitation. And the guidelines for anonymous proposals document includes a work to example. Here's some other requirements which I won't go into in detail here but I just wanted to point out that the guidelines for anonymous proposal document actually spells them out in significant detail. So how will your proposals be reviewed. The flow of the reviewers as follows. You'll start off with an anonymized scientific review. This is where the review panel discusses those anonymous proposals. All assessments get completed or grades get finalized or panel summaries are written. And then number two, the expertise and resources document will be distributed to panelists for a subset of proposals. Typically the top third, a panelist go forth and assess the team and resources capability to execute the proposed investigation. What do we tell our review panelists. We tell our review panelists to consider proposals solely on the scientific merit of what's proposed. We ask them to not spend any time attempting to identify the PI or the team. And even if our reviewers think they know the identities of the team, we instruct them to discuss the science and not the people. We have NASA appointed levelers in each panel room to ensure that the discussion does not veer off. We also tell our reviewers to keep in mind that language can be very important. We ask them to use appropriately neutral pronouns so what they prefer or what the team has evaluated when they write their panel summaries. As the discussion unfolds, we have NASA appointed levelers in each room. In addition to panel support staff, the role of the levelers to ensure that the panel discussions focus on the scientific merit. And unlike the chairs, they're not listening for issues pertaining to the science, but what they're actually primarily focused on the discussion itself. Now, if the discussion does appear into comments on the proposing team or their password, the validity or their identities, we make sure that our levelers simply refocus that discussion. But if that discussion gets out of hand, which very rarely it will, levelers have the authority to stop the discussion. Let's move on to the discussion of the expertise and resources, not anonymized document. So again, the scientific evaluation of all the proposals is completed. Then we distribute this document to panelists for a subset of proposals, typically the top third, according to the distribution of assigned grades and projected selection rates. And we asked our panelists to assess the team team capability to execute the proposed investigation using a three point scale, uniquely qualified, qualified or not qualified. And we provide details of what each of those criteria and ratings means. And we set the expectation that the vast majority of the proposals, say 98% or so, will fall into the qualified category. Here are some answers to the questions that were submitted in advance of the web. The first one is this, if the identity of the proposing teams and institutions is shrouded in secrecy, then how are proposing teams and institutions to discuss their track record, their ongoing work, their complementary endeavors, their institutional assets and so forth. Our answer to that is as follows, the anonymized proposal set absolutely no prohibition on discussing these aspects. But we do ask that they be discussed without attribution to a particular investigator or group. And in situations like this, we simply recommend writing previous work. Instead of our previous work or using words such as obtained in private communication. Now remember, as always, proposers should be able to make their case through their description of their proposed program of observations and analysis. And of course, there is the opportunity for them to demonstrate and validate the claims using the expertise and resources, not anonymized document. And again, remember that the goal of dual anonymous is not to make it completely impossible to guess the identities of the investigators, but to shift the focus of discussion away from the individuals and towards the proposed science. Another question that was highly upvoted is this. Why not just solve the demonstrable problems in selection practices at the Space Telescope Science Institute, rather than change the systems across all of NASA's Science Mission Directorate. Well, firstly, I'd say we would actually dispute that there are any problems in the selection practices at Space Telescope, we actually think they run an extremely competent, fair and thorough review. But we would point out again, the key goal of dual anonymous is to level the playing field for everyone. And once more remember that this is a pilot study for just four programs out of over 100 SMD roses elements. Another question. While it's not possible for the proposing teams not to show any information proposals that might reveal their identities, such as the context and motivation of the proposed research, unique methodologies cited references, why keep reviewers guessing. Leading to undesirable consequences. Furthermore, the track records of the proposals should be part of the merits of the proposals. Once more, we respond by saying that it's entirely appropriate for the context and motivation be addressed in the anonymized proposal. But again, you must not attribute these to a particular investigator or group or claim ownership of that work. And again, the track records of the proposing team will be addressed in the expertise and resources not anonymous document. Three very similar questions for us, and they all circle around the concept of demonstrating whether there are sufficient institutional resources or access to custom built facilities or laboratories. And indeed, how in general does the proposing teams ability to accomplish the research get evaluated. I hope this webinar has served as a reminder that what we're really trying to do here is to separate out the anonymized science proposal from any claims of validation, which is subject subsequently get discussed in the expertise and resources, not anonymized document. So I hope that this webinar has provided a little bit of confidence to the proposal community that the qualifications teams and experience of the PI and track record are not entirely eliminated. But they are evaluated rather in a separate step after the scientific evaluation of the proposal is completed. Another question was that what do you expect the unintended consequences of this action to be. And does this really serve the meritocracy. The experience for the Hubble Space Telescope indicates that there were very few unintended consequences. But NASA is proactively taking steps to ensure the following, but the SMD programs in this pilot, then themselves to dual anonymous peer review that they were very carefully chosen. We're also ensuring the proposals have sufficient information and guidance to adequately anonymize their proposals. We're also making sure that review panels are sufficiently briefed about dual anonymous peer review. We're also trying to make sure that the duration of each panel is not significantly increased. Once more, we're trying to ensure that conflicts of interest are identified ahead of time and not during the review if at all possible. And then finally, we're trying to make sure that high risk, high impact proposals are not disproportionately affected. We have a new SMD blue ribbon panel to ensure just that. NASA understands that dual anonymous peer review represents a major shift in the evaluation of proposals. We understand that there may be occasional slips in right to be anonymized proposals. However, NASA does reserve the right to return without review any proposals that are particularly egregious in terms of the identification of the opposing team. We further acknowledge that some proposed work may be so specialized that despite good faith attempts to anonymize the proposal, the identities of the PI and the team members are readily discernible. But as long as those guidelines are followed, NASA will not return these proposals without review. Finally, we make sure and we encourage all of our PIs and proposing community to plan adequately and feel free to contact your program officer or to email this address very much for listening.