 So welcome to the Dual Anonymous Pure Review Town Hall for Planetary and Exoplanets Research Programs. My name is Delia Santiago-Madarice, the lead for the Dual Anonymous Pure Review process under the Planetary Science Division out of NASA headquarters. And we'll have program officers from the programs that will be doing Dual Anonymous Pure Review and they'll be introducing themselves shortly. First of all, please submit any questions you have during this presentation via the link here. It is also posted in the chat. You can upvote questions already posted as well. In addition to that, you can ask questions in the chat of the presentation. Due to the large number of people, we're going to field questions in that manner and you might get some of the answers back to you from the other program officers in the chat but we'll also be answering some of the questions at the end of this town hall and we will also have, we already have some pre-prepared answers for the questions that were posted earlier. So this slide is going to be a quick overview of Dual Anonymous Pure Review, which we fondly call Dapper. And I'll be going more into depth into all of these points later on. But overall, we have this great website on Dual Anonymous Pure Review. Also, note that these slides will be posted there later on so you can access all this information and a lot more at this website. The goal of Dual Anonymous Review is to reduce implicit or unconscious bias in the evaluation of the intrinsic scientific merit of proposals. And the way we do this is that in addition to the proposers being unaware of the review panel member identities, the reviewers themselves are not told the identities of the proposers until after the evaluation of intrinsic and scientific merit. So the process for proposers, and again I'll go into this more in depth later, is that proposals are written to exclude any personally or organizationally identifying information in the scientific portion of the proposal. And they must upload a separate expertise and resources not anonymized document, which contains all of the personally or organizationally identifying information. For the reviewers, this means that they evaluate the intrinsic scientific merit of anonymized proposals without knowing the proposing team who they are. And after the scientific evaluation is finalized for proposals, the panelists will review the expertise and resources anonymized documents to assess whether the team has the qualifications and capabilities necessary to do the proposed work. So we'll be going over what programs are going to be converting to Dapper dual anonymous peer review, what exactly is dual anonymous peer review, and why is it important, how to make your proposal compliant, and how it's going to be evaluated. So which programs are converting? So first of all, we've already done a number under roses 20. These were all inspired by the work of the Hubble Space Telescope Reviews by the Space Telescope Science Institute starting in 2018. But under roses 20, SMD did a pilot with one program from each of the visions. So the astrophysics data analysis program, earth science US participating investigator program, habitable worlds from planetary science, and the heliophysics guest investigator program. So habitable worlds is still under review, but the other three have all been completed. And I'll be showing some information on that. In addition to that, the astrophysics guest observer, guest investigator programs are permanently converting to dual anonymous peer review. And a note on the programs I just mentioned that were piloted under roses 20, all of those will be continuing under Dapper under roses 21, habitable worlds is not being called under roses 21, but the other ones are going to be continuing under Dapper. For 2021, of interest are the columns on the right of planetary science and cross divisional. We have our DAPS here, and the exoplanet research program, which is largely managed out of planetary science. So to go more in depth about the planetary science division programs, first of all, under roses 20, we have habitable worlds, which is managed by Lindsay Hayes and Becky McCulley-Rench. And I believe Lindsay is here, if you want to say hello. She might not be able to say hello right now, but Lindsay is here, and they've been working on this review. And the proposals have been submitted, the review process is underway. Just a few proposals were returned without review because of egregious rule violations. But most other non-compliances were allowed through PI's were given a warning letter, but the proposals are still being reviewed, and we expected this under the first year. So I'll go more into depth about some of those violations, non-compliances later on, once you have a more clear idea of what one should be doing, but really quickly, the most common issues are failure to follow the reference numbering scheme that you're supposed to follow, and then just slips, basically. So someplace, it will say, you know, CoI and then give a name, and then later it refers to them anonymously. So just small little slips like that. Under roses 21, we have all of the data analysis programs, as well as the cross-divisional exoplanet research program. So here are the list of program officers, and we can start from the left if folks want to introduce themselves, starting with Cassini Dap. Hi, this is Henrik Schrupp on the Cassini Dap, a CDAP, and also on New Frontiers Dap. Thank you. Hello, I'm Doris Thao, and I'm the program scientist for DiscoveryDap. Hi, I'm Shisha Noida. I'm the lead for the Lunar Data Analysis Program. Glad to see so many people online today. And I don't know if Mitch Schulte or Adrian Brown are here, but they lead the Mars Data Analysis Program together, and I believe our XRP leads here. Hi, everybody. I'm Megan. I'm the lead from PSD for XRP. It's good to see everybody. Thank you. So now we'll talk about some of the motivation. In summary, this is also called Double Blind. We tend to say dual anonymous now. Dual anonymous review is to address issues related to implicit bias. So a classic tale is that in the 1970s, the top five orchestras in the U.S. had fewer than 5% women. When blind auditions were instated, that went into the 30s, 30%. And another example is with behavioral ecology. They switched to double blind review, which resulted in a significant increase in the number of female first authored publications. So first, it's difficult to completely interrupt implicit bias through training. It's also referred to as unconscious bias because we aren't always aware of our biases. So structural changes are needed, which is why we're doing this. We want to thank the Hubble Space Telescope team for pioneering dual anonymous peer review. And here's some data from them. So this was when they, before they switched to dual anonymous peer review, you can see the proposal acceptance rates between male and female PIs. And so this is the percent of male PIs that were accepted and the percent of female PIs that were accepted for a given cycle. And you can see that there is a difference. And theoretically, they should be, there's no reason they shouldn't be accepted at the same rate. So when DAPR or their dual anonymous peer review was introduced, this difference decreased. It didn't completely go away. And you can see for cycle 26 that females were slightly overrepresented in the next two years. They were slightly underrepresented compared to the percent acceptance rate of men of males. But this difference decreased. And that's the overall point. And one point is some preliminary data we have from the astrophysics data analysis program. So this is a slightly different chart. This is the percent proposals with inferred female PIs. And the total pool is the leftmost group. And then on the right are the top two and top three proposals from every group. And blue is non-DAPR. And you would, theoretically, there's no reason that the, you know, between year to year that these wouldn't just go straight across. But you can see that the percent proposals with female inferred female PIs were underrepresented in the top two and three proposals in each panel before DAPR. But with DAPR it reached more parity with the percent that submitted. So gender is just the tip of the iceberg. Another thing we're always looking at is success rate by institution type. These are the success rates for the ROSES programs in the SMD pilot under ROSES 20. And you can see that government contractors and NASA centers are selected, have their proposals selected at a higher rate than non-R1 or R2 academic institutions or minority survey institutions. And there are a lot of factors at play. We don't claim that bias is everything, but the idea is to play, to level the playing field for everyone. It's not a silver bowl, but it doesn't solve everything, but it's one tool to help level the playing field with everyone. So what exactly is it? Again, the proposals are unaware of the identity of the members on the review panel, but the reviewers do not know who the identities of the proposing came during the scientific evaluation of the proposals, of the proposal. And the intent is to eliminate the team as a topic of discussion, not to make it impossible to guess who might be on that team. We know that people have very specialized fields and there are, you know, you might feel like you can narrow it down, but the idea is not that you can't guess at all, but the idea is we're not talking about this. And we want to create a change in tenor of the discussion, away from the individuals on the proposing team and toward the proposed science. So it's not completely blind process. We have proposers submit the anonymized proposal, but also a not anonymized expertise and resources document. And the first step is the mayor of the proposal is assessed and separately the qualifications of the team are assessed. So the qualifications track record and access to unique facilities is indeed part of the evaluation. It's just separated out. Feedback from Hubble panelists is that proposal discussions were characterized as more collegial and efficient. Focus was squarely on the science rather than the scientists. There was a noticeable shift in the depth of discussions as well. It was clear that the reviewers had read the proposals very diligently and that without the distraction of names or institutions, there was no recourse but to focus on the proposed science. Discussions at both the panel level and the attack, that's the time allocation committee level, focused predominantly on whether the science was novel, impactful and feasible with HST and not on whether the proposers had the expertise to carry out the proposals. Several TAC members noted that they felt the discussions at both the panel and the TAC level seemed more collegial and less emotionally charged than previous TACs perhaps because either positive negative feelings about the people involved with the proposal were largely removed. So how do you make your proposal compliant? We're giving you lots of information but first of all step one is not anonymized. It's submitted per normal requirements and step two is anonymized per the guidance that follows. So first the inspires program page. Each element will have a document entitled guidelines for anonymous proposals and it has a lot of detailed information about submitting the anonymous proposals and a quick start tutorial and FAQs are at the website listed here which is the same system. So first of all you need to exclude all names and affiliations of the proposing team including anything in figures or references to personal websites. Do not claim ownership of past work. So instead of saying my previously funded work or our analysis shown in whatever you would say previous work shows or previous analyses in this reference you would just have it be you would have no ownership of it and references are cited in the passive third term. So you can see the yellow this prior analysis brackets one indicates that and that just kind of draws attention away from you know who's frequently cited. You can still say we when you talk about you know the work you're going to do you can say we propose to do something or we will do something like that you just can't say it in terms of ownership of past things and you'll include the separate not anonymized expertise in research document. So how do you publish how do you reference unpublished work for proprietary results? You might need to do this occasionally or maybe non-public software things that haven't been presented in public but they're not citable that could imply the investigators but we want you to kind of again refer to it without attribution or ownership. So you can use language such as obtained in private communication or from private consultation which people can do anyway but you will do that in reference to yourself as well. And just again recall the goal of dual anonymous is to shift the tenor of the discussion not to make it impossible to guess the team members. Regarding institutional access to unique resources another common issue is that you know you might have unique access to certain facilities to do the work and you can still talk about this in the STM the scientific and technical management section of the proposal you just should do in a way that does not identify people. So you can say the team has access to this ultra high resolution IRMS which will enable the required measurements. And in this kind of situation we would recommend you can support that in your expertise and resources document. And this is an example of anonymization and I'm actually going to just go to the next slide where we highlight so the yellow goes to yellow so instead of saying in Rogers 2014 we concluded say prior work concluded if our model from this is correct you say if the model from this is correct and we will compare with our first epoch it's a first epoch was obtained in 2007. Those are examples of how to translate that text. And a really big thing is that many people obviously resubmit proposals. This is where it gets a little trickier. We really ask you to really check things when you're resubmitting a proposal or reusing text from something else. Just double check that you're not claiming ownership of password. Do not include the names of personnel associated with the proposal or their organizational affiliations. Write the references in the format as indicated use third person neutral wording. And then again you can go to this website for information. But how is the capability of the team to execute the investigation account? I know there's kind of a concern that somebody is going to write a proposal but they don't actually know how to do that and it's NASA can be funding people that are not qualified but we're going to make sure that they are evaluated for that. So we have the expertise and resources document so it's a second document you upload and it is distributed to panelists for a subset of proposals. Typically the top third it depends on how the program decides to do that. But it's more than we would presumably slightly more than we would potentially select. And it will have the following elements. So you have the list of the team members and their organization and their roles. A brief description of their expertise and what they're bringing to that. A discussion of the contributions of each team member will make to the specific investigation. A discussion of the specific resources required to perform the investigation. So access to laboratory equipment, specific samples or sites, etc. A summary of work effort to include the to include an anonymized table of work effort. So you'll have you'll have a anonymized and not anonymized portion for that and that's outlined more in the guidelines document. Biographical sketches as required. Statements of current and pending support as required. This is all in the expertise and resources document. And any letters of resource support. So if there's some facility that you need to have access to or letters to the PDS things like that that all goes in the expertise and resources document. And how will your proposal be reviewed? So we have the two parts. We have the science review and the expertise assessment. The anonymized scientific review will take place and everything is finalized. The evaluations are written. They're signed off. They are done before the expertise and resource document reveal occurs. And then once that occurs it'll occur for that subset of proposals and panelists will assess the team and resource capability to execute the proposed investigation. And this is how what the panelists are instructed to do. They're instructed to consider proposals solely based on the merit of what's been proposed for the first part. And they're instructed to not spend any time trying to identify the PRA of the team. And even if they have an idea they're supposed to discuss the science and not the people. And there are NASA appointed levelers present to ensure that the integrity of this process that are going to be present in each panel. And they're also instructed to make sure that their language is important discussing the proposal. So they should use you know neutral pronouns such as what they propose the team has evaluated data etc. And the NASA appointed levelers which in some cases are program officers but it's always somebody that's you know job is to be listening to the conversation. They're going to be there to support all of this. They are there to ensure that everyone's focusing on scientific merit. They're not necessarily listening for issues related to science just more that the they're focused on the discussion itself. And if the discussion ever becomes inappropriate and trying to guess anything about the proposers the leveler can refocus the discussion and they have the authority to just stop the discussion of the proposal. Then the discussion of the resources of the expertise and resources document occurs. So again the scientific evaluation is completed. And then these documents are distributed to the panelists for the subset and they assess the team capability using the scale. So proposers are either uniquely qualified qualified or not qualified to do the proposed work. We set the expectation that the vast majority of proposals are going to be qualified. And there actually is a requirement that if the if the reviewers find that the team is uniquely qualified or not qualified they actually have to write something about it and justify that reason because we do expect the vast majority of teams to just be qualified. And uniquely qualified does have a very both of them have a very high bar. So for example to be uniquely qualified you have to be exceptionally capable to do the proposed work and also have singular access to resources to complete the work. And being not qualified means you have severe deficiencies in the necessary expertise and resources to execute execute the proposed investigation. And when this is distributed we instruct panelists to base their assessment based on what's written in the text and not say things such as oh so and so is clearly qualified. So we understand that DAPR is a major shift in evaluating proposals and preparing them and there might be some slips in writing these proposals. We do reserve the right to return without review if there are particularly egregious violations and non-compliance in terms of identifying the proposing team. So we also know that some work is so specialized that even if you're you're trying to anonymize the proposal that identify identification of the PI and the team members can be discernible. But as long as you follow these guidelines we're not going to return your proposal without review. That's not our goal at all. So a quick review of DAPR experiences to date. We've only had three egregious violations on the anonymization guidelines which resulted in proposals being returned without review and that's out of hundreds. Here are some common pitfalls that we see maybe 10 to 15 percent of the proposals claiming ownership of password. So our previous analysis PI has established record including metadata. This is kind of a tricky one. That might reveal the name of the PI. So bookmarks for example might have some PDF bookmarks that might have the name of the PI. When people recycle proposals prepare piratidual anonymous peer review and they kind of skip a sentence or something and they don't anonymize everything. Providing the names of the investigators and the contents page we've had I would say we've had some like wonderfully beautifully anonymized proposals but then they just accidentally left in a title page or contents that had their name like right at the front. So trying to that's something to be aware of and double check. Providing the origin of travel for professional conferences that's something to leave out in the anonymized portion of your proposal. You can just say that you're traveling. You don't need to say that in the anonymized portion you don't need to stay your starting location for example. Mentioning the institution name and the budget narrative. So this is something to double check with your budget people. I know that a lot of times the people who prepare the budgets are working you know they might not always be aware of the latest guidelines here and their default template has the institution so that's something to check. And including PI or CoI name on budget tables it's another thing where we'll see like a perfectly anonymized proposal and then the budget table just like names the PI because again it was prepared you know a different person prepared that. And then from panelists the reviewer surveys thus far have indicated that the discussions were more focused on science content than on non-dapper reviews and there was overwhelming support to continue Dapper and Future reviews and that's those are results from NASA SMD so far. So I'm going to answer some submitted questions. These were submitted as of midday yesterday. Why did the planetary science division not go with open peer review? So the goal of Dapper is to reduce implicit bias which revolves removing the identity of the proposer from consideration when evaluating the intrinsic of a proposal. So open peer review might address other concerns for goals. I know it's been credited with more you know collegial interactions and such but it's not directly designed to address issues of implicit bias. And we're basing this off of HST's process which suggested that removing the identification of the proposer that inferred female PI's were selected at a rate more on par with their submission rates than in previous years. So this process has been shown to address some issues of implicit bias. How does the dual anonymous process work with continuation proposals? So this is some language from the guidebook for proposers which you know the kind of key point as at the end it says therefore NASA will consider all proposals received in response to a notice of opportunity for funding notice of funding opportunity as new proposals and will review them impartially. So proposals submitted to under a Dapper program are like other programs reviewed as new proposals and reviews as such. So they should not refer to previous funded work in the anonymized portion of proposal. You can refer to that work in your E&R document. How do we refer to relevant previous work without outing ourselves? So I think I just addressed most of this in the presentation but you but to just emphasize you are free to refer to the work itself you just cannot claim ownership of it and you should discuss it without attribution. For unpublished work we recommend saying previous work instead of our previous work or obtained in private communications. And then there are a couple of questions. So how do we describe the use of analytical labs to be sure the reviewers don't find out where the work would be done? And the second question I will confess I did not completely I was not completely sure of the intent of it but it was that dual anonymous makes sense when everyone was proposing the same use use the same equipment. How can it work in an open proposal call? If if I'm not addressing that here please feel free to use the chat because I just wasn't quite sure what the nugget was of that but I believe the the questions are about how do you how do you talk about specialized equipment? And you can talk about these aspects you just can't discuss them with attribution towards your group. You can refer to having access to specialized equipment facilities and if that's required the panel is going to flag that and then when it comes time to go to the for the ENR review the expertise and research review then they can check for that. So you know the panel might say wow this is really you know they're going to have access to this like not many people have access to that and that's not going to be a weakness or a strength they're just going to flag it and then in the ENR it'll be apparent if the group actually does have access and so they're they'll perform that full analysis using the three-point scale and again just remember that the goal is not to make it impossible to guess the identities of the investigators but to shift the focus away from the individuals and toward the science. And then there were kind of a group of questions that kind of addressed the same thing but the idea was that a scientist track record is an excellent indicator of future research but shouldn't this be part of the evaluation. So the proposers should be able to make their case through the description of their proposed work that they have the necessary skills to achieve research success and the track records are addressed in the ENR document and they're going to be voted on and we're not trying to make it impossible to guess who it is we're trying to shift the focus of the conversation and this is a follow-up is how do we evaluate proposals without knowing their track history and there were several questions all related to this and there were in the past there were programs that funded purely on track record but it's not something we're currently doing and we evaluate the proposals new every time and I would also like to note that to date no highly rated proposals evaluated under DAPR have been submitted by a team where they were not deemed at minimum qualified and this is out of over 500 proposals submitted to NASA absentee how do we handle proprietary restricted access resources the example given in the guideline was not convincing so we just asked to use the guideline given we'll flag it and we'll verify it later and I keep saying the same thing you know we're just trying to ship the conversation of toward science not people and if you follow the DAPR guidelines we're not going to return it without review and if if we are seeking to fund or use a specific observatory where the set of PIs are known and sorry I cannot completely see this can we no longer reference that observatory you can you can still reference it in the STM you just can't write it in a way that you're identifying the team ever so you can say the team has access to telescope time on the WNK observatory which will enable follow-up of the galaxies in the sample and in this situation we encourage you to follow up and say how you have access in the expertise and resources document and yes that's amazing so some final remarks here oh actually this is a good place to bring up any other questions I want to list all the program officers here you can ask any of them for questions about this and if it's very DAPR specific you can always reach out to me but I encourage you to reach out to your program lead as well this is Henry Throop I'm going to read a couple of the questions Deli that have come up in the on the web page they've been voted up the first one here is for technology development proposals often build on previous work and with a small field proposal teams are identifiable how does that work so are any of these I did not think any of these were technology development proposed programs I mean I know they're data analysis programs we so that is an interesting point we don't have any technology development explicitly in any of these programs so I can I definitely see that point but we're we haven't also extended DAPR to those programs at this point in time so that will be something you know we'll need to decide if that's the right match for that thank you next question is can you address how the institutional and management sections of the proposals should be adapted to the two DAPR that was this last slide that I thought was missing and I think I just clicked over it for it went really quickly because I actually remember spending time on this question and then I was all of a sudden at the final remarks page okay I have a slide for that so this is a couple of things a lot of this is from the guidelines but I'm just breaking it out so the STM portion itself is written anonymized the summary of work effort is included in an on my fashion you can say Hawaii one quite to in the main document and then in an anonymous non-anonymized fashion in the E&R document so you'll have the table twice once anonymized another time non-anonymized and then as usual roses proposals include a redacted budget so so the redacted budget that is provided to the panel doesn't have salary fringe or over doesn't have salary fringe overhead but it doesn't have you know names of people etc but it should you should have a very generative that discusses financial support for the CoIs you just don't you can have it anonymized so there's it's the same thing where you've always had a redacted budget it's it's just anonymized and then in the unredacted budget you can use names and all of that and the facilities and equipment section is not included in the main proposal document it's a shortened version of this information including letters of resource support are in the expertise and resources not anonymized document so I think that addresses most of the components of that question thank you next question is similar to the question addressed about labs and access how do we anonymize a proposal that uses a code that is not open access and I can imagine for instance a proprietary code might be linked to one institution or might even have the institution's name in that code how do we address that you can um you can say that you have access to it I mean you can consider while an institution might have access like um you somebody can share code right like you could have a collaborator who share who shares that code who's not that institution for example so the point is that you just you can refer to it you just can't say you can't say that you can just say that you have access to it it's the same basically the same as facilities you just say that you have access to it um and and again I mean yes it does that possibly give a clue but that's you know it's within the rules and um you know the point is you're saying you don't say that you created it but you can say that you have access to it um on the review does Dapper in any way in any way affect how conflicts of interest are identified and managed um so to some extent um I need to double check so we we are having to the onus is more on the program officers to manage these um but we do we have changed some of our internal processes um and I think that's all I can say about that moment um will the proposal be sanctioned in any way or penalized in any way if proposal accidentally reveals too much information about the key members and works they have previously done um so the first year we're trying to be more forgiving we understand that slips may occur and we've been able to draw a clear line so far of agree just versus not it's been pretty straightforward um where you know it it was clear that somebody didn't do it at all anonymous a proposal at all versus like oh they had like two slips or something like that um so if we in general find that there's been a good faith effort um we you you might get a warning this year um or but you could still it could still be reviewed um but if it's if it's free just it will be pulled but that's has not happened extensively this year we had a couple questions about formatting references uh has NASA provided any templates for formatting references in the correct way with numbers rather than the pi's names no um so there's in the guidelines document basically it's numbered brackets and then you number it so it'll just be like a numbered list and you refer to it in the numbers in the proposal so and presumably there are standard latex templates out there uh or other templates that one could use to format the references that way um question about the review panel for the flow of the review panel does the evaluation of expertise extend the time that the panel will meet for does it take uh longer to discuss this than it does normally um it's pretty brief actually um it's um I would it's it's not a substantial it adds minimal time to the panel I would say like maybe yeah this is a very rough you know it's going to depend on each group but maybe an hour is that and um but there are other parts of it that are you know you're actually spending less time discussing some of those things during the actual discussion so it kind of evens out in some ways we have a question about external reviewers uh will external reviewers uh also see only the science portion of the proposal over those where they see the uh the non-anonymized section as well they will just see the anonymized version um let's see for facilities uh what if you have uh sort of private 100 ownership of the facility and identifying that facility would definitely identify you I mean it would so in general I mean you could it's you would still you would just say that you have access to it I mean you don't need to say I mean you would just say we have access to it um you don't need to say we have access to this and only the people that own the lab have access to it so you would just refer to having access to it and and if somebody happens to know that it's you know something that only those people have access to then that's what it is um but the point is to just say you have access to it and one more here uh may have PI volunteer to be identified as author of the proposal I don't understand what so I think that the question refers to whether someone may opt out of the dual anonymous system and identify themselves by name in the proposal I mean from what from what I'm hearing that would be outside the guidelines so I'm not quite sure if I'm understanding the question I think that's what the question is yes okay okay um and just a note too on the um the very specialized equipment issue and I know this comes up a lot um is that you know we don't know there could be this could be a lab of five people and there could be postdocs or people who graduated from them and we don't know who is who and that's that is still you know anonymized and um anecdotal information is that um a lot of times people guess and they're wrong and they think they know who it is and they don't always so um again and the point isn't to try and guess and see if you're right or wrong and the point is to to not be talking about it as much uh let me pass it over to Doris now okay she has questions press on uh yes most of the questions that were asked are already have already been responded to but there's a question here about uh can we have any anonymized description of expertise in science uh uh in stm e.g the team has decades of experience in counting craters along with recently developed experience in numerical model that's something that just um by default belongs in the expertise and research document um and it's um yeah it just that's where it belongs it's not that it's um that's where it goes like by structure that it would go in the enr section there's another question about the conflict of interest which you've addressed and also I want to let you guys know that then Evans also uh addressed that question so if the answers are not uh uh full for you uh feel free to uh pose it again there's a question about uh the program officers when we are preparing selection document will we be doing this blind as well uh uh with anonymized proposals so we know the identities and that helps us manage the process uh there was a lot of conversation and I think that already has been addressed about what do we do if we have access to facilities or past past papers publications so I think unless I see something new that's about it for for the questions on the chat let me have one question here okay I have one question here is there any change to the page limits and what can you say about the page limits of the expertise document um there is no page limit to the main part of the document um you I mean you kind of end up having a little bonus space because you're excluding some of the stuff that is normally in that portion um and the enr document does I haven't looked at the call in a long time because we're like way off cycle but it does it should not have any page limits um and also Dan Evans is uh who I believe joined us during this he is the the lead for SMD doing the dual anonymous peer review and so thanks to Dan for pioneering this and if you had anything to say you're um it would be you're welcome to chime in and say hello to everyone no you're handling it beautifully already deli and that needs me to chime in and there is no no limit to the enr document but typically speaking um a few pages um maybe up to five pages is about right including you know biosketches and stuff like that but for the written material um you know lists of expertise and so forth that's typically about a page or two we also had a question about the slides for this town hall um would those be posted and available yes they will be on the dual anonymous page um and actually if somebody is able to post post that page in the comments that would be great so that people know where to find that it's been referenced numerous times but obviously if you don't have the slides you can't access the page so do it all right deli if you have time I have a couple more questions here okay great all right so a question about the citations if the citations are fully listed at the end then what is the point of changing the in-text to number so so the idea is to just draw attention away from that um and it's kind of a compromise right because because if we say if we had put the references you know if the references were in the redacted portion people are gonna be like oh they're citing these crazy references that are completely invalid so we might have a way for people to properly cite things but it's not upfront in center um and we you know we do rely on the honor system of their viewers to not make it you know a like a hunt to figure out you know who cited a lot but um another question uh asking if there are any statistics on the proportion of proposals with excellent to very good science ratings but very low qualifications rating so we've only done um the reveal for the top rated proposal so like if if something was rated fair or poor we weren't doing the reveal um and then of those that um were made it to the expertise and resource reveal um none in SMD have been deemed not qualified and so it's just those three categories not qualified qualified and I don't know how many have been deemed uniquely qualified um but my understanding is the vast majority have been just qualified all right uh I have one more question uh for oh for a follow-on proposal do you still need a section on accomplishments from the previous award how can that be anonymized or should this be left out so um I think that was hinted at in the continuation question um so that should be in the expertise and resources and whether or not it's required at all you can you can talk to the specific program lead for that so I would talk to the specific that person to see if that needs to be included at all but if you are including it it would be in the expertise and resource section and it would you see it's considered part of your expertise and there's a question about um let's see would the review process differentiate between the new proposals or and the follow-up proposals yeah so you don't refer to your previous work it's it should just be a standalone so you're not making any reference to previously funded work including you know your your predecessor proposal I think that's answering that right okay one more I have I think this is the last one I have here and it's quoting the questions on the Q&A page as an important sub-questions on referring to previous work how do we talk about preliminary work for this proposal could we say we have done the analysis shown in figure one since this is still anonymous but demonstrates its capabilities which I haven't heard discussed basically yeah if it's if it's not published anywhere then yeah I mean if you obviously say if you published it in LPSC recently then that's kind of your your owning you're making that connection but if it hasn't been published anywhere and that can be attributed to you I see no reason why you wouldn't be able to refer to to your preliminary results thank you Delia I think that's all I see in the chat if I missed your questions please type it again I do any of the program officers have anything they want to chime in and they might just have to say a lot of people are putting thank you notes I hope you'll get a chance to read them after this well thank you everyone for coming here and there are a lot of resources available and feel free to reach out about any questions I know that this is a big change but I we're excited to work with everybody on this and and I'll say thanks to my colleagues for being here and supporting this event and thanks to the community so have a great day everyone and I don't know Shoshana if we stopped recording but