 you know, pre-proposal telecoms for the world's 2021 price-rate sciences call, because we're doing, using, you know, for earth sciences at least, a pretty new concept with a dual anonymous peer review. And I think personally, there's no downside to it. I think it's moving the right direction. But I also understand there's some apprehension in the community for doing it, you know, going that way. You know, I've got, you know, several emails, why are we doing this? Emails, or some specific questions, etc. So, you know, that's why we're having these two pre-proposal telecoms to explain a little more. We're coming from how it's implemented and then, you know, answer, hopefully, any questions you have. For us, it's, you know, it's new as well. So, you can pick what's the next slide. I'm going to do this. You know, like with other NASA town halls, you can anonymously post questions on a website, which is the latest given here at the top. So, you can, you know, support a question, etc. or upload an existing question. It's very user-friendly and you would see how this is working. There's also a Q&A window on websites that you can use if you have other questions. I cannot see the chat or the Q&A window, but Caitlyn will be monitoring it and, you know, it suggests we just, you know, go through and then we'll answer maybe questions. At the end, if there's a question, Caitlyn, where you feel you should answer that right now, just interrupt me anytime and I will, what? Sure. Stop my presentation and just, you know, we can talk about it. So, I think you're all automatically muted. So, and, you know, because it's, you know, anonymous, etc., you know, proposals, but so you use the chat, use the question answer or use the URL that's on the screen right now to post any questions you want to have. So, as I said many times when I, you know, started this job as a program manager for CRIO, I wanted to have, you know, regular calls and for the CRIO third program, science program. And, you know, one year I have a very open call basically, you know, bring your ideas or bring the panel, you know, I should say. And then we go from there. And then I have a second call, a smaller call, which is this call in August 2021, where I'm trying to address a very specific topic in this case, it is, you know, the stability of the ice salves, of green and an octetar, which is smaller call. And the idea is that scientists, the eyes selected through this call will actually, you know, work together to achieve something, you know, that is tangible, something where we feel we may really progress in understanding in this case the stability of ice salves because you talk to 10 scientists who are doing ice salves science, and you get 10 answers what the most important process is. And so all these processes are, you know, competing, you know, or have positive feedback on each other. So that's the idea of this call in general. And I don't want to talk much about this, the call specifically, but, you know, we can talk about it in any specific questions to the con. This three proposal, tell the con is more really for, you know, to understand the double anonymous peer review, where as I said in the beginning, you know, there's a certain nervousness and a sense in the community for going that long. So we will talk a little bit of, you know, which programs are converted to do an anonymous peer review. We will talk about the motivation, what it means specifically, and then, you know, give you some tips and tricks how to make a proposal compliant, which I think personally is not as complicated as it may sound in the beginning. So these programs are converted to the app. And, you know, we are behind. This has been establishing an astrophysics for years and very successful. And you can see in astrophysics, we are 10 plus programs who are using, you know, the dual anonymous peer review to evaluate their comps, they have their proposals. Similarly for planetary, they are five this year. So Earth Sciences, we have one call last year, the USPI program, led by Richard Eggman, and I participated in that panel. But here's just the second panel in the Earth Sciences for a second call in Earth Sciences who's doing double anonymous peer review. Other divisions are much, you know, much ahead of us. So what's the motivation behind it? You know, the goal really is to do anonymous peer review, is to level the playing field for everyone. You know, we want to create a change in the ten of the discussions away from the individuals on the proposing team and move towards that purely talk about science. And it's not a dual, you know, it's not a civil bullet, it's not about all the problems, but it's a step towards it. And I have to say, like, in these panelists, I, you know, ran participated co-led in the last, you know, almost two years from now. I don't know if it was 28 or 10 or so. There's always this problem. People, you know, there's not always this problem. People just cannot really say, you know, they cannot really separate person from proposal. And sometimes, you know, panelists are struggling with it. And basically, also, every panel I, you know, I led, I had to step up at one point or speak up at one point when panel members said, well, you know, this is not very well articulated in this proposal, but I know the proposer, and, you know, he or she will certainly deliver. And, you know, then I always have to step in and say, hey, you need to evaluate the proposal because otherwise, in the military or scientists, you know, wouldn't have a chance that their proposals needed to be so much better, so much more perfect because they wouldn't get away with, you know, their reputation, their proposal. So it's really, you know, leveling the field, focusing on the science and the proposal. So at panels, according to bias, BDO, to make people aware of bias, he said we all have, but it's difficult to completely interrupt biases through training alone. So, you know, the staffer process, though, does not remove the need for structural changes in order to improve diversity, equity, and inclusion. But I think it's a useful step towards that goal. You know, it's nothing new. You know, there's this article from Business Insider, you know, in 1970, the top five orchestras in the U.S. and fewer than 5% women. And then auditions were made blind and that number increased dramatically. And that with the most famous orchestra in Germany, they did the same thing. They changed, it was all men, they changed to blind auditions and, you know, the ratios changed tremendously and so far. And, you know, the reason why astrophysics is so much ahead of us in a way is because they saw when they looked at proposals for the Hubble Telescope Program that the success rate of female scientists was throughout the years lower than the success rate of their male colleagues. And they went to double anonymous peer review and that number changed. It got more even. I mean, immediately the numbers changed. So, you know, we all try to be as unbiased as we can, but, you know, there is a bias that is very hard to overcome. And similarly here, this is a success rate of proposals organized by institutions and you see, you know, government contracts NASA centers, federal government are one universities have very high success rates, but then once you go to minority-serving institutions, you know, or even like two universities, you know, the success rate you know, the success rate become much smaller and, you know, the question is how much is the cognitive bias part of, you know, this, you know, inequity in success rate. So, this is not my slide, but this is like, you know, showing this slide because this is something that is coming from, you know, Thomas Zubu from the science mission directorate's level. Thomas talks about this all the time. He's very committed to it and dual anonymous peer review is there to stay and I think it's just going to be an experiment. And the reason is because NASA science mission directorate strongly committed to ensuring that a leap of proposals is performed in an equitable and fair manner and to this end, motivated by the successful study conducted for the Hubble Space Telescope, as I said, S&D is adopting S&D is adopting the dual anonymous peer review for numerous program programs. So, in the system, not only are the proposals unaware of the identity of the members of the review panel, but also the reviewers don't have explicit knowledge of the identities of the opposing team during the scientific evaluation of the proposal. So, what is this process exactly? So, this is the first sentence assessed, a repetition of the previous sentence. The panel does not the identity of the proposal. And the primary intent of the dual anonymous peer review is to eliminate the team as a topic during the scientific evaluation of the proposal. And you talk about the science. And we talk about it later. It's not always completely possible to not have an inkling who the proposal may be, but it's still, and that's what experience has shown is the focus is way more on the science than on the proposal, you know, compared to the dual anonymous peer review process. So, it's not the process itself we talk about this detail as well. You know, it's not completely by proposal submit proposal, which is pretty much the end of proposal, which is anonymized, and we talk about this. And then there's not an anonymized part, expertise resource document that talks about team qualifications, teams, the teams prior experience, etc. The list shows a list there, and but the merit of the proposal is assessed anonymously and will be determined separately from the non-anonymized qualifications of the team. But nevertheless, the qualifications, the track record and access to unique facility will form part of the evaluation. So, how do we make our how do you make your proposals, I should say, you know, compliant? How do you make it? How do you anonymize your proposals? So, I'll give you an example in a minute. You know, so basically you exclude names and affiliations of the proposing team, including figures and references, you know, to personal websites. You do not claim past ownership saying my previously or our analysis shown etc. You just, you know, make it more person and say prior analysis has shown. But you can do say we propose because you are the team and you do propose something. It may occasionally be important, that's kind of important, to cite exclusive access to data sets non-public software, unpublished data or findings that have not yet presented to public before, but are but are not, you know, but are not sightable. Each of these may reveal a strong implies investing data sets of proposal and these instances propose of, you know, must use language such as obtained in private communication or from private communication when restoring to such potentially identifying board. And, you know, we call that the goal of the dapper is to shift the tenor of the discussion not to make it absolutely possible to get the team members. Another thing is institutional access to unique resources. So another common situation that cures in proposals is when the team member has institutional access to the facilities, you know, for example, an observatory or a laboratory are required to accomplish the proposal. And an anonymized proposal does not prohibit stating this fact in the scientific, technical management section of the proposal. However, the proposal must be written in a way that does not identify the team member. And here's an example. The team has access to telescope time of the WM Tech Observatory which Bill and Adler have said along. Note that in the situation NASA recommended to team provides detailed reporting information if needed to validate the claim in the expertise and resources document that is not anonymized. So we can expand on that access to the data said or to the lab or whatever. So here's an example and I think this is quite a good example because I feel it's easier than a, you know, initial, it may initially sound to make a proposal anonymized proposal and although I realize that devil is in the detail so this example let me go to the next page actually just better. You know, for example in the sentence starts in we conclude it in the proposal in this case we just say prior work concluded and the citation, there are still references in the proposal but instead of we just it out we have like other journals we just have a number there for the reference now people can still look it up and see who is the first author who wrote this paper but I think the idea is here that you know, the text is not full of Rogers et al 2014, Rogers et al 2019, Rogers et al 2018, Rogers et al 2020, it just says 12, 14, 16, 29 or something like this so I think it's more perception than an elimination of your own references and then, you know, as the queen highlighted text shows you know instead of writing if our model you know is correct, you write if the model is correct things like this and I think personally I was thinking about it more it seems you know if you just write your proposal as you normally would write your proposal and then go through the text that you know makes it anonymous shouldn't be hopefully I'm saying this so easily because I'm not going to write a proposal obviously but hopefully it's not you know as you know laborsome as you make here so how is the capability of the team to execute the investigation accounted for so in the expertise and resources document that is not anonymized it's a standard stuff in the way that you know is normally in your proposal list of team members description of expertise specific contributions from members you know you all write this in the management set management sets you know whatever you specifically call it specialized resources some of your approval effort you have by a standard paragraph sketches in the standard current employment support and then letters of resource support if applicable so much of these months of this information you know that you are used to providing you provide just like that just like you did in the past but just as you know as a separate document so there is in the inspires call there is some detailed guidance it's like a 5-6 page document guidelines for anonymous proposals and I think it's actually pretty well read it's pretty clear what you know what you should be doing you know what you know there are some examples there so there may be specific questions in the process as you write this as you write your proposal in that and if you if you still run if you still run into problems you know don't hesitate to send me and Kate any time you try our best to answer it and you know we may also prefer ask our experts here we have done this many times to solve a specific problem and then there is also a quick step tutorial as well as frequently asked questions in the link and maybe Kate you can just put it in the chat so people just click on it rather than having to write it down for the presentation so I will the proposal be it's a two part process now we will use the anonymized scientific proposal and have a normal panel and you know all the assessments of your proposal we waited just as you know between one poor or an excellent five and once all this is done as a panel summary so then we go look at the documents so we will give probably the top of the proposals the proposals that have a realistic chance of being funded we will provide the expertise and resources to the panel so that the panel then can evaluate the team and the resource capabilities to execute the proposal investigation the proposal itself will be evaluated just as any proposal in the past will be evaluated assessed based upon scientific merit to the solicitation program as I say we will talk about proposals it's really the solicitation that is well it needs to be relevant to and then cost we will have a budget narrative which is anonymized still in your your core part of the proposal with regard to expertise and qualification the the panel will look at the second part of the non-anonymized part of the proposal and will then evaluate whether the team is uniquely qualified qualified or non-qualified to carry out the work and that seems to make real sense and what I heard is that it's basically impossible to write if you're not qualified to write the great proposal some people may be really good at it but I think it hasn't happened yet and so most people especially those in the top in the top third are all qualified to do the work and some may be you know some may be uniquely qualified so success metrics so far that's taken to look into our last calls the two ISA2 calls the cryo call and to look at the success rate there I mean it's actually doing pretty good in cryo for the studies with ISA2 the oldest 2019 one the success rate for male PIs was 20% the success rate of female PIs was 32% for example the success rate for male PIs was 24% 24.6% and 17% for females and then for the reasons second ISA2 call the success rate for male PIs was 35% for females was 57% so our numbers are very good but I also want to say you know what we track is right now female male we started to track early career which is hard to have for we do not track race it's something that has come up before quite a bit and it's something that with the anonymous period we will address as well and race is harder to track because it's more guessing by the name of the PI so while cryo is doing in terms of gender biases this is the astrophysics data analysis the astrophysics data analysis program results so in 2018 there are many programs to solve it's like 10 programs it's not only one panel it's many panels before in 2018 the number of female PIs was 26% so there's a number of selected or it's the number of female PIs that are in the top 2 who are in the top 3 category where it's just 15% or 20% it's way below their submission ratio when they implied the dual anonymous period in 2020 when they used this dual anonymous period process in 2020 it I think it's very impressive even though the number of proposals with female PIs was 30% and the top the number of female PIs in the top rate of proposals was 30% as well so it's kind of surprisingly clear an adjustment now as people are using the double anonymous period view also they look at the view as what do the panel members think does it work does it not work and I think it's roughly a thousand numbers or something or maybe it was a thousand proposals I forget I have to be careful not what I'm saying but it's because other disciplines have been using it for quite a while those are statistically it's not small number statistics and you can see that most people the vast majority of people believe that the dual anonymous period procedure improve the overall quality of the period people really thought you know the focus of the discussions is so much more on the science and on the processing most people you know strongly agree with the majority of proposals on my panel that prepared in accordance with the guidelines so it doesn't seem it's a writing proposals that are anonymous you know people did it very successful that's pretty much what it says so out of a thousand proposals I think that's pretty number five only three proposals had really bad violations of the anonymization guidelines and those proposals were trying to work with you you know reviewed and yeah so some common minor proposals that they saw is there are such good sentences in there where the P.I. as a proposal changed the ownership of passports the P.I. has established a method there are some PDF I'm not really sure how this would work but everything is in one PDF for our proposals that would reveal the name of the P.I. some people recycle old proposal text and then I'm not being careful of anonymizing it and sometimes names still appear here and there maybe in the budget narrative maybe in some budget table or you know as the P.I. says you provide the origin of the travel for your travel budget so which also reveals your identity yeah that's too quick we're not we're not to understand that dual anonymous peer who presents a major shift in the evaluation of proposals and as such occasional slips inviting anonymous proposals and now the results are right to return proposals you know you know anonymization is really ignored but you know as I said it happens very very you know and then NASA further acknowledges acknowledges that some proposed work may be so specialized despite attempts to anonymize proposal the identities as a principal investment entity members are really discernible but I think it's very true for I can easily see this happening in a community as a small as a cryo community where people have those specialized expertise etc but as long as the guidelines are followed by the proposals and it's all I think it's all good and I think because we are a small community you know you know going this way will actually help because we know each other and pretty much everyone knows everyone after a while and in this you know doing this anonymous peer review we as I said we will focus the discussion on the sounds because many people and whether we want to have an opinion of a fellow scientist whether he or she does great work or good work you know not too great work and because we know each other and we do have opinions on people's you know people's work and as I said we'll help a lot also I think early career scientists do not have the name recognition with them when they write their proposals they don't get away with I should say you know sloppy proposals that you know some established scientists may get away with so I think personally I think there's no harm at all in doing this and you know if we cry or bearhand this you know making you know proposals more equitable I think it's a good thing to do and we are considered a progressive cryo progressive discipline that is you know sometimes you know ahead of the game you know I think it's a good thing I think it's a win-win for everyone you know we're good with gender it looks like at least from the statistics but we have also could have also biases with regard to institution biases with regard to race stages etc so that's really all I had in the slides so we can have half an hour so if you need to talk about questions concerns, opinions you have to hear your opinions you know I'm serving you I just want to do the right thing let's see my biggest you know community as a whole yeah Caitlin maybe I'll stop with my presentation if we get out of it yeah great thanks Torsen so there were a handful of kind of different questions both in the QA in the Webex and then in the online town hall tracker about proposals being precluded from review if minor mistakes are made and I think the answer to that is minor mistakes are going to happen especially because this is the first time that the cryospheric sciences community has done this so actually maybe a question that you could answer proposals won't be returned for minor violations right those will just go through and will we have the opportunity to kind of scrub those before they go to a panel for review and you are muted that was all the time yeah don't sweat it if you put in reasonable effort to minimize the proposal it's all good I mean I realize it because you know lots of people in our community have specialized experience yeah so I think I wouldn't worry too much about it the top rated question on the town hall is are conflicts of interest still considered when you review a signed review so that's a good one that's a good one so technically there shouldn't be conflicts of interest because you wouldn't know who the proposals are sometimes you know the figures, the layout, the fund so well that you know that a panel member knows how does that person and I would talk to the panel members after we've done we clearly know who that proposal is coming from you know let us let us know what we do in reassignment and yeah that's what we're planning on doing that's a tap on it and there will be those cases because I've seen proposals where I immediately know who the proposal is so we will do the reassignments and of course knows alright the next one in our proposals how do we refer to COIs and also unfunded collaborators by number rather than name I see what you're saying so do you want to call it COIA, COIB, COIC I think yeah that's what the PDF document that SMB put out kind of outlines for folks and then in that non-anonymized document that's where the summary of work kind of impending funding that's where all of those things are kind of broken down anyway so the the only part that's really anonymized is the discussion of what science is going to be performed so yeah you can do like that all you just use a T but if you go into specific roles totally so I think this is a good question assume project managers still see the names of the proposals how does this approach help remove bias at all levels including at program management not just at the viewer panel level so that's a very fair point you know I'm in my approach as a program manager I'm very very much you know following the evaluation of the panel the ranking of the panel I'm pretty much going down top to bottom maybe you know there's a small gray zone where you need you know to be a brilliant actor versus an actor or something like this I should say maybe for this one but overall I you know follow the panel so that she's going strong but also once the panelists done with their ranking and evaluations Caitlin and I we have to present to a steering committee and where we have to justify our selections and if it's you know straight down you know start at the top and go down until you run out of money you know then it's really easy if you start to skip you've got to have a really good reason why you know programmatic reason you know to to skip a proposal it's a good point you know I think I handful these questions were answered in that presentation this is an interesting one how will international collaborators writing that supporting whether anonymously or anonymously be handled so putting a lot of that could be an anonymized part so that could be just normal just you know the international collaborator can cite this person you know a provider or whatever and you can also anonymously say you know if it's part of the proposal that something will be provided by say international institution space agency whatever then you can say that and you can justify it or you know later or not justify it but you know you know prove it later if you will with you know anonymized that our support from international collaborator does this make sense though I was just traveling I think it's okay so yeah it would be part of that not enough expertise and research portion of the proposal um actually yes someone just submitted this question um so this since this is essentially a DEI action how will we ensure that the selected TI is included after they are funded so that's um I love that question because that's it's a tricky one and and that's to the community that's on the community to to do this and you know this early career event with AGU last last week organized and that crack that similar question came out and this is really something on the community and I remember well let me get started because I've read this system the other week with somebody high up and I said and when crisis and many people many of you know many of you know that crisis is you know the KU center for the center of IHC you know the science and technology center funded during this 10 year project they also partnered with some minority universities and so there was a waste work this was way before my time is like you know 7 years, 8 years ago so it's a waste meeting then there were you know a bunch of you know like students out of you know technically came to this waste workshop and this person in this app told him that he said he observed that yes there are like you know students at Wains but in the evenings it was total separation and it's just and it's something that is you know on us community to actively address and make that and be more conscious about I think you know when on purpose I think you know way more conscious by now than you know but I'm saying this easily but I think it's an asset community I like the point I like the point and then Torsten there are a couple of questions I guess because it wasn't on the list of the documents that were in the expertise and resources, non-anonymized document so when are budget, budget justification and current pending support documents introduced into the Ruby process and I would assume that yes those would accompany the non-anonymous documents the budget and budget justification portions of the proposal what about it where are those introduced are those accompanying the non-anonymous documents so the budget narrative and you know the overview will be part of the anonymous part where you know where we see you know the money the funding is you know it's not in the proposal any more anyhow so but there is a table you know with cars that has this work effort whether it be I you know seeking 0.5 FTEs or you know 6 months because this much support for Saturday and then travel and etc that will all be you know anonymously in the body of the proposal otherwise costs could not be evaluated the other questions were addressed in the in the presentation I made sure to send the PDF link that Torsten referenced earlier that has the more detailed data proposal submission guidelines in the chat it was also in the email announcement that got sent out about this town hall see another question just popped up this is kind of another good one after conducting a proposal cycle under this emerging do you plan to evaluate whether this approach is achieving its intended objectives if yes how would that evaluation be conducted so it's a good one and I think we'll do the statistics just as we started the statistics on other panels and other calls and we will see how it goes we will do statistics on we try to do the statistics on you know getting as easy you know but you know why is this more difficult than the career statistics but you know a single call is you know as you all know it's very low number statistics so it's going to be a little tricky in a way just from one call to say hey super success or not I will talk as I do with every panel talk with the panel afterwards what they think how it works but I will also see I think I want to have a post selection of proposal for the telecom with you all again because I'm curious to hear what you think and how the proposal went because I would be curious about that as well so we will do the statistics big picture the statistics on our panel will forward into the statistics of other panels and then we will see whether it was a success bigger picture or not and maybe other science disciplines will probably see do panelists have an obligation to decline review of a proposal if they can guess who the PI and the CoIs are even if they haven't personally worked with them so we can all guess right so and you know we may have an ink in here and there and you know but people don't know so I think I think it should be you feel it's probably what you don't know and so we would still do you know evaluation of the proposal we will talk with the panel as well but luckily we're evaluating the proposal so I think it's the less of a problem than it may sound that makes sense because the conversation steered away from the PI from the proposal and it's more just on to the science how can mistakes in like egregious identification criteria by the reviewers being mitigated so I think this question is a little bit interesting word so if a proposal cites a certain author many times and the reviewers assume that the cited author is the PI while it's not so we shouldn't assume you just evaluate a proposal while it could very well be larger set out or larger as an example but you wouldn't know and I think you know you may personally guess etc but the discussion would be more on the proposal than on guessing who's the proposaler you just let it go, you don't know you have an inkling maybe is that the PI in the bigger picture you will still discuss the scientific narrative of the proposal are there efforts to diversify channel participation so that's a good one and that's we are working very actively on any other you know my parents have agenda balance you know the career stage balance because that's something also that's very important you know from a senior scientist to post up you want to have that on the panel racial is more difficult because I'm thinking about it but we in cryo we are not racially diverse communities and we have lots of ways to go there ways to go there and we are thinking of ways to actually include this and we will see what will come out in the next few years because it's not a switch you can do and sustain for them at some minority or historically black college established but with regard to gender and career stage we are definitely very aware of that so we only have a few minutes left so for NOIs do those need to be submitted with dapper guidelines in mind as well that's a good question no don't sweat it NOIs are just for us for the people who see them it's taking myself and for us it helps a lot to see how many proposals can we expect on what topics so that way we can start thinking about how many panels do we need how big of a panel do we need what expertise do we need from the panel or on the panel so do those NOIs go nowhere other than to me and my friends so don't worry about it and then another kind of philosophical question are statistics enough to evaluate whether or not this new approach works wouldn't the determining criteria be whether all proposals were made truly anonymous to all panelists so without even I guess the expertise and research non-anonymized documents at the end for the top rated proposals so the one panel I was on as I said earlier pretty much everyone was qualified so it makes second quality papers that you think less important maybe less important but you can also be uniquely qualified so that be anonymized as well I think I mean expertise is certainly something that is something that is positive or can be positive on a proposal to think about it an interesting question I think all the other ones have been addressed what if the community doesn't want this so that's a question so but it's if you really talk to me and I'm willing to listen and to make this proud because as I said I'm serving the community if the community does not want to do it but that means all of the community I think honestly I don't see it down side too I really don't because you write proposals you write great proposals and if you are specialized in something you know and that's your feel you know people know but then you just simply write the best proposals almost automatically you are the expert in that field and I still I do not see it down side too I only see proposals about it because you know as I said level to playing field it's all about you know it's all about the science and you avoid some many of the biases that you just naturally have and that come out of it and you know when I was editor for T-dive you know there was clearly a racial bias you know you can out the tone of it again if I try to meet it with you as for a paper submitted by you know Asian Sciences that you just spit it out it was very very difficult to get through yours and I thought about it then this is not you know I think I don't see any downside for you know I think it for questions outside of the content that was presented I don't know if you're also looking at the list of town hall questions first yeah I think there's a question how do we show preliminary results done for the proposal I think that there was an example kind of shown how to do that citing personal communications or something like that what are the slides available we can make the slides available we're also recording this presentation we'll be recording the presentation that we gave in a couple of weeks as well so yeah all this information will be will be available to folks afterwards how do we do this paper do you want I'm not sure I imagine the WebEx recording we can probably ask for that to be posted on the solicitation I'm not sure about the slides though we can we'll see how we can send out we'll see we can send us an email to send it to you and we know we can do that as well if you don't find out a good way I'll send it to you that's a good idea do you have anything else you want to add here at the end for us one thing I really appreciate you working with me on this for me it's the first time I know that some people are nervous and apprehensive as I said you know I'm a little apprehensive as well but I think it's a good plan I think it's the right thing to do to increase fairness in sciences and we will avoid racism that we receive about scientific neocotism we will receive about when we selected the ISAP-13 we will be almost the whole we selected the question came up how much may work with this this way people really like better proposals because they have more insight into ISAP-2 and these these suspicions if we put it this way the biases just won't be taken away I think it's a good plan so thank you for working with me I really appreciate it I'm because I know we have a really great community so I assume that people will actually embrace that trying to prepare our plans to set any philosophical yeah if you have any concerns we are there for you and we know about us we will respond that's a good plan yeah like NOIs are due tomorrow but proposals aren't due until the very end of July so you have plenty of time to ask questions if there's anything that we didn't get to today that you're still unsure or uncertain about please don't hesitate to reach out if you have any more questions or concerns alright thanks everyone I wish I could see you in a way goodbye I know thanks for joining us today everyone