 Welcome everybody to day two of the Year of Open Science Conference. We're delighted that you came back and are joining us with this. This morning we are going to have a session that's focused on research assessment. All of the enthusiasm, engagement, excitement about open science, the increasing adoption has been an amazing thing to witness over the last decade. But none of it will be sustainable without attention to how researchers are assessed for the quality of their work. Who gets a job, who keeps a job depends on research assessment. And if those don't align with the values that we have for scholarship with the things that we're trying to advance with open science, then the basic evolutionary pressures of who gets to stick around what the makeup of the Academy of the Scientific Community looks like will constantly be pushed and pressured by those assessment systems. What is a rewarded incentivized for researchers to do. So this morning we are very lucky to have three representatives of the most important movements in transforming research assessment. The declaration on research assessment or Dora Haley Hazlick will represent that the higher education leadership and open science Helios represented by Caitlin Carter and the Coalition for Advancing Research Assessment So what we'll do this morning is each of them will provide about five minutes of context for what their assessment work is about what they're trying to do, how it is trying to change and reform the system. Then we'll have some group discussion and we will welcome conversation and questions from the audience as you have them. I encourage you to use the Q&A feature easier for us to monitor than the chat feature in this context, and we'll get as far as we can in the 45 minutes or so that we have remaining. So let me start with handing over the presentation with audio to Haley Hazlick for presenting Dora Haley. All right. Thank you so much, Brian. Brian already gave me a lovely introduction. I'm Dr. Haley Hazlick Program Manager for the Declaration on Research Assessment or Dora. And I want to say a really big thank you for inviting me to give really a lightning tour of what Dora is and the type of work that we do to implement change. So the Declaration on Research Assessment or Dora is a global nonprofit initiative of the American Society for Cell Biology. And we work to encourage best practices in the professional evaluation of researchers and the outputs of scholarly research. We call this working towards responsible research assessment. You may have noticed the word declaration is in our name, and that is because Dora is an organization that grew from a declaration of recommendations for how researchers, publishers, funders, data providers and institutions can better try to better assess research on its own merits. And now individuals and organizations can sign the declaration to signal their support and commitment to reforming research assessment and to date nearly 25,000 individuals and organizations have signed Dora across the world. Now, when you signed Dora, you're committing to several broad themes that run through the declaration. And these include the commitment to not use journal-based metrics as a surrogate measure of quality to be explicit about the criteria used in hiring promotion, tenure and funding decisions, and to consider the value and impact of a broad range of research outputs. And I want to note here that Dora recently celebrated its 10th anniversary. And while we may have started as a declaration, Dora is now a global initiative that actively works and will continue to work to support and develop best practices and research assessment. So how do we do this work? How do we implement change on a global scale? So Dora's approach to reforming research assessment is really threefold. We really work to engage with the community and support their efforts. We work to facilitate knowledge sharing and also develop resources. And we do this for all members of the scholarly community from research funders and researchers and beyond. And so one of the big mechanisms that we use is facilitating collective learning. So for example, we host communities of practice for research funding organizations and also for other initiatives like Dora so we could all be in dialogue with each other and move together towards a shared goal. We host free virtual events with community members on topics relevant to them like how the pandemic impacted hiring and promotion practices or how preprints could be leveraged for early career researchers to gain recognition for their work. Because one of the biggest hurdles to organizations who are working to reform their practices is a lack of examples of where to start. Dora has really done a lot of work to create infrastructure for the sharing of knowledge to showcase what reform looks like practically. So for example, Dora hosts and curates a resource library. We manage a repository of case studies. And this year we launched reform scape, which is a database of openly available responsible assessment practices and policies from academic institutions. And the real goal of reform scape will be to help organizations share their work with their peers and also find practical examples from their peers to help them implement change themselves. Finally, Dora also develops resources to support change and we often do this in collaboration with our community members and these can all be found in our resource library and include tools with guidance on how to implement change. And I want to just highlight that this week our collaborators at CWTS Leiden released a qualitative study of faculty hiring promotion and tenure assessments in the United States. And this is as part of Dora's Tools to Advance Research Assessment or TERA project. Alright, so wrapping up my five minutes. I want to spend a little bit of time talking about what success might look like for Dora. What does success look like for Dora? And we know that research assessment is very closely tied to research culture and it's really a systems problem that impacts everyone solving these types of challenges requires a collaborative systems approach that addresses the underlying culture, infrastructure and conditions of the entire scholarly ecosystem. So from research funders and publishers to academic institutions, research administrators and researchers. And Dora takes a systems approach to support change, meaning that we work with all actors in this system around the world. Because we're a global organization that serves all members of the academic community. I really want to stress that success for our signatory organizations, you know, that's going to be deeply context dependent based on country organization institution, etc. So for Dora as an organization, I think one benchmark of success would be would look like individuals at research institutions and funding organizations feeling supported. As they champion and implement more responsible research assessment practices. And ultimately, we really strive for a world where researchers are hired, promoted and awarded funding based on their most relevant work and outputs. And with that, I will hand it over back to you, Brian. Thank you. Thank you very much Haley for that great overview. Next we will have Caitlin Carter presenting about the Helios project higher education leadership and open science. Caitlin. Thanks Brian and thanks Haley so Helios I think wouldn't exist if not for the efforts of Dora and we too are inspired by Kauara so I just wanted to also thank center for open science for inviting us to speak and for this wonderful event. So Helios as Brian said so it's the higher education leadership initiative for open scholarship or Helios open, and it's a cohort of 101 colleges and universities committed to collective action to advance open scholarship within their campus and across US campuses. So it is a project that has emerged from the National Academy's round table on aligning incentives for open science. Now it's open scholarship. It's an inclusive term because it's something that we have heard from a lot of our institutions and a lot of the disciplines within them that sometimes the social sciences humanities does not feel like they saw themselves in this movement. So we're really happy to use that language to make sure that we are inclusive. Again, we have 101 members that covers about three and a half to four million faculty staff and students across the United States. We also have leading this effort so Michael Crow is the president of Arizona State University, Ron Daniels is the president of Johns Hopkins University, and then our strategic lead is Gita Swami who is the associate vice president for research and vice president or sorry vice dean for scientific integrity at Duke University. Those two presidents along with President Roslyn artists and President Danny Anderson were a participants in the round table, I believe around 2019 and the round table really brought together different leaders of different industries different sectors professional societies and the group saw that there really was a need for the highest level of the institution to help with what we've been shouting for the roof from the rooftops for so long to help you know decision makers at the table to help us advance open scholarship norms for awards incentives. So they issued a dear colleagues letter and said, now is the time this is a call to action join us in advancing open scholarship collectively across us higher education 65 institutions signed up and joined the call and said we want to be a part of this. We've grown to 101, which is great it's a lot, but yeah so we from 2022 to 2024. We surveyed our institutions and asked you know what do you want to work on together, and for areas emerged you know we asked you know what are your, what are your measures of success, if you were to join this coalition and truly advance a more equitable transparent research ecosystem. There's a group institutional departmental policy language that was the number one priority surface and that really is focused on policy reform research evaluation and assessment reform. There was another subset of institutions that really wanted to explore how open infrastructure or open scholarship infrastructure could advance some of these ideas. A third group that emerged which was good practices and open scholarship really focused on making open scholarship easier for researchers and the institutions that support them. And then cross sector alignment really staying true to the, you know the the makeup of the round table and seeing how that translates into this context and it's especially important for example, when advancing reform ideas research assessment change that to to honor some of the the norms within you know professional societies and disciplines, align with federal agencies and public access policies so there's a lot of opportunities there. So our theory of change is really you know if we we believe that if the research incentive system rewards open scholarship explicitly like 10 year promotion review and hiring policies and open scholarship is easier for researchers and the institutions that support them through alignment with those other sectors. This environment makes change easier. For us success truthfully we're still figuring it out. I don't think that we ever expected 65 now 101 institutions to answer the call. That actually is it's a success but it's also a challenge so it's success for us is not growing in numbers of members. So when we did ask our institutions that that that's the approach we took we had 65 we asked them what we thought success look like and try to work from there. We learned a lot and done a lot in the last two years so we've produced issue briefs that really make sure that leaders, presidents and provost specifically know what the issues are when the Nelson memorandum came out. If anyone knows higher ed institutions they're very on top of current events but at the same time some are still grappling with for example the NIH data management and sharing plan and and didn't quite yet even have time to focus on the Nelson memorandum. So we're here to inform and also educate about what the opportunities are to basically align our reward systems with those that are referenced in the public access policy with equity at its core. And then our approach moving forward, we consider, you know, three institutions working together to do something bold like changing their tenure and promotion policies within their school as something that is a success. So, sometimes again in our in the way that we are pivoting our work and how we've previously done our work successes sometimes just awareness raising among the highest level of leadership, knowing that they have so many priorities that pull their attention away, and making sure that they stay engaged and see opportunities and then take that action when we need them to do that. So, back to you Brian. Thank you, Caitlin for that great overview. Our final introductory presentation will be from Eva Mendez who will be presenting the coalition for advancing research assessment. Thank you very much Brian. Thank you very much for the invitation. Thank you, Nessa for organizing the year of open science. I think we are more than in the era of open science. And I'm very happy to hear my colleagues that we are on time to advance on research assessment in the in the global spectrum of research. My name is Eva Mendez. I'm a faculty member. I'm a researcher and a meta researcher. I'm the head of the Open Science Lab at the University Carlos the Federal Madrid. But I'm also a member of the steering board of Kauara that you may have heard about that. And people think that Kauara is a European project. Kauara is a global endeavor. It's a global initiative that is building on progress made so far for other big alliances, which is Dora, Lady Manifesto, Concom principles and all these initiatives that they have been heading before us and a common direction to evaluate research in a better way. And more accordingly, with the right with the time that we need in pandemic and challenges of climatic change, we cannot keep evaluating research as we did in 19th century, when the journals have only a printer to publish them and is the was the only way to communicate knowledge. We all agree on this. And the initiative of Kauara was creating not yet another declaration. They are more than 200 declarations all over the world in the in the context of open science. And I always says that is a very Spanish sentence that is a toast to the sun. We all agree on the principles we all agree on the we have to change the way we measure science. But so what, at the end of the day, we keep looking at a index, we keep looking at a general impact factor, and nothing happened. So the initiative of creating Kauara was in that it was creating not yet another declaration was creating a community board on up. The Commission, the European Commission was kind of our facilitator, but it's not a top down initiative. Thank you European Commission to let us happen this and make a community board on up and be capable to join people and institutions. The difference between Kauara and Dora, Dora is 10 years old. We are only one year old. It was created in 2022. We create the community in December 22. So we haven't just running for one year. We are just creating the structure of a bottom up community based on institutions, which is a very difficult thing to manage. But we have to distinguish between the agreement and between the coalition. People that they join the agreement, they can or cannot join the coalition. The coalition is the coalition of doers, the people that they just step forward and say, I want to change the way we measure science. With my hat, in my institution, if I have the role of being an agency of evaluation, okay, good enough. But if you have the role of being just an institution and university, or a funding agency or whoever you are, you have to step forward and have a roadmap to change in your kingdom, the research assessment. And the other thing about the community is that you have to share what you do and learning by doing with the whole community, because this is something that really, really encourage us to do it if we do it together. We know that nobody on their own will change the way we measure science, because we want to do it in a global perspective and bottom up. And the other thing is that the commitment, the commitments that similar to Dora and Helios and all these initiatives that we are exactly in the same page, we want to underline that we want to recognize diversity of the contributions, not paper centric communication. We create data, we create other outcomes, and we want to base the valuation, not only in one kind of contribution, as well as abandon the inappropriate use of the research assessment, general impact factor, completed based on publication metrics, and also avoid the use of rankings. And the whole commitments, the whole commitments come along with commitments to share and to communicate what we do to the community. So that's why we have created different working groups based on the topics that they are crucial. And the topics that they are crucial indicators, we cannot blame people to use general impact factor if we don't provide them with something else, we have to agree upon together without the new indicators that they can't measure at some point science. Also, we want to just underline the value of peer review and the qualitative research, but we have to really legitimize the qualitative research, because for some reason people think that is biased. And also, we have another approach, which is the national chapters. I would love to have a national chapter for the US because this is a global endeavor, and we want to move forward together. But the problem is that it's a global endeavor based on institutions, not in individuals. Dora, as an individual, you can't find Dora. You can be a member of the research data alliances, but here you have to commit your institution to do something. So we want to move forward on this initiative. And the national chapters also legitimize the real boundaries that we have in the research evaluation. It's not the same that would happen in the United States, it could happen in the Netherlands or in Spain or in other countries. So we have to also communicate these common endeavors with the global world, including the global self, because we have sciences global, research evaluations should be global, and it's not only the year of open science, it's the era of open science. And we are very happy here to know that we are on time to change the bottleneck of open science, which is research assessment. You are very welcome. Just join us. Thank you very much, Eva. Thanks all of you for those excellent overviews of your initiatives. And what is so clear just hearing them back to back is how complimentary they are, because so much of this type of reform work requires not just a singular solution, but a coordinated effort with many different approaches that converge on some of the shared values and approaches that we're trying to accomplish. So we will transition now into some open discussion among the panel. Anybody that has questions is welcome to post them into the Q&A. But what I'd like to start with among the panelists is some reflection on how this gets started. There is an obvious tension in any type of reform movement of trying to ask the people that can make reform to do something meaningful enough that it would make change, but not have such a large ask that they can't possibly get from where they are today to where they're going. So I wonder if each of you could talk a little bit about how your initiative wrestles with that tension of something that creates meaningful progress, but not so far that you can't get people engaged with it, and how there's variation among these an interesting way to resolve that tension. So Hailey, maybe you could start with. In the hot seat. All right. So I think this is a really fantastic question and I think that there are a few ways to answer that in terms of kind of building that bottom up support or making people feel like this is something that is doable. We really try to advocate for open and consistent communication around all issues, you know, from basic ideas around say the most basic avoiding the use of journal impact factor as an inappropriate metric to measure research quality all the way up to conversations around how to iterate and expand on policies that are already in place. I think meeting people where they're at and respecting that and helping them start those conversations because at least for Dora, any any movement and any advocacy is is good, and that is what grows grows a movement within in this case say an academic institution. It's one of the reasons why we accept signatories who are departments. That's makes up a really big base of organizational signatories in the US in particular so if a department wants to make a change. What we'll hear, I can say anecdotally from those departments that do make those changes is that other departments will look at them and say, Oh, well, that's actually more doable than I thought it would be so I think really supporting folks at whatever level they're willing to give and trying to foster them there would be would be a good way to do that without setting setting the bar too high. Yeah, so it sounds like that is really leveraging the power of norms pure influence of Oh, this isn't as hard as I thought it might be and sort of gaining momentum through that that's excellent. Caitlin or Eva, do you want to elaborate on how your initiatives wrestle with this. Sure, so I'm going to do it. That was my fault. Go ahead. No, no worries. So it was just going in order as you described last time so following directions. Yeah, so I'm going to start with success because I'm going to talk a bit about what failed and also again just kind of admire the work that Haley and Eva are leading. So our success or our sorry our approach now is awareness raising as the lowest bar for entry, but with a specific group and peer influence with provost vice presidents and vice presidents for faculty affairs which is a bit more difficult but that group specifically, and then getting what we'd like to do is then get a small cohort to commit to action and reform, which is much harder. And then our role after that is to shout the successes from the rooftop so others can explore what might work for their institutions I think that part is easier. And obviously that creates that peer influence and so we have a very knowledgeable and robust community of almost 200 designated campus representatives who are there supporting their leaders who have been engaged for the past two years, and can support some of her and intra institutional or campus activities, like the ones that Haley described. So our approach at first it did not work, but I still think it led us to a more successful place. So again that that number one priority was you know change tenure and promotion but you got everybody in a room together and we, we heard. I'm not the right person to be doing this advancing research assessment reform even though I am a leader, I don't have any power on my campus it was sort of like a diffusion of responsibility and pointing fingers of like well, if I do it as a provost even like I'm a friendly provost and I don't want to do something that my faculty aren't on board with and if it was a department chair it was. Well I need to not sure the provost would support it and so it was working through some of those institutional change challenges that are happening within department schools and just the institution in general. So we wrote a joint statement thinking this is easy we will just articulate our values in higher education and how you know they pretty easily align with open scholarship. We suggested some actions campuses could take and then asked our reps to socialize them with leadership. No institutions were interested. They said we will not sign anything that is actually more dangerous for us as leaders because then we're committing our name and our institution and our brand is something and that's not something that we want to do. October 2022 we changed it so we said let's make it a little bit more flexible and say, we commit to exploring conversations are starting conversations thinking you know what we could just commit to exploring that's commit to exploration that's easy. That didn't work either. One institution shout out to Whitman College you had the provost as the designated representative only institution that signed. So we reassessed our approach and continue to learn that the specific role on campus potentially could be that VP of faculty fairs and provost hearing, it kept going up the chain so we said let's go ask and so thanks to NASA, who we were collaborating with during the year of open science like Center for open science is they hosted a workshop or funded a workshop through Florida International University one of our Helios members where we collaborated, and it was specifically for presidents provost and vice provost for faculty affairs and research and 50 joined and said, we think this is true and we're going to explore some action plans to basically advance research assessment assessment and evaluation reform. Those examples are remarkable because it's so easy to presume that if one is in a leadership position, right, it's just a matter of wanting to do it. All right, I'm on board with the concepts, we're going to do this as a university. The other examples here highlight that that is not that easy. You could have people that are completely on board hearts and minds and still not feel like they have the opportunity ability efficacy to act because of how decentralized the culture is and the challenges of bringing everyone along so that the both of those who are working on Helios efforts of working on the leadership and bringing whoever is the coalition of a willing along sort of play that complimentary building role together it's great to hear the the challenge as well as they where it's moving. Eva you want to sort of give some perspective from core on this issue. Actually, I completely agree with with what it has been said so far, because this tension comes from that we have to maintain the system at the same time that we are creating another one. And unfortunately, we don't have money to do that. I would love to go to the to bed and overnight tomorrow have a new completely academic system and new complete research system with sciences more dedicated to the society to societal changes challenges, and that is not going to happen, because we have to maintain both systems. So the tensions comes for that, because research is a human enterprise. We are no researchers for the 29th century that we are looking around butterflies and we have a lot of money to do that. No, we have a tenure track. We have a career involved on this. So we want to have the guarantee that we know what we have to do, publish in the Q one. Okay, I will do it. I don't care. I do. We do whatever it is. But we have to recognize that the web have changed every single thing in our lives. The way we communicate when we have a boyfriend, the way we chat, the way we read everything except the way we communicate science, the way we found science and the way we evaluate science. Why for because of this tension, because we don't want to do this change overnight, and we have to keep the thing movement. It happened to me when I used to be the deputy vice president for research policies and they have in the agenda of the day. This is super important. We have to sign choir. We have to join this movement, blah, blah, blah. Okay, yes. Wonderful. Everybody agrees. And in the second point of the agenda, we have to evaluate a researcher to have this career. What is his age index in the same conversation in the same room. So this is the reality and this is a tension. So that's why we inquire, we want to have a roadmap and say, Well, go as far as you can. Perhaps you cannot change overnight everything. Listen to the early career researcher. For me, this is crucial. I have a joke that everybody always say on the 40, there is hope. And I do really believe in the early career researchers that they are absolutely willing to share science, willing to compromise with the society and get rid of papers. That's the whole thing. But I think the tension is there and just we have to move a little bit. Yeah, thank you for that. But it's such an important point that the principles of open scholarship are so easy to embrace, especially when one is early in the career and not jaded by the system as it currently is. And so the opportunity really is to leverage that enthusiasm engagement by ECRs that can then empower those that are in leadership positions to actually make some proactive steps to support that change. So the comments from all three of you sort of circle around a interesting challenge of the buy-in and there's variation. I don't want to get overly provincial, but the difference between US level of buy-in on these and EU and even more globally, right? UNESCO has gotten substantial engagement from a variety of national contexts on these issues. Are there unique things about the US context that are barriers or other places that are resistant that we can look at commonalities that we might be able to start to poke at to see how change can happen? And we don't have to go in any order. So anybody that has comments, welcome. Well, I think I can start from the global kind of top-down sort of view vantage point. What we have found at DORA, so we can, we accept organization, as I said, we accept organizational signatories and there's a very clear difference between the types of organizations that sign DORA, say, in Europe or the UK. Usually, those are entire academic institutions, universities, as opposed to in the United States, we really most often see, if we're looking at academic institutions, departments or even smaller units within those departments that want to signal their support for DORA and they become signatories of DORA. And what we have heard is that it can, because US academic institutions are more decentralized and departments, as opposed to other parts of the world, departments really have the say in how hiring promotion and tenure practices are set out and what types of criteria are implemented at that level. And so that, I think, is a really big difference. And I think, you know, as Caitlin said, you have to meet people where they're at and you have to find who is the best person to talk to to implement this change. And so that's one of the reasons why we have our case studies, because, you know, we showcase how this change goes about and we're currently in the process of working on a case study from the US, our very first one, so I'm very excited about that. So this is top of mind. But I think that would be that would be one of the biggest challenges and just knowing that just having that information is really, really helpful to know to know who to talk to, as you said, Brian. So that's that's my initial, that's my initial thought process and I think can be open communication is just probably one of the best ways to facilitate buy in at any level. It's really helpful. Yeah. Great. Thank you, Caitlin or Eva, you want to elaborate. George, I think I said a bit about just the reluctance to sign on to anything publicly that's definitely a trend. I can't. I'm not an expert as to why but it's definitely something that I could make some guesses. But another thing is just the understanding and this also may be the case in in different countries but that that siloed nature that Haley talked about also. There are efforts that you know from University of Vermont after working with us and doing their own advocacy on campus for changing tenure and promotion. They passed a faculty Senate resolution from the entire institution saying we because we're members of Helios and open and because we are participating in the National Academy's round table and because the Nelson memorandum. We want all of our schools and departments to change their tenure and promotion or change their policies to embed open scholarship open science within them. And that is really wonderful that's a huge signaling effort but the implementation has to happen and that's where some of the challenges and getting into the weeds of what doesn't work for each institution and how different institutions are across the US. That's where we can just really get bogged down and continuing to admire the problem or just you know we recognize that there this is an issue, but it's so hard to change in the US, at least what I'm hearing. Because of the big differences between our institutions and then our schools which a lot of times are have the autonomy to kind of do what they want and the departments. That's the problem and so again because we're a leadership effort we said let's go right to the provost and president and explore this. Thank you for that. Just to finish because I'm the only one that is not American here, North American. And I think it's not a huge difference because the problem is that you probably have a lot of autonomy in the institution that I agree completely with Kathleen when I was in my period of full radius color and in the United States they found that you have a lot of a lot of autonomy in the universities for example but this is perfect because this is can produce a domino effect. Say you know if MIT do something the Stanford want to do the same and that's what we need a very big snowball that comes together because we all have the same problem. And that autonomy just ask for who is the bravest who can be brave enough to step forward and say well we will do this little tiny bit. We are going to I don't know change the tenure system no but in as a pilot next year we are going to use core principles to evaluate the fellowship of the students as postdocs. Okay, that's it you promote the early career research in a different manner, but whatever you do is valuable. And the only thing is that you have to step forward. And is what you said Ryan is not it's not time to wait is no time is going to happen. UNESCO has a working group on how you motivate and how you finance research and at the end is a question of funding. It's a question of the golden rule who has the goal has the rule puts the rule the rule. If the NSF say you have to do this way and they do to give you the money people will change the mind. So that's that's the point and also there is a crucial thing that could happen that is seduction. The seduction is something that is not is the best motivator of will is something that happened when you do it because it's easy. Because this is what I call the WhatsApp effect of research. Do you remember when you start using WhatsApp? No, you just use it. I want to that happen and say why I'm using now know the journal impact factor I'm using something else just based seduction. It's not that nobody has to convince me is the system that convinced me. Right. Yeah, because we're already in a system and we just accept that part as it is and we may decry it, but we don't feel powerful enough to change it. But as it evolves, we may likewise. Oh, actually, this one is more aligned with how I want things to be great, great points. All of you. The there's a sort of an interesting challenge that I think is embedded in all of your comments around inclusion, which is the obvious need to bring everybody along across the Academy. Caitlin raised this early in talking about the movement of language to open scholarship to make sure that the humanities saw themselves in it. The customization or need for different departments Haley brought up as has been an important element of how it is that people see their own work themselves and all of this. But the challenge, of course, is that if it's a general movement that's trying to reform assessment practices. To make it as inclusive as possible, then means it helps us to be as custom as possible to all of those different needs across many different communities. So if there are no standards, then how does anyone figure out how they should change in an effective way? And does that create a huge burden on every individual actor? Well, it's up to you. It's up to you to figure out how you want to change. Well, I need some tools. I need some ways to figure out what to do. And this may also play into that challenge that articulated about leaders. A provost may feel like, how could I possibly institute a new policy that's going to apply to the chemists and the poets and the business school professors? I just, I can't figure out how I would do that, even if I could. So any of you have thoughts about how that challenge of maximizing inclusion, bringing everyone along, but still giving people tools and pathways to be able to make changes efficiently for their own domain, their own work. I can start. So I'll just say that that's been a big emphasis from the beginning. So we actually asked that question. One tactic we used was we asked one of the original presidential signers on the initial letter that called action. Danny Anderson, he's the president emeritus of Trinity University to work with us to have those conversations with leaders. And he really emphasized the opportunities as a provost. So he's been a he's been a dean, he's been a faculty member, a dean, a provost and a president. So having that experience and having that peer to peer engagement and conversation, it lands better when it comes from him. So when he's talking to presidents and provosts and vice provosts or presidents for faculty affairs. And if he says, this is an example of what's happening, you know, exerting that peer pressure that peer influence at, you know, MIT. So Stanford, here are some opportunities for you as provost to do this. So here's all these different pathways. So it's validating that it looks different, that your campus is special. But it's that peer influence that I think really in knowing how to speak the language and he has done so much research in his own sort of executive coaching training to understand like what are the what's the complex model for change. And so he can point out like what happens when you don't do something. But as a specific leader, he talks to his peers about not having a reactive mindset and about the opportunities. So that's been incredibly transformative in our work. Excellent. Thank you. Hey, we're able to do you want to elaborate on that? All right, I'll go ahead. Yes, I really think that one thing, building off of the previous conversation. And I think that this is applicable here is doing a good job of showcasing when people are making that change because that's that's one of the biggest barriers that we have encountered is folks will say, well, I don't know where to start. I don't know what to do. If you can point to examples and getting those initial examples, that's I think one of the toughest pieces, right, getting those really good practical cases of what change looks like. They don't have to copy it one for one and they don't want to copy one for one what another organization does. You know, there is that domino effect that Eva mentioned that once folks see that everyone around them is making change, then they want to apply that in their own context. I think the recognition and the recognition and rewards program in the Netherlands is a really great example of this. They have a very flexible. In addition to Quora, of course, but the recognition and rewards program has been around for several years and they have a really flexible set of guidelines that are being implemented across the country and they have all of these different examples of how these guidelines are implemented in a way that is unique to each department or school, but still falls within that umbrella. So it's really the balance of not being overly prescriptive while also being, you know, having firm enough guidelines that people can build off of. But I think the flexibility is really important and, you know, just allowing people to make changes that work best for them because good doesn't look the same from institution to institution or, you know, what is valued is very different from country to country or region to region. So we can't be too prescriptive with that but therein lies the challenge. Yeah. Yeah. Thank you. Eva, any final comment on that? I do. I do really agree. So on behalf of the time, I will just tell you if there is a burning question in the, in the Q&A because I completely agree with you. And also Quora has a license in order to share action plans that you can look for that and I can share the link afterwards. I think it's in my slides. So this is exactly what it would have happened. Excellent. Thank you, Eva. Well, we are at the within the minute of our time period. So I will use this as an occasion to wrap. We're really just scratching the surface of research assessment and how to do it. But what's clear is that it's a it's a ground game. A lot of this is working person by person institution by institution program by program to try to build energy around these issues and advance experimentation and make visible. Those things are working to share knowledge so that the change can spread. Both Jenny and Paul in the in the Q&A offered other pathways that I think are worth everybody thinking about when they're thinking about change for research assessment. I think they're very resonant with the discussion that we had Jenny, for example, noting that well departments may be the locus of change because the norms of scholarship are within scholarly disciplines rather than across an institution. So maybe societies, scientific societies are a great partner for departments and disseminating those changes across scholarly domains. And then Paul offered that, you know, maybe you just go even go down a level individual faculty or groups, maybe ones that can enact change more effectively at the outset and then build capacity across programs from there. I think both of those are very resonant with all the comments that you made. So thank you everyone for coming to this session. Thank you Haley Eva and Caitlin for presenting the work and just for the work that you're doing. This is a critical part of scaling and sustaining the open scholarship movement to advance transparency, equity and reproducibility as the aims are of the global work. So thank you everyone for attending this session and enjoy the rest of the meeting.