 I would like to welcome our last speaker, Erin McKernan, and she will be speaking about evaluating researchers. Are we rewarding what we value? All right, so we're gonna send you into lunch needing a drink, because Sarah had some bad news and I have some worse news. So my name is Erin McKernan, I'm the community manager for the Open Research Funders Group, and I'm also a professor at the National Autonomous University of Mexico in Mexico City in the Department of Physics. It's always hard to stand between folks, sorry, how do I move slides here? There we go, yeah. Always hard to stand between folks and lunch, so if anyone is really, really hungry, you can just read this slide and then skip out, I won't take it personally. The questions that we're trying to answer in this research are, what do universities value? What do they actually reward? And are those two the same thing? And I'll just tell you no, they're not, at least not yet. So I'm gonna tell you some bad news with our data and then tell you a little bit about what we're trying to do to hopefully correct the situation. So if we, again, ask what is it that universities value? We can take from their mission statements some of this. So if we look at university mission statements, and a lot of you may see your own university in this, those often tout the importance of community engagement, public engagement, working for the public good, especially if you work at a public institution like I do. So if we think about those values, how might they be reflected when we're looking at faculty behavior at the time of evaluation? So if we look at faculty work, there's lots of different aspects that we might think of as public dimensions of that work. Lots of different knowledge exchange that happens outside of a traditional journal publication. There's sharing information through articles, through preprints, they're sharing data, they're sharing code, they're sharing educational resources. But there's also science communication through blogging, through educational outreach events. We may be interacting a lot, depending on our research with communities, trying to reach out to those communities, get them participating, collaborate with those communities. And so where are all of these things in the promotion and tenure process? So when we asked these questions, we realized that we really didn't know a lot about what promotion and tenure documents actually say. So this is the RPT project, review promotion and tenure in collaboration with the Skalcom lab. So Meredith Niles at the University of Vermont and Juan Pablo Albrin at Simon Frazier University, primary collaborators with me on this work. So what we did was collect over 800 documents related to the RPT processes at 129 universities. So this is a representative sample of universities throughout the U.S. and Canada. We had 381 academic units represented there. We had R-type, M-type, back-type institutions and also various disciplines within those academic units represented. And so what we did was analyze those documents to find out what they said. So if we first look at a few key words here, we can search those documents and look for mentions of public and community and public or community engagement. We see that the mentions across all institution types are relatively high. So 60%, 70%, 80%, even 90% in some cases. So again, we are at least in this sense seeing an alignment, right? So institutional mission statements mentioning those aspects and the documents mentioning those aspects. But when we look at the language surrounding those words, here's where we start to see a problem. So the most common word we saw next to public was service. And the most common words we saw next to the word community was service and university. So first of all, what community are we talking about? We're still talking about the academic community, community within our university walls. That's not necessarily what we mean when we say community engagement, right? Or working for the public good. And then what's the other problem? What do we know about service? Service is the most undervalued category in our RPT trifecta of research, teaching and service. And not only is it the most undervalued, but it's also the one that falls most heavily, often on the shoulders of women, faculty of color and other underrepresented groups. So now, not only are you not rewarding the very things that could help you achieve your institutional mission, but you also have a major equity issue in terms of who are performing those activities and are you selecting them out of your system? Are you selecting out the very people that are holding up your institutional values? Then we went and looked at a few other words, impact. Everybody loves impact, right? Metrics, traditional outputs, traditional outputs over 90% in every institution type. But again, is traditional output, is that really measuring community engagement or public good? A lot of those traditional outputs and those metrics like citations aren't measuring those aspects of community work, right? One potential one that could measure whether we're getting to the communities we want to get to is access to literature. But when we looked at how often open access was mentioned in these documents, it's a very depressing number. So it was around 5% and a lot of those mentions were actually negative. They were cautioning folks against publishing in predatory open access journals. So again, are we rewarding what we value? So the public dimension of impact, when we analyze this language, the public dimension of impact is rarely mentioned explicitly in any RPT document. So only 9% of our type institutions and 11% of M types. That's a very, very low number if we compare that to the overall number of mentioning public or mentioning impact. But these two things don't seem to go together in the ways that we would like them to. So here's that bad news. Really high incidence of the terms public and community across RPT documents, but neither explicit incentives or clear structures of support for those actual public dimensions of scholarship. So we have arrows going in opposite directions, right? We have the university saying this is our mission, but at the time that they're going to evaluate faculty for review, promotion and tenure, they're not rewarding what they supposedly value. And faculty are also feeling a mismatch here. So as part of this larger study, we went and surveyed faculty at the same institutions for which we had RPT documents. And we asked them, why are you publishing in certain venues? And this will surprise none of the psychologists in the room, but when we asked them why they published in certain venues, they said, well, I'm interested in readership and audience, but all my colleagues only care about prestige and impact factor, right? So I'm doing the right thing, but everybody else is not. But if everybody thinks that, what's going on? Why are we not having more conversations about our actual values? And part of it had to do with they think they're being evaluated on these things. They think everybody else is concerned about these things. So again, the results suggesting a disconnect between what academics value versus what they think their peers value and particularly what they think their peers are evaluating them on. So that's all the bad news. Got it out of the way. What's the good news? The good news, as Sara was saying, is that there are actually a lot of efforts going on throughout the world focused on promotion and tenure reform. There's growing momentum. I think we can say there is definitely growing momentum. Whether or not those initiatives are having the immediate impact we would like to see, I think the answer again is not yet, but are people talking about it increasingly? Yes, right? And so I just want to mention this is not a representative sample of all the things going on, but just a few things that I'd like to talk about. So last year, the Department of Psychology at the University of Maryland was the first, if not the only one I'm aware of in the US that went through and revised their promotion and tenure guidelines with the championship of Mike Doherty as the chair of psychology there. And you'll now see that they've taken kind of a values-based approach. Again, what is it that we actually want to incentivize? And they have embedded open science language and practices throughout their tenure promotion guidelines and also their merit review guidelines. So I encourage folks to check that out. I'll provide the slides afterwards where all the links are available. The other thing we've been doing with Mike Doherty is now going around and talking to department chairs about this values-based approach. So a few months ago, we ran a workshop at the COGDOP meeting. So this is a meeting for chairs of Departments of Psychology. And we did that. We went in, we were using the values-based approach from Hugh Metrix, if folks are familiar with that project. And we went in and said, well, as psychology departments, as chairs, what are you trying to incentivize? Do you want rigor? Do you want reproducibility? Do you want inclusivity? Do you want diversity? Okay, if those are the things you want, then what are the behaviors or practices that you need to measure that would enable those things? And then, how can we rethink our promotion and tenure guidelines based on that? How can we get out of this research, teaching, service, metrics, traditional outputs box that we're stuck in and think about a values-based approach? So we did that and it generated a lot of conversation and we're now thinking about writing that up. The good news is, too, they're going to be running another workshop for any of you that are going to be at the APS meeting. That will be with Mike Doherty and the lead of another initiative I want to talk about. That's Caitlin Carter. She is leading up the Higher Education Leadership for Initiative for Open Scholarship. So this is Helios. This launched the last year and it is 91 institutions now. It started with 65, grew to 91. Across the U.S. that have raised their hand and said they are interested in Open Scholarship, committed to this as one of their enabling strategies. And we have a working group within that initiative that's focused on institutional policy, so reforming promotion and tenure. What can we do there? They put out a joint statement talking about can we at least start those discussions? We know promotion and tenure reform is hard. We realize it's going to look different at every institution but can we start those discussions and have them together? So they are starting that work and really hope to see great things coming out of that. At least now we're having those conversations. We're getting folks together to say how can we better align our institutional mission and the sharing practices and the community engagement that we would like to see with what we're evaluating in the promotion and tenure process. So just a few examples and I think I'll leave it there but thank you very much for your time. Thank you, that was super interesting. We have almost three minutes for the questions. Hi, Kathy Zeiler, Boston University School of Law. This is super interesting and I've been writing to try to give some prescriptive steps. All sorts of actors can take including school administrators and including tenure and promotion. And I gave a talk recently and a former dean of the Yale Law School was in the audience and he raised his hand and said, why would any dean wanna do this? We're winning under what we're doing now under the old standards. And by winning I think he means we're doing well in the rankings. So why would we wanna rock the boat? So how do you, what encouraged, I'm interested in why Marilyn is moving forward and these other schools are moving forward. What is the incentive? What are the motivations? Yeah, that's a good question. I think there's a few answers. So one is, are you really winning, right? So do university rankings again really represent those things that are touted in our mission statement? Are we measuring community engagement, public good? Those things don't often get factored into those calculations. So when folks say we're winning I would question whether that's actually the case. The other thing I'd say is that there's growing momentum also on the federal policy level. This is gonna look different for different disciplines but these changes are coming. So if you think you're winning now you might not be winning a year or two from now if you haven't already set your faculty up to succeed in terms of sharing. Because these things are, these are coming, right? They're in the OSTP memo that came out last year is gonna require article sharing, data sharing. I think we're gonna see a growing trend with that with funders, we are seeing a trend. So, and that's another one, you want the funding? You might not be winning if you're not encouraging faculty to share. So that's how I'd entered that, I think. Sorry, there is a lot of questions here. There's a lot of questions, it was a very nice talk. Thank you. Wolf Kinkers North Assistant Professor on a tenure track Michigan State University. I've been an open source open science advocate for like roughly a decade and I feel like what I took away from this is like, the main thing that we can do is win at publications and show that open science is a better model. I'm wondering what are the suggestions you have for young tenure streams, young tenure stream faculty as well as people in the open science community to push and make us transform things towards a better future. In two seconds. No, now it's going the other way. So yeah, it's really hard and I'll say, I'll just be completely honest. So I went through the tenure and promotion process just last year. It was almost not successful. I was almost one of those people that got run out and now have 10 years, I can say this probably. Congratulations. Oh, thank you, thank you. But part of the reason that it almost wasn't successful is because I had a lot of shared products like data sets and software and preprints that just weren't counted. So now trying to go to my department and say, again, what is our mission? What are we trying to do? We're the largest public institution in Mexico, right? So if I have shared educational resources, that's putting some benefit back into the community. Why are we not evaluating this? Why are we not valuing it? I'm not sure I have a great answer for you. I would just say one of the things that helped me was sharing all this raised my visibility as a researcher when I was on that tenure track, right? And helped me create a name for myself in a way that I don't think I would have been able to do with just journal publications in a niche area of research. And that helped me in the end to eventually win that case. And so just kind of pushing back and saying, part of this is visibility, part of this is getting out there in front of people and openness is helping me to do that is one strategy. I won't say it's easy. I nearly failed, but that's what I'd say, I think. Thank you very much. I think we should move to the lunch break now. The next session will start at 1 p.m. At 12.30 we are starting poster session in the Great Hall. So I suggest you get your lunch and go to the posters and we meet here at 1. Thank you so much. Thank you to all the fantastic speakers.