 Good morning. My name is Dan Laramore. I am an associate professor at CU Boulder's Bio Frontiers Institute and Department of Computer Science, where my group studies academic faculty from faculty hiring networks and prestige to the socioeconomic roots of the professoriate and the impacts of parenthood on academic careers. And it's my great pleasure to serve as moderator for today's kickoff panel. First, however, a small public service announcement. When you look at the coffee, the little label for decaf is shiny and at the very top. So don't make the mistake that I made and get the decaf by accident. OK, so DEI efforts and initiatives are really common, particularly in the science and science-related fields that are of interest to the meta-science community. And while many of the motivations for today's DEI efforts are rooted in observation, in evidence, and generations of advocacy and organizing, many of the interventions to improve DEI are rooted in theory and may therefore benefit from an evidence-based approach. As one panelist mentioned to me, inclusion is sort of a vibe, but to include is a verb. And this sort of separates the difference between something that is less measurable and something that is more measurable. So this panel asks broadly, how do we, as a meta-science community, a community interested in a more scientific understanding of how our communities work, discover, and thrive, move toward evidence-based practice in the DEI realm? So today's panelists approach this question from very complementary perspectives. And while you can read their full bios and accolades in the program, I want to introduce each briefly by mentioning the way that they approach today's topic. So first, I'd like to introduce Dr. Catherine Alfano. She draws on deep expertise in the field of cancer research and behavioral interventions, including randomized controlled trials. Next, Dr. Hannah Rubin brings together complementary backgrounds in philosophy and evolutionary game theory, studying not only individuals, but teams and collaborations. Welcome. And our third panelist is Professor Jordy Goodman. She also studies teams, but with a unique perspective of a legal scholar and former patent prosecutor, probing the ways that intellectual property and under-representation intersect and interact. Welcome. So I'll get out of the way in just a moment and hand it over to the panelists. But first, the order of events. Each panelist will share a small portion of their work, spanning the first half of our time together today. And then I'll ask a few questions on my own before opening to audience questions. So with that introduction, please join me in welcoming our first presenting panelist, Dr. Catherine Alfano. Hello. All right, today I'm so thrilled to be able to talk to you about our MAVEN program, which is our senior scientist leadership program and its evidenced-based training. First of all, thank you to the last speaker for queuing this up. We're talking about diversity, equity, and inclusion. And you beautifully articulated the needs and the differences that people of color, women of color, face. I want to call out that it was my colleague, Dr. Felicia Hill-Briggs, who was asked to give this talk. Unfortunately, she was unable to be with us. She and I are both co-investigators on the MAVEN grant. But obviously, I am an Italian-American of white skin. She's an African-American. What we've come to, what we bring to that team are different, and I want to acknowledge that. So the mission of MAVEN, what we're trying to do, is really expand the national pool of qualified, underrepresented candidates for senior scientist positions. Across all areas of science, we know this is a problem. We want to help accelerate their advancement into leadership positions. The aims of this grant are to increase their career satisfaction, also increase their peak productivity, build their scientific networks, help them ascend to those leadership positions, and then help them transform their institutions to further DEI back home. So as Dan talked about just a moment ago, when we have a lot of universities across the country, and in fact, across the world right now, taking theory-based or best-guess sometimes approaches to, how do we fix this problem of DEI in science? And it's kind of like throwing spaghetti at the wall. And when you throw spaghetti at the wall, your impact is a bit like this. And again, Italian-American, every kid in my family has some baby picture that looks just like this. That's fine. But we can do better. So what we're doing with the MAVEN leadership experience is an RCT. And we are so excited to have gotten funding from the US National Institutes of Health, from NIGMS, the National Institute for General Medical Sciences. So we randomized, we scraped the public records for female scientists across the country who have at least one R01 or R01 equivalent funded by that particular institute, NIGMS. And they have to be at least 10 to 15 years from their terminal degree. So these are scientists who have established their street credibility out there. But they're at that point where they're emerging leaders. They're ready for a next step. And then we randomize them to what we tell them, more effort or less effort. But we all know as intervention versus control. Nobody likes the control group. So we call it the less effort group. We are enrolling four cohorts over the five years of our grant. So we enroll a new cohort the first four years of the grant. In fact, just last night, we kicked off cohort number three. And it'll be a total of 160 scientists that will be participating in this over the course of five years. So what are the components of this leadership RCT? So the intervention program gets all of that knowledge that we feel like has gotten passed down by the majority patriarchy in back rooms, men's rooms, backhand deals, things like that over time, right? Career planning, organizational know-how, mentoring, how you improve your academic productivity, how you build your scientific network, how you get on thought leadership panels like this, how you get board assignments. And so we're trying to pass down that information to teach them what they can't get through those back room conversations traditionally. So what they get are two summer institutes of didactics, which are virtual interactive workshops. They also get a lot of mentorship. They get formal training. They get peer mentoring pods, which I'll talk about in a minute, and the opportunity to do informational interviews with experts around the country in areas that are things that they want to build in their own leadership. They then, between Institute One or Year One and Institute Two or Year Two, take on an institutional change project, something that they're trying to do back home to improve DEI, to improve their own career, to improve their own leadership. I'll talk about that in a minute. And on top of all that, they get access to the Harvard Online Course Pack, where they have any number of self-directed leadership courses. So as I said, the primary outcome measure is career satisfaction. Because if we're doing anything in DEI, it should be improving equitably the access to a satisfying career. So that is why we chose career satisfaction as our primary outcome. In addition to that, we are scraping public records to look at whether their productivity increases, whether their professional network size increases by the type of title that they have in their job, whether they ascended to a higher leadership position. Some will, some won't. I frankly think if you look at that and decide it's not for me, that also will win. So we have those additional outcomes. These are just some examples of the topics that we teach, leadership essentials, negotiation, strategy building, values-based leadership. So leading from your values, how to improve your network, how to create change. They meet with an executive search firm and understand how to present themselves. We teach them how to be better mentors, how to create bidirectionally helpful mentoring mentee relationships. Scientific thought leadership, graceful, the art of graceful self-promotion, how to build your lab, many, many, many other things. So in addition, those are the didactics that we teach them in their two summer didactic programs. In addition to that, we have their institutional change projects, and when they tell us what they wanna take on for their institutional change project, we thematically group them into peer mentoring pods of people who are tackling related things. So maybe one group, for example, here are the last year's cohort, one group wanted to focus on improving the impact of their science, whether that was to industry, to patients or participants or community members, to potential funders, et cetera. Another group wanted to talk about how to build better productive collaborations and how to improve their leadership. And a third wanted to talk about negotiating and advocating for resources for them and for their junior people, how to build their lab, how to be better mentors. So we host these peer mentoring pods where we help them enact good, solid change management plans. We help them take big goals and break them down into small goals, making sure that what they're doing in terms of their institutional change project is bringing in all of the appropriate and available resources to them and is based in their strengths. I also said that they are able to take on one or two informational interviews. We ask them, what are types of experiences that you would like to have? Who are types of people that you would like to have access to in your scientific network, in your career development network? And we approach these people because all of us on the MAVEN leadership team, we have access to a lot of the leaders around the world, thank goodness. And we can just approach them and say, listen, are you willing to just talk to people for 45 minutes? And people have found that this is a really excellent way to, for example, explore whether, I wanna talk to a provost and see whether that may be something I wanna do, or I wanna talk to someone who's commercialized their science with a pharma company or some other industry to see if that's something I wanna do. And most of our scientists take advantage of those. We have, I can't give you the results of this program because of course we're smack dab in the middle of this RCT, but I can tell you that we do a yearly kind of evaluation of what our participants think of it and there we go. You can see of the 15 respondents that came back, people in general are really happy with this program. They feel like it reenergized them, especially coming out of the pandemic. The topics were relevant. It really helps to empower them to become leaders and expert faculty. I always like presenting some of their participants' own words, I'll let you read these. I like in particular the people who realized that, you know, I was waiting for other people to recognize me and now I understand I need to do a better job of promoting myself. We built a website, you can get it here. It's Maven, it's the Maven program. You can just Google that. I had the address on the bottom, but it seems to have fallen off. And finally I wanna end with what gets me up at every morning to do this, which is this wonderful quote by Amanda Gorman, our US poet laureate who says, it is how we empower others that makes our power so vital. So thank you, thank you to NIH for our funding. Thank you for the ability to be on this panel. Okay, so hello. My talk here is, I titled it, evidence-based DEI without RCTs, because I didn't quite know what to call it. It's basically somewhat of a bridge between Catherine and Jordy's talks to say something about what we can do when we don't or we can't have data from RCTs. So I'll talk about how mathematical modeling can help us to incorporate other kinds of data, data of the sort that Jordy will talk about into the evidence base for DEI relevant policies. So I'll only have time to give a couple examples from my own work of models playing this role, but I'll also give some quick general thoughts about how modeling can play a role in evidence-based DEI. Okay, so first, one thing that's concerning is that in many fields, there's lack of diversity within collaborations. So we see evidence of this, for example, when it comes to co-authorship patterns in many disciplines, specifically with regard to race and gender. And when it comes to diversity, we cared not just about the presence of diverse people, but actual engagement of ideas across social identity lines. So we might think how best ought we to promote diverse collaborations. And we might, for example, enact a policy where we give particular grants to people who are trying to put together a diverse team of researchers. On the face of this, it sounds like a totally worthwhile policy. It encourages diverse collaborations, and it rewards those who are making an effort to fix what we see as a problematic situation. However, models show that this also has the potential to backfire quite terribly. And the problem is, is that if you don't understand what's responsible for the lack of diversity, you're not able to predict how this policy plays out. And one reason for lack of diverse collaborations is an inequitable division of labor. So the dynamics of interpersonal interactions in scientific communities can lead to inequity in collaborations, and this can lead to partially segregated communities where there's not full engagement of ideas across social identity lines. But what this means is that a diversity initiative that incentivizes diverse collaborations serves to encourage people into those inequitable collaborations. And what's worse, in the long term, we don't fix the inequitable norms, and so we still have that pressure working against diversity. Now, my co-authors and I make this argument based on mathematical models, not actual interventions. These models don't give us new data regarding interventions, but they indicate what data is important for assessing interventions and why it's important. So for instance, evidence about how credit for joint work is distributed, personal experiences of members of marginalized groups and diverse collaborations, and the reasons people have for choosing their collaborations all matter to policies promoting diverse collaborations. And I'll note that this includes both quantitative and qualitative data, data we already have, and data that we ought to collect. This and other work showing DEI policy backfire is relevant to the business case for diversity and related argument strategies. So this business case might be a term that's unfamiliar to people, but likely the kinds of arguments that are given are familiar to many people here. They're arguments that we ought to increase diversity, promote diverse collaborations and so on, because inquiry will be more effective or some other ends will be achieved more readily. The issue is when we argue for diversity initiatives in this manner, while ignoring underlying social structures, it can lead to ineffective policies which fail to achieve the desired benefits. This isn't to say we shouldn't care about these benefits or we shouldn't make these arguments at all, but doing so while treating people as a means to an end or without paying attention to experiences of members of marginalized groups is likely to lead to policy backfire. Additionally, attempting to improve efficiency of science while ignoring background inequities can also backfire, decreasing efficiency and also increasing inequity. This means that DEI efforts are relevant to policies that aren't strictly speaking about DEI. So for instance, take proposals to switch to what's called crowdsourced peer review or post publication peer review, popular topic in meta science. These proposals differ on their details, but generally call for some sort of system where people post their paper online to an archive in effect publishing it and then peer review happens afterwards. So instead of journals assigning reviewers to a manuscript, people choose to review papers as they see fit and authors choose to respond as they see fit. So I'm not going to stand up here and argue whether this is a good idea in general. I just wanna point to one feature of these proposals or one possible outcome of these proposals, which is what Hayeson and Bright call a runaway Matthew effect, where the well positioned in the community are more visible, therefore their work has more impact and then they become more well positioned and so on. So Hayeson and Bright argue that there's no evidence that switching to post publication peer review will create runaway Matthew effects where there previously were none and therefore they dismiss this worry for their proposal as highly speculative. However, we can use mathematical models to show how runaway Matthew effects would be worse with crowdsourced peer review. And the idea is that unlike in the current peer review system, in order for a paper to receive post publication peer review, it must first be seen by others. This increases the importance of social positioning relative to our current system. We're publishing a paper in peer review journal that answers as a mechanism by which an academic can gain visibility or reputation that's at least much less dependent on your position in a community. So this I show worsens existing runaway Matthew effects by increasing the importance of certain network features that can amplify over time, creating a feedback loop whereby people on the peripheries are post further and further to the peripheries over time where their work is overlooked. Okay, so you might reasonably ask, well, you think there are going to be these policy backfires, why don't you just get some data, do a beta test or something, and then see whether these bad things happen. Well, with respect to the runaway Matthew effect, this would be hard to detect before fully implementing the policy. Beta testing limited to certain journals won't create that feedback loop because there will be other quality journals that academics can submit to and gain visibility. In this case, I think the model gives us what beta testing can't. More generally though, running experiments is expensive and we need some reason to take certain possibilities seriously. I think providing a model motivated by evidence with plausible assumptions about how people act gives us a reason to take these potential negative consequences seriously. While we might not need the models to come up with the possibilities, the models add something extra in terms of clearly demonstrating mechanisms by which a policy backfire can occur. So this also relates to what I said a few sides ago that we might not know all the evidence that's relevant to collect and models can give suggestions as to what effects to look out for. So mathematical modeling, I think, can help in developing a strong evidence base for the effectiveness and possible consequences of DEI relevant policies. Thanks. I'm so excited to be here to talk about the Patent Equity Project with all these wonderful panelists. And so the question that I come to ask is who invents? So this is Thomas Edison. I think everybody here might think of him as an inventor, certainly true, but that's not the answer that I want us to come to today, that one person is an inventor and that's because the work of Dennis Crouch and the United States Patent and Trademark Office and so many others show us that groups invent. The average number of inventors on a US patent since 2021 has been over three and has been over two since 1990. And so instead of thinking one person invents, we need to think of groups inventing. Well, we don't really show representation in groups yet. And so that's where my data begins. So what I did was me and a bunch of RAs who you will see at the end of this presentation collected three million patents filed from 2005 to 2022, which is basically all the ones that we were able to collect. And we gender matched them with a world gender named database 2.0, which means everything here is going to be done in binary. If anybody here has data to acknowledge more than just gender binary in this type of analysis, I would love to speak with you after this presentation. But what we did was we divided the results by size of an inventor group. And the way that we calculated the probability of a group appearing versus the group that actually appeared was kind of akin to rolling dice, right? If you wanna say that a baseline of a dice is one out of six, that the chance of rolling a one on one die is one out of six, but the chance of getting two ones is one out of 36. Could you multiply them together, right? You take the baseline, you multiply it the number of times in the group of dice, and then you get the answer of the probability. So I did that with women in the workforce. That is, I took the baseline, which I'll simplify for 20% for now. The likelihood of two female inventors appearing on a patent together in a group of two should be 4% of the time, 20% times 20%. For an all-male group, that's 64% of the time, 80% times 80%, and the rest you can figure out mathematically. And you can extrapolate this to basically any group size that you choose. And so what I did was what I'm calling the equity test that is calculating the percent difference between the baseline probability calculations and the actual representation of this type of group. And I used two baselines. The first baseline is what did the US census say was the female STEM representation of that year, 20%, 15%, 25%, whatever it was, that was my baseline, which you'll see first. And then at the end, I took a baseline of the named female inventors on the patents that year. So were the inventors distributed well once they got through all of that credit, all of that time, all that effort, all that inclusion? And those are the results you'll see today. Now, we're gonna be looking at a chart. So I wanna explain what this chart is. You're gonna see a bunch of these circles. We're not gonna think of these circles as a woman being better represented or not represented, right? That was the Thomas Edison thing. Cross it out, doesn't exist. We are thinking this as teams. So in this, it's a team of two women and two men. And this would be underrepresented because it's green and there's a negative. The negative represents, and the number represents the percent difference between the group baseline calculation and the actual representation of that group. The greater the pink, the greater the overrepresentation, the greater the green, the greater underrepresentation. Now, this is in 2005 to 2007. You can see the ratio of the women and the team on the vertical axis and the group size on the horizontal axis. Meaning that the ratio of women and the team zero, that's at the bottom, right, the all male teams, you can see in 2005, all male teams are overrepresented and any team that has a woman is underrepresented. That was expected. What's interesting though is we're gonna progress to the present and I want everybody to pay attention to the upper right hand corner relative to the middle portion, that middle portion being the mixed gender teams. And so we can see as we advance over and over, things are getting a little bit better but only for the groups of all women at the top, right? We can see that in greater groups of all women, we're seeing an overrepresentation. Actually very similar to that overrepresentation of the all male groups, but the mixed groups are stagnant, right? The mixed groups didn't really change very much. This really fits with the data of the number of groups that we're missing, right? If we're looking at underrepresentation, that group of all five women shouldn't statistically appear very often. And so that gap of overrepresentation or underrepresentation is important. But looking at the number of female inventors of a group of say five, the big gaps are in that middle. The group that men and women should be working together aren't appearing. And there are many reasons why this isn't happening. But it's something that we need to focus our DEI efforts on. And the reason for this is the trend. Now I simplified this to groups of three inventors. You can see the all male inventors at the top. And you can see this group of three female inventors is trending pretty well towards equity. That is the percent difference is trending towards zero and we haven't quite reached equity yet. But in 2026, that number is supposed to reach the equitable representation threshold. But what about those groups of one female inventor and two female inventors in groups of three? Well, those aren't hitting equity for quite a while. And I know I've taken some liberties here with the calculations, but we can see that this mixed group inventorship as Hannah was talking about is really something that we should be concentrating on. Now remember I said earlier about how I used two baselines. The first baseline was the women in STEM representation and the second one was are those inventors distributed well? I thought, I hoped that maybe if you had gotten through the process of getting your STEM degree and getting your PhD and getting into an industry and getting your name on a patent that maybe at least the inventorship was distributed equally, it wasn't. And this was what really concerned me. You can kind of see this diagonal. What I like calling it just to make a few people mad is the token group. So for the most part, it's the group of all men minus one. So all men is at the bottom, minus one, that's the group of women that is underrepresented. And you can see here that we're starting mostly with ones as a representation of a difference. So you can see a percent difference of maybe 12%, 13% over time. But if we continue with this underrepresentation of the token groups over time, not only can you see that more often, over and over and over again, these groups are underrepresented, but in fact this token group is getting more underrepresented over time. That's a problem because we're seeing that there's greater segregation and greater underrepresentation of a group that should be appearing over and over and over again. This could be that the group isn't forming. This could be that the group isn't as successful when it does form. It could mean that somebody is being left off of the patent credit at the end. And these are all different DEI initiatives, but they're certainly different DEI initiatives than what we see most often in terms of getting a representation of one person because this is a group effort. And because it's a group effort, I wanted to thank everybody who has helped me so far with this project. So I want to thank all three of our panelists for their presentations. Sorry about the technical difficulties. You can step up to the microphone if you have a question, but as folks are doing so, I want to ask a question about unintended consequences. So humans are famous for creating social systems where there are plenty of endogenous feedback loops. And so when you impose some kind of intervention or you're measuring some kind of dynamics, there may be some unintended consequences because of those sort of endogenous feedback loops. So what do we know about, or how do you think about those kinds of endogenous feedback loops or unintended consequences or setting up your study to measure those when they may arise? Katherine? Yeah, I'd love to start with that. Jordy, I want to call out your talk in a minute. So I think one of the first things we learned when we started implementing the MAVEN program is that we thought it would be a great idea. I mean, these are all women and women of color. And we thought it would be a great idea to tell their deans, hey, this is such a great prestigious thing that your person, your faculty member, has been invited to be in. And that completely backfired. These women said to us, it feels like we're being remediated. You telling our deans feels like we're being called out as being different. We hate that, and it just, and the other thing that I'm thinking about is, here we are, they also said, one of the things they didn't like about this program is that we're putting all of the emphasis on them to fix the problem. And it made me think about your token group, right? If I were to design the perfect, I think, intervention program, it would be a multi-level intervention where we'd be trying to help, yes, the identified token, right? But also we'd help everyone else to try to meet them in the middle and work together. And then we'd also be influencing the policy in which those people were interacting. And I think your research really speaks to the need to do that. Thanks so much, yeah, I think that that's right. I think that a lot of the conversations that I've had with women and with people of color when we're trying to look at inclusion is that they feel like a token. They feel like this prize that they're in a group and somehow they're the ones who are getting all of this DEI funding and they really kind of want that. They want recognition for what they've done, not for the color of their skin or their race or something like that. And so I think that work in groups really helps to say that it's not just you, it's everybody in that group who's underrepresented. You have a group of five, there's four men and one woman who is underrepresented in that group. So I think it's important. And something also that you had brought up is the DEI efforts of just talking about business modeling and making that economic argument for progress and something that we had discussed was that in the past and certainly even in my work I talk about the economic arguments but the more that I think about it, the more that I realize that if I emphasize the business aspects of why DEI is important, that you're gonna be making more money, that you're kind of going to have better inventions or something like that, the more I'm telling rich old white men that they can still build their businesses on the backs of women and people of color and be more economically advantageous, which doesn't change the system and infectious and trenches the system. So it's really important for even researchers who are trying for us to constantly learn about how these can improve because that was really helpful for me to learn about. Yeah, I mean, so when it comes, at least in my view, when it comes to things like the business case for the diversity or all these interventions, we might have to promote diversity, equity and inclusion. I think talking about the unintended and negative consequences is very important. But it's important also to remember that just because there's unintended and negative consequences, that doesn't mean that we give up or we don't do the thing or we can't ever talk about any benefits that come from diversity, equity and inclusion. It just means that we have to do so more carefully or we might adjust our intervention to make it so that these negative consequences don't happen. Question from the audience. Hi, Jessica Polka from ASAP Bio. This question is for Hannah and I think building off of what you just said about how to avoid unintended consequences, we see a lot of benefits for open pre-publication peer review. I'm curious if in your modeling, you have identified any ways to modify the interventions to prevent these runaway Matthew effects, which of course is not what we want, but is there some way to change the visibility of papers or who is shown what, et cetera? Yeah, so this is one thing that, so hasten and bright, I reference them partially because they're philosophers and so I've engaged with them a lot. This is one thing that they feel quite confident that we can just adjust the way that this post-publication peer review is done to prevent the runaway Matthew effect. I'm not an expert on post-publication peer review. I do think there might be certain things that you might be hopeful would work, like promoting work from underrepresented groups. The one hesitancy I have is that when you have feedback loops like this, they are constant pressures working in a direction. If you enact a policy or some reform that creates a feedback loop, you're going to constantly have to be working against it and so you have to be working against it from the start and you have to be collecting the right data to make sure that what you're doing is actually preventing the runaway Matthew effect and things like that. While I think there might be, it seems plausible that there's something we could do, it needs to be done very carefully, I think. So I wanted to ask a question about DEI as like three things, so diversity, equity and inclusion and depending on which acronym you choose, there may be like a fourth or maybe even a fifth thing in there, maybe justice for instance. I was wondering if folks think in medicine research if we treat those three or four equally and if we overemphasize one, if we tend to prefer to study one because it's easier to measure or easier to somehow quantify and therefore graph. What do you think about the sort of biases with which we approach what we study in this area? So I think that it's important to first define what we're talking about in terms of DEI, B, and J. So diversity is you pick kind of a race or gender or some other variable and you can quantify the number of people who are women or men or as I've talked about how if I want to find people who identify as non-binary, that might be more difficult to quantify or to gather data but the data can be there and it's certainly there in a quantifiable sense. Equity is whether each of those people once you've picked that variable has an equal chance of succeeding in whatever success metric you choose, whether that's publications or promotion or money or something like that. Career satisfaction. And career satisfaction, exactly. And inclusion though is less quantifiable in terms of numbers and more qualitative research and I think it's an equally important thing to look at although some of the research might be overlooked and that's really the feeling that you have now included this person into the group, into the establishment and that person feels like they belong. Now some people will divide up inclusion generally of the majority group including somebody in a minoritized group and belonging that is the feeling of the minoritized individual into the group that would be kind of different and then others will talk about justice that is should we have equity as the reward or as the ultimate goal or should we be giving more people more interventions in order to make sure that eventually they will catch up if something is kind of a problem. So this would be anything from extra support for nursing mothers to extra math help if somebody comes to college and doesn't have the basic math structure to succeed but is otherwise considered a capable student. Those are kinds of questions that we have in the DEI, B&J world. Now I know that numbers seem to be more persuasive than what people will call anecdotes but as Ray Wohlfinger has said and George Stigler and many others the plural of anecdote is data and at one point we're going to need to start listening to these voices and the wonderful anthropologists and sociologists and people who are collecting this data which should be elevated when the quantitative data fails or can't quite capture that. I'd like to add anything. To the audience. I'm Diana Cardia and first of all I just want to thank you. I got my own doctorate in higher ed and it's the discipline that should be doing the work that you're doing and it's not. It's still just describing the problems and I am so grateful to the three of you who spoke and even you as a moderator. The work that you're doing is so vital. I do work to make change in academia too and I want to describe just an experience I had and see what you have to say about it. I was called into a department that did a climate study that revealed gender. This is a STEM department that had high representation of women but the climate study to everybody's surprise there's a gender issue in this department and I started meeting with the faculty and what the women said to me they could tell me every detail but they were very clear. They could not say it out loud in that department even to each other. So my question to you is the work that each of you are doing how, what do you know about us getting past that survival silencing of the conversation? I'll take a stab at that. So in our MAVEN groups I talked about how one of the components of the intervention is that they take on, our participants take on an institutional change project and we lump them into peer mentoring groups. I actually run those groups, I facilitate those groups and I hear so many stories like what you just said. I hear stories like certain states right now in the United States that you're not allowed to use the words diversity, equity or inclusion. You might want to create policies to improve those things but you can't even say those words out loud anywhere on campus right now. So how are you supposed to do what you're doing? So to answer your question, how some of the ways that we've talked about that in the groups that I facilitate are it has to be more than that person's voice. It has to be more than one person trying to make this change. So how can you bring in supports? How can you bring in resources? How can you find the right provost? I used to work in DC and the health policy front. We often said in health policy that making change in a policy level is about telling the right story to the right person with the right data. So how can you collect the anecdotes that become the data that make that a compelling voice and speak with, as you say, not one voice that maybe isn't even unspoken but speak with a megaphone. So part of what we try to do is help these participants amplify their voices so that they can speak with that megaphone and feel comfortable in doing so but it's an uphill climb right now and especially in some states where you're not allowed to utter these words. That feels like a broader issue related to quantification, right? There are certain variables that we feel are easier to measure, easier to quantify and yet it seems like it's very important that we design our studies so that we're sort of open to receiving this kind of feedback or this kind of information. Like that interview seems like it's incredibly valuable whereas measuring the composition of the department by a gender at a high level is not going to be. Yes? Thank you. Again, very appreciative of the work that you're doing. I'm involved in some DEI work back in Edinburgh and one of the issues that we often come across is around intersectionality. I just wanted to get your thoughts on, I guess, I don't know if this is a hard question but the best way to do it or are there some things that we should be doing that we're not doing to really bring intersectionality into DEI work because we often just focus on women or we focus on people of colour or we focus on people with disabilities but obviously intersections affect how you engage in surviving in this environment. So have you got any thoughts on that? Intersectional work is incredibly important. It's something that I'm working to incorporate into my research. It's difficult in the United States sometimes to collect some of the intersectional data and conversations about race differ in every country in terms of how you would categorize somebody of a certain race. So, for example, a project that I'm working on with Paul Gugluza looks at lawyers and their representation and he did an entire gender sample but now we were looking at trying to identify attorneys by race and to see if people who are attorneys of colour and women have more of a negative effect than if they are women or people of colour and looking at kind of that intersection. But to do that we couldn't collect the data on somebody's self-identified race because one that requires IRB approval to a lot of people didn't want to respond to that and three, we didn't want to alienate a very small population and so what we did was we used the work of some sociologists which was collect everybody's picture and we asked a racially diverse group of research assistants to identify how would they perceive somebody? How would they perceive them by gender? How would they perceive them by race? And in so doing with this type of intersectional work we're looking not only at whether somebody identifies what people's identification might be but also how society might identify them. This doesn't include disability work. It doesn't include areas of the world that they might have grown up in. It doesn't include educational background. There's a lot of things that we're missing from the sample which then leads to... If we included all of these, the trouble is we then get statistically insignificant results which I think is another reason why qualitative data would certainly help bolster the analysis that we can do with the limited amount of quantification data that we can collect at this time. Yeah, so when I, I guess, Jordy knows more about the technicalities of data collection but a lot of the studies I read, so for instance when we're talking about citations and who gets more citations for their work based on race or gender, a lot of the studies that look at race specifically can't get enough data points in order to separate out different categories. So oftentimes it's just white versus non-white which is, it says something but it's not necessarily what you thought you wanted going into the study. And I mean it's not the fault of the researchers, that's just the methods they use demand more data. So yeah, I think this is, and then of course when you get into intersectionality, the groups become even smaller and the data becomes even harder to get. So like Jordy said, I think this is a place where we can switch to more qualitative understandings but also a place where mathematical models can be used to kind of complement those qualitative understandings. So take some things that we know based on that research about how people act, about how people perceive their interpersonal interactions and say, okay now how does that figure into the community structure or these sort of structural effects that we observe? So show, use the model to kind of bridge the gap a bit and show how these effects scale up into like what we observe in scientific communities. One other thing to add to that, I think I'm aware that the US National Institutes of Healthy Office of Portfolio Analysis at NIH here in town did an interesting set of papers looking at, okay if we look at researchers' productivity, their ability to get grants funded or their ability to get papers out, is it really that there are differences by gender? Is it really that there are differences by race ethnicity or is it topic choice? And I point you to that work because I think there's some fascinating data coming out of that world, that I think we are seeing some differences by people who are from marginalized groups or people or women tend to take on different topics in their research and those topics may be less funded or not as well funded or differentially funded. Anyway, there's some very interesting intersections coming out of that world as well. Final question for our panelists. Thank you so much, Muhammad Hussaini from Northwestern University. I was wondering what you think about reactions to some of the DEI policies that are implemented and the reason why I'm asking this question is because we did this study in Ireland where they have a top-down approach. They have this program called SALI, Senior Academic Leadership Initiative where they said, okay, next four years we have 45 position, women only, in the beginning it was women only and then they said, oh, we will also include other underrepresented groups and then to your point about anecdotes and narratives, I interviewed 16 women from my own university and I asked them, what do you think about this? What do you think about SALI? The response was, I think it's great. Okay, would you be happy to get one of those? Out of 16, only five said yes, I want to be a recipient of a SALI position. And the reason why there was that sort of sentiment towards getting one of those was because they thought they would be looked down on, not just by men, but also by other women who thought they have elbowed their way through the ranks and now there is this new person who's just almost, not that this is the case, not that they have no merits, but in comparison with the other people who had to elbow their way through, they are almost like given this opportunity that wasn't there at the time. How do you think we can deal with this culture? Yeah, I mean, this is one of the biggest questions for our time and another aspect of what you're talking about is, at least in the US right now, the participants in our MAVEN groups are telling me that in the basic sciences right now, if you're a person of color, especially if you're a woman of color, you're getting something like a $3 million startup package and no one in that department got anywhere near that when they came in and that's creating some real animosity. So talking to your first question about unanticipated consequences, I don't have the answers to this, right? I would love to hear what you all think of this. This is one of the biggest challenges of our time. I think the same way that people don't want to hear that they can't succeed because of things outside of their control, like where they were born or how much money they were born into or the color of their skin or their gender, people also don't want to hear that the only reason why they succeeded was also because of circumstances outside of their control. People certainly want to live in a meritocracy and I think that that's important in academia to feel like you earned the position that you got into and I think that it's also important to acknowledge that there are implicit and explicit biases external to women, to people of color, to people with disabilities in academia that are preventing them from reaching those positions. So if the reason why somebody is getting a Sally position is simply because they're putting a thumb on the scale that helps counteract those biases and I would encourage everybody in that university to understand what they're doing but if it's explicitly the only reason why you're getting this is because we made a mistake in the past and you're here at the right time, then I completely understand why somebody doesn't want that position because they want to be told not only are you getting this because we need better representation and because we'll help the next generation but also because you absolutely deserve it just as much as everybody else in this community. Do you think there's any ways of creating that culture? How to create that culture? Good question. I hope a lot of people here can talk about their own personal experiences for creating that type of culture in their own universities. I think a lot of these are just, it's just different for each area of the world, for people who are there. I think it takes leaders though. It takes leaders at the table who are trying to set those policies and have those conversations and set those expectations of being in a meritocracy. Yeah, absolutely. Well, it sounds like we have our work cut out for us. Definitely. Please join me in thanking our panelists for their remarks.