 So, my name is Molly. I'm an MD-PhD student at Oregon Health and Science University, and I'm going to be talking about using research and technology to tackle gender bias. So I actually want to start with disclosures, because I'm a really early learner when it comes to programming, so there was a certain level of intimidation about coming here today and talking to everyone. But I'm happy to, if you have questions about some of the stuff that we're doing, plug you in with people who are way above my level of learning. And I'm trying. I'm trying to keep up and learn all the things. So I wanted to start with the way I think all of us enter school. For me it was medical school my first, you know, first couple of weeks and you enter excited. You enter like ready and jazzed. You're entering academic medicine. It's gonna be great. And then you go to a lecture by one of the professors that you most admire and she talks about salary gaps in gender and how you're gonna get a degree but not make as much money. And then she talks about you're also not going to become a professor because the whole leaky pipeline is a thing. So it's gonna be harder and longer for you. And then she shows the leaky pipeline. So you might also not even stay in medicine. You may end up, you may never become a dean. Women are less likely to sort of make it all the way to the top. And then she does my favorite study of all time, which is where someone looked and said, are there more women in leadership positions in medical schools or are there more men with mustaches? It's called the mustache index. And unfortunately, as you might surmise by the fact that this got published, there are frequently more men in leadership with mustaches than women. A similar study was done in business and they looked at men named John versus women. You might, again, similar outcome if your name is John, you're more likely to be in power than if you're a woman. So as part of this talk, she talked about letters of recommendation and how letters of recommendation and evaluations influence all sorts of different aspects of a woman's career trajectory, grading, promotion, hiring, tenure-track discussions, and how that could potentially influence part of the issue that we see with the leaky pipeline. Yeah, I was less excited about academic medicine at this point, you know? Like, suddenly I'm like, what have I gotten into? And so I started thinking, what do we do? And so I was thinking about this letters of recommendation thing. And then I was thinking about, like, well, I know Excel. And one of the things I know about Excel is you can use if then statements. And I know people have looked into this because she was quoting literature in her talk. So I'm going to look into the literature. So what does the literature say? And what's been done? So I decided to think about it in terms of rules. And that's one of the great things about qualitative research is they kind of go through, they look at themes, they look at what comes out of the data. So I thought about it in terms, what are the rules? What are the if-then statements that we can make about gender bias as it relates to letters of recommendation and evaluation? And I looked at as much of the literature I could, medicine, science, there was a great paper in geology, business, computing, as you might imagine, this is not an issue unique to the world of medicine. So I found that letters of recommendation for these rules are more likely to include minimal assurance. She's competent. She can do the job versus she's excellent. She excels in this area. She's one of the best people I've ever worked with. They're more likely to highlight effort. She's hard working rather than accomplishment. So they're not talking about what you've accomplished with your work. They're talking about the fact that you work really hard. Well, you can read that as this person's hard working and potentially not getting anything done. And then they they discuss personal life. So family, children, things that you normally wouldn't think to include in letters of recommendation or evaluations per se. They'll often lack formal titles. And they'll include gender stereotypes or motion focused words. So some of the literature talks about this idea about communal versus agenteic language. So for for gentlemen, usually it'll be they're independent, they're self motivated, they're driven. And for women, they're compassionate, they have empathy, they're a great team player, they're interpersonal. And this really has an impact when you're thinking about hiring. Because a lot of times when you're thinking about who you're promoting, you're looking for self directed, independent, and team player might be great, but not necessarily if you're hiring for leadership. And then finally, they raised doubt. And this is sort of that, you know, they're they're competent. Well, why are you only saying that? And they're often shorter. So I looked at what's out there. And there are some gender bias calculators. I reached out to text IO because I was like, they're doing this for hiring. They're doing this for job searches. They'd be really super willing to help. They weren't. And the Hemingway app sort of does looks at complex sentences. So I was like, there is a way that we can do this with text analysis. This is feasible. And then I was like, well, there are other rules too, that these text analysis programs aren't going to capture because they're highlighting what's there. They're highlighting the presence of female associated words or motion associated words. And they're not highlighting things that are missing. And the rule the the letters for written for women and evaluations, they're less likely to mention publications. They're less likely to talk about projects and research. They're less likely to include superlatives. And to use nouns. So, you know, she, you know, she what was she taught, and he was a he was an excellent teacher. So this idea of a noun versus a verb. And then finally, they they're less likely to include this repetition throughout the letter of positive qualities. So my background, I was a theater major. And one of the things I loved about theater is this concept of a penter pause because emotion takes place in the middle of things. Emotion often doesn't take place only on our words. So he puts in pauses so that we can have space for what needs to be felt. And I also did work briefly in social justice when I was in New York. And so one of the things we talked about there is whose voice is heard? And who are we not hearing from? And how do we represent and bring those voices in? And then finally, I have dyslexia. So when I look at a page of text, I see the white space instead of the black space makes coding super interesting. Let me tell you. And so I'm really fascinated by this idea of space and silence. And what that means for what's missing from the letters, which really gets at the heart of implicit bias and how that impacts everybody. So how do we make this happen? How do we take all those these ideas to and put them together? And what ultimately do we want to create? So I dove in. I learned Python. I took a like an intro class to Python and started learning HTML and did a crash course on GitHub and got involved with Mozilla Open Leaders, which is incredible. I got a mentor. And then I ended up on the pool side because very quickly to do what I wanted to accomplish, we needed natural language processing and lint and APIs and view and the work that goes into developing a plugin. And it got to the point that I could read the code, but probably couldn't totally understand what was happening. And that's where the value of a team really, really comes in because a lot of people in the open source community came in and are helping me make this happen. And I wanted to highlight them here, because otherwise, this never would have happened. And I want to show you where we are right now. So I'm going to have to pop out of this for a second. Oh, it's not going to let me. Sorry, did not expect. Great, everything just went black. So all right, I'm going to keep talking and see if it stops being black. So one of the things, okay, it's thinking great. See if I can show you. So here's what we have so far, you paste in your letter of recommendation. So example of the letter of recommendation, you're going to press submit. And right now it's going to underline violations of the rules that I went over before. So what and then it's going to tell you how you violated that rule. So for example, willingness was violated. So it gives you feedback on that. So what's interesting to me and is what we're working, I know it's well, I can talk through it, and then it'll come back on hopefully check is what the summary statement, right? So we're going to we're also looking for things that are missing. So how do we get that information to people? So right now what we're working on is creating a summary statement so we can talk about missing superlatives, missing publications, missing projects and address that part of it. And one of the things that was interesting is I've done some beta testing with it. We've started some beta testing with it. And they're these great blended rules where it's like, yes, they have superlatives, but it's the best woman, as opposed to being, you know, the best person I've worked with. So we've been finessing it. The other one that came out is the issue with sort of nouns versus adjectives. I did evaluations for med students. You can be patient or you can write a patient history. And so we're trying to tease apart and add that context into the tool as well. So it'll be more useful for people. And digging into the summary statement, I put a lot of thought into that. What does feedback look like? How should feedback work? We know that if you want behavior change, it's got to be positive reinforcement. So we need to say, hey, you did this stuff great. You included superlatives. You had this. You included a lot of statements about research and publications. Great work. Here are things to consider changing to improve the letter. The other thing is I really I wanted to be frequency based because I ultimately I don't want to take all the letters written for women and turn them into letters that sound like they're written for men. We should be valuing and considering the value of emotion focused words of communal language of interpersonal skills and empathy. And a lot of the things we might not think about in hiring and promotion, but it's still really important. And so ultimately the idea is to say you have more of these communal words consider adding agentic or like leadership or sort of more independence focused words from this list instead of deleting really good content content that's there. And my hope going forward once we have the tool is to find out if people will use it. Can the tool change behavior? What level of specificity does it need? Do we need something different for applications? Something for promotion and hiring letters of recommendation or evaluations? And does it need to be specific for fields, right? What differences are going to be there for a letter for a teacher or a professor versus a letter for someone in engineering? And then also going to help with personal statements because I for one tend to not follow any of the rules about good writing and personal statements because I don't want to like highlight that. And can we develop rules for other types of bias? That's where I really want to go next. What are the other types of implicit bias that we can look, make rules for and add in? And I started with rule based instead of statistical based because it's what I knew. It's what I was sort of the one that I could the easiest one to start with. But I would love eventually to get to the more statistical sort of learning based approaches to this. So I told the story in a really specific way today. And the reason I did that is because this idea came because I went to a talk and sitting you're sitting together in a room with different people who have different lived experiences and different skills. And there are there's amazing power in these moments. I also want to talk about the idea. I wanted to highlight the idea that I didn't have any expertise. I was new to academic medicine. I had done zero programming. And I still could find people to support me and make this happen. So don't limit yourself by your expertise. Don't think I don't know how to do that. It's not my field. Therefore I can't. And finally bring your creativity to problem solving. One of the reasons I thought about the absence of things I think is because of my history with dyslexia. And sort of my own experiences in different communities. And so I want to encourage all of you to sort of take this as we go forward. And I wanted to leave time for questions. So thank you so much.