 So my name is Ethel Azar, I'm the, I'm the project leader with Humanising Machine Intelligence Crank Challenge and I'm delighted to welcome you to the third H&I Day seminar. I'd like to start by acknowledging the fact that I'm, we live and work on, well, I live and work on Narago lands and I'd like to pay my respects to Elders past, present and future and acknowledge the traditional custodians of land throughout Australia. So today we have Ria de Atiago who has recently finished a PhD at the Carnegie Mellon and who's just started a job at the University of Texas at Austin as an assistant professor in the Information Risk and Operation Management School. So Maria is going to be talking about semantic representations. Maria, as we've only got an hour, it'd be great if we get started. We'll do the usual about half an hour for a talk then we'll have the Q and A and then we'll shift over to the slack. So try to go ahead and share your slides, that'd be great. Thank you so much for having me. I'll share my slides now. You should be seeing my slides. Yes. Yeah, perfect. Okay, well, thanks everyone for the invitation. It's great to be here, especially following the two fantastic talks by Claire and Lili. And today I'll be telling you about my work on semantic representation bias. This is work that was done in collaboration with people at Microsoft Research and at LinkedIn. In recent years, we have seen an increase in the use of machine learning to assist experts in high-stakes decisions. Examples of this include healthcare, child abuse hotlines, hiring decisions. And a lot of this work has been motivated by research that showed that machines were better than humans at making predictions. But these results often make simplifying assumptions about the task in hand and the data available. And in deployment context, several risks have been highlighted. In her book, Automating and Equality, Virginia events discuss how predictive models can further marginalize communities that have been disadvantaged in the past. And two examples of these that we have seen in the media include Amazon's automated recruiting tool, which was scrapped after showing bias against women. And what was happening here was presumably that the algorithm had been trained with data of historical hiring patterns. And so bias says that discriminated women were being reproduced. And in another piece, we saw how on a healthcare algorithm, Overmiguer and his co-authors published a paper showing how the medical algorithm was showing bias against blacks. And the nature of the problem here was a little bit different, stemmed from omitted pay of bias in which the label that was being optimized for did not match the label that they cared about perfectly. And that led to the bias. So they were using medical costs as a proxy for medical conditions. And one of my research focuses on the question at the center of my work is what are the risks and the opportunities of using machine learning for decision support? And as part of this question, I consider the nature of the labels available for training, including issues of omitted pay of bias, the selected labels problem. I also consider issues of human-computer interaction components that affect how predictions ultimately impact decisions. But the part that I'll be focusing on on today's talk is the data representation and the machine prediction. And in particular, how societal biases can be reproduced and amplified in this stage of the pipeline. So one domain in particular where we've seen a drastic increase in the use of machine learning is human resources where machine learning is being used as part of hiring and recruiting tools. And here, the question that drove our research was, can we quantify the risks of gender bias in automated recruiting? And can we mitigate these bias? And as I mentioned before, this was research that I did. Well, I started while I was in training at Microsoft Research. And it was also in collaboration with researchers at LinkedIn. And to summarize the findings that we'll be discussing today, our first compounding gender imbalance effect in a large-scale study that we conducted. And here, we show a general result of how this compounding effect can happen in supervised learning. And moreover, we propose methodology to leverage biases present in word embeddings to mitigate biases without access to protected attributes. So let's assume if we had a machine learning algorithm that was going to help us recruit, what would we like that to look like? So for example, if we're a recruiter who's looking for a computer programmer, we may want an algorithm that is able to pick up on the fact that a candidate is a software engineer and has experience in Java and Python and so on. Of course, we wouldn't, for example, want the fact that their name is Jane to affect the probability that we identify this person as a computer programmer. The challenge here is that even if we obfuscate information that explicitly allows us to infer these, there are other elements of the information that is included often on structure form that can leave the algorithm to infer information about the demographics. So for example, the fact that she was software team captain at a historically black college can leave the algorithm to infer that she's a black woman. And we would not want this fact to influence the probability that we see this person as a potential candidate. Increasingly, our online professional presence has become more and more important and recruiters invest vast amounts of time navigating through our personal websites and so on to determine what are our patients, what are our qualifications and who they should target. And presumably, this is a task that could be made easier with machine learning. These, by the way, are all of the co-authors of this work. And so what we said was, well, in this unstructured data scenario, what would be, because it's not really easy to access data of one of the companies that are actually using for recruiting, we wanted to construct a data set that would allow us to simulate what would be the risks of deploying a machine learning algorithm to do something like determining who to target with a job advertisement based on their occupation. So to study these, we collected 400,000 online biographies through Common Crawl. And the structure of the biographies were biographies written in third person. So the pronouns allowed us to extract gender. Here, it's important to note the limitation that by extracting the gender through pronouns, these results in a simplified binary notion of gender that fails to capture important aspects of gender and the races and the represents part of what we want to study. Here, so we're using the third person pronouns to extract the gender. We have the title from the Bureau of Labor Statistics. So in this biography, for example, what we would want to do is to, from the text that is highlighted in orange, we want to predict these occupations. So we want from text in orange to predict that this person is a professor. So the machine learning task that we end up having is a classification problem where we want to classify the title out of 28 potential categories from the text in the biography. And the pipeline then is we have as input data, the biographies. We have, we've scored different semantic representations. So we have a bag of words with a logistic regression or an embedding with a logistic regression or embedding with a big neural network. And this is because we wanted to understand how different complexities of the algorithm would play out the bias differently. And in particular, there's a very big difference between using a bar of words and a word embedding. And it's that the word embedding is also importing bias from other sources, right? The word embedding has been pre-trained in another corpus and so there's a potential of another source of bias. And the objective is to predict the occupation. So this was my biography at the time. And so if we fill my biography into the deep neural network, it predicts that I'm a teacher. And this is a network with attention. So the words that are highlighted in red are the ones that it pays the most attention to. And if motivated by the research of Bertrand and Lenathan who studied differences in callback rates when names were buried in CVs so that the race that was inferred would be different. And they studied how callback rates when DCVs were sent out of deferred. We wanted to see how the algorithms prediction would vary if we change gender indicators. And here an important thing to notice that this is not a counterfactual analysis of what would have happened if the person changed gender in which case like all of the features would change is just looking at the effect of the gender pronouns by themselves. And so in my case, for example, if I were using male gender pronouns it would predict that I'm a software engineer. And this is something that we saw at scale. So here what we're seeing are the most common swaps in which the algorithm gets the classification correctly if we used opposite gender pronouns but got it incorrectly with the gender pronouns that were used that were originally in the biography. The reason why we wanted to do these analysis in particular is because there's a hypothesis of saying, well, maybe it's just that women professors are just writing biographies that are impossible to infer that there are professors. So the question there is, well, if they were just using different gender indicators how would this change? And these are the patterns that we're seeing here. Of course, looking just at gender pronouns, first of all, well, maybe the answer is we should just scrub the gender pronouns, right? And then we get rid of the problem. So we want to look at the bias beyond gender pronouns and look at what is happening. And here what we're looking at is in the X axis we have the percentage of women in the occupation in the data that was used for training. So for example, the way to interpret this is surgeon, for example, here is approximately 15% women in the data set. And in the Y axis, we're looking at the gap in accuracy. So if there was no gaps in accuracy and the algorithm was equally accurate for both men and women across all occupations, what we should be seeing is that these all lie on a horizontal line. What we're seeing instead is that there's a correlation. And what this correlation shows us is that the algorithm is more accurate for women in occupations that are predominantly female, more accurate for men in occupations that are predominantly male. And this is not surprising from a statistical perspective, of course, the algorithm is just inferring patterns and saying, well, if I learned that women are more likely to be nurses and surgeons and I infer that these biographies of a woman, I'm gonna say it's more likely that they're a nurse, for example. So the question is, why does this matter? And from all of the metrics that we could be considering as algorithmic furnace metrics, why is this one one that is concerning? And here, and this is not the formal formulation of the theorem, but in the paper you can find the formal formulation. If the fraction of women in occupation is less than 50% then and the gender gap is negative, meaning the algorithm is more accurate for men than for women in that occupation or in that class, this is a general result, not just for this domain, then their fraction of female in the true positives in those that are correctly identified by the algorithm as being part of that class will be less than P, less than it was in the initial data set. And what this means, this is something that we can connect to the notion of compounding injustice in political philosophy. So what is happening here that if the initial imbalance constitutes an injustice, then what the model's prediction is doing is the model prediction is informed by and compounds these previous injustice. And so let's look at what this compounding in balance effect looks like exactly. So as I mentioned before, in our algorithm approximately 15%, a little bit less than 15% of surgeons are women and the algorithm identifies 71% of male surgeons correctly as surgeons, while only 54% of female surgeons are surgeons. And so as a result among true positives, the percentage of women drops to 11.6%. And so again, why does this matter? Well, if we were using these for a task such as determining who to target with an ad for a job, then what this would mean is that those that were targeting correctly, so those for whom the ad is actually relevant have a further underrepresentation of the group that was already underrepresented in that occupation. And this is something that we find across the different representations and both with and without the gender pronouns. So what we're looking at in these plots is the same plot that I showed before ex-access percentage of women in the occupation, why access the gap in accuracy, the gender gap. And what we're seeing here in blue is what happens when we use all of the biographies, so including gender pronouns and so on. And in green, we see what happens when we drop explicit gender indicators. So he, she, Mr. Miss, and so on. And there's two key things to take away from these results. The first one is that if you look at that curiosity plots we see that we actually have substantial gains in terms of dropping the compounding in balance effect by removing these explicit gender indicators without having lost in accuracy. And that is a topic that is a theme that we'll come back to later in the talk. But additionally we see that this compounding effect remains and so this is something, so all of the details if you want to look more are in the paper including some simulation in which we discuss how if the algorithm is encountered multiple times, then you end up having this phenomenon of kind of like a leaky pipeline in which at each iteration of encountering an algorithm that has this property then a group becomes further underrepresented. But this phenomenon we can relate it to what happens in orchestra auditions. So in orchestra auditions at some point add some way to trying to mitigate the biases that played into the judges assessment of musicians. They said, well, maybe we can ask musicians to play behind a curtain and that way the judges won't be seeing the musician and they'll just be judging the music. And what they found was that this response was insufficient to counter gender bias because they were still able to hear the shoes of the people playing. And so we can see these as analogous of the algorithm even though we can drop these explicit gender indicators the algorithm is still able to kind of hear the shoes and infer these information. And again the reason why this compounding effect matters is because of why this gap in actress matters is because of this compounding effect. So the question then becomes can we mitigate the compounding balance effect in a way that doesn't assume access to protected attributes? And this is important because these are attributes that are often unavailable or illegal to use under certain jurisdictions for certain applications regardless of whether we're trying to incorporate these attributes in the data for the purpose of mitigating biases. And additional, we want to account for intersectionality and here we want to account for two things. First of all, the phenomenon that we analyze on gender is something that we were able to analyze in this data set by using these proxy of extracting gender indicators but this phenomenon and what is giving rise to this compounding effect is likely to be happening on other demographics that we cannot test for like for example, rates. And in addition to this intersection of these attributes and how groups at the intersection may be affected in specific ways that are as different as Lilly mentioned in the talk last week is different to just considering like one access and the other access and that same. So how can we go about doing this? In previous work that we did, we looked at biases in word embeddings. And so previous work had showed some word embeddings are semantic representations of text. Basically each word has a vector representation and then proximity between words in that vector space is meant to capture semantic representation a semantic similarity along other properties of word embeddings. And so there has been research that showed that word embeddings contain biases for example, gender bias. And in this work, what we did, this work that we presented at AIS, we presented an algorithm to enumerate potential biases in a word embedding in a non-supervised way. And we showed that widely used word embeddings contain harmful biases associated to people's names. And through crowdsourcing, we also showed that these are lined with societal stereotypes. So for example, cluster of Latinx names were associated in widely used word embeddings with words such as drug trafficker, cartel, undocumented and illegal immigrants. So the question there becomes if we know that word embeddings contain these biases, can we use bias to fight bias? And so the underlying idea of the work that we presented at NACL last year is captured in these fraction of Shakespeare's from Anjulia that is what's in a name. So here, for example, the probability that we correctly classify someone as an engineer should not depend on where their name lies in the word embedding. That's another way of putting it. And so one way that we can go about preventing this is by directly penalizing a correlation between errors that the algorithm in classification is making and the location of the name in the word embedding. And we do this during training and then during these have the advantage that during deployment or testing, you're not using the name for prediction. So you don't need access to the name. And furthermore, an important thing of this is that this doesn't rely on whether the benefits of mitigating biases and the benefits of mitigating biases that the algorithm brings for a given individual don't depend on whether the demographics of that individual are well-proxied by their name. Meaning what we're doing is during training, we're nudging the algorithm to not focus on signals that are predictive of the names that lead to these parties in the classification errors. And then during testing, those signals are no longer used. That's like the underlying idea. And so those games extend beyond to those individuals whose demographics are poorly proxied by their name. So for example, a woman named Alex would not, like the fact that we can infer from their name, their gender is not a prerequisite for the games to be had. So how do we do this? So typically we minimize a loss function L. And so we propose to regularize the accuracy gaps in two ways. So two loss functions that we propose is first add cluster constraint loss. And so here basically what we're doing is we have clusters of names in the word embedding and we penalize whenever different clusters have different, there's differences in the error rates in different clusters. So for example, if a certain classification leads you to be very good at classifying engineers in one cluster of names and not in the other one, that is penalized. An alternative is to penalize these directly and we do this through the covariance constraint loss. And so here we're simply directly penalizing when the location of the name in the word embedding correlates with the probability that we correctly predict these individuals, the individual occupation, for example. And here this is something that, these are the results on a semi-synthetic data that we created using the UCI adult data set. So here the task is to predict income and you have the demographic attributes. So we use census data to create semi-synthetic names and look at how the algorithm would work. And we also applied it on the bias and bias data set and those results are in the paper. But here what is important to note that is a result that you have throughout different fairness and forcing algorithm is that the way to read this by the way is the X-axis is showing us the gender gap or the race gap in the way that I defined before. So this is basically like the differences in accuracy. And in the Y-axis we have the balance, accuracy. The balance is just capturing the fact that there's, the classes may be in balance in terms of professors. It's a much bigger class than rappers, for example, in this data set. And here the unconstrained, so if we don't enforce any constraint, we are in the top left for me, of the plod and as we enforce a larger constraint we see how this progresses. And this curve, the way that these accuracy fairness trade of curves look like is actually quite general to fairness enforcing algorithms. And the important thing to note here, oh, it's not circle A, is that these phenomenon here in which we see how we have gains in mitigating the gap without a cost in accuracy is actually quite general. And the same that we saw before in the other results. And so the important thing here is that we need to explicitly encode these values in the models because while there's not a cost in accuracy, it won't happen by chance. And well, all the details of this paper, of this work are in the paper and I'm also happy to discuss more in the Q&A and over Slack. And to summarize, we characterize the risk of compounding injustices in supervised learning and through a large scale study of automated recruiting we have showed both these compounding effect and the fact that it is not enough to remove explicit gender indicators to get rid of these effects. And we have proposed methodology to mitigate biases without assuming access to protected attributes. Some limitations and open questions from this work are first, how can data representations be leveraged to penalize this proxy discrimination without relying on names? So the underlying idea of the methodology here is how can we move past assuming that we have access to protected attributes, but we're relying heavily on having these access to names. And this is something that is A, not feasible in many domains and it can have privacy considerations and so on. So are there ways in which we can leverage data representations and the bias that is captured in them to penalize this proxy discrimination in other ways? Another question is, how do we mitigate this compounding bias effect when it stems from bias labels? So throughout the methodology when we're doing things like penalizing a correlation between the errors and the location of the name in the word embedding we're assuming that we can actually capture the errors and that if we're correctly predicting the label that we have in our data, that's what we want to do. But often cases the bias is in that label that we're trying to predict. So how can we mitigate this compounding and balance effect when we have that problem? And here clearly the methodology that I presented is not the right way to go about this. And then the other question is how do we mitigate this compounding bias effect and how does it actually play out when we're not making predictions in isolation or when we're actually giving a recommendation to a human expert? And there then the prediction is interacting with the way in which the human makes use of the information to impact the decision. Thank you very much. Hey, fantastic. Thanks so much, Maria. I think maybe we'll leave the slides up for now if that's all right, just so people can answer clarifying questions if they need to. That's great. If you don't mind me starting. So people in the broader audience, if you'd like to put questions into the Q&A panel and then please do and I will sort of pass them on. We'll start with one from there and ask you to sort of expand out a bit on something you said at the start of the talk. So Jessica called asks what happens to people using the then pronouns? Do they get that correctly classified? Now you mentioned obviously at the start the limitations of using pronouns. I'm curious to know how you think, how you think sort of thinking about gender in a less binary way would influence the sort of practical upshot of the kind of research you're doing here? Yeah, so I think that there's two, three things to note here. So the first one is that clearly in the characterization that we did, these words doesn't accurately capture how individuals that do not identify with binary gender pronouns would be affected. And so that is a clear limitation and there when you apply method, so that is goes for the characterization portion. Then when you look at methodologies of how do we mitigate these, then if we assume that there's a portion of the bias that could be mitigated when we penalize for the correlation between clusters of names in the word embeddings and accurate classification. So in that portion we're no longer relying on gender pronouns. However, we are relying on attributes being captured by names. And so in the work that we did here we found that word embeddings, captured biases that concern individuals, races, gender along binary dimensions. It's whether, here for example, a question that is open is for example whether a bias is targeting transgender individuals would be captured, right? And so here you can assume if there were some, because of a social phenomenon there are certain type of names that individuals were transgender individuals were more likely to adopt. And so this cluster and this was captured in the corpus that was trained in the word embedding then that could mitigate this. But of course that's a lot of if, if, if, right? And so that is to say how the way that we approach gender, what are the limitations of the ways in which it may or may not mitigate biases. And the other thing there is that the, it goes towards intersectionality and is the fact that the dynamics of the bias there we cannot assume will be the same. Like the social dynamic, for example, one of the things that we found in what words they, like what are the type of words that make it makes it likely for the algorithm to predict whether someone's a woman. Is there are certain words that like, certain like very confident words that would leave the algorithm to predict that it's a man. Because there's, so here I should mention that in the paper we have an additional part in which we're just looking at the algorithm's ability to infer the gender alone. And so for the purpose only of looking at whether this is something that is being captured in the rest of the data. So again, the social dynamics that lead to these type of data looking a certain way and the way that the different biases are represented and how people choose to represent themselves in the biography is going to likely be entirely different. And so that's another component there. Fantastic, thanks Maria. Okay, the next question is from Lixing Xie in computer science here at A&E, Lixing. Hi Maria, thanks for a very interesting talk. I have two questions. The first one is about word representation. The second one is about notions of fairness. So the first one, as you know that in the last two years or so the language models are increasingly moving to subword units such as biparing coding. Basically frequent for our physical science friends who may not have heard of like substrings that are frequent and they are below the word level. I was wondering whether the biases you observe still exist on the subword level and maybe, and whether this could be one thing, one representation that could hopefully mitigate the gender bias that you described. Yeah, so there's, I know there's some work that has been done on trying to look at bias in contextualized word embeddings. So I think that that may be the type of work that you're interested in. There's some from, I can look up the pointers. I believe that there's an LTI group at CMU that has been doing work on this. So Angelie and Julia. So I'm gonna look up their work and I can send you a pointer afterwards because yeah, that's a great point that is the word representations that are being used now have changed dramatically. And so the way that biases is being captured there may be significantly different. And yeah, that has been addressed at least in part by that word. So I can send you a pointer afterwards through this slide so that others have access to it. Yeah, we could do that on Slack. My second question is that the algorithm be proposed to mitigate the biases is trying to equalize the classification accuracy across say gender groups. So as we know, there are a kind of variety of fairness measures in like that's been proposed in the past few years. Do you think like your algorithm or similar ones are amenable to take into account other notions of fairness? Yeah, absolutely. Yeah, so what we're doing here so one tiny clarifications that here we're no longer looking at the gender groups because all the things that we're wanting to do there is not rely and that connects to the previous question that is we did not want to rely on the single attribute. And so there we're penalizing with correlations with the word embedding with the location of the name in the word embedding. So that's just a tiny, a tiny clarification and then with respect to other measures of fairness the reason why we were focused on mitigating the differences in accuracy was because of this compounding effect that was the justification that we were using for why isn't that we were cared about this measure but you can replace the laws for anything else. So any other group fairness measures. So for those who are not familiar with these there's different notions of algorithmic fairness in terms of the metrics of fairness. And so any group fairness measure that is looking at different statistical disparities in groups you can plug in into. So maybe here it's useful that I have the laws. So yeah, basically like these laws here could like how you're computing the difference here, right? For example, here it's easier to see you don't need to be looking at the differences in the predicate probability but you could be looking at other metrics, right? So for example, instead of looking at here what you're doing is you're looking at the true probability for that case, right? So you could be looking at other metrics you could just be looking at these priorities and you could just be looking at statistical priority for example. And so group notions of fairness you could plug in to the laws very directly. Oh, thanks a lot. Then you have thoughts on who might tell you which one, which loss function is the right one to use. Sorry? So which loss function is right for certain application scenarios? Have you had thoughts about how to go about that? Given that this is so flexible and we can essentially plug in any group fairness loss. Yeah, yeah. So the reason why we focus on the disparities in accuracy is because we were concerned with this compounding effect. And so in this case, that is the justification that led us to focus on this one in particular. And there's other cases where for example as I mentioned the limitations if you're concerned with bias in the label, right? In the target label that you're predicting then penalizing differences in accuracy is not going to solve your problem. And so there may be reinforcing something like statistical priority may be more desirable in certain contexts. So definitely in addition to why certain notions of fairness may matter because of elements like their compounding effect and so on, depending on the nature of the bias of the data and the domain expertise that can also lead you to choose different fairness metrics. Thank you so much. Thanks for those questions, Leshing. Okay, the next question is going to come from Sarita on this topic. I want to push back a little bit on the constraint that you're imposing here of assuming that the ultimate goal is to not use these labels at all or not have them inform the ultimate outcomes. And so I mean, that makes sense. Like it's a pretty common thing in a fairness measure. But it also in the context that you're saying you're applying it in this like a hiring decision you admit that this is the sort of thing that happens in the context of this is informing a human decision maker who was in turn acting in a situation in which there's underlying injustice. And I'm wondering what happens if you let go of the need to talk about fairness in terms of there being no information from the label to the outcome and instead sort of allow. Yeah, so this is a great question and also like a great opportunity to make a clarification that is in no way am I advocating for trying to forget about the protected attributes and trying to do a fully data driven approach to fairness in which we can completely ignore the context and the domain. So yeah, I know the things here is that the motivation for how can we mitigate biases without assuming access to protected attributes is more coming from constraints in a real world applications where you have for example, in certain jurisdictions in the US you cannot use protected attributes for making for example, like hiring related decisions. So if you're deciding who to show an ad you cannot use these attributes regardless of how you're using it, right? And so of course in the US in particular you get into very yeah, like whether you can do affirmative action type approaches and so on is something that is very different to other countries. So that is a big portion of the motivation here of why we want to think about how do we mitigate these? And another thing as you mentioned here is that there's a difference between the context that we were thinking about in large part was also in automated systems where you're for example just choosing who to show an advertisement, right? Or something like that. And so there it's not to say that you should that they protected attribute and the asymmetries of these doesn't matter, right? Like of course whether for and to begin with whether there's a compounding in balance effect that constitutes a compounding injustice depends on whether this initially in balance is an injustice, right? And so there what is the protected group that you're concerned about matters? And so that is an important thing to note in these characterization, but yeah in these case we're looking also at automated systems but I fully agree with you that once you're taking to account the dynamics of the human then again what the protected attribute is is also something that plays a super important role. Yeah, just one quick follow up there. It sounds like in terms of trying to actually abide by the letter of the law if you're thinking about the agent who's not allowed to discriminate on the basis of these protected characteristics as inclusive of both the algorithm and the humans involved then you might think for example that given that the humans are going to bring their own biases to the table the augment of the algorithm might be one of counteracting rather than sort of being a neutral presence if that makes sense, does that? So what you're saying basically is that if the effect, so I think that what you're saying is that you cannot be using these in the in the ultimate decision and so you think that it's legitimate to use it during the prediction that is informing the decision because you're countering the effect, is that capturing that correctly? Well, yeah, essentially I'm curious if the law applies to like basically what do you think of it as applying holistically to the decision maker in general or to each individual aspect? So, because if you're thinking about it as applying holistically, you might say, well, the decision system isn't bound by this requirement because the decision system is really there to help make the entire decision process more balanced in the end. Yeah, so I'm not a law scholar so that would that be caveat in the legislation in the US according to my understanding, there are many cases in which you just cannot use the protected attribute. Doesn't matter how you're using it or how that is affecting the ultimate decision. You just cannot be using it as one of the features. And so there, of course that opens a lot of questions of how not using that can actually lead to discrimination as you're saying and as you're pointing out but there's still that constraint you just cannot be using this feature. So we, one of the nice things about this team is that we have people from lots of different disciplines and we have a lawyer who has a hand in the cube that's a couple who we'll be getting to in a moment. But first we have Atuza Kassisa there, Atuza. Hi Maria, thank you so much for the great discussion. So my question is more about what if in the, like what if you have good reasons to assume that in the data set we have two different sources of bias. For example, in word embedding or like the first, the first onto like identify gender bias and mitigate on the gender bias by like kind of changing the loss function, but like we have gender bias and for example, racial bias. And so then it seems to me that in such cases even in the first case when we have just one sources of bias, it seems to me that there is no principle like systematic answer to how to go about choosing these loss functions. So what if we have these two sources of bias, how are we going to choose the loss function? Is it gonna be something really random? Is it gonna be something that we just, yeah, like or fine tune somehow to get something that we want to get the results or is there more like theoretical insights behind the choice of the loss functions? Yeah, that's a great question. And that was actually a big portion of the motivation for these methodologies. So here what we're doing is if you look at it, for example, the clusters, right? What we're doing is we're penalizing correlations between the cluster. So here we will completely beyond the portion of looking at gender and we're penalizing whether clusters of names in the word embeddings are correlated with errors. And so there's, I don't think I have it in these slides. So maybe I feel like we see, no, I don't think I do. So, but in the AIS paper, we have the tables looking at the groupings of names that arise in the clusters of the word embeddings. And so there you do capture not only different demographics, but you capture intersectionality. So you'll have, for example, a cluster of young Latinas, so like female, young, and from like certain areas, right? So like you do capture a lot of these intersectionality. And interesting thing here is that depending on the corpus that you're using, so depending, and this is part of the followup work that I'm doing, is looking at how the biases that you capture change drastically depending on the context of the word embedding. So for example, when I was looking at it in Spanish, and so where names end up, or like what the clusters end up looking like are picking up in entirely different societal biases that are present in the corpus, in the context that gave rights to the corpus that you use for training. But so that is what you're saying is precisely and what motivated this methodology that is how do we go about thinking about multiple dimensions of a bias. Again, caveat is here your constraint to bias is capturing, start capturing multiple and intersection. Could the same idea also apply in the first work that you presented, and not the word embedding case, but when you are trying to minimize gender bias or something? Yeah, so here what we're doing is, and you could do that directly without the use of the word embedding, but think on the side you care about. So for example, class A occupation, right? And here you could do it directly. So if you knew that you can act, you only care about gender bias, for example, and you have access to this information, then you could just directly penalize that. This is thinking about what happens if you care about penalizing, mitigating the bias that we found in the first portion, but you don't have access to gender data or you care about other aspects as well. And so that's how you go about this, but eventually what we're doing here is precisely addressing the bias that we characterize in the first portion. Well, thank you. Okay, next question from Marple Yang, a PhD student in computer science working on individual and group fairness. So this topic is right on point for you. Yeah, thanks, Seth. Thanks for the talk, Maria. I think it's always wonderful to see case studies because they help motivate why all this work is important in the first place. I think my question is related to the current, what we've been discussing so far. In the first paper, could you explain why we saw differences in accuracy for identifying the same occupation for two different gender groups? I assume my assumption is that it's about how the representations of how people write their biographies differently. And I just, I building on that, I wonder if there's something like about the way in which the biographies are gendered, right? Even for the same occupation, like how people write their own descriptions. Yeah. Yeah, so this is a great question and there's several parts to my answer. So one part is in terms of what gave rise to these bias, right? And so there what you have is that the data is in balance, right? So as long as you have differences in how the different genders choose to represent themselves, then that is what is giving rise to or are represented in their biographies. If they're written by someone else, then that is what gives rise to these like, like you have this like statistical, the algorithm has this statistical incentive to say, well, if I can infer the gender and I know that women are less likely to be surgeons than I'm more likely to not classify the women as surgeons. And in terms of what you're saying of the choices of representation, that is something that we started a little bit in our analysis and we found some interesting things there. So for example, well, the portion of swapping the gender pronouns allows us to see that it is not just that one of the genders just write biographies that are impossible to tell what they are, right? And so that's kind of like the area that we were covering with doing that. But then when looking at, and that's why we build the model to predict gender from the occupation, one thing that was interesting was at one point that this is a little bit anecdotal, but it's something that we found at scale. So there was, at scale there were certain words, for example, whether the word husband or wife appeared in the biography was much more likely to be the biography of a woman. And then there were certain words like renowned that were way less likely to be a woman. And so there, when I was running some of our biographies through the network and looking at the prediction, I run mine that I showed you, and when I run Jennifer, Jennifer, who's that called, Jennifer Chase, I run that one and the word renowned was marked as one of the words that the algorithm paid attention to to predict that Jennifer was a man. So, and of course, using attention to infer what the algorithm for explainability has like very clear limitations, but it still allows you to, in that because the algorithm is a small parenthesis, because the algorithm is not paying attention to something doesn't mean that it's not using it, but you can see that it is using something for these predictions. So that, I think, I hope gets to your question of what type of patterns we saw of what gave rise to this. Yeah, that's great. Thank you for that. Okay, so we've only got time for one more question, but those of you who have questions that are hanging over, please do take them over to the Slack channel and carry on the discussion there. It's been a super presentation discussion and a really wonderful model of how to present this kind of work for a general audience. So thank you for doing that. So to bring it back to a question in the Q&A, so Joe Hopers raised the comparison between this kind of work and the use of recommendation algorithms in particular for things like Amazon, what have you, where there's a narrowing effect that it seems that everyone wants the same things each time, leading to a very narrow model of the person. And he points out that the work was being done on building recommendation algorithms that can sort of avoid this bias and incorporate, I suppose, a certain measure of exploration as well as exploitation. Do you see kind of analogies between what you're identifying here and the same sort of problems that arise with recommendation algorithms? Yes, I think that here we had something that allowed us to make these assumptions that is not too well, that is people are interested in job postings within their occupation, for example. And so that is how, what you're predicting is evidently relevant to the individual and also you, that allowed us to decide that we wanted to focus on true positives. So that also like the effect of true positives, which is something that may be very different. And even here, there may be caveats to these that is different types of errors may have a different effect, right? And I think that here is where you start also having parallels with a lot of the personalization and targeted advertisements are things that is, if you're showing an ad for a job that is slightly above yours, and so that incentivizes you to apply, it's very different to if you get consistently shown jobs for something that you're overqualified. So looking at, looking at that I think is something that it could be done as follow-up from this work looking at how not all mistakes are the same here. And so the type of mistakes and the effects of like, for example, showing a job advertisement for something that you're overqualified versus underqualified may be different. Thank you for that, Maria. Okay, so my little helper has come to tell me that it's time to finish. So thanks so much, Maria. I think Michael is the one who could end the broadcast right now. I think only by ending the whole recording. Thank you to everybody who's stayed on. Thanks for all the great questions. Wait for it. And let's head over to Slack to continue the discussion.