 Thank you. Okay, so I am here to talk about bias auditing. Auditing, artificial intelligence systems is now all the rage, but we began working on this topic some time ago with a lot of people. We have tried to amass diversity of people, so here are the people who are permanent in our work team. These are people, but we actually have had hands-on workshops with diverse people from extremely different, no, not extremely different, but quite different backgrounds, so people working in nutrition, journalists and data scientists as well to get the grasp about how bias auditing could really, really work. So I'm going to talk a little bit about what bias is and then how we have approached it. So bias, as per the definition of the Oxford languages, is an inclination of our prejudice for or against one person or group in a way considered to be unfair. And how does this translate into our everyday life? Usually in our everyday life this translates in the form of stereotypes. Stereotypes are subtle but very, very useful ways of packaging bias, and I am going to give you an example just so we are sure that we are in the same page. For example, would you consider hiring Sylvester Stallone for your film? You probably, maybe not, maybe you would, but probably what you would not is consider hiring them, hiring him as a screenwriter. Although he was shortlisted for the Oscar as best screenwriter for Rocky. I'm not very much into this kind of film. So this is an unfair situation for him because you are not giving him the same possibilities as other people for this job. Even though he is a white male coming from a central country and strongly recognized professionally. So imagine what is left for the rest of people who are not so well positioned. So this is how stereotype hurts. And in language technologies we have lots and lots and lots of stereotypes that produce unfair situations for people. For example, if you type the poor people are, this search engine tends to complete your search because it assumes that you will probably look for what most of the people have been looking for or have been writing about this. And I am going to give the completion in Spanish because in English it has been word railed already and probably in Spanish it has as well by this time. And you can see that some of the auto completions that this mainstream search engine gives for los pobres son are things like los pobres son responsables de su propia pobreza. Which means the poor are responsible for their own property which if you are poor and you are using this tends to have an effect on how you are to perceive yourself. Which is not so positive. And you have more auto completions in the same line. It's not just one error. It's a systematic error. And that is what we call bias in data science, artificial diligence or the marketing name that you choose for this is systematic error, non-homogeneous error. So this is something that we want to audit systems for. We want to audit systems and see if they have this kind of systematic behavior that discriminates against some group. And in some cases these biases are huge like gender bias. Gender bias is something that we all know that is pervasive in most cultures. And we kind of have a consensus on how to model it. And we have resources to do that. And so we kind of have been advancing in how to measure that. But some others are not so huge. They are more subtle and they are difficult to perceive, to formalize, to transform into something that can be used systematically by anyone. And most of them are culture dependent. And this means that they are also language dependent. So you cannot reuse what has been done in one language for other languages. You cannot reuse what has been done in the same language for one culture to another culture within the same language. So this means that it is critical who defines bias. Because only people with bias experience can define bias. So only people with experiences in discrimination can get to define bias. And experiences in discrimination come in all colors, not only in white. But we have been having a huge problem with this. Although bias auditing has been advancing in academia. So it's a hot topic in conferences. And there are many, many papers and toolkits and frameworks and proposals about how to measure bias in language technologies in particular, but in general also in images and whatever. The very big problem that we found when working with this is that these approaches are very technical. Are highly technical. That not only that they require you to be able to program, some of them do not exactly require programming skills, but they require that you manage mathematical concepts. And they want you to install complex software. And they resort to terms that are very, very, very technical. So people with discrimination experiences have to climb over these very steep technical barriers to be able to express their discrimination experience. And that is suboptimal, highly suboptimal. So what we did was a tool to facilitate that people with discrimination experience, and when I say discrimination experience, I'm talking about not only that they have suffered discrimination or that they consistently suffer discrimination, but also that they have studied it or worked with that. That these people can express their knowledge without the technical barriers. So we, in this tool, we resort to intuitive concepts and we have extensively tested that these concepts are actually intuitive for people who are not in data science. And intuitive concepts like, okay, you think that the technology that you're using is biased. How do you see that? Well, you can see that, like actually see that this way. For example, these are the words fat in masculine and fat in feminine. And this is ugly in masculine, ugly in feminine, in Spanish. Okay, sorry, sorry, sorry, sorry. And you can see that they are far in space, yeah? But you can see that fat, sorry, fat feminine and ugly feminine are very close in this dimension. They are almost both at this line, while masculine, fat, and masculine, ugly are much more separated. So this means that in some respects, fat feminine and ugly feminine are equivalent, while fat masculine and fat feminine, fat masculine and ugly masculine are not equivalent in that same dimension. And this is something quite intuitive. Underlying this, there are vectors, there are distances, there's a cosine and principal components and embeddings, and neural networks and all the blah, blah, blah, technical things that you want. But this can be understood. Actually, this is not something that we came up with. This is something that people working in the social perception of eating habits intuitively knew, and they checked it in our tool and they found this. This is a finding that these people found about the underlying artifacts in language technologies that actually deployed. So how does this translate in discriminative behavior of technologies? For example, like this, the completion of the fat masculine plural is Gordo Spondiola, and the fat in Alice in Wonderland, and the fat suffer less the cold. And when you have the fat in feminine plural, they are disgusting, they can get pregnant, which are less positive complications. And to do this, we did not need to resort to any of these concepts that have been mentioning and that populate all the literature about bias auditing in academia and available frameworks. So we are happy to be working with the intuitive approach because we find this is useful and productive for people to express their intuitions about bias. But this, as you can see, is a little bit anecdotical, and you need to be more systematic in the way that you audit bias. So this is why we provide something like this. Yet this is more systematic than that can be collapsed into a number, a figure that you can report to the competent authority or whomever. This is, for example, to the right you have feminine, to the left you have masculine, and you can see that verbs like argument or think or ask are closer to the masculine meaning than to the feminine meaning, while knitting, sewing, and dancing are closer to the feminine. And this kind of systematization can be achieved by just listing words that represent the feminine meaning, like woman, she, mother, girl, whatever words, the people with the experience know that represent the meaning that they consider as feminine, the same for the masculine, and then check whether words that should be in the middle are in the middle or not. With the typical example of these are professions, you have a representation of feminine with she, blah, blah, blah, a representation of masculine with he, man, whatever, and then you have professions, and you will say, okay, but you have then, for example, professor feminine, and then professor masculine, and they are feminine masculine, but you can see things like nurse feminine, which is very much into the feminine side, nurse masculine, which is the least masculine of the masculine professions, and this is clearly biased, and this is quantifiable, and this is reportable, this is systematic, and this all available in this tool that is graphical, and you do not need to know that this is measured using cosine distance with woman, whatever, yeah? So what is critical here is not that you know about vectorial space and distances and similarities, but about word lists representing the biases, the stereotypes that you're experiencing or that you have experience with, and for that, the only thing that you know is experts, experts have a hierarchical role in auditing systems if you do not have technical barriers, yeah? When we were working with this in the first hands-on workshops, the feedback that we got from experts was that this was very limited, working with words was limited, because words are ambiguous, we knew that, yeah, I'm a linguist myself, we knew that from the start, but this was the first thing that we had, so words are ambiguous, they are inherently vague, they need the context to specify their meaning, and then there's also multi-word expressions, things that you express with more than one word, and then there are non-binary semantics, like age, you do not have old and young, you also have child and young and teenage and middle-aged, so it's inadequate to express that with this kind of diagram, isn't it? You don't want that. We also have problems with non-marked meanings, like we said, okay, we want to check about indigenous, but then what is the opposite of indigenous? Non-indigenous? No, it's not, okay? So to address that, we worked with language models, we worked with language models, language models are all the rage now with chatGPT, but this was before that, when language models already existed because they have been existing for some time now, and what we did was work with sentences, people express the biases, the possible biases in sentences, and if your language model is not biased, it should give equal probability, it should prefer all sentences equally, and if it is biased, it means that it is more likely to say one thing than another. For example, here you see the Bolivians cannot manage their money, the Argentinians cannot manage their money, the Germans cannot manage their money, guess which of these sentences the language model is more likely to produce, like GPT. GPT is more likely to say the Bolivians cannot manage their money than the Germans cannot manage their money. This is a very nice way for people to express their experience of discrimination, their experience of stereotypes, and it is quite easy to measure if a given artifact, a given language model, like the one that's this underlying chatGPT or Lama, or whichever language model you like, has a preference to say one thing or the other. So now we are quite happy with our instruments, but we are people working in computer science, I am a linguist, but I have been working in computer science for a long time now, so my imagination is beneath the standard that any standard that you can get to, so we have only been working with gender and race, there are many, many, many more biases to be studied, so we need experts to create these resources, word lists, sentence lists, and we are expecting, we are working now toward people using this or any other approach, but we are very confident that our tool is useful to create these lists that represent the stereotypes that they experience so that these lists can be readily used for people working with artifact, these artifact, these language technologies to have a first audit of these, of their technologies to assess what next steps need to be addressed with the results of your audit. So we encourage you to create these word lists, contact us, we are working towards having these approaches more widely used and to have experts in the hierarchical central role that they deserve in the development of audits for language technologies.