 For those that have joined our conversations before, you know that this is part of our series of conversations with data science folks called Making Data Science Work. So I'm Indrang and along with Venkata from Scribble Data. We play host to this conversation and we learn a bunch along the way. So this is as much for us as it is for you. At Scribble Data, of course, we are an ML feature store company but our conversations here tend to be fairly broad-ranging and in fact today's conversation is a sort of testimony to that because we are now talking about something that gets a lot of lip service nowadays but actually infusing it into our DNA, that's a harder call when we have to actually make sure that we are walking the talk. It's a little bit harder. So today's session is about doing data science with ethics and we had originally had two panelists slotted. Unfortunately, Ashwin from Fidelity couldn't be here because of some logistical reasons and others but we do have Suchana and Suchana said, well I mean there's a lot I can tell you about her. I'll just start with a few highlights and then I'll ask her to speak about it. So Suchana in the past has been a Mozilla open web fellow at Data & Society and a fellow and affiliated at the Berkman Klein Center for Internet and Society at Harvard. Her research focuses on ways to operationalize ethical machine learning and responsible AI in the industry and she has a broad range of interests from auditing algorithms for fairness, accountability and transparency. She's interested and very passionate about closing the gender gap in data science and leads data science workshops with organizations like Women Who Code. So we have a lot of landscape to cover with her and a lot of questions because even in when we were discussing about the various topics we touch upon today, we found so many different rabbit holes that we wanted to go down. Sometimes it felt like the one hour wouldn't be enough but let's see where today's conversation goes. So Suchana welcome. Thank you for joining us. Thank you Andha and thank you Venkata and thank you for putting together this lovely series of sessions and the topics are incredibly interesting and relevant. So it's great effort. Thanks for putting this together and having me here today. It's entirely our pleasure. So Suchana one of the things that we'd like to quickly sort of bracket right at the start is maybe a couple of points along what got you into doing what you're doing right now. Some of the some of what the meandering finally landed on. Yeah just before we started off Indra and I were talking about how you know sometimes interesting career paths tend to have some unexpected tours along the way or they take a meandering route that you would not necessarily have expected at the start of it. So my background is in theoretical physics and I did my masters and I have a few years of research under my bed and you know while I was doing my research in theoretical physics I became increasingly interested in sort of the intersection of machine learning with theoretical physics and then eventually it dawned on me that you know there is a lot of scope for application of ML in the industry as well and data science was just sort of taking off in India at that point. So I made the switch to the industry and I think all along I have been interested in a lot of these rabbit holes that again Indra was talking about. A lot of it is philosophy and hopefully we won't digress too much into philosophical tension here today we will sort of stick to more hands-on practical checklists, tours, workflows, rubrics that kind of stuff but I was definitely interested in what are the best practices for machine learning. How do I do responsible machine learning even before these things became a buzzword maybe and along the way I was very fortunate to have volunteered with organizations like BetaKind who you know who bring together data scientists who want to do pro bono work with non-profits who have data but not necessarily the resources or the talent to utilize that data right. So when I was mentioning data science sessions with organizations like BetaKind I began to realize that there's something of a gap in terms of you know we as data scientists either you know whether we're self-taught or whether we go through formal courses we pick up a lot of technical stuff about what are the different ML models we could build but what we do not learn are some extremely important pragmatic stuff like what's their scope what's their applicability what could go wrong when we build these models how do we make sure that our end users sort of understand how best to use these models and apply them and make sense of them right. So as I began thinking about all of these things in the course of my work as well I think you know I was very fortunate that Mozilla started their open web fellowship programs and they really wanted to build a whole network of public interest technologists which is sort of a fantastic thing to happen and I think there's more need for it than ever that we look at technology through the lens of public interest as more and more of our you know I would say critical infrastructures like democratic infrastructures our social and commercial infrastructures are sort of being dominated by different technologies are being dominated by different corporate players I would say. So I think there's definitely a goal for public interest technologies to make sense of all this and and sort of you know protect our rights and to push tech in the right direction. So I feel very fortunate that the Mozilla fellowship you know sort of connected me with this wonderful network of people across the world who sort of think similarly and work on similar issues and then you know my stint at Harvard also is like extremely rewarding. So that's been my path and like I said it's been a little you know I couldn't have expected or anticipated all of the steps in advance and I sort of I want to say this you know for the sake of pretty much everyone out there who wants to go on an adventure and is perhaps wondering where will this lead? Go for it. Yeah absolutely. So Sushant actually you touched upon a few different points when I already in in this introduction and I would love to understand from you. Do you think where do you think ideally the slow permeation of ethics into data science should come from? Is it from good-hearted data scientists who are thinking about doing the right thing? Is it within corporations who are thinking right from the start we shall not be evil even if our motto changes along the way at some point or is it in the form of governments looking after their people and instituting strict guidelines whether it is data privacy guidelines or other ethical boundaries where what's the ideal path? Maybe it's a combination but what's an ideal path? Yeah well I think it's obvious that it's a combination but having said that I want to call out a couple of things. These are personal opinions and I do recognize that there have been different points of view as well but the first thing I want to say is that the regulatory landscape and pretty much any effort that any government across the world makes to regulate technology is going to be somewhat reactive at best. So it's the pace at which technology changes and I mean I for one sort of love the fact that we innovate so quickly but it also means that it's very difficult given the amount of time and the amount of human effort that it takes to put together credible regulations the regulatory landscape will necessarily be reactive. It cannot anticipate every possible advance and it cannot provide legally for those situations that might arise. So maybe it's best not to place all of our hopes in that one egg in that basket and I think in terms of corporate ownership of ethics I think a lot of companies do really well but again having said that I tend to be a little cynical as part of my professional work and I would say that very friendly is our friend. So while good intentions are fantastic and no one's doubting them but at the same time I think AI systems should certainly aim for verifiability for all of the claims that we make of them. Whether it's a robustness claim or whether it's a fairness claim or a transparency claim I think we should have verifiability built in and I think that's really key to having accountable systems as well and having auditable systems as well and we will touch upon all these aspects accountability and auditability hopefully the course of our talk today but yeah no verifiability doesn't work for me. That came to my mind is that it is not that we have not been here when lasers were developed when nuclear technology was developed and a whole lot of bio weapons for example when they were developed over a period of time we developed the the institutional framework, the accountability mechanisms for all of them. It almost seems like if you look at all of them they took 30, 40, 50 years for them to develop whether it is a START treaty, whether it is the biological review board that departments have and so on. It almost seems like that whole process is now extremely compressed and it is happening at a time when we don't know the the parameters still of the whole ML escape in what is the class of applications, what is the degree of impact, nature of impact, time frame of impact and so on. It's how do we cope with or how is the coping happening today? Given that it's only a large number of people. For example all the loans increasingly are going to be scored you will be measured against some model will assess your risk and then give you. So the impact is here the institutions and the the mental frameworks to understand and evaluate all of these things are lagging behind already even before the regulation comes. Right so I think we already do have some robust you know institutions and institutionalized learning that we have in place. For example IEEE is developing a whole set of standards that relate to algorithmic bias right and I'm part of one of those efforts the P7003 standard but there are also a few other standards within that same family that have to do with algorithmic transparency that have to do with autonomous vehicles that have to do with ethics of intelligence systems and so I would say you know that's just one example of a non-government kind of a self-regulation developing checks and balances kind of an approach and I don't think it's a bad approach at all so I think some of our older institutions will translate quite well to this new landscape as well but having said that there are I think a few crucial aspects that make this whole machine learning and AI regulatory landscape a little different you know which is one algorithms have the capacity to amplify harm right or to amplify benefits so the thing is you know if you have a machine learning algorithm you suddenly have the capacity to impact people at scale you know and that scale is very very different from you know the scale at which let's say if you're selling an individual vehicle even an autonomous vehicle to an individual consumer the scale of that is very different from you know the scale with which social media is entering our lives and touching almost every part of our lives right and the other thing I want to call out you know this scale and amplification is one aspect the other aspect is the networked nature of machine learning and AI you know so for instance if you take data privacy right it's not a simple matter of me protecting my privacy I have to worry about what my friends are saying about me on that you know for my privacy to be fully protected and that's something that I don't necessarily have any choice control or consent over so this networked nature is again something that we should shape carefully around and I think deeply about when we build products and make regulations right so you know if you consider things like IOT for example right here are possible ways that devices in your home can be hacked so call me an alarmist call me paranoid right but that's my job so you know here are ways that you know that privacy inside our homes can be compromised right take wearable devices for instance take you know the obvious marriage between wearable devices and social media a lot of us are you know doing exercises you know logging our runs and stuff on Fitbit sharing it on social media or NQDOS whatever it is right so these are our examples of the networked way in which we need now need to think about our rights our privacy the impact that machine learning algorithms can have and I think these are two key differences I would say yeah wasn't there that matter about Strava the wearable company that was highlighting where CIA bases or armed forces bases work because soldiers are wearing them and exercising I'm not knowing that all of that is being tracked so so you know if you step sort of almost into the field of data science now as a data scientist what should how should they think about this what should how should they think about their responsibility when when when they're asked to be ethical or when you know they're doing data science and they have there's this voice in the back of their heads that's saying hopefully it's saying I should be ethical especially in the context of their KPIs their individual KPIs not necessarily being aligned to what is considered ethical so I think you know this this whole alignment problem I mean you know in the in the field of AI we tend to talk about outer alignment and inner alignment as to whole problems in AI safety and I think it has analogs and mirrors in human life is where right how do we get our KPIs to align to our yeah stated values or our not so clearly stated values right so that's definitely an org culture problem but I would want to begin from the fact that data scientists have a tremendous amount of power let's not forget that so if you look at the whole ML production chain right I think data science is the best place to understand the big picture as well as the details so we understand where the data is coming from both the data is like what are the challenges with the data we understand why we made certain decisions while building the prototype you know in terms of choice of model choice of hyper parameters and so on we also understand you know what choices we may have made during deployment you know maybe we decided to put you know we decided to consume the prediction of the actual model by coupling it along with some handwritten heuristics or rules based on our business understanding right so we understand the entire chain perhaps better than anybody is involved in the production of ML systems you know so given that power I think we are best placed also to call our potential harm that can arise you know a lot of consequences can be foreseen in the sense that you know I hope we would talk about it in some detail later in the conversation but if you think about this business of choosing the right metric to optimize right sometimes in hindsight it seems blindingly obvious that if you know since we chose this particular metric we could have almost predicted how users are going to game it even if we do optimize for this metric right so I think these are conversations that should not happen in silos definitely data scientists should engage with all stakeholders as far as possible we should talk to product managers we should talk to actual users you know we should talk to ML engineers we should talk to you know folks on the business side of things when we try to identify you know what kind of consequences or harms can arise or what are really the right metrics to optimize for but there are also some very interesting nuances to things like accuracy right that again we are best placed to understand so for instance we should be calling out what is the cost of a false positive what is the real cost you know what is the real cost of a false negative with the prediction that my system is making and so you know what are some realistic accuracy goals that we should be setting for ourselves and you know what is that goal really translate to in terms of a business action right so what happens if we have a false positive what is the real cost to the users downstream so just to give you an example right there's some terrific research by Joy Bulwamini and Timit Keblu MIT and Microsoft who showed that IBM's facial recognition software had an error rate imbalance with respect to gender and with respect to race so what I'm talking about here in simple terms is that the facial recognition algorithm would make more mistakes with people of color and it would make more mistakes with women and it would make even more mistakes with women of color you know and uh this is the sort of thing that but hindsight is totally but this is the sort of thing that is quite easy to correct in the prototyping stage where you're making your model choices where you're clearing your hyperparameters it's a very simple question to ask right should be part of everybody's checklist that if I were to take my accuracy and I were to look at my error rates across different demographic classes that I have whatever is of interest to whether it's you know gender or whether it's race or whether it's ethnic identity or whatever other demographic parameter might be of interest to me right it's quite simple to do with check and it can you know prevent the kind of embarrassment that later where IBM did to be fair IBM went on to make some massive improvements to the way that they were constructing their training data set so their facial recognition software training data set is now much more diverse and much more explosive so you know great that this outside research kind of made that change happen but perhaps that research shouldn't have been necessary because yeah and just on the IBM point though my recent understanding is despite them having made some efforts to correct the imbalance they've actually taken it offline because of other ethical considerations right okay in fact several other companies amazon included have actually chosen not to sell facial recognition software for surveillance applications particularly to you know police departments and to governments and this is you know I think this is sort of a fantastic example of what happens when an organization articulates its values you know perhaps does an internal reality check in terms of what do our employees want where do we stand with regard to values and it takes a call based on that right I think about a couple of years back Google decided you know based on employee pushback and you know that they were not going to do an AI project for the US Department of Defense so that's that's again a fantastic victory yeah yeah yeah I have a question actually this is from one of our panelists uh Yannis Yannis thank you for asking this he asks what about building technology for controlling technology in obviously within the ethical landscape are there any do we know of any such examples and if there are any concrete standards that one can base such technology on any anything comes to mind technology yeah so so there's a whole spectrum of technologies that are actually being developed for this right so uh what sort of technologies have to do with being able to audit algorithms and you can automate that to the degree that is possible although I would always favor human in the loop systems simply because we are not quite there yet in terms of yeah imagining every possible thing that there are always unknown unknowns and humans are still a little bit better at at getting a handle on unknown unknowns than AI systems are just now so uh there are ways to interrogate AI systems right where basically uh you know imagine that you have this API right there is a prediction system it's a black box you don't know what it's doing but you know what its input is supposed to be and you know you can give it an input and you can get a prediction as output and you can look for different kinds of discrimination you can look for different kinds of biases you can uh and also you can uh kind of interpretability probes you can ask you know counterfactual style questions what would happen if I were to change the input just a little bit how would the prediction change then right so these are all and there is a whole spectrum of uh tools that are being developed which are at different points in the automation spectrum I would say but they all you know they're all in a sense an automated way of integrating an algorithm if that's what Yanis was true true except I mean not except but rather maybe adding on to that point which is that it seems to me that while these technologies can be built can probably do exist probably are useful it depends on the context in which they're being used because one narrow and probably uh extensive use case for all of this is just to see how well the technologies how well the algorithms are performing rather than putting on the ethical lens to look for biases especially for biases in race and gender because you're thinking about fairness because you're thinking about you know social equality and all of that instead you could just use that same technology and say uh how well is it performing I am you know I'm under indexed in this in this field versus that etc etc etc I think that human in the loop like you said not just because of the unknown unknowns but also because somewhere there needs to be an infusion of values about how you're doing this too I absolutely agree with you Indra but I also want to call out this perhaps false dichotomy that and it's a provocation that I'm throwing out there for the audience as well to consider maybe you know there isn't always a tension between accuracy and fairness so let's say that if you were to have a disproportionately high error rate for a certain group of people then that's a kind of unfairness in itself right so even if your goal was a very narrow goal of just optimizing for accuracy right even simply by optimizing for accuracy you would end up optimizing for certain kinds of fairness and that sort of brings us back to the reality check in terms of tensions and trade-offs right so um there are any number of possible definitions of fairness for instance right there's a fantastic talk by Arvind Narayan and a couple of years back at the Fatima conference where he talked about talked about 21 definitions of fairness and their political implications their political context in which they are to be applied right and there are tensions and trade-offs between them optimizing for one kind of fairness you do so at the cost of optimizing for another kind of fairness right so this uh this this challenge of first articulating our values and then translating those values into metrics is something that is there pretty much in every system you consider it doesn't go away and that's at the heart of doing AI ethics right how do I go from articulating something subjective and not very tangible like a value or a preference that I have or how do I elicit that from my users you know their value preferences and then how do I map it on to a tangible metric which I can get optimized for and then how do I also articulate the tensions around well if we optimize for this particular metric how does that play against my interpretability requirements how does that play against the accountability structures that we need to set up how does that play against my accuracy goals right absolutely the other sort of provocation I want to throw out there is who do all of you who do you know who all of us collectively think should be deciding these things you know should it just be a single ML engineer who maybe defaults to system settings or should it just be the entire data science team uh you know should it be the entire organization you know in town halls yeah so uh while I'm prescriptive about the questions we should ask I'm not prescriptive about the answers so these are definitely things that we should start thinking about collectively and you know I'm not saying that uh data scientists should not own this on the contrary we should definitely be driving these conversations but I am saying that we need inputs from every stakeholder possible you know system up one counter uh example to that whole concept of if we look for accuracy we will try and get fairness as well uh one no no no no no thanks but one prominent example comes to mind where if for example the ml recommendation engine for youtube videos was doing its job well it would get you to click onto the next video or watch the next video whereas at the same time that that success there by that accuracy might end up doing the kind of polarization in views exactly mindset that you might not want yes and this is you know there was quite a bit of uh uh quite a bit of noise in the in the press about this as well right so this brings me back to the tensions that I was talking about right so are we choosing the right metrics to optimize for right and none of these conversations can happen in isolation so uh you know just like in the whole data science workflow you know there is uh there is kind of a loop like quality to it so we build a model we test it out we're not quite happy with its performance we come back to the drawing board maybe choose a different model or choose to optimize the model differently right I think ethics needs to have that same loop like quality to it in the conversations that we do because certain consequences are easy to anticipate certain consequences are not and so sometimes you will only learn these things by doing a field test or maybe by shadowing some existing models and you see how your own model plays out and the the metric it optimizes for and what the consequences are so then we need to go back to the drawing board and ask okay perhaps we need to change the metric we are optimizing for which in turn means we need to optimize you know we need to change all of the other considerations as well we need to ask what fairness criteria should we be thinking about now in the light of this new metric that we have chosen right but it's not a uh uh you know uh I apologize for being a little vague perhaps but it's difficult to discuss this without like a concrete case in mind yeah well on the one hand the concrete case and on the other hand the underlying philosophy as well make it I'm sorry I've been completely hogging the channel please interrupt as you see topic see the the cases that actually bubble up in the in the community like the youtube case right those are um after a point it is it is obvious to everyone that there is there is a ethical question here but when I look at the life of a average data scientist you know you're trying to sell coffee and all of those kinds of things my big concern of the the question that I have in mind is not the clearly identifiable case but the the cases that are slightly nuanced that you have to spend a little bit of time thinking about them and then they pass just because it is not very obvious in the first go to you for example this even if you for example in the youtube case one is the the most obvious clique related problem but if you look at the complex ecosystem of youtube itself how many other places are there uh unnoticed ethical issues where people didn't think much and they thought it was just a mechanical activity and somehow it is built into the process the tooling the the interface the language there is some bias so much I'm worried about the dog that didn't bark then the dog that yeah yeah absolutely agree with you Venkata in fact you know I think that's where sort of uh you know the the term scientist you know which is part of the term data scientist really matters I think this kind of critical reflection should be part of every stage of our workflow right and it helps if at the beginning of the design process we are asking about the possible kinds of harm that can arise and we can provide a little bit of a structure to that question by asking things like what are the representational harms what are the allocational harms because as long as you're formulating the problem in terms of a vague ethics goal right it's it's difficult it's nebulous it's difficult to know where to begin but when you make it a little more concrete and ask who can this system and I don't mean just the machine learning model but I mean the larger software system or the or the larger platform that it's a part of right who can this actually harm and uh you know what I mean by representational harm is that am I you know is this product or is this solution by making a prediction is it reinforcing entrenched stereotypes right am I continuing to represent a misrepresented group of people in a bad group right that's representational harm and then you can think about allocational harm right which is where you're asking am I depriving people of a set of resources by making the prediction in a certain way so you know there are documented instances of where uh women are not necessarily shown advertisements for jobs that are very highly paid or that are senior leadership roles as compared to men right it's a thing so so that's you know that's like a classic example of uh allocational harm after I get over my embarrassment that the state of affairs the intention wasn't uh to no no I know I know I know I know this is just so I was amazed as well when I kind of learned about it a couple of years back although there there are more I think egregious examples are you guys familiar with the one where uh you know that there are these uh tabs with motion sensors that don't work so well for dark-skinned people I thought that was even more embarrassing like this when you're talking about tabs in public info like you know airport bathrooms and things like that I mean the pervasiveness of this problem is what requires this conversation and that are going to come because there are all scales of application in all uh uh different experiences that we as you know citizens humans employees yeah so one uh related question again from Yanis uh Yanis is complaining saying that I'm talking too much about selling coffee that might have been as I know I yes it is I go in my prayers like I was selling coffee so with my data skills that's the problem okay so he he writes um um since you have written extensively about the the uh this is Zainab Venkata this is all Zainab oh Zainab Zainab not Yanis um can you share we can take it now or later share can you share how the the law and the privacy bill that is in the works how will it impact ethics or the impact the role of law in the whole ethics conversation itself yeah so I think you know this is a really complicated question you know if only because I think privacy is a small part of the of the conversation around ethics right it's only one small piece of it and so you know there is this whole um product liability angle to it as well which is another rich vein of discussion that maybe we can you know consider as a standalone discussion in itself but you know if you sort of look at uh you know legal efforts around the world starting from GDPR in the European Union right so there is no rights and protecting rights based approach to governing AI which is great but then from a very bread and butter you know e-commerce just machine learning and industry perspective there's the product liability angle as well what happens if my AI system does something that it wasn't supposed to and it uh you know it causes harm so who is liable to what degree for it and and how much and all of these pieces also intersect with the privacy conversation right but having said that I think I think again there are a lot of dichotomies in this conversation right it's it's it's not either or it's not it's not that we can have privacy or we can have nice things right that's not true so there are a lot of privacy preserving technologies that are being developed I think you know a lot of the contract tracing apps that are being developed you know certainly the ones that google apple uh you know these do take privacy and encryption pretty seriously right they're doing a critical job and so uh maybe the we should be reframing the discussion a little bit and asking privacy should be the default right how do we make sure that that happens so how do we use techniques like you know federated machine learning for example how do we bring code to the data as a default practice rather than taking data to the code right I think these are the conversations that we should be having as a community actually this is one to point out that one upcoming session sometime down the line with case studies there are particular cases where data scientists have taken a position on the privacy and built the entire tooling to meet that even it was just a constraint in their design and we will we have some exciting stuff coming up and if Sushana is open to it I'd love to see if she has any views on it as well because in addition to the systems and tooling there is also the entire angle of the economics of it well initially when you see something that is free and nice your gmail your google search your social media profiles you don't know the cost of it up until much later but today what we knowing what we know about the the economics of it and how that privacy sliver of the of the whole ethical by the privacy sliver how it's being exploited and whatnot if you could change the economics of it that itself might be very interesting so for example there's a new search engine that's coming out called niva and eva unlike duck duck book which is which is a privacy privacy based search engine but they do serve you contextual ads in niva's case it's a paid subscription you want to search you will forever be completely private completely anonymized but you pay you pay for it so what earlier thought of as being free and we got used to it I mean for the longest time at least you know for for many years we could not have imagined with google existing why would be paid for it but now when we see what is what can go awry then potentially the the thing opens up again and of course it will have to be subsidized if it only takes up it'll have to be subsidized by the very rich who are most concerned about their privacy eventually driving all of the price down well that's where a role the role of economics and carrots aligning can can play a big part so I think this is where you know I'm kind of definitely wearing my ethicist hat and I'm going to ask you know again throw out a provocation out there you know is privacy a right or is it a luxury you know should privacy be a premium feature in a product or should it be baked in as a matter of course in the free version of the product and and this is something that I've written about before and I find you know this is something that kind of troubles me deeply as well that when I see you know a lot of people and I'm going to get very kind of practical here right so when I see you know the women who help me at home for instance when I see the way that they use their smartphones right their understanding of privacy is not very sophisticated but they do have an understanding of it and they do see that there is a gap here that you know they do have an understanding that they're having to sign away some of their rights and you know that they do understand that data is an asset which they no longer have any control over so is this something that troubles me quite deeply that you know should privacy just be a luxury feature or you know should we be using I think these are areas where regulation and and sort of government interventions can really help should we be requiring privacy as a matter of course yours is a very practical question because I think if somebody can if a world organization I think it was the UN that can declare the internet a basic human right this seems like only one or two steps from that it's not far fetched what you're saying. Yes and of course if you look at whether we want to admit it or not right social media platforms you know they have become part of our democratic infrastructure and so given that reality you know should we be asking these questions once again you know privacy is a right then how do we you know I'm not necessarily saying that we should be nationalizing social media platforms in order to protect our rights and privacy you know but how should we be governing these platforms. Yeah agreed and actually to that point and to Venkata's earlier point any thoughts on beyond the individual mindset and the individual value system what kind of systems and processes might actually aid this from a little bit more of a technology flavor to this. Yeah absolutely you know I was I was sort of hoping that we would get into that right so okay so let's see I want to break this discussion down into kind of four parts if that helps right so I want to talk about the early phase of the data science workflow where you're essentially doing split-readed analysis you're maybe framing the problem you're doing solution design right and then the latter part of the workflow where you're actually prototyping a model and then you know the end stage of it where you're deploying it and you're monitoring its performances in post-production right so in each of these stages like what are the things that you can do and what are the things that you should consider right so okay so let's see one of the things that I would strongly recommend doing upfront is this idea of registering your ethics goals and what I mean by that is not that you're registering with some external third party right although if third party audits of algorithms become a thing then that might be a good thing to do but the idea is that whenever you're building an AI system it's a good idea internally to at least articulate your values and to talk about your ethics goals for the system right what are the fairness criteria that you wanted to meet what are the interpretability criteria that are relevant for the system to have what accountability structures are you going to set up how are you going to communicate with your street holders about it right what are sort of foreseeable points of failure what kind of adversarial attacks do you anticipate you know what and if you expect data breaches are inevitable I would say at some point so what would you do to guard against something like that right so developing a checklist of this kind right at the beginning right and then also kind of recording your data governance procedures like your entire data pipeline where is the data coming from right what kind of consent and choice have users actually had in generating the data and how are you recording that and keep track of that I think all of us have had these poor experiences with these cookie policy pop-ups that we now get on almost every single website that we visit it's such a it's a classic example of ethics washing right you know you you were well I don't need to explain that any further so are you are you really giving your users any choice or control or any real meaningful consent here so there are no right or wrong answers all of these is a long spectrum and again I know I sound big but it's hard to do this without concrete please tell me right so it's a good idea to articulate where on this sort of spectrum of values you are and where you want to be and where the gaps are and how you're planning to address those gaps right so that's in the design phase and if I I mean I'm constricting back to the life of an average data scientist right in one of the many companies and many folks that we meet the the challenge that I am seeing at some level is that this requires an overall organization level alignment that this needs to be done this needs to be done is a certain way and that this is actually be this will be a valid consideration for the discussion because later when the model will be evaluated against some of these things right again some of these considerations so the can an individual how how how how can a how can a data scientist function in a system where not everybody is aligned to this objective whether it is top down or bottom up the even if the leadership wants the average algorithm the average data scientist looks at increasingly looks at himself for herself as an algorithm engineer and trying to maximize a certain accuracy measures as opposed to the broad based measures that we are talking about so how do you how do you see that friction in the organization around these objectives right so I think you know one of the simplest ways that I see of addressing this friction is to ask what is the cost of being wrong you know what is the cost to the to this particular product what is the cost to the organization so I think we are already seeing in the European Union we're already seeing cyber insurance policies for corporations that take into account what GDPR fines might have to be paid for violations right so insurance policies are already protecting organizations against that so the reason that I'm calling this out is that I don't think it's the case that there isn't organizational buy-in for thinking about ethics at every stage of the product because if you at a very pragmatic level just boil it down to the cost of being wrong if the cost is high enough no organization wants to be wrong so it's about in going back to Indra's earlier point is about aligning the economics of this thing there is a price for not being fair right because of liability or any other reason the entire toolchain will align itself the people the process the product yes and in the end you know isn't that why we advocate for market forces isn't that you know we hope that eventually market forces will correct things right so I you know I don't always I think I could you know we could go down this whole rabbit hole of like edge cases of capitalism and I don't want to go down that rabbit hole just now but yeah I'm almost imagining and even as I imagine it I also discard it which is imagine a data ethics score a certification score that this is where you're used to when we certified you based on a number of different audits but because I know from your own past experience you have helped audit some of these things has that does that play into any of what we're talking about here have you ever gone back with recommendations to people that you work with on the dimension of ethics right you know so this is something that's very close to my heart and I really want to explore this idea for a few minutes right so ethics is not so far from best practice ethics is not so far from doing your job but so you know just to give you a complete example right so I think I'd be surprised so you know one of the tools that I love using in the exploratory data analysis stage is partial dependence plots right because and the reason that's so is because it helps me get a sense of the relative contribution of the relative importance of features to whatever I'm trying to predict you know whatever I'm trying to build and this is in and of itself a great exploratory data analysis tool even if there were no ethics questions at play it just helps you to maybe think through your modeling strategy better but it's also a great ethics tool right because it helps you to ask questions about process fairness for example so when you think about the fairness of prediction systems right there are two ways you can think about it you can think about the process fairness right what are the inputs to this prediction system and are they are each of these inputs fair are these things that I want to use to predict about people so for instance do I want to use somebody's race in order to predict their likelihood of committing a crime right these are questions that society should be asking itself right so are these inputs fair and then there is sort of the outcome fairness part of it right so what I predicted is it fair across the board to the people I'm predicting on now for the process fairness part of it something like a partial dependence plot is a great tool to use because it tells you what are the most important most predictive most information carrying features in your model and are these features that you are ethically okay with using for this particular prediction task so this is what I was talking about when I said that best practice goals are not necessarily so far away from ethics goals and maybe we need to reframe the problem in that manner particularly in our data science pedagogy right so when we where we mentor our junior colleagues or when we teach data science to people newly entering the industry maybe this is how the problem needs to be framed that it is not a tension between it's not ethics not aligning with your economics objectives and therefore it's a close talk rationalization exercise no but it's more a question of its best practice any not sure but if you zoom into this whole you the partial dependency graph that you're talking about right now what you're doing there is effectively applying your human judgment on the the or you're trying to infer the meaning of the use of a particular feature in the model which you are to bring the cultural experience you have to bring a lot of other experience to be able to say what does it mean right when I'm using a particular feature as opposed to that other feature now generally we are you know as a from whatever I have seen on the on the the street people are not asking that value judgment laden questions I was wondering somewhere in this whole process of development of the skill of the data scientists over a period of time somewhere that in the conversation we have to incorporate the ethics discussion as well right from fairly early on any sense of how do you train a data scientist to be very aware and have a questioning mindset or interpreting mindset yeah yeah you know that's why the science part of doing data science comes in and frankly I don't think there are any easy answers because how do you teach any human being to be aware and reflective and critical right not just you know not just in their work but in their life as well there are no easy answers here but I think we need to so that there are two approaches to it right you can start at the beginning of the pipeline where your your data science curriculum explicitly includes ethics and not as an afterthought but as very much a part of project based learning and you know ethics discussions happen at every stage but it's also that we need to begin practicing it in our own work so that you know our junior colleagues look at the way that we do data science and learn from us it was very nice to see one of the biggest database names H.V. Jagdish I saw his his course contented out all of this this I would not have expected this as a young graduate student even like some you know 10-15 years back what was it specifically Venkata that addresses he's he's one of the biggest names in the database he has he so one of the stalwarts in the community has actively recognized that this is an issue and he is you know discussing debating running a course and training the next generation of database as well as data engineers around these kinds of questions it's good seen it from other other big names for example there's a lot of discussion around technologies but even the fact technology oftentimes gets boiled down to mechanisms and it becomes and misses the the larger choices trade-offs and things like that it is yet another metric that needs to be optimized that is I mean algorithm engineering is what I call it yeah no absolutely I think you know we need to acknowledge the fact that a key part of a data scientist's job is to exercise their professional judgment to choose the right tools and algorithms and this is a skill and it needs to be taught and it needs to be developed right so there is always going to be a bit there's always going to be the next full algorithm that comes along right and you just sort of blindly apply it everywhere no that doesn't make any sense you need to look at your problem in its context and decide what algorithms and tools are appropriate right so that's already a part of that that kind of a critical evaluation of algorithm's metrics accuracy metrics is already a part of our workflow I mean that's what makes a good data scientist right so it's a matter of just extending that to asking you know all of these ethical considerations checklist as well so you you talked about the first and the second phases you were before we broke off into this side thread right so so you know going back to sort of yeah so we talked earlier a little earlier in the discussion about what is the cost of going wrong right and this is this is something that has I strongly advocate for in the design phase as well and in the model prototyping phase as well once you do have something tangible in the form of a model that's predicting something you know you start asking what is the cost of false positives and what is the cost of false negatives right this is something that I strongly advocate for and then the the next thing is also you know just looking at documentation and reproducibility because these things have a domino effect on your interpretability requirements later on so you know let's let's just sort of talk about these different terms I think they get thrown around a lot so there's there's interpretability there's explainability there is transparency you know accountability so here's how I'm using these terms right so I'm talking about interpretability with respect to a specific machine learning algorithm and when I say transparency I'm talking about the largest system as a whole so this ML model sits as part of some software system you know which predicts something and then takes a business action and I'm talking about transparency as the transparency of that whole system so unless you have good documentation good data governance practices analysis you're confident that your code is reproducible your work is replicable it's very difficult to have real transparency at the end of your pipeline you know it's not sufficient that you choose a particular interpretability strategy you know let's say that your model splits out shafty values or your model spits out you know some confidence score in addition to predicting what it's supposed to predict right and it's very critical so this whole critical thinking piece right it's it's not a good to have it's almost essential to the quality of the work we do so there was some very interesting research recently out of I think Dubingen in Germany you know where they asked this simple question what's the you know what's the difference between a machine learning system and a human brain when it comes to processing pictures of cats and dogs and animals and classifying them right and what they found was really intriguing they found that human beings tend to focus on shapes in an image right so if I'm shown the image of a cat I would tend to focus on the shape and not so much on the texture but a deep learning system trained for the same task would typically tend to focus on the texture because the texture gives it an easy win you know and this is this has all kinds of interesting it's very exciting for me it has all kinds of interesting implications for for generalizing to other tasks well for robustness you know and the reason I brought this up is to say that look critical thinking is not a good you know it's not a good to have skillset right it's essential the reason that this team was able to generate these super exciting results which will probably guide research for the next couple of years is that they were doing a lot of critical thinking and and what's really going on here is once you understand the difference between how a machine learning system might approach a problem and how a human being might approach a problem it allows you to anticipate a lot of things that might go wrong downstream so why is it that the deep learning systems are learning to you know latch on to texture as the the most informative piece of the puzzle because it allows them to optimize their accuracy it's as simple as that but what does this imply so texture is also the you know is also that set of features which are particularly vulnerable to noise so if you think about you know just close your eyes and imagine for a second what would happen if i were to take a picture of a cat and add either certain kinds of specific noise or just general white noise to the picture right so the shape information of the cat would likely be preserved but the texture information would begin to lose a lot of detail and so that's why when you add noise to these images a lot of these deep learning systems are you know vulnerable to adversarial attacks that are noise based they their performance degrades pretty quickly so what does this imply this implies that in order to build you know more robust less vulnerable systems you actually want to build systems that are able to understand both the shape and the texture and pick up on both of these right so this is taught you something you know that kind of critical reflection has taught you something useful about building less vulnerable systems this I said earlier point about ethics not being very far from best practice right exactly exactly documentation all of it is a big part of the conversation these days for us also because you need the model to be itself predictable and stable if you will today if it is results tomorrow if it gives some other results how can we even interpret any at any point in time or how do we even debug it right so yeah interestingly what is happening is separate thread is emerging from the need to build robust systems all of these mechanisms are being built and it seems like a small jump from there to be able to now understand these add a little bit of interpretation color discussion around these best practices itself to be able to the next stage I'm so glad you brought that up because you know that was going to be my next point about this particular study as well so you know what they found when they sort of extended the research was that if you ask the returning system to articulate its decision-making mechanism right in a sense you asked it to generate an explanation it actually got better at recognizing shape in addition to texture because it was given this additional constraint of explaining it explaining its decision right so here is you know that the reason that I'm perhaps kind of harping on this is that I really want to highlight the connections that exist you know these are not these are not isolated considerations you know fairness on the one hand and answer of you know robustness to adversarial attacks on the other hand or interpretability these are not isolated parts of the puzzle right they are all quite interconnected you know having an interpretable machine learning model allows you to probe it for lack of fairness for instance so there are all of these connections you know and so the thing that ties it together is really critical thinking and and sort of you know practicing that critical thinking at every stage of your workflow and not just at the design phase or not just during deployment or not just during performance monitoring right and the end to and discipline yes and that's that's the science part of doing data science absolutely we are going to in the next session we are going to talk a little bit about the science in data science the experimentation mindset and things like that so we are up to our Indra we can't hear you I said that's unfortunate that we are up to the R because like I had predicted earlier there are so there were there were so many threads that emerged out of this where is consent and harm you know where where you draw the line where do values play a role and even this this thing where I was doing my quita about the difference between accuracy and fairness where you can you know define accuracy going down one path and fairness being somewhere else whereas you hold a contrary view that sometimes we can come together there's so much to probe but we do not have the time here Zainab says Sushana we want you back and I completely completely agree thank you Zainab I think we have lots to talk about so thank you so much for making the time any any closing thoughts from you Sushana that you want to share with the audience absolutely so I think you know we raised a whole bunch of questions and that was quite intentional you know I hope that the audience is not disappointed that we didn't come up with a very handy limited convenience set of checklists and tools we'll try to put out a resource in a few days you know where we do have some of these links you know tools and resources for you to explore further but I really wanted to sort of you know bring up to surface all of these questions to make the point that we should be thinking about these things as a community there aren't you know there's no one size all fits right answer for and AI ethics is hard precisely for this reason you know and so I think I would invite on every data scientist you know every ML engineer every product manager everybody who's been part of the discussion today to to start thinking about these things and to tell us where you feel we should stand you know like like what are the what are the values that we should be articulated you know are there certain values that should be like I was talking about earlier you know are there certain values that are really non-negotiable you know it's privacy non-negotiable for example and you know who should be deciding these things if we don't yeah absolutely so on the one hand all the listeners will have the opportunity to go to the has geek page where they registered for this or found out about this and add comments if you have questions for her which we will relay her way but if they want to connect with you is there somewhere online that they can find you yeah Twitter is great yes how can they find you what's your handle it's wonderful thank you is there in our posts as well yes as host this is this is a very important topic for us we'll stay on this for many weeks and months to come we are happy to facilitate further conversations on it the audience has views definitely they should write to us and especially as an ML engineering company we are very interested in mechanisms that will help with all of this auditability big words for us very central words for even us as an organization right transparency and we are happy to engage in more conversations around the mechanisms yeah wonderful wonderful well thank you so much Sushana Venkat and thank you Venkat this was a really wonderful discussion and as usual thanks to Amogh and Zainab from has geek for facilitating this all of the intro all of the streaming thank you very much thanks take take care everyone take care bye thank you