 Dr. Chin is the Richard Perillo family professor of healthcare ethics and an associate director of the McLean Center. Dr. Chin is professor of medicine with extensive experience caring for vulnerable patients with chronic diseases and a national expert on healthcare disparities. He's an associate chief and director of research for the section of general internal medicine here in the Department of Medicine as well as the director of the Chicago Center for Diabetes Translational Research. Dr. Chin leads initiatives to improve disparities in healthcare on the national level with the Merck Foundation and with the Robert Wood Johnson Foundation. Additionally, very importantly, Dr. Chin was one of 80 physicians nationwide elected last year to the prestigious National Academy of Medicine. Today, Marshall will moderate the panel on health policy and community health and will begin with his own talk, the title of which is Behind Me. Please join me in giving a warm welcome to Dr. Marshall Chin. Thank you all for being here. This is the 4.15 session on Friday and so you guys are the loyalists in the hardcore here. So thanks so much for hanging out with us this last panel. And the other three speakers, Ellen Fox and Stacy Lindell and New Pellani, they're all outstanding engaging speakers. So we'll try to keep this lively for this appetizer before dinner. And so just a couple questions that how many people this is the first McLean conference you've been to? What's the question? The first McLean conference you've been to. And let me preface this once by saying that this is the 4.15 session so you can't raise your hand. You've got to respond with noise. If you're clapping, you're saying, yes, again. So how many people is the first time at a McLean conference? Okay. Yeah. And how many people this is more than the first time they've been at a McLean conference? Okay. Great. Yeah. And how many people are enjoying this conference? You can do a lot better than that. That we've had some great speakers and it's incredibly important for Mark that you're enjoying this conference. So who here is having a good time at this conference? Okay. Great. So Ellen and Stacy and New Pellani will try to rate that energy as we bring you home to the 5.30 dinner. Okay. So I was telling Kay at the break that this is in some ways a scary talk but one that can have a happy ending. Important disclosure. So the co-authors on the main paper I'm going to talk to you are from Google. And three main goals here with this particular talk to define machine learning and why it is important to identify ethical issues with machine learning and to outline ways to ensure that there's fairness with these different machine learning approaches. I'm going to start with two case studies looking at identifying patients for care management, both with goals of reducing length of stay and identifying high cost patients. I'll explain what is machine learning, spend most of the time talking about the health equity issues that can arise in machine learning and end with recommendations. And so this is a paper that came out about a year ago in Annals of Internal Medicine. And it's one of my papers that has had the most media attention. You see up there that the popular press, the Wall Street Journal, Forbes, Consumer Reports picked it up. The scientific journals like Nature picked it up. IEEE which is the journal of like 400,000 engineers picked it up also. Also a lot of interest in this particular article. And one of the things they all asked about was case two in the paper. Let me explain case two. So imagine there's this hospital, this hypothetical hospital which we will just say unartically has the initials randomly you see and comes from a city that has a map that sort of looks like the one on the bottom there. And let's make believe that like the folks that were running the medical center thought very justifiably that it's a good idea to get people out of the hospital sooner. I mean it's good for the patient because we all know the hospital is a dangerous place so as soon as someone is healthy enough it's really good for them to leave the hospital. It's also good for the hospital because first that you know if you're being paid on a per admission basis you save money that way. Then also it frees up a bed and it's good for the hospital and it's good for the community if you have you know an empty bed filled with a patient that needs it. There's a lot of good reasons to basically have efficient flow there. And so they thought that if you could identify the patients who were most likely to leave the hospital sooner you could then give them additional case management resources to make sure that any missing barriers were eliminated so they truly did get out of the hospital sooner. And so they have a data analytics shop at this hospital and then they decided to yeah we're all good. No problem. We're all good. So anyway they have a data analytics shop actually a very excellent data analytics shop and they came up with a model using clinical data to predict who would be able to leave the hospital sooner. And they found though that like if you add patients zip code they came from you could actually improve the accuracy of the model. You did better in terms of identifying who would be those patients who would leave the hospital sooner and therefore it would get additional case management resources by the plan. The problem was they found that the patients who were ready to leave the hospital sooner were basically from the more affluent wider zip codes. So basically the people who were doing well that frankly didn't need the resources as much as the patients who had more social issues other issues that truly require case management. And to their credit the folks who were in this operation identified this as a problem and caught this soon enough and said well we're not going to deploy this model. We're not going to deploy this case management system yet and we're going to basically come up with a system to ensure that this does not happen and in fact we can bear yet design systems that would more proactively advance health equity. So actually the University of Chicago has become a leader in these different efforts. The second case study is one that you may have seen in the papers. You've also got a lot of coverage the past two weeks. High profile article in science by researchers that were originally in the Brigham Women's and Mass General at the time I believe. And basically what they studied was this company called Optum was a data analyst company and there's a lot of health centers and medical centers that hire companies like Optum to do data analytics. Sort of the same thing I'm trying to identify who are the patients that you might identify for care management. In this particular case they use the outcome of costs. They want to predict who are the patients that are the higher cost patients and then like oftentimes then health centers will then use those that list of patients for their care management list of patients to devote or resources to. What they found was that when you use costs as the metric if you're an African-American patient you had to be sicker than a white patient to be identified. In other words an African-American patient and a white patient who had identical costs generally that African-American patient was sicker. Had more bone marrow conditions. And you sort of think that through and it probably has to do with like well we're not spending as much money on African-American patients in terms of like either the barriers to access or maybe there's differential care unless people enter the system. So a problem. And this is like the tip of the iceberg. So Optum's analytics affect millions of patients and it's probably not just Optum's analytics all the other companies too. So literally the estimate was like a couple hundred million people in the country are affected by algorithms like this which are biased. So contrary to popular opinion this is not a picture of Tom Cruise doing a presentation at a Scientology convention. This is actually from a movie called Minority Report which who was serious in Minority Report. It's actually I think it's a great movie that if you haven't seen it's worth looking at. The basic premise and I won't spoil it. You see two thumbs up there on the movie review. The basic premise is that Tom Cruise is a police officer in the future and there is this analytics system where they can actually predict who's going to commit a crime and predict it before they actually commit the crime. Then Tom Cruise's unit goes in there and basically prevents the crime from happening before it happens. But you see the ethical problem here in terms of crime is not been committed but you're basically predicting who's going to have the crime. And when I say it's sort of a scary story and this is what I meant Kay that like it turns out like this is embedded in basically almost all important aspects of life whether we know it or not. And so this is book weapons of math destruction by Kathy O'Neill so she goes through things like how you qualify for a mortgage, the college application process, things like how judges decide how long a criminal sentence should be, which persons should be the subject of inquiry by health and human services in terms of child neglect issues and all. Basically all these different areas there are these algorithms that are biased and that have real implications for people's lives including probably everyone here. So it's sort of hard wired into the system at this point. Okay step back what is machine learning? Who here has taken a computer programming course like in high school or college or at some point about half the people. So you probably remember in your course these statements. If grade the 100 then do this. If less than 100 do that. So a lot of if then statements. Machine learning takes it to another order complexity. So basically your mapping is learned by the system given only input examples represented through a set of features together with their desired outputs referred to as labels. The example I will use for this is something like the electronic record and all the information electronic record. These machine learning systems will like not look at one piece of information or a simple if ands but look at this whole body of information in an electronic record these so called features and then find out what's correlated with the labels. So it's a different order of magnitude in terms of the amount of data that's required and complexity. And so I'm going to describe where the biases come from both model development high development model and high deploy the model. This is the main figure we have in our paper and I'm going to go into each section in more detail. The left part is model development the right is model deployment. Next slide is going to be the model deployment development side. See this particular slide I'm not going to be complete that you can go to the article for like all the different examples but it gives us a flavor of the types within each category. So look at the left hand side of the figure there's like for example minority bias where for example if you just have a data set that doesn't have a lot of minority patients well you know you're not going to have an accurate model if you just don't have a lot of patients to develop it upon. You see in the middle there's that missing data bias so for example if you have good insurance you know probably you're going to get most of your care in one different place. If you don't have insurance, you have bad insurance you're more likely to basically get care from multiple hospitals, multiple systems so your data is going to be spread out across different systems so any type of algorithm that's built upon data from one system won't be as accurate for you because you have relatively incomplete data compared to most people here probably have all their data most of the data in one healthcare system. The right hand side, label bias example that would be that there's some evidence for example that I think minorities for many types of mental illness may have more somatic physical symptoms compared to white populations who like a lot of the DSM criteria are these like behavioral psychological terms. So if the underlying criteria by which someone is labeled with a condition is erroneous well then again lots of data in, lots of data out. The next title will be model deployment one. This is the patient side. So for example you see the bottom left there agency bias would mean that well maybe if you are a poor person or a risk-free minority you weren't part of the process by which these models were developed and deployed. Informist trust would be for example if there's a legacy of distrust in the community has for the medical center they may distrust any type of algorithm or a system that's built upon something like the healthcare system or the developers. Something like privileged bias maybe you develop a model that requires technology that is not affordable by the community. So for example maybe you are using an algorithm developed for smart phones as opposed to texting phones. This last one this is the clinician side so a couple of examples here. Automation bias would be for example if the clinician is blindly accept what the algorithm does without questioning it the flip side could be dismissal bias where clinicians know that the algorithm is lousy for some populations they may automatically dismiss it and so they can be biases built in those ways. The full model. Okay so here's the ethics part. So I've talked about the problem what are, specifically it's a distributed justice problem in terms of these inequities and here are three potential approaches for a solution. Equal patient outcomes, equal performance of the models, equal allocation. So equal patient outcomes the idea here is that the model would lead to equal outcomes among different patient populations or better yet equalize meaning that if there was a disparity beginning with you've narrowed that disparity the challenge with this though is that the assumption is that we know what to do to intervene and that it will be done. I spend most of my time in the day job working on the intervention side so we actually do know a lot about what works to reduce disparities. The problem is that these interventions are largely not being implemented as well as the way we pay for care does not support and incentivize. For example things that David talked about this morning or something that someone like Selwyn talked about at the address social terms of health just largely aren't paid for. So short of outcomes you might think about performance and then think now back to your clinical epidemiology or epidemiology course that you may have taken. The idea here is that you can make a model that performs equally well across groups so like a more and less advanced group for metrics such as accuracy, sensitivity, specificity, positive predictive values. You remember those buzzwords and epidemiology and let me make a concrete. So for example it may differ which characteristics you would want to prioritize to optimize depending upon the issue. So for example like at the University of Chicago there's a group that has created these algorithms to determine who is looking like they're on the general mess and floor and looking sick enough that we think there's a high likelihood they're going to start going downhill and require ICU admission. And so the idea here is that you don't want to have a sensitive model. In other words that if you're in that category of participant benefiting from ICU admission that the algorithm will capture. You don't want to have another diagnosis for example of a poor patient, for example, or a race or minority in that category. You want to have a high sensitivity. Example of specificity where you would want to avoid have a high specificity would be something like say like which moles have a high false negative rate because otherwise you'd have a lot of unnecessary biopsies in your disadvantaged population or positive predictive value. Example of that would be I mentioned that thing about like the automation dismissal where you don't want to have a lot of false positives because if you keep having false positives the clinicians would start ignoring them because they realize the algorithm is not good. It's important enough that equal performance does not necessarily equal outcomes because there's a lot that happens after like a clinician would be fed the information. So again, equal performance does not necessarily equal outcomes but again, so it's complimentary to an approach that emphasizes equal outcomes. The third approach I'll discuss is equal allocation. So getting back to that University of Chicago example of like remember the case two where they talked about trying to allocate more case management resources to the people you thought would leave the hospital sooner. So it's an issue of like who would get the case management resources. So basically you can pick cut points for who qualifies that would ensure that well for example in this case like you know if you're from a South Side African neighborhood you have equal access to allocation of that case management resource as if you're coming from like an affluent white suburb. So you could do that but keep in mind too that this does not necessarily correlate with actual need. In other words you could pick a criteria for who would get then like the care management resources but again it doesn't necessarily correlate with need. So the example I'm going to give you is like coronary artery disease and cardiac procedures. So there's some literature that indicates that if anything white people get too many unnecessary cardiac catheterizations. So for example if you were to create a system to equalize that and make sure that more African women who are getting fewer of these catheterizations actually get the same rate as white patients you could equalize that but you actually may be getting equal access to an unnecessary procedure. So I'll say that it's not a clean in terms of what's the best way to do this. It has to be sort of thought through. So it's no cookie cutter solution but I would argue that like if you start thinking about the example I went through you can actually think about like this rational way to sort of think things through and so in practice a good team of people sort of thinking this through you could probably come up with a reasonable actual set of choices of solutions coming up with both a reasonable algorithm cut points and ways to allocate. So I'm going to buzz through this so we have some time for question and answers. Recommendations. Basically the major point is that at each step of the process from developing to deploying the algorithms you can design ways that basically intentionally ask are we introducing an equity problem and basically or better yet again can we proactively advance health equity. So for example I'm going to highlight the bolder parts here when you think about the goal of the algorithm you should review with diverse stakeholders including patients that to ensure that the goal makes sense. So for example I would argue that like when you have like optimal picking like cost as the major metric as opposed to a health outcome I would guess that like patient groups would say this does not make any sense and would push for the more worthy goal of patient outcomes and patient health outcomes as a metric. Discussing ethical concerns, deciding which groups are the vulnerable groups, making sure the data is not biased. The trained data or the data that the algorithm has developed upon making sure it's adequate representation of the vulnerable groups within this data talking about the fairness goals that we have discussed and then once you think about deploying the model then track it to see whether you're running the problems with different metrics. So for example, you know looking at well is there some type of disparity who's actually receiving the resource for example does the deployment data that you're actually implementing the algorithm is this similar enough to trained data that is a valid algorithm is the algorithm useful to clinicians and this actually recommended before we actually launched the algorithm again testing basis with all stakeholders and again once you actually deploy the model systematically monitor it with these metrics over time you need to think of a clinical trial to study whether it improves outcomes or not and of course then getting feedback from clinicians and patients. So I'll end with a slide that there was an editorial from a Stanford group that accompanied our paper and I love this quote which was maybe like in their last paragraph the only solutions is to apply to artificial intelligence algorithms the very thing they are designed to supersede human intelligence. So maybe that's one of the themes of today in terms of the importance of the human touch human communication the human person so even a high tech thing like machine learning you got to insert the human judgment in there to make sure that there aren't these problems that arise and again to decide in the best possible way. So that's my talk thank you very much we've got about a minute then for question answers and so maybe if people love to answer questions. So you were talking about how so the case two that got a lot of attention was enhancing inequity and then when you were talking about what you do with machine learning to change that is to make groups affected equally rather than equitably is that something that can work like how can you have machine learning work in actual ways even introducing this complexity to create different groups is difficult and increases the level of complexity that you're actually doing Yeah so the question was I'm not referring this way the question is this is complex using the University of Chicago example how can you actually sort of address this in a way that addresses that complexity so let me actually finish the story so it's a really good group they're both really good technically and they're really good people and really sort of values driven people and so again they were horrified when they found out what was happening and so they ended up partnering with the university's diversity and equity committee and folks who tend to have an equity lens on things and so there's a lot of discussions and sort of debugging this process and thinking about what are the different steps in the process by which problems could occur and then how can we hard wire into the system checklists or like checks on ourselves to make sure that at each stage like the last set of slides we specifically figure out or think about how can we make sure nothing bad's happening and again better yet how can we be making good things so I guess some magic solution but if you bring good people together and a diverse enough crew and people take the equity lens then people come up with reasonable solutions Yeah So my name is Kevin and I'm the third year family medicine resident over at Northwestern so in my career in medicine you know I've seen this question kind of gets at how important you think it is for physicians to have competency and continue to be at the table when it comes to the development of technologies such as machine learning and clinical use because you know as I've moved throughout my career there's always that like one or two people who are like really feel super comfortable with like that or building applications that can do really great things for clinical care and patient care and then they graduate and like everyone's afraid to touch it or you know there's a problem with the EHR and everyone's just like I can't change that I'll never be able to change that I'll never be at the table for the discussion like what thoughts do you have on that and do you have advice or people on like thinking about actually getting that human touch to the technology in terms of like thinking about as individuals that you know you need to sort of focus as an individual things that you're most passionate about having said that there are some things which are just so important so I mean the thing about machine learning is not just medicine but it's just a system it's like college applications it's mortgage applications etc all different parts of our life if we didn't have basically the average person so the patient the public healthcare clinicians I mean it has such a big effect on what we do it's crucial that we're at the table and so basically it's an issue of activation empowerment which again is one of the themes of today and so again it doesn't mean you need to be the expert in machine learning however you know your input in terms of how for example machine learning affects you or your common sense in terms of like is there a value or ethical problem here you know it's really important to have clinicians at the patient at the table so this is a great question I'm actually going to move on I will move on I'm sorry in terms of like keeping us well on schedule our next speaker is Ellen Fox who well-known in our community a fellow an attorney that he's made for many years for many years ran the ethics center at the VA a huge leadership role Carmel has a couple of hands both as head of her own ethics consulting firm at Altarum Institute so very influential and Ellen's going to speak about a new national study of ethics consultation in U.S. hospitals current practices and perspectives of ethics consultation practitioners so Ellen