 Hello, everyone. Welcome to the Organizational Ethics Consortium, which is hosted by the Center for Bioethics here at Harvard Medical School. I'm Charlotte Harrison. I co-chair the consortium along with Kelsey Berry and Jim Saban. Today, I'll be your moderator as we think together about data ethics in the hospital systems of today. Julie, can we have the first slide, please? Thanks. So with the development and use of new technologies in our society, there's often a lag in identifying and understanding the ethical issues involved, as well as a related lag in devising appropriate U.S. governance. That's the one. That's the slide. Thanks. Many uses of social media provide familiar examples of this. Today, our panel will describe some parallel issues in the context of healthcare technology, where novel uses of data science, including artificial intelligence, are challenging widely accepted tenets of healthcare ethics, and outpacing existing public policy and regulation. In the absence of accepted governance from professional and political sources, how might healthcare organizations such as hospitals respond to some of these challenges? Before we hear from our panel, just a quick note about this consortium and the ways that you may want to be involved in the program today. Julie, can we go to the next slide? Great. So the central focus of this consortium is the ethics of organizations in healthcare. Primarily, we consider challenges and possible solutions for healthcare provider organizations. Occasionally, our guests have represented professional medical societies, biotech companies, independent research organizations and others. Another focus is the building of a learning community of practitioners and scholars who are interested in this comparatively underdeveloped field of healthcare ethics. It's a great benefit to everyone presenting and attending when we have the active participation of audience members in various modalities that are offered in this webinar format. We hope over time to facilitate introductions and collaborations of my audience members if folks identify themselves in the chat and Q&A. With that in mind, let's look at the ways that you can participate today if you'd like to. First, you can submit questions for the panelists at any time using the Q&A feature. Selected questions will be addressed at the end of the panel's discussion. Also, you can use the chat box, both to ask for technical help from center staff and to share your thoughts about the issues being presented again at any time. If you're so inclined, we hope you'll identify yourself when you join the chat or ask a question so that others will know a bit about you. Now, back to our panel. Next slide, please. Today, we have the privilege of learning from a multidisciplinary group of experts, and it's my pleasure to introduce them. Dwight Barry is a principal data scientist in the operations side of Seattle Children's Hospital with focus areas in critical care, neurology, quality and safety, and clinical effectiveness. He also works with hospital center for diversity and health equity to assist clinical leaders with quality improvement efforts to promote equity in the clinical operations. Dwight has degrees from Yale University, Texas A&M, and the University of North Texas. And after an initial career in environmental science, moved into healthcare analytics in 2012. Eugene Day, studied systems engineering applied to healthcare delivery at Washington University in St. Louis. He's been a principal investigator with the Department of Veterans Affairs. He's been in operations and analytics in pediatric hospitals since 2013, including the Children's Hospital of Philadelphia, and currently with Seattle Children's Hospital. He has published about equity issues in both the research funding arena and operations analytics and hospital operations. Elizabeth Matague has almost 10 years experience working in healthcare analytics. She studied biology at Wellesley College and then completed a masters of science in data science in the inaugural cohort at University of Washington. She's interested in ways that we can incorporate data ethics concepts into healthcare analytics. She currently serves as manager of surgical services analytics at Seattle Children's Hospital. David Danks is professor of data science and philosophy at UC San Diego. His research interests range widely across philosophy, cognitive science, and machine learning, including their intersection. David has examined the ethical, psychological, and policy issues around AI and robotics in transportation, healthcare, privacy, and security. He currently serves on multiple AI advisory boards, including the National AI Advisory Committee. Douglas Decoma is a professor of pediatrics at the University of Washington School of Medicine. He founded the Truman Cat Center for Pediatric Bioethics at Seattle Children's in 2004 and has chaired the IRB at Seattle Children's Research Institute since 2000. He is past chair of the Committee on Bioethics of the American Academy of Pediatrics and a former elected board member of the American Society for Bioethics and Humanities. Doug currently chairs the Secretary's Advisory Committee on Human Subjects Protection. A warm welcome to each of you. Dwight, I believe you're going to start us off. Hello, everyone. Good morning from Seattle. Good afternoon from the East Coast. My name is Dwight Berry. I'm a principal data scientist at Seattle Children's Hospital. So about five or six years, seven years ago now, I had a vice president come to me and say, we want to be able to evaluate the stress of our units around the hospital and maybe predict things that could allow us to intervene sooner. And so it's sort of straightforward to do these things with, with inside the hospital measures such as things like the number of nurses over time shifts, the acuity of the patients, the census of the hospital, things like that. So that's pretty straightforward. And Cincinnati Children's had already done it at that time, and we were thinking of doing it ourselves as well. But then a further question was, how would we measure the stress of units where the metrics aren't so obvious, like HR or IT or supply chain? And how would we figure out what they were doing in terms of their stress levels, their their happiness, what that sort of thing? Because it's not a straightforward. I was talking to one of the data engineers that I worked with. And just casually he suggested, well, we could just scrape social media. And I thought about that for a minute. I was like, huh, in one sense, we're already monitoring our employees, because if you use a computer that's the company's computer, you expect to be monitored. But this was something else. This was going outside and just scraping social media or our employees by name. And it made me feel a little uneasy. And the whole premise behind machine learning and AI these days is you can put a bunch of social information together to make inferences about people that they may not want you to know about. And about the same time, and this happened, I had read a paper by Jake Metcalf and Kate Crawford about the difference between sort of the big ethics trend that was becoming really popular and IRB protections. And they point out in this 2016 paper that the research regulations that we have in the US exempt projects that make use of already existing publicly available datasets. So even if this were research, which it was not, it was meant to be quality improvement. If it were research, the IRB wouldn't cover something like that. So a question for legal came up. Do we want to go ask legal if this is legal? Since we didn't have IRB protection for this, it is legal. And it turns out that because there's no IRB protection, because it's a single company, and it's the terms of services allow such things in many of these social media companies that, yeah, I mean, strictly speaking, it was legal. Was it hard? No. In a matter of a couple of hours, I had put together a way to scrape Twitter or sentiments about Seattle Children's. And I could use any keyword there instead of Seattle Children's, it could be the names of our employees, it could be the names of our particular clinics, it could be anything like that. And it's really easy to just get a sentiment analysis, it sort of breaks out the individual tweets, in this case, by what people are sort of thinking. And the way this happens is that is that the words in the tweet are scored in certain ways to provide that sentiment value. So higher levels would be more positive and negative values would be more negative. Truly easy to do this. But of course, there's a problem with these tools. You would expect that in the instance where there shouldn't be any differences in sentiment measurement, that there wouldn't be. But in fact, there is. Some of these things have been repaired by now. This is kind of an old example, but some things are not. As we can see by the workarounds and chat GPT that have been coming out. At this time, I'd also discovered a paper that was published in 2016 also and as a preprint, and then full, it was a conference paper that came out in the year after. But what this paper pointed out to me was that you can look at any sort of data ethics, data analysis problem, machine learning, whatever predictive and analytics. And if there is any imbalance in the groups, say there's more of white people compared to black or African American people, if there's any imbalance, then there is no way to maximize all definitions of fairness. Mathematically, just a mathematical statement. It's an issue of ethics at that point. So you have to choose. You have to choose what definition of fair you're going to maximize for in these models. So as a result, it's sort of like, you can see that we have to make some choices about these things because they're not fair by default. And they're not going to be fair even when we make a choice based on certain definitions of fairness. And people like me are coding these things. We're coding these things into our algorithms. And they are going to have profound effects on society as these things play out in the real world. So the nature of an AI algorithm means that it's designers, people like me have to make explicit what values and political choices it is going to serve. We can't hide behind. Oh, it's just math. Well, at that point, I knew that I had a problem. And that the ethics of trying to predict sentiment of our employee population was something that I figured we needed help for. So I asked for an official consult from our bioethics service. And with that, I'll turn it over to Doug to talk about his side of it. Thank you, Dwight. So yes, at this point, for perhaps the first time in our institution, data met ethics. That consult was done by one of my colleagues, along with one of our bioethics fellows. And then not long after that, I was approached by other members of the team, requesting that I put together some educational sessions for those within the data analytics group who might be interested and and that led to additional work. Next slide. So the context that this consult came in was an interesting one. We had our data science group that I think it's fair to say many of us were completely unaware. I mean, we knew that there were some information technology people at the institution, I don't think we knew the extent to what they were doing in terms of using data to inform the institution on issues. And the we really had very little that fit well for dealing with that as Dwight has already mentioned, the IRB really didn't have much of a role here because most of what our data science team was doing fell into the category of quality assurance or organizational work and didn't really fit the definition of research. So the IRB had no role. We really didn't have an organizational ethics group at the time. Most organizational ethics was handled by high level administrators and many of those decisions were being made without an ethics person in the room. Occasionally, I would get pulled into to offer some advice, but it was fairly rare. And so what we really had was this clinical ethics team. And this really wasn't clinical ethics either. So it really didn't have an easy place to fit. Next slide. So for most of us who were more used to sort of the kind of consults we got on the clinical realm, this was a bit of a light bulb moment. Most of us on the ethics side, and I think it's fair to say the clinical side had no idea what big data was being used for at our institution that that that it was being used for advertising and marketing and identifying new markets for the hospital services and the assignment of resources and the assessment of patient care. And lots of other things, some of which, you know, depending on the algorithm you use could introduce as again, Dwight pointed out some pretty significant biases. And also which, you know, when you do this work with AI, it's devoid of context. So to go back to Dwight's example, where he they just he simply just very quickly put together an algorithm where you could plug in, let's say my name, and then see on the web what adjectives would come up. What what what you don't know is whether those adjectives are related to me as a physician or to my academic work or to some other aspect of the work I do that has an online presence. And and so judgments end up getting made without the context that may be really important to to evaluate that. Next slide. So the concerns that were expressed by our data science group included some discomfort with the kinds of questions they were they were being asked to answer or provide solutions to the way those questions were asked and the potential for that to introduce bias. And also the implications of the answers they would come up with related to privacy and fairness and equity. Because again, with those algorithms, if you if your algorithm is biased, you're going to get a biased result. And then lots of other questions. And I mean, I one of the recollections I have, very clearly is one of the data scientists when we first met expressing their discomfort at actually, even though they were bound by confidentiality of the institution, being able to see patient level data that in some cases was quite sensitive. And it just gave them, in some cases, a feeling of being intrusive. Next slide. So just a couple of observations about this unique request to involve ethics in in as as the data science team move forward. It was in my experience, incredibly unique and forward looking. First of all, the request came from frontline data science people. I mean, these these were people who are in the trenches doing the hard work. And I think we were really impressed that that they were struggling with these ethical issues and willing to reach out to ask for some advice and help. And then as as time went by, they remained incredibly invested in this activity and and have continued to to be so through today. They as a group also highlighted issues of equity and fairness as as as they use the data and and we're struggling with that they were concerned about the ethics and not the legality, which again, I think surprised many of us because a lot of people want to know if they can get away with something, but they fall short of asking, well, should we do this? And this group definitely did. And what I want to say is that these requests from the data science team came before at Seattle Children's anyway, we had any kind of formal organizational ethics team or approach. And before we had a we're having a major emphasis on equity, like we do in 2022. And because this group actually, you know, was formed and started doing the work of 2016 or so, which is now six years ago, or seven years ago. My last slide, but and I also want to say that, you know, the emphasis on bias and equity, although we were paying some attention to that at the time, it didn't raise to the level that that people are trying to pay attention to to it now in 2023. And I think it's fair to say that in raising these questions. Our data science team was one of the leaders within the organization in in terms of having a concern about bias and inequity that that might arise from some of the work they were doing and wanting to be able to think about that. And so on. And you know, what it reminded me of was was sort of the evolution of our approach to medical error, where for decades, the approach was really, you know, either to hide or cover it up, not talk about it, ignore it. But then eventually, there was a move toward transparency and disclosure and systemic change with which, although not entirely the same, I think there's some similarities to what was happening, you know, for our data ethics group, and our data science group. And, and I think that's the goal moving forward that the use of data really not simply at our institution, but but more broadly needs to move more toward a a mechanism that that values transparency and looks at bias and equity by acknowledging and confronting those issues in an effort to change. Alright, on to our next speaker. Thanks. So as Doug discussed, there was a real engagement with the data ethics team. But one of the challenges is for analysts, data analysts, data scientists, to know when they should actually contact a member of the ethics team. If all the analysts are trying to do is to, for example, improve the performance of the servers to optimize that sort of task, there's no particular ethical challenge. So the question that came before us when Dwight reached out to me was, how could we help our the data analysts at Seattle Children's at the time I was at Carnegie Mellon University? How could we help them know when to call somebody? And so we've thought about this as a kind of almost ethical triage. How could people who are not trained in ethics know when there was a potential crisis moment in front of them, in this case, in the form of a problematic data analysis project, and thereby reach out to people who do have that training, the members of Doug's group, for example. And so we worked together to develop a checklist where the idea of the checklist was to make it so that very quickly, a data analyst could figure out whether there was an ethical risk that they should talk to somebody about. So next slide, please. We did this by trying to have a small number of questions, two for each of the major topics that were raised as concerns by both the data analysts and the data ethicists that could quickly be assessed by the person who's considering the new project. So if it's Dwight considering whether he ought to be scraping social media of all employees at Seattle Children's, he would ask, for example, these kinds of questions. And the idea of the questions is that they are written in language that is appropriate for the data scientists themselves. So some of you on this webinar might be looking at these questions thinking, that's not exactly right from an ethics perspective, or that's not even written the way I would expect to see it. But we worked collaboratively with members of the data analysis team to develop questions that were appropriate for the kinds of ways and language that they were using within their team. So, for example, just to take this first privacy question, as Doug was noting from the data ethics side, many people outside of the data analysis team were not aware of the amount of data that was being collected. Or if we look at the second privacy question, people were not aware of the kinds of inferences and targeting that was being done. Of course, targeting is the language of a data scientist. Other people might interpret that word rather differently than we would as data scientists. And so, next slide. We also looked and developed questions around not only privacy, but also bias, where the issue is whether there are biases in the data collection procedures, as well as in the data analysis process, because issues of bias and equity can arise in both of those places. Now, again, you'll notice that what these questions are deliberately not asking whether the data analyst could fix the problem. So, the first question asks were these collected or measured or processed differently between groups? If the answer to that question is yes, there actually are a number of technical responses that we can make using data analytic methods to try to compensate for those kinds of biases in data collection. Now, not all biases in data collection can be compensated for, but some can be. But you'll notice that this question doesn't ask, have you figured out how to fix the problem? And in this way, the checklist that we developed is rather different from the tools that are being developed at many of the leading technology companies to do things such as determine whether some AI system is trustworthy or responsible. We weren't trying to figure out whether the data analyst could fix the problem. We are trying to figure out when the data analyst ought to reach out to a data ethicist to help better understand the complexities of the problem they were trying to solve. Next slide. Especially in the context of medical institutions, transparency is really critical. It is important for the patients, their families and community members to trust the organization in the institution. And so we wanted to think about whether that trust could be supported by explanations about the kinds of inferences and predictions that were being made. So if we were to scrape a bunch of social media and say somebody is potentially a problematic employee or overall sentiment is trending downwards, could we explain it in terms of changes in the language that were being used? Could we account for predictions about how medical resources were going to need to be allocated in the coming months so that we avoided creating sort of the medical analog to a food desert where certain communities were being systematically denied or deprived of medical resources simply because of historical injustices? And so again, these were questions that were designed for the data analysts to figure out whether there were potential issues lurking inside of their project that could benefit from consultation with ethical experts. Next slide. We explicitly included questions around equity. These actually emerged organically through the discussions as the checklist began to be considered outside of the immediate data analysis team, that it was important to highlight and explicitly think about not only whether there are biases within the data or biases within the analysis, but whether there were larger structural and systemic biases that needed to be considered before moving forward with a project. Sometimes in data analysis, people will think about something analogous to the medical dictum to do no harm. And so people will say, well, my data science effort didn't make things worse, so therefore it's perfectly acceptable for us to use it. On the other hand, as these questions indicate, it was really critical for the data analysis team at Seattle Children's that their systems not only do no harm, but actually could potentially improve matters. Next slide. And so then these questions are useful. It's useful to think about various issues that can arise when we think about particular dimensions of the project, but of course we needed to have overall ones. And these are sort of the are we sure kind of questions that are really critical when we are moving forward with any data analysis project. We need to think carefully about whether, in fact, what we are doing is going to be legitimate and perceived as such by the communities that were relevant. Next slide. These then became brought together into a kind of overall scoring system. And the design of the scoring system was explicitly to find ways to minimize what we might call false negatives. If a member of the data analysis team had to talk to an ethicist and the after a 30 minute conversation they said, yeah, it's okay. There's no real issues here. That was far better than the alternative of somebody moving forward with a project that turned out to have real ethical concerns that were discovered only later. And so we developed this system, a scoring system, explicitly to try to make sure that the conversations were more likely to happen than not. And we tested this on a few isolated cases, sort of came up with our own intuitions about what ought to happen. But then the system began to be deployed. And for that I'm going to turn it over to the next speaker to talk about some of the experiences that the team had using this checklist. Hi, I'm Elizabeth Montague and I'm one of the analytics managers at Seattle Children's Hospital. And I'm here to talk about how I've used the checklist and kind of how we've used some a specific use case. So we've used the checklist as a tool to facilitate conversations about data ethics. These conversations have included data analysts, data engineers, data scientists, clinicians, leadership and others across the organization. You saw David talk about kind of the questions in the checklist and these questions and concepts in the checklist really distill data ethics concepts to be approachable by a variety of audiences. So in one use case, a department we work with was looking at a tool that analyzed text data, so an external tool in deciding if they wanted to use it. We met with the leadership from the department, members of the department would use the tool and then some analysts and data scientists as well. We went through each piece of the checklist and discussed together and we determined a score on each of the sections as David showed. In doing so, it really opened up the conversation and we covered concepts including bias, privacy, consent, transparency and impact. And the open discussion provided a forum to explore these concepts and consider the impacts of using the tool. The result of the conversation was that the team had some open questions to bring back to the vendor so then they could kind of further evaluate and determine if the tool was right for use at Seattle Children's. So I'll turn it over to the next presenter. Hi, I'm Eugene Day. Thank you very much. So I want to talk about like deploying those checklist insights into production within the hospital. One other example on the inpatient flow analytics side was the double rooming dashboard that was designed by a former colleague of ours, Aaron McKady, who was instrumental in this in the data ethics group. So like many hospitals, we will occasionally double room patients and when we were developing a dashboard to track this, we employed the dashboard, the checklist to identify ethical issues with such a dashboard. And while we found no such issues with regard to the idea of building a dashboard and tracking the double rooming, we recognized it as an opportunity to include information in the dashboard, which otherwise might have been neglected and essentially to surface equity and potential biases to the personnel making the decisions around double rooming in ways that they may not have otherwise done if they weren't tracking it and if ethics were not a fundamental sort of bedrock aspect of the data analytics and building these dashboards. So Aaron developed a trend visualization that essentially was able to track equity markers of patients placed in double rooms, race, ethnicity, language of origin, gender, etc. And that compared known patient attributes that are at risk for disparities to overall averages over time. This allowed us to make rapid, both qualitative and quantitative comparisons of subgroups to the overall population. So patients of a particular ethnicity should be over time on average in double rooms at the same rate as the overall average patient at our hospital. And this identified and enabled us to track those disparities because of course we did find some disparities and to correct them in double rooming as we moved forward. Next slide, please. So with that type of insight and what I think of as a sort of fundamentally different use of the checklist, it was not so much to identify potential biases in the data, so much as it was to identify opportunities to use the data to advance ethics and to correct disparities. What are the future directions for this? First of all, we want to proliferate the actual checklist across Seattle Children's Analytics Environment and to yourselves as other interested parties in data ethics across the healthcare space. I think it is especially important to consider these types of tools in the essentially unregulated environment of internal data product deployment. Research has IRBs and external structures for ethical oversight. Medicine, medication, the practice of medicine, medical devices are regulated, but building a dashboard to examine patients who are double roomed within the context of an internal deployment of that information is essentially an entirely unregulated environment and therefore ethics become having internal structures for those ethics becomes all the more important because there is no formal oversight. And so to do that and to embrace these things, we need to move towards the culture of equity as a value. Doug spoke about the evolution of medical errors and how we think about them from trying to conceal them to trying to publicly air them. And I think that that equity and bias is exactly the same way. There is sometimes the idea that if we conceal inequities that we are managing risk, unfortunately, I think that that cultural shift needs to be to if we are not publicly searching for and eliminating inequities that we are a fact de facto embracing them. And that requires essentially a process of vigilance. You know, we talk about pharmacovigilance and and studying approved drugs in the population where their use has been longer than their trial period. And therefore, rarer events that may not have been caught in the trial period can be identified after use in the community. Vioxx, I believe is a great example of pharmacovigilance being effective. We need algorithmic vigilance that that as we have these mature data products that are developed and deployed in our organization, we must not forget to consistently review them and to have a culture of examining the lifecycle of data products and to make that part of data governance so that we are appropriately monitoring the ethical use of data and data products in our organizations. And I will hand that to the next speaker. We have the checklist online when there's both the QR code as well as the we have the checklist online and that that's a quick link that will redirect you to a red cap instance that shows the data ethics checklist. And there's a QR code if you want to make it even faster. And then we also published a paper about it a couple years ago. And the link for that is at the bottom. And the QR code is there as well to discuss many of the things that we discussed today with you. So we can turn it back over to Charlotte for the moderation. Thank you, Dwight. And thank you, each of you on this panel for outlining the thinking and the process around the data ethics program that you've been developing at Seattle Children's. It's as Doug commented, remarkable to have this kind of work coming from the people most involved up to brought up to the ethics service at the hospital and to others to see what ways the institution could respond and really to be developing those ways yourselves. So it's a wonderful foundation for continuing to talk about these issues. I know a number of us will have questions here even more. But this is the moment when we turn to our external commentator for another perspective. And I'd like to introduce and welcome Alex John London, who will be our commentator. Alex is the Clara L West Professor of Ethics and Philosophy and director of the Center for Ethics and Policy at Carnegie Mellon University, an elected fellow of the Hastings Center. He focuses his work on novel technologies in medicine, biotechnology and AI. His new book for the common good Philosophical Foundations of Research Ethics is available as an open access title from the Oxford University Press. He was a member of the WHO expert group on ethics and governance of AI, whose report was published in 2021. He's currently a member of the US National Academy of Medicine Committee on creating a framework for emerging science, technology and innovation. He's the author of more than 100 articles and other publications and has so much to offer us today. I appreciate your coming, Alex. We welcome you warmly and look forward to hearing your comments. Thank you very much, Charlotte. I appreciate that generous introduction and the chance to be here and participate in this panel. So I'm going to offer some comments on the checklist that we just heard about. I have to say in the beginning that I don't have any conflicts to disclose regarding the material in the talk. And when I mention systems or products, that's not an endorsement. So the first general point that I want to make is that I really do applaud this initiative. There are ethical responsibilities that are diffused throughout organizations. And we sort of pick places where the ethical issues are particularly salient to sort of focus on. But we've got to do a better job of disseminating awareness of ethical responsibilities more broadly throughout these institutions. And so that's going to be a theme for my talk. But in general, you can't solve ethical problems that you haven't formulated. And so raising awareness and helping people formulate the ways in which their workflow is implicated in ethical issues is really critical. It comes to AI and data science articulating the ethical dimensions of problems as early as possible so that they can be incorporated into every aspect of system development is really important. So having said that, though, I think in order for initiatives like this to have a bigger impact and then in order to expand the reach and the scope of initiatives like this, we need to broaden our focus. So that includes three elements. The first is on the values that we're focused on. So I think it is absolutely critical that we focus on equity and privacy and the other values that my colleagues mentioned. But I think there's two others that need to also be highlighted throughout process. One is risk assessment that we really need to be sure that doubts about the accuracy or the efficacy of our systems get addressed so that there's sufficient evidence to believe that they're going to add value. And the second value is social value. So you know, we don't want to use data science just for the sake of data science or automate processes for the sake of automation. We want to do them to make our practices more effective and more efficient and also more equitable. And so we want to have incredible evidence that adding automation to a workflow or adding evidence from automation or scraping sentiment analysis from, you know, social media is going to actually figure into decisions in ways that are going to make those decisions better and not just make them noisier. So that's about values. The second is broad in our sense of the choice points at intervention development where we can intervene or where ethical issues might arise. And then the third is who are the stakeholders who need to incorporate these values into their decisions. So it's great to have a checklist for data analysts. They're an important stakeholder in this process, but they're one stakeholder with a much larger organization. And, you know, you might sort of there are sort of business ethics or ethics classes where the example will be something like, you know, your boss wants you to do this task. You know, you have ethical qualms about it should you do it. And now that's certainly a good way of framing the problem, but it leaves unaddressed the question, what's the environment in which the boss is working, where they've asked you to do this task. So let me take you through, you know, amplify each of these points with some example. So this recent paper of mine from cell reports medicine, I talk about challenges at each of the stages in the system development that come from kind of structural features of medicine. And I think a lot of these points are relevant into thinking about the present initiative. So task development and making sure that you're matching the task that you assign to system developers with the capabilities of AI systems is really fundamental. And medicine is an area where we've seen some pretty dramatic failures in that regard. So, you know, the slide up here, Watson conquers jeopardy. And then, you know, on leaving jeopardy, IBM says Watson's going to go and conquer medicine. And if you read some of the work that was published at that time about what Watson was going to do, it's really, it's really fabulous, right? He was going to, or it was going to, you know, autonomously automate the gathering collection synthesis of information. And it was going to make decisions autonomously. And of course, the architecture of Watson was that those things were were unlikely to ever come to fruition. And then you had institutions like MD Anderson that spent 62 million dollars in its cooperation with with IBM Watson and Watson was recently Watson health was recently sold it's reported for about a billion dollars to a private equity firm. But it's also estimated that acquisitions alone for the Watson program totaled $5 billion. At one point there were 7000 employees working on this project and they just couldn't make it work. And part of the problem was a mismatch between what they wanted it to do and what it was capable of doing. The idea of, you know, sort of having AI that can kind of search through medical evidence and synthesize for it and make the job of scientists easier is a perennial one. It's sort of what Watson wanted to do. It's what meta wanted to do with the Galactica one of the, you know, so there's been a spurt of large language models that have come on to the scene. Meta put Galactica out with this the idea that it was going to do some of these tasks in science. And I think it lasted three days. It wasn't just that it produced racist language, but also it made facts up because the architectures for large language models are just not well suited to logical inference confined to an existing database of information. So getting your task right with system capabilities is critical. Bias in data, there's a lot of discussion on this. It's very important. But in healthcare systems, the bias can stem from multiple factors and not just under representation of marginalized or disadvantaged groups or groups who've been subject to racism or other forms of prejudice. But the fact that same dynamics influence the data generating processes in our health systems. So, you know, the graph here in the picture, you know, is sort of an example where peripheral oxygen saturation measurements tended to over estimate oxygen saturation levels in black patients in comparison to white patients. And so the data then would sort of systematically assume that black patients were oxygen levels were sort of satisfactory when in fact they wouldn't be. That's part of the underlying data generating process. And so we've got to make sure then that we're not just thinking about bias and data as a relationship among the data that we gather, but about how the underlying data generating process generates that data. Otherwise it's going to be the equivalent of worrying about smoke damage without putting out the fire that's causing the smoke damage. If we skip ahead to, you know, assuming that we can produce models that are going to do what we want them to do, testing in validation is so weak spot for artificial intelligence. You know, in medicine, there's so much uncertainty. Most interventions in medicine and innovations in medicine fail or don't work. Half of those things in drug development context, half of new drugs fail at phase three after firms have spent hundreds of millions of dollars trying to develop an intervention. So they're often supported by, you know, solid evidence of efficacy, but it doesn't work out. Most AI interventions also don't work out. So we need real gut check about whether the systems that we've developed on the data that we've used to develop them produce the benefits and translate into the gains in effectiveness or efficiency or equity that we were hoping to get out of them. And so I think that's also going to require changes in organizational practice and the participation of a wider range of stakeholders in these organizations. The last piece is training and monitoring. So Eugene said we need algorithmic vigilance and I couldn't agree with that more. So the, sorry, so the paper that was here is a systematic review of, you know, the degree to which AI systems are subject to prospective evaluation. And the graph at the top is from a group that looked at sepsis alerts before and then during COVID. And, you know, this is an example of what's called distribution shift where, you know, sepsis detection systems are trained in an ecosystem where there's a normal rate of the various medical conditions that bring people into the hospital system. And, you know, the COVID changes the demographics or the distribution of disease in the population. And so, you know, you go from a certain rate of sepsis alerts to a much higher rate of sepsis alerts and the authors of this paper say one of the things you have to do is train people in your organization, not only on how to use the system, but to know when to decommission that system and what to do when that system has been decommissioned. And so I think this is about at various stages in the life cycle from conception all the way through product testing and then deployment, people have ethical responsibilities and we need to sensitize them to these responsibilities. And the last thing I want to say then is we need to broaden our sense of who these stakeholders are who have these responsibilities. So I think senior management gets a buy too frequently. A lot of our ethics discourse, especially in medicine and medical research focuses on researchers and clinicians. In these organizations, though, clinicians and researchers are often constrained in what they can do because of their relationship to the people who have the real power and who make most of the decisions. So I think senior management needs to be sensitized to the way that they play a role in each part of this and that the signal has to come from the top that ethical issues are really important at each of these stages in the process of using data to improve the way that we deliver care. There's other stakeholders as well, product procurement people, the EMR vendors, quality assurance teams and educators. So these interventions in order to succeed, they require the commitment of a broad range of people. And it's important then that each of the members in this ecosystem have a sense of why what they do contributes to the ethics of making sure that health systems perform their required function. It's, you know, safely, effectively and efficiently. And so with that, I will turn it back over to Charlotte, and I'm looking forward to the conversation. Thank you, Alex, for knowing this environment so well, as well as raising these issues which so complements the work that the panel had presented lots to discuss here with the questions that you've raised. And I'm wondering if we might, before we go further into audience questions, if we might have a period when people on the panel ask each other questions, or there's some things that you would like to discuss with each other. I know I have some questions in the moderator role, but I'd give you the first shot if somebody wants to do that. Well, Charlotte, I there's a question in the Q&A that that I'd like to know the answer to and I don't, which is I'm sure Dwight does. But you know, how our institutions using social media data, we talked about some of the issues related to that, but we didn't actually talk about whether that's a common practice. Yeah, so the a very long story short is that we did not move ahead on that project. And interestingly enough, it wasn't because of the ethical issues that I was concerned about. It was because of it capability. And so we didn't have the ability to put in some sort of monitoring system that outside of the EMR that would last that would be up 24 seven. And it wasn't willing to help us set up a system that would do so for a variety of reasons. So in the end, I didn't have to make any sort of decisions or pride people about the ethical dimensions of this because it just got cut off at the at that initial stage before we even started building models. I was happy to see it die though. Dwight, do you think that what happened in that project sort of word got around? Have there been other cases that other suggestions of doing that kind of thing again? Or do you think people have sort of come to know and be aware of the issues that concerned you? It it often comes up when when folks like me or Libby or Jean are interacting with with our stakeholders, the clinician leaders and management and so on. Because it's often things that they they haven't even thought about it. They just they've heard some sort of thing that I can do this for us. Let's go do it. And and then we end up having discussions and in some cases, test cases showing that here are some of the ethical dimensions that are associated with these technical problems. And it's we've seen that it's a lot harder for the junior analysts to do this. And it's a lot easier for those at the higher levels of the analytics group to do so. And so there's a definite power of symmetry there. Well, that actually leads nicely into the question of the voluntary system that you have and the fact that the institution isn't telling people with less power that they'll be protected if they raise these questions and educating across the the institution for folks who will be asking you for projects, you know, to come to appreciate the issues that you raised. What have you seen, Dwight, and the rest of the team at Seattle? What do you feel you've seen as sort of the advantages as well as the limitations of the grassroots approach that you've chosen and, you know, the the the attempt to really develop a culture in one sector of the of the hospital, the sector that people have to come to when they want a project like this, you know, how well has that worked and how would you compare it with other possibilities? If I may, I think. So, I mean, the the response from our partners, what, you know, we on the analytic side, we think of our partners as the clinicians and the hospital leaders who, you know, provide us with the the demand for the work that we do. The response from our partners around using the checklist and using and injecting, you know, sort of the ethical framework into the work that we do has been pretty uniformly positive. I, you know, I think Kelsey asked a question in the chat about, you know, the tension between moving fast and the slowdown culture of ethics. And I think that we are fortunate in that people recognize that here, that if we move too fast and we end up doing something which has consequences, that the the cost in human impact, time, money and rework outweighs the cost of doing things right in the first place. And that's true in a great many different environments of human endeavor, not just ethical data science, right? And so. But yes, there are a lot there are a lot of places, I think, in the world where this is seen as an unnecessary burden or a way of slowing things down. So I think that we're we're fortunate that we don't have that perspective. At the same time, no, there is not an institutional policy about being required to use these checklists. I think we didn't want that when you add an administrative burden to this work, it can become wrote and therefore essentially neglected. It is it is technically complied with, but essentially neglected. And I don't have the answers to solving that problem. I'm not going to pretend I do. But I believe that when you. Demonstrate when leaders in the organization, not just technical managers, but but leaders like Dwight and Libby, who are at the forefront of the work that we do, when they demonstrate that commitment to authentic and sincere engagement with these kinds of ethical tools, it is infectious across the organization. And that leads to a culture where we can say ethics is at the forefront of what we do because it's the right thing. That's a really helpful answer, Jean. And I wonder if anybody wants to add anything to that. Yeah, I will say for the some of these conversations, one thing that's helped is having multiple of us attend the conversation, some multiple kind of analysts or data scientists or data engineers to just help facilitate that. And especially kind of when you talk about the junior analysts, right? So making sure that we support each other. And Charlotte Charlotte, you know, from where I sit, which is definitely not in the data science group. Although I now have a son who's in data science, so I have learned a lot. But this was very exciting to me because, you know, I've. I've been doing ethics for a long time and the organizational ethics issues were always tough ones for us to crack because we weren't welcome in many cases. And one of the clear lessons to me was you can't force your way in. That if I had approached this group with any kind of a sort of you have to do this or, you know, you guys need some ethics or whatever, it just would have fallen flat on its face and all likelihood. And I think what was really an exciting thing to see was we actually had a group thinking about it internally and then coming to us as a resource. And I wish we could clone that to other parts of the hospital. I think there's clearly been things have clearly changed to some degree over the last, you know, 30 years that I've been doing this. But this was probably the most impressive, internally driven, let's, you know, let's take a hard look from an ethics perspective of what we're doing that I've seen. And it's what makes it successful. You know, it's a very common experience in clinical ethics and research ethics, IRB work, right? That people are worried about. Yeah, we want we want the advice, but we don't want to. We don't want to slow anything down as Kelsey has mentioned that, as well as obviously the tech culture. I'm wondering if there are other, you know, cultural or organizational issues that were competing goals that anybody cited in pushing back against if anybody has against the sort of checklist approach. If you look at the values in the checklist, it's hard to see how anyone would sort of disagree with the importance of those values for the institution and for this kind of project. But I wonder if there has been, you know, other feedback from in the organizational perspective saying, great, but here's a different priority or some other reason. Has there been anything like that? I sat in on on one use of the checklist that I could tell that the senior leader who was a senior director that's just below the EP was impatient and wanted to get it over with, but didn't cut it off either. They were willing to, you know, spend almost an entire hour walking through this with the rest of us and talking it out. But I could tell they were impatient. Well, I can I can see that. Someone in the chat has a very I'm sorry, and the Q&A has a very different kind of question. I wonder if just because there it's there, we might jump to that and then come back to our internal discussion. Brian Nice is asking, what are your thoughts on the executive orders and policies being released from the White House on equity and AI? Or the NIST AI risk management framework? Does anyone have familiarity with those or want to comment on them? I've looked at the White House one, but not the NIST one. I don't know if basically just in short, I think the the White House one is necessary, but it's pretty vague still. One of the things that I think is so powerful with the checklist that we have is that it really it really brings up the hard conversations that need to happen. Whereas the the White House AI statements are just pretty vague. I was like, here's the kinds of things that we should be doing. And it's necessary, but it's certainly not sufficient. Anyone else? Specifically with the RMF, the challenge is that it is still being translated into practice. I think we don't know exactly how well it's going to work. It seems to be good at identification of risks, but not necessarily at resolving the question of what we ought to do given the risks. And so I think that it is still a work in progress. I think it has a lot of promise, but we'll know a lot more in six to nine months when we see people trying to work it out in real cases, which didn't largely happen prior to the publication of it. Yeah. So shall we go back for a moment to the institution and what you've been doing? I'm wondering about the broader question that many of you have talked about, about trust. And thinking about patients and families trusting the institution, certainly if one of these projects went bad and people were aware of it, that would be a major concern. I wonder if proactively you've thought about ways to maintain patient and family trust and staff trust as well, I guess about this kind of thing and whether there are disclosures that the hospital should make or explanations about uses of data or assurances of some kind, some broader information campaign at some point. Have you thought about that or what do you think might be helpful or not? I have a few thoughts and it looks like Jean does as well. So yeah. You wanna go first? Sure. What perked my ears up about the question was about disclosures of the use of data. One of the, I had a really fruitful collaboration once with a professor of psychology at the University of Saskatchewan and we did a small deceptive study in which we asked people in a consent form whether they consented to the reuse of the data from the study as in open data to share it. And what we found of course is that nobody read the consent form at all. But then when we debriefed them and we asked them how do you feel you consented to open data with this study how do you really feel about that? A very large proportion of them said that they would not have consented if they paid attention to it and that they were not comfortable with the resharing their data for new studies that were not part of the original IRB approved study. And so I think here we have a really thorny problem that is for people like Alex and Doug rather than people like me, which is hospitals and medical science in general need to be able to use data to take care of current patients. They need to be able to use data to advance the science of medicine and they need to be able to use data to improve the quality of care. So people are appropriately concerned about the use of data but they are also inappropriately concerned about the use of data, right? People get scared about data they panic about their data being used in certain ways. And so how do you negotiate the edge of that blade so that you can use the data that you need to use in order to take care of the patients in advance of science of medicine while simultaneously safeguarding not only the privacy of your patients but also their sense of security with their interaction with your hospital. And I wish I knew the answer to that question but I doubt there is one. Yeah. And do I eat you? If I just sort of follow up on, I think that's an important point. And I think there's a larger issue that data science in general is sort of raising which is our system for regulating medical practice is premised pretty much on the idea that there's a relatively clear distinction between activities that we do to learn and activities that we perform in the general practice of medicine where we have standards of care and oversight for physician practice that's supposed to ensure that they act within a responsible clinical practice, right? So they're drawing on the established knowledge base when they deliver care and then the knowledge base gets established through a kind of separate research enterprise. And then we have a very different oversight and regulatory system for human subjects research. And data science often falls in a very precarious region where you're inundating, so you're trying something new, whether that's a diagnostic or an intervention and you can use existing data. So you may not necessarily be doing human subjects research when you do that. And if you don't subject it then to explicit testing in the wild, then you don't have to visit your IRB. So we're okay with a kind of innovation that happens on the bench and then can be deployed at large scale in the wild with its own risks. The way that I think is really out of touch or discordant with how sensitive we are to the risks that we expose human subjects to in comparatively small clinical trials. So another way to put it is if you are very concerned about the ways that people's interests might be implicated in explicit formal research, then you should be as concerned or more about the way that people's interests are gonna be implicated by the widespread use of things that haven't been validated as carefully as what we validate when we do human subjects research. So I think the use of people's data, it's certainly important, privacy is important, but also these larger issues about the ways in which we're using those data is the task we're using them for. As people try to develop biomarkers for things like from when you call the hospital, if a machine can analyze your voice and then give it an output or risk score for various types of medical problems, it's generating new private identifiable medical information perhaps without your consent. Is that a research ethics issue? Is it a clinical practice issue? If it falls outside of our current silos, then it might sort of escape explicit even though the ethical issues are as big or bigger than the ones that we regulate in our traditional division of labor. Yes, I wanna follow up on that really very quickly because I think it's very important that our current understanding of what needs to go to the IRB and what is subject to ethical scrutiny can lead to problems because it is so easy to justify, I'm doing this for the greater good. I'm doing this in order to improve patient care and therefore, dot, dot, dot, right? And we know that the history of human rights excrescences is punctuated with people who thought that they were doing the right thing and thought that they were doing a noble thing. And so whether or not we are required to go to the IRB in certain cases, we need to have new ethical structures or expanded ethical structures that are capable of reviewing these kinds of things because well, this doesn't need to go to the IRB because it doesn't fit into one of our current human subjects designations. If you're using human data, whether or not it is legally human subjects research, you need to think of it as human subjects research and you need to think of the consequences to the person at the end of the thing. Yeah. Do you folks think that the IRB, thank you, Jean, and bringing up the whole data subject concept, I think is really important as well. Do people think that the IRB should be an expanded, educated IRB should be the structure that takes on this approach or something else? Something else, is there an opinion from our panelists? Well, I don't think it's the right structure. I mean, IRBs are so driven by regulation that, I mean, I can tell you that if the data science folks here started sending this stuff to us and analysts would look at it and mark it, not human subjects research, basically not ours. So, but Jean is right and the whole history of research, ethics and regulation is founded in people who really, really believe that what they were doing was good and they ended up being scandals and that's really how we got regulations at least in the United States for human subjects research and I think data science runs some of the same risks if the field itself can't figure out a way to do this that avoids that, you're probably looking at regulations 20 years from now or maybe sooner but it took a long time with research. And quite a few scandals. Yeah, the question for me is, are we going to learn from prior industries mistakes or are we going to recommit those same mistakes and abuses and have regulation imposed from the exterior? And I know which of those that I prefer. Yeah, I don't want to thank David for his comment in the chat adding to this and did you want to add a comment to that, David? Well, my internet is not very good so I'm trying to avoid people listening to my glitchy voice but I'll just note that whether in technology companies, transportation, self-driving cars, social networks, universities, these questions of how to ethically evaluate data-centric projects is front and center for a lot of people and I think it's an opportunity for kind of cross-sector interaction and learning as we all try and figure out what to do. Well, thank you. I want to invite Kelsey into this. I think she has a question. Thanks Charlotte. You know, I'm really struck in ways by the surprise that has felt when suddenly even people within the institution begin to recognize the ways that data are being used to structure for clinicians the way that they practice. The organization of how they interact with one another and with their patients and the markets that are being sought for different sources of care. So imagine that if that surprise is happening for people within the organization and how surprising that might feel for patients and community members who really are only interacting with the outcomes of those decisions as they engage with the people who deliver care and kind of greet them at the door. And so one of the questions that I'm kind of thinking about is that in this incredibly complex kind of modern healthcare system where there's so much interprofessional collaboration where you've got data scientists and analysts and other types of professions, now perhaps just as involved in the delivery of healthcare as your kind of traditionally recognized roles, clinicians and such. Whether there is a role for education and work that comes prior to the engagement of these professionals in the healthcare system itself. So we obviously have a lot of ethics education that focuses on clinicians, right? Happening in schools of medical education all the time. Is the way that the modern healthcare system moving to have more interprofessional collaboration placing demands on schools of higher education, training institutions for other sorts of professions beyond those that have already been identified. And so I just, I think it's really fantastic about the work that's being done here is that it uses Seattle Children's as this convening space to really bring attention to the ethical dimensions of the issues that different professions are now interacting with within the healthcare system. But is that almost too late? I guess it's the question I'm asking or do you feel like it only makes sense once you enter into the healthcare space as a data analyst to really be grappling with these sorts of questions? Well, really important, it's just Kelsey. So I mean, I think it's not too late. And I think our group shows that, but in most places you're not gonna have that sort of core of people who really wanna do this and get educated and so on. What's not gonna work is one more self, not self, impose module to make our employees do about ethics and data science. I mean, I'm not a big fan of modules at all. I mean, they're just busy work and rarely impact the decisions people make. So I do think you're right that early on is important. I'm thinking back to my son's experience, getting his master's degree in undergrad work. And I don't think he had, I don't even know if he had a class session on ethics much less an entire course. And so there's I think a lot of work to do there. It also, I think we've learned from healthcare that it has to be done the right way, right? You have to teach ethics in a way that is accessible to people who aren't philosophers. My guess is most data scientists don't have a lot of interest in hearing about cotton mail. And so it's gotta be a case driven, engaging, really thought provoking sort of teaching. And that's gonna probably take a little work to get to that point. So I, if I could just, because I think Kelsey's question was or comment was really important. And I think it goes back to who we regard. And by we, I think I mean educators and sort of the citizens in a open society more generally, who we regard as having important moral responsibilities. And so there's a long tradition of seeing healthcare providers as having very important moral responsibilities. There's less of such a tradition of seeing managers and management people as having that same type of moral responsibility. And a much longer tradition of seeing them is in the state of nature where they're competing for scarce resources against other firms. And I wanna bring this back to how different medicine is from other domains. So, and this is one of the things I try to emphasize to the students that I teach, many of whom are data science students. You know, that if you, if you go back to the comment about Google earlier about, you know, they have a fast-paced culture, you know and you sort of move fast and break things. And that's totally fine if everything you're doing lives inside a browser and, you know, it's sort of serving up people's search results on the internet because very little hangs on that, you know, I mean, in the individual cases. But, you know, when you're talking about things that affect people's health and wellbeing and also when you're operating in such an area of uncertainty, I think a lot of people don't understand the pervasiveness of uncertainty in medicine. And so if you're doing data science in medicine, it's gonna be different than if you're doing data science for Amazon where you get to see people's purchasing behavior and what you're doing is predicting their purchasing behavior. And if you're in management in medicine, then one of the things that you've got to confront is how is your institution going to manage the pervasive uncertainty that it faces and learning and learning in an ethical way so that you improve practices and try to make them more efficient? That's fundamental to your task. So I think we've miss-sold ethics as a set of constraints that are supposed to get in people's way rather than trying to sell ethics as part of the, you know, fundamental obligation of people in these various places in these large institutions and confronting uncertainty in medicine is everybody's responsibility from the top down and the bottom up. And that requires sort of a change in lots of different cultures, management being one of them. Yeah, again, if I could piggyback off that just a tiny bit, I think one of the great challenges in getting data scientists to be engaged with this kind of ethical work is that the chain of implications from writing code to outcomes of the patient is so long and variegated that it is often not obvious to us that the effects are hitting the patient, right? To take, for example, the double-rooming dashboard. Obviously there's a disparity associated with who is put into the double-rooming dashboard, you know, into double-rooms but those implications may go further, right? They may result in outcomes, they may result in medical experiment, experiences that result in, you know, long-term impacts to patients and to the hospital itself. And so, you know, we've all heard the term that, you know, when a physician makes a mistake, a patient can die and therefore they are held to this incredibly high standard. Well, in these circumstances, when a data scientist makes the mistake, hundreds and hundreds and hundreds of patients can die, right? It just, you'll just never see that line of implication because the tools that we build are used across such a broad swath of our charges and the people that we are responsible with providing care for. And so it makes it all the more important but all the more difficult because it is so easy to say those outcomes aren't actually the effect of my work when that line of implication may be sublimated and it may be lost in a lot of noise but it is still there. So well said. Jean and I think that as we're reaching the time when we have to close the session, I'm not sure there could be anything more powerful than that statement of responsibility but I think that we wanna thank every one of you who's spoken for the really rare opportunity to get inside an organization at the beginning of people trying on the ground to translate ethical concerns into practice in a way that is respectful of your fellow professionals and educates and presumably is changing culture. You certainly supported by professional organizations in your field all new in what the people are saying but an opportunity to really try to move forward and pioneer an approach that may really be helpful to others to use as well. So I think we've explored additional ways of thinking about this but most of all we've had this opportunity to be inside the organization and to talk about the issues as well as practices. And we're so grateful to each of you for making that possible and bringing these issues forward for our community thanks to the audience and thanks to Alyssa for your last note in the chat. Kelsey, do you wanna take us out of this? Sure, really huge amount of gratitude for all of you. We are going to pivot a little bit in our next session of the Organizational Ethics Consortium. March 31st, we'll hear from a group at Beth Israel Deaconess Medical Center on an emerging medical legal partnership and the ways again in which healthcare systems are perhaps thinking beyond traditional clinical delivery. Their role is traditional delivery of clinical care to advance the good and well-being of a broader patient population and community. So thank you so much to all of you and we look forward to seeing you again. Bye bye now. Bye.