 Speaker, one of two key notes in this session will be from Eric Kodish, the Director of the Cleveland Clinic Center for Ethics, Humanities and Spiritual Care. He's the F.J. O'Neill Professor and Chairman of the Department of Bioethics, Executive Director of the Cleveland Fellowship in Advanced Bioethics and Professor of Pediatrics at the Lerner College of Medicine of Case Western. His areas of expertise include childhood cancer and blood diseases, pediatric ethics, end-of-life issues, and research ethics. He's been principal investigator in a series of three NCI-funded multi-site studies of informed consent in childhood cancer. The American Cancer Society, Greenwall Foundation, Cleveland Foundation, and Josiah Macy Foundation have funded his work, and he will be speaking to us on quality attestation in clinical ethics. Stepping up. So please step up. Good afternoon. Always such a pleasure to be back on the South Side and such a pleasure, Mark, to be back reunited with the wonderful community that you have created here. I'm going to dedicate this talk to the memory of Bill Steuben, who many of you knew, passed away recently, President of the Greenwall Foundation, and it was actually a conversation with Bill that got me interested in this, the experience that led to it also related to my going into a secure testing center, surrendering my belongings, as someone mentioned in an earlier talk today, taking the recertification board in pediatric hematology oncology, walking out into the parking lot, looking at my beeper, and noticing that someone had called me to do an ethics council on an end-of-life situation in the ICU, and I had an epiphany, or if not an epiphany at least, a moment of realization where I said to myself, how come I have to take a test to prove that I can take care of kids with cancer but I don't have to do the same thing to do an ethics council? It didn't seem right to me. I began to think about it, talk to people, including Bill Steuben. One thing led to another, and here we are. So I am going to just mention a quick disclosure. I didn't think it mirrored at a slide, but I'm a consultant to Biogen IDEC for some issues in research ethics, having nothing whatsoever to do with this particular topic. Quality attestation is a, yeah, thank you, Mark, it is an intermediate step. It's meant to be a review of an individual's ability to carry out ethics consultation in practice in a paper that we had in the Hastings Center report. It was envisioned as a two-step process, a portfolio, which I'm going to talk about today in detail in which we've accomplished, at least in a pilot phase, followed by an oral examination. It's meant to assess education skills and experience to ascertain whether an individual can either independently perform ethics consultation or lead as a lead consultant in a group council. I think as many of you know, there are other concepts out here which are important to delineate. Credentially, it has to do with a local hospital allowing somebody to practice to write a note in a chart. It will vary from venue to venue. Formal board certification, we're not there yet, but many think that this quality attestation would be a step on that pathway. And then programs, as you see in the bottom slide, which can be accredited to complement the individual issue of certification and accreditation of ethics training programs is a different issue that I'm not going to cover today. I want to say a word or two about measurement. There's at least two important concepts. One is precision is a measurement, something that will yield consistent results when repeated. And the question there is how close are the results to one another. And then there's accuracy, which is ideally a level of measurement with no inherent limitation. And the question there is how close are the results to the true value. And I want to apply these for a second to thinking about measuring quality in ethics consultation. Since the 80s, at least, maybe the 70s, people have been doing ethics consults. There hasn't been a lot of progress in how we measure quality in ethics consultation. And I think what you see here would be a measurement that had low accuracy but high precision, right? All of these are near each other on repeated measurements, but they're not very accurate. High accuracy with low precision would look something like this. And ideally, one would have high accuracy and high precision. And I would submit that for now, in the place we are in history in measuring quality in ethics consultation, that accuracy is more important than precision. And I fear sometimes that we're going to get a little hung up on the precision. So let's aim for at least modest accuracy, and we can worry about precision later. This has been decades in evolution. In addition to Dave Scheedermeyer's wonderful literary skills, he and John Lepuma had a paper in 91 about the qualifications of ethics consultants. John Fletcher wrote about it in 94. ASBH has been active in this area for a long time on core competencies. Ellen and her colleagues at the VA showed a lot of ethics consultation going on, but very little training, diverse backgrounds, calling out the need for standards. Nancy Dubler also was concerned about this and brought together a group you'll see here on the bottom on the credentialing project with an important paper in the Hastings Center. I'm going to go through the history pretty quick here. ASBH Clinical Ethics Task Force was created after the conversation that I had with Bill Stubin. I, with some colleagues, came up with an idea for a four-step process that, in addition to a portfolio, an oral exam included a written exam and a simulated ethics consultation. This was published in the Journal of Clinical Ethics. And we put this out there as four separate ideas that could be modular. It wasn't necessary that all four of them would happen. It was in conversation with the Greenwell Foundation about work toward that. And then I was approached by my friend and colleague Joe Finns when he became president of ASBH to look to partner to move this forward. Mark Kaczewski was president before Joe and was also very committed to this. And the ASBH Board has been supportive of this, working forward. And the work I'm going to talk about now involves a quality attestation, Presidential Task Force, which I co-chaired with Dr. Finns. We initially had support from the Macy Foundation. I mentioned this paper in the Hastings Center that laid out a two-step model. And then the Greenwell Foundation provided some pilot support for the portfolio segment. So that takes us through a few decades of history in a couple minutes. The point I want to make, though, is that ethics consultation is here to stay. In the New York Times, a month and a half ago, my colleague Marty Smith was quoted having to do with our work on patients without surrogates, which we see all the time. We do 600 consults a year at the Cleveland Clinic across the system. Many of them involve patients without surrogates. And ethics consultation is key to helping, at least one way, to help manage these difficult situations. So we have no basic credentialing requirement now. I think consensus has emerged and that we're going to be kind of left behind if we don't take this internally and act. We also have the risk, I think, that external forces will come into the picture and tell ethics consultants how they ought to be doing it. If we don't take some ownership in this. We have a challenge. We have many challenges, actually. There's no prescribed entrance exam or educational requirement now to conduct ethics consultation. It is true that anybody could hang their shingle, say that they're an ethics consultant, and it would not be really possible to challenge that in any meaningful way. It is a high stakes endeavor, as I said before, we're involved in end of life issues, reproductive ethics, all sorts of complicated, challenging things without oversight or regulation. And it's not in conformity with the overall trend in healthcare, which is emphasis on safety, quality, accountability. And I think we do have general agreement on the core competencies and we can test. Why a portfolio? I find portfolios to be pedagogically compelling. I don't know how much experience those of you in the audience have had with them, but there is precedent in some other areas of assessment. It's essential, I think, to allow candidates to present their best case. And that's what a portfolio can do. It is amenable to individuals with variable training and experiences. And that is, I think, one of the real strengths of ethics, clinical ethics consultation is that we have people from many fields who do it and do it well. A portfolio allows many different disciplines to show their best stuff, if you will. And I think it's compatible with residency programs to make you board eligible. Our thinking here was that if you pass the portfolio stage, you would become eligible for the next step. I'm going to move into some nuts and bolts of how we went about this now. Now, we, the quality attestation presidential task force, put together these elements of a portfolio and made some judgments about how many points would be assigned to the various components of the portfolio. And you'll see here that these six in-depth consultations were worth 48 points. So almost half, this adds up to 100 points, and that's the scale that we'll be showing you data on next. Almost half of the portfolio is attributable to these six in-depth consultations. But education, training, experience, 15, a statement about your philosophy of ethics consultation, which I'll say more about in a minute. Some letters of recommendation, and then six kind of one-pager brief consults that were meant to allow candidates to show more breadth of their mastery of ethics consultation. So that adds up to 100, and just I want to say something because I know Mark is particularly interested in this in terms of education. We set the bar at a master's degree or higher. Now, you could argue one side or the other of that. We did make an allowance for candidates who had not achieved the master's degree to raise the possibility that by virtue of their own experience, they would be exempt from that requirement. I think some people feel that it should be a doctoral degree. Others may feel that an undergraduate degree is sufficient. But right now, the education criteria states master's degree or higher. This philosophy of ethics consultation is kind of a tip of our hat to the value of being concise. We figured that we should be able to say in 500 words or less what their philosophy of ethics consultation is, and we set up this scoring system of five criteria each grade at zero to three, and I'm not going to read through them for the sake of time, but these are the things that are the components of that statement of philosophy that we scored together as a task force, and I'll go through our methods here in a bit. And let me just pause here to say that a manuscript has been accepted in a job, the American Journal of Bioethics, and this particular journal has what they call target articles, and Target is an interesting name for this, but a job has decided to put this paper out as a target article. The email, I think, went out yesterday, so it's pretty serendipitous timing, and those of you who want to read the article, certainly I would encourage you to do that, but also to propose to a job what they call an OPC, an open peer commentary to give us your ideas and help us think better about this. So we're excited that there's going to be an open peer commentary period from age up, and I'm going to move on now to tell you a little bit about EQUAT. Our colleague Bob Perlman, who is on the quality attestation task force, really in very beautiful timing, had done a lot of the preliminary work on this with his colleagues at the VA ethics program under Ellen's leadership, and they have a system that they've been working on to measure quality of an ethics consult note. Now does an ethics consult note, if you write that prove that you're competent to do ethics consultation? I would say no, but hopefully taken together with your statement of philosophy, with your training, with those other elements that I showed you, gives a more holistic picture. This is the critical tool though, because this one is the 48 points that I showed you before, and the scores are based on integration of four elements. Is the ethics question or concerns stated clearly in the note? Is there consultation specific information provided in the note? Is there an ethical analysis? And are there recommendations or conclusions? And in the EQUAT model, each case is initially scored as acceptable or less than acceptable on a holistic basis. Then the reviewer rereads it and gives it a one through four score, and we had dyads of reviewers. The notes were blinded, I'll get into that in a minute, but we had people reading them, and if they were discrepant between the dyads, if one raider gave it a one or a two and the other gave it a three or four, a conference call was arranged for them to reconcile their views and come together with a final score. So we did our best to make scoring consistent and fair by going through the training ourselves. Everybody on the task force did hours of training using the EQUAT model. We reviewed the consults without knowledge of the consultant or the institution, and it took the staff a lot of time to de-identify all of these cases, but I think it was time well spent. And then, as I said, pairs of reviewers independently reviewed all six cases. So here was the sequence of events. Initially, I think at the end of 2013, we sent out requests for letters of intent, and we were gratified that 82 people in the bioethics community said, sure, I'll do a portfolio. I'll partner with you to learn from this. We knew that we couldn't review 82 portfolios, and we did a stratified random selection of 32 of those, and these were individuals. This was not to be done by institution, and some people were confused about that. This was open to any individual who wanted to submit. Of the 82, we stratified by years of experience into quartiles, so we had eight in each of four quartiles on years of experience. Interestingly, 23 of the 32 who we invited submitted a portfolio, which is about what we expected. And then around the table at the task force, we ascertained conflicts of interest. We did the ECWAP training that I mentioned, and then the dyads did their reviews. After that, there was an unblinding, and the remainder of the portfolio was scored. There's no way to score the other components, really, or at least most of the other components without unblinding, but we had removed COI. And then we had a consensus meeting and discussion, and we made a group decision on each portfolio. A bit about the characteristics, you can see a pretty even split between men and women, and here's the years of experience. I don't have time to show you the disciplinary background, but it's in the paper, and take my word, there's a nice mix of professional disciplines as well for what you can do with a sample size of 23. We elected to use an inductive process, just a reminder for those of you. I always have to remind myself to go from observations to tentative conclusions is an inductive process, and a deductive process would have been to set an a priori cutoff without seeing the data first. And we certainly plan on an iterative process as we get more portfolios, if this moves forward, that the cutoff could change and probably will change based on the data. But here's the data, and I think if there's a slide that's most important in my talk, this is the data. This is our 100 points scale, and the range of scores was from 52.5 up to 95. The mean was about 80, and the median score was a little higher at 85, and the standard deviation was 13. So here's the 23 pioneers, if you will, who submitted portfolios, and this is the distribution of their scores based on what I showed you. If you take 1.5 standard deviations beneath, thank you, beneath the mean, the cutoff is right about here at 60, and with that threshold, about 90% of the applicants of the candidates would be considered to have a satisfactory score and would qualify for the next step in quality attestation. And I'm going to give a few conclusions here. I'm not going to take time to talk about the limitations because they're in the paper, there are substantial limitations, we recognize them, I'm sure the open peer review comments will give us more limitations because we're really just getting started with this. But I think that we got a good sample of candidates that varied by discipline and diversity of ethics consultation. The scores of individual components were correlated with one another, but not completely, so we think each element makes some useful contribution. We didn't see enough redundancy at this point that we would eliminate anything. We know that the attestation process could be further improved in the future by more transparency about what we're looking for, but we'll be honest, we didn't really know what we were looking for because we were just getting started. The vast majority of candidates were asked to proceed, which suggests to me that ethics consultation in 90% of the candidates now are being adequately performed, but with any quality improvement effort, we're committed to this idea of gradually, I think, increasing the expectation because that's a key issue for professionals. Everyone wants to get better all the time, so we admittedly said, you know, we want most people to get a passing grade this time. We owe a great deal of appreciation to the candidates that volunteered. This was a very time-intensive effort, and we really obviously could not have done it without those who were willing pioneers. We know that people will know what constitutes quality as we disseminate this. We think that's a good thing. And finally, the portfolios, I think, are a promising tool in measurement of quality for ethics consultation. Thanks to the members of the Quality Attestation Task Force, you'll see in the asterisk here that Art Dursey, who's here today, Art, where are you? Raise your hand in the back. And Anita Tarzian are the current co-chairs. Joe and I have rotated off. The project lives on, and I know we'll continue to do great work. And finally, a little wisdom of fiction. This great novel I read, I pity the poor immigrant, had this little quote in it, and I think it says something about ethics consultation. It was not in the hope of finding an answer, but in the hope of creating a space in which to think through the question. Thank you. We have time for perhaps one or two questions. Laini has first. Great. Thank you for that great overview of what you guys have been doing. Thank you for doing all that work. Going back to this morning when we heard Stacey talking about sustainability, the amount of time and training for viewing it, and going back to what's been talked about, where we do two ethics consults and we get paid a little to do it. How are you planning to make this sustainable because it's an expensive process? Yeah. Well, we've been batting around some ideas around that. I think what we've been geared toward now is kind of the high end. People that are really active ethics consultants doing a lot of consults. I think there are also opportunities potentially to broaden the market for this. People that are on ethics committees, people who would like to sort of climb the ladder of the profession of clinical ethics. So potentially a written test that people would pay less for, but there needs to be a sustainable business model for sure. Our foundation partners have been very generous and we hope will continue to be very generous, but they don't want to bankroll this forever, nor should we expect them to. Again, I think we need to demonstrate quality to hospitals fundamentally, and they will hopefully in the area of value buy in. That's the best I can hope for. One more question. Yes, Lexie, please. Hi, Lexie Torkey from Indiana. I have both a nuts and bolts question and then just ask you to reflect on something more qualitatively. So do folks have to have a master's degree in ethics because both as a graduate of this program and as a director of an ethics program, I'm curious how you perceived ethics fellowships. And then my qualitative question is, can you just reflect a little now that you've been through this process once on what tended to make a strong portfolio or red flags for portfolio? Thank you for those questions. The first one is, is the master's required to be a master's in bioethics? And the answer is no. A master's in divinity. I think many people have, you know, it could be a master's in electrical engineering, I guess theoretically. It could be anything, but the portfolio hopefully will reflect quality regardless of what kind of master's degree it is. With regard to reflection about what made a strong portfolio, rigor of thinking, clarity of conceptual analysis of a particular case, I think restraint at least for me. I can't speak to my colleagues, but our sense was if someone came in, like my last slide, to say, here's the answers, that's probably not as good an ethics counsel to someone who says, here's the questions and tries to enable the person who requested the consult to reflect. And someone who teaches, like if I saw a consult note that someone had referred to a paper that Dan Solmezzi wrote to and the consult note on the chart, I'd say, Bingo, this person, you know, is probably doing good quality ethics counsels. Thanks. Thank you.