 It is an absolute honor to present to you today at this year's art medicine conference. My name is Rich Hannah. I'm a data scientist in the cell and gene therapy data ops team at the Children's Hospital of Philadelphia. And it's my pleasure, along with Dr. Stephen Kodalki, to talk to you today about an art shiny application that we developed to assist users with stem cell transplant recording and transcription. I have no conflicts of interest to disclose. So let's give a little contextual background on stem cell transplantation, which should come as no surprise that it's a life-saving treatment for cancers and other diseases. But what are the actual steps to it? So the first is to collect stem cells. The donor is hooked up to an apheresis machine which can separate stem cells from other cells. In the image here, you see the bag into which the stem cells will go on the right. Next, the patient who has cancer or some other disease requiring stem cell transplant gets chemo therapy to clear out the cancer cells as well as any of the patient's own cells and make room for the new stem cells in his or her bone marrow. And after the chemo is done, the patient receives the stem cells from the donor intravenously. The stem cells then go to the bone marrow and start making new blood cells. So how do we know that the stem cell transplant was successful? The function of stem cells is to generate red blood cells, a kind of immune cell known as neutrophils and platelets which help with blood clotting. When the patient gets chemo, all of their own stem cells die and as a result they stop making these cells. Red blood cells live for a long time, but neutrophils and platelets are short-lived on the order of just a few days. So a key outcome we monitor after-stall stem cell transplant is the reemergence of those short-lived neutrophils and platelets, which now come from the donor's own stem cells. This reemergence of platelets and neutrophils is called engraftment. To aggregate this data, cancer centers across the United States are required by law to report their outcomes data to the Center for International Bone and Marrow Transplant Research, or CIBMTR. Registries like CIBMTR are instrumental for providing the data for cancer treatment guidelines, which is a prime example of real-world data and so it comes in no surprise that it's imperative that the data be accurately recorded, such as in the case of engraftment. Two key values when talking about engraftment are neutrophil counts and platelet counts. Neutrophils, also known as white blood cells, are responsible for the body's ability to fight off infection. After a transplant, the level of those cells is considered to be at appropriate levels when it reaches an account of 500 cells per cubic milliliter. The other key indicator is platelet recovery. Platelets are responsible for proper clotting of blood and having too few can result in bleeding. We are particularly interested in if and when the platelet count reaches 20 mil... cells per milliliter. So what's the current process for our reporting of neutrophil and platelet recovery? To be honest, it's quite a slog. Here's the rundown. Our research staff has to reference the patient medical records on the electronic health record system, locate specific tabs, expand to underlying tables, flip over to associate flow sheets, then identify the target values we need to meet specific date criteria, and then finally take those manual calculations and transcribe them into the database registry. It is a very labor-intensive process and a bit frustrating, and our data manager spent a lot of time on the CDS work. This was a starting point for us wanting to develop an app to automate most of this work. But the problem itself was actually much bigger. As part of the pre-work of the clinical implementation of the app, we went back and reviewed submitted data between the year 2016 and 2020, and we discovered an astounding number of data entry errors. We found that out 226 patients, 64 had incorrect platelet recovery and graphing dates, and 29 had incorrect neutrophil and graphing dates. A nearly 30% error rate is incredibly high, and we are just one of many institutions that are part of this registry. So we decided to make an application using our Shiny to address these issues called the Engraffment app to help our team with this work. In order for that app to be successful, it was helpful to nail down the specific needs of the team to make adoption and trust of the app as seamless as possible. A successful application design had to essentially ease the work currently in place. It needed to decrease transcription time, remove the need for manual calculations, increase the accuracy of those calculations, and deliver results fast and with minimal user interface effort. All right, so here we are at the CIBMTR forms assistant application itself. I'm going to give you a quick demo. Everything you see here is going to be using test data. So let's quickly append this URL to make sure that that is in fact what we're using. And here we are in the testing environment. And right off the bat, you're going to see that we put up some guidelines in place for the only input on this application. It's just taking a patient MRN in. Now at CHOP, we have two requirements mostly for patient MRNs. It has to be all numeric values, and the length of that string of numeric values has to be 8. So if we put in something shorter than that, you'll get a little error saying it has to be a length of 8. If we put in something longer than that, it'll still say length of 8. If we put in something that has a letter, it'll change to an error display showing that can only take numeric values. So we always have the guidelines up in place to make sure that we're helping our users as much as possible. So let's put in a fake patient. And anyone who's a fan of Game of Thrones should recognize this is not real patient data. So for Asgard Lannister, we have two transplants. And that's shown here in this nice little DT display. The top row here shows that this patient never engrafted for ANC 500, which is the neutrophil count. So for 500 cells per cubic milliliter, they also never engrafted for PLT 50, which is another metric that we take into consideration. But they did engraft for the platelet 20 count on September 26, 2020. So if we click on this, we should see a graphical display also showing that. So we get a really, really nice display thanks to High Charter and a nice little button here thanks to our clipboard. And this is interactive. So you can actually go ahead and look at all the different ANC values. And we see here that there was no call out. There was never a point in this first transplant. You'll see if this patient actually had two transplants also showing the DT table. During this period after the first transplant, they never had that ANC engraftment. They did have a few platelet transfusions shown by these flags. And right here is the first valid point where they had a PLT 20 value. So that checks out. Now if we wanted to actually take this value, we're actually going to put this into the CIDMTR forms app. All we have to do is copy and click on that button. Now if I open up the text editor and paste it, you see right there that we have exactly the date. So you can put it right into the target destination. Now let's look at a second one. So for the second transplant, this one had all three values available. And we see here that that is in fact true. So the first ANC value reaches that ANC 500 threshold right here on October 28. Right here is displayed in that button. After the second transplant, we had a PLT 20 value here on October 30 and a 50 right here on November 11. Now let's look at another patient. So for this patient, for Hodor Greyjoy, we had four transplants recorded for this patient. Now you're going to see something here. I'm going to take your attention to this fourth transplant. It's missing a value in the manipulation column. The manipulation column is responsible for describing any kind of processing done in the stem cell product. For example, if there was a T cell depletion to decrease the chance of something called graft versus host disease. So this is actually more of a clerical error rather than an indicator of contextual data like what we saw on the blanks from the previous patient. If we click on this, we actually get a really nice shiny bootstrap warning telling us that there is actually possibly a data entry error here and to contact the appropriate person. So that's another thing we put in place to make sure that we target these corner case scenarios where something might be amiss. So now that we've seen the app in action presented it to our team, how do we assess that it actually fulfilled what we promised? Well, remember that maze of steps we took in the original process? We reduced that down to a single user input and only two clicks backed up by visuals to confirm the validity of the end result. Custom error messages help set up guide rails for our app users. Our clipboard allowed us to make quick copy pasting a possibility and the shiny bootstrap package added additional oversight for corner case alarms. Now under the hood, we developed a robust testing suite using the use this and test that packages. Combining that with the visual markdown output of the cover page package shown here allowed us to show our team that our software was adequately tested and inspire confidence that we were doing all we could to give them reliable software. And lastly, designing with a modular framework in mind was made possible by Golem. So implementing additional functionality or deploying the package at another institution or with connections to a different database only requires rewrites of those underlying connection functions, nothing more. So remember those concerning errors from earlier, the 30%, we conducted a prospective post implementation validation and here are the results. Does our app reduce errors? Yes. Not only does it reduce errors, but it completely squashes them. I'll probably never be able to present again on something that has a 0% error rate. So I'm gonna relish in this, but indeed every calculation was nailed every time for an error rate of 0%. So let's start concluding and wrap things up. So data entry in healthcare is hard, doesn't have to be, but it is ubiquitous and difficult. But by leveraging automation and the power supplied by R Shiny, we can eliminate errors and reduce the burden on staff responsible for data entry. By designing with a modular architecture, the footprint built can be delivered to other databases, departments or other institutions entirely. And this in turn can support a robust, reliable data registry that accurately powers cancer treatment guidelines. Thank you so much for attending. I hope you enjoy the rest of the conference.