 So next up, we have Dr. Stefan Kodaki and Rich Hannah from CHOPP. They're going to do a presentation on CAR T cell clinical trial analytics at CHOPP. So I'm moving from red cap to R Shiny with the red cap tidy. So saw that he's the co-host now. So if you're ready to start a little bit early, that'd be great. Not all good. All right. I'm going to share my screen here. See if that works. So thanks to the organizers for having me back here at our medicine. I'm super excited to see how things are coming together and how many people we're going to have. We have at this conference today. So when I get started, the title of my talk is CAR T cell trial analytics at CHOPP from red cap to R Shiny with red cap tidier. And I'm Stefan Kodaki. I'm an assistant professor of pathology and laboratory medicine at the Children's Hospital of Philadelphia, also known as CHOPP. At CHOPP, I wear a couple of different hats. I work with patients that have blood disorders like sickle cell disease and autoimmune disease. And I co-direct the cell and gene therapy laboratory. And this is a place where we manufacture cellular therapies for patients. You can almost think of it like a small pharmaceutical company that's built into the hospital. And finally, I lead the cell and gene therapy informatics team and Rich Hanna from our team is here is going to answer questions in the chat for you. And in the sound gene therapy informatics team, we leverage tools like our red cap and others to create data products that aid in our hospitals cell therapy operations. So here's the outline of my talk. And so we're going to cover a lot of ground but it's because I feel that you'll walk away with a much better picture of what it is that we're doing. I'll first tell you why and how we're doing the clinical trials we're doing, and for which we want to streamline the analytics. So in part one, I'll shed light on why and how we conduct CAR T cell trials, focusing on pediatric BLL, be a cute lymphoblastic leukemia. In part we'll explore our quest for streamlining the CAR T clinical trial analytics at job. The challenges we faced in building this pipeline and how we're overcoming them with the help of red cap tidier. Leukemia is the most common cancer in children. And although the prognosis is generally good with cure rates after chemotherapy alone close to 90%. And only only one in three children who relapse after that chemo, initial chemotherapy treatment, or that have leukemia that doesn't respond to the chemotherapy will survive in the long in the long term. So the approach in this case involved intensive chemotherapy, followed by a stem cell transplant or a bone marrow transplant same thing. However, at the advent of CAR T cell therapy a decade ago has significantly changed the treatment landscape for these challenging cases. Let me introduce you to Emily Whitehead. In 2012 at just six years old, Emily came to chop with relapsed BLL that wasn't responding to chemotherapy. And she was so sick that a bone marrow transplant was off the table. Emily became the first pediatric patient to receive CAR T cells through a new clinical trial led by Steve Grubb. The treatment eradicated her cancer cells, and they have not reappeared since. So fast forward to last year, in March of 2022 Emily celebrated a decade post CAR T treatment, marking her as the first patient to be cured by this revolutionary therapy. But still early to determine overall cure rates for CAR T cells. We believe that a significant number of patient with relapsed or refractory BLL could potentially find a cure with just a single dose of CAR T cells and, and this is nothing short of amazing. In oncology is incredibly uncommon to see a single treatment with a single therapeutic agent achieve such a remarkable outcome. Let's delve into the fascinating world of CAR T cells. Essentially CAR T cell is a genetically modified T cell which is shown here in blue. Can you guys see my cursor? Yes, I hope so. And it's designed to produce a chimeric antigen receptor or CAR visualized here in purple. And this car enables it to identify bind to and kill a tumor cell shown here in orange. And it does so because the car can recognize some target protein that's on the tumor cell. So the car is built of an extra cellular component derived from an antibodies antigen recognition domain and an intracellular part that consists of a signaling domain that can activate the T cell. An important aspect of CAR T cells is they're a living drug. They can divide and persist in the patient's body and each car T cell can eliminate many tumor cells. So what makes a good target for BLL? CD 19 is a compelling answer. And here's why. So, so what CD 19 CD 19 is a protein expressed on all normal B cells throughout their development until they differentiate into the plasma cells that make antibodies is your function of the B cells make antibodies. So the CD 19 protein is restricted to B cells. You don't have any CD 19 on other cell types. You don't have a blood stem cells and or other tissues. And fortunately, most B cell cancers remain the expression of CD 19. So most B cell cancer such as BLL or CD 19 positive. The key point is that humans can survive without B cells, and that decreases the risk of CAR T cells attacking vital tissues. And our group and others have shown that CD 19 targeted CAR T cells are safe. They achieve complete remission, complete response in 70 to 90% of pediatric patients with relapsed or refractory ALL with some patients achieving durable mission like Emily. However, like I mentioned earlier, we don't know the exact cure rate because we're only now reaching a decade since the first treatments and outcomes from a large multi center trial of CD 19 directed CAR T cell product the same product that Emily received led to the approval of Tissa Gen Leckluse cell or Tissa cell brand by the FDA in the United States in 2017, and that's the first FDA approved CAR T therapy. So as of now the FDA has approved six CAR T therapies, four of which including Tissa cell target CD 19 these first four here. However, only Tissa cell is currently approved for children so this is what we, this is the only drug that we administer, the only CAR T drug that we administer and chop here. However, we have a lot of clinical trials that we're conducting with experimental CAR T therapies for other conditions, including acute myelid leukemia, Hodgkin and non Hodgkin lymphoma T acute lymphoblastic leukemia neuroblastoma and other solid tumors with many others in sort of development. And successful implementation of these innovative therapies really critically heavily depends on effective data collection analysis for clinical trials. And to illustrate this I want to explore a CAR T cell trial that that we launched less than a year ago. This trial is titled a phase one to be trial of humanized CD 19 CAR T cells, manufactured using prodigy for pediatric be a l l. Let's break this down. Phase one to be indicates that we're assessing both the safety, that's the phase one part, and efficacy, that's the phase two part of these CAR T cells humanized refers to certain tweaks that were made to the car to try to enhance its function. As for manufactured using prodigy, the prodigy is a novel automated manufacturing platform that allows us to produce these clinical CAR T cells in our own laboratory. And in our hands this cuts down the cost of making these things by over 90% over 90% compared to commercial CAR T products. And we believe that this cost reduction could potentially broaden access to this life saving therapy, both in the US and globally. So here's the study schema, we have two cohorts patients who've previously received and failed CAR T therapy and first time CAR T therapy recipients. And after consent and enrollment T cells are harvested from the patients by process called aphoresis and following this we manufacture the CAR T cells in our lab, and then patients undergo lympho depleting chemotherapy and this makes room for the incoming CAR T cells, and then they receive a single dose of humanized CD 19 directed CAR T cells. We then evaluate the effectiveness of the treatment with a marrow biopsy and lumbar puncture on day 28 or around day 28, followed by a 12 month follow up period, during which we see the patient on multiple visits. So now after the 12 months patients are then enrolled in a 15 year long term follow up study. And today, we have successfully manufactured products for eight patients we've infused six, four out of four patients from which we have 2028 day data have responded so this is working really well so far. So, with that, let's move on to the second part. And analytics for CAR T trials at shop, where we will look at how we've previously collected patient data from these trials, and how we're trying to do it now or in the future. So, as you might imagine, all of these trial visits that we just talked about they generate a lot of data, and we use to capture all of this on paper. So each, each patient on a CAR T CAR T trial. So each binder is not a trial each binder here is a patient, and we've treated almost 500 patients and clinical trial so far. So what you're seeing here is just a small fraction maybe 10% of the binders we have here at chop, just for CAR T patients. So a vast effort goes into this data collection but there isn't even a clear pathway to getting any value out of this. So, so there's, there's trial monitors that look at these binders and then we go through them and collect data on spreadsheets then they'll then turn into data say data and safety monitoring board reports. On the other hand, researchers who would like to write papers often ignore these binders completely and instead choose to manually extract relevant data from the medical record directly. So, so as you, you can really imagine here that there's a lot of papers that were never written over published or written and published much later than would have been possible because of these challenges. And so our objective here was to overhaul this process by digitizing the entire CAR T clinical trial data capture to analytics process by developing automated real time analytics. And I think this will have many benefits. For example, we should be able to identify issues that require intervention earlier, and this might improve patient outcomes. We completely expect that this will streamline the generation of research abstracts and publications and they'll boost researchers productivity by allowing for more and earlier publications from each trial. And also a standardized and partially automated approach to data capture and analysis should reduce the labor required to initiate and conduct new clinical trials, leading to more and cheaper trials overall. So let's discuss how we built the end to end trial analytics pipeline. The basic idea is that the data gets abstracted from our electronic health record or EHR and then goes into an electronic data capture EDC system that we're custom building using red cap. Then that data gets automatically pulled into a shiny dashboard that displays the relevant tables and graphs. Forward. Oh no, it's all pixelated. Anyways, okay. So exploring the red cap based electronic data capture further and you'll just have to believe me that this is a screenshot of a red cap or red cap data capture doesn't matter. So we chose red cap for for a bunch of different reasons. We have it available here chop. It's free to use for us and it's secure for storing sensitive information. And we have importantly we have complete control over data entry instrument configuration and data handling. So this database shown in this super blurry screenshot is longitudinal. And it allows us to. So this means that it allows us to document repeat measures such as repeated labs, once for each visit and there may be many, many visits for for each patient in a trial. In addition, we capture adverse events and adverse events don't really fit so neatly into this longitudinal paradigm, but red cap less is managed as using a repeated instrument setup. So this arrow is going to point out is sometimes now you'll just have to believe me. We've also incorporated clinical data pool or CDP, and CDP is a relatively new red cap feature that automatically pulls data like demographics vital signs and labs from the EHR, and that saves considerable considerable time and reduces data entry errors. Oh no. Why is it keep. All right, I'm going to try to fix this. All messed up. All right. All right. Well, this is, this is what the. This is what the with the shiny based real time Carti dashboard looks like if you have really terrible vision. So, so this is written in shiny. We decided to use shiny because obviously we love our, and because we can securely store sensitive information on our studio connect instance that's behind our hospital firewall. And what if you want I could I have one prototype up now if you would like me to share that. Yeah, yeah, why don't we do that. I think we have we have maybe two or three extra minutes because we started early. I really don't know why it's doing that. Can you see my screen. Yeah. Can you open the, this is great. Can you open the the adverse event summary that's further up right here. Okay, so, so what you see here is the DSMB or data and safety monitoring report view with an adverse event summary table that breaks things down by phase or phase of a trial and whether the event was serious or not serious. So we architected. So this is this is we architect the dashboard so that each analytic object like this table or any any graphs that are in here are independent units that can be unit tested. We also set up and this is really similar to the talk we just heard we we we initially the dashboard was really slow. So the way we solve this is we set up a data entry trigger and caching mechanisms that make the dashboard always up to date and lightning fast. So whenever now whenever somebody enters new data data into the red cap database, the affected table gets rebuilt and stored and we use pins for this on our studio connect instance. And I think I can take over again. Thanks Rich for saving my life. Okay, all right guys. Okay, can you see my screen again. Yes. Okay. So what I wanted to say next is that we really owe a great deal to the our community for for building this we use several packages including BS for dash GT summary obviously golem. And of course the tidy verse suite. However, we did encounter a significant hurdle early on and, and that's that the red cap API output is far from user friendly when you deal with a complex project like the one that's longitudinal or has repeating elements or both. We recognize the need for some functionality for a package that makes it easy to import comp to import complex red cap projects into our, and that led us to create one ourselves. And that's how red cap tidier was born an open source package that we first launched a year ago actually at our medicine last year ago. And the red cap tidier makes it easy to import any red cap project into our even the complex ones that contain launch tooling or repeated instruments or events. And it does so by integrating data and metadata into a single object. And we call that object a super table. It's easy to explore that super table in the R studio ID and manipulate using tidy semantics like the pipe operator. And it's both robust and efficient. So it can be used in a production environment. It's built on top of the red cap our package which itself is robust and efficient. It follow we follow tidy verse SDLC recommendations. We have pretty good test coverage. And we have, and we have continuous integration enabled on GitHub. We have extensive error checking with helpful helpful error messages we think they're helpful. And we have used profit and other tools to profile and replace a slow tidy verse code with base are where we found bottlenecks. So let's delve deeper into the red cap are red cap tidy are super table. The object gets generated when you read a red cap project using the read underscore red cap function read red cap function is really simple to use. It takes it has to require arguments the, the URI of the of the of the red cap server and an access token. This is very similar to red capper, but it returns it returns this object here. And this object here is part of the super table from the project trial. When you display it in the R studio ID viewer. Each role of the super table corresponds to an instrument. So you can see the name and delay the human readable name label of the instrument. There's a couple of additional columns here and here's really important ones. And then you have two list columns, one is red cap data one is red cap metadata, and each of these list columns contain a set of tables one for each instrument. So this, this, this contains the data and the field level metadata from that instrument. And so, so what you can do in the R studio idea can do it right here right now but you can click on this table icon and you can drill down into the data and this is a really, really easy way to do initial data exploration on a project wide level. What else can you do with the super table. So we have two ways to extract data instruments data from from from this the first is bind tables bind tables makes the data tables magically appear in your environment. And extract table is another mechanism where attract extract and returns an individual data table that you specify, and both of these functionalities pair well with a pipe operator. You can, you can use make labeled to add variable labels we love variable labels. So we use the labeled package to attach the red cap field labels, which are used at the as the data entry prompts and we use them to. Now turn them into descriptions, variable labels of the of the data columns. And finally, and this is hot of the press with the version point for that we just published a cram, you can use right red cap XLS X to export to Excel, and this feature is super handy for collaborating and it supports labels. So your collaborative can see descriptions of these variable. So here's what the project database would look like as an Excel sheet. So you, so you have a table of contents and then each instrument is in its own tab. All right, so, so the latest version of redcap tidier point four just came out of one cram a few days ago. So you can install it by typing install packages redcap tighter you get the latest and greatest. Please, please, please do test it out with your databases, and let us know what works and importantly what doesn't. We're very interested in getting feedback we really want to make this work for any, any redcap database no matter how, how weird it is. And do tell us about features that you'd like to see. If you scan this QR code here, you'll take you to the package down documentation site. And then a getting started tutorial and a bunch of vignettes, including one that that tells you exactly how that redcap tidier supertipple works and exactly what kinds of transformations. It makes to make the data more user friendly. So, let's recap. We talked about how BLL is the most common cancer in children and a CAR T cells living drug that consists of genetically modified T cells are capable of killing cancer cells. And CD 19 directed CAR T therapy has proven to be a groundbreaking treatment for relapse refractory BLL and children. And we use the trial that tests a new variation of a car that's humanized and a new manufacturing platform the prodigy to establish an end to end analytics pipeline framework for CAR T clinical trials. And redcap tidier built to support this pipeline, and makes it easy to import any redcap project into our and it's robust and performs well, it can export a collaborator friendly version of the project to excel. So now that's a very briefly consider the impacts and the benefits of this approach to date, building our own redcap databases has dramatically increased the satisfaction of our data entry personnel. I created a database build standard which enables us to use a template approach to building new clinical trial or CAR T trial EDCs. And this in turn is allowed us to build new databases for five additional trials in a small fraction of the time that may took us to build the first one. And then one key function of the dashboard is to automatically generate what's called a data and safety monitoring board or DSMB report which the view that you just saw. So these DSMB reports are created regular intervals to play a vital role in clinical trials to ensure the safety of the patients on the trial and they usually take a highly trained professional multiple weeks to complete and and our automation has reduced us to mere seconds. So I want to thank, I want to thank everybody I want to thank everybody on the CGT informatics team thanks, rich and and and Ezra. Thanks for saving my life just now by showing a view of the of the project reporter. I want to thank the clinical team and, and our funders and everybody. Thank you for letting us in our comedian everybody who came to attend my talk. Thanks very much guys. Happy to take any questions. Awesome talk Stefan. I think they're rich at most of the questions in the chat there's one quick one that I saw about having to get your red cap instance. And if you download the slides from from from sketch, you're actually going to see the high resolution slides. So anyways, that was also a question. Did you have to get your red cap instance certified party 11 compliant. So that's just an excellent question and we And our, our groups. Just this okay I'm going to take a quick step back because it's a super important question. And it's very contentious that we as a group as the wholesale therapy program have decided that part 11 does not apply to us. And this is because you know, when, because the the FDA wants in the guidance wants every center to make that own determination. And then there's and then it's very, very vague about what it is how to make that determination. One critical thing is that if you want to if you do work that's in preparation of a marketing application for for for a pharmaceutical product, then you got to do it. Otherwise, you don't. And this is how we're taking it so we're saying categorically, no we're not doing part 11. We're not going to tell anybody we're part 11 compliant. And this is this is this makes our lives much easier. However, we, we are bound by other standards there is a there's a foundation for the accreditation of the therapy. So if we as and they just also promulgate some new rules that we have to good for for for electronic records that we have to abide by. And so, so for some applications that we build we actually have to be fact compliant, but not part 11 compliant so so it becomes complicated and I mean the other thing is the question is, like do you like how important is quality in your software and quality I think is very important we try to do a lot of unit testing and continuous education and kind of stuff. But, but we want to we always take sort of a risk based approach to things, like if something cannot ever like really break if really harm a patient if something. Something doesn't work as it should then that gets validated and test very rigorously. If that isn't the case and then we don't bother or like you know for these databases we don't bother validating them because they're not really going to hurt a patient. If, if there's a data and sharing it but we still do our best to try to, to try to make them make them work well. Thanks for that question I appreciate it. Awesome thanks. All right, we're going to move on to the next talk. Thanks for stepping out super inch.