 wonderful. Thank you so much for the introduction. I'm just gonna make sure that I'm displaying things appropriately. Can I confirm that I am displaying the PowerPoint and not the slide set? Yes. Okay. Yeah. I don't trust anything. So my name is Jeremy Michael and I'm a pediatrician informaticist at the Children's Hospital of Philadelphia. And thank you for coming to my talk entitled Development and Application of an Immunization Validation Engine Utilizing R to evaluate appropriate completion of immunization series across a large primary care network. I have no disclosures relevant for today's presentation. One of my main areas of research involves the updating and maintenance of the immunization decision support used by Children's Hospital of Philadelphia. The constraints of our current system led me to investigating how R could be used to augment the support we currently supply through the analysis of clinical data. I'll begin by giving a brief overview of clinical decision support for immunizations. I'll then be describing the current clinical decision support we use at CHOP, including its strengths and limitations. I'll next discuss how we develop the R script for immunization analysis using our CDSI as a reference. And finally, I will be showing some results from ongoing projects from which we use the script. Immunizations remain one of the best public health interventions for combating the spread of communicable disease. They work by promoting a person's natural defense mechanism against specific targets, basically training your body to fight illness. Immunizations in the United States are intended to be delivered according to the routine immunization schedule developed by the Advisory Council Committee on Immunization Practices, ACIP. This schedule was designed to provide the most appropriate level of immune response for specific vaccination targets based on the risk of exposure and severity of illness. Clinical decisions for support for immunizations, or CDSI, includes tools which help clinicians determine which immunizations are due for a given patient at a given visit. Numerous studies have demonstrated that CDSI has improved adherence to routine immunization schedules. CHOP developed and maintains its own immunization care system to provide immunization recommendations in the EHR. This system has been in use since 2012 and has been used in more than 1.5 million routine health visits. The immunization care system has multiple components. It has a clinician-facing web-based interface, a Python-based immunization rules evaluator, and a database-driven store of the immunization rules themselves. When a patient's chart is opened, the system uses the current information to run through that Python-based rules evaluator against the database to generate a list of what is due for a given patient at that visit. As opposed to a system provided or out of the box model, CHOP maintains local control of all guidance provided by the care assistant. Having local control of the immunization rules allows for rapid adjustment in response to changes in ASIP recommendations. The data generated by the care assistant is primarily stored in the clinical record, but log and rules data is stored within the application itself. The care system works by first analyzing a patient's chart to determine which immunizations could be due, then uses the patient's medical history to select the applicable immunization recommendations, and then fine-tunes those recommendations based on specific modifiers and applies site-level data to ensure that the immunizations that it recommends are actually in stock. Finally, it provides a link to place orders. The clinicians do maintain ultimate responsibility for confirming and placing orders through this system. In addition to supporting order placement, the care assistant also supports documentation. At each visit, the care assistant triggers content to be inserted into the clinical documentation. This information is stored in the EHR backend at the visit level, using something called smart data elements for those who are not familiar with EPIC, that is the internal structure in which it can store information. It also uses constrained text options to support any text-based note analysis. The information inserted into the note includes what immunizations are due at the time the note is generated, which were ordered, any modifications to the routine recommendations, such as the personalizations or supply issues, and decision-making surrounding vaccine ordering, including refusal by families. After using this in practice, I can report that it's a pretty nice to use in the clinical setting. However, the system was designed with the clinician and patient dyad in mind. There are notable barriers to the system and other CDSI for data analysis. First, the care assistant is optimized for episodes of care. Each time a patient chart is loaded, the system validates historical immunization data and presents a list of recommendations. As information changes in the system, for example, a pneumatically relevant diagnosis is added to the chart. The system needs to recalibrate and provide updated recommendations. This mutability of data can make it difficult to analyze populations in real time. The system also only has access to clinical data reconciled into the EHR. Other data, such as data from the state immunization registry is not evaluated. While we have shown our internal aid to be fairly comprehensive, it does have the potential to impact evaluations of subpopulations who may not have their data in the system. There's also no way to use the current system to evaluate patient's panels or population level data. Analysis on the patient level will be two-time consuming to perform on large panels. Additionally, whether our EHR supplied mechanisms, such as Slicer Dicer, those do not work well for us because they can't see our rules engine. Finally, there's no way to evaluate historical data. For example, if I wanted to look at patients who are seeing pre and post COVID and what their vaccine rates were, it would be impossible for me to do that using the current system, because everything in the care assistant is evaluated in real time. Given the topic of this conference, it should not be surprising to note that we leveraged R for analysis of our immunization data. Our objective was to create a reusable script for analysis of immunization data across patient populations. We sought to use software that is endorsed by our department to support collaboration. This meant for us using the beaver to extract information from the clinical record and generating datasets and then using R to analyze the data. We used our previous work with the care assistant validation rules authoring to support our script development. First, we identified the minimum data needed for validation of immunizations. This turns out to be not a lot. The included data we needed was the immunization product itself, the age the immunization was given, and the date the immunization was given for products where rules have changed in recent years. We then used R to merge the data from the electronic health record with registry data to develop a comprehensive picture of the immunizations given at the shop. We expressed immunization validation rules from the CDSI using R scripts. This involves separating out for analysis, all immunization targets within a single product. For example, the product pro quad includes two immunization targets. It includes MMR and varicella. We then iterated over the immunizations to determine each was given at the valid time and age to develop a comprehensive picture of the immunization status for a given patient for each immunization target. Finally, we assessed immunization status on patient cohorts using various metrics we encoded into R. To determine which antigens were included in each product, we need to use a combination of regular expressions on immunization product names and curated lists. For example, for the product, if a product to determine if a product covered hepatitis A, we looked for the following things. We looked for the term HEP followed by a white space followed by an A. We looked for the specific project named TwinRx and we looked for the term HEP A HEP B. We then had to exclude keywords such as globulin as these products with those names show up in the immunization table, but they are not themselves immunizations. To adjust issues related to divergent validation rules for adult and child hepatitis A products, we had to use the epic product numbers as the historical names were not sufficiently granular and their adult and child hepatitis A products are managed slightly differently. As each product could contain multiple antigens, we can create a column within the data table and store a true false evaluation of whether a product contained that antigen. When validating an immunization, each antigen need also need to be considered separately. It may be the case that an immunization is given when some components were valid, but others were given too early to be considered valid. By performing the remainder of the analysis at the antigen level, we could avoid partially validating or invalidating data rows. Therefore, from the curated immunization list, we extracted and created a table of administered antigens. When multiple doses of the same antigen are given to a patient, we needed to calculate the amount of time between the doses. And we did this using a mutate function. Calculating the interval between doses is essential because unless a dose is given at the right amount of time from the last dose, it is also not considered valid. At this step, we also took the time to create placeholders for information that would be necessary for validating later doses in many of the primary series, such as the first hepatitis B vaccine or the first tetanus vaccine. Since validation of each subsequent dose in the series depends on validation of earlier doses, we performed a stepwise validation of each the routine immunization series. We determined the validity of the first dose in the series using the patient age and immunization data. For example, the first tetanus vaccine was only considered valid if it was given after six weeks of life. But certain hepatitis A vaccines had to be handled differently, depending on if they were given after the first birthday or after the 19th birthday. If a dose was invalid, the second dose was promoted to become the first dose and again was checked for validity. Once a first valid administration was identified, for each of the routine vaccines for each patient, the system begins its loop for subsequent vaccine doses. Subsequent evaluation loops may have included depending on the vaccine that we were looking at or depending on the antigen series that we're looking at additional validation assessments, such as the minimum time between doses or the minimum time since the first dose. This is the validation loop for the first dose in each routinely recommended immunization series. From the data table, we see all rows with an admin counter indicating the first potential dose in a series that is over the age for validation. We select those using these rows. We're then able to establish a definitive dose counter and set the validity of that antigen to true. If this dose would be the last in the series at this point, we would note that note that. And we'd also note if the vaccine was given on time or delayed any dose within an admin counter of one are not that are not validated in this system are then determined to be invalidated invalid doses and removed. The admin counter is set reset for that patient for that antigen and the loop is repeated until there are no first instances of a vaccine series that have not been validated. Of note, depending on the patient's age, there may have been different validation roles. For example, for pneumococcal vaccine, there are two roles that we created. One determined if the valid dose was given before or after the second birthday. This allows the system to quickly determine if a pneumococcal series is complete. We also used R to encode public health metrics and other analytics functions. I'll be briefly showing two examples. The adolescent immunization combination two or the heatest measure and our calculation of up-to-date status. The adolescent immunization combination two is one of the simplest metrics to calculate, which I think contributes to its popularity. To evaluate this metric, we only need to look at three antigens and only vaccines that are given over a brief period of time. We used R to evaluate if any dose given would signify completion of one of the components of the series using the age that it was given and the dose counter variables. We then merged this with our patient data to create a yearly view of patients based on completion of this metric. Using this, we were able to demonstrate that immunization rates were improving among adolescents in our practices and that this was driven primarily by an improvement in age appropriate acceptance of adolescent immunizations in suburban populations. As a more complex test, we sought to determine the up-to-date status for against all routine immunizations for each patient. Up-to-date status involves calculating the number of valid doses received by a certain age, recalling that in some circumstances the age at which one gets a dose changes the number that are needed to complete a series or to be considered up-to-date. We also had to account for something that makes this more complex in that up-to-date status in itself is a moving target. For example, if you are age three and you have one MMR vaccine, you're considered up-to-date, but if you're age five and you have one MMR vaccine, you're not considered up-to-date. Additionally, some immunizations are not started after a certain age. For example, Hib and pneumococcal are not started typically for children after the age five and therefore would not be counted against up-to-date status for older children. Children have entered our practice after age five. We used a combination of series completion determinations made during the validation process, the maximum dose counter in the series, and the patient's current age and the age of the first VEX visit to estimate the up-to-date status for our population. Here we note that up-to-date status of patients as of June 30th 2022. Having the ability to run this as a reference data has allowed us to examine how subpopulations are compared doing compared to the general population. We're also using these scripts in multiple other projects and evaluating them, particularly looking at the uptake of COVID-19 vaccines in general and subpopulations and where and when vaccines are given. Also looking at routine immunizations particularly investigating acceptance and refusal. And finally we're also considering its application this for the script and identifying patients who are delayed on vaccines and may need to be called in thinking about some of the quality improvement aspects that we can be using this for. So in conclusion we successfully developed in our script to validate immunizations, leveraging our experience with immunization validation within our clinical decision support systems. We've also evaluated different use cases for this validation script and particular projects that it can be useful for. Finally we note that validation, up-to-date status, determination, and quality metric evaluation have strong similarities but did have differences in their data needs for evaluation. I want to thank the various project teams who have been involved in this work either by testing this or asking questions that led to the creation of the script. My hope is to compile this as a resource that can be used repeatedly to support other immunization data analysis. Thank you very much for your presentation. We have one question is A1C2 metric achievement associated with CDC social vulnerability index by census tract. I don't know the answer to that question. One of the things that I can tell you the main thing I can tell you about the the adolescent immunization thing is it actually is better in the urban populations and the primary reason for that is not refusal to get HPV vaccine for the urban population compared to the suburban population or at least not refusal to delay it until kids are 13, 14, or 15. Peter, partially joking says can you identify neighborhoods with low vaccine rates and send out a vaccination ban that plays the ice cream truck music? It's possible but I don't think that would work well and I think I would get fired if I tried. Well thank you very much. There are more questions. I'll let you go but thank you very much for your presentation. Tempting to stop?