 My name is Kevin Mullen. I'm the chair of the Greenbound Care Board and I'm gonna convene this meeting to order. We usually start out with the executive director's report. Our executive director is under the weather today. So I'll make a few announcements on her behalf. First of all, we have a couple of open public comments. One being the Vermont Health Information Exchange strategic plan. And we're looking for public comments by November 25th. And the second open public comment is the ACO budget. And we're hoping to take any public comment on the ACO budget by December 1st. So those were the announcements that I had other than just to remind everyone that the Greenbound Care Board has a meeting Monday morning at 8.30. And the purpose of that meeting is the Springfield Hospital budget. So with that, I'm going to turn it over to Sarah Kinsler to tee up today's discussion. Sarah. Thank you so much, Mr. Chair. I'm just gonna take a moment to share my screen. Are you all able to see that? And does it look normal? It looks normal. Great, thank you. So for the record, this is Sarah Kinsler, GMCB's director of health systems policy. And I'm here to just give a quick introduction. So today's main event will be a presentation from the researchers at NORC who were hired by the federal government to evaluate Vermont's all-pair model from the perspective of Medicare. This presentation is going to summarize their first evaluation report, which was released in late August. And that covers the first two years of the model 2018 to 2019. I wanna extend our huge thanks to NORC and the federal evaluation team for taking the time to be here with us today to share those early findings with us. Following NORC's presentation, we will have time for the board to discuss what they've heard as well as for public comment. I wanna just take a moment to provide some context and situate today's presentation. Vermont and the GMCB assess APM performance and track it over time in a couple of different ways. First is the data collection and reporting that we do related to the targets in the all-pair model agreement. The board staff prepare the lion's share of the reporting required by the APM agreement with help from our analytics contractor, including regular reporting on the agreement targets for total cost of care, for quality and health outcomes and for scale. All of these reports are posted to our public and are posted to the all-pair model reports page on our website and the performance is summarized in a Tableau dashboard produced by our data team. I do want to take a moment to note that the state recently received notice from our federal partners that they are waiving enforcement of the agreement scale targets through the current agreement term, though the state will continue to report on all-pair model scale. And there's a note about that on the next slide, including a link to that letter. In addition to the reporting required by the agreement, we also look for other ways to assess the model and its successes and challenges, including through payer-specific data, through qualitative and stakeholder input and special analyses as well. And finally, today's presentation, CMS has contracted with NORC to evaluate the model over the course of the agreement. And this is definitely the most methodologically robust evaluation of the model, though I'll note that the analysis does emphasize Medicare. Zooming out a bit, I also want to remind us that this is not the same as the board's work to regulate Vermont ACOs and assess ACO performance, which I know is top of mind for many of us at the moment because we have one care Vermont's annual budget review hearing next week on Wednesday the 10th. But these processes do inform each other. Finally, I want to note that this meeting is a bit different than most of our GMCB meetings in that NORC is not able to take live questions due to the required federal clearance process other than to clarify specifics related to the presented slides. The meeting will still include board discussion and public comment as it always does. And with that, if there are no questions or comments from board members, I would hand the mic over to NORC. Actually, Sarah, I'm going to jump in there right quick and introduce myself and I think just kind of reiterate some of the things that you said. Hi, everyone, my name is Franklin Hendrick and I am a centers for Medicare and Medicaid Services Federal Employee who is the government lead on the evaluation of the Vermont All-Payer Accountable Care Organization model. For some additional context, each center for Medicare and Medicaid innovation model of which the Vermont All-Pay-ACO model is one must be evaluated to assess the impacts on spending and quality of care. As Sarah mentioned, CMS is contracted with NORC at the University of Chicago to conduct this required evaluation, the Vermont All-Pay-ACO model. And so without further ado, I will then turn it over to Sai Loganathan from the NORC team who serves as the project director on the evaluation to kick the presentation off. Thanks for letting me jump in there real quick. Take it away, Sai. Thank you, Franklin. Good afternoon, everyone. And thank you for the opportunity to present our findings today. I'm Sai Loganathan. I'm a researcher at NORC and I serve as the project director on this evaluation. The NORC team is tasked with designing and carrying out an independent evaluation of CMMI's Vermont All-Payer-ACO model. And today our team is pleased to present the early findings reported in our first evaluation report. Just a quick note about NORC. NORC at the University of Chicago is an objective non-partisan research organization. We provide comprehensive and integrated services that span the research cycle across five main areas, economics, markets and the workforce, education, training and learning, global development, health and well-being and social, society, media and public affairs. I'd like to take a moment to acknowledge the contributions of our entire evaluation team. If you can move to the next slide, please. Including our partners, Actuarial Research Corporation, the Center for Healthcare Strategies and our advisors at the University of Minnesota. My colleagues, Erin Ewald, our impact analysis lead and Rachel Singer are qualitatively will be joining me today in presenting the findings in the first evaluation report. We'll begin our presentation by providing an overview of our evaluation approach, including some of the evaluation design considerations we had to address. And next we present our findings on participation in the model, implementation as well as impact. And we'll conclude with a summary of our key findings and some of the next steps in our evaluation. So this slide presents our evaluation objectives. Our evaluation is assessing impact of the model on population health outcomes, as well as spending utilization and quality of care for the Medicare population, as well as the Medicaid population. In addition, we're assessing model implementation, as well as replicability and sustainability of the model. Our evaluation utilizes rigorous mixed methods design. We draw on multiple primary and secondary data sources and methods, including CMS administrative data, key informant interviews and a survey of practitioners. The findings from the survey will be presented in future evaluation reports. We just got finished collecting the data. And I will turn it over to my colleague, Rachel Singer to continue the presentation. This conceptual framework informs our understanding of the model and our approach to evaluating the effectiveness of both implementation and the impact of the model. Our evaluation looks at the role of contextual factors, including Vermont's history of healthcare reform efforts and the regulatory role of the GMCB, as well as the state's healthcare market. We also consider the model's design features, such as the GMCB's oversight authority for the model, and as well as Vermont's flexibility to determine ACO outcomes, set ACO benchmarks, structure risk arrangements and payment mechanisms, investing care management and monitoring and enhanced benefits. The evaluation includes an assessment of the perspectives of the model participants and implementation partners on the implementation of the model, which in turn inform our interpretation of outcomes. Now to the next slide. In evaluating the all-pair model, we recognize that this model builds on decades of innovation in the state. The blueprint for health set the stage with the focus on supporting primary care practices and achieving and maintaining PCMH certification and a focus on community-led strategies. Under the multi-payer advanced primary care practice demonstration, Medicare, Medicaid and commercial payers provided a per member per month payment to certified PCMH practices and also supported community health teams and Medicare additionally supported the supports and services at home program. The Vermont Health Care Innovation Project under the CMS State Innovation Model Initiative set the stage for the implementation of the all-pair model with a focus on value-based payment models, including participation in the Medicare shared savings program and the launch of the Medicare Medicaid shared savings program. As we acknowledge later in this presentation, the context in which the all-pair model is being implemented impacts our interpretation of our impact findings. Moving on to the next slide. Today we're presenting findings from our first evaluation report, which is the first in a series of public reports summarizing evaluation findings. This report focuses on the first two performance years of the model, 2018 and 2019. It includes state and ACO level model impacts on Medicare spending, utilization and quality of care and presents findings on implementation experiences and participation. In future years, we will incorporate impact findings for the Medicaid ACO as well. Okay, so this slide summarizes the impact analysis designed for our evaluation. We use a quasi-expend from experimental difference and differences analysis to assess the impact on Medicare spending, utilization and quality of care as Rachel said for the first two years of the model, which is 2018-2019. So using this design with a comparison group and a pre-intervention period allows us to isolate the impact of the model apart from any secular trends in the baseline period. The comparison group, for instance, larger nationwide shifts in Medicare payment and spending over time. Due to the model's multi-level accountability and incentive structure in which effects are monitored at the level of both the state and the ACO, we estimate the model's impact at both those levels as well in our evaluation. So we define the population for these analyses based on where and from whom Medicare beneficiaries are receiving care. Both levels of our analysis include fee-for-service Medicare beneficiaries residing in Vermont and for the ACO-level analysis, we limit the treatment group to Vermonters who have received the plurality of their care based on paid claims for E&M visits from model practitioners. For the state-level analysis, we include those who received the plurality of their primary care services within the entire state of Vermont, regardless of whether their provider was participating in the model or not. We'll go into more details about the comparison group in the next line, but at a high level, the comparison group structure mirrors the structure of the treatment groups, as they both also included Medicare fee-for-service beneficiaries, except for the comparison group that were residing in 26 comparison states that we identify, and we can get more into the details in the next slide. But for the ACO comparison group, we identified beneficiaries who received the plurality of their care from Medicare Track 1 SSP ACOs. So by using a comparison group with similar experience in Medicare ACOs, we're able to measure the additional effect of the all-pair framework on the existing ACO model. For the state-level analysis, the comparison group comprised of Medicare fee-for-service beneficiaries residing in our 26 comparison states without regard to any specific provider's experience with Medicare ACOs. Next slide, please. Okay, so this slide shows the four stages of our comparison group construction with the Vermont treatment group depicted on the left-hand side and the comparison group shown in parallel on the right-hand side. So because this model and other initiatives in Vermont were implemented across the state and we expect impacts of those initiatives be realized statewide and not just for individuals attributed to a specific provider who is in the model, we chose to use an out-of-state comparison group. So in stage one, we identified 26 comparison states with a similar history of healthcare reforms as Vermont based mainly on whether they had existing NCQA-certified Patient-Centered Medical Home Programs and multi-pair reform initiatives in the baseline period, which was 2014 to 2016. So multi-pair reforms include other CMO models such as the state innovation models and the multi-pair advanced primary care practice demonstration, both of which Vermont participated in. So that's why we chose those. Once we identified the 26 comparison states, in step two, we drew a stratified random sample of Medicare beneficiaries residing in the states over sampling rural areas to increase the likelihood that the comparison benes would reflect a similar geographic distribution as those in Vermont. And the reason we took a sample to include all eligible beneficiaries in the state was just due to computational constraints. And in step three, which was only performed for the ACO level analysis, we identified Vermont treatment beneficiaries using concurrent attribution to identified benes who got the plurality of their care from a provider who was participating in the model. For the comparison group, we used these same attribution rules to identify benes who were aligned to practitioners participating in, as I mentioned, track one, Medicare SSP ACOs. And in step four, we used beneficiary and market level factors to weight the comparison group benes identified in step three to make them as similar as possible to the Vermont treatment group. This weighted comparison group that comes out of stage four is the comparison group we use in our impact analysis regression models. Next slide, please. Thanks. So before we present the results later in the presentation, we wanted to highlight some of the limitations and the methodological challenge we face specifically related to the construction of the comparison group and how we mitigated those limitations. So Vermont is unique in its socio-demographics, healthcare market and history of health reform as we saw a little earlier in our presentation on the graph. And because of this, identifying candidates for our comparison states and comparison beneficiaries was pretty difficult. So at the beneficiary level, there are meaningful differences between Vermonters and those who reside in similar regions of the country in terms of reality and demographic characteristics. So to get around this to the extent possible, we used a method called entropy balancing to weight our comparison groups so that they look more like Vermonters in terms of their individual and area level characteristics. But even after we took those steps, some differences remain between our treatment and comparison groups. I will mention in the technical appendix for our first report, which is available on the CMS website, we provide detailed statistics and more information on each of these variables that we used, including the ones that were still dissimilar between the two groups after we applied these weights. So we also found that Vermont and the comparison group had different trends in the baseline period, 2014 to 2016 on some of our key outcomes indicating that they were on different trajectories even before the model started. So to mitigate this in our regression models, we included an interaction term essentially that allows these baseline trends to vary and accounts for that in our impact estimates. We also observed meaningful market level differences, specifically in the rate of Medicare Advantage penetration, which is much lower in Vermont compared to most other states and the prevalence of upside risk ACOs, which is much higher in Vermont relative to other states. So to account for this, we limited our ACO level comparison group to only beneficiaries seen by Medicare SSP ACO providers. We hope by limiting our comparison group to those providers who had Medicare ACO experience, we might be able to adequately balance on these other market level factors. But because Vermont was such an outlier among other states in these factors, there's still sizable differences between the treatment and comparison groups. Okay, and in this last row, we just want to reiterate the fact that it's likely that preexisting health reform initiatives in Vermont have been driving some of the early results we see in this model, especially as many components of the existing initiatives in the baseline like blueprint are now incorporated into or continuing under the model that we're evaluating. We try to mitigate this to the extent possible by choosing comparison states based on whether they implemented some of those similar health reform initiatives, like we said, SIM, MAPCP, but again, there were still meaningful differences between Vermont and our comparison states after we did that. We provided additional context around this issue in our report and highlighted that as a potential driver of the results we're seeing as well. The first evaluation report includes findings also from two rounds of semi-structure interviews in June of 2019 and July to September of 2020 with state officials, one care leaders, blueprint staff at the state level and from across the state, hospital leaders, physicians, and representatives from designated mental health agencies and FQHCs. In total, we conducted 49 interviews with representation from 12 of 15 HSAs. We also reviewed existing documents such as state and ACO level budget documents as well as a wide array of public information available on the state and GMCB websites. Moving on to the next slide. Before we get to the impact results, we wanted to briefly set the stage and provide context on who is participating in the all-payer model, particularly in the Medicare ACO initiative, which is the focus of this report. Next slide. Participation in the model is voluntary for both payers and providers. Home hospitals are the primary disappearing entities and can choose to participate in each payer initiative, Medicare, Medicaid, commercial. Providers who choose to participate are not required to do so with all participating payers. Unless the home hospital and a given HSA opts to participate in the model with a given payer, other healthcare providers and practitioners and the HSA will not be eligible to participate. Which we'll go back to in some of our results. Moving on to the next slide. With respect to payer participation, Blue Cross with Shield of Vermont was the only commercial payer in the model in 2018 and 2019. And in the first two years of the model, the two largest self-insured plans in the state did not participate. With respect to hospital participation, while 13 of 15 hospitals serving Vermonters participated in one or more of the three payer ACO initiatives, Medicare, Medicaid or commercial, in 2019, eight participated in the Medicare ACO initiative, which was up from six in 2018, still around half. Only two of seven critical access hospitals opt to participate in the Medicare ACO initiative. Moving on to the next slide. Moving to practice and practitioner level participation. The number of practitioners participating in the all payer model increased between 2018 and 2019. However, the share of providers who were eligible for an opt to participate in all three ACO initiatives decreased slightly between 2018 and 2019. Practitioner participation mirrored that of hospitals with the Medicare ACO having limited presence in the more rural areas of the state per hospital participation. Moving on to the next slide. As you know, Vermont is responsible for meeting skill targets for model participation, namely to attribute a minimum percentage of the Medicare, Medicaid and commercial populations to the model for each performance year. Medicare beneficiaries are attributed prospectively to the model each year based on beneficiaries past care utilization patterns. As you can see here in GMCB's analysis for both 2018 and 2019, the model didn't meet the all payer and Medicare specific skill targets. The participation did increase for both Medicare and Medicaid between 2018 and 2019. In 2019 GMCB's analysis of the model's progress found that 30% of the eligible insured Vermont population was attributed to the model across all payers falling short of the all payer scale target by 20 percentage points. For the Medicare ACO initiative, 47% of eligible Vermont beneficiaries were attributed. Commercial participation remained low and steady during these two years. Moving on to the next slide. Again, here we're focusing on Medicare scale target performance only. When you look at who is actually receiving care in the state, the model has a much better reach. Specifically, over 25% of eligible Vermont Medicare beneficiary of the eligible Medicare beneficiary population did not receive any qualified evaluation and management services within the state in 2019. Taking this into account, we use an alternative approach to assessing scale target performance excluding these beneficiaries who did not receive any care in Vermont. With this alternative assessment, the model performed better covering 65% of Medicare beneficiaries in 2019 compared to 47% in the GMCB analysis but still did not achieve the 75% scale target. We're now going to provide some high level qualitative findings around implementation of a model based on interviews conducted in 2019 and 2020. With regards to implementation of the payment model, state level leaders and providers interview described a positive experience with Medicaid unreconciled prospective population based payment. One state leader noted that this payment structure allows providers to get quote, the predictable chunk of money upfront for Medicaid to more flexibly spend. This was particularly valuable with reduced volume during the COVID-19 pandemic. Neither state level stakeholders nor hospitals understood at the outset of the model that there would be a shadow fee for service component to the Medicare all inclusive population based payment that Medicare would recoup the all inclusive population based payment against fee for service claims. When negotiating with CMS, state level leaders expected that the Medicare payment model would be full capitation, which is how the state designed the Medicaid model and hospital leaders had similar expectations. State level and hospital leaders underscored the challenge of operating in both fee for service and value based payment models simultaneously. With limited scale provider organization described operating with their feet and two canoes while continuing to rely on traditional fee for service reimbursement as a large portion of their operating revenue while trying to move towards enacting care delivery reforms. Administrators described the challenge of managing projecting the revenue with two payment models, especially with one being reconciled and providers expressed a reluctance to provide different models of care based on patients payers. Moving to the next slide. As we noted earlier, and as you're aware the all payer model built on previous initiatives. The funding structure of the model allowed for continued funding of existing blueprint initiatives including the patient center medical homes, community health teams, sash with funds now going through the ACO. As we will come back to later these initiatives serve the entire community not just ACO attributed beneficiaries. One funding stream that is new under the model is support for complex care coordination for patients considered high and very high risk based on a risk stratification algorithm. In the first two years of the model this funding went to primary care practices and community providers including designated mental health agencies, home health agencies and was intended to build capacity to provide care management. The increased funding has allowed expansion of some services already provided through the blueprint such as community health teams. However, some community providers and organizations were reluctant to use this funding to hire new staff and increase capacity because they were concerned about the sustainability of the position should the model be terminated or should the ACO change the payment structure. Stakeholders at the local level reported that the model is strengthening relationships between hospitals, community organizations, designated mental health agencies, primary care practices and other providers and provides a mechanism for collaboration across the continuum of care and reduces competition and to address population health goals. While the model is still in development the care coordination model and particularly in the first two years even early on providers and community organizations shared that the model is beginning to bring hospitals to the table in communities where they had not previously been involved in community level initiatives. Moving on to the next slide. On provider engagement in the first two years of the model one care has struggled to engage critical access hospitals federally qualified health centers and independent providers. We found that there was a perception among some non-risk bearing non-hospital providers that they've been sidelined. Stakeholders also share their concerns that one care was not positioned to support critical access hospitals and independent practices in the model's financial structure and that there were limited incentives for non-hospital providers. They also expressed concern about the ACO's relationship with academic medical centers. I'll now turn it back to Erin to go through our impact results. Thanks Rachel. So next we'll walk through some of the impact analysis results from the first two years of the evaluation. As a reminder we're focusing on impact for only Medicare beneficiaries in this section of the report. So as I mentioned earlier we're estimating impacts in our evaluation at two different levels the ACO level and the state level which have different comparison groups although both are made up of Medicare beneficiaries residing in our comparison states that we identified. The ACO analysis has a comparison group of Benny's attributed to other Medicare ACO providers and so the results reflect the impact of that all-payer ACO framework beyond any single-payer specific ACO models that were operating prior to the model. The state analysis on the other hand has a comparison group of beneficiaries residing in those comparison states regardless of whether providers whether their providers were in an ACO and results from that analysis may reflect the impact of Vermont statewide payment and delivery system efforts for Medicare beneficiaries. And an explainer on the two different types of costs on this presented on this slide gross spending and net spending. So gross spending we calculate the impact for Medicare parts A and B spending and net spending is the impact for Medicare parts A and B spending after accounting for CMS incentives to model providers and comparison providers in the baseline and performance years. So providers in the comparison groups are participating in SSP, Pioneer, Next Generation ACOs and they are also receiving CMS shared savings and losses during the baseline performance years. So we account for those in our net estimation of the net Medicare spending. So overall we found that the model reduced Medicare spending for beneficiaries who were in the ACO as well as statewide relative to our comparison groups in the first two years of the model. The cumulative impacts on this table for reference represent essentially a weighted average of the beneficiary per year estimates for PY1 and PY2. So we see a decrease in gross Medicare spending of $607 for the ACO analysis across those two years and a decrease of $783 in the state level analysis both relative to the comparison group both of which are significant. We also see a significant decrease in net spending at the state level of $748 after accounting for those CMS incentive payments that I just described. And as a note, asterisks in this table and throughout the rest of the presentation indicate significance of the PY1 level or lower. Next slide, thanks. So these charts, they show the same gross in net spending estimates that we summarized on the previous slide with the dots representing the numerical point estimates. But we've also in these charts depicted the uncertainty around these estimates with 90 percent conference intervals which are the little bars on each side. And it's important to note that all of the numbers we're presenting here and throughout our presentation they all have some degree of uncertainty associated with them and we should be interpreting our findings in light of that level of uncertainty. And so we'll be presenting the impact estimates with the conference intervals with this level of uncertainty for the utilization findings as well and later in the report. So, and as a final note on this slide we'll discuss more about potential drivers when we present the utilization findings but due to the issues we discussed earlier in terms of overlapping and existing initiatives keep in mind that we're not able to attribute model impacts to one specific programmatic driver. Next slide, please. Okay, so this slide shows the trends in total Medicare spending for the ACO and state level across all the years that were in our analysis. So 2014 to 2019. And as a note here, we consider 2017 as a ramp up year as we indicate in these churches PY0 as the model implementation was just starting. So 2017 is not included in our analyses. Our analyses when we talk about PY1 and PY2 are just 2018 and 2019. So looking at these charts we can see that the reductions that we observe in Medicare spending for both the ACO and state analyses generally reflect rising spending in the comparison groups, which is in the gray dotted line and relatively flat spending in the comparison groups which is the solid orange line during the first two performance years. And we can see that this is a trend which started during the baseline period for Vermont again likely due to Vermont's long history of reform initiatives ongoing in the baseline period. So as we discuss these next slides with utilization findings keep in mind that these are all relative to the comparison groups which may itself also be showing changing trends over the baseline and performance years. Next slide, please. Thanks. So next we'll turn to the utilization findings. To provide some framing we're looking at these utilization impacts to better understand the potential drivers of the cost impacts we're seeing. So this side shows our main findings in terms of hospital utilization which includes acute care stays days and 30 day read missions. So in PY2 we see decreases in acute care stays and acute care days both the ACO and state level relative to comparison group and also a decrease in 30 day read missions for Vermont beneficiaries in the state level analysis in both performance years. So due to the cost of these types of inpatient episodes and hospital based care generally we think that the utilization decreases on this side are the likely main drivers of the overall spending reduction we're seeing. Next slide. Thanks. In addition to those decreases in acute care we also saw some changes in ambulatory care. First we see reductions in specialty E&M visits across both ACO and state analyses. From our analyses it's not clear whether the reduction in specialty visits is intentional or whether this is a fact of access to care, specialist shortages or other factors in the healthcare market. We also see a decrease in annual wellness visits at the ACO level and again keep in mind that the decreases we're seeing here are relative to the comparison group. So for the annual wellness visits the decrease we're seeing for the impact estimate is due to the track one SSP beneficiaries and the comparison group essentially catching up to the already high level of annual wellness visit utilization that Vermont beneficiaries had in the baseline. Because the annual wellness visit utilization in the comparison group increased much more in the performance years they had more ground to cover essentially than the Vermont group did. We're seeing an overall net decrease in terms of the impact for this outcome. And as a reminder in the technical appendices for our report we showed the baseline and performance adjusted and unadjusted means. If you have additional questions about specific measures and increases and decreases for the comparison group versus Vermont. So going back to our initial framing to wrap it up of looking at the utilization findings to explain what we're seeing in terms of the overall spending reduction. We think that the declines in acute care are so likely the main contributor for the overall decreases in spending based on the fact that acute care utilization is much more expensive and a larger driver of healthcare costs overall but that these decreases we're seeing here on the ambulatory care measures maybe secondary drivers in addition to that. And so I think with that I'll hand it back over to Sai to summarize our findings. Thank you, Aaron and Rachel. In conclusion, the early findings show that although the model did not see broad participation across all payer models in a Medicare, Medicaid and commercial the model achieved statistically significant cumulative growth in Medicare spending reductions at the state level and then cumulative growth spending reductions at the ACO level over the first two performance years. So Vermont's culture of reform and the model's continuation of years of investment in population health initiatives may have likely contributed to these early outcomes. In addition, a qualitative findings show that the model participants reported the model providing a unifying forum for hospitals and community organizations to collaborate. However, a lack of widespread understanding of the model and perceived lack of transparency and distressed among some practitioners contributed to the challenges with engaging these practitioners and achieving broader participation in the model. And a stakeholder has also noted that beyond payment reform, delivery system transformation will require more comprehensive transition to value-based payment and more of a focus on upstream investments in population health that address social determinants of health. As for next steps in the evaluation, we will continue to assess the model's impact on the Medicare population in future performance years. In addition, we are working on assessing the impact of the model on the Medicaid population. We also plan on assessing the model's impact on population health outcomes. And finally, we plan on integrating findings from our practitioner survey in future evaluation reports. So that brings us to the end of our presentation. Thank you for your interest in our evaluation findings. Thank you. We really appreciate you coming before us and showing us what you found and helping Vermonters to understand what you found as well. And I think that the board is fully understands the position that we can't ask direct questions, but any comments from the board will gladly be taken and I'll start in alphabetical order with member Holmes. Well, very much appreciated. I know that we can't ask questions. You know, I'm jotting down tons of notes and thinking ahead about our work that we can be doing at the state level, even currently before we think about our next model, you know, particularly among some of the findings around lack of understanding of the model and lack of transparency. So thinking about how can we improve that? So there's greater understanding about what we're trying to accomplish here. Thinking ahead, you know, as we, you know, obviously there's a lot more work to be done to evaluate the model. This is just the first couple of years, first two performance years. So I'm very interested in understanding the population health outcomes analysis that's gonna be forthcoming and the results of the provider survey. I think that will be really eye-opening. And I guess I think about all of this as, you know, possible information and education as we're thinking about our next agreement with the federal government. What can we learn from this? Where can we improve in our negotiations? Where can we improve in the way in which we're sharing information about what we're trying to accomplish here, improve our transparency, improve our understanding? What is the appropriate time period? I was struck by the notion that some providers felt that the time period was too short to encourage investment in the staff and the care coordination staff. So how do we think about that? So I guess I'm just, I'm processing this all and thinking about how do we build on this to improve, you know, health outcomes and reduce costs for remoders in the next round of what we do. And what can we do in the meantime to improve understanding and improve transparency? So those are just my initial kind of first pass comments, Kevin. Member Robyn Lunge. Thank you. And thank you for the presentation. I think it's important for us to, as Jess said, learn from the evaluation material as we go. And I feel like Jess covered it. So I don't really have much else to add other than it was interesting to hear the information and to understand some of the positives and negatives in the first two years. Thank you, Robyn. Next we'll turn to board member Tom Pellum. Tom. Well, I too wanna thank you for all this work. I find it fascinating. I do take it with a grain of good salt, organic salt that it's looking at 2018 and 2019, which is kind of the beginning of the all-payer model. And the results are good and they look good. I don't think the results we can take to the bank, but I am very grateful that you are an independent organization and that you are aligned with CMS and yes, this material is very complicated and hard to explain. And I think that you've kind of cut through that kind of murkiness of it to underscore kind of the essentials. I think the best thing about the report is that it provides a baseline for future evaluations. So we have a starting point that is a point in time that might be foundational and that as we move forward, we have a point we can look back to as a starting point in a kind of thorough and technical analysis of this. I do hope that those trajectories that you showed in terms of the spending reductions, those are significant percentages. I don't know which ones of them are statistically significant and which ones aren't, but they were definitely trending in the right direction. So I'm hoping that 18 and 19, 2018 and 2019 are a good starting point and have sent us in the right direction. And in the next stage of your evaluation, we will find more of the same and but we'll be at a point where the people in Vermont and the Vermonters can begin to see these results on a day-to-day basis or a week-to-week basis or a month-to-month basis and how they relate to their healthcare system. Just a technical question for the other board members. I had a message pop up while I think it was when Robin was on that Teams detected a network issue. Did anybody have any problems hearing anything today? So for me, there was a short 20 seconds where there was a technical problem, but it wasn't, it wasn't bad. Okay, Robin, are you okay? Some of the slides were slow to advance, but I had the website version, so it was not a problem. And I assume that's probably internet on my end. Okay, it seemed to be okay on my end, but I had lost the internet connection earlier this morning. And so I just wanted to make sure that everybody was able to hear everything. And so that's good to hear. So with that, I'm going to turn it over to public comment and any member of the public who wishes to comment at this time, if you could raise your hand, that would be appreciated. And I'm gonna start with Julie Wasserman. Julie, Julie, if you're speaking, you must be on mute. Yes, thank you. I appreciate you reminding me. My first concern is why did the study focus only on Medicare lives when these lives comprise roughly about a third of all ACO participants? Now, I understand and I ask that question rhetorically because I understand that CMS will not permit the researchers to answer questions so much for transparency. But compounding the problem of studying only one third of all ACO participants, those ACO lives were a minority of Vermont's Medicare population. So in essence, we're talking about a minority of a minority. The ACO Medicare lives that were studied made up only about 33% of all Vermont Medicare lives. That was in 2018 and in 2019, 47% of all Medicare lives. Now, granted, 47% is much better than 33%, but neither are representative of Vermont's Medicare population. Now, the study would have been better off studying Medicaid ACO lives since there are far more Medicaid lives participating in the ACO than there are Medicare lives. And we all know Medicare participation in the all payer ACO model has been robust. The same cannot be said of Medicare. So we're hoping that this was not because the Medicaid ACO financial losses were so significant, especially in 2019 that it needed to be avoided. Regarding savings, I think it's important to be clear about the data and look at statistical significance as Chairman Pellum suggested. First of all though, the report underscores the fact that Vermont Medicare spending has been relatively flat since 2014, long before the all payer model came into being. But more importantly, the ACO had no net Medicare savings. I repeat, no net Medicare savings. Now, net savings are the true and most accurate indicator of savings because they remove things like CMS's pass-through payments for the blueprint. So given that net savings are the most accurate indicator, I find it rather interesting that there were actually net savings with the state level group. So that means that the state level group actually did better than the ACO. And if the state level group is outperforming the ACO in Medicare savings, then we need to question the value of the ACO. And lastly, I think it's quite unfortunate that the ACO had significant declines in terms of their beneficiaries receiving annual wellness visits. Now, these decreases, declines of annual wellness visits occurred in both performance years, minus 43% in the first year and minus 34% in the second year. Now those are big numbers. And I find it a bit depressing given that the fundamental underpinning of value-based care is to keep people healthy. And how do you keep people healthy? The answer is annual wellness visits. Thank you. So Julie, just a couple of points. I do think, and Sarah Kensler, you could correct me if I'm wrong that the evaluation itself was paid for by Medicare and that it was required under the model. And so Sarah, do you wanna jump in? I see you're back on screen. Sure, yep. I think you said it exactly correctly, Kevin. This is a model that, or excuse me, this is a model evaluation that Medicare is required to perform as our colleague Franklin noted prior to the presentation. It is comparing. And I would also add that it's looking at Vermont, not compared to past Vermont performance, particularly related to your last comment on primary care utilization, Julie, but rather in comparison to the comparison group selected, based on a huge number of factors by the researchers. And so we're not looking at comparing Vermont to Vermont in years prior. And so there's kind of an interplay of factors there. I know that NRC has said that they may be able to answer some questions that are specifically related to clarifying the slides. So I would offer the mic to them if they would like to comment and feel able to comment given the requirements to have pre-clearance. Thank you, Sarah. We just note that assessing impacts on the Medicaid population is part of our plans. There's usually a longer lag when it comes to assessing impacts on the Medicaid population given the data quality challenges with Medicaid claims data. So we're working towards the goal of assessing impacts on the Medicaid population. And I'll just very quickly note that when we compare the impacts at the state level to the ACO level, we just need to keep in mind that the comparison groups are different. There's a higher pressure for the ACO level impact analysis because we're comparing other ACOs. So it's the additional value of the all payer framework over and above an ACO model that we're assessing. So just something to keep in mind. Thank you. Also, Sarah, if I could jump in here on Julia's point and I appreciate her comments about the annual webloomness visits. I just want to reiterate Aaron's description in case it was missed that the negative 43% we're observing is the consequence of essentially the SSP ACOs catching up to the high levels of annual wellness visits that Vermont had in the baseline period. So to a certain extent that it's not really a degradation or a significant reduction in annual wellness visits for Vermont. It's just that Vermont has been relatively flat while everyone is catching up. And so that's where that negative 43% comes from. That's it. Is there other members of the public who wish to comment at this time? If any member of the public wishes to comment at this time if they could raise their hand. If anybody is just calling in and not on Teams you could just speak up at this point in time. And Julia, I do see your hand raised again. Yes, thank you. I had assumed others would ask questions or comment but I guess my last point would be that there's great irony in the fact that UVM Health Network has just created its own Medicare Advantage plan. Now enrollees in this Medicare Advantage plan are ineligible for the ACO. And UVM Health Network owns the ACO so you would think that they would want it to succeed. Yet the UVM Health Network has just created a new insurance plan that will reduce the number of eligible lives for its ACO. Chairman Mullen, I'm curious, what do you make of that? Well, Julia, I have concerns about Medicare Advantage plans in general but I don't think this is the proper time for me to go on a rant about why I don't think that they're necessarily helpful to reform efforts. Thank you. I see RH with a hand raised, RH. Hi, can you all hear me? We can. And for the record, could you just identify yourself? Sure, Robert Hoffman. I wanted to point out what an abjectly lacking study this was in terms of rigor. The folks involved have committed the most basic errors that we would get in a master's program in public health. Conjecture, using terms like possibly likely maybe. Lack of regression analysis, lack of curiosity that as the state is experiencing people literally dying in pain and committing suicide in pain because they can't access specialist care. And these folks can't be bothered to take a look at whether or not there's a relationship between a 10% decline in specialist care and lack of access to specialist care being reported anecdotally. To think that we're gonna talk about savings against a backdrop of people not being able to access care. And we're all gonna pretend that a critical examination is not gonna evaluate whether or not there's a relationship there. It just defies the ability to suspend disbelief here is beyond. In terms of the study design, it lacks the most basic rigor that any significant study would utilize a difference in difference. I've shared with you all researchers who have demonstrated when we use an RCT to examine the impact of care coordination on purported savings, there's zero relationship. And these folks didn't even bother to try and demonstrate some statistical relationship between decreased emergency room visits or readmissions with the savings. When we talk about people that can't even get into ERs leaving after 12, 16 hours, 24 hours in an ER people sitting in hallways. No wonder ER visits are down. That's a great way to save money. You put a global budget on hospital. The most direct way to save money, the most variable line item in a budget is your FTE. We know that for a fact at this point that the hospital that controls 52% of spending in the state refuses to raise compensation for nurses with even a remote degree of parity with the rest of the country. And we're somehow expected to believe against that backdrop that these savings were generated through care coordination, I mean it's laughable. And for CMS to suggest that somehow this is a cause for celebration in a model that's supposed to be used as a demonstration project for potential national implementation. Frankly, it's just unthinkable that this level of software work research is going to be advanced. And we're all going to somehow accept and believe that this is worth even spending time looking at. We wouldn't even, this wouldn't even make it into a mainstream media outlet, the level of software work research that was performed. That's all I have to say. Thank you. And I will give Nork a chance to respond because there were some pretty sweeping allegations there that I don't think were quite fair. And Sai, if you want to say anything you can or if you don't have to. Chairman Mullins, actually, if you wouldn't mind if I jumped in, I think I just wanted to, I appreciate Robert's concern and I could definitely hear the emotion there. And I definitely think he's coming from a good place. I will say, in terms of that, the evaluation did not use any regression models. If you read the report and as we mentioned, as NRC mentioned in the presentation, they used a difference in difference regression model. In terms of not having an RCT design, I think that that is a question that we can talk with or in future iterations of the model, the way that the current model was designed, you can question whether or not an RCT approach would even work. But I just wanted to clarify those two points that he made. Okay, thank you, Franklin. Next, I see a hand raised by Betty Keller. Thank you. Can you hear me? Yes. Hi, this is Betty Keller. I live in St. John'sbury. Thank you very much for providing this public opportunity to see this federal evaluation. I share the concerns about the challenges with this study, comparing us to 26 other states. Admittedly, not very comparable. You can't find three other states that would be comparable to us for what we were doing, the quality that we were doing and what we're doing with Medicare before this study began. So it is pretty disingenuous. You could choose states and look at numbers and kind of juggle around what you wanna get for a result. It's, as a physician, it's concerning to me that physicians for good reasons are required to use evidence-based medicine, but economic policymakers are not and the evidence for ACOs is so poor. The global hospital budgets question, I differ on that from the last speaker. I do believe that global hospital budgets that are actually funded in advance and that are in a long-term system, such as run by a state government, that would actually have a reliable amount of money coming in on a quarterly basis would provide that stability for hospitals to know what to do as far as increasing staffing levels and that sort of thing. The way it is working in this sort of a situation, I can appreciate that it'll be very challenging for people to know what they can spend and what they can expect to still be making in the future. I think maybe something we learned from this is that in the place like, you know, in, well, we haven't seen it in these two years yet, but we would see it in later years assessment, looking at what happened in the pandemic or having some reliability of some money coming in, but you aren't saving all that administrative money that you would have if a hospital did not have to do any of the fee-for-service billing if they had a global hospital budget and could, like, eliminate entire billing departments and reliably know that they got money back, that they got money and did not have to bill, fight denials and then have long lags before. They're basically young insurance companies serving as financial institutions, trying to keep the money as long as they can because they can make money on the money that they don't spend on their medical losses. So I just wanted to share my, share that I have some of the same concerns with Julie Wasserman, but I also wanted to note that I would like to see us go in the direction of global hospital budgets if we could figure out the financial way to do that before the federal government puts something in place. Thank you. Thank you, Betty. Is there other public comment? Robert, I see you have your hand raised again. Yeah, pursuant to her comment on the global budget issue, I wasn't disparaging global budgets. What I was saying is the evaluation doesn't adjust for the fact that the entire state is being impacted by global budget. So that's a confounder in the causal pathway that they haven't accounted for. They don't even discuss it, that it's possible. The savings were achieved as a consequence of global budgets as opposed to ACO activities. That would be just a basic preliminary type of analysis that anyone would perform on an experiment like this. So that's just a clarification. Thank you, Robert. Is there other public comment? Sorry, one more thing. Since North can't comment, Franklin certainly is not precluded from commenting. Franklin, you're paid to you know, administrate programs like this across the country. Are you not at least curious that we see documented declines in specialist care in the analysis and concurrently both anecdotal reports and hospital report outs in the annual hospital budget process that demonstrate months, if not years of wait time for specialist care and people dying, waiting for care? I mean, is that not at least curious to you enough to wanna perform some more rigorous analysis than what we've done? You know, Robert, we'll definitely take this back and think about, you know, what we can do in future evaluation reports. You know, we do live in a world of unlimited resource and, you know, budget constraints even on here on the federal side. But, you know, I think that, you know, as we pointed out that some of the qualitative evidence has identified this specialist shortages. And, you know, to the extent that the Vermont all payer model is related to that, to the extent that they were occurring before the model, you know, we could take a look at the report to kind of look at those trends. But, you know, I think I'll end with just saying that, you know, we could take your comments back as well as everyone else's comments back and think about, you know, what we can do in the future for the other annual evaluation reports. Thank you, Franklin. Is there other public comment? Hearing none, I really wish to thank the team from NORC for presenting today. And we at least appreciate all the information that you have delivered to us. And it's useful information that we as a board can use as we start to try to consider what might be possible in the future to improve reform efforts in Vermont. And so, thank you very much. And with that, is there any old business to come before the board? Hearing none, is there any new business to come before the board? Hearing none, is there a motion to adjourn? So moved. Second. It's been moved and seconded to adjourn. All those in favor, signify by saying aye. Aye. Those opposed, signify by saying nay. Let the record show that it passed unanimously. And our next meeting will be Monday morning at 8.30, focused on the Springfield Hospital budget. Thank you everyone and have a great rest of the day.