 Good afternoon again. The House Committee on Government Operations and Military Affairs is switching gears here for the second half of our afternoon to take a walk through draft language for H702, which is our Act relating to legislative operations and government accountability, trying to capture the recommendations of the summer government accountability committee that was set up by H125 last year. And a number of folks have been doing work trying to put a draft together and we have a few folks here to testify. And so I'm going to first turn it over to Legislative Council to walk us through this new language and then we'll hear from some of our guests on it. Thank you very much for having me, Chair McCarthy. For the record, my name is Tim Devlin, Legislative Council. Can everybody hear me okay. We can now. We just needed to crank your volume up on our end, but we've got you now Tim. Good. Again, for the record, my name is Tim Devlin, Legislative Council. Before you. The committee members should have draft number 2.1 of the strike all amendment to House Bill 702 and actually do legislative operations and government accountability. The new language has been highlighted for easy reference, very minor changes since the first version that you looked at I believe last Friday. We found a page two under section two. Having to do is creating the joint government oversight accountability committee, specifically under the to be added. So we have say section 971 creation of a committee subsection to do with the composition of the board. And this will now read the committee shall be composed of eight members, four members of the House to represent that more than two shall be from the same party, appointed by the committee on committees. And this was modified in reaction to the committee's discussion of how to best make sure that not everybody's on the same political party that be pointed to this board. That is actually the only update here by be happy to walk through any of the language or answer any questions about the words on the page here. Just a nitpick. I think it's a typo Tim of maybe not line free page one. It just happened. When I was rereading the top and agree relating to act relating. Did you mean to say that? Yeah, that is indeed a typo should just act relating to legislative operations. I'm not an attorney, but I think that's wrong. Okay. Okay, it's a glitch in the matrix. I'm now worried about Tim. Thanks for Morgan for the evil eye. Just happy to see that. Great. Any questions about the new drafts. I think that's what we saw last week. Okay, Tim, stick around with us case questions come up, but I think we'll move on to hearing testimony from some of our guests, and I want to welcome auditor hopper for his perspective. Thanks for the opportunity. I was the state's chief accountability officer. I was happy to support the work of the summer committee and pleased to see you acting on some of the recommendations. I assume you're anxious to get this through the house before crossover so I'll keep my remarks brief. First I support the creation of a new committee to focus exclusively on performance, especially since it can meet after the session when time is not so precious. As I said that I feel strongly about the summer committee's first recommendation, which was quote to educate members of the general assembly on the importance of government accountability. That obviously is a longer term project and I'll get back to it later but I do think it's critical, because while the joint committee will take the point on this if you go forward with this format. I know that all the other committees in the chain in both chambers and the members will take seriously the need for and value of a systemic approach to performance measurement, which also likely would take some time although everybody's been talking about it now for a couple of years. This will take time. And the bill outlines a modest beginning. One clear thing one staff person at JFO dedicated to this is great but they can only do so much. In my view, going forward not something that you were or JFO is prepared for now but I'd love to see the entire staff trained in some level of performance measurement. Not quite gag of standards like my office but so that all of them are capable and have that extra tool available for their regular routine work so that everybody can contribute and not just the one individual. But that's subject for down the road I presume for now. Some thoughts should be given although probably not in bill language, but as a conversation amongst you and your colleagues and Catherine and JFO and others, what standard of review the JFO staffer and others may apply. Obviously, my professional staffers use gag us generally accepted government learning standards which comes from GAO the so called yellow book, and there's no way one person could do that, literally one person. It's very tedious. It's rigorous. And I don't think that's required because we do that. And I'm not sure that's what you're expecting or the committee would be expecting but there should be some conversation. At some point about how that will play out and my office. We have some really cracker jack people are happy to help JFO if they ask. For the committee's first recommendation. I think I can offer something as we move forward. As you know periodically when we release gag us audits we typically inform the committees of jurisdiction. And in many cases they invite us in to talk about the primary or major findings and recommendations and that's as it should be. I'd be happy to add another layer to that process and upon request. Or by invitation I should say we can add a layer and inform the members of both the new committee and the committees of jurisdiction about how we did the work. Which we typically don't talk about. And I don't mean to bore people and it wouldn't have to be that way but my audit managers are really very good. And it might help your colleagues just as a routine thing every now and then we only do five or six gag us audits a year. So it wouldn't be terribly burdensome but I'm happy to offer that going forward. And finally it's kind of interesting. It's been almost 30 years but I did some contract work for then state auditor Ed Flanagan back in the late 90s. And at that time state auditors were limited pretty much to financial and compliance auditing all very important stuff. But the governmental accounting standards board GASB had this crazy idea in the mid 90s and said you know we really should do more than that. It's important to track the money and compliance with statutes and rules but that's the beginning of a much more important conversation about whether the money is being well spent and is effectively achieving the goals that you guys hope for when you pass these bills and create programs and so forth. And that was a new thing. A lot of state auditors didn't jump at the opportunity. But Ed Flanagan understood its value to his credit and he asked me to dive in and I did a little work. And it's been almost 30 years. And that was a really important seminal moment for Vermont Ed's successors all of whom you probably know or may have forgotten. But they all carried the ball forward from that point. And here we are. And I'm so pleased that now you guys after some starts and stops over the last few years are ready to institutionalize this for the legislature which I think is overdue and I'm very pleased that you're doing it. So thank you and thank the summer committee for sure. And that's about it. Our offer any specific recommendations or things that we missed in this draft and you have to answer that now but if you given that the clock is running out on us over here if you do have recommendations about things we might include before we're out I know you were able to weigh in and really appreciate your help with the summer government accountability committee but if there's anything specific you'd like us to clarify or see as improvements to the bill definitely let us know. Thank you. Happy to get I'll get back to you. Thanks so much for being with us today. Thank you. And by the way, with respect to the folks who are going to follow me. I have a couple of time sensitive things to do so I can't stick around. I know that the gentleman from New Mexico is going to be a great guest and I gather the summer committee made good use of his knowledge and expertise. So I look forward to seeing the video later on. Great. Thanks so much. Thank you. Take care. Great. I had asked will Clark to maybe help us contextualize some of this work that we're doing a little bit. So he is joining us and is kind of double booked from NCSL so will I don't know if are you available right now to talk with us. Good afternoon. Yes I am chair McCarthy. For the record will Clark and yes I'm on a state visit right now so yeah that it was the time they worked out really great so I'm happy to be here today. Let me see if I can present my screen. Actually typically so we can see you we've got your deck on our committee page and we all have devices we can use to follow it so if you want to walk us through it. We'll we'll just look at you on our big screen and you don't have to do a screen share. We've got your slide deck. Excellent. Okay, great. Well again for the record will Clark I'm a program principal with the National Conference of State Legislatures. I work for the Center of Legislative Strengthening, which is a section of NCSL that's focused on helping legislative institutions. And so one of my topic areas is legislative oversight so that fits really well with the work that you are doing on this bill and then with the summer government accountability committee. So today I'm just going to go over just a, you know, kind of high level review of some of the most common tools that legislators have at their disposal for conducting oversight. You'll see any slides that first slide just again, legislative oversight tools at a high level are focused on the relationship between the legislature and typically the executive branch though also with some concern to the judicial branch. And at the end of the day, these tools are helping you as members determine whether or not the statutes that you've passed and the various policies that you've implemented are falling legislative intent that they're effective in achieving the statutory goals that they enable good stewardship of taxpayer dollars and then to determine whether or not they're in need of any kind of modification. So again, a lot of a lot of these tools are providing you with metrics in order to help you ensure that the legislative intent is being enacted. So we've got five major tools that I'll be talking about today, very briefly, and they're clearly more that you could use but some of the most common and kind of some of the better differentiations between the different types of tools though sometimes they bleed together. The first, the first major tool is the one you'll probably be the most familiar with the one you spend the most of your time on and that is legislative committees. Some of the most important oversight work happens and these this is where you're able to take testimony, you're able to hear from agencies about particular bills and also to to vote the most directly on pieces of legislation that are either directly related to specific policy focus committee or on a fiscal committee. And then as you saw with the summer summer government accountability committee they can also be focused in special and interim committees to look at particular topics and to focus on particular areas and so there's certain states like Colorado and others that really take that opportunity to have an interim committee to look at a specific issue and so doing or conducting some oversight studies and analysis again very common approach. So the main thing that differentiates these different types of tools are who requests them and how. So, if you have a research division, many legislators do the conduct research over the summer the interim or possibly during session. They request simple request from a policy committee, or from a speaker or Senate president, or it can be something that they undertake themselves. But often, some of the more important researches and analysis and studies come from legislation. So oftentimes, you know, members can either as part of passing some sort of large bill that has a lot of funding they may require some sort of annual analysis and a couple years to look back and then so you can pace you can place the burden on the agency itself to document that for you and then that can be part of a review in the future. And also you can, of course, come up with statutes to look at an existing programs as well to get some of that information to have some examples on the studies and analysis page administrative rules review is probably one area of oversight that varies the most among states, this you have kind of two ends of the spectrum you've got one end where states have the power to review rules and regulations but they, in order to act upon them they need permission and must work with the executive agencies or the governor's office or must pass legislation in order to modify an existing role or regulation. As an example of that they actually do have a committee to look at rules, but if they want to make any type of modification, or to disallow the rule they have to fall through the normal bill process. On the other side of the spectrum you have Illinois which has the joint committee on administrative rules, and in that particular committee, they do have the power to veto certain rules without consideration of the general assembly. So again that's that's one end of the spectrum and in the middle you have Idaho, which uses the germane joint subcommittees. So various rules that come before the legislature reviewed in subcommittee and then through a joint resolution, the legislature can either modify or revoke some of those roles. And but at the end of the day, regardless of the type of path that state legislature takes they typically have language or provisions that say that if you're going to modify a rule or disallow rule, you need to establish a violation of legislative intent. So again that's tying it back to a lot of these tools are to help the legislature establish the intent and ensure that it is it's happening as legislators wish it to, and if not to modify it. We also have sunrise committees and sunset committees sunrise committees are typically addressing an issue where there may be a gap in oversight or regulation so they're looking at harm competency protection so you're looking at an unregulated practice and then determining does it harm the public is that due to incompetent practice and is the public currently unprotected if the answer is yes sunrise committee may establish an entity or a border commission to address that gap. And that is opposed to pardon me sunset committees, where they handle kind of the opposite end of the process, looking at existing boards and commissions and agencies. And on a regular time scale typically 44 states have used this tool at some point in time. The first office was created in Colorado in the 1970s. I think Texas is a very prominent example of how that's still currently being used and and essentially just on that time scale. We also have regular reviews of a border commission agency or another entity who will be terminated unless the legislature takes action to reinstate it. So it's a tool that was used far more back when legislators had perhaps less strength in terms of oversight or fewer resources in order to handle some of these relationships and so that as as time has moved forward and as their staff has more access to more tools. Some of these some states have decided not to use this tool anymore. Well you might remember from talking with the summer government accountability committee that the creation of that summer committee came out of. I think we had a five year sunset committee that really was looking at all of our boards and commissions across statutes. One of the things they recommended that we disband was the government accountability committee because it didn't have the powers and authority to do what we wanted it to do. And that's in large part that the sunset committees report led to the summer committee that led to this bill that's before us. So we're we're we're dealing with a fallout of a sunset committee right now before us. And then again that gets that gets to the end of the end of the day gets back to the just extreme variation across states in terms of how these tools are set up and statute and then how they are implemented. Because again I think maybe other states ran into similar issues with with Vermont where the tool wasn't working as they attended or as they wished it to. And so I think that's how some of those some of those tools diminished, which does lead into the current bill and I think a more modern development. And that is the development of audit offices and performance evaluation. And so these are becoming more prominent. You just heard from the independently elected auditor. And so I think legislators are more and more turning to these tools audits can take a lot of time, but they can be a very thorough and helpful tool for legislators and also just policy implementation in general. The most common audits financial audits I think we're all aware of those he made us someone who's been audited or been part of an organization who has been audited. They're typically looking at expenditures and revenue. They're also compliance audits. They can kind of bleed into internal audits. And then you have performance audits, which as as the auditor formally mentioned, typically follow gaga's they fall specific standards they can be very time consuming. And so they're audits, but oftentimes they're taking a deep dive into a particular policy area or an agency or some sort of practice and really getting down into the weeds, and then providing recommendations at the end. So those tools can be very effective for lawmakers, especially when the auditor reports those reports to a specific committee or to committee of relevance. And so program and performance evaluation, they kind of go hand in hand. Sometimes, sometimes they are this one in the same and sometimes they're different performance evaluation, I think is a major differentiation is that they're not following those standards necessarily. So maybe think of it more as a research and evaluation, but overall they're getting kind of at the same goal which is to review a program review an agency at the behest of either the legislature or some other purpose and then come up with report about what's going on and then developing recommendations that can be tracked. So most of these are attached to the office of the state auditor where they're part of the legislature both. There are there are 34 states that have some form of legislative legislative the appointed auditor audit office and that may be joint appointment process with the governor. And then there are some independently elected auditors of course, and then some states have both. And I'll talk about one of the talk about an example of that here in a minute, but most report to a joint bipartisan legislative committee that's the fairly common practice. And again that's to help make sure that the results of that audit go on the record, and then there are individuals either in a specialized committee or in a policy committee that receives those results and then potentially is in a position to implement those recommendations. And so one example that I'd like to highlight is just the joint legislative audit and review committee in Washington. So they are legislatively appointed auditor office there's also an independently elected auditor in Washington who provides their report results to legislators in the joint legislative audit review committee. The jail art committee itself directs its own spotted staff. They provide information to the public as well as to members. So this is just some screenshots of their website. And you can see here they have a very extensive audit plan that goes out to at least 2030. And so a lot of their audits are directed by legislation to look at specific issue areas. And then they conduct reports as well that provide information for decision makers. The second to last page will find some resources to some pages we have on additional legislative oversight tools. And that concludes my spoken remarks and I'm happy to answer any questions you may have. Well I really appreciate you like framing up sort of what what the different policy levers are reminding us the context of what we're doing here and see what some other states are doing to focus on them and the committee have any questions for will. I think that was a really this is I'm going to hang on to this deck forever. Thank you. That was really. I love what Washington is doing that hybrid model too is interesting and I think you know if we adopt this bill it's somewhat similar to what we're proposing here where we have a state auditor but we have the legislature and there's some overlap at their separate authorities. Well thank you for taking some time away from whichever state you're visiting right now. Absolutely my pleasure and happy to help and then if you have any further questions feel free to reach out. Thanks. Take care. Dr. Davis enjoy to join us at the table thanks for coming in person it's nice to like not have everybody beyond doing today. Great to have you with us. Welcome back to go box. I shared for the records was on a Davis racial equity director for the state. I shared a testimony document with the committee shortly ago so I don't expect that you all have had time to look at it. But I'll be drawing from that today in my remarks is actually joint testimony from me and from the chief performance officer Justin Kenny. So I will be doing some partial reading particularly of the items pertaining to the chief performance officers per view having to do with accountability and government performance management. And then partial on reading style a little bit for the portions that have to do with equity and inclusion. So first we just wanted to acknowledge the positive steps that the legislature is taking on this matter. We are really really glad to see this move toward institutionalizing accountability. It's often seen as the work of an individual or a particular office or entity or just some observance loud member of the public to point out flaws shortcomings waste or other bad things that government is doing. And of course we know that that's not an effective way to monitor governments progress so we're very pleased to see this. We think that it's a really positive step toward upholding public trust and trust is key in governance. And I think that creating a culture of accountability is extremely important because a lot of times we see that word or that concept as if it needs to be something scary combative punitive and externally generated. As opposed to something that we're doing with and for ourselves. We're really pleased to see that we also appreciate all of the efforts that you all have been doing around the rulemaking and just creating that consistency and accountability especially with things like making sure that we have accessible reports and that they are actually being utilized and followed through with. I also think it's really important this this notion that you all have been grappling with about making sure that committee members and staff are regularly reviewing past legislation and past reports and just checking back in on stuff that we've done to make sure that it's working. So that's the stuff that we like. And in kind of talking with the chief performance officer. We had some and I have the benefit of being two doors down from him. Next door so we have these chats regularly. And so there are a couple of things that we wanted to highlight some concerns and some things that we just want to make sure that you all are considering as you move forward and consideration of this bill. So first, I want to talk a little bit about knowledge and about the skills shift that's going to be necessary. So I'll read here comments from the CPO some of the recommendations put forth by the summer government accountability committee on how the summer deck will require a significant shift in knowledge and skills toward program evaluation beyond mere auditing. So if you are valuable for assessing compliance and identifying your regularities program evaluation knows deeper into understanding the effectiveness and impact of programs and government initiatives. It requires expertise and reviewing theories of action, theories of change, assessing program results and recommending improvements based on data and evidence based practices. To be successful, it's essential that adequate resources and training be provided to equip legislators and staff with the necessary skills to undertake comprehensive program evaluations effectively. It's also essential that both of our offices be consulted and involved in the establishment of any program evaluation mechanisms. It's partly because that just makes sense, but also because there are certain existing statutory mandates that kind of require that our office is being involved. I'll give one example. The act nine of 2018 that's currently three VSA chapter 68 50003 states the director of racial equity is required to overshoot statewide collection of race data. The first that statute was amended in 2022 to add the division of racial justice statistics, which is empowered to create a centralized mechanism for collecting analyzing and reporting on those disaggregated data. It's one example of a few that help us be that remind us that there is an important role for the chief performance office and the office of racial equity in creating that sort of enterprise wide program evaluation mechanism. I will know we mentioned earlier data and evidence based practices. Obviously, we feel strongly about those being good things in general and also I have to be really clear in saying that there are sometimes that the evidence may lead us in wrong directions, or that it may create disparities. We have to remember that data collection systems and the sciences are only as objective as the people who devise them. And so it is often the case that a lot of what we know through research or evidence is already tainted by bias. One clear example is most of the medical research that we have on the human body was conducted on neuro descended people. And so there's a lot that we don't know about the impact of certain conditions and diseases on people, not of European descent, which creates difficulties in treatment. An example I use a lot is the fact that we all can recognize certain clear signs of heart attack, right, the left arm thing, the pain but for women that's actually not one of the more common signs of heart attack that many of us don't know that we're conditioned to know how it affect the male body, but not so much the female body. So that's one example of how sometimes evidence based practices can lead us astray. We know, of course, seatbelts were famously designed using male crash test dummies and so they are less safe for women. They, I think, fix that they got around to it. But yes, so knowledge and skill shift. Next, I want to talk a little bit about how any efforts at program evaluation really have to be grounded in a shared understanding of what equity actually means, right when we're talking about the impact of the programs that we're implementing. We want them to work for all residents and visitors of Vermont. And that means we have to understand how our programs and how our work is going to impact different people differently. For us to be effective. We have to be able to design our programs design our initiatives in ways that are going to use metrics that will surface any disparities that are going to show up. So for example, we know that it's one thing to say, oh research tells us that this particular initiative is going to be great for learning environments. I don't know pick whatever it is. We have metrics that we're going to be using maybe classroom size, maybe number of hours of instruction test scores. If we're not using metrics that we can disaggregate by things like race, ethnicity, gender disability, then we could be missing key differences in the ways that our work is impacting people. So whatever program evaluation that we're looking at and whatever mechanisms we're using to track our success as government really have to come with adequate opportunity for us to disaggregate whatever data we're collecting. And then in terms of the shared understanding of equity and justice, it also means that when we're doing that evaluation when we're looking at those disaggregated data. If we don't make sure that the people looking at the data know what they're looking at that disparity when they see it, then effectively we could be just missing glaring things. That's why continuous and repeated, consistent and repeated training are so important. We know for example that a lot of staff here in the legislature tend to be seasonal. And so from one biennium to the next, you may have different people looking at the same issues the same topics or the same programs who may have very, very different of how those programs are going to impact different constituencies. And so having that consistency and the repetition in evaluating things through an equity lens is going to be really critical. Next I want to talk about technical challenges. So I know that the committee has considered implementing a program kind of like Legista out of New Mexico. So we are the opinion that implementing a program like that would definitely require a big technical lift consolidating vast amounts of data from disparate sources and ensuring data accuracy and integrity and developing user friendly interfaces for data collection and analysis are just a few of those challenges that we're going to have to address. So for this to work, it's really going to be imperative that we have an investment in robust data infrastructure that's going to leverage modern technologies so that we can streamline our data collection and analysis and reporting processes. CPO and or we have experience doing this and we would love to be able to help continue that work. I do want to mention when we talk about data accuracy and integrity. Again, that is another point where we can make sure that we're supporting equity and inclusion. We know that demographic data in Vermont are so fragile, partly because we're dealing with such small population numbers for some of our demographic groups. And partly because we have inconsistent ways of collecting those data. For example, let's look at policing. We know that in some cases, demographic data are perceived it's imputed right what is the officer think you are in cases like mine they're almost always going to get it wrong. In the courts, it may be self reported. So now we're taking data maybe from law enforcement and from the courts, putting them together in the same databases and we're going to have inconsistencies in those data, particularly the one you have one person represented in encounters in each of those spaces. Now listed as different racial groups, right. So, it's really a garbage and garbage out situation we want to make sure that to have that data accuracy and integrity that we're utilizing consistent data collection practices. What I want to know is that we talked, I talked about user friendly interfaces, and that's important for those who are going to be using data or evaluating it that may be members of this new committee, other people in the legislature, or people in the executive branch. But most importantly, those user friendly interfaces really should also be aimed at who from the public are going to be utilizing our data sources. So, research partners, academia or just regular old residents and visitors who are looking for state data. So as we think about how can we not just collect it and analyze it but package it and report on it. It's important that in order to ensure equitable access to government that we're providing an ample opportunity for members of the public to have the data and not just technically have it. Oh, you can download a comma separated values file if you want to and it just looks like ones and zeros, but really presented to the public in a way that's digestible so that we don't just say we've technically complied with minimum requirements for access. But you can have it and know what you're looking at a pause there I'm talking a lot. One of the things that you're bringing to mind for me is is when we're thinking about data collection the data you really need to make a decisions about lawmakers. The people who are collecting the data need to have their user interface be easy as well. We talked a lot last year about for instance, the requirement we're going to have for law enforcement, want to make a traffic stop to do data collection so that we can have that disaggregated data about interactions between law enforcement and the public. And then we were hearing from different smaller law enforcement agencies that they just didn't have the technical capability to change the way that they do it and these kinds of things and so, you know, we talked about a director of shares operations and trying to do that. Are those the kinds of things you would envision this new joint committee sort of holding all of our policy committees to new standards when we're looking at bills. When we're talking about where we get the data that we used to make decisions, or do you see it kind of how do you see it living in the system because I'm imagining us trying to create a new culture of how we do data collection how we use data evidence based practices. And in this bill we're trying to arm this joint committee with things like subpoena power, you know, things that we haven't had before, in order to be able to get at the information that we need like where we feel as legislators start for the information that we need to make decisions. And I'm wondering where you see that kind of residing and who's responsible for it. Yeah, and I'll talk more about it in a moment but I really see that as being threaded through everything that we're doing so yes I think that this new committee should be empowered to be able to ask those kinds of questions and do that follow up with the policy committees. But most importantly, I see that as frankly, of a laminated flow chart right here in the middle of every policy committee's table that says, are you making a new committee, yes, go over here. Is it going to be, you know, a timeline of law, or, you know, how many people are going to be on it, what's it going to mean that just just maybe that's reductive maybe not maybe that's a good idea I don't know. But what I'm getting at here is that it should be the responsibility to double check those kinds of questions of this committee. But the primary responsibility should really come at the earliest stages of consideration of a bill which means during the drafting process and during the early deliberations process of the committee it's really got to be threaded throughout the process, not just as a final check. But I can speak a little bit more about that in a moment. That's really helpful. Thank you. On to readiness and leadership commitment. So readiness and leadership commitment are really essential for the success of the recommendations that were provided by the summer gap. We want to adopt the culture of accountability and that's going to require more than just procedural changes, it'll necessitate a fundamental shift in attitudes and behavior towards transparency, openness, inclusion, equity and continuous improvement. Leaders have to lead by example and they have to demonstrate their commitment to accountability for their actions and their decisions. It's not enough that we just put out a press statement that says yeah we love accountability. But don't ask us questions. They should actively champion accountability initiatives and provide necessary resources and support and hold themselves and others accountable for upholding standards and delivering results. We also want to recognize the important interplay between positional leadership and dispositional leadership, and we really want to encourage efforts that are going to empower non managerial and non supervisory staff to also participate in accountability activities and be reflected see themselves reflected in those efforts. We also note that while the attitudes and the commitments of leadership are critical. There's a disproportionate demographic representation among leadership, and with that across all three branches and that is a factor that's really contributing to that picture of readiness. Right, we want to make sure that our policies and our work is informed by communities who are going to be most impacted. So what are doing that means putting people in the room who are from those same communities. And of course our office leads with race it is the most sadly the most stable predictor of life outcomes when you account for all other disparities and demographic factors. So despite the fact that we lead with race, when I talk about demographic representation, I am talking about culture, very broadly, not just race and ethnicity, but gender, sex, gender identity, disability, socio economics that is educational attainment, etc. So really thinking about who are the people in the in positions of leadership, not just how ready they are but also who are they and how ready they are. So let's talk about something that gets at what you were asking Mr chair, which is risk of responsibility application. There's a risk that the creation of an accountability office or the hiring of dedicated staff work program evaluation might end up leading to an application of responsibility among individual legislators or other members of the other branches of government. Well, establishing a dedicated oversight body can definitely help centralize our accountability efforts and ensure consistency in our evaluation practices. We really want to make sure that regarding against the perception that accountability or equitable governance are solely the responsibility of a special office or committee or person. So for myself, I know it's often the case that people invite me to things sometimes so that they can say we've checked the box, we're certified not racist because Susanna says it's okay. And that's really not how it works. Similarly, we just want to make sure that in the same way that one person isn't the ground, who stamps something not racist, or the woman who stamps something as not sexist or the one person living with a disability who says yeah this is fine. We want to make sure that you don't just have one person on or one office on whose shoulders accountability will solely sit because there's a large volume of proposals that come out of all of these committees. And I just can't imagine having to be the one person or office having to evaluate all of them. Instead, we want you all as colleagues to that potential committee to sort of help them help you by doing some of this consideration on the front end. So we want to make sure that they're recognizing their role in holding government and programs accountable and actively engaging in oversight activities and advocating for transparency in your own respective domains. Learning from past efforts. So I was privileged to participate in some of the act 186 population level outcomes review that took place. Was that was that two years ago three. Man, asking us about time right now. I think it was three, I think the act 186 was three years ago. Thank you're right. And that year, the committee this committee also had, let's reapportionment and pensions on for the same session so the recommendations that came out of that act 186 work did not actually move because we had a lot going on in the go box committees. But the work was valuable it's it's still out there right and the population level outcomes still exist. And if we're reflecting on those past efforts, we really want to make sure that we're raising important questions about why previous mechanisms, like the annual outcomes report and the PPMB that's the programmatic and performance measure budgeting report aren't being fully utilized. I know Mr chair you said earlier that legislators are starved for a lot of these data. What I find is that simultaneously we have a lot of robust data existing in pockets that don't get looked at. And so for us, it's, it's something that we're, that we're really committed to is how are we bridging that that gap. So, there have been significant investments in developing accountability frameworks and reporting mechanisms. But there's a lack of institutionalization and I think that in some spaces there's a lack of which is undermining the effectiveness of that reporting framework crucial to conduct a thorough review of past initiatives and identify lessons to learn and address the barriers that hindered their implementation by learning from past experiences and building upon existing frameworks we can ensure that accountability efforts are more robust sustainable and impactful moving forward. That's the polite academic way of saying we need assurances and confidence that the reports were being mandated to write are getting read. I think the Office of racial equity published a language access report in January of last year. And in the year plus since then, we are repeatedly asked the same questions that are answered in that report by policymakers, and we regularly direct people to that report and I don't expect everyone to memorize it it's a hundred pages cover to cover our staff went hard. However, I think that again bridging the gap between who are the key players who need the information. Where is the key information living and how do we make sure that it's accessible and just findable is going to be really important. I would like to apologize and say that we actually had a little taste in our age 629 grab by from ways and means to the five languages be available from the tax burden for translation, which I think that level of requirement in that bill is a direct result of that. That's because representatives cornhizer and her crew are deeply tuned in, and that's not the same in every committee, and I'll take responsibility for in here. We have bills flying through all the time where I probably if I was looking at it through the lens would say, ah, we really should think about language access when we're doing x, y, or z. And there's so many lenses that I don't know that we really do have that institutionalization of some of the things that we've asked you to give us this information, but we're trying ourselves to use it when we're not like looking at the words on the page about the thing is a cultural challenge that I don't exactly how we get over it here but I appreciate you like raising it as something that we need to try to do during this process. And, you know, again, I, maybe it's overly simplistic, but I mean, I love a good flow chart. Honestly, are we putting out public facing materials. Yes, are they being translated, if so into which languages, if not why not right. I also think this is what I'm so glad that you said that because it allows me to mention the impact assessment tool, which I think is also another really great way to make sure that we are asking ourselves the right questions at the time. I think you heard me talk about this before, probably at the beginning of biennium. But since 2020 the executive agencies have used an impact assessment tool that is required for any new budget or policy recommendation coming out of the executive agencies. And it requires us to ask ourselves a series of questions to ensure that we're not creating disparities or inefficiencies in whatever the proposals are. Those impact assessment tools are time intensive, but we believe we strongly believe that they're very much worth it. They will ask questions like, will this policy, if you're proposing a budget cut is it going to be fortunately harm any particular group in the state is this going to have a disparate regional impact. How is this going to incorporate cultural practices including but not limited to spiritual or religious practices or anything related to particular demographic groups, etc. I'll give you an example of one piece of legislation that was passed in recent years that did not go through that process and that created a disparity. The tobacco 21 legislation that criminalized minors from possessing tobacco under 21 did not include a religious or spiritual carve out for indigenous people who see tobacco as a sacred plant it's used in the conveyance of prayers and purification rituals. As a result, we ended up with a disparity that specifically impacted indigenous people in the state where minors could not practice those spiritual practices without coming under risk of criminal penalty. We know that that kind of a carve out is actually pretty common. For example, if you are a person who was raised Catholic like I was you were drinking wine as a minor because the law said you could. So it's not uncommon to have that kind of a carve out but we didn't have an I a tool at the time that made sure that we were asking ourselves about spiritual or cultural practices. So that's one way in which we can make sure that we are asking those questions and they are top of mind. Each time so that we don't have gaps where some bills have it and some bills don't. It is my goal before I leave this earth or this state that the legislature adopt the tool. Not done until you guys take it. Okay. I want to talk next about the that PPMB the programmatic performance measure budgeting report. This I'm going to read directly from this because this came directly from CPO. And I do not have enough expertise to be able to freestyle about it. So there is specific mention of the programmatic performance measure budgeting report and the timing of that report. The timing of the report can be shifted but a few things would need to be understood first. The report itself is released when it is because it's associated with the budget and includes financial information that comes from the governor's recommended budget. That recommended budget is not available until May. So if we were to change the timing of the PPMB report, then it wouldn't include that information at the time of its release. We all know that only half of the measures are reported on a state fiscal year period. The rest are mostly calendar year with some of them being federal fiscal year. So because of that, regardless of when the report is released, there's always going to be some data that's not as current as we might want it to be. Further, there's not an actual requirement for the chief performance office to submit this report. The PPMB report actually came about as a response to 32 BSA section 307, and that law required that as a part of the budget, there would be a strategic plan for every state agency department office or other entity or program that includes a statement of mission and goals that support the relevant population level outcomes and a description of the program performance measures that are used to demonstrate outputs and results. So technically, all of that gets submitted in the budget already. And so what the CPO produces is actually above and beyond that. So it should be noted that the current version of the PPMB report is not far off from the way that the line item performance measures work, excuse me, the line item performance measures reporting work in jurisdictions like Utah. So not as far off as maybe we thought we were, but the way we utilize the data and the way and the timing of it being reported doesn't allow us to really process it in our legislative committees to the fullest extent. Yeah, and the way that I understand it is that regardless of when in the year we put the PPMB report, we're always going to have that lab just because of the different reporting cycles that all of the data are using. So it doesn't seem like any time is really the ideal time. But that is probably better for CPO Kennedy to follow up on with you all than me because that is the limit of my understanding of this. I understand. All right. Thanks for your patience. I am almost done. I just want to talk a little bit about support for program improvement. So finally, if there is an office that's established to scrutinize program performance, we really, really urge that there be commensurate support to help improve those programs. Accountability efforts should not be seen as punitive measures. This was something that I worked really hard to demonstrate to my colleagues when this role was first created. That when I come poking around in your business, it's not a gotcha moment. We're not looking for a flashy press release, right? We're trying to fix things. And so looking into things to make sure that we're on the up and up or that we're doing things inclusively or efficiently is not intended to be a punishment. It's an opportunity for learning and for improvement. Programs and program staff often face a lot of challenges that are preventing them from being able to achieve whatever desired outcomes they have. And sometimes it's limited resources or technical constraints or just competing priorities. So alongside the accountability mechanisms, it's really crucial that we provide support structures and resources to help programs actually overcome those challenges, not just point them out and say you're horrible. This might include providing technical assistance or knowledge sharing or disseminating best practices, fostering collaboration with stakeholders so that we can really get at those systemic issues and what are the barriers that are stopping us from meeting our goals. Again, both the CPO and ORE would really love the opportunity to continue talking more about that because it is certainly our goal, probably our primary goal to support programs and staff across state government. And so to the extent that we can be helpful in figuring out how to best do that we would love to. I will say that when we talk about ways that we can, you know, foster collaboration among stakeholders or what have you, what I don't recommend is just making a new council for every new task that we create in state government. I think that adds to government bloat and sometimes makes things more complicated than they need to be. But in terms of providing support for programs and for initiatives. I think that's going to be a really critical step. It's not just about identifying our flaws and shortcomings, but really saying, okay, what can we actually do about it. The last thing that I want to say is at least for our office the most important, which is the summer GAC was charged with four main things over this past year, and we're given four meetings to do it all. And several of those things related directly to equity and inclusion so naturally our ears perked up. We were extremely disappointed that there wasn't enough time or runway to be able to address them adequately. But we had a lot of thoughts so many thoughts. One thing that I am really urging this committee to do is, as you consider age 702, I'm not seeing really anything on equity. And it really needs to be in there, because accountability is really about more than just the numbers, right it's about the people who those numbers represent. So, particularly one of one of the things that summer GAC was asked to look at was equitable inclusion and participation on the state's boards and commissions and per diem rates for people who serve on the state's boards and commissions, etc. We provided a presentation that is not just in the visit in the written documents that summer GAC has on the legislative website, but it's also online and a recording. We provided an initial letter of testimony and a guidance document about equitable and inclusive committee participation. I would love love to continue this conversation with you all I don't want this to sort of passing thing in the night. I know that auditor Hoffer mentioned that you want to get this through before crossover. And also, the mention that I made earlier of competing priorities. We really need equity not to be competing with anything else. Because we're often told, oh, we know this bill is unjust, but we'll fix it in January will fix it next year and so what that what that means when we say that is, there's something more important than equity that is motivating us to act. I have a hard time with that. So, we have a lot of recommendations for how we can incorporate more equity more justice and more inclusion into our performance management and our program evaluation. And I would be extremely happy to either sit down with the committee again or just to provide whatever materials would be helpful so that we can see how we can incorporate that I will also say there is a very. There's a quiet riot, bubbling up among the many working groups in the state over the per diem rate so I really think this is going to be an issue that this legislature is eventually going to have to tackle, because it's become really clear that the demand on people's their labor their expertise is not commensurate with the with the compensation and it's it's really leading to us having struggles appointing people to boards and commission and retaining them this morning. I received word that two of my appointees on one particular advisory board have now resigned. And I have to find two new community members who are familiar with certain topics in certain areas of expertise, who are not already sitting on six other boards and commissions to appoint to this body. And they're certainly not the only ones so I do strongly recommend that this committee pick up that thread from where summer gap left off and again and offer lots of guidance and suggestions. So that concludes my testimony but I'm happy to take questions or other thoughts. So I wanted to start by saying, I. When I first got back the draft from the summer gap, and it said, we're focusing on one to three and four or for your box committees good luck. So what we had before us in the first few weeks of the session. We made some decisions about what we were going to do and how we. I think if there are ways we can make sure that we make this bill as good as it can be in the next couple of weeks. The bill is the beginning of a new process and some resources. I want to make sure that we don't miss prioritizing equity, but I also think that we need to spend some time after crossover with you and other stakeholders looking at all the boards and commissions that we have. You put together some incredible materials in here that show just how many things you have to sit on or point people to that are just in the kind of like equity space and because you're the office of racial equity every time we put together an advisory council. We appoint somebody through your office in order to be on it. I think we need to create space when we're not under the gun to actually go through that and do that work and I'm committed to doing that. When we get past crossover and bringing some of that to light here and then we'll have opportunities because we have several vehicles that are about boards and commissions and sort of the way the government works in order to add that in. And take action on some of those things this year so they didn't want to hold up things like the truth of reconciliation commission doing it or just getting this joint committee off the ground. So those things have not been forgotten. The other thing I wanted to say to you is that I'm really committed to taking some testimony on the report you sent us on January 15 about your five year work back. There's some things in there around a government accountability that definitely made my eyes pop. No one had noticed. We noticed didn't know what to do about it in a few weeks so but we need I think to elevate some of the things that you brought to light there especially ahead of future flooding events we're going to have opportunities. To get the flood bill that has to do with emergency powers and some other things from the Senate I think to actually address some of the things that you brought up in that report so I didn't notice and other people in this room notice so I just wanted you to hear that. Yeah. Two questions for the director. Thank you and thank you for coming today and I have maybe it's more of a comment but again in the beginning of your presentation you talked about a positive step. These measures signal a commitment to upholding the public trust and ensuring that governmental actions are in line with the best interest of constituents. I've spoken to this committee about that particular thing about in the best of my constituents. And I guess the issue that I had and I've talked about before is what's in the best interest of my constituents may not be the same as representative Hooper's. That's where I have a real issue and I also have an expensive changing here which I appreciate but the makeup of this as well with the floor members presented by the speaker the house for minutes by committee committee. I think it's it's really important that we do get a diverse and I don't know if you can say about a non partisan group to consider this because again. A few examples over the years like even a childcare issue. I mean I think the state of Vermont did the childcare industry in with its regulations around education, water quality, you name it and then we have another full blown expensive childcare thing that might work might not. There's some issues there. Another example might even be the PCB testing. What we started out is, we have to do it if you disagree with that and you don't care about kids. And now all of a sudden it's looking at like we may not continue the program. So again, there's just two examples in my mind of how is the government account a bit. How would they look at that. And what would they, what would they, what would we do different. So, again, I, I understand the efforts here, but I, but I think it really depends on outcomes and and those outcomes are subjective is get I get what I'm saying. Yeah, thank you representative, you know, I think I think that you're right different constituent groups and different regions around the state have different priorities for themselves. That can lead us to feel like folks over here need one thing and folks over here need another thing and so we can't come to agreement or, or, or what have you. I completely agree with you that we want whatever committee comes out of this to be as reflective of everybody as possible. I agree with you that it should be non partisan that it should be really diverse and I mean that not just in terms of the demographic identities of its representatives but also their interest areas expertise region, etc. A really important thing that I want to make sure that we know is that we're planning for and making decisions for not just the Vermont we have today, but the Vermont we expect to have in the future. And so I think sometimes we've got a lot of policy that's built on what our communities have historically wanted or preferred or dying. And what that does is it's pattern matching, right where we're getting the same results, because we're catering to the same perspectives and desires. And as we're seeing a demographic shift that we hope is sustained and grown in our state, it's going to require us to do things sometimes a little bit differently than what people are accustomed to. And, you know, I'm reminded of a phrase that says, at first growing might feel like breaking and I think to a lot of people sometimes it feels like breaking. But in the long run what I have found is that really a collective benefit to equity, you know, curb cut effect, whatever we want to call it. But I think that there is a way that we can communicate to the public that what we're putting into place now it's going to help everybody and it's going to benefit everybody. And I think that that's, that's going to be key. But again, I do just want to affirm what you're saying about the importance of making sure that any new body recreate is is really going to be taking into account of those perspectives. Okay. Any other questions for the director. Representative checks. To the point about us asking you to point somebody to so many of these things you have any information or feedback on the people that have been appointed if they feel valued and heard and that's if the committees that they're appointed to are appreciating and incorporating their expertise. Thank you for asking that we rarely get asked that and it's, it's very validating to be asked that a lot of the so so first I just have to say, it is hard for me to make appointments to things. Because I'm not pulling from the same list of 12 melanated people everybody knows, which makes it harder to identify people who are willing to be civically engaged and participate on these boards and commissions from Vermont. Because it's not the regular few who always get appointed to everything that being said, some of the folks we've appointed have not felt valued have not felt heard and have not felt like it's worth their time. Case in point the two I was just informed have stepped down this morning. That's not true. I think they were they, well, it's complicated. I think some have not really felt valued and and that's what I have found is that that has less to do with the working group itself and more with the work product that they generated. Sometimes it feels like we make recommendations we write reports we do a lot of inquest and research and tabulation and say this is what we think should happen. And then when it becomes policy it's distorted by the time it gets to the other end. And that can be really discouraging for people, particularly for community members who were told, you're going to be hurting you're going to impact change. So I think that it's the answer is probably yes and no, I know a lot of people who feel very rewarded and fulfilled being appointed to these kinds of body like they are visible and have a voice. But I think at the end of the day, it's the more important piece is what we do with it. At the end. And that is really what demonstrates to those community members, whether their efforts are in vain or whether it's just perform it over just being tokenized by being brought into the room. And that's one of the things that we want to make sure that when we appoint people that they know their ups there are downs. You know, we just, we want it to work but but for us we really see the the critical piece being once you generate a work product, we've got to follow through on it on our end. Thank you. We're going to have you back just point you to know that I'm sorry this is the first time you're sitting in here this year and I think it's reflection of we have a lot on our plate. But we're prioritizing this as the start of some work to do some institutional transformation and really value this really thorough input and I appreciate you putting the links to the materials that you sent the summer gap into your testimony today. I highly encourage everybody to look at that. It is quite illuminating what director Davis said to the gap. Thank you for the summer. Thank you for your hard work. Right. Anna, are you aren't great right on cue. Thanks so much for joining us. Thank you for having me for the record. My name is Anna roulette and I'm the policy and program director for building bright futures, which is Vermont's early childhood State Advisory Council. I want to first briefly introduce building bright futures role in Vermont's early childhood system. And then I will speak to some key considerations for you all on each 702. Many of which I think you'll find are pretty aligned with observations other have made thus far today. It is an early childhood public private partnership charged under title 33 chapter 46, and the federal head start act to serve as Vermont's early childhood State Advisory Council. It serves as the mechanism used to advise the governor and the legislature on the well being of young children and their families from the prenatal period to age eight. So I'm going to dive in just want to call out that we do not directly support or oppose any specific proposal or bill. And instead our role is really to convene and elevate the voices of families and communities, early childhood partners, monitor the system by identifying and providing high quality up to the data to inform policy and decision making. With that I'm going to dive into describing a little bit of our role in monitoring and accountability within Vermont's early childhood system through title three title 33 chapter 46. And more recently we are charged under act 76 with monitoring the new investments and policy changes being made to Vermont's childcare and early education system. In addition to this work BVF is also conducting the monitoring and evaluation for the preschool development grant, which is bringing over $7 million a year to the state with activities spanning five agencies and organizations. I will share more in a moment but first I just want to outline our key considerations or highlights from the testimony that I also have submitted for the record for you all to review. Based on our role and experience with monitoring and accountability along with some recent policy recommendations endorsed by Vermont's early childhood State Advisory Council. First we want to note that defining success and selecting measures to track and understand the impact of policy change, as others have pointed out today is a really complex process and requires partner engagement and expertise. It is also key to ensure that those with lived experience relevant to a proposal are consulted in defining that success and that their role is explicitly named in the system or infrastructure that we are creating to monitor and provide accountability to the system. As others have recommended we also encourage you all to think about when policy committees are aiming to use the work or learning of previously legislative mandated reports studies or committees to move forward on a related bill that they slow down and receive a thorough overview of that work and its findings before moving forward. Additionally, the committees of jurisdiction should also discuss the ways in which previously mandated work is or is not informing their work on a new proposal. We will note that the proposed shift in legislative deadlines in each 702 could potentially support committees and utilizing the work of previous legislation. Committee also sorry was our question. No, I think somebody just coughed in the room. Sorry. Every single sound gets picked up. Committees also currently don't have specific guidance or requirements around when to designate an entity with duties related to data collection monitoring and reporting back to the legislature on a given policy change or investments. Exploring a process or series of guidelines I liked the previous recommendation related to a flow chart of some sort. For committees to utilize may support building a culture of accountability and data driven decision making throughout the legislative process rather than isolating this work in a specific committee. And finally, I'll note that overall staffing of the state legislature should also be considered when making recommendations to strengthen accountability. Based on 2021 research from NCSL Vermont has the third smallest legislative stash staff during session insufficient staffing capacity may limit the legislature's ability to use data to inform policy and implement new accountability procedures. With the remainder of my testimony I'll share a bit more about bbs role and monitoring act 76 and elevate related policy recommendations on monitoring and accountability made by the bbs state advisory council that have informed these key considerations for you all. First, as we outlined in our annual report to the legislature on act 76 monitoring, one key learning of our role has been that no singular indicator is sufficient to measure overall impact. Over the course of several months we held focus groups and conducted surveys to ask partners what success for this law looked like, and to name the top three things that we need to measure to understand its impact. Significant variation existed among partners based on their sector lived experience and the elements of the law that they anticipate to be the most successful. Even among legislators whose expertise had shaped the bill every partner thought differently about how to understand the impact of the laws investments and policy changes. Additionally, some recent policy recommendations endorsed by the state advisory council related to government accountability and the importance of partnering with those with lived experience can also lend lessons to this conversation. Over the past several years the SAC has elevated the importance of providing more structure through guidance or a protocol for legislative committees creating new bodies. This guidance should make explicit the ways in which those with relevant lived experience in our communities are including and included in decision making processes. An additional recommendation that our SAC and partners continuously have elevated is the need to make critical investments in data and technological infrastructure. Investments and additional staff capacity are paramount to being able to successfully build a culture of data driven decision making and continuous quality improvement. Next, I want to speak a little bit more to our role in providing evaluation support for the preschool development grant. As I mentioned this grant is bringing over $7 million a year to the state with activities spanning five agencies and organizations. I just want to note that this effort has taken over a year as well as a full time dedicated staff member to define measures of success for this specific grants and to begin to build systems and tools to monitor and evaluate activities and the overall project. As a result to say that building a culture of accountability and data driven work takes significant time and content area expertise and that what the bill that you are considering today is attempting to do it feels quite ambitious considering the amount of resources that is being given. We are excited to see this committee and others considering how best to improve the legislature's government accountability systems and practices. Building a culture of accountability and data driven decision making as the potential to strengthen the work of the legislature and create more systems to ensure that individuals with lived experience are heard. Thank you for your time. I'm happy to take any questions. Any questions for Ms. Brulette. I really appreciate your detailed written testimony. I think we're getting to the end nearing of what's been a really long day and are starting to run out of gas a little bit. So I apologize for that. Representative Higley had brought up the childcare bill as an example of like one of the big things that we've done. And I think of that as an example of one of the kinds of big initiatives that. You know, this new joint committee and the director position at JFO would. Look at and monitor and I'm wondering from DBS perspective, you know, did we build in sufficient. Look backs and accountability provisions into the bill that we passed around. Setting up the new childcare system or. You know, are there things that we should have done differently from an accountability perspective or just things that. You know, BBF has noticed that we should be tracking in terms of data that we're not. Great question. I would say that act 76 the childcare bill actually is kind of a unique circumstance in which the legislature was really intentional about thinking about what a successful monitoring and accountability could look like so. As I mentioned in my testimony, building bright futures was charged under that childcare law to be the organization or entity that is holding the vision and strategy for both monitoring implementation that is happening at CDD and other agencies. As well as tracking data over time to understand the law's impact. And through through act 76 we are required to submit an annual reports to the legislature on our role in doing that in our provide observations on implementation and how it is going but also as I said, what impact has looked like so far and we're providing up to date data to the legislature but also the general public on on the impact of that law so I would say. So, I would say it's actually a kind of a unique and potentially helpful situation or example for this committee to consider about what successful. What successful examples can be looked at for how the legislature might be able to charge more entities with with supporting some of that government accountability work. Great. I was hoping you would say something like that knowing how deeply involved at least representative Bromstead was on both of these efforts. Any other questions for Anna. Thank you so much for being with us this afternoon. Of course, thanks for having me. I mean, Charles, are you still with us? I really appreciate your patience and and hanging out there. I'm excited to hear your testimony about your efforts in New Mexico. Yes, I'm still here, but it won't let me turn on my others. I could start my video now. There we go. Great. So I think you all have a short PowerPoint slide that I put together for you. Really just to provide some context about the legislating for results framework that I'll get into a little bit more detail. The first is my name is Charles Sully. I'm the director of the legislative finance committee, which is the New Mexico's joint interim budget committee. A couple of key points for that committee's work. It basically has jurisdiction overall of state and local government, higher education and and public schools. One of its main functions is to develop a budget recommendation separate from the governor's for the full legislature to consider. Oftentimes you hear people in New Mexico, we've we just got out of a 30 day session and in odd years, we have a 60 day session and oftentimes the public is wondering how you put together a budget or legislation and that short amount of time. And my answer is we don't we start in September, putting together the budget for the on behalf of the legislature. And so that's really important for the committee to be able to start that work. A couple of key things. It does have subpoena power. It's never been used, but it's a key piece that's an important component for the committee, because there is another statute that requires agencies to cooperate with the staff and provide any information. That the staff team necessary to carry out the work of the committee. That's a really important cooperation provision we feel in New Mexico. It does have a limitation that I don't advise other states to emulate and that is that it prohibits. Well, it says that agencies don't have to give us information. That's otherwise confidential by law. That has inhibited our work throughout the years when agencies throw up a roadblock. We've been able to work with this administration to get workarounds. But if you're considering sort of the authority of your staff to be able to get information. Those are important things to consider. The next slide 3, just as an overview, we've got about 41 people on staff. We have a team of economists that work on the revenue estimates. We do a consensus revenue. Estimate with the executive branch. We have a team of fiscal analyst that work on the budget recommendation. We do use a performance based budgeting. Approach in New Mexico. So they are also charged with working with agencies and developing quarterly report cards that I'll get into at the end of the day. Your appropriations committees are building a budget right now. And why it's you hope to accomplish something and the performance measures help tell you whether or not you're heading in the right direction and accomplishing what you wanted to accomplish. So the fiscal analyst work on doing that work. And then we're somewhat unique in that we house our government accountability unit, our program evaluation team to be able to go and do the deep dive. The performance information that you will be getting if this bill passes and you move forward is good. It tells you if you're heading in the direction that you want. It doesn't tell you why or what to do about it. There's a different level of analysis that that's needed in order to dig into that. And just I wanted to note that the legislature this session in the feed bill greatly expanded our staff by another six FTE. So we'll be expanding our work into more of the health and human services area as well as additional program areas. On slide for again just context where our general fund budgets about $10 billion for next year are all funds budget almost 29 billion. The general fund has increased this will be a represented increase a well over 50% since FY 19. So dramatic changes and dramatic new investments that have been made in our state and really we need to up our game to say whether those investments are in fact dealing better results for the people in New Mexico. And you can see the breakdown of our current budget and major cost centers that include public schools and health and human services. Mr. Chairman slide 5 gives you a very simple simple overview of what we call legislating for results framework. And it's a continuous process and improvement type of framework. And we've built it out over the past 20 plus years since the state started implementing the performance based budgeting system that requires agencies to have strategic plans metrics to tell you whether you're heading in the right direction reports back to the legislature on the results of implementation of recommendations of our program evaluations. And it really starts with these performance metrics that we're looking at. And we use these through our report cards as almost a surveillance tool when you think about all of the different agencies that you have which ones matter the most which metric should you be paying attention to the report cards to help us kind of flag where we need to do that deeper dive to understand why and what to do about it. But it tells you how are we doing and the committee gets for the largest agencies, basically a state of the state every quarter on how our tax departments doing how our human services department and Medicaid are serving individuals in the state, higher education and K 12 even though those data points aren't always available quarterly. The next phase is, if we're not doing as well as we want, then what should we be doing different and doing that deep dive to answer the, the why and what to do about it. Are we spending money the way on evidence based interventions that are likely to yield good results. Are we not appropriating enough to solve the problem. Analysts will go in and do that analysis and then sometimes we'll assign a deeper dive for the program evaluators to take three to five months to help answer some of those questions. How do we get more third graders to read at grade level. One investment would be through our research, our pre K program, and we develop performance metrics to monitor how that was working as we are continuing to expand that program. We also use the program evaluation to identify things that weren't leading to better educational outcomes so we tell our members, if you're trying to impact early literacy outcomes. The healthcare assistance program is not the investment that you'd want to make you'd rather make that in into pre K. And then you build that information into the budget development process how many slots for pre K can we afford how many teachers can we hire how do we how do we do that over time. And then the budget gets passed and that's just oftentimes that's where things tend to end. We've passed the bill we passed the budget to solve a problem. But really, we advise our members that that's the just the first step. The next is really monitoring the implementation and the use of those investments that you've made to see whether or not it's paid off. Are the kids that participate in our pre K program getting the kind of outcomes that we want. And our answer has been consistently yes over the last 10 years and that's informed further investments in that to the point that we've got a universal four year old pre K program and we're heading towards about 50% enrollment. In our for three year olds. So that's the continuous process and the framework and how staff help do analysis to advise the committee on the types of investments that it should consider. Slide 6 just gives you an idea of all the different tools that we try to deploy. We've got a pretty sophisticated cost benefit analysis system where we can take the best research about what works. Apply New Mexico specific performance and cost data and project out for the committee what the return on investment would be making for say a substance abuse treatment program in our prisons what kind of impact we would expect it to have on. Recidivism rates and the like. All of these are in addition to your traditional budget development tools. Looking at vacancies of staff incremental cost for the budget. Page 7 just gives you a snapshot of what our quarterly report cards look like again. We only use these for the largest agencies all agencies including us report annually on our performance metrics we have those we've got a strategic plan. But for those biggest ones we want to know what's going on with our child protective services. Is it operating at the level we need or is the media attention that it's getting focused on isolated cases or do we have systemic problems. These this tool really helps start start the line of questioning it doesn't necessarily answer all of them but it really helps. A new tool that we've been implementing over the past 2 years on slide 8. Is what we call legislation. And if you've heard of stat it's really a performance management system often directed. And the private sector but also adapted in the public sector usually by governors mayors. Or agency heads and that's really looking at the data on a regular basis and trying to manage around that. And we've tried to adapt this to a legislative environment where we're taking a subset of all of those performance metrics that we're getting. And saying what are the 3 key performance challenges that we've identified and maybe collectively with the agency that we want to focus on over the next year or 2. And have legislative oversight consistently come back to those performance challenges and understand from the agency what's different. So you'll hear the committee members ask when we leave this hearing what are you going to what are your action steps what are you going to do what's going to be different when you come back. And when we start the next meeting what have you done what did you accomplish so strong accountability. The other key component of this new tool because we've always had hearings on agency report cards and performance is it flips the script where the agency only gets about 5 minutes the fiscal analyst from our staff. We'll go through the actions that the agency has been taking any changes in performance that we've been seeing. And then we open up the hearing for members to have a more of a collaborative deep dive discussion with agencies rather than being talked at for an hour and a half or 2 hours. Because the committee you know as the board of directors and not the CEO. It's a look the dynamics have to be a little bit more collaborative as opposed to what a governor could do in terms of accountability. So that's something that we've been working through with the committee. It does take some extra work in between meetings and prep work on the part of the committee it's hard to sit down. You know and get a 3 page document and be able to really get into the meat of it. But so far it's been working fairly well. We've already exited our tourism department because they were doing so good at improving their performance challenges. The committee wanted to focus on, you know, close to 60% of our budget, which is the hardest part to do for this because our higher education and K 12 system is such a decentralized. Governance system with not very good interim metrics, but how do you ignore 60% of the budget if you're going to embark on this kind of activity. It has been good from the standpoint that I've talked to agency heads who have made it very clear that they do not want to go back in front of the LFC without having taken measurable actions and being able to demonstrate improvement. And that's more the next time there before the committee and those were even collaborative meetings but they really want to work on those those topics that we've collectively identified to to make improvement so very action oriented with that Mr chairman I know you've had a long day and and I'd be happy to stand for any questions. One of the things that came to mind was I'm wondering how your committee, when they looked at some of those decentralized questions that are big numbers you know we're having enormous policy conversations as well here in Vermont about education finance, and our very decision making system around the local control of school boards. I'm wondering how how did they sort of bite into that did they focus on, you know, certain funding streams of how do what how did they even sort of start to look at accountability and what measures that they wanted to put in place in order to understand what the performance was of the system would be. Sure, we've got a pretty sophisticated and and far reaching performance measurement system for K 12. The problem is the most of it is snapshot data at the end of the year. At the end of the year when you get test data, or your high school graduation, and there's not as much interim data so we tried to focus in on for K 12 to leading indicators and the first we started this when we were coming out of COVID in the state was not doing any testing at all, and didn't seem like it was going to be moving in that direction and so our first goal was to get the executive to agree to actually see where our kids were at, coming out of COVID and actually administer the test, which it did, and I don't think it would have otherwise done it without legislation. The second was to shine a light on attendance gaps that we were noticing. You know, we've done a lot of research that it's pretty obvious if you're not at school you're not learning, and you're probably not going to be performing as well. And the rate of kids not attending school has really increased quite a bit and so there's been a lot of focus on doing deeper dives on attendance as an interim metric. We are, we did have a school finance lawsuit adverse ruling against the state in 2019. And we've put in well over, you know, a billion, probably close to a billion five cents into a suite of interventions that were all informed by a lot of this research that we have been been doing over the past decade, everything from pre K to an extended school year to not just increase teacher pay but really trying to support high quality teaching school leadership, and members are increasingly questioning why those investments aren't yielding better results. So they're using these tools to try to dig in to understand that. I will tell you that our summer government accountability committee was really excited by what you're doing in New Mexico. They looked at these, you know, sort of your legislative reports are almost like a heads up display of what's going on, what performance measures that we identified, how are the agencies doing. Getting from where we are now, where, you know, we're, I was describing in that previous testimony that a lot of times we feel a little starved for the information that we need to where we want to be. You know, we are considering empowering this committee in several different ways. I appreciate the specific recommendation on access to confidential records and we'll definitely take that to heart as we look at how we give this committee tools, but I'm wondering did this process kind of phase in and grow and become part of the New Mexico legislature's story over time or was it something where there was like a big push in a whole bunch of investment in a year or two. So this has been growing over time over the past 20 plus years. So it started with in the early 90s are some of the chairman of the committee were frustrated because they were making appropriations for specific things and didn't know whether it was making a difference or not. And we were one of the last states to include and build out a performance audit unit. They originally house that over at the state auditor's office. And then there was a change in state auditor, and he didn't want to have anything to do with it so the legislature took those FTEs and put them in the LFC. And then in the late 90s, there was a big push for performance based budgeting and New Mexico implemented that and that that placed the foundation of the performance measurement system into place and accordingly reporting. And then about 2005 we instituted the report card system and started when I started in 2005, reforming how we did the performance audits to answer more of the outcome and impact evaluation questions that the legislature had at the time. So we revamped that performance audit unit into a program evaluation unit to answer those questions as pre K make a difference or not, versus compliance audit or auditing performance measures, whether they're accurate or not. And then we worked with Pew and their results first initiative to build out our staff capability to do that cutting edge cost benefit analysis, continue to refine how we, the analyst worked with agencies around understanding the performance measures and how to make meaningful use of that. And then most recently worked with a consultant through a grant through Pew to help the committee develop the legislative model. So it's a, it's been a long iterative process. I would, you know, suggest you all consider what are some priority areas that you'd want to start with. Maybe even looking at areas where you're already getting some performance data reported, you know, federally funded programs tend to, to already have a pretty sophisticated apparatus. You know, it might be a child protective services, which has an extensive performance reporting system. And be able to start there and then build that build it out over time. Those are all great suggestions. Do you have a question representing agent? Yeah, thanks. I'm wondering the report card system, which. I love that was, was that hard to implement and like, who helps. Make it happen as far as like agencies or. Right. So, all of the agencies. I mean, many of the report cards. Are actually written into the budget or the performance measures are written into the budget and the legislature sets the performance target. Other measures are agreed upon between the governor's budget shop in our office that we should be measuring them in collaboration with the agencies. And so those are all for the big agencies that data is reported from the agency to the governor's office in our office on a quarterly basis. And our staff put it into the template and do analysis around what's the data telling us. The legend is that kind of format on a smaller subset of key metrics might be a good way to start as well. You could talk to our members and when we do our state of the state quarterly report. Sometimes I feel like it may be too much of a data dump on volunteer board on volunteer legislators, but they've got that information in every budget hearing they've got that information. And anytime an agency's up, they're going to have their report card and some of them are latching on to asking key questions of agencies every time they're before the committee, even in other committees we go and present those report cards. That's a way to link Mr. Chairman, some of the work that the staff that you you're considering funding to your policy and standing committees is to be able to report out if you're, if you've got a performance report, how your corrections departments doing on recidivism. You know, do your courts and criminal justice or whatever the name of it is during the interim. And we've, we found that that force multiplication helps better unify the legislature to understand how the executive branch is really operating. Oh, that sounds like a dream. We're at right now. It's I think we're a little less far along in our journey, but happy to stand on your shoulders. We're happy to provide any resources and assistance if this comes to fruition, or even if not, if you continue to work on this, we're happy to be of assistance to to your legislative staff to help them learn more about how we've developed this. These these tools. Great. Any other questions for Charles from the committee. All right, we have, I believe, maybe a familiar face to you. Mr. Silly, our chair of the house ways and means committee who was one of our members of the summer GAC, the summer government accountability committee is going to wrap up our work on 702 for today. And I really deeply appreciate you being with us represent cornhizer. You have had a long day as well. I haven't got to eat some chips in my way here and that really worked me up. Hi, I'm Emily cornhizer representative from Brattle bro. Like a step like a few years back from the summer government accountability committee work. Is that okay with you? Yes, please. I wanted you to kind of like help us remember the story of how we got here. I'm going to start with the story of my career, if that's okay. So I pretty much since graduated from college have worked for nonprofits, primarily in the international development space. And I was focused on what government accountability looks like when we sort of leave the continental us and our acting overseas. And what accountability means when we primarily contract for services rather than doing those services ourselves. When I stopped doing international development and started working locally in Vermont, I continued that real focus on what is accountability look like in the context of contracted services. Which is as you all probably know better than anyone in this building primarily how government is, you know, deliver spirit Vermont. And so, as part of that work, originally with a and then with a number of other community partners. I was a member of the Vermont accountability group, which was a statewide collaborative of folks who are working in government accountability. That was facilitated and shared by drew wrestling, who was the performance accountability director at the agency of human services for a decade. And had stakeholders from across and beyond state government contractors consultants folks who did the bulk of evaluations for state government folks who were on the ground doing this work. I met trainers who had been trained at the national level and sort of beyond that. I met representative brumstead for the first time before I was elected to the legislature and the context of the Vermont accountability group. In those meetings, we began talking about what it would look like to strengthen the ability of the legislature to carry out this work to carry out meaningful accountability work in collaboration with state government. In collaboration with the administration, especially and including our contracted services. At that time, we were talking just about performance notes and really trying to move that together because representative brumstead was elected before me has really been focused on this since her first, you know, since her first day in the legislature as well. When I was elected in 2019, I think that's right. I actually have no idea at this particular moment, but when I was elected, I was appointed to the previously conceived government accountability appointed by speaker. Previous government accountability committee committee, which you all sunsetted last year. Thank you for that. That group was conceived of originally as a committee whose work it was to make sure that we were holding the administration and the legislature accountable to the performance measures that are spelled out in long ago is Act 186. We did not have the resources to do that even slightly effectively, let alone well. I want to spend a moment describing sort of what we attempted and why it didn't work in order to draw a very clear picture for you of why I think the legislation that you have in front of you as you've developed it is important and why our summer work pointed in that direction. So, the first thing that we were really focused on in the government accountability committee was essentially like how do we look at these performance measures and these population outcomes that are spelled out and statute. And how do we understand what is happening, like in the real world, as regards to these population outcomes like we say we want all Vermonters to be healthier. All children to be safe. But like what does it actually mean in terms of how we do government. I'll tell you the folks on the government accountability committee one did not have the expertise to understand what that sort of level of data science meant. And two, did not have the time or the bandwidth to dive in to how specific performance or programs or spending lined up with those big picture population outcomes. So we knew that the administration was creating those data sets for us giving us to them, but the expertise to dive into that and to really understand what meaningful and useful data is where to disaggregate where to analyze it where to say that the curve should be going in this direction versus this direction. The committee did not have that expertise because we are a citizen legislature, and we did not have the staff available to help us have that expertise. So we were all as we all do every day and I'm sorry to say this on the record making up as we go along. And that really didn't work at this like sort of essential crux of like what do numbers mean for the well being of Vermonters. And so we tried. That work was just like a summer committee right and as you all know the connection between sort of a summer committees work and the standard committees work is tenuous at best we don't have formal connections between those two things. And so what we then try to do is say actually the meaningful work of government accountability needs to happen at the committee level right or human services committee needs to dive in and say, okay we set this child care bill forward and statute. We decided actually making a difference for Vermonters if it's not making a difference for Vermonters is it not making a difference for Vermonters, because we're not administering it well, because we're not spending what we said we were going to spend, or because our whole idea was really dumb to start with, right. And those are sort of the three ways that you fall short when you're achieving outcomes it's like your original theory of change didn't work. You didn't actually or you didn't actually carry out your theory of change which could be sort of like personnel and systems or it could be not fully funding an idea. And so all three of those different things point to very different solutions. But our standard committees did not have the expertise to dive into sort of those three very separate questions, which are sometimes described as how much are we doing how well are we doing is anyone better off. But you can just describe it in like the regular plain language way I did before. And so we tried to do trainings for the whole as general assembly, and we had. I conducted the training with through wrestling who is actually internationally acclaimed and qualified trainer on performance accountability. And we did that for. I think like 80 members came to an off session full day training, which is like, still to this day I look back on that and like, oh, I don't know why there's so many people show up this is incredible right. And that points to me like that this is an issue that's incredibly meaningful to people we had is incredibly nonpartisan who came into the room there was a mix of committee chairs and new members, and people really spent the whole day with us going through this really comprehensive thing. We gave workshops that people could use to create outlines before they even gave a bill to legislative council. We created outlines for specific ways that people can ask questions in standing committees. We really handed over every single tool, including training and how to use those tools to the general assembly. And then afterwards a number of committee chairs asked us to come into their committees and do follow up this was three years ago. This was pre pandemic. So it was more than three years ago. It was however many years ago that was. And I'll tell you, that was really fun, and it felt really effective and it made no difference at all. None. Because we're all lost in the day to day Muck and Meyer of the workers right in front of us in our committees. And so after that process. The process of serving on the government accountability committee. I just been I basically having worked together on this for all these years came to the summer reimagining of the government accountability committee and said to ourselves and to all of our witnesses. Good intentions are not going to cut it around here. We need an actual systemic supported reform if we are going to carry out our essential obligation to Vermonters which is to responsibly spend public dollars towards public good. And that's essentially why you have this bill in front of us like we need to firmly and comprehensively staff this responsibility so that we all have meaningful guidance when we make decisions every day about whether or not we're making a difference. And I think until we do that until it's someone's direct responsibility to help us with this we are not going to get anywhere else. Thank you for your. Representative Nugent has a question for you. So the in the summer group. I'm just curious like if the overall goal, because like statewide you can see population level outcomes, you know, like education reading levels stuff like that you can see how well it's working as like part of the hope with what we're doing is to be able to assess like the individual parts that make that up. So we can know like what parts need more. Whatever it is to improve the overall. Yeah, so let's say we passed Act 173 years ago that reformed special education funding. That's one that I was just talking about committee. And we know that educational outcomes have gone down since then we know that special ed funding has costs for special ed have gone up. So it's really as far as we know, and we don't necessarily have the expertise to help us figure out what the right questions are to ask to dive deeper into understanding that data well, in order to say how can we make changes. And so without that, I think we make changes based on our guts which are great that we're all here because I think we have really good values and really good guts. And we need to also be driven by data. And I think without guidance on what quality data is and what questions to ask of data we're a little bit. Other questions for represented or has it. Your last statement reminded me of what the gentleman from explore talked about, you know, there was a big increase in spending for education and they're finding that there's there's standards are coming up to stop too. The only other thing and I had some questions and concerns about the whole process in a sense and then you had talked about the childcare bill in particular as far as you know, is it doing what it was supposed to be doing. Do we need to put in more money, or is it just a bad idea. And, you know, I think I would have liked to have heard you said to is where we're getting the money from hurting the industry as far as payroll tax. And again, because then we're going to, we're going to go down the road and another year or so for, you know, the pay family and sick leave and that was looked at as a payroll tax. So again, I guess, you know, I have to go back to the accumulative effect again of what we're doing to Vermont's and again you expressed it to is what we do for Vermont is is it is a good for the public good. And that's a huge question. It's just such a huge endeavor and yeah, I'm not convinced that, you know, New Mexico has done it completely and I'm not convinced that we're going to we're going to do it. I don't think we have anyone staff to actually help us answer that question. So, we have folks who are, you know, can understand the immediate fiscal impacts of our decisions or project out the fiscal impacts of our decisions but they don't do not dynamic modeling. And they don't help us figure out how we might want to understand the future. And then we have to figure out, you know, when we pass the childcare bill, what are the questions we need to ask on that bills, then we look back, we know that we did make a difference, we know what data we want to track to make sure we made a difference so we can make course corrections. You know, we pass a number of tax credits really great tax credits that we sort of know from national data, make a really big difference in Vermonters lives, and that costs money from the public purse right. And we want to make sure that as we go along, we are really making sure we're understanding and exploring exactly as you said, does that make a difference in Vermonters lives. And did we put into that legislation what we need to understand in the future, because you know, we need to understand the data from the beginning of a process right to do a good job. And, you know, I started my testimony explaining to you how much I think about this stuff in the course of my day to day life and I don't put that stuff in my bills and like, this is like my professional career is focused on these questions. And when I'm like, you know, putting together legislation I don't even remember to ask these questions and so that's why like, you know, we put before historic like you know those used to be those little yellow cards that were on a bunch of committee tables. That didn't do the trick. And so I don't know if New Mexico has it figured out perfectly I know their legislature meets even less often than we do. So they need even more staff support. But when I met them at a conference and heard a presentation from them. I was really like they are certainly a lot closer than we are and figuring this out and I think there's a lot to learn from them. I think in if you look at the deck that Mr. silly give us they've got three buckets of staff that are related to this. It's pretty comprehensive and I mean they're not. What's the scale of New Mexico is essentially by population budget like three times is Vermont size it's not totally out of the realm. You know, they're still relatively small state. And I think what they do they, you know, they're doing comprehensively in a way that would be a lot to jump into for us, but I'll, you know, in our work over the summer it became clear that almost every other state has someone within their legislature does something like that's right. The federal government has the GAO the government accountability office. But there are much smaller scale versions of this in almost every state and it goes beyond, you know, you don't need a full scale economist you need someone who is practiced and performance and population accountability. Well, I think we're going to take a look at the draft that we have before us and try to encapsulate some of the things that we heard today. There's a couple of recommendations that Mr. silly made that I think are really important for us to think about in terms of the information that the joint committee has access to. That we might want to look at and I think in particular, I'm. Director Davis was in here from already and. You know, I know that we tasked the summer summer government accountability committee with too much stuff. And now it's back on us and I made a commitment to her that we would definitely be taking up some of the data that she's gathered on various topics but I think that we might need some more explicitly build in and the next draft of seven or two, what we. How we're going to build into our work here and the analysis, you know, the lens of equity and whose voices come into that data are we asking the right questions and is it getting interpreted in a way that looks at. Equity as a priority for us as a state. So. Those are a couple of the areas, but I think we're awfully close and I deeply appreciate what the summer gap did in a really short amount of time to get us on a path to. Adopting some real performance accountability staffing and resources here. Anybody else have anything for on this. Alright, so we have 1 task before us. That is not related to each 702. So I want to thank our witnesses who stuck around on zoom and in the room on this bill.