 Welcome to a joint hearing of House and Senate Education Committee and I am Senator Phil Baruth for the Senate Education Committee and I'm joined by Kate Webb, House Education Committee. And thank you to our guests from Education Commission of the States who have traveled to talk to us about proficiency-based education. Would you like to join us gentlemen? I'm not sure if you'd like to come up together or separately. Joel's going to go ahead and defer to me, I'll do it. Okay, yeah. Great. So if you could just introduce yourself and give us your affiliation. Great. Matt Jordan, Director of Strategic Initiatives and Education Commission of the States. Go ahead and start. Please. Great. Thank you. The pleasure to be with you today. We have hard copy notes of slides that I'll go through and we can provide any additional follow-up information if you find any additional. Let me tell you a little bit about Education Commission of the States before we get started or as we get started here. So again I mentioned Matt Jordan. I've been with the organization about four years as a policy director. We are based in Denver, Colorado. We were created four states by states. So what that means is we've created to be an unbiased source of information for state leaders like yourself and the executive branch. So it's just really the whole spectrum of players. So it's really a few things I hope you know about us. We are unbiased and non-partisan, which is to say I'm not here to give you a policy outcome. I'm here to let you learn from what other states are doing and what is down the road. I was researching practices shown. Also I want you to know that we cover the full spectrum of education issues from early learning all the way through post-secondary workforce. So that gives us a nice view across the full spectrum of education. In this slide I added it so this is a little bit diversion from your notes there. The other thing I want to make sure you know about us though is that we cover the full spectrum and serve all key players at the state level. So we work across all 50 states. And our commissioner structure, our government structure is set up where our chair is a governor and rotates every two years between Republican and Democrat. And then our vice chair is always a legislator of the opposite party. And then our secretary is usually a higher ed official and our treasurer is usually a chief state school officer. So our mix of representatives, our constituents if you will, really is the full mix that you all work with every day across the education spectrum at the state level, which is governors, legislators, chief state school officers, higher ed officials and other important policy leaders. So again, we were created to be your resource and so we really appreciate the time to be here. So now jump into the presentation. It's my understanding that I'm going to run through this fairly quickly and cover quite a bit of information. You all have given us a number of questions to address. And so I have a team of colleagues in Denver and a few here with me in the back here. So we're going to go through your responses and then leave time for Q&A. So these five questions. What is proficiency based learning? How does it work in other states? How is it incorporated in transcripts? How is it called admissions? And what are the effects of the implementation? So we'll go through those questions today. Let me start with a little spoiler though. Again, we're not here to give you advice on what you should do. We're just here to help you think through what are the practices that other states are doing and what we've seen through research. So our underlying belief, though, is that effective policies come from thoughtful design and careful implementation. So thoughtful design, careful implementation, which means we really applaud you, Joint House and Senate, getting together to study issues like this. So we call to give testimony like this around the country and we really want to applaud you to take the time to get together in your busy schedules to take a look at this important topic. So what is proficiency based learning? It's also been called competency based learning or education, mastery learning or competency based education. In proficiency based learning systems, students demonstrate desired learning outcomes through the learning process. So that means it allows them to move at their own pace. And so learning is the constant. Time is the variable. Now, you all know in traditional systems, there's a set of calendar and students go through the curriculum based on the calendar. Whereas in proficiency based learning, time can change based on the progress of each individual student or subgroups of students. proficiency based learning or PBL can produce different types of measurements. So you may be aware, one of the key issues is, do you use the traditional A to F grades or do you go through a different numbering system? Perhaps it's like one through four or one through five. So that's one of the features that you see sometimes in PBL systems is using a different grading system. So those are the basics. Now let's talk about what does it look like? How does PBL work in other states? So 2012. This was a snapshot. You can see purple are the advanced states, green developing, orange emerging. Only a handful back in 2012, only a handful of advanced states at that time. Now this is an analysis by the Aurora Institute from a snapshot that they gave last year. So let's fast forward to 2019. So you see proficiency based learning is a growing trade, growing interest across the states. And the number of advanced states, the purple states had grown. Now back in 2012, Vermont was shown as an emerging state, now it was today the advanced category. Now there are, as of last year, 17 advanced states. And these are states that have a comprehensive policy alignment that, you know, it does an established state active state role. In the developing states, in District of Columbia, which are in green, the state has more of an open policy and flexibility. And in the emerging states, there's limited, unlimited. So this is a gradation of sort of the range of state policies that I was trying to show you. So definitely growing trade across the country. Except Wyoming. Except Wyoming. Except Wyoming. Nothing going on here. As of last year. So we're going to give you three case studies, three examples to look at now. First with Illinois. So Illinois is an example of pilots, local districts piloting proficiency based learning. So they passed in 2016, the Illinois legislature, competency based high school graduation requirements. And at that time, 2016, 10 pilot sites, pilot districts started. And it's growing to 45 now. So from 10 to 45. And what the districts were doing in Illinois, or are doing in Illinois, their graduation requirements are replacing certain parts of their current system. So in one or two ways. Either they're replacing some core curriculum elements like math, language arts, or both. Or which years they intend. So meaning two years of math or years of ELA. So the point here is that they're implementing changes in graduation requirements across different districts. So districts get to pick which competency based elements are going to implement in place of one traditional curriculum. Now there are some required components though. And they have a mapping between competencies and academic standards. A plan for determining mastery. And data collection. So you're going to hear that theme throughout. There's going to be different choices that you have for districts. Data collection, professional development, which are things that they built in here are important themes. Districts trying out graduation requirements. In our home state of Colorado, a little bit different approach. Here the State Department of Education is allowing districts to explore different strategies. So less requirements in terms of what they can implement and more support around different ideas. So here it's a more open-ended approach where the department is working with a number of districts of all sizes to help them to a study group facilitated by an outside agency or outside organization called Achieve, which many of you may have heard of. They're exploring different strategies but that will last prescription. I can tell you that a personal note is a parent of a seventh grader in Denver Public Schools where our school is implementing many of these standards. I have a chance to see this as a parent and as a member of the site council. So it's kind of an interesting comparison here from a parent and professional level. In New Hampshire, which you all may be aware of this one, this is an example of a pilot around assessments. So the pace, the performance assessment for competency education is the first of its kind pilot allowed by the U.S. Department of Education that is designed to measure proficiency more reliably and collect local assessments to state assessments, so more connecting local and state around proficiency-based standards. And the idea here is that local districts will take the time to develop standards based on the different proficiency items that they create. So it will give more real-time formative types of feedback but still keep it connected to the state level. This would be for grades 3 through 8. Now students in the high school level would still take the SAT, so the pilot is really of the elementary and middle school grades. Now this kind of assessment brings up some interesting policy questions for you to think about that we'll explore. The questions around comparability, validity, and transparency. So the pilot is, again, still early but a great example to look at what variability will allow across those. We'll now move on to the third question around how is PBL incorporated into transcription? So the states that have advanced proficiency-based learning tend to leave design of transcripts up to local decision makers. But there are certain components or standards that experts would say local decision makers should take into account. The first is clarity is important. So clear expectations of what the transcript means. In addition, cross-cutting or non-academic competencies such as habits of work, of communication, or problem solving, of skills can be included. But you need to have good descriptions of what those are so they're understood. And you can also have descriptions of capstone projects, internships, and other learning. But again, in a way that makes sense for folks who would be reviewing and whether they're parents or post-secondary. So item number one, clarity is critical. Another component is transcripts, another point under clarity is critical. Transcripts can balance traditional and non-traditional measures. So there are a couple of examples. The Great Schools Partnership, which is basically nearby and we'll work here. They have transcript models, templates you can look at that help cross-walk the GPA to the less traditional measures to these competency-based measures. So the Great Schools Partnership has a number of transcripts you can look at. Another group is called Knowledge Works based out of Cincinnati. They do work across the country. And here they show students mastery through various subjects and then they break down the mastery into subcomponents. So instead of just a letter grade, you're looking at where students are across different dimensions and subdimensions. So Great Schools Partnership and Knowledge Works are resources you can look at to really see what transcripts and how transcripts can handle these traditional and non-traditional measures. So another point is less is more, right? Simple is better. Too much information can be overwhelmed. And so making sure that teachers, parents and post-secondary admissions folks know what's in a transcript is going to be an important component. So they'll want to make sure that they're clearly delineated. Another component, and this is one that you all have given attention to, is this flexibility around grades that you, while the PVL system in its purest form gets away from A to F grades, you have allowed in your practice here, which is, we've seen in other states, more flexibility. So districts can choose. Supervisory unions can choose whether they're going to implement the full grading system or adapt across the traditional system. And in this case, again, the idea of assessments comes in. So we use the, excuse me, the New Hampshire example for you all to look at. As we, as you think about grading systems, you want to take a look at what the local assessments are and how they connect to the state. So the standards are the core and how students are achieving mastery to those standards is the core of it, not necessarily what their grade is. So the next question had to do with college admissions. There have been a number of surveys of the college admissions officers around the country, and they tend to consistently indicate that PVL systems do not put students at a disadvantage. So that's good news. In fact, many say that they embrace the goals of a PVL to present a fuller picture of knowledge. Again, this idea of looking at state students' mastery across different competency areas. However, it's important to remember that admissions officers have thousands, many times thousands of transcripts to look at, like we talked about earlier. So this idea of transparency is important and having clear definitions. So we're going to explore that and unpack that here a little bit more. This is a survey of admissions officials talking about GPA. One of the issues to really think about where a problem can emerge, which is something you want to avoid, is when you have inconsistent standards, or that you have a lack of information so that transcripts don't convert well across different systems. So that is looking at an example here from the New England Board of Higher Education in the New England Secondary School Consortium, where let me just read a few of these quotes here. Students with proficiency based transcripts will not be disadvantaged in the highly selected admissions process. Features of the proficiency based transcript model shared with the group could provide important information for institutions, seeking not just high performing impact academics, but engaged life-long learners. So many colleges already receive transcripts like this, these non-traditional transcripts. So this challenge is an entire new. Selected admissions leaders at this meeting stressed how their institutions receive applications from across the country and around the world, which represent a diverse range of high school environments, and a variety of grading scales, terminology, and transcript designs. They're used to seeing variety in transcripts, but again, clarity is key. And it's critical that the admissions leaders interpret the proficiency based transcripts so that having the ability to engage in that is something you'll explore here next. So implementation effects. So we've talked about how transcripts are critical. We've talked about how you can engage with higher ed leaders to work on those transcripts. So let's talk about what implementation looks like a little bit more fully over the next several slides. So effective assessments of proficiency. Experts will tell you assessments need to be valid, generalizable, and comparable. And this is one of the most important parts for PVL that you have solid assessments. In other words, assessments need to measure the important proficiencies aligned with the standards and then measure those standards, you know, link those standards to local, to what is being taught in the classroom. The implication is if some students are held to a higher standard than others because tests are inconsistent or unreliable, then transcripts can't be trusted. So that's why you really want to make sure that the core focus on assessments is critical. So what can you do about it? How do you go about these timely assessments? It's where you have the regular connection between, and I believe we had a chance to visit the folks from the agencies of education. The ongoing communication, the ongoing implementation strategy between the state and locals so they can make sure that their assessments are staying aligned and are meeting the standards. And so state summative assessments can be measured on a reliable proficiency-based standard. Second, timely completion. So students are working at their own pace. It's important to make sure that they don't get too far apart or that you keep them moving at their own pace. So if some students take more time than others to become proficient, they can fall behind. So that's the time in this aspect to think about it. The states need to monitor the impact of PBL and think about accountability in terms of the amount of time state students come through. And it's also important in these systems that the motivation factor for students is kept there. So they see the reason why they should keep moving beyond the letter grade, that their mastery of the subject is what the motivation is for them. Then the next time, so students now are talking about teachers in terms of professional development. So what teachers need in environments like this are strong materials, construction materials, and understanding of the evaluation. So working with teachers, the lesson plans align to the standards and they understand what is in the assessment. So ongoing collaboration with educators is a hallmark of what we've seen in the thoughtful design and careful implementation. And then as I mentioned earlier, engaging with post-secondary partners. If they understand what is in the transcripts, if they have the ability to shape those, and this oftentimes happens at a local level. Again, I mentioned the Illinois example earlier. What they're oftentimes doing is local districts working with community colleges or with universities in the area. There's how they can work and collaborate to make sure the transcripts are understood by post-secondary leaders, post-secondary admissions officials. Another element is CTE, current technical education. There's a lot of common ground between proficiency-based learning and CTE. And so they can match up really well when the standards align. Again, your Act 77 created personalized, flexible pathways. And so that's very much in the mode of what career technical education would allow students to be moved into those career tech tracks. So there's an opportunity to match your PBL system with your CTE system. That's very much of a nice acronym of late sins there. But it's an opportunity that these types of systems can't work well together. And this point about engaging stakeholders, so parents and students, to the most important stakeholders. It's important, as you may have already seen, that parents understand why there's a move to these systems. Educators, while not universal, there is a growing consensus in the education community that these systems are superior. There are advantages to that. But if parents don't understand that, it can create problems as you all know well. So continuing to engage stakeholders, including parents, it will be an important part of this. Because the reality is, students get used to the traditional system, right? They know AQF, they understand how to operate in that traditional system. So helping, hopefully through teachers guiding them, and through other policy communication, helping them understand the advantage to them and their parents will be important. Because you all may have studied what happened in May. So May, back in that 2012 slide, was an advanced state, right? And they established certain standards that were to be implemented by a certain date. And like many states that were moving on that way. As you all know, in 2018 that was rolled back. And analysis, there's been several analysis of why that happened. I want to summarize a few of them as sort of summary here. Now again, this is an analysis by Chalkby Competency Works. So it's not our analysis, but it's one that we think was helpful for you to think about. In summary, as we get into a conversation. So there was a report that there was too much confusion. So this point about not enough clarity is an important thing to think through. Second, stakeholders, students and parents were very much invested in things like class rank. Or valedictorian status, right? So they didn't understand why this other system would be worth giving those up. Third, teachers felt overwhelmed. And the analysis says that the pace went a little too fast. They weren't brought along in the way of the professional development. So again, remember the slide about creating resources for educators. So having an opportunity to equip teachers. Students themselves, this idea that they weren't fighting to the system. That they again thought it was, for example, homework. If it wasn't graded, why is homework important? So there are changes that indie systems that encourage learners to grow at their own pace. So that's a good thing. But if the students don't understand why that's happening, it can be a problem. And the last thing is, this is again, the analysis, not our words, that the state moved too fast, requiring schools to do too much too quickly. So my spoiler alert earlier about effective policies. Again, thoughtful design and careful implementation. That's why you're taking the time to review these policies as a good thing. So you can give good feedback and stay abreast of what's happening. So clearly, proficiency-based learning isn't growing policy area across the country. Almost every state, except Wyoming, is dealing with it in some way. The question is at what point? And again, it's been my understanding, my reading, that the agency of education is taking steps to local districts to let them make changes as they need to, to meet their local conditions, the local needs. The question is then, how do you help continue momentum over some of these barriers to move forward for what many believe is a worthwhile change but go overcome, thoughtfully, some of the resistance and address some of the concerns? So covered a lot of ground quickly. Maybe we can clear up any questions and add any value to your questions. A quick question about the nationwide outlook. And it sounded from your description as though most places are allowing pilot communities or individual districts to take the lead rather than having a single top-down solution. I'm wondering in terms of transcript notation, does any state mandate, let's say, a hybrid system? So in other words, we're going to put in proficiency-based learning, but we're going to have a provision in law that says you have to have, as you go forward, a traditional grade grid as well as proficiency-based charts, graphs, whatever you need. Well, the Illinois example and the pilots that they have, so they, again, this is in their graduation, high school graduation requirements. They do, so they give districts flexibility, but it's a choice of menu options. And we can send you additional information and this is followed, but the menu options include things like mapping between competencies and standards, a plan to determine mastery which can include out-of-school experiences. Again, this idea of how they're going to collect data and what is their approach to grading. So the Illinois example does have certain requirements in the snatching, but it gives districts flexibility in how they go about it, what they choose, how they have to choose to apply. I would be interested if you could send that on. We'll do. Can you let me know how many states in the current academic year, even in the next academic year, require proficiency-based model for all of their high school students? Because it seemed like for your discussion here, where some districts played here, we required it for all students in the state. I just want to see how many other states are doing something similar to us. Yeah. Again, the slide that we had related to where things stand now, we'll go back and see if we can find out how many of the states, I think what, well, let me say, I think what the advanced, what another way of looking at the advanced states is basically the same, they're the same state that had worries for model. But you just said main role back and you said main listed as an advanced state. Yeah. I think what might understand what they did in main is they rolled back the timeline. So, but not this current school year, they're not requiring that all students in the state be under this model. Yeah. What I'm saying is that what you're seeing here within these purple states, which main still stays there, they're still looking at a comprehensive statewide approach. That's why it's purple. The way they go about in there and it gives more flexibility in terms of what districts do at what pace. So, but the question I have is how many states this year currently, I mean, I get that they're moving that direction. Yeah. We've done it. Yeah. How many other states are where we are that it's done? Yeah. That's what I'm trying to say. According to this analysis, there are 16 other states, the purple states are in the same category, but it's still new enough that it's, you have to go back and look at each state in terms of where they are exactly. It's not like every, it's not a uniform measurement. Right. There's different strategies and different paths of different paces. So, when the best thing to say is you're in a category, the advanced category of certain kind of states, the story of what each state is doing varies. But it's safer to say though, because of those 16 states, not all of them have all of their students under this model. That's correct. That's correct. So, it's less than 16 states. Yeah. I don't know if there's any state that has every student underneath their model, underneath that. But again, the flexibility that you all have too though is, I don't know if you're that much farther along or further behind is what I'm trying to say. I think it's my understanding that there are other states that are further ahead in other ways, in some ways, but not every state is doing the same thing. Does that make sense? Yep. Good. Could you, we hear different things about the impact on college admissions. Do you have a sense of are colleges united and yes, we can deal with these or is it a mixed bag? You know, some colleges have got, or small colleges that have a large admissions department, other colleges that have a large admissions department. Yeah. And again, I mentioned this survey of university admissions officers. And here in New England, your own board of higher education, the consortium of higher education has been looking at that. And again, what I read to you was that basically the consensus is most higher ed embrace the idea. So from a conceptual standpoint, there isn't a problem with proficiency based learning standards or systems. Like most things in life, the question is how it's implemented. And so if you have clear transcripts, and if admissions officials understand the transcripts, that's what you need to look at. But that's why I also mentioned that admissions officials have, they have a lot of variation in transcripts already. So it's not like this is differentiating from only one kind of transcript. I think the point here, whether it's a PBL system or not, you want your transcripts to be as clear as possible. Does that make sense? Yeah. So I think what the research we should have found is this isn't something that in and of itself is a problem for post-secondary, but there are certain elements in mind in engaging and continuing to engage that important stakeholder group as well. Yeah. So I have two questions. First, on this map, similar to leading on from what Senator Parran asked, is there any data on how effective it has been across these states? When we have, what, 16 states or purple states, at least seven years where between 2012 and 2019, has there been any national research on is this improving learning for students? Is this better than what was there before in terms of? Yeah. The research isn't conclusive because it's still too new. And that's why Senator Parran would ask part of who I was trying to explain. This is, so 2012 to 2019 what you can see is from a policy movement standpoint, there's clearly more interest in these systems. Now, your question is asking what impacts they have on learning. Right. And it's still too early to answer that question from what we've been able to find. Is anybody collecting data right now? Now is the time to collect it. I mean, or 2012 was really to see what are we starting with and where are we going. Is anybody working on that to see if this is actually having an impact on super learning? The answer is yes. Now, there are a number of organizations that study this closely. But again, I think the question is at what scale. So again, the best answer I can give you is it's not conclusive yet. And yes, they are. Because I think it's a little telling that some of those emerging states, Massachusetts strikes me actually. That's a state that is always at the top of the national standards for student and high test scores, really, really excellent record for educating students. And they're not doing it or they're just now thinking about it. And that raises questions for me as to why they're linked to this game. Maybe they don't think it's worth it because they are in such a good system. But that just struck me. And then my other question is there was a slide that you had on equity. And I didn't quite understand what your point was there in terms of I wrote down connected and clear or it can worsen equity. So I wonder if you could say something more. What I was trying to say is, let me answer both of these together. So these systems, one of the reasons why they are becoming widely studied and implemented is the idea that the standardized approach, well, there's a couple of problems with it. One, it doesn't meet enough students where they are individual. So this is an attempt to create more personalized systems based on confidence. Embedded in that is the idea that it hopefully will help equity, help those concerns around traditionally underserved students. Let me use the example, my own example of my own students. My child's school is an example. So that school happens to be an inverted bell curve where it has high learners and well it has students who are ahead of grade and below grade and hardly any in the middle. The reason why they went to personalized approach, comes to based approaches so you can help move those groups along based on where each student is in their own learning continuum. So that's the theory behind it, the promise of it and that's why I think you're seeing a number of states trying to figure out how to make this type of system work is it creates opportunities to help a broader range of students. Does that make sense? Yeah, but if it's done badly, it could make inequity. That's right, that's exactly the other side of it. Again, not here to be an advocate for the system. But if the implementation happens in a way where, again, the theory is that the students move along at their pace and that all students are moving along. High performers, right? But just the high performers are the ones moving on or the band gets wider, that makes the problem worse. Okay, I have usually a third question if I may. And that is about cost and it seems to me that this has a potential and we're already, as you noted in the answer, maybe the barn doors, whatever that's saying, of course the cows in the barn door is out. But we're expensive because we're trying to create personalized learning paths for each student that are based on proficiency standards and so you might have to do something different for our schools or little, so a hundred or two hundred students. That seems to me like that has potential to be a lot more expensive. You know, honestly, I haven't seen the research in that but I can't tell you whether there's research that we can look in to see if there's any research on cost. But I think the point is that you want to make sure there's an investment in resources for educators but I think that's true in any system. So I don't know if this system is requiring any more or less resources for educators. The point is you want to make sure that you have them. Does that make sense? Right, but how much do those resources cost? That's what, again, we didn't research that and I don't know that. Senator Root. Thank you. You said that learning is the constant and time is the variable. Is there some limit to the time though? Does that go across great levels or is there some expectation that at the end of every school year a child, no matter how slowly they've gone through, that they have to reach a certain level in order to get moved on to the next grade? Yeah, theoretically I think it could be flexible but I do think implementation does tend to fall within the traditional academic years. So there is an idea that you're trying to help students move through the great bands. So yeah, it's my understanding, yes, there is some thought about where they should be in terms of their grades but there's more flexibility within those great levels. Okay, and is there any evidence about, say, in Vermont in certain areas we have a lot of English language learners and sometimes there's difficulty in knowing where to place them when they first enter this country. We have refugee resettlement in a lot of areas. Is there any evidence remaining in the other states about how proficiency-based learning has integrated with English language learners in particular and whether that's been helpful to them or not? We'll have to get back to you on that one. I don't have that up top ahead of how English language learners are impacted by this type of system positively or negatively. Again, theoretically if you have systems in place it allows for more variation. But again, I think that would depend on what resources are set up and why they're implemented. So, I've always thought that, and I think that would be a good idea to try something for education. And this proficiency-based learning is concerning to me. It's been seven years since we've done inclusive data that's through millions of kids out there that are going through systems that even make you stupid because of the instruction. It's not for our men, but they don't know that and they may not be learners. So, I don't understand how you can start an assessment of how people use that. See if it really does advance through that learning. And the other thing for me is calculation. I mean, how do you go looking from one student in Canaan with transfers to South Burlington at every school and develop their own proficiency? How do we know that a fourth grader of a young Canaan goes to California or anything? They don't land. They don't come into that grade with the skills that all of you need to access the instruction. Yeah, I think that's where the state standards come in. That if the local districts are, their local assessments are aligned with the state standards, that creates the consistency that you're talking about. The key issue here that we're helping highlight is that proficiency, this move toward proficiency based standards again is trying to help create more flexibility and more variation in how teaching happens, learning happens. Again, theoretically that's a good thing, right? It is theoretical. Right, exactly. And so this idea that helping, that implementing these practices takes time, so to your point about seven years, I think one of the things that it's good, maybe that, another way to think about this, maybe more incremental change is a good thing because I think one of the messages we hear across the board is initiative fatigue, right? Too many things thrown at us, right? So that maybe is an argument for allowing things to move forward, initiatives to move forward at a pace that they need to. So it may take a while for a state system to get implemented and having engagement with stakeholders and you may go too fast and so you have to back up or you may not be going fast enough and you have to hurry up. But an item in the area we're maybe offering to consider is not whether you should implement it or not, but if you're going to implement it, implement it at the pace that you need to do it right and there tends to be more consensus of maybe let the initiatives that are happening move forward because again, this idea, so the worry is maybe initiative fatigue. Don't jump in and make a move in a way that would change direction too quickly. But that's not to say you shouldn't get involved. They get involved in a way that maybe is more incremental, but this is a suggestion to consider. Does that make sense? Yeah. And sometimes these things do take time and sometimes seven years seems like a long time. I don't know if that's too slow or too fast. Yeah. You know, under child's life, it's a children that are, you know, it's working, it's not working. The other ones that are going to feed the recipients. Right, exactly. But I guess the other thing about it is education changes are probably an ongoing process that are never finished. So hopefully it's getting better, but there's probably always some amount of change in the system happening. Yep. Thank you. How do we involve or get our teachers ramped up to do these learning projects? Has that been addressed or have they been surveyed perhaps in terms of their... Yeah, I'm not familiar with what surveys have happened here, locally, but again that's what we pointed out in the Colorado example where... What they've done is the State Department set up basically a work group, a pilot study group of various districts with the idea that they're engaging their stakeholders at a local level and then having basically a shared study, a shared experience going through together, acknowledging that this is a process and that it needs to be thoughtful and moved through. We'd have to do more research, and again I don't know what the answer is in terms of where things stand now, but from the concept that's one of the things we're suggesting states take a look at is how they continue to engage stakeholders throughout the implementation process. Representative James. I think my question was similar, but just anecdotally it seems as though the districts that are taking their time from out with a really thoughtful rollout that involves a lot of teacher support and training are doing well and districts that just dropped it on people or not. So is there a best practice toolkit or a state program you would point to or that really has put the focus on teacher support, teacher training, stakeholder buy-in? It sounds like Colorado? Well Colorado uses an example in terms of the state agency engaging districts in an implementation process. The resources I've mentioned, the Great Schools Partnership and KnowledgeWorks are two, several organizations, there's more we can point to. I know for example KnowledgeWorks, they have a state toolkit that talks about how to engage policy, excuse me, engage stakeholders and how to think about these policies in a comprehensive way. And again my understanding the Great Schools Partnership, they have, I mean here locally, they have resources and tools to help districts think through different implementation strategies. So there could be others, but those would be ones we point to. Thanks. So hearing, we hear anecdotally, we get a lot of information anecdotally. And we hear that Vermont is way out in front and colleges don't know how to deal with it. So in reality you're saying we're sort of in the first third, but it's a national trend. That's right. And again, there are organizations that study this more closely than we do, but when I actually spoke to one of those organizations in preparing for this presentation and it was their assessment that Vermont back around 2013, 2014 was more in the lead. But other states have maybe caught up or surpassed from a state policy standpoint. Oh, that's good or bad. But I think as you look at this map here it's clearly a growing trend that states are looking into. A growing area of interest among states. And the challenge that we hear less on proficiency based learning but on proficiency based grading. That's what we hear most of the confusion. Transcripts and grading, right? I think they're the points that come up and again, educator preparedness. And that's why I think of all the issues you deal with there's maybe more of a smaller subset of some of these things. And that's why I was talking about the clarity around transcripts, the engagement with post-secondary stakeholders and equipping educators. Those are some of the key components that we pointed to. And you said that you have a research study of college. Yeah, we can make it available to you. In fact, the slides that I am reading from have a number of resources in them already that have links to these so we can resend it. And so there was, in 2018, study of two different studies. One of them was a national one of college admissions officers and there was a reference to regional, here within New England, of some of the conversations that have happened among the New England Higher Education Group and this secondary school consortium. Is there anything more you'd like to share with us or are we at the end of your presentation? Understood. Just one comment. Yeah, we've covered everything I have. Okay. Jim. I just had a question, kind of on the coattails, so I was already asked, but I was not interested in parent involvement and participation and understanding of all of this, which I've received a lot of emails danglingly, also with their children. So it's from teachers to parents to the students themselves. There's a lack of understanding in me that I'm hearing on my end. I was wondering what I would go to or help those folks. Well, I wanted to say two ways. One, the idea that the schools would be providing information that the schools would already have. So hopefully it's whatever information you have, your local schools would have too because if the parents are getting information that the schools don't have, that can create a problem. So hopefully it's in coordination with what your own superintendent or your own administrator would have. Now that's the first point. But again, this is where there are national organizations that have different resources that are available for parents and for educators. And so I think the question is, is it a lack of information or is there some other concern below that? And oftentimes I think what's required is an ongoing conversation. And it's through that ongoing conversation is where we can address these. Again, we can make the information available and other organizations can too. But in positive, again, just all this thinking about as parents. The question is, the parents, is it a lack of information or is it a concern that they have about the system itself? So we do have John Carroll with us from the State Board of Education. John, anything you'd like to ask? Well, I think it's something we've discussed a bit. It's the question of faculty. And it goes to the center. I've got this question about measuring performance. I think a lot of us feel that while some sectors of our student population may be inconvenienced, I think especially those who are kind of transcripts, just kind of colleges, often convenient certainly. I think many of us are hopeful that PBL really reaches to those students who are not so comfortable in school and that's what we'll serve. And so I guess the question for you, sir, is what is the, just to refer to, so one of the rationale for PBL is that he didn't participate in a lot of opportunities for students. Everybody in this room, what's the data that's showing that we're achieving it? You're a Vermont? Anywhere? Anywhere. I think that's one of the key issues. I think what you're asking is if you look at the name scores and look at the scores, I think there's, I don't know if that's an answerable question because of all the variables that go into outcomes. So we're getting kind of going back to the original question. I don't know if there's enough information to answer that question. Sorry. I just don't have that. I just don't have the answer for you. It's entirely, first of all, if it can't be answered, then that's a serious problem. I mean anything worth doing is being able to assess. But it seems like there might, and this is Ross, to be thinking about, there might be ways beyond name scores and that sort of thing. It might be simply possible to survey teachers, educators, maybe even students to find out, is the world working definitely for the student who is perhaps marginalized for any reason at all? Is that students' world of opportunity and enterprise are four years of experience of rolling this out? Can I ask as much a question to the educational organizations in the state is to use their artistic. If this is the reason why we're doing it, are you sure, secondly, how far to find out if we're succeeding now? We're moving the needle? No, fair enough. And without any national data, it's a little alarming to see the colors shifting so quickly with the whole country moving in that direction. Ordinarily, you'd have the early adopters would be providing the data and then that would be what's driving the second wave but doesn't seem to be the case. Other questions? Thank you very much, gentlemen. We appreciate the trip and the deeper dig into it. I would be personally interested in those resources you mentioned around, I believe it's Illinois and what they do require. Thank you very much.