 Now, I think the others pointed out the need for quality is a really overarching theme. It's one component, but it also over-arches everything. So we are going to talk about quality assurance in post-secondary education in the US, and we'll ask some questions. And let me introduce the panel first very quickly, again, looking at your bios. So we have a J-Box, Chancellor of Kentucky Community and Technical College System. Kay Gilter, Director of the Accreditation Group at the Office of Post-Secondary Education at the US Department of Education. Andrew Kelly, the Director of the Center on Higher Education Reform at the American Enterprise Institute. And Iris Palmer, Senior Policy Analyst at the Education Division at the National Governors Association. So before we get into lengthy questions, just to break things up a little bit, quality assurance in the US post-secondary system, the first three words that come to mind. No more, no less. Three words. Kay, do you want to start? OK, I'll say first, challenged. Second, fractured. And third, this is one word, poorly understood. OK, Iris. A black box. OK, Andrew? Simultaneously rigid and flimsy. I feel like that's four, but that's fine. No, it doesn't count. OK, all right, Jim. Less than needed. OK, well, so we're going to have a whole conversation about this, but I think that's a pretty clear signal of where things are, where at least the panel thinks things are. So this seems to jive with one of the findings of the report, and one of the findings of the report is that in order to fix this, we need to fund, that we need to link the funding to outcomes. I think you just said that other countries are much more demanding about accountability, especially when their public funds provided. And what's interesting about the report, and that I'm appreciative of, is even though in the title, it talks about career technical education. In this country, we have all these different funding streams, and one is a particular funding stream for career technical education. But it's actually a very, very small percentage of I think the federal dollars, at least, that go to what we consider education and career technical education. Sort of the big ones are student loans and Pell grants, over $150 billion a year, which is all part of the, as they refer to the Title IV programs. So I appreciate the fact that you called those out. I mean, that's where the money is. And again, it sort of gets to this idea that we're talking about this false dichotomy between sort of money that's in the higher education pot and money that's in the career pot. And no, in fact, all of these systems need to be talking to one another, and let's use the big levers that we have in Title IV dollars. Federal financial aid dollars are a big lever. So with that in mind, let's sort of jump right into this. So the report spent a lot of time talking about the role of the federal government and accreditation in the quality assurance process. So Kay is our resident expert here. Could you just sort of briefly talk through sort of what this triad is and sort of describe the federal role for us because it's still confusing to me and it would be helpful to hear. Okay, well, as Simon said, we have an extremely decentralized system. And that leads also to decentralization in terms of responsibility for higher education quality assurance. The triad is the Department of Education, the states and accrediting agencies. And many years ago, when we had concern about the provision of dollars from the federal government to support students in their educational endeavors, we made use of an existing system, the accreditation system that had been in place since the 1880s in some cases in order to take care of the main part of the quality assurance challenge. But also in existing are all of the states and many of those states have very robust oversight, particularly at the career and technical education level of the quality of the provision to their students. The Department of Education primarily has the role of oversight of accrediting agencies, that is we recognize certain accrediting agencies as gatekeepers to federal funding. We also have oversight of all of the administration of the monies that go out of our Title IV Student Financial Assistance Program. So there are concerns about the financial stability of an institution, the ability of the institution to actually administer those funds responsibility. So we try to, ideally this triad would be one where there's a lot of communication across and among the players. This has been particularly challenging because the states are not a single entity each. So within each state there are varieties of entities that are responsible for different parts of the educational system. And you may have heard something about the state authorization challenges that plays into it as well. We have tried to specify what it means to be authorizing an institution to offer post-secondary education and we find, of course, that there are great differences in the states. So that being said, the accreditation system is a voluntary system, except that it isn't. As I mentioned, in order to have institutions eligible to participate in the Title IV programs they have to be accredited by a recognized accreditor. So our role in the federal government is to take what the Congress has given us in terms of the statute of what's required for recognition, to elaborate that in regulations and then to apply those regulations in our review of accrediting agencies to see whether or not they meet the requirements. The statute has in it 10 required standards. Leading in those is the standard on student achievement. The standard says that we have to look at, there has to be a standard of student achievement and as appropriate it should include measures of completion, placement, and licensure passage rates. As appropriate is certainly true for CTE programs and when we are reviewing the entities that accredit CTE programs we do ensure that they are looking at those measures and that they have established some thresholds, benchmarks in those particular areas. We are however forbidden from regulating on those required standards areas. So there is very wide variation among the types of agencies that accredit CTE and other programs on how prescriptive those standards are. And there's a rationale for that if you think about how a very large research one institution and the kinds of programs that are offered there may be anywhere from four year program to graduate programs or even a comprehensive institution which has at the associate level as well as a baccalaureate level. And there's a very strong resistance in the higher education community for the federal government to be doing anything very prescriptive in the areas of quality assurance. That being said there is a lot of pressure that's coming from a lot of sides in the current discussion and debate on accountability and transparency. In CTE it's particularly complicated because there are varieties of types of agencies that accredit CTE programs. I don't know if you're aware that we also recognize some state agencies for the approval of CTE programs in an accrediting agency role. So there are four state agencies that we recognize, Pennsylvania, Puerto Rico, the New York State Board of Regents and Oklahoma. In the past there were a number of others but the difficulty of sort of taking the state registration and oversight role and then trying to add in what was required in an accreditation role became particularly challenging for a number of states and so they dropped out. Those are gatekeepers to Title IV funds. So those institutions are not otherwise accredited except by the state agencies. We also have regional accrediting agencies mainly for degree granting institutions and a lot of the community college-based CTE provision is under the accreditation of those entities. There are also regional accrediting agencies focused on CTE offered primarily through K to 12 schools. This has been a real culture conflict in that the K to 12 kind of oversight and accreditation is very much on a school improvement model and our requirements in terms of recognition of accrediting agencies is much more compliance-oriented and we have had actually recently four of those commissions of regional accrediting agencies no longer being recognized by the Department of Education because of that kind of conflict. There are also national accrediting agencies that focus and have historically focused on career and technical education. These are primarily accrediting for-profit providers and they have been relatively good at setting very clear benchmarks and looking very closely at each program at the institutions that they accredit. I won't say uniformly good at do that but there is certainly a lot of practice in those areas to doing that. I think that the one other thing I like to say is that you probably know about our gainful employment regulations which did reach a little bit of an impasse with the court intervening. We are of course rising up and trying to do those again and we will be negotiating new gainful employment regulations. We do believe that those are a very significant way to try to identify the successful programs and those programs that are failing to meet the needs of students. Great. Thank you, Kay. So Andrew, one of the primary recommendations which I've talked about is tying federal financial aid to outcomes. Do we need to do this? What outcomes matter? Should we do it? How could we do it? Tell us and let us. Wow, well should we do it? No pressure but take notes. Yes, that's a lot of questions. So first I want to say thanks to the OECD folks, the researchers for doing this. You're the latest in a long line of people to critique the quality assurance system in the United States for higher education more generally but also for CTE. Maybe some international prodding will drive the change we've been talking about since the early 70s. But the question about regulating on the basis of outcomes and sort of basing decisions about who has access to title foreign outcomes is something that is intuitively very appealing. I think it's, we've heard a lot about how complicated CTE is and how complicated the measurement of the outcomes of CTE are. But I actually think it's much simpler in many ways than the fight about the degree granting for four year colleges and that there's a legitimate argument at that level that folks are there not necessarily just for job training. They're there to become enlightened citizens to go do a mission somewhere abroad. Whereas in CTE it's pretty clear why you're there. You're there to get a job. And what's striking to me is how if you look at title four eligibility versus the way we evaluate, say, WEA. WEA you actually do have to make your employment outcomes public and states have to post them publicly and say this is how we're spending our WEA dollars. This is how we've placed people into jobs and so on. Title four, we don't really expect much of that on the part of providers. So I think the logic of basing quality assurance on outcomes is a great idea. I would posit, though, that what we need is not, we need to get away from binary outcomes, which is what we have now. We have binary outcomes for both state authorization and for accreditation. So what happens when you have a binary outcome? Well, it becomes incredibly high stakes and you see what's happening in San Francisco, with the Community College of San Francisco. They are threatened with losing their accreditation. Looks like they will lose their accreditation pending an appeal. What happens is if you just have a zero one variable, you either have it or you don't, it sets this stage for these high stakes decisions. It's also massively uninformative to students. And one of the things that appealed to me about the talk from the folks from OECD was the notion of posting program reviews and process audits online or in public so that folks can see what's actually happening in these programs. Accreditation reviews aren't even made public oftentimes. It's sort of a private kind of almost collusive deal. So what I would suggest is certainly focus on outcomes for quality assurance, but also don't lose sight of the fact that we need to move away from a binary system of eligibility for these programs. There should be sliding scale for eligibility. If you don't perform well, we should slowly ratchet down your eligibility for aid, I think is the way I would do it. Makes a lot of sense. So Iris, working with NGA, working with governors. So we're talking about the triad, talking about the federal and you have a lot of experience with states. So what is the role of states? What should it be in ensuring quality and post-secondary education? Are they doing this? I mean Kay said some states are doing it really well. Are they doing it by and large? And what are some good examples we can learn from? So I think Kay had a very good point, which is that state authorization is very inconsistent across states, how they actually implement it. It's also by and large focused on consumer protection, which is really important, but doesn't necessarily get to the heart of quality. And most states are pretty happy as far as the quality of education to leave it up to the traditional arbiters of quality, the institutions and the faculty. And that is how higher education has been judged as quality throughout the history of the United States. But some states are really trying to get more of a handle on quality. And sort of the push for this is that they are putting more of their funding through outcomes measures. And I think the worry is, is that if you don't hold quality constant and you don't ensure that quality is constant, it may decline as you tie funding to it. So states are looking at outcomes and quality in two ways. And I'm kind of gonna go put state authorization and the compact for state authorization aside for a moment. They're looking at measuring learning outcomes through standardized assessments. And they're looking at connecting labor market outcomes to programs, particular programs. And in those, some of them are just doing a public reporting piece. And some of them are actually connecting it to their accountability systems and their funding systems. So if we start with measuring learning outcomes, Kentucky is actually doing this for their four year schools. They are posting online the assessments that their students are coming up with. And Nevada is doing the same thing. There are several other states. Then for instance, Tennessee actually has this as a piece of their performance funding formula for their two year schools. And in South Dakota, students actually have to get a certain score on a standardized assessment to leave the general education core. So those are some states that are using it in part of their accountability systems for students and for the institutions. For labor market outcomes, posting it online, I think we've all heard of college measures and there are states, Arkansas, Nevada, Texas, Colorado, who are putting labor market outcomes online in a way that students and families can actually see what the outcomes are for their programs and hopefully use those in making decisions for which programs to go into. Hopefully, I mean, we don't wanna misuse our labor market data. I'll put that caveat out there. But some states are also using it as accountability in their funding formulas. So the Texas State Technical College System is allocating a large amount of their funding based on their labor market outcomes for their programs. The Wisconsin Technical System is looking at doing the same thing and Missouri is looking at the placement of recent graduates as a one option in their performance funding formula as well. Okay, and to the last piece of the triad, although not representing a creditor, but a creditor should represent institutions. Jay, what can institutions do to respond to the ever-changing skills that are needed by industry? It's constantly turning. How can industry and institutions work together to increase both the value and demand for the credentials that colleges are delivering? Thank you. The traditional way that we've always dealt with the changing skills needed in industry is to have advisory boards that work with our programs and give us feedback and then we make changes to curriculum and move forward. That's partially effective, but the reality is those advisory boards meet maybe once a year. By the time the feedback gets rolled into curriculum change, then it's two or three years down the road and we're not as responsive as we need to be. So in Kentucky, we've started turning more into technology and current reporting that kinda helps us understand what we need to be doing on a just continuous basis. For example, we've always used labor market information to track employment needs in certain careers and that helps us make determinations of when to ramp up a certain program and when to scale back. But even LMI information can have a delay time. We've recently started using a more real-time LMI, which has been very, very helpful. But the most- Sorry, do you folks know what LMI is? Labor market information. Yep. Yes. The most recent thing that we've done has been very helpful is that our entire state, through our workforce investment boards and our workforce development area plus our colleges, our community colleges, have bought into a software called Burning Glass and I'm not just promoting the software. I'm talking about the tool that is provided, which takes the job openings that are posted out there by industry, which requires skill sets that are required for that job opening and they post that and we match it back up to the actual programs, credentials that we're offering. And it's been very telling for us because we're able to see what real-time, what the job opening is, and see where we have matched up the skills that we need and if they're missing. For example, what we found just recently, we did a look at our registered nurse, our ADN program across the state and we saw an anomaly that we hadn't seen before and that was in certain parts of the state. All of a sudden the hospitals and the other healthcare providers were requiring Spanish of all of the nurses that they were hiring. That was not in our curriculum. It was not and we said, oh gosh, we're missing something here. So it opened up two opportunities for us. It opened up the opportunity for us to immediately change that as a requirement for our degree, but it also opened up the opportunity for our workforce training division to start going out to those healthcare providers and saying, we can come in quickly and train your current nursing staff and bring them up to speed in nursing. That's the kind of immediate feedback we need to be able to be responsive to the industry. Great, thanks. So there's been a lot of talk about recently, at least in this town, I guess, isn't that the name of the new book? So this town on quality and accreditation and the house just held a hearing to a few weeks ago on accreditation and I think there was really some sort of, not even confusion, but just dumbfoundedness at really what some members of the committee sort of expressed as disbelief that this system, that the accreditation system, which is this voluntary peer-based system, basically holds the keys to the Title IV Kingdom and to these $150-plus billion worth of financial aid and that we have no real guarantee on the other side of basic things like students graduating, getting jobs, getting jobs that allow them to pay back their loans, et cetera. So there's a lot in the ethos and in the air around this accountability question. So I know we want to have time for questions, but just sort of quickly, sort of thinking about that and going forward, what do you think are the biggest opportunities to improve quality in US higher education? What are the biggest challenges and obstacles right now? So who wants to start? Andrew, do you want to start? I made the mistake of looking at you. You did. So I think one of the things that the report highlights and that we don't pay enough attention to here, particularly in quality assurance is the programmatic accreditation versus institutional accreditation. To me, CTE seems really well-suited for programmatic accreditation that potentially spans institutions. The institution-centric view of the world, that the entire accreditation system is built around that. I think that is sort of, we're sort of moving past that, moving past an institution-centric view. And so I think that's an opportunity. I think thinking harder about programmatic accreditation or programmatic certification as it relates to eligibility for various pots of federal money. The biggest obstacle, frankly, is that this has been, this is a game of kind of kicking, handing the hot potato off to one other. So we decided to hand the ball to accreditors, right? To ensure quality. And they said, well, that's not really our job. We're not gatekeepers. They said, well, we'll do it anyway. And then the accreditors say, we don't want to do this. And we'd rather not be doing this in some cases. And but everybody kind of holds their hands up like this and doesn't want to touch it either. And then the last thing I would say is just, I think that the collaboration between employers and the higher ed system is a big area of frustration on both sides of the coin. I think employers feel also that it's not just that they're, I think they're not, they don't feel like they have a very receptive audience often in the higher ed community. And I think that's a big problem. Kay? Well, I think there actually has been a little bit of light emerging out of the higher ed community itself. There has been a drumbeat for many years on trying to better define outcomes and measure those outcomes. The resistance has been within higher ed institutions, I think more than within accrediting agencies. And accrediting agencies have been trying to drive this for a number of years. I think that there is a real sense that the time is getting very short. And in fact, you might have seen this morning within the inside higher ed and the Chronicle, both had articles about yet another new system for that several institutions came together in trying to define what would be the measures that would be made public about their performance as institutions. And there's a lot of thinking that's going along those lines. So it's still very slow, but I think there has been a wake up call that's been heard. I would actually say something that we didn't really talk about here. And that's the sort of rise of competency-based education to which you have been a party at the federal level. And I think that that actually has a huge possibility of certifying and demonstrating actual student learning and holding student learning constant rather than time constant would be a huge breakthrough in this. I think the challenge is what the challenge always is in demonstrating quality. And that is making sure you have valid assessments that actually say these students actually know what we need them to know. And I think that continues to be a struggle. And I think that it's going to continue to be a struggle as states and the federal government struggle to change their policy structure in that way. Thank you, Iris. I stole your one. Yeah, you know that. But of course, Kentucky is highly involved in the driving the competency-based learning approach. And no offense, but accreditation is critical because we have to have the accreditation. But in career and technical education, the bottom line is, are our graduates getting employed? And are they doing a good job for the employer? And so we have a feedback system. What drives us the most is are our graduates successful? And so we are turning much more attention to what are the certifications our graduates have? How quickly do they pass licensure exams? How quickly are they employed? And what is their average salary upon employment? Those are the kind of feedbacks we need to make critical decisions. The accrediting looks at things just, in my opinion, way too broad and doesn't look at really the end results. And that's why I said that at my three words, less than enough for accreditation, I really think accreditation can get a little bit more directed into what we really need, and that is to look at the end results of our graduates. Great, thank you. And actually, I want to open it up for questions until Claire tells me that we have to stop. Hold on, Sharon, she's very excited. But I would actually just, I love hearing your three words again. I would love to hear the other panelists' three words again just so that I can write them down also. Kay, do you remember what yours were? Challenged, fractured, and poorly understood. Oh, right. Mine was simultaneously rigid, and then I used a symbol like an ampersand. Flimsy, and if I could explain what I meant by that very quickly. We don't have very good quality control, but yeah, the system keeps a lot of people out. Right. Do you remember what yours were? Yeah, a black box. So I'm counting the article, and I think it's pretty self-explanatory. Great, thanks. Okay, we have some questions on the front. Yes, the mic is coming. If you could identify yourself and where you're from so that we... Hi, my name is Sharon Boyvin, and I'm from the National Center for Education Statistics in the U.S. Department of Education, and Jay was mentioning the difficulty in responding to employer needs because of the time that it takes to set up a new curriculum and get it approved and stuff, and we know that as a result, a lot of the occupational education in this country takes place through the non-credit system in community colleges, and yet those non-credit programs are not subject to these kinds of policy levers that we have. You can't use federal aid for them. States don't count them because they don't fund them with the exception of a few like North Carolina, and there's virtually no data or outcome measures on non-credit, so what policy levers do we have to assure quality in that very large portion of the occupational education in the U.S.? A black box. We've done two things in Kentucky. First of all, we've modularized all of our credit curriculum and encouraged in our workforce training side that they apply a credit type of curriculum in what they're using when it fits, and that allows them for employees who are getting training to be able to work towards a transcripted credential. Now, the other side of it is we've been doing that for several years, but we had never talked about the value of working with that employee and employer to say, how can we transition that student toward a credential, to toward some kind of certification down the road? So for the last four years, we've been tracking how many of our students will actually transition from a workforce training initiative over into a credential seeking student, and we've set performance measures for our institutions to work with those employees to transition them in towards credentials, and we've seen we're up to 8.5%, our goal is 10% by next year, so we're making progress. I think that's a really important question. Three quick points about it. One is it, I think it points out a lot of how we differ from other countries especially, as we don't count a lot of stuff that they do. That's partly what explains our lagging behind, if you wanna call it that. The second thing is on the one thing I would sort of push back on a little bit in the report is that at times it feels like the quality assurance problem is only in the for-profit sector. The for-profit sector has a lot of problems. We know that. I would suggest though that the point you started with, which is nimbleness to respond to labor market demand, that's one strength the for-profit sector brings to this question in particular. And then the third thing is just on the non-credit side versus credit side, one of the things that I think is interesting or I'd be wary of is bringing those programs under Title IV eligibility and what that might do to the pricing of them. So. I think there is an opportunity in terms of the competency-based models so that where you earn your knowledge or achieve those skills is less important than you're being able to demonstrate them. So I love Kay for saying that. So we agree here at New America Foundation and we are actually we are embarking on a research project around non-credit and would love to work with those of you in the audience who are interested in this issue. So look for an event maybe next year on this topic. Other questions? Yes, sir. Good morning. I'm Barry Stern with the Haberman Educational Foundation. Most of the OECD countries, or at least many of them, have what they call employment qualification frameworks. I think we mentioned Scotland as being one of them. It's been very difficult to get anything like that going. We had the old Noik and Soix back in the 80s and 90s. But Kentucky is doing some interesting things in terms of sector-based economic development, trying to align the educational institutions, the colleges, schools, universities and so forth with the economic plans of different regions. Then we have accreditation agencies. But the interesting thing about the European and Asian employment qualifications frameworks is that they're really dominated by employers. They pretty much say here's what you gotta know and be able to do to get employed. So try to help me out with this. How do you relate accreditation, employment qualification frameworks, and what Kentucky is doing in sector-based economic development? And if folks who aren't from Kentucky also want to start. I guess the only thing I would say to that is the call-out that was earlier to programmatic level accreditation. I mean, I think there is more responsiveness at that level and more knowledge of what is required in terms of employment in particular fields. But not a good answer, but... Is anyone... Oh, I was wondering... I was just gonna say, I think this is, I think again, this is sort of a... The political economy here is different, right? Our employers relate to one another in different ways. And there's a high level of competition, a high level of sort of emphasis on asset-specific skills, right? And that makes it harder to sort of collaborate. I think there is a movement, though, to develop certifications across. I know there's something going on in aerospace. There's a bunch of work in manufacturing on this front. So it's not exactly the same as sort of... I don't think it's the same as the employment qualification profile, but it's a similar effort. I would also say that competency-based programs are working very closely with employers to create the competencies, but they're not widespread and scaled in the way you were alluding to. And I think I've gotten the time signal. So I know there are lots of questions. I think we could talk about this forever, at least we all could. But I appreciate all of your questions and appreciate all of your thoughts. And thank you for your three words, we will remember them. And now we'll have the next panel. Thank you guys. Thanks.