 Good morning. I'm Kevin Carey. I'm the Director of the Education Policy Program here at the New America Foundation. Thanks so much to all of you for coming this morning. I was downstairs on the third floor and we have a, our conference room is full of people watching this. So if you're somewhere else in the building, hello. Thank you also for sticking around for what has really been a terrific conversation this morning. Our next panel is going to focus on the credentialing part of this discussion. We have really a fantastic group here, people that are working in the private sector, in governments, on the research side of things with some great perspectives. Rather than having introductions, I'm going to just kind of dive right into the discussion and start with Tony Carnivali from Georgetown University. He's really one of the nation's premier experts on the intersection between higher education and the workforce and the labor market. Tony and your colleagues at the center have written quite a bit in recent years about the rise of certificates in the ecosystem of training, higher education and the workforce and that there really has been a remarkable and in some ways under-recognized increase in the number of people earning certificates granted through some combination of post-secondary institutions and industry. How and why did that happen and is it a good thing? I think in general, and in reading this report, it struck me that in general we're talking about a very rich and fast moving environment. Certificates are probably the primary indicator of that in the American system. That is they are, to a certain extent, a unique American invention. We need something shorter, faster, more direct. We got a lot of it. Some of it with economic value, maybe about half of it, some of it without, but had other kinds of value. Some people do want to learn basket weaving. So you know, there is a, the churn in this arena is what's striking and when I read this report, what struck me about it was that its analysis of the American system was largely what we often hear and that is that the American system is diverse, chaotic. There's lots of churn and change. There is quality sporadically here and there, but we have trouble bringing it to scale and making it consistent. That is a very old story about the United States written by the OECD many, many times in the last several decades. But the answer from the European perspective is usually that we need more committees and more meetings and more governance and more explicit standards and things that are really quite un-American and would probably lose us most of the kinds of creativity that we are associated with, including the development of certificates. I think the truth of this is when I look at the OECD report, my instinct was to give this to a graduate department like the Harvard Business School and say, here's another management problem. Tell me how you solve that and I know what they would say. They would say that the way you solve that in modern management as distinct from industrial management where you did have high governance, high touch, rule-based, top-down bureaucratic systems is that you run this system as we hear over and over again in all the comments before, that you run this system by specifying outcome standards. You don't interfere with the workings of institutions. You use information and then as we hear over and over again this morning, the first thing you get is transparency. The big issue, of course, is what do you do with transparency once you have it and that's where government and individuals and institutions intervene because transparency, as the higher ed folks know, is really the slippery slope to accountability. I think the first step in, the reason we all say transparency, is it has a how can you be opposed to it, but people are right to be nervous about it. It is the first condition of building an outcomes-based managerial structure in the modern sense. I think there are some of us here in Washington, D.C., that would like nothing better than to get higher education on that slippery slope and kick it so it slides down as fast as possible. Sharon Boyvin is the acting associate commissioner for post-secondary adult and career education division at NCES, the National Center for Education Statistics. You can go on the NCES website and get a pretty good sense of how many people are earning bachelor's degrees in America and where they got them and what they were granted in. It seems that our basic infrastructure of information about career and technical credentials is not as precise. I understand this is something that you're working on. Can you talk a little bit about that and what we can do to improve it? Absolutely. I'd be happy to, Kevin. If you look at the American Community Survey, the educational attainment of the U.S. adult population, if you look at people ages 25 to 34, these are people who are entering the labor market and supposedly need to have some kind of credential or competency or something to show an employer that they have the skills to do the job, if you look at that group of people across pretty much all the states, about 30 percent of the U.S. adult population of that age range says that they have some college because that's what we measure in educational attainment in the ACS is some college. But what does that mean? It doesn't mean that somebody necessarily has any kind of a credential that they can show to an employer to say I have the skills to do a job. As a result, back in 2009, an interagency working group came together under the, was commissioned by the Council of Economic Advisers, Martha Cantor, the Under Secretary of Education, and Kathy Wallman, the Chief Satistician of the United States at the Office of Management and Budget. And we brought together members of the federal statistical community from the Bureau of Labor Statistics, the Census Bureau, NCES, and now the National Center for Science and Engineering Statistics. And we've been working to develop new measures of the prevalence of industry recognized certifications and educational certificates in the United States. We've also recently started to develop new survey items that will help us understand the role of education and training, including non-credit education and formal on-the-job training in helping people acquire the skills that they need for work. Back in April, NCES released a methodology report based upon a 2009 pilot study called the Adult Training and Education Study, and it included recommendations from the panel, the Interagency Committee on Expanded Measures of Enrollment and Attainment, which we abbreviate GEMINA, and if you Google GEMINA, it's not the city in the Congo, and it's not the Ethiopian drama on TV, but if you go a little further down, you will see GEMINA from NCES, and that is the GEMINA I'm talking about, and there's a lot of information in there about the work that we're doing, but this pilot study report that was released in April contains recommendations for items on industry recognized certifications and occupational licenses that can go into federal surveys or into other surveys, if anyone is interested in grabbing these items for your own data collections, that will help us understand from individuals what is the stock of human capital in the United States. We heard Marlene say earlier that LMI is giving us better information about the demand, but what we really need more information about is what are people learning, what credentials do they have, what competencies do they have, and what can they show employers that will help them understand what the human capital is within their local labor market, within the region, within the country. And at this point, we are working both to deploy those items that had been validated and recommended by the GEMINA group to other federal surveys that, to all the federal surveys that have to do with households or individuals, and we are also working to develop a new survey within NCES that will go periodically to adults to get a broader picture of what the education training and credentials are in this country that people need to get and to use in order to get jobs. Thanks. Dr. Roy Swift is currently the senior director of personnel credentialing accreditation programs at the American National Standards Institute. Dr. Swift, you spent a lot of your career in the armed services. 28 years. Which is an enterprise that is highly skills and education focused. Very much so. How did the armed services measure the acquisition of skills and are there things from that system that are broadly applicable to the issues we're discussing today? Well, probably the military is the best example of competency-based education and has been doing it for many, many years in this regard. But I first have to say, I have to get this out, that certification is one of the most abused words used. Everybody has a different nation. It's kind of like Gomer Pyle saying that certification is like a box of chocolates. You never know what you're going to get. And that is very true. It is a word that has many, many meanings and probably the majority of people who say they have a certification program do not have a certification program based on national accepted standards regarding certifications. So when we talk about quality assurance and higher education, we also have to talk about quality insurance on the certification side of the house that more things are abused in the word of certification. And it's no wonder that employers sometimes say, well, the certification is no good. They arrived and they didn't have the knowledge and skills that they said they were going to have. Well, it's because of measurement. We have seen it in apprenticeship programs. We have seen it in certificate programs and we certainly see it in what are called certification programs that the quality of measurement is the weak Achilles heel. No one is really ground. When we see some of the certificate programs in this country, one of the biggest nonconformities in the accreditation is in the assessment that when you talk about a criterion referenced examination, looking at pass fail scoring, nobody knows what that is. So the academic preparation evidently is one of the issues that we need to talk about. Now in the military, you don't leave, you either are watched out or you don't leave until you have demonstrated performance. I remember I used to get very frustrated. I was in the health care department. I was involved with occupational therapy at the Academy of Health Sciences, where we were teaching what would be called a two year associate degree, occupational therapy assistant. And the rule is that there are a lot of performance examinations in the military because it's very skill oriented. And one of the things is that when a soldier fails a performance examination, they don't just wash them out because they've spent a lot of money in training to that point. They say you have to reteach and retest, retech and retest. I used to think, well, I don't know if they fail, they should just be kicked out. But you know what? Some of our best people who had to retest three times, but eventually learn the skill and prove their skill. This is one of the issues. The assessment again is the weak link because everybody can talk about competency based education. But first of all, lastly, you have to measure it. You can have all the competencies you want. But it's measuring whether they actually do it or not. So this is my plea, plea that skill based learning is really performance and that there is strengthening of the assessment tools. And that we've got to get out of credentialing terminology chaos. We're talking past one another. We don't know what you either talking about. When you say certification, you might be in one thing. You say certification, you might be in one thing. Certificate, sometimes it's certificate of attendance, sometimes it's certificate of completion, sometimes the certificate of achievement. We are credit agencies that are assessment rates certificate because that's what we chose to look at in this game through an American national standard. And we believe that there is a quality standard for certification on ISO standard. But what I think the report was referring to is the sector specific standards that would be a compliment to what a quality certification program is the ISO standard, which is also an American national standard is a generic standard about if you say you certified people, this is what you have to do. Because often a lot of people certify, but they don't have a recertification program. And it's not transparent as to what that is. Certifications can go all away from paying your money and getting a diploma or a certificate in the mail to one that is based on a job analysis that looks at what the stakeholders and industry is saying on an ongoing basis, because they have to update that examination or it doesn't maintain its content validity in this regard and its reliability. And the one thing that people forget about in certification, a true certification program has procedures to take it away. The certificate does not belong to the individual. The certificate belongs to the certification body who is to monitor that for the industry in this regard. And there's a lot of criticism, even if the ones that it will be considered premier certifications in this country, that they are not monitoring their certificates in a manner that would allow business to understand that there is someone there that first of all has an automatic way of updating because certification says that if you don't update in a certain way based on their standards, then you lose the certification. So industry has ready made mechanisms to help people continually update with the various things. And also then a monitoring system that if they are a complaint is lodged, there has to be due process. So that's more long-winded than I know you wanted, but you can tell I'm passionate about this subject because it's the other part of the scale. We talk about quality assurance and higher education, but the credentialing world is in just the much of a mess. And guess who controls the credentials? Not higher education, the professional societies, and we don't talk about the professional societies who are the gatekeeper. I was a specialized accreditor. I chaired the Accreditation Committee for the American Occupational Therapy Association. There's no communication between institutional accreditation and programmatic accreditation. We tried to do it at the SUNY Buffalo one time. We couldn't even get them to talk to us on site, you know, and we're both asking for the same information. So the universities get frustrated and say, I've got 55 accrediting bodies coming to me this year. They're all asking the same questions. There's the overlap of information. This is nonsense. And so the accreditation community needs to be called the task about looking at this in cooperation with higher education in looking at that. Sorry. Thanks. Brent Wild is Senior Vice President and Treasurer at the Manufacturing Institute, which is the 501c3 affiliate of the National Association of manufacturers. You know, part of the conversation around credentials that you hear a lot these days has to do with this idea of stackable credentials. And when I ask people, well, who's, you know, who's actually doing that? Like, where's the, what are good examples of stackable credentials? What does that mean? People tell me that the Manufacturing Institute is kind of like really leading that conversation. Can you tell us a little bit about what that means and the work that you're doing? Sure. But I'm going to start by putting it in context to start by saying that for many manufacturers, the majority of which are really 20 people or fewer, the career and technical system is really teetering on the verge of the relevance, essentially. And I mean, know that because we've asked them and when we asked them, where do they turn to for a qualified workforce? Where do they turn to for workers entry level on up? Top of the list is word of mouth. Second on the list is staffing agencies. And the bottom of the list of somewhere near margin of error is career and technical and community college system. Now we represent manufacturers and when I talk to manufacturers, I tell them they're out of their minds because we have a career and technical system that exists to train people into the skills needed for manufacturers. On the other hand, when I'm here talking to educators mostly, and so I'll say we are out of our mind as educators because we're really missing an opportunity to engage and we're missing a clear opportunity to have relevance for manufacturers that do need a very specific set of competencies. When we when we approached that, we did start with the advanced manufacturing competency model at U.S. Department of Labor really around the competencies identified by industry that were needed. And again, a lot of times the emphasis on education pathways look at the very top of the chain. Look at the engineers and the scientists. And we do need those, but we're not going to have them unless we have people that have basic skills in mathematics and applied mathematics being able to locate information that they're going to use being technicians that then can grow within their within a career pathway. That's really a lifelong learning opportunity. So industry actually has taken the lead. Manufacturers have taken the lead in defining a series of credentials that do lead up to from the basic levels into more advanced specific occupations and high demand, such as advanced machining and welding and the like. We did that really by setting our own criteria for what is what is going to be a solid credentialing system that's going to work, what industry certifications are there. And we base that on they had to be nationally portable, third party validated certifications, industry based and recognized data supported and time back to the competencies in the competency model. We have identified now 14 certification partners and pathways then that we endorse on behalf of the National Association of Manufacturers and are gaining traction in making in making that happen. Different and having very successful different models of how that can work. So when it's tied to industry certifications, then that gives a bridge from two plus two programs from high school into the community colleges that gives an equal footing and understanding of what are those competencies to it's not it's not just a matter of seat time but based on the competencies that are there. Articulation from two year into four year bridging from noncredit to credit programs all on the basis of those certifications that get back to do they really get to the competencies needed by manufacturers. Some of the success that we've seen in then has been for example in accelerated models. We know that the vanishingly small the number of people they're going to finish a two year degree in a two year time span. So really how do we can we accelerate in a credit model for that includes credit and industry certification in a way that does have market value there really bringing in manufacturers to support from the outset for example in a machining program what is needed in a CNC machining operation program that would be in an accelerated way. So students come in they take the national career readiness certificate as an entry they earn certifications by the National Institute for Metalworking Skills which is an industry group in the process and in a semester time frame then are ready to go into into a paid internship by manufacturers on the other side that then completes that circle. It's in a credit program so students can complete go to work. People can complete continue on in their education or they can do both because most manufacturers have a tuition reimbursement program to allow people to start to work and then continue on in that program that we call right skills now. The we know that these are working because first of all the manufacturers have been tied in closely have to support with internships are brought into that into that process from the very beginning and we know works on the part of students because classes that had been declining programs that had been being cut in areas where right skills now has been put in have been full classes additional tracks being put in and looking to expand on that in a stackable way to then include. OK what comes next after that first semester that does tie back into the industry certification. It really does come down to close attention to to the manufacturing community and I'd say one more thing in relation to the OECD report which had many solid examples. There's a recognition that there's a lot of this does have to be voluntary. We do have examples of a voluntary declaration having some impact but again we think it's got to come from the employers themselves. One is we did endorse particular certification pathways that that became a symbol of quality. The other is that we came out this year with what we call the M list which is right now 42 institutions and growing of institutions that do tie to industry certifications that are endorsed by the National Association of Manufacturers and being that bridge then that helps ensure that the programs truly are relevant and truly are making difference for students and for and for companies. So this is a question I'm going to put to all four of you. There's a kind of a theme running throughout the OECD report where they say kind of reflecting what Dr. Karnavali said that yeah the American system is it's very open and it's very accessible it's diverse it's flexible but all of their recommendations have the foundation is a need for quality to understand quality to measure quality to instantiate quality in the way that we decide what a certification a certificate is or isn't in the way that we regulate organizations that are accessing federal funding. It's hard to argue against quality in the abstract. So what are the specific barriers to making quality matter more in this conversation. Well I can I can say that we have to have it when we decided that we wanted to accredit certificate programs we looked for a national standard and we couldn't find one. So we ask one of our standard developing organizations since ANSI doesn't actually develop standards we accredit people to develop standards and eventually American national standards which are accepted most of the time by federal government we had ASTM International create a national accepted with input from trainers and educators as to what a certificate program would be that has an assessment at the end. So there is an American national standard for certificate programs but the probably one tenth of one percent now we've this program is only three years old we've accredited maybe 23 out of what 500,000 or whatever kind of thing. So it's just a drop in the bucket and as you say we haven't figured out how to have the scale ability of something like this. But I just wanted to point out that there is a standard that has been developed but we are not finding the educational community very accepting of it. It says it's too stringent and we go back and ask people and it's basic education really and yet they think it's too costly to try to meet in that regard. I'd like to say that I think there are two kinds of quality there's theoretical quality. We can say here's what we think a program ought to entail and then there's empirical quality. There's whether the people who have gotten a credential from this program are able to get and keep good paying jobs and in order to assess empirical quality we need better data and Tony has encountered this in the work they've been doing at the Georgetown Center because the lack of data is a real barrier to us understanding the return on investment in different kinds of credentials that are recognized by industry. I'll mention one thing about data I mentioned the federal efforts but I think it is critical for us to understand the return on investment on these kind of data for state longitudinal data systems to start collecting and using data on workforce outcomes of students who go through their educational programs and to collect whether students get certifications get licenses get certificates and follow those adults and young people over time to see how they are able to use those within the workforce. So I would encourage all of you to look at the work that is starting at the workforce data quality campaign to start including these kind of measures and state data systems and those of you who are in states to please be very aware that this is a critical way that we will be able to measure empirical quality of these kinds of things. I think I mean my favorite is the 48 hour inspection. I can see Marlene Seltzer and I who've been at this for 30 40 years now kicking down the door at some training program with our Westat badges or something demanding quality but the first thing we'd have to know is what do we mean by that. And one the employability kinds of metrics only one kind of quality if you want to go learn basket weaving. I mean we have to decide as a people we're going to pay for that. What are we what outcomes are we going to pay for in the public sector and which will be privately paid for. I have no problem with paying for a certificate of liberal learning from Nova which takes about a year has no economic return. But why would I deny somebody that and let them go to Georgetown and take Latin we have Latin majors and classics majors and so on. So you know there is a the transparency conversation is the doorway into the other conversations which are much harder. Ultimately I think they come down to what are we going to pay for. I think in the end we'll pay for a lot more than higher ed is is concerned that we won't pay for Shakespeare will only pay for HVAC. I don't think that's going to be what happens. Dr. Swift I'm curious just to elaborate a little bit on your what you were saying. So you came to this point of conflict with institutions you set standards that you thought were reasonable they thought they were unreasonable. What specifically were they objecting to. What do they think was too stringent or or. Well it it the the ASTM twenty six fifty nine I can I'll pick out three or four things that I think they don't. One it has a quality management system component attached to it where there has to be continuous quality improvement involved in where they have to ongoingly look at themselves which is not a usual phenomena in some of these certificate programs. Also it has the requirement of the oversight stakeholder and they're used to having like was mentioned in the previous panels of once a year meeting where this sort of thing requires ongoing dialogue and other types of input and then the the fourth one the third one is really the assessment. It requires a criterion referenced examination with a variety of stakeholders involved to determine the assessment tool and determine what constitutes pass fail. This is the issue about quality is that and the problem with certification being a garbage pan term that everybody uses and nobody knows what it is. It's really hurtful to the consumer to the public and to the employer because we may be passing people that shouldn't have passed and we may be failing people who should have passed because the instruments that we're using have no validity to them. And so when you talk about quality I do think naturally I'm from the American National Standards Institute. I think there should be standards of course. And we've also talked to this emerging what I think because there has been a lack of quality that this concept called badges which is sort of scaring me to death because it's just another one of those credentials you know but I think it's a symptom of what's not happening in education is the reason they've come. So just give me a badge that I earn and then that will show the employer what I can do in this regard. And we've got to get a handle on this because this is going to proliferate more and more in our country today. So I think those are the three elements we hear the most about the people that we have accredited so far interestingly enough as the federal government the FBI has been one of the Army Safety Center has been another and professional societies like the apartment complex with the maintenance people who has certificate programs that they show the employer not one community college has stepped forward and this is where most of the certificate programs come from. And once community college said to us if we follow that standard we would have the best program credit or non credit in the whole institution. Now that I did I thought that was kind of a sad statement actually because and I think it's this ongoing systematic thing and the assessment seems to be the areas that seems to be the most frustrated to the educational institutions. We have a little time for questions from the audience so please raise your hand and someone will bring the microphone. Yes. Right here. Hi my name is Michael I'm with the Education Writers Association. This is a question about whether certificates gel with what employers and industries want. So last year there was this new cycle of of certificates sort of being a high entry barrier to jobs. There is that core case in Utah about a hair braider who didn't have a certain certificate and so she couldn't have a job and then she thought to actually be a hair braider in Utah. So I'm wondering if there is this like tension between having a certificate and actually being able to work in a specified field and whether there's resolution whether that Utah case was something in the extreme or whether this is widespread. Well I can tell you a couple of ones and Brett can talk about manufacturing. We see certificates and being used sometimes in the areas of which it involves health and safety. And we have two states now that have passed legislation that says that in order to be a food handler in California and in Illinois you must have a certificate that is accredited under the ASTM 2659. So we do have two states who said oh there is a standard about certificates and food handlers is a safety health problem in this country and therefore we want a quality certificate so therefore they have mandated it through legislation. And so we do see these sort of mandated certificates where there is national security safety finance and health involved. I would certainly agree with that. I mean there's you see licensure requirements that typically are not going to be found within manufacturing and I can't speak to to hair braiding and kind of the what's there. I can say that frequently I do hear from manufacturers that say this would be great looking at for example a certified production technician that matches up to really what all manufacturers require is an onboarding and the kinds of safety and health kinds of things that they would require from their employees and say but or even a national career readiness certificate and what that means in terms of reading and writing and being able to use fractions on the job and say if I ask for that essentially the bare minimum I will get nobody. So therefore they ask for very little and that's really what sometimes they're getting. And so as a as a country we have to look at that and say OK we can do better. Now you're not good what you're not going to find is companies requiring an optional certification and with very rare exceptions because of human because of HR requirements and EEO and what that means. But we are working with companies to step up and say we will recognize that because we recognize that it matches up to the needs that we have. And I have examples of a company for example a large oil and gas company that their operations managers took a look at the certified production technician which is typically can be offered in a short time frame and said that that certification alone if we had people coming in with that would save us 30 days in recruitment time and 30 days of on the job training time for my company when people come in. And that's a real benefit that companies are starting to pick up on in areas where education institutions are stepping up and offering them. The only thing I would say about that is that this is a very old question in the United States and that is we as a nation have always been anti-credentialing. It's a frustration for educators but we're very anti-credential Lincoln was a lawyer. He read for the law didn't go to law school. We had a movement in the 19th century to eliminate all qualifications for jobs. So if you start to push the American worker especially immigrant workers too hard on credentialing you got issues. The other piece is we have a labor market that is essentially driven very much unlike European labor markets. That is if in Europe to use an example I'm not sure is right. If you're going to make a cake in Europe you've got to have a baker's certificate. In America if you can make a cake somebody will eat you can be a baker. So you know that ideal you'll always get pushed back because we do know that credentialing is a way that people set up barriers to entry. I can tell you I'm a PhD economist in Georgetown. There's credentialing is a way you set up barriers to letting other people get your job. That is that's you know it's done in health care it's done and it's done all over the place. So there is a tension here are we so that when you do set up credentialing you want to make sure that the manufacturing credential doesn't keep a young man or woman with a strong back and good work ethic and decent skills because they didn't have the certificate from getting the job. And in the end that's where you'll get tension from supplies and places the FL CIO will push back on that. Not just the employers. Other questions. No one. Okay. Seeing none please join me in thanking our panel for a fantastic discussion. I think we're going to be.