 Hello, everyone. Welcome to our webinar Beyond Standardized Test, using performance assessment in college admissions. Thanks for joining us today for our 75-minute webinar. While we give a few minutes to make sure everyone gets logged in and online, how about if those who are here use the chat function to let us know who is on? You can list your name, organization, state, or region, whatever is most comfortable for you. You can also engage in discussion in the chat throughout the webinar. We'll be listing resources in the chat function as well. And if you have any questions, please submit them using the Q&A function at the bottom of the screen. Thanks for great, thank you. I love that you guys are populating the chat. I'd also like to let the audience know that this webinar is being recorded. A video recording along with a summary of the webinar content will be emailed to you in a few days. The slides are currently available at the link in the chat box. Very excited today that we have almost 700 folks registered. Today's webinar will examine the use of performance assessments in secondary education and in admissions specifically at the City University of New York, CUNY. The webinar is being held as part of the Reimagining College Access Initiative, lovingly called RCA. It is a partnership with Education Council and Education First. We believe that high quality performance assessments, if organized in easily reviewable form, could be used as an additional source of information about students' achievements and potential for post-secondary success in college admissions as well as for placement decisions or advising. My name is Monica Martinez and I am the Director of Strategic Partners where one of the initiatives I support is the Reimagining College Access Initiative on behalf of the Learning Policy Institute. I'm relatively new to the Learning Policy Institute so I'm really excited about our being part of our first webinar. For those who have not heard of RCA, we began in 2017 when LPI with Education Council brought together a group of individuals and organizations to explore interest in and the value of using K-12 performance assessment in higher education admissions, placement, and advising decisions to improve the quality and equity of these decisions by providing better information. Following this 2017 exploratory meeting, three taskforce were created to shape the initiative. As this slide shows, through combination of taskforce meetings and convenings with a network of interested individuals and organizations over the last two years, a series of these specific recommendations were made aimed at achieving RCA's vision. One of these recommendations was to work through local and regional partnerships to pilot the use of performance assessment in higher education admissions and to identify technology-based tool or platform to capture student performance information. To facilitate this, Reimagining College Access Initiative partnered with the Common App application, also called the Common App. For those who may not know of the Common App, it is an undergraduate college admissions application that applicants may use to apply to any of more than 800 member colleges and universities in 49 states and the District of Columbia, as well as Canada, China, Japan, and many European countries. Given the familiarity and usage of the Common App among high school students, as well as counselors, they've been great partners through Common App and the platform they use, Slide Room. Applicants can upload performance assessment materials at the same time and in the same way as their other application components, background, personal statements, letters of recommendation, transcripts, thereby allowing admissions officers, admissions reviewers, to assess and score their applicant files, including performance assessment. This way, we are not creating an additional barrier for students but are using a platform that integrates the inclusion of performance assessment into the existing workflow of a student's application and reviewer scoring process. This year, we've been working with and learning from five higher education institutions in New England who have piloted the use of performance assessment in admissions and used Common App and Slide Room as their application submissions platform through, even though at least another 120 higher education institutions that are part of Common App are capable of doing this. There's also another 750 organizations and 1000 academic programs on Slide Room independent of Common App that are able to use this platform to submit performance assessments. We are excited to see and support the momentum around reimagining college access and success as a growing number of colleges are seeking more ways to recognize and encourage the development of student abilities that go beyond standardized test scores. We are proud to be working with a broad-based network of K-12 and higher education policy and practice leaders including the National Association for College Admissions Counseling, NACAC, the American Association of Collegiate Registrar and Admissions Officers, also called ACRO, State Department leaders, school network leaders like Linked Learning and Visions, the New York Consortium. We're also working with the Mastery Transcript Consortium, a growing network of public and private schools who are introducing a digital high school transcript that shows in detail evidence of what students have done, strengths, abilities and academic history, the coalition for college which created an application with a digital locker that students can use to apply to 150 colleges committed to access, affordability and success, and also making caring common a project of Harvard's Graduate School of Education in a coalition that is focused on reshaping the college admissions process to promote greater ethical engagement among aspiring students. You will see on the slide that is just a sample of the many partners who have brought us here today to reimagine college access. Moving on to our speakers, our first speaker is Linda Darling-Hammond and she will talk about what is equitable and high-quality performance assessment and their role in K-12 education and higher education. Linda is one of the nation's most prominent education researchers, president and CEO of the Learning Policy Institute, the lead partner for RCA, and president of the California State Board of Education. Our next speaker is Joanna Kurcharski. She will tell us about CUNY's pilot program to accept performance assessment as part of the admissions application and how they were integrated into the admissions process and decision. Joanna serves as an associate director of admissions and the recruitment for the City University of New York. City of University of New York serves at least 500,000 students through 25 to and four-year colleges with almost 1,800 degree programs. Our final speaker is Michelle Fine. She will share early findings on the impact of the pilot program at CUNY to use performance assessment in admissions for students who did not meet the minimum scores on the SAT. Michelle is a distinguished professor at the City University of New York Graduate Center. Her research includes the topics of social injustice and urban education and she has received multiple awards for her scholarly contributions. Linda Darling-Hammond will wrap up the session with some final thoughts and we'll leave time at the end of this webinar for Q&A. As a reminder, if you have any questions please click the Q&A button at the bottom of your screen. If you'd like to engage in discussion you may click the chat button and type in the check box at the lower right side of your screen. With that we will start with Linda Darling-Hammond who will help everyone understand K-12 performance assessments and why they can help colleges and universities evaluate student admissions applications. So I'm thrilled to be able to talk about this question of what our performance assessments and why would we want them as part of the higher education admissions placement and advising process and I want to start by noting that what higher education does in terms of admission both gives them a sense of who they may be admitting but it also sends strong signals to K-12 education about what is valued and what is valuable and therefore it really has influence on the curriculum and the teaching that occurred there which is one of the reasons that performance assessments are an important part of this process. Performance assessments simply put are opportunities for students to demonstrate what they know and are able to do through actually doing it not by choosing one answer out of five predetermined responses. Ideally those are also opportunities for students to get iterative feedback and opportunities to revise their work so that it becomes more and more proficient towards meeting explicit standards. It can include open-ended problems, evidence-based analysis, quite often different kinds of research investigations in different disciplinary areas, exhibitions of learning where students show what they've learned in a variety of ways, presented sometimes to a jury of teachers and others from outside the school and defensive their ideas in response to questions, the kind of intellectual work that we actually expect to see students able to do in college. And now I'm trying to make the screen move. So performance assessments develop a wide range of cognitive skills as well as social and emotional abilities and I put this little cover of authentic assessment and action which is a book that I did back in 1995 which includes some of the performance assessments we're going to hear about today that were being begun that many years ago. We know that performance assessments can improve students' ability to analyze and synthesize information, use evidence, communicate in multiple ways, but they also require that students plan and organize themselves, manage their work, that they encounter challenges and learn to be resourceful and to persevere as they do complex work to collaborate in many cases and then to take and use feedback which is a critical life skill as well as developing a growth mindset as they iterate and evolve their work. And what we've found over many years of research on the use of such assessments is that they do, in fact, improve students' abilities to engage in higher order thinking, critical thinking and to develop these life skills as well as supporting their ability to dedicate knowledge in other ways. The New York Performance Standards Consortium is the site of some of the most sophisticated performance assessments for graduation from high school in the country. Some of the schools began doing this work as early as the 1980s, but certainly since the early 1990s the collaborative of schools have been graduating students by portfolio with performance tasks in English language arts, math, science, history, often the schools of the 38 of them use performance assessments in world language, the arts, internships, and other things. These are tied to the standards of disciplinary inquiry in those fields of scientific investigation, social science, research paper, literary analyses, mathematical modeling. They are undertaken in a process over the four years of high school, defended before a committee much like a dissertation defense, and students learn to respond to a wide range of questions and inquiries and think deeply about their work. The revision process occurs in relation to those standards, and what we've found in multiple studies over many years is that these young people, most of them from high poverty schools, many recent immigrants, as well as students of color, are graduating from high school going on to college and succeeding and persisting in college at higher rates than their more advantaged peers. And when you talk to them about what has supported their success in college, this experience is one of the things that they will talk about as a critical piece of their ability to succeed. This is a little challenging to see, but when you get the report on which this is based, you'll be able to see the rubrics for these tasks. They're called PBATs, performance-based assessment tasks, and this one happens to be for mathematics. If you can see along the left-hand column, you'll see that the criteria have to do with problem solving, reasoning and proof, communication, drawing connections, and representation of the mathematical ideas. This other one is for experimental science, and you can see that the standards have to do with the way in which the material or the problem is contextualized, the experimental design, the ability to collect and organize and present data, analysis, and interpretation of results, the ability to revise the original design, so this idea of iterative thinking, and then of course the defense of the material itself. This kind of scientific investigation, by the way, is part of the assessment systems in many Australian states, in the UK, in Singapore, and other places where the idea of assessment is to develop these critical thinking and performance skills. And so I'm going to pass the ball now that you know a little bit about the New York Performance Standards Consortium to Joanne Kaczarski. The one point that I want to make as we are continuing this discussion of the work is that these kinds of assessments for these students are not only a piece of information for college admissions, as you will hear, but they are also the way in which students grow up in an intellectual tradition and learn to use their minds well. Sorry, thank you, Linda. Thank you so much for gowning us in the research and giving us these examples. We actually have a couple of follow-up questions for you, Linda, and we have time before moving on to our next panelists. And so the first question that we have is, how does using performance assessment in K-12 create a different approach to teaching and learning? Well, the approach is one that emphasizes depth of thinking and of inquiry. It creates the disciplinary standards and facts are still taught. In fact, this portfolio that we're talking about was mapped against all of the region's standards. When new standards came in in the early 1990s, I was actually leading the curriculum and assessment council in New York at that time. This portfolio had to demonstrate that students were learning what was in the standards and thereby got a waiver from the other region's exams that were being used to measure in a more traditional way. But what happens is that teachers are framing big questions and themes, helping students see the dilemmas, the problems, the questions, the inquiries, giving them the opportunities, sometimes in groups, sometimes individually, sometimes in a combination, to go deeply into those to inquire, to use the mode of inquiry in the discipline to uncover the answers to meaningful questions, to write those up, to present them orally in writing and graphic representations, sometimes in the math and science arenas through a variety of quantitative representations, and then to apply that in real world settings. What we know about this kind of learning is that it's both more student-initiated. There's a lot more student agency purposefulness. Students are deeply engaged in their work because it is their work, and they are able to transfer what they've learned more effectively to other problems and situations and areas of knowledge in the future. Great, thank you. We actually have time for one more question, Linda. What features of student performance, what features of student performance can performance assessments reveal that are useful for admissions? Well, when you think about what people want to see students able to do in college, and I will say truth in advertising, I was a graduate school professor at Columbia and Stanford for 30 years, so I've heard the discourse and seen the needs. You want people who can think analytically, who can engage in inquiry and research, who can write effectively, who can communicate effectively, who can weigh and balance evidence, who can use ideas and knowledge to promote answers and solutions to problems. All of these are the kinds of things that are developed in this kind of work, and you see the difference in the ability of these students to engage in what the purpose of college really is when they've had this kind of experience throughout high school. As a college professor, I cannot tell you how many times I've heard folks talk about, oh, you know, students can't, you know, many of our students can't write, they can't think analytically. That is not the complaint with students who've been through four years of high school education focused directly and intensely on the development of those abilities. Great. I'm going to ask you one more question, so I'm not going to take myself off video. Sorry, but we have time. Thank you for this extra time, Linda, but this question just came in. Is the idea of that performance assessment would supplement or supplant standardized testing for college admissions? It could, you know, be used in any way that seems appropriate in a college. Many colleges are now doing holistic review, and I know at the University of California system, we just went through a vote about the admissions process last week. There are 14 indicators in the holistic review system in the University of California, which in some cases are used very, very widely and deeply. So this could be one indicator in a system of holistic review along with a test score or more than one test score, you know, including international baccalaureate, advanced placement, and other kinds of assessments. It depends on the college and its need. Of course, in some fields like art and architecture and music, this kind of material has been submitted, you know, in the application routinely, and is heavily weighted in the process. Depending on the size and shape of the University and its framework, one might look at these kinds of indicators for students in a smaller group along the margin of admissions or in, you know, the broader group. And one might look at different things. The tools that are being used that you introduced earlier, Monica, allow people to decide where to go deep and what to look at. It is also true that universities are beginning to use this kind of material for placement and advising as well. And I think we should think of that whole continuum. But there's no one answer to the way in which these assessments and others might be used in conjunction with one another. Great. Thank you so much, Linda. I appreciate that. And I really appreciate the audience sending in those questions. So thanks very much. We will move on to our next speaker. And Linda will come back and wrap up our webinar today. But thank you for this question. So Joanna, as previously stated, she comes from City University of New York where she works as the director of admissions for CUNY and has been part of this pilot program from its inception. Joanna is going to share why and how a system as old and as large as City University of New York uses performance assessment as part of the admissions selection criteria. Thank you, Monica. So in this section of the presentation, we'll discuss how CUNY and the New York State Consortium Schools engaged in an admissions pilot. I'll start by giving you a brief overview of CUNY, describe our traditional admissions process, and then talk about how we reviewed applicants who participated in the pilot. So just a little bit of history. CUNY was founded in 1847 with a mission to provide access to a quality education to all students regardless of background or means. So we are viewed as a vehicle for upward mobility for our students. And as part of our mission, we continuously explore opportunities which focus on access and students success. So from opportunity programs to our nationally recognized ASAP model, there are many initiatives across CUNY that support our mission. CUNY consists of 25 colleges in New York City, and we currently serve 275,000 degree-seeking students across our campuses. Now our colleges include community colleges, four-year colleges, graduate, and professional schools. Now let's transition to talking a little bit about application volume. CUNY receives a sizable number of applications. As of earlier this week, we received applications from over 86,000 freshman applicants, and those applicants have generated over 350,000 applications for our colleges. Separately, our transfer application has given us over 30,000 applicants and right around 67,000 applications for our colleges. Because we operate on a rolling admission basis, we expect increases in these numbers over the course of the summer. The admissions experience at CUNY is relatively streamlined. Students who are interested in applying to be considered would use the CUNY application. One application enables our first year students to apply to and receive decisions from up to six CUNY colleges. We also have a centralized university admissions office, serving all of our undergraduate campuses in application completion, evaluation, and recommendation. In addition, each campus also has its own admission office, primarily responsible for admitting and enrolling students. At all points of this process, all college admissions offices have full access to view their applicant pools and take action. Now let's transition to the consortium pilot. So historically, students from the consortium would apply to the university much in the same way as the rest of our applicant pool. Over time, we started receiving feedback from educators that students who would have historically been admitted to CUNY were not being offered admission. This, in part, was attributed to our four-year institutions, reducing the availability of remedial coursework and looking to college readiness benchmarks to determine readiness. These benchmarks were primarily determined by tests, standardized region's exam scores, SAT tests, and ACT tests. Our partners at the consortium noticed that with performance-based assessments, students were already engaging in a type of work that would be expected of them in a college setting and had the capacity to succeed in our most competitive campuses and programs, even if numerically they did not meet the benchmark for admissions at some of our competitive campuses. These conversations were truly the catalyst for our pilot work, which started in the spring of 2015 looking at fall 2015 applicants and expanded for the fall 2016 cohort. Here is a general overview of the process. So first, high school administrators and teachers were asked to identify students who are a strong fit to our four-year programs, but did not meet the typical academic profile of the traditionally admissible student. Next, we asked for high school transcripts and SAT scores to be submitted for application completion purposes. Additionally, we asked all applicants to submit personal statements, letters of recommendation, and a sample performance assessment that they felt reflected their academic ability. Centrally, we completed and reviewed applicants holistically. So broadly speaking, we used personal statements and letters of recommendation to gauge non-cognitive attributes that could determine or that could contribute to student success. Specifically, we looked for signs of leadership determination and personal circumstances to shape our understanding of the students. Performance assessments were primarily used as evidence of an applicant's ability to clearly communicate ideas. We focused on grammatical execution, development of ideas, reasoning, and how well ideas were connected together. And then quite frankly, in all of this, we were reading for fit to see which campuses would best fit the needs and goals of our students. We ranked each component and submitted final applications to our colleges, giving them context about the type of work that all consortium students do and asking them to admit applicants a student of promise, even if they did not meet the traditional benchmarks that our campuses were seeking. So just a couple of notes here. Performance assessments currently display as pass or fail on student transcripts, and some of the PBATs that we received were not accompanied by rubric. Being that we did not engage faculty in the review of performance assessment, this to some extent limited our ability to gauge the accuracy of the work, unless our staff lacking in the specifics of how each student was evaluated for competency. So this is an area that I think we could see some improvement and I'm excited to explore a little bit more. I also wanted to point out that the structure of this pilot in itself is quite significant. We asked students to participate if they felt like they were a strong fit to the university. We asked counselors and teachers to confirm to us that these students were capable of doing college level work. So by the time we received these files, we have strong reason to consider these applicants seriously and advocate on their behalf. So for implementation and expansion, and this is really quick. At the start of this review, our office did not have a platform that encouraged the supplemental submission of documents. Since then, we've moved to a PeopleSoft product. And as a university, we've been able to scale asking students to submit supplemental documents. And so this has really made the process much simpler for us. As far as timeline, early submission of applications has been key to our success. It gave us the best opportunity to communicate to our individual institutions early in the cycle, identify students of promise, and advocate on their behalf. And then finally, the one piece that we've been encouraging as we've got more familiar with performance assessments are more regular submission of rubrics. Throughout this process, we read work that had so much range, both in quality and subject matter. And since rubrics were not always included, it made it difficult to translate the work in a way that could be analyzed for trends and implemented at scale. So I think this is one of our takeaways is how do we potentially scale some of the information that we're seeing, especially since it's so promising. So just to conclude, university-wide, we're seeing support that a student's work in high school serves as our strongest indicator of persistence in post-secondary setting. And that factors, like self-regulation traits, are important predictors of persistence. Our work with performance assessment complements what of our research shows us and quite excited to see further evolution of this work. And also for you to see some of the results for students who chose to enroll with CUNY after participating. And with that, Monica, I'll hand it over to you. Great. Thank you so much, Joanna. I so appreciate what CUNY has done, the vision, the partnership with the consortium, and how thoughtful you were in integrating the student admissions process to support equity and student success. And you're really a north star for the Reimagined College admissions initiative, and so glad you made time for today. Before we move on to discuss the outcomes from this pilot, we have a question from one of the audience members. And they're asking specifically for an example of what you look at with a PBAP. So you talked about the fact that on the transcripts it comes through as a pass or fail. But when you're actually looking at a document, what are you looking at? And for the audience to know when it comes to the consortium specifically around the performance-based assessments they use, they're graded on a four-point scale, outstanding, good, competent, or needs revision. And of course, if it needs revision, it is because part of performance assessment is that you continuously revise your work, developing agency as well as integrating feedback and becoming a better learner and to improve your skills. So there are four, there is a four-point scale. But Joanna, when you receive a PBAP, what is the admissions office looking at? So we implemented over time, and I think there was some evolution, as we got familiar with the performance assessments, we really graded things on a three-point scale to say like this exceeded our expectation, this falls in line with what we would expect from a high school student, or we felt that this particular assessment really needs some improvement and additional work. When we were in receipt of the rubric scale that accompanied some of our performance assessments, but not all of them, we would use those as a guide to make decisions. And then quite honestly, we read the performance assessments anyway. It was an educational model for us to be able to see how those two complemented one another. In the absence of a rubric, we really focused on just the content and how well it was executed, the strength of the grammar, the development of ideas, as well as reasoning and connection of ideas. Again, because our undergraduate admissions process, at least from our central office, does not include review by faculty members. There were portions of assessments where we simply didn't have the staff to be able to speak to some of the accuracy of the work, particularly as it related to math and how our favorite subjects and admissions that I'm sure we could speak about. But broadly speaking, we were looking at the whole package holistically. We were looking for feedback from the community and the partnerships that we were developing with the consortium schools. We were looking at the quality of the work and then to some extent also seeing whether or not that was a match with the GPA that the student had. It's fantastic. I mean, essentially you guys were looking at content, you're looking at the school the student comes from, and you are looking at the rubrics when there is an assessment, but you're looking at student work, which is really laudable by you all and I know challenging for large admissions offices like yours. I just have a quick follow-up question that we actually have a couple of minutes for that's very related to what you just talked about and that was how did your admissions staff develop the capacity to review performance assessment materials? You said like with math sometimes that wasn't where your expertise was. I know at our next webinar MIT is going to share with us how they do it and how they involve faculty, but tell us how you develop the capacity of the admissions staff to review the performance assessment materials. Yeah, so I wanted to emphasize here CUNY's mission, the performance assessments that we reviewed and this pilot represents one of many ways in which we advocate for students centrally and so advocacy has always been part of the work that we've done centrally with the university, but broadly speaking we assessed for the information that we could reliably assess given our skillset and so some of those benchmarks that I mentioned about the applicant's ability to communicate ideas and the grammatical execution, those were things that we could agree upon as a team. As far as the timing, so typically the pilot consisted of between 85 maybe 100 applicants and so that generated roughly between 300 and 500 applications to our four-year institutions and so we had the capacity to be able to engage with that material more closely. I think again my thoughts are always with with scale how do we take some of this content and what other information can high schools provide to us so that we could use something like the rubric with a little bit more consistency to say if a student is outstanding in this particular measure this is generally what this indicates for us as far as student success is concerned. So I think there's a little bit of work and development and opportunity there. Great, all right so just kind of with Linda, I think I kind of lied to you I am going to go ahead and ask one more question. So you touched on the equity issue at the beginning of the conversation and just a little bit now in our conversation, but how do you and I know that Michelle will talk about this as well but but how has using performance assessment really addressed some of the equity challenges CUNY was facing when you guys began this pilot really five years ago? Yeah you know I think a large piece that could be a separate and very lengthy conversation that I initially touched upon was the use of cuts to determine access to college level courses. As we were doing this work with our consortium schools the university was was also examining how to change some of our practices around placement into college level courses and I think that this pilot just complemented that so nicely. I mean previously many of our students would need to take a standardized exam in order to prove that they could be successful in a college level course and this type of work has supported the idea that in fact it's the work that you're doing in high school that serves as your strongest predictor of post-secondary success and not to say that the performance assessment was the only thing that we were doing but but I think there were a lot of projects university-wide that supported this move to looking at student achievement in high school a little bit more closely and taking it a little bit more seriously than perhaps we had. Great thank you so much Joanne I really appreciate your perspective and it really almost ends where Linda ended around the role of holistic admissions and how do you integrate this in holistic admissions and much like what University of California may be able to do it's around student work and so I really appreciate those comments and I'm sure we'll have more questions for you at the end of this webinar. So we're going to go ahead and move on to our next panelist Michelle Fine so certainly last but not least at all Michelle comes to us from the Graduate Center at the City University of New York where she and her colleague Karina I should have asked you how to say her last name Priam Ka have designed an evaluation on the impact of this pilot on student outcomes so we're so glad we can amplify their research and Michelle will talk about the outcomes a paper is forthcoming and LPI will be distributing this paper this summer and for now we are fortunate to have Michelle live with us to share the results from the study they're conducting. Hello all I uh I think I can't can you hear me it's saying that I'm here yes we can hear you Michelle the co-host yes I can't hear you yeah though your video is not here I come I believe maybe yeah yep there you are cool hello everybody I'm Michelle Fine I teach at CUNY and I'm delighted to be here thank you Monica Linda all of the staff for making this possible Joanna thank you for making the time and to all of you in the audience I hope you are coping well enough with COVID and the violence in Minnesota and all the ways in which the current moment is affecting our our lives but I think we gathered to imagine a moment of possibility in difficult times and so I'm going to sketch the the origin and impact of the pilot in the CUNY system and I encourage you to look at the report for a more detailed assessment of the data I need to say that our study emerged out of a significant partnership between K through 12 schools the consortium schools the university that is CUNY the high school version was initiated by Anne Cook and Phyllis Tashlick from the New York performance standards consortium the pilot was launched with the bold leadership of interim chancellor Vita Rabinowitz at the city university of New York who was searching for strategies that could braid equity and access um it was launched with the bold leadership of the admissions director Joanna Kurcharski and then this assessment comes from a collaboration between myself and Karina Priyanka who's an advanced doctoral student in psychology at the Graduate Center and she's also a faculty member at Lehman College our final report draws from data from the New York City Department of Education as well as CUNY's databases and I encourage you to look at the data on the 38 consortium schools because you will see from those data that young people come into those schools as Linda said often at significant disadvantage high housing precarity um larger than city-wide rates of English language learners lower socioeconomic status um highly diverse racially and ethnically they come in slightly below the average for the city in terms of reading and math scores and yet they graduate at much higher rates of college readiness and gaps between race and class narrow within the consortium schools let me also say before we even get into the data that as uh folks have indicated the consortium schools are organized around a performance assessment system so this isn't a technique added at the end for college access as Linda indicated these young people are growing up in schools that value inquiry depth teacher professionalism and then at the end of their PBATs they have to present their work to an external examining committee of scientists people like myself university professors so it's it's a culture immersed in student inquiry multicultural responsive curriculum teacher generated um PBATs and external evaluation the pilot was born because as Joanna indicated there was a moment after 2008 when some of the educators in the consortium schools particularly the counselors noticed that young people who used to get into the CUNY four-year schools and were getting into private universities that were SAT optional were no longer getting into those four-year schools and were being um invited to attend community colleges this was particularly true for black and um and Latino low-income students and a prominent community-based organization policy organization in New York the community service society launched a pretty substantial public relations campaign to call CUNY to account for why black and Latino New York City graduates were decreasingly gaining access to the four-year colleges CUNY decided to build a pilot the consortium was involved I and Karina at the public science project to CUNY and the pilot was born what we did with the pilot was as Joanna indicated we uh created a process whereby graduates from consortium schools were doubly vetted they were vetted at the consortium schools and then by CUNY assessing their performance-based assessment tasks teacher recommendations their GPA and in the pilot they were permitted to come in below the SAT cutoff which at CUNY at the time in the four-year colleges was 500 on language arts and mathematics so we had been systematically tracking these young people over time we now have three cohorts and I'm going to give you some preliminary data uh just want to make sure my time is good preliminary data in tracking the students as you can see on this slide we had two simple research questions how do students who are educated in the consortium with performance assessments fair over time in terms of college persistence uh GPA and credits accumulated in general and then where possible we wanted to disaggregate the data by race and ethnicity and we were comparing them to CUNY students in general first time first year full-time BA students secondly we were interested in particular in those students in the pilot who scored under as uh the 500 uh on the SATs how they fared over times same outcomes college persistence GPAs and credits accumulated I'm going to give you an overview of the finding and then we can do a deeper look assuming we have time um I will say we have been delighted at the at the research and um let me just say that in the beginning the university agreed to do the pilot in exchange for the fact that we agreed to do a transparent documentation of the impact can we get these young people into CUNY and how do they fare so this was deeply committed to a notion of accountability that the consortium schools were more than willing to undertake what we have found to date with three cohorts of students is that first year full-time students pursuing a BA in the CUNY pilot that is they came in with below cut off on the SATs have a higher rate of persistence after one year compared to their peers from New York City public schools secondly is that a higher percentage of these students earn 80 percent or more of attempted first semester credits and third is that the students in the pilot have a higher grade point average than their peers so on those three indicators of academic performance persistence credit accumulation in GPA the pilot students were outperforming their peers when we looked at the larger sample of consortium students which included pilot and non-pilot we did a disaggregated analysis particularly for black males and we found that there was what we're calling a kind of consortium boost black males who are pursuing the BA at at CUNY who come from consortium schools have a much higher rate of persistence after one year and have higher GPA than their peers in CUNY so this will give you a sense of the cohorts the first year was a funky year happy to answer questions we were late the materials weren't clear so of the 52 who applied we ended up with 15 the numbers have increased over time number applied number admitted and the percentage who end up coming to CUNY I think this year we're up to 127 who have applied and 84 percent have accepted we don't know yield yet this is the first year full-time students pursuing the BA in the pilot again the pilot are the young people who scored below the SAT cutoff doubly vetted in their high schools and at CUNY and you see after one year 90 almost 94 percent are coming back higher percentage of these students are earning more than 80 percent of their attempted first semester credits again if you just look at the pilot line it's almost 89 percent and if you look at the GPA again you see students in the pilot have a slightly higher than average GPA for their first year I need to tell you also that the students are coming in with below SAT 500 cutoff they're also more likely to black be black and Latinx than the CUNY general population is they're less likely to be white and Asian and they're more likely to be coming from the Bronx so you're getting a more diverse community of students who are achieving at a rate at the level of their peers if not surpassing it if we go to the full-time black males you'll see that for the consortium 90 percent of black males are retained after one year compared to 78 percent of the New York City public schools specialized high schools are the highly selective high schools and the consortium african-american males are still out performing those young people same thing with GPA the implications for equity access and persistence speak to a number of simple findings one is that our early results reveal encouraging patterns of performance assessments in terms of equity access credit accumulation GPA and persistence the early evidence on race ethnicity equity is promising but as we've said our sample size are small we're about to get a new data dump about which we're quite excited but the pilot offers extremely encouraging evidence that college admission policies rooted in performance assessments can strengthen equitable college admissions achievement persistence and eventually we predict graduation and both the statistical evidence and the interviews we conducted with administrators suggest that even large public universities are beginning to recognize the need and desire to develop the means for more open or holistic admission processes opening up our universities to more diverse student communities through a multi metric framework and I'll end on this note as private universities more than 1200 move toward test optional admissions one might ask is it not the responsibility of public and private universities to develop admission policies that widen access strength and equity and deepen the creative intellectual development of our students at the moment because of COVID and the bold moves of the University of California we we have access to a natural experiment on equity access and higher education it has been born from a public health crisis it gives us an opportunity to move toward an ethical authentic and deeply accessible admissions process Monica you're on thank you so much Michelle and thank you for the profound kind of charges to us as as educators as higher ed and k-12 leaders in this world um we're so excited about the valuation um and that and that CUNY had the wherewithal to really design a pilot alongside the pilot or a study alongside the pilot um and we're especially excited about the finding showing the promise of performance and admissions to higher education and we're also really excited about what it says around equity um and and the holistic admissions a few folks have been asking if this report will be published and yes it will be published it will probably come out in July and uh we will have a summary that's available around the report after this webinar will be emailing it to folks so just want to reassure everyone about that right away before we move on to Linda, Darlene Hammond to wrap up this webinar for us um we're seeing a lot of questions and so Michelle I'm just going to go forward so we can get to everyone's questions after Linda wraps up um there's a lot of questions right now around performance assessments um and as well as scaling um and so we'll move on to Linda wrapping up our webinar and then we'll move into questions and answers with everyone and I appreciate all of you sticking with us as we now are at the top of the hour and we'll go on for about 15 more minutes. Linda I'm going to turn this over to you. Well I want to thank you Michelle for the fabulous research, the fabulous report and the great um presentation of the findings. I note in the chat box if you haven't been able to look while you were talking that Young Wan Choi has written you a note um uh you are as inspiring as ever 21 years ago as a graduate student who just speak in New York City. I love that because Young Wan is now in Oakland leading the performance assessment work in in that town which requires a capstone project of all of their seniors and is part of the California Performance Assessment Collaborative which really learned how to be a collaborative from the New York Performance Standards Consortium so it's an interesting cross country connection uh that I love in so many ways. I just want to point out a one uh thing from the wonderful data that Michelle just presented which is that the persistence rate for the pilot students in CUNY um is 94 percent uh at uh after one year at the beginning of the sophomore year which is an extraordinary uh persistence rate um in a public university for students um of color who are um coming from as Michelle said I mean we also saw this effect for the number of students from high poverty schools uh being much more um pronounced than um for um other students so I think that there's something about these practices that develop what I might think of as academic resilience um you know both the finding that you have about the pilot students about the much higher rates of success of black male students about students from high poverty schools because when you come to college the question of do I belong can I do this work how do I do this work uh is a very different question if you've had the experience of doing this kind of work uh throughout high school if you've had the opportunity to develop serious research and uh investigations and refine your work in response to standards and defend your work and learn that you can in fact meet the standards and continue to improve your own thinking and your own analysis uh the academic confidence and resilience that that provides I think is part of the real storyline here and that's my that's my last word well Linda yeah thank you so much um well I appreciate all of the panelists and Linda Joanna and shell um for sharing this with everybody and we are going to move to questions and answers now and um and like I said we've been getting lots of great questions we've received some interesting ones we'll have time to talk about a few of those some folks gave me the name and affiliation so hopefully I can attribute the um questions to the proper person um I'm going to start off a little bit with the performance assessment piece um Linda there's been very related questions around um kind of the timing of performance assessment the um kind of objectivity that is used in performance assessment um one person asked do parents help the students and therefore give them a leg up so to speak um so maybe if you can talk a little bit about how performance assessment is embedded um in the classrooms and I think some folks are seeing this is kind of very traditional end of the course end of the year one person asked if it's um like the SAT and ACT is it offered during certain times of the year so maybe if you could talk a little bit more about performance assessment but we certainly have questions for Joanna and Michelle as well yeah I was going to invite Michelle to offer some comments on this too because she has been tracking this process for many years as well in terms of what goes on in the schools um and of course this is really embedded in everything the students are doing that you know their work is um continually organized around investigation research writing and communicating to a standard you know revising their work and so that's just the way that education is done in the consortium schools and in many other schools that are involved in this kind of deep approach to um both project-based inquiry-based learning and performance assessment and so uh it's um not something that happens at testing time it's not something that happens at the end of the course it's not something that happens only in senior year it's a process of learning to think and inquire and evaluate and you know apply evidence over a long period of time uh you know the question about whether you know and particularly in the families that we're talking about in this performance assessment consortium you know a lot of uh families are um you know working two jobs are you know not didn't graduate from high school themselves um are not you know going to be there doing the students work for them but I do think that there are other contexts within which we do this kind of work goes on in school uh where uh it is quite possible to frame the work such that it is the students work and it is their ability to take the feedback and revise the work and meet the standards that is being assessed. Michelle do you want to add anything on that score? Am I good yeah um two quick thoughts um this tends not to be the crowd who has the over-involved parents who's writing the paper for them that's us that's not who's attending these schools to the most part but I have sat on at a performance assessment roundtables where there is presentation and there have been parents on the roundtable so the young people have to present their work you know a 25 page paper comparing two genres of literature or a moment in history with um where an ethical debate was being undertaken or science experiment they have to write it they presented to the reviewers who are teachers outside like academics or scientists or business or politicians that I've seen parents on that and then we interrogate and I have been on beautiful um uh panels where you can't believe the work is so gorgeous and I've been on panels where we say linda you're gonna be great but your book's not quite ready or the text isn't ready or your science experiment needs a little help and nobody runs out of the room saying oh my god I failed they take the feedback they revise and and and they re-defend it's a it's a beautiful community of feedback revision intellectual growth and when Laurie Chaget I'm happy to see the citations in the report when Laurie Chaget followed consortium students into college she found not only higher persistence rates but three big distinctions one is they know how to write a paper they were so surprised that their um their roommates were flipped out about writing a paper second when they hit a bump in the road they find someone to help them they trust adults and a lot of kids just as Linda said who feel like maybe I don't belong to college anyway they hit a bump in the road and they exit these kids find somebody at the college or somebody at their high school and the third thing is that they know how to revise they know how to take feedback and revise which feels to me like a lost skill of the 20th century um so I think the culture builds that I like that you call it academic resilience um because I think they learn how to navigate and navigate with an idea toward inquiry and their own intellectual signature not just compliance and taking notes that's great I'm gonna um do a follow-up question really that builds on both of what you and Linda asked uh we're talking about and and Jonah it's it's back to you and really one of the things I heard you say when you were looking at the performance assessment is you really could see some of those critical thinking skills and problem solving skills when you looked at student work and so one of the questions that came in was and again it's about kind of objectivity and subjectivity but one of the audience members asked do you worry about the subjectivity of of certain readers you know in terms of your admissions staff and and how do you kind of um calibrate for that um as an admissions officer certainly done you know within these schools that use performance assessment visit an admissions person how do you kind of calibrate the objectivity of what you guys are looking at and integrating that into your decision yeah so so for for this pilot we had um two admissions readers take a look at the performance assessment of each individual um and then we would compare to see whether or not we shared an our assessment of of the work um and so I think that helped us internally be a little bit more objective in our final decision um you know again there is um I think I wanted to second that you know what Michelle and Linda had said that the work that we're seeing um it's quite evident that this is an ongoing project that students really put a lot of effort into and so so while there are different disciplines that are being covered and different materials that we're seeing throughout our assessment um you know unanimously I think we all agreed that that we are seeing work that has uh been thoughtfully compiled together and so um and so that's very important to us as well great and then go ahead and finish yours just one other follow-up question to you Joanna was um what's your advice to admissions offices or the enrollment you know managers and deans um if they were to you know move into this kind of direction use performance assessment as part of their holistic review what would you say are some of the first things they might need to do I I think initially just get some familiarity with performance assessments um to understand how the work looks um and that work doesn't always mean that it's an essay so similar to Michelle's I've I've also sat on a couple of round tables in the past couple of years um and that has helped me and a couple of my staff understand the type of work that happens with performance assessments but but I think um number one understand what it is that you're measuring figure out how to measure it um and then see if you can work with the institutions to to set up some type of a standard I think uh the the key to making this work and to making this work at scale is to work collaboratively with the institutions figure out what is being graded what is being presented with some of these performance assessments so that we have a meaningful way to track some of this information to see if there are any particular indicators that are pointing to student persistence and student success um you know to to that end we we still have a long ways to go we we implemented this pilot in 2015 it really took off in 2016 Michelle um shared some of those results we are now starting to incorporate some of those results and and thinking about the the next steps of streamlining some of the work that we've done that's great um can I just add to that point um you know one of the things that Joanna mentioned is that it's very important for the work to arrive with the rubrics so that you know what it has been evaluated on and uh obviously you want to train people to be able to read that work but uh in many cases it's also a fun show can you trust the scores that come in with the work that have already been scored they're using the rubrics in the case of the New York Performance Standards Consortium there is moderated scoring that goes on across schools during the summertime where people come together and score work as they would if you were an AP score on the AP essay elements if you were working in the international baccalaureate you would learn and get calibrated in your scoring to a common standard that is true in a number of schools that have developed very thoughtful well-developed portfolio systems I think of um Envision Education I think of the link learning schools that are working on this I think about the work that's going on in Oakland that I mentioned earlier uh and a number of other places where um there's a system there's a set of well-defined rubrics there's a calibration process for staff and what universities would want to know is that that system has gone on what is the assessment system how has it been scored we accept the scores on the AP including their capstone and inquiry courses that are all projects and performance tasks we accept the scores on the international baccalaureate which is primarily papers and projects scored by teachers because we know what goes into it we know what the process is so as the this gets developed being able to have a high confidence process and a knowledge base about what the competencies are that are being evaluated how they're being evaluated what the evaluations can allow people both to accept the work but also just to accept the judgments that have been made if the process meets a high standard of rigor so I think there are a lot of ways to think about it the mastery transcripts consortium has a way that you can see the work that students have done associated with the competencies they're trying to measure how the how the students did on those competencies they're you know courses and and grades and and so on but also the standards and the pieces of work related to those so I think as Joanne has said we need to develop systems that convey the work in a usable and understandable form with a level of assurance about what is behind it as well as then helping people in admissions figure out how to use it efficiently as well as effectively and I think we now have a study of the predictive validity of these assessments so that questions about subjectivity and objectivity get kind of turned on their head now that we can demonstrate that young people brought in and and we're just growing the samples so we'll have more and more data for you but that young people come in with a complex thoughtful process but then even using just quantitative measures whether it's GPA, accredited accumulation or persistence we can see that they are faring as well if not better than their peers I know you want to go we got to go thank you. No you don't have to go quite yet but I do want to thank you but also Michelle also thank you again for ending on that profound note around the predictive validity and and the potential and so I just you know we have a couple more minutes but I can't thank the three of you for the valuable information the questions that are coming in around performance assessment the use of such in college admissions at CUNY to support Equal Access and Success and and Joanna again thank you for sharing your lessons learned and some of that will be captured in the paper that we are publishing and we do really look at CUNY as as our North Star we have questions around people asking about how does the University of California integrate some of this work and so I'm sorry we won't be able to get to all of them but I think what this webinar shows is we're just really starting to generate some evidence about the potential use of using high quality performance assessment in admissions thanks to CUNY's and thanks to the Graduate Center's study it's the right time for higher education to recognize and value performance assessment in high school we've had a lot of questions about what's the impact of COVID well again this is the right time as so many schools are starting to integrate test optional and holistic but we believe that performance assessment will help colleges and universities more equivocally and holistically review their candidates specific to a broad set of competencies and alignment to their mission and to ensure students can succeed at their campus I'm going to as our next webinar will actually take off from this it's part of a series reimagining college access and success it will be on June 23rd is going to be called performance assessment and college admissions how students show what they know and can do it is purposely designed to take off where this one ended our panelists will include Scott Anderson senior director and board at Common App a major partner with RCA Judy Purdy director of admissions for Wheaton College who piloted the use of performance assessments this past year and Stu Schmill dean of admissions and student financial aid services for MIT who's been using performance assessment for a few years now the recording of this webinar as well as a webinar summary and all of the resources we've shared today will be sent out to everyone via email in a few days we'll also be releasing the study this summer and we'll also send that out via email please feel free to share any of the resources with your colleagues and finally I'd like to mention that a survey will appear in your window when you leave this webinar and we appreciate your feedback I know we went a little bit over time thank you for taking the time to join us today thank you for your interest thank you for your questions and most of all your commitment to reimagining college access and success we hope you'll have a wonderful day and please let us know how we can support your work thank you for all of the participants bye bye