 Well, welcome everyone. It's so wonderful to see all of your faces. This is one of those times that I really love Zoom when I get to see our whole community together on one screen. So as most of you here know, that our program is a fairly intense two years. And this is always one of the, you know, if not the highlight of every year is to see all of the members of a cohort present the work that's come to fruition over the course of these two years. Two years is a, you know, a lot can happen in two years, as we know, but also two years can go by very, very quickly. And we ask a lot of our students in that period of time, our students come in with an idea or two or three or more and then very quickly we ask them to focus in on a thesis topic and then do the research and then do the writing. And, you know, that happens at a real clip. And I'm especially happy to be here for this cohort because last year was the first time that I taught the second semester course in which students are asked to first formulate a thesis proposal. And so it's really exciting to be able to see where things have come to from that point, that early point in this process. The way that we're gonna do this today is that we will, I think most of you probably have the schedule. We're gonna have three presentations in the morning session, then we'll have a break and four more presentations in the afternoon. For each student that's presenting, I'm gonna call on their thesis advisor to do a short intro and then I'll just try to keep things going as we move along. So our first student up this morning is Roya Musapur and I would like to ask Justin to get us started. Welcome everyone, really wonderful to be here. Can everybody hear me? Yep, yep, terrific. Yeah, Roya has been just such a crucial and fabulous part of our team for the past two years and we're incredibly grateful for all of her contributions. I know that there are lots of folks from the teaching systems lab here today who are really excited to hear more about her thesis which is born, starts from a personal experience that kind of got underneath Roya's skin and she really demonstrates in a powerful way what some intensive research to try to understand things in the media ecosystem that feel problematic to us can look like after deep study. So really proud of Roya's work and excited to hear it in its newest form. All right, so I'm gonna pass it to Roya. The title of Roya's thesis is caching in on student data, standardized testing and predatory college marketing in the United States. All right, Roya. Thank you. I'm just gonna take one second. I'll apologize first off because I have a couple of screens here. So if you see me look over to one of them, don't be alarmed. Coming up okay? Great. Well, good morning everyone. Thank you to Vivek and Shannon and Justin for the intro and for setting this all up. I think it's been a wild year to be doing all of this research and I think I speak for all of my classmates and saying thanks for supporting us through all of this while we've been scattered across the country. So today I wanna introduce you to some core aspects of my thesis research. So I'll set this up with a bit of a personal story to tell you why I'm motivated to study the standardized testing and predatory marketing that happens around the college admissions process. So 10 or so years ago, I was a high school junior starting my college process. And as the oldest of three, I was the first to go through the process and I had really quite excellent access to college counseling at my high school. But I felt like I had a lot to learn and I know that my parents are on right now and I'm sure that they would probably agree with that. So when I took the SAT for the first time, I realized I was eligible to sign up for something called the student search service, which is a college board service that provides students with personalized college recommendations and all kinds of outreach from universities. So I really eagerly signed up. I was excited to get mailers and emails from universities that thought I should apply. So I wanna kind of take you back in time and show you what that looked like on one of the first days I started receiving emails from universities. So here's my inbox as a high school junior. I would say it's pretty typical. I have emails from concert venues, 17 magazine, Groupon, but what really strikes me here is the number of universities that had emailed me on this day. So not long after I started receiving these emails, I actually revoked my consent from the program. I found the emails overwhelmingly annoying and really so generic that they provided me with very little information that was actually aiding my college search. But I continued to receive emails and I even reached out to the college board to try to figure out why. And they actually confirmed that I was not subscribed to any email newsletters at all. Later on that year, they emailed me three separate times to ask if I wanted to rejoin the student search service, which to me was proof that I wasn't actively signed up even though I continued to receive emails. And I ended up actually receiving emails for nearly a year after revoking my permission from participation. So 10 years later, I found myself really puzzled by that fact. Why did I continue receiving emails after revoking my consent? Where was my personal information in this process as a high school or a minor? And how was it being used? So after all of this time, a lot more school and much, much more reading, I think I finally have the right words to describe what this was. So in my thesis, I argue that programs like the college board student search service facilitate something called surveillance capitalism, which is a phrase coined by a Harvard researcher and professor Shoshana Zuboff. I'll explain a little bit more about surveillance capitalism in a couple of minutes, but at the most basic level, surveillance capitalism is the idea that your personal information, traits and behavior can be rendered into data, feeding into predictive products that help large companies provide things like targeted advertising. And then ultimately allows those companies to make predictions about your future behavior and even shape your future behavior. So I argue in my thesis that while we need more transparency around these practices, our ultimate need is greater regulation of companies and organizations. So I see my contribution to academic discourse around surveillance capitalism and more broadly what I'm referring to here is critical data studies. I see this as a key example of surveillance capitalism that I hope to convince you in the course of this talk at least is entirely unethical and the potential implications for such practices. So very briefly, just a quick overview of kind of where I'll take you in the next half an hour or so. I'm gonna take a moment to just situate this thesis within the sort of broader academic discourse. And then I'll introduce you to the privacy framework that I build through my thesis to rely upon to analyze the actions of the college board. Very briefly touch on methodology and then really dive into two key case studies involving the student search service and then talk about how those kind of translate into predatory marketing and the ethical implications for data collection and analysis in this process. Finally, I'll talk about sort of where do we go from here? What are the takeaways? So I see this thesis and this is sort of a question I go back to pretty often within media studies at least. I see this thesis sitting at the intersection of what I'm calling education, critical data studies and law and public policy. So I draw from each field in order to understand the implications for programs like the college board student search service in the broader discourse around data collection, algorithmic decision making and predatory marketing. So these are some examples of who I've been reading for the last couple of years and I wanna bring these up as individuals thinking critically about data studies in various spaces. So on the right you see Justin's book, Failure to Disrupt Within Education. We have people thinking about this within financial institutions, on social media and in many cases the implications for what all of these individuals are studying tend to impact people of color and people impacted by poverty most directly. So I'll take a second. You see on the left surveillance capitalism to introduce you to a little bit more about that. So because my thesis uses a case study around the college board to demonstrate what I'm calling an act of surveillance capitalism impacting K-12 students, I wanna take a moment to outline a critical transition that Shoshana Zuboff outlines in her book on surveillance capitalism. And so I wanna explain what she calls the behavioral value reinvestment cycle which is the beginnings of surveillance capitalism. So in this cycle, and this is a figure from her book on the left, when information is collected from individuals especially in an online setting, it's rendered or turned into data, analyzed and then used to improve upon that surface for that service for an individual. So there's a certain amount of data that's being rendered that doesn't get used and this is at the top of the cycle. And Shoshana Zuboff considers that to be data exhaust or surplus information which is a byproduct of this cycle. At some point companies realized that that very small amount of data exhaust what we're now characterizing as behavioral surplus was actually enormously profitable. So in the second figure on the right we see this original cycle, the behavioral value reinvestment cycle actually on the bottom of this much larger image. So it used to be considered exhaust is now fed into a much larger cycle. This is not intended to benefit the user at all. This entirely is intended to benefit a company. So data previously rendered as extra exhaust is fed into this much larger system that can predict our behavior as users. And that can be extremely profitable. So I think what's important to recognize here is while this does feed into marketing and advertisements we might expect this cycle really goes beyond that. So this new marketplace here on the top left of the figure operates around future behavior. So it's probabilistic in nature and companies that make the best predictions about people's future behavior end up winning in the end. So that way marketing and advertisement can be used to shape people's behavior based on predictions and how people act. This is far from the predictions of AI taking over the worlds that many have made but it speaks to a significant reduction in individual's autonomy over their actions. So what's the ultimate outcome here? Individuals lose their autonomy over daily decision making. And while companies like Google have been doing this for years, I argue that we can and we should do our best to minimize the impacts of this on other aspects of society such as within our K-12 and higher education systems. So now that we have a better idea of the core tenants and impacts of surveillance capitalism I wanna pivot back to the college board and share some of the theoretical basis for my research that has helped me understand my own relationship as a high schooler with the college board and the wider implications for the college board's student and university-facing services. So in order to do so, I built a model of privacy based on the work of two individuals, Helen Nissenbaum and Ari Ezra Waldman. And I use this model of privacy throughout my research to analyze different aspects of behavior from the college board. So here on the left, kind of upper left is Ari Ezra Waldman's privacy as trust. And this model is one in which we recognize that all of our relationships, personal, professional, et cetera, are built on trust. So we build a sense of trust with individuals or organizations to share any kind of personal information, just as we do with doctors or with financial advisors. And in those situations, those individuals are legally required to keep our information private. So this is exactly what Waldman calls privacy as trust. And it's enforced legally. Nissenbaum on the bottom right here, her model of privacy as contextual integrity is one under which the distribution of data can happen under two specific norms. And you have to meet both of those norms for distribution to be okay. And those are called appropriateness and flow. So they help us ask questions such as, is the data being collected in a certain situation appropriate? And then is the transfer of that data from one party to another appropriate? So just as an example, it might be perfectly appropriate for a grocery store to collect information on the types of food people buy in order to restock the proper amounts so that they always have enough in stock. But it would be less appropriate for that same grocery store to ask patrons of the store for a list of all of the restaurants they like within a one mile radius of their home. Similarly, it might be reasonable for that grocery store to share the purchasing habits of patrons with their supplier for restocking purposes, but it wouldn't be appropriate flow or distribution for that supplier to then share that information one step further with patrons health insurance providers. So here I have sort of my sketched out model of privacy that I use throughout my thesis. And I'm arguing for a combined framework of privacy as trust and privacy as contextual integrity. So I'll start on the left in terms of assumptions. So when we analyze this instance of surveillance capitalism within the college process, we have to assume in some sense that data sharing will continue. So this draws directly from Waldman's work. Second, we have to assume that students and their parents and even their teachers and administrators are not always aware of the data sharing that happens and how these data are shared. And this is exactly what happened to me as a student when I was in high school. So next are sorts of requirements for data sharing of some sort are as follows, exactly the norms of appropriateness and flow as defined by Nissenbaum. So we ask again, are the data collected appropriate to a specific context? And then when considering distribution or flow, is that distribution appropriate and how can relationships of trust enable and constrain distribution? So finally, and this has been sort of my biggest question throughout all of this, how do we actually do this? So I say through two primary methods and the first is transparency, which helps minimize any difference of information between a data provider and a data collector. And the second and arguably most important method is through legal enforcement. And this echoes Waldman's work as well. So we need to build in legal mechanisms that actually enforce data privacy legislation in order to minimize potential harm to students. So I wanna briefly highlight the different methodologies I use in this research just to show you a bit of the varied approach that I take. So I rely upon historical research in order to look at the history of standardized testing, normalization of data collection within our school systems, policy analysis to review all kinds of federal and state laws surrounding student data privacy. And then I dive into primary research in order to analyze documents that are associated with the student search service and the college board. And that even includes some of my own documentation from 10 years ago. And then interviews to sort of situate this work contextually within the voices of individuals connected to the college admissions process. So college enrollment consultants, college counselors, admissions officers, et cetera. So now that you have an understanding of what I'm using to analyze the actions of the college board, I'll bring back to my example of the college board student search service to specifically look at two examples of how the college board violates that framework of privacy that I just built. So one example relates the inclusion of transparency in the privacy framework. And the second uses my own experience participating in the student search service as a high schooler to illustrate a violation of the norm of distribution. So as I mentioned before, the student search service is the college board's flagship program to provide students with free personalized college recommendations. When a student participates in a college board exam like the SAT or PSAT, a student can opt into this service and provide all kinds of information about themselves to the college board, such as their address, their ethnicity, gender, financial aid need and educational aspirations and interests. The college board then takes this information, plugs it into its proprietary recommender systems and then provides students with recommendations for college. However, to kind of pivot towards the college facing service, the college board also collects this information and I'll quote here to create the largest most effective admission search database in the United States. So that database, it then markets to universities and other loosely educational organizations through the college board search. And these organizations and universities have the option to lease student names from the college board in a variety of ways, but at the most basic level, it costs colleges about 50 cents per name to gain student information. As written here on the bottom with the college facing example and these are actual quotes that the college board uses to market these two services, whether it's student facing or college facing. The college board is very explicit about marketing the college board search as an enrollment solution to help universities market themselves to students. And what I think is most striking about this quote on the bottom is specifically this focus on marketing return on investment. This is not towards the universities, this is not intended to benefit the students. So we know that about 80% of students using the college board opt into the student search service at some time. Even if we approximate this based on the number of people taking the SAT in one year, this is about 1.8 million students. That's a big data set. But research shows that the student search service doesn't really benefit students at all. So according to an internal college board paper cited by the Wall Street Journal in 2019, a student whose information is sold as part of the student search service is about 0.1% likelier to apply to a university from which they've received marketing and materials of some sort and only about 0.2% likelier to enroll in one of those universities. So who benefits from the student search service? It's purely the college board and universities. The college board brings in about 10 to 11% of the organization's revenue every year from the college board search and associated programs. Universities gain access to large quantities of data, student information for marketing and analytics. Individual students are barely impacted at least in a positive way. So this I argue is exactly what feeds into surveillance capitalism. Large amounts of data are collected and distributed to third parties. In this case, universities and scholarship organizations rendered at scale, these data can be used to make all kinds of predictions about incoming students who will be successful, who might drop out, who might major in what subject, who can pay and who can't. So I wanna take you through my two small case studies to illustrate some of the challenges surrounding the college board student search service and college board search. So this slide kind of demonstrates the kinds of information shared either towards students or for parents about the student search service. So a student considering signing up for the student search service online is told that colleges pay for the service. Online, however, parents are not told that colleges pay for their children's information through the student search service and college board search, unless they navigate to a separate webpage where they're told that universities pay a license fee to participate. This has nothing about the explicit per name basis of the service. So in a test setting, on the other hand, students are not told that colleges pay for their information. And this is not in the proctor's script. So a proctor administering the exam will never say anything in the entire exam that says that there's a cost associated with the service to universities. If and when parents are provided with a consent form for their children to participate, which is not required on the federal level, but certain states or districts might require, they're told the same information as the FAQ online, which is that universities pay some kind of license fee to participate. So this difference in information is very similar to what economists call an information asymmetry. And one that makes it very difficult for individuals to proceed within what we're calling a relationship built on trust between a student and the college board or a parent in the college board. So the college board explicitly holds more information about these programs than students and parents who are most responsible for consenting to the service. So in doing so, the college board has created a reseller's market between students and universities, one which the larger customer, the university, will default as the most important. The information students are provided with in different situations to participate in this market is quite minimal, but the lack of transparency between the college board and students is missing what Shoshana Zuboff in her book on surveillance capitalism refers to as the shadow text. And that's this invisible understanding of what's really happening under the surface. So the college board tells students that the organization is helping them, but the shadow text here tells a very different story that students are helping universities market themselves and make predictions about their incoming classes of students. This is possible simply because of the nature of a reseller's market where the larger customer is more likely to benefit from then the smaller customer. And in this case, the smaller customer isn't really a customer at all and is just a supplier, the students. So my second example is around the norm of flow. And here's a brief snippet of how the college board allows students to opt out of the student search service. So while the college board claims to lease student information, I'm arguing that the organization explicitly violates the norm of flow or distribution outlined earlier by misinforming students and parents about how they distribute information. So as a reminder, the norm of flow circled here on the bottom is one of two norms described by Hellenism bombs privacy is contextual integrity. And this asks whether the actual distribution of data from one party to another, the flow of information, whether that's appropriate. So back to the college board, a student signs up for the student search service just as I did as a high school and likely many of my classmates consenting to the distribution of their data. The college board by claiming to lease student information implies that such data sharing is temporary. If a student were to agree to such distribution of their information, then it would be expected that after a student revokes consent, i.e. opts out of the program, that their student data would be removed. However, the college board while claiming to lease student data is explicitly giving universities data for a cost, which is just selling. So the college board will not redistribute student data after a student opts out, but just as highlighted here in this FAQ on opting out of the program, universities that already have student information can continue to use that information and reach out to students. So to provide a parallel situation, which is especially funny given that it's snowing outside for many of us today, imagine you rent a car on vacation. This is the wishful thinking of the day. You've signed a lease with a rental agency. On the last day of your lease, you're required to return the car. If you keep the car after your lease ends, you're likely going to face pretty large penalties as you violated the terms of your lease. So with the college board, a student revoking consent from the student search service is the end of a lease. If a university maintains access to student information and continues to use it, then by definition, if that is within the scope of allowance according to the college board, then the college board is very explicitly selling rather than leasing student information. And this feeds directly into surveillance capitalism. An individual signs up for a service thinking it will provide one specific thing and unknowingly becomes a part of a completely different process. A student can individually unsubscribe from university's email lists one at a time, which in many cases could be in the 10s, 50s, hundreds of universities reaching out. But that student has absolutely no idea how that university might continue to use their information. And this misdirection is a classic component of surveillance capitalism. Students supply their data for free into an economy that ends up feeding off of their data for a price. So these types of data can then be used in different forms of marketing and analysis, not unlike the emails that I received from universities 10 years ago. And so all of this marketing happens within a field called enrollment management. Enrollment management as an industry exists to help colleges recruit and retain students. So one educational consultant I spoke with shared with me that this extends to a college's image or brand. And maintaining prestige and publicity is very important for colleges, both for nonprofit and for-profit universities. So at many universities, this extends beyond what we might think of as students showing demonstrated interest by attending an information session or participating in a campus visit. This has even turned into measuring student engagement with a university through clicks, marketing campaigns and metrics. So there are many companies that provide these services to universities. And some of them, such as Slate here on the right, provide not only marketing services, but also what was described to me by an admissions officer at a large university as CRM services or customer relationship management. So here, students are customers for universities and universities need to catch their eye. But looking more specifically at the quote I've highlighted here on the right above Slate, many of these companies provide all kinds of analytical services through data dashboards and other systems that allow universities to make predictions about student outcomes and success. So this form of prediction is exactly what many scholars have warned about, taking personal information in order to make predictions about which students will be successful, graduate or be able to pay. Many times this even happens at schools historically known for discrimination. So this is what Shoshana Zuboff refers to as rendition of data into prediction products and then even further behavior modification. Student information is used for predictions that allow universities to market themselves to students and gain customers through appealing to students. It makes us ask the question, do students maintain autonomy in this process and what does this set students up for in the future? So in certain cases, the types of marketing previously mentioned can be helpful and that type of marketing has the ability to bring in populations of students otherwise marginalized from higher education. But in many cases, this marketing can be extremely dangerous. So the for-profit university industry is a discriminatory and predatory industry that preys on students. These schools often target students based on zip codes for recruitment to figure out which students might be eligible for federal loans. Interviews I spoke with shared the focus on recruiting military veterans as well as veterans come with some kind of educational funding through the GI Bill. So these types of universities use large amounts of student data in order to optimize their potential incoming class for profit and programs such as the student search service have the ability to enable their actions. So here's just a little bit of a doodle of where student data might go in this type of a situation. So at the federal level, students are not legal stewards of their own data within school systems until they're 18, but exceptions are made in many cases when organizations are deemed educational in some sense. Organizations like the College Board act as middlemen in these processes, distributing highly personal information to other organizations and never asking for the data back. So this third level of organization or university here is no longer subject to the same level of legal oversight of how data is used and this level acts opaquely and it's really anyone's guess how information might be used and further distributed after it reaches that level. So what we do know is that student data is not used only to recruit diverse applicants, but in many cases it's also used within enrollment management systems for predatory lending practices for use in predictive machine learning models and in targeted advertisement. So organizations such as the College Board misdirect students to share personal information and enable the for-profit university and predatory lending industries feeding into surveillance capitalism involving K-12 students. So where do we go from here? So we know that organizations like the College Board are enabling this often inappropriate distribution of data that really does have the impact, the potential to impact someone's life enormously. So I argue for three primary action items around this. So first, we need to build some kind of transparency of company data policies in order to clearly communicate what companies hope to do with student data. This goes back to the way that organizations like the College Board describe their own actions. When students and parents see different information when trying to learn about services where parents are not provided with readable consent forms or data policies when a student takes something like the SAT within school, there's an information asymmetry created that makes it quite impossible for individuals to make educated decisions about how to act and whether to sort of enter this relationship with an organization. So the College Board has created a reseller's market through the Student Search Service and it provides great benefit to itself and to university customers while providing little to students in the process. Second, there needs to be greater oversight of organizations collecting student data and enforcement of laws and legally binding agreements such as something called the Student Privacy Pledge which is a pledge that lots of educational organizations and companies have signed, including the College Board that says that they don't sell student data. And when an organization signs this type of a pledge, it is actually legally enforceable through the FTC but we've seen very minimal enforcement if any of these types of pledges already. So third, I argue that we need greater privacy protections for students and this really extends to consumer data privacy as well. This is something individuals and companies agree with. Individuals feel better protected knowing their individual information is protected and companies would prefer to comply with federal laws rather than the fractured landscape of individual state privacy laws. So these takeaways are by no means exhaustive but they feed into a greater area of consumer data privacy and lots of individuals and activists fighting to protect consumers data more broadly. So I hope that I've convinced you that programs such as the College Board Student Search Service are unethical and feed into surveillance capitalism surrounding our K-12 students by creating this opportunity for student information to be rendered into predictive products helping universities make predictions about student success to inform their marketing. We need to regulate these programs and others in order to make sure children and parents understand how their information is being used. The College Board over time has become ingrained as this odd not-for-profit acting as a gatekeeper to higher education within our K-12 and higher education areas. And my hope is that by fighting back against surveillance capitalism within our educational spaces we can ensure more equitable access to higher education for students and minimize the impact of predatory practices within the college application process. Thank you. All right, thank you, Roya. We have a little bit of time for questions and so for those of you who are on screen if you can use your hands, your either virtual or actual hands, I will try to keep track of the order that they're coming in. So first, William. Roya, thanks, that was terrific. It's terrific, it's important, it's logical and it's ethical and the legal system I'm afraid is neither of those. You've made a compelling case logically and ethically. The Student Privacy Pledge is an interesting example. If there is indeed violation here, there's a class action suit waiting to happen and this is a relatively well-endowed organization. Why hasn't that happened? Has it happened? What's the legal advice on this? Because I'm sure there's some mighty powerful loopholes that are blocking this. And that I think it is exactly those powerful loopholes that have been the problem. So there have been a couple of cases that have actually been tossed out where people have said, the College Board has sold my information and I would like to sue them because of this. There's a class action right now that's being argued. I think it was filed last spring that we don't know about yet. But the actual issue, and this is I think the most challenging thing about doing this type of work, has been that no one has been able to demonstrate in the legal court actual harm from this very directly because there's no direct kind of tracking that's possible from an individual student into some kind of college system, into some kind of marketing or a for-profit university's predatory lending practices. So courts have thrown these issues out purely on the technicality of demonstrated harm. And this has been, and you could ask Justin this question as well. And he and I have gone back and forth on this many times in terms of how do you sort of beat one of these issues before you see the implications happen? And we've seen the implications of for-profit colleges and we know that they target students based on where they are and we know that zip code has a really high correlation with race. And so it ends up impacting people of color most directly. But it's almost like there needs to be some massive investigative journalism project that is able to actually get someone to say from point A to point B, yes, we have this direct link because we have people who are saying we need better protections for students and we have people saying we have this enormous industry of predatory lending but the actual connection at least in a court has not yet been made is what I would say. And it's something I think about and I'm thinking about as I'm doing all of my revisions now as well. Thank you. I think Andrew was next and then T.L. Well, thanks for that. So I just wanna make sure I understood one thing that you said that there is no benefit basically to schools in terms of application rates and yield on this data that they got. No, that hasn't, that so it has- Yeah, I wanted to understand- Yeah, that's a good thing to clarify. Point zero at one and point zero at two, yeah. Yeah, this is a great thing to clarify. So for an individual student, there's very little impact but for universities it has a huge impact because you basically can scale up while it might for an individual student have a really minimal impact on what ends up happening in terms of their application. When schools are buying hundreds of thousands of names all of a sudden a very little impact for one student over that 100,000, 200,000 or more number of names does scale up. So it is, this is, there's actually a chapter in Kathy O'Neill's weapons of math destruction that discusses this exactly from the perspective of yield and of the number of applicants to university. And I think this all sort of started in the 70s and 80s when U.S. news and world reports started reporting college rankings and this sort of directly fed into that at the very initial outset. Interesting. And then real quick, a second question. What's the history of selling data? Cause I know the college board's been around since early 1900s I think and it was set up as a way to basically create the SAT to standardize testing for admissions offices. But there must have been a presentation that somebody gave at some point saying, oh, I've got this great idea. We should use it. Yeah. So this started in 1972 under, I think it was called the student data questionnaire at the time. It was a little bit different. But what's really interesting to track sort of in the history of this is that while the college board has been at least somewhat consistent about leasing student data, media reporting on this service has always used the word sell. There has never, there's never been a point where someone had said, yeah, they're leasing student data and we totally agree with them. But it has always been since the absolute very beginning, it's been the college board selling student data, selling student data over and over again. Right, thanks. I think TL was next if TL still has a question. Well, I guess maybe I'll just, I don't hopefully communicate. It was really interesting presentation and I love the kind of close read and analysis of the system and how the data is flowing. I guess my question might have actually been slightly answered in piggybacking on Andrews because it felt like there was an implied model of human agency running through the talk. At the same time, what I felt was some undermining of the actual efficacy of the predictive system. And I think this is one of the interesting critiques that's emerging is it's a little bit, sometimes a hype and a promise of prediction that doesn't actually pay off. I think this is one of the things we're seeing around the critiques of predictive policing where actually there's lots of ways you can poke holes in the kind of impact and efficacy of those systems. So I don't know if I have a way to ask this quite right. I mean, you might have addressed it with Andrews. If you have a sense of where my question was heading and I'd be happy to hear more thoughts, but I do wonder sometimes these systems a little bit promise more than they actually can pay off. So anyway, I'll leave it at that. Yeah, so I think what sort of strikes me about this in terms of where the predictive models in a sense are coming in is that they're happening at sort of all different levels. The College Board has their own system. The College Board ends up maintaining all of this information and they have an entire research organization within the organization itself that does all kinds of analysis with this data. But I think what's challenging is every single university that has data in this regard is using it in a totally different way and has a totally different system. One of the universities that I spoke with, I spoke with a researcher who said, we know for us that student success correlates really, really strongly with SAT scores. While other universities, I went to Bowdoin. Bowdoin said a very long time ago, we're no longer gonna require test scores because we don't think that they really have an impact. And at both of those universities, you have highly successful students, highly intelligent students. And in one system, they've decided that their predictions are that the SAT scores really matter and in others, they've decided that it doesn't really matter at all. So it's, I don't know if this entirely answers your question but it's very fractured. And I think sort of, I noticed in the chat, Justin has been doing a great job of moderating the chat for me. Thank you, Justin. But one of my colleagues at TSL at the lab asked, should we just be calling for the abolishment of the College Board? And there are sort of two different ways that this could happen. I personally, we've seen over the last year how many schools are just totally fine without asking for SAT scores. And this year has really been a reckoning for that. I think MIT just announced the other day that they're not gonna be asking for SAT scores again. So it is this odd time of being able to say, maybe we should just start getting rid of this organization but even then there are plenty of other places that have gotten in trouble before that have been collecting student data in other ways that have gotten, they've run into issues legally surveying students for personal information and they're still selling data to universities. So it's sort of, the models are all spread out but the data is coming in from very, very many places that are sort of doing it under the books in a sense. Can I just say, I appreciate that answer and I think it points to part of what I'm trying to think through is, there's a long history of the sociology of education and educators themselves critiquing systems. What I'm wondering is if the theoretical models that come with something like Zuboff's surveillance capitalism are perhaps a more, a bigger and a kind of unwieldy, more of an unwieldy hammer than we actually need to make the critiques we wanna make because there are really interesting theories of agency packaged in. And I love Zuboff's especially her first book but I mean, so anyway, so thank you for that answer because it begins to get at some of what I think I was trying a little bit softly pointing to. All right, Eric, and then we will be moving on to our next presentation. My question actually builds on the some of the stuff you just talked about there at the end which was the past year. Yeah, there's been a lot of disruption that most schools haven't required any standardized exams over the last year. And there's some data starting to emerge on that in terms of what it's done for diversity. And it's a little bit hard to tease out exactly because there's also pandemic impacts on that as well in terms of who's applied to where. But it's been a massive experiment. Do you sort of see that as a disruption to the entire system or is it gonna sort of settle back down to equilibrium? I would love to see it as a disruption to the system. I don't think we're far enough out yet to see how it might impact. I've traced a little bit, not in the scope of this work but just in my own reading trying to follow what's happening this year. And I read an article a few weeks ago that specifically talked about how we've seen application numbers jump enormously at all these sort of elite universities that are not requiring test scores. But what we've also seen is numbers drop terribly at community colleges and all kinds of sort of other less not the non four year universities which I think are still extremely important parts of our educational system and trade schools and such. So I think it is sort of a reckoning point and we're not yet far enough to see how it goes but I would be really saddened to see, oh well now that we don't have SAT scores let's say hypothetically, we have everyone applying for this set of schools they all get that much more competitive. And then we see all these universities which are catering to non-traditional students which are catering to sort of students who need who are gonna jump from an associates to a four year in the way that I think like the UC system has done really well over the years. I would be really sad to see those sort of dwindle because of all of a sudden this focus on elite schools without test scores. So it really could go either way. Right, thank you so much, Roya. Yeah, thank you.