 Thank you very much. I'm in a, you know, I wear the jacket for the obligatory jacket, but it's warm in here so and I never teach, you know, my native habitat is a t-shirt and jeans, so this is a So I've worn the jacket, you saw the jacket. Tell my wife I wore the jacket. So the first thing I have to do is, you know, in this paper You'll see I have a number of co-authors on this paper So far they actually haven't seen the results on this paper So this is very preliminary in terms of what we're doing here The experiment itself is one that we actually have done quite a bit as Sue mentioned It's one that actually was one of the kind of rare instances in my career where, you know Somebody in government really said look There's something we can learn from this and really think about trying to apply the principles here And in a larger area One of the things that I recommend to doctoral students is oftentimes that you when you try to do one project You know try to hit the home run, but as you're trying to hit the home run Embed some other projects in there in terms of your data collection So you have some options and one of the things we were really interested in understanding was the role of information and Especially when we think about a high school student What kind of information do they want when they're thinking about whether they go to college and was there a way for us to Provide that and to provide that in a very customized way that might help those students So we're going to come to that So so the project you know does early information improved college access by early information here What we're trying to get at is Information in the earlier part of their high school careers as opposed to information in their senior year Now in terms of so the first place I'm going to start with it's a nice little quote and the quote is a high school senior anywhere in this great Land of ours can apply to any college or any university in any of the 50 states and not be turned away because his family is poor Who said oh come on? You had it So for those of you who weren't listening the this was Lyndon Johnson actually in and 1965 and At the time Okay, I Can't argue You know so it's a later on I'll have to tell some stories about Sue and she can tell some stories Let's see how this time in our goes we might spend the last 20 minutes shopping stories here So What I wanted to you know if you think about that quote in 1965 in 1965 and it and as you kind of shoot forward One of the things that we have in the United States is just a persistent gap Between low income families and high income families the bottom 40 percent You know we're somewhere between about a 25 percent gap and while they're going while the poorest Families in the United States are going to school at a higher rate There's still that gap and the gap for in many ways is a puzzle and as you start to think about different reasons Why that gap might exist there's a number of different reasons why you might throw out there One of them is just the difference as an academic preparation. They go to different high schools There's different qualities, you know, it's certainly a viable explanation It's not one that I'm going to go down, but it's certainly part that contributes The second one is just thinking you know these are low income families They have less resources they have and so thinking about the the relationship of aid The two that we're going to talk about today. The first one is information and the idea here on information is does the family have Information you think about what information does information is the core of our expectations As I'm going to talk about it here in a second the core of our kind of models of decision-making Means that we we are going to look at the kind of net benefits and the net costs of any and net benefits of two different things And we might make an investment if the net benefit in that investment is higher than the net benefit of some alternative Investment well, how do we know the net benefits? It's we think it's through information And so one of the things we want to try to help students understand is both the benefits as well as the cost and some information about those things We'll come back and we're going to talk a lot about that The other one is on complexity and it's going to be kind of one of the things in the background here because in this particular Realm of financial aid, it's very hard to separate the two Oftentimes information is about how to navigate complex processes And the complexity is going to be one of those things that we think is going to deter individuals and if you've taken a class from Sue Assuming that Sue, you know, two to her own horn what she should in in in this So so if we want we can go and I can tell some stories about Kevin soon because Kevin was an undergraduate in one of the classes that I TA'd when I was a Long time ago in a in a macro galaxy long away So the point that I want to make on this complexity is this is a place where there's a lot of active work going on here and When we started this project most of the information was actually most of the kind of policy initiatives We're at the information phase and that's where I think people had really kind of pushed very hard And as time has gone on in part because of the other paper Just in part because of the literature. We're learning more We've realized that perhaps somewhere down here which invites kind of behavioral solutions to help people overcome these small barriers And in particular one of the things if you go and you look at we have a lot of studies most by Sue That suggests that there's this impact of aid But as we look at the impact of aid one of the things that's interesting is the places where aid has had the most impact are also the places where Information has been the most plentiful to students as well as the processes to acquire the aid have actually been the simplest Okay, so so it's hard as we start to think about this relationship of aid policy It's somehow embedded into this in these notions of information and complexity The other one is just more generally as we think about these behavioral notion these nudges There's quite a bit of work right now being done And so if you're looking for kind of a good dissertation This is a good place to start to think about these principles and we'll talk about them now This is a slide. I'm going to use a lot of Sue's work here This is a slide from Sudnarsky and Judas Scott Clayton's paper and one of the things that they tried to do is to try to demonstrate The timing of the process of trying to go to college So many of you remember back, you know some some fall a long time ago when you were thinking about going to college And you started to actually get your stuff together to apply to a college you apply to a college Well, the process if you're a low-income individual for getting that financial aid Comes sometime after your taxes are done. You're going to submit this FAFSA form as many how many people have done a FAFSA How have you enjoyed every minute of it? I did a FAFSA one year, but my my my my father refused to do it in the second year because it was telling big brother too Many things about us I've always appreciated that and in my student loans So what's gonna happen is after that FAFSA comes now It's a little bit more streamlined because of the online nature of it But at the time that we were writing this there was a significant lag between the time that you would submit the FAFSA and the time That the information would come back Now the hard part is if you're a student like think about just economics if I'm gonna go buy a good What good is there that I'd buy where I don't know the actual price of the good until almost six months after I've actually Indicated a willingness to buy the good But that's what higher education largely does we indicate over here in the fall that we want to buy the good or make the investment But we don't know the actual cost of the investment until after all of this processing has happened Oftentimes up until March of our senior year in high school And so one of the things we do right now is we provide information right here to students But that information it's it's it's not always transparent And so one of the things that we you know trying to understand this was we wanted to really say well Where are the informational barriers and how can we try to break those barriers? Now in terms of the FAFSA at the time we did this this was the FAFSA in all of its great detail There's actually a number of good papers talking about the complexity of the FAFSA So you know just kind of reinforcing that there's both informational issues and complexity issues here We have to deal with now I put this in here because I wasn't quite sure You know how many people are economists by nature in the ed school when we when we At Stanford, you know because we're not the best program in fact. We're number five this year Number one in education policy, but number five One of things that happens it is you know almost as always sociologists who come to my talks there And so as soon as I start talking to me capital model it immediately they turn off But the basic idea that I already kind of started to think about is the way that we think that people make decisions is that we think as Economists that people basically weigh two different alternatives and what the benefits are going to be over a lifetime a Lot of times we talk about the monetary benefits and the cost So we think about you know what kind of earnings I could have from this major What it's going to cost me to go to this college versus another college But we also mean the kind of non monetary benefits as well those other benefits that accrue from actually accomplishing a college degree as We're thinking about this the place where we're kind of stuck is When we look at the numbers We think that a lot of these low-income families should be making decisions and we don't understand why and I just want to Illustrate that there are good reasons why they may be choosing to under invest For example, you might have individuals who realize that the cost that they're going to incur in terms of the effort that they're Going to have to do in college is just too high And so they don't want to invest that and that might be a very rational reason why I don't invest in kind of human capital And one reason or another Others might actually have a realistic idea of what the returns to college might be and they might see that for their particular case It might actually be that the returns are too low to actually justify going to college again That's a good reason to actually choose not to go to college bad reasons Have to do with if we start to think about the numbers and we have the numbers wrong So for example in our sample one of the things we asked families was how much would it cost for the tuition at a two-year college? Any ideas what tuition at a two-year college now? We're in 2008 is when we're talking about anybody have any ideas for just tuition no not fees But just how much would tuition be at a two-year college in Ohio or North Carolina? Isaac for annual Okay, thousand fifteen hundred other thoughts Too low too high way too low Your guess is accurate The average family in our sample was guessing ten thousand dollars The truth is three thousand dollars Now if you're thinking about a family who's making a calculation where they're thinking about the cost and the benefits of going to college And they have in mind that college cost ten thousand a year and it doesn't cost ten thousand a year They're handicapping that decision now I would call that a bad reason why an individual might actually make a bad decision in that human capital model and And so one of the things we want to try to sort out is if we can fix some of these places where they might Misestimate the cost or misestimate the benefits then we might be in a position where we could actually improve their decision-making Okay So did you do any follow-up questions? We sort of said say it with three thousand Would you consider that something you could pull off? We didn't We didn't do any questions But as I'll show you in most of what we're gonna actually show them as part of our intervention is that the tuition is 3000 and for most of our sample, you know the Pell Grant and the state grant that they're gonna get would actually cover the tuition So okay, so, you know, I've already made a few of these points There's a few other kind of extra points that I want to try to make about information in this process One of them is in terms of the FAFSA now We're getting to a point that people really recognize the FAFSA at the time that we are really starting this research There was still somewhat of low visibility There's some estimates that about a million students who could have received aid were not actually filing FAFSA's The other one is late information You know this goes back to this kind of idea earlier that we you know We're not actually learning what the price is until very late in the process And then the missed deadlines becomes an important one in the state of Ohio where our research was largely based There's a deadline basically at March 1st if your applications in by that point you're guaranteed aid It comes in after March 1st Basically if there's aid left it's yours, but if it's too late and for a lot of our students We're gonna see that actually some of their applications were going in in July or Well after the deadline of having kind of a guarantee When you look at the research There's a few that are going to be kind of nice causal study I'm going to show you those at the time we started this most of the stuff was very descriptive so for example Sally made's done a series of You know basically surveys of low-income families asking them What do you think about information? And so what you'll see here is the lowest income families are the families who basically say we either agree or Strongly agree that we needed more information about financial aid and Largely those were the minority groups as well who are kind of more likely to be the ones saying that we need that When you ask students if they're if they would have made different choices had they had better information You're getting basically somewhere around 59% of students saying that they would have made different choices had they had information And it was a survey like this survey that kind of led people to kind of say this information Must really matter. We would have had we had that different information. They might have made different choices The other one just to kind of lay out some other places the college boards has two different panels that have convened One was rethinking financial aid and one of their their big Adaptations was the denarski postcard and you know the denarski postcard was basically this idea of using the tax return to generate Information for the family to help project them What their finances would look like in college and help them start to understand what might be available Soon, I just participated in another panel that was very similar thinking about the future and again information played a part in terms of the Recommendations we had at the end of the day though. They're a little bit more nuanced advisory Committee on student financial aid has been playing the kind of information card for some time and some of the early efforts by the Department of Ed was all about trying to improve information As recently as you know a year and a half ago I was on the phone with the White House and the basic charge to us was come up with some ideas to improve financial aid That didn't cost any money and so you know it turns out that providing information is a pretty cheap way to do things And and that's about the only thing you can do without much money There are this isn't the first paper and there's been a number of papers that have started to explore more this information The ones that you might be familiar with which I think are really gaining ground and in particular the The recent paper by Hoxby and Turner's received quite a bit of press is on thinking about this kind of mismatch hypothesis That there's some students that don't have the right information about you know kind of what college They might be able to succeed at and if we provide that information We might actually be able to change decisions and in Hoxby and Turner Basically targeted kind of the low-income top 4% of all students nationally of finding students who were unsupported provided information and a little bit of aid in terms of Fee waivers and they were able to get those students to instead of applying to kind of homegrown schools to really switch their menu Of schools to more selective schools other places the counseling program Boston coach Tom Kane and Chris Avery ran a project for many years out of the Kennedy school where they would go on to kind of the inner city Schools and really try to elicit what the students expected in terms of costs and benefits And they actually made the claim that students basically had a pretty good accurate model of the human capital model But they tended to essentially not only increase the cost of college But also increase the benefits that they might expect There's a few other ones that are kind of interesting. I'm going to come back to this paper later It's one of my favorite papers the student loan paper one of the things in the Netherlands They have a nice student loan program and so they but they had a low take up so they invited students in and they essentially Randomize which students they gave information about that program and then they basically sat back to see if Students made some change their decisions had greater take up in their treatment group Now I'll come back a little bit later and tell you what they actually found But the other thing they did which was really interesting was they also gave a post-test to the students to see if they still remembered the Information and lo and behold the students in the treatment group really understood better the student loan program There's some other ones that Phil areopolis has worked on in Canada again finding this evidence that at least soft evidence that somehow Providing information might change a decision-making It didn't change their behavior Well, I told you I was gonna come back to it later because you know as a preview of upcoming attractions We're gonna have a whole lot of zeros today And this is at least one that suggested that we should have seen that coming now in terms of our experiment Back in 2007 or so H&R block was taking a beating in the press Just because of the way their business model which you know really thrives on kind of high fees and for short-term loans and So H&R block basically asked for a call for proposals about things that they might be able to do To kind of community outreach. It was also their business development group, but that never materialized for One of the things that's interesting about H&R block is they're a pretty good firm for thinking about trying to help students with financial aid They specialize in trying to make complex forms simple It's about three quarters of all welfare recipients go to a place like H&R block to do their taxes So it's a nice venue Most of their clientele actually at the time we were looking had adjusted gross incomes under 40,000 in the year We were interested in 45,000 for a family of two would have got you a Pell Grant And so so they were a good partner in many ways The other thing that was really nice was they were scalable if we could get it to work in H&R block It'd be easy to think about actually putting this in many tech centers And I'll come back at the end and tell you what we're trying to do now So what we basically did was we would come in the individual would come in and do their taxes and our software was kind of embedded in Their tax software and we would look to see if they had an income under 45,000 Well if they had an income under 45,000 then we looked for college somebody who was kind of college age And we defined that fairly liberally being 15 to 30 the 15 and 16 year olds being those sophomores and juniors that We wanted to understand a bit more and then those 17 year old to 30 year olds being those who really you know Might be in a position to go to college next year and we probably focused on three different groups dependents were basically the high school seniors and Our IRB actually forced us to we couldn't actually do the 18 year olds because the students aren't in the office with their parents And so we couldn't do the 18 year olds because we would need their consent So we had to focus on those who were who are kind of young for there They were 17 year olds in the in the spring of their senior year the Independents were those who basically we didn't need their parents information They were either over the age of 24 or they had a child or they were married or they were a veteran or an orphan Anyone of the conditions that would make it to where their parents finances were irrelevant and all that mattered was their own tax Return and then the final group which is really the group. I'm going to try to focus on is this group of high school sophomores and juniors and With these group of high school and juniors sophomores and juniors what we really identified was the parents of these individuals And we provided information to the parents now It was the same model in terms of the dependents But as you'll see we actually have a very different result in the dependents versus the independents or versus these high school sophomores and juniors So just to walk through kind of the experiment so for the high school and seniors in the independents they'd come in They do their taxes. It's about a 45 minute process if you're there Our software would identify and we'd have a few additional questions to gather some information And then we'd have a consent form after the consent form. We didn't implement a randomization Now in our main one, which is the study. We've already published we had three different groups group number one was about fast simplification assistance and information We included information here, but we also tried to simplify that this kind of complex process for them These were kids who are on the who could potentially go to college in that same year And so it made sense for us to try to help them actually file the FAFSA You know, it's kind of the extreme case and you can think about this part being kind of information and assistance We also Provided a second treatment group just with information and I'll tell you exactly what we mean by that in a second and show you an example, and then we had a control group now Each of our block really wanted everybody to feel like they were getting something and so the control group It's you know somewhat unsatisfying if you sign it consent form and we're like, thanks. Goodbye And so we actually provided a little brochure about the benefits of college to everybody So, you know and and that affected all three groups now. I can't tell you the impact of that You know, we tried to make that as general as possible Just basically as something that if you were to just Google benefits from college, you'd probably come up with yourself Because we wanted it intentionally not to have an impact But it was something there that we you know So we're gonna focus in terms of our information more on the cost side because we've already provided information on the benefits in a different way Now just to kind of outline what we mean in that fast for treatment group One of the things we did was we basically said look You've just told us everything in your taxes It's almost two-thirds of the information from the in the fast for come from the tax return And so the first thing we did was we wrote a crosswalk where those fields in their tax return We would just automatically grab that data and put it in a fast for form for them The second thing that happened is this personal assistance where essentially we created this kind of interview then that would go through And ask whatever questions were missing and then the final part here We would actually give them information and I'm going to come back This is the same information that we're going to talk about in terms of what we're doing But the key thing was it was very personalized Both in terms of location as well as in terms of their finances and eligibility And then the final thing that was kind of interesting was the Department of Education as a pilot Allowed us to actually submit the fastest so just like when you hit that button You know I did it at 11 45 on Monday night to send my taxes to Washington We could do the same thing with the fast as soon as we basically hit that form Essentially their fast was automatically transmitted one thing that's interesting is this is actually something the Department of Education It's going to institute in tax year 2014 we've been pushing hard on this part and it's finally going to happen. We believe The information only group, you know, basically we're just going to provide information But we're not going to help them at all with the fast So we're going to basically say look there's this form called a FAFSA and if you fill it out This is the type of benefit you're going to get Now in the group that we're going to focus on for most of what I want to try to do today It was that high school sophomores and juniors and it doesn't make any sense for us to actually fill out their FAFSA for them They can't submit it for another year or two years and the finances, you know in that year or somewhat irrelevant But to the extent that they're related to what the future finances might be they might give us a picture And so all we did with them was really that information But the key thing in that information that we said was we basically said if the student was to go to college this year This is what the financial aid package would look like and now this is what our information looks like So this is a student from Cleveland and it basically says look It appears that you're eligible for in this case the student was eligible for about thirty five hundred dollars in state and federal aid That's a big difference from what for those of you you've had your kind of student aid report Your student aid report typically tells you your EFC how much you can pay And you have to wait until you get that price from the the the financial aid office But what we basically said is look this is how much aid is sitting out there for you And then what we did was we knew where they lived and we chose the four public colleges That were closest to them making sure to have at least some two-year and some four-year options And then we showed what the tuition would look like so the tuition at Cleveland State and they share was almost $8,000 and then there were three community college ranging from thirty three hundred to about twenty nine hundred terms of their cost And we put that right next to how much they'd be eligible in eight For a few students who had lower eligibility we'd also provide a little bit more information about government loans But it wasn't a big part of what we were trying to do this part Here was the big part of our information now One of the hard parts in this part or you know and kind of going forward in some sense What we were trying to do is replicate the information that at the time Policymakers were advocating if we gave students this type of information it might change their decisions And so that's what we were trying to do now Looking through you know kind of five years after the fact as we start to think about this information It's not clear that this is the right information and that's a fair criticism of this project To say well you should have provided different information and this is you know in some sense We made the best choice at the time we had but I want to come back to that because if we find no effect One of the hard parts is do we just provide the wrong information to them? And if we provided the wrong information then of course there wasn't an impact on it Questions up till there. Yeah, please Of the last two or if you're a part-time You know we actually this text box here where we amplified the text covers up this You can see the bottom of the it actually said if you were to go half time This is what you're financial aid and all the awards are just halved and the tuition as well There's another question right here Why were you not allowed to include 18 year old college seniors? The issue that we had was the first year in our pilot year We actually tried to do them and our IRB Wanted their signatures on a piece of paper saying that they were going to put that they Consented to be involved in the research study Well, the problem was the 18 year olds not with the parent doing their taxes How many times did you do your? You know were you sitting with your parents when they were doing their taxes or having that conversation and so what we We tried our first year was we literally had these $50 gift cards that we would send home with the parents And if they sent back the the consent form we would then activate the card And you know we had like five returned the entire season You know it was it was actually very difficult for us to get the 18 year olds It's also one of the places where IRBs were learning at the time You know we are winding up in this is a case where we actually had electronic consent throughout the way And because of the fact that this was a low-risk project to the individual could really only bring them benefits Today an IRB would probably approve us doing an 18 year olds In fact, we did get a waiver for this year when we did 18 year olds But at the time IRBs weren't willing to do that So it's a big omission 18 19 20 year olds are basically just out of the study all together other questions We were and I'll come back to that because one of the things that we're gonna see is we We're gonna have a big impact on both You know filing the form as well as attendance in the FAFSA project not in the information project and a lot of You know it's there's a little bit of selection here, and I'll come back and talk about it But basically most of that comes through the people who filled it out at the moment Um, and basically had everything submitted before they left the office Okay, yeah In the 2015 tax year So the 2014 tax year essentially Right now If you were to go into an hr block office or any tax office today If you wanted to do their fast the FAFSA they'd have to log in and do fast on the web And what the department of education is has claiming that they're going to have ready for next tax season Is an interface where a tax preparer would actually be able to Upload your information to fast on the web and it would just populate fast on the web for you And so that's one of the things they're trying to do is to make it to where it's more seamless with your taxes No, no, no they they they're only what they what they're going to do is actually You have to actually provide the plan of how it's going to help low income families Before you actually get permission to actually, you know make this decision and they'll be auditing those Okay, so the key things that we're going to look at here is we are kind of research question is we want to understand If this information Influences one did they submit the FAFSA and we think of that as kind of a baseline, right? If they really understand that this FAFSA is important Then hopefully if this if this information is kind of piqued their curiosity about their eligibility They're going to get the format Number two we can track college attendance and number three We can actually look at whether they actually receive financial aid Now the college attendance we're going to rely largely on the national student clearinghouse But we also have data from the department of education So if they actually went to a school that's not covered by the national student clearinghouse Yet they receive financially we'll be able to observe them as well So this one here basically is is the universe of all students who are going to attend and actually receive aid Who did their FAFSA? So in any randomized experiment the first thing you worry about is did we get it right? Did we actually have some kind of nice randomization and in our case? We think we did I mean essentially we had an algorithm based on the last two digits the social security numbers The last two digits of social security numbers are uniformly distributed throughout the population And so basically we had the last two digits corresponding to whether they would be in their treatment or control In all of our focus groups. Nobody ever guessed that that's what we actually did And so we think that we we're in a nice position to you know Kind of make it to where the tax professionals couldn't cheat and put people into the other one If indeed we have a good experiment then just a simple t statistic Or in our case we're going to run basically a very simple regression Of our outcome on whether they were in that Complete FAFSA treatment or whether they were in the information treatment And then of course in the high school sophomores and juniors they weren't in that main treatment They weren't eligible for that kind of extra assistance. So it'll just be really Is there a difference by whether they were randomly assigned to this treatment group? Sometimes this is called our kind of intent to treat effect Now the first thing I want to show you is this is a table that h and r block Prepared for us because we actually did not have the data on individuals who did not consent so What you'll see over here in the first column Is the number of people who are software determined might be eligible for the study Now what's a little bit hard is up here the dependence You know put high school seniors this actually is not just high school seniors This is all the sophomores juniors and seniors Okay Now what you'll see is here's only people who were families that were flagged as being eligible The next question we say is well, are you interested in knowing more about college and 15 as in 15 left or 15 as in Okay Okay, let's just have fun Okay, so the so what you'll see here is pretty even and all of these are kind of proportions of the original group Now the place where I want to kind of draw your attention where we did have a problem in this study You know pretty much were pretty good up until here in terms of finishing the office interview But one of the things that happened was h and r block wanted an electronic consent in the process But the irb wanted written consents And so what they would do is they would print the consent at the end of the study Well, the problem we had is h and r block when they print things they print things first the print out that stays in the office Second all the print out that stays home They would do two copies of our consent form one that was supposed to go home One that was supposed to stay in the office with that second batch And so the first month maybe the first two weeks I was pulling I didn't have any hair left But um, I was you know, I was really nervous because essentially we weren't getting consent forms in and we were getting electronic consent And it was because they were sending the consent homes form And so we we changed the experiment a little bit and we started saying look when we receive the consent form That's when we'll kind of activate the money that we were giving for tax professionals who were doing this But the one thing that you'll see here while it doesn't impact in our dependent sample The balance so this value here is a p value on an f test that these ratios are equivalent In our independence. We start to actually lose the randomization that we had There's some groups that we're getting the it can send form back a little bit more And you can see here when you compare this column, you know If you just look at this column here as a ratio with this as the denominator We're losing about 25 of our sample and Now we think we're balanced if we in the sophomores and juniors If you just look at any of the characteristics and then you look at the kind of t statistics on the differences We don't find anything that's significant Um, I would have put stars on it You know because i'm just trying to get more differences on here than the standard errors If we did an f test on kind of the joint significance of or insignificance of this We basically can't reject that basically all these differences are zero, which is what you want to see um, just to give you a couple of of places you in whole in The hispanic population in Ohio just basically doesn't exist So that's why that number's so low the african-american population. It's actually very close to the national average So we're quite overrepresented here in the sample that's going to h and r block offices The other thing that's interesting is the adjusted gross income. It's about 22,000 Um, I actually have data on all the facile filers in 2000 and the the mean in 2000 was actually around $50,000 in Ohio for facile filers. So quite a bit lower It's about 68 percent of the parents say oh, we're definitely interested in knowing more about college for our sophomore or junior Okay, um I wanted to put up here some of the dependent students So this is the high school seniors and the high school seniors here We have the control group the fast food group and the information group. They look pretty balanced as well though There's low samples because we had to focus on the 17 year olds The incomes are in the similar range is what we saw with the other dependents Most of those students are actually current high school students Although some of them had actually finished at the time of it and again overrepresented african-american Relative to the rest of the population By contrast, this is what the independents look like So this is the independents who'd never been to college And you can see their incomes are quite a bit lower because they're not relying on their parents Their ages. They're somewhere in the mid 26s Here's the students who had prior college experience their incomes are a little bit higher This is a returns to schooling. We think some college might matter They're also just a tiny bit older than the other sample Now, um I'm going to just kind of jump right in and show you our first result So this is the first year after they graduated. Did they file a fast fund? So our 10th graders 40 of them in the control group Filed a fast fund The difference is about one percentage point if you're in our treatment group Now for those of you who who have had your regression class and are thinking about your regression class Our standard error is pretty big here. It's 2.5 percent So if you think about, you know, kind of a 95 confidence interval is going to be kind of two times that plus or minus So that's a big number, right? That goes between five percent up or five percent down from what we're at So that could actually be a fairly big range. We could have something as big as four percent or as Or something, you know negative four percent So here's the 11th graders again big standard error very close to zero nothing statistically significant If you pull them it starts to pull down the standard the standard error here because we're Essentially getting much larger sample, but we basically are finding no effect on this likelihood that they complete it Now the two percent still might be a pretty big standard error, right? Because it still could be a three percent effect or lower So a different way to say it is here's our 10th and 11th graders Here's those 12th graders that we did offer the information to Here's independence who had no prior college experience. Here's independence What if we pooled all these controlled for which group they're in and tried to see what it tells us on information overall And when we pool those numbers together you get something 0.003 and now the standard error is actually around one percent So we're looking now at a range somewhere between, you know 1.7 or negative 1.7. It's tighten that range quite a bit But this is just on filing the FAFSA and we're basically finding that providing this information just did not influence the likelihood that families Filed the FAFSA later on We provided the information on returns to everybody and this extra information telling what their benefit might be if they file the FAFSA What what the costs were? No impact whatsoever Of course if it's not going to affect this You can guess what the rest of the presentation is basically going to show you here Here's a college attendance zero zero zero zero zero and a one percent standard error If you look at on the likelihood of them getting financial aid zero zero zero zero zero with the one percent It's the first time I've had like precision in my zero in the actual you know We kind of didn't believe I mean in particular I think Bridget just wanted information to matter we looked through basically, you know things on whether they How much the aid was their take up of the aid The only thing we found a significant difference was students who had participated in our treatment Filed their FAFSA two weeks earlier than the other kids That was the only difference that we found across any of those and of course it's not going to stand up to kind of a a multiple You know a multiple Hypothesis testing But that was the only thing that we found that was anywhere different between those two groups now We said well, let's look at some interesting subsamples So in some of our prior work, we found that students who were very interested in college were more receptive to these programs zero We also found the students who said that work was a barrier to them going to college were more receptive to these programs Zero The families who took a loan immediately for the rebate or did not Zero zero males females black white zero zero zero zero The only sub sample that seemed to have a significant difference Was those who were not very interested in college? And then, you know, it we kind of chalked that up to multiple hypothesis testing videos There's there's no way that that's going to stand up to be something that's significant Um now I wanted to show you what we found before to kind of then try to think about putting these in context So in our prior study, this was the high school seniors The control group 40 percent of them went to college Or excuse me 40 percent of them filed a FAFSA. It was about 16 percentage points higher if they were in our treatment Remember the big difference here was one, they were high school seniors But we were all doing both information and assistance with these individuals The group that we did assistance who were grade 12, we saw them earlier here That was their numbers here in terms of whether or not there was a likelihood on their filing the FAFSA But the group who are grade 12 here that Basically we provided the full treatment with the assistance and the information. It was 16 percentage points higher Their college attendance 34 went to college. It was 8 higher If they received that extra assistance If we use kind of a bigger measure where we use the department of ed data It's about a 10 percentage point or 11 percentage point increase in the likelihood. They're going to college And these are big effects if only 30 percent of them are going these are one third Increase in going to college and the big difference is we're just helping with the FAFSA and giving that that information Um, these are the independent students. So the first column here showing filing the FAFSA You see 16 percent who'd never been to college filed the FAFSA in the control group It was up by 27 percentage points 32 percent in the students with some prior and that was up 20 percentage points in terms of going to college It's about a two percentage point increase in college with this measure a three percentage point increase with this measure The other students is basically zero You know, um In terms of the amount of aid they're getting uh, they're likelihood of attending college This is the result we saw if this is actually looking at persistence If you look at this one down here enrolled in college for two consecutive years You'll see that the eight percentage point effect goes three years after the project But what's interesting is it's actually negative here One of the things that was happening in that FAFSA experiment was it was getting students who might Some students about four percent were coming from students who wouldn't have gone to college and now go to college The other four percent were going from students who would have gone the next year But now go a year earlier And that's what really that that information and assistance was doing But regardless three years later from the time we did the experiment They at least they were much more likely to have actually finished two full years of college um So then the hard part is you started back this out. Well, what's really going on here And so the the question that was asked earlier about was this really about finishing things in the office Now we didn't have a good experiment here like in in hindsight. We wish we would have all we observed was Uh, people could choose whether to finish it in the office or take it home And what we actually find is if we as we look at it the file rate was 82 percent if they finished it in the office And 82 percent because sometimes it'd be kicked back to ask for more information And they may not have completed it if we gave them a piece of paper and they said they would do it on their own It was only about 14 percent So we think that a lot of it is kind of this kind of You know in some sense catching them when they're kind of thinking about it And giving them that little nudge at the right moment with that assistance Now in terms of information, this is kind of the pressing part for us. There's just no impact You know, we've we've got 15 000 people here that we've You know run this experiment on and we don't find anything in terms of information So the question is well, what's the issue here? Well, the first part is Maybe the problem is we just didn't go early enough. Maybe it's not enough to tell us sophomore About the cost of college. Maybe we have to rewind it and go to middle school or maybe elementary school Or you know, I found out today that you know, my my my sister-in-law is expecting You know, I'm going to send her a little recording of me talking about college finance to play for the baby You know, I don't know at what point but I mean, maybe we have to go earlier The second one is we could have the wrong information Maybe it's not really the cost Of college or the amount of aid I get maybe it's more about the experience of college Helping me see that the kind of cost of that study and the hard work might actually Be as high as I think What's interesting is you start to think about it There's kind of most of the studies that are looking at the attendance decision on whether to go to college Basically find that information doesn't matter when you look at it as an example of the kind of netherland study on take-up of that loan The exception of that is in developing countries. There's a nice paper UCLA, uh, what's his name? Rob Hanson Jensen, sorry has a paper from dominican republic where he's basically giving people Information about the returns to kind of secondary school and he finds massive changes in enrollment there But the studies that we have that actually suggests that there's an impact Almost always that impact is happening in terms of where they decide to go and not whether they decide to go Um So uh-huh experience earner it provided both information But they had different treatment arms and the treatment arm that is there was the cheapest and the most effective Was a treatment arm where basically they were providing fee waivers and some coaching on how to use those fee waivers to apply to those colleges So again, it had an element of information, but it also had an element of assistance as well Um, so, you know, I actually really want to see this paper But caroline has decided not to present it at stanford because she's too busy in the east coast presenting it is what she said She lives there, okay Um, so then how do you start to think about it? And this is a place where I think there's an opening for those of you are able thinking about behavioral You know trying to figure out little ways to give nudges to try to help uh individuals And I'm not necessarily saying, you know that this is one of those but for for some reason the psychological cost We don't have a good measure of how tangible these are for students And if we really think the human capital model is the right framework There's got to be cost or benefits kind of either benefits disappearing or costs appearing that we're not capturing in what we're doing Yeah, um the final thing, you know, and I'm going to just kind of show you two more slides here And then let's talk, um, you know the first bullet point just basically is a summary of everything we have We just find zero zero zero zero zero and we actually in the in the end have fairly tight standard airbands So then the question is is this the end of information? Can we just drive a stake in and say information doesn't matter, you know It's information and assistance and I think that uh, there might be a case to be made for that But the one thing about information is it's so cheap And even if we were to find that information had a one percentage point increase in the likelihood of students going to college It would probably pass a cost benefit analysis because it's so cheap um The other one is we're focusing on a very specific sample a sample of students whose parents go to h and r block So as you think about it, you know, there's a lot of other populations out there who might be more sensitive to information And more receptive to it, um than than the setting we had so A few different things just in terms of what we actually have You know when when we think broadly about this kind of overall intervention where we actually provided the assistance and the information You know earlier I mentioned these were some of the different problems we see in the process We when we were actually assisting people with the FAFSA. It was taking us about eight minutes to do so Um, and when you did that assistance and the information these were the benefits you seemed to have They filed their FAFSA. They went to college. They received aid. They were had a higher attainment but the point here Only the information It just doesn't seem to actually be the thing that's actually moving things and and so that's one of the places in the limitation So the final slide just to kind of tell you what we're doing right now Um, the vita sites are the kind of free tax sites and what we uh have funding to do right now Is to try to expand this service into all of the vita sites And so this year we actually ran a pilot in about 144 offices where basically the office would just provide this service For anybody who came in and wanted to complete their FAFSA and we actually put it right in the software Working with the software provider for all the vita sites We've randomized at the office level. So some offices receive this other offices doing There's some problem with there The biggest problem for scalability is the people who run these offices and the people who maintain the software There's no model to get revenue here And so one of the things right now our grants are basically supporting the company who provides the software To maintain that software, but that's the kind of sustainability problem in terms of the long run So we're hoping in 2014 that this will actually be in all the vita sites It'll certainly be embedded in their software We'll still have a randomization happening to try to see if the scale up is leading to Differences in offices in terms of enrollment rates, but that's where we're at in this and if it gets to the vita sites You know, this is actually going to be a big one because that's our way You know, we've got two and a half million kids across a country whose parents do their taxes here at vita sites Yeah, Michigan actually was a is an interesting case for us because the vita site the biggest vita site here is actually a mobile vita site Where they literally it's it's a set of laptops in a car and they basically drive to schools and set up a vita site for a day So it's one of the places we're thinking about the randomization that if we can actually get them to randomize Set of at the office level at the where we park the car for the night That we actually might have a a nicer research project Okay, so that's what I have You know big picture the information just doesn't seem to do it a lot by itself And there seems to be something more to the puzzle and I'm happy to you field your questions and talk a little bit more about information here So probably how is there really disperse the one thing that we can say is that so in the High school sophomores and juniors we don't find any impact on attendance and when we look institution by institution We just don't have enough power to actually identify an effect for individual institution When we do that for the overall what we actually found was uh, sinclair and ohio university We're actually the two biggest recipients and those were places where actually, you know It increased and almost doubled the number of students who were going to those respective schools But in the high school and sophomores program We don't find anything on on which college in terms of it affecting it So the just the information we find nothing in ours in terms of where they go to school When we did assistance and information that was the ohio and Sinclair That's a good idea to do that. I mean we haven't uh We we can do something more nuanced We've actually looked at whether it increased the likelihood Of them going to one of the four schools that we listed on their form and we didn't find anything there But but but that's worthwhile to kind of collapse it by the by, you know, selectivity Although most of these students are not, you know, thinking about highly selective schools Yes In one of your slides you mentioned that perhaps you should go earlier and give me information earlier And so I'm wondering For the the sophomores and juniors, for example, it may be for them that at that moment They're not thinking about college yet. So maybe it's actually and it's actually their parents, right? It's their parents that well, so and so in my mind I'm thinking like I know if my mom would have received that information She wouldn't have known what to do with it. She would have just probably passed it off to me And their junior year I wasn't thinking about college yet But the hard part right is when we provided it their senior year, it didn't change either Yeah, and um, you know a different way to go with it is to go and a different way to kind of couch a lot of the sociological Interventions that are happening, you know, the the kind of operative word is changing the college culture Right and if I try to translate that to what I think it means in economic terms It's a more realistic picture of both the costs and benefits of college But also a more realistic kind of expectation of the social norms as well as the potential individual costs That may be non financial that students might have from going to college And so, you know, in some sense, maybe that's the right information That somehow changing the way that they think about it that might somehow change their perception of the kind of non monetary costs And I think that that's one place where You know, there's there's I wouldn't say there's good experimental evidence But certainly it's a place where a lot of people have explored and seem to suggest that there's a relationship there But we're just not finding it Let's make it it's mark. Yes. Um, have you thought about changing up the delivery method of the information? like if you look at like Dean Coleman Martinez And their experiment in chile and oriapolis and done when they use they use videos Yeah, I mean they thought about like changing up the intervention. You know, we didn't have a lot of We didn't have a lot of well first off remember that we're you know, in some sense We we proceeded all of them like oriapolis was actually designing the videos. Yeah Yeah You are in in our setting. We didn't have that opportunity to do that Um, you know, phil was uh, you know oriapolis and done You know phil was working on this project with us and phil was basically He's one of the co-authors and we were trying to figure out kind of a more creative way But we had the limitation of in terms of both how long they would allow us to give information We tried we were allowed in the 2009 tax season to actually send an email reminder to each family about the information at a later date But they but then basically is the Within two weeks of tax season ending h and r block actually fired everybody affiliated with the project in 2009 And so a lot of that data is just gone Please Is is there any indication whether or is there any way to tease out whether or not? assistance without information That's a great point I mean in retrospect, we wish we would have done both assistance and information information and then assistance But it was one of those things we didn't realize how important the assistance I mean, I think while most of us were optimistic that there'd be something from information Because at the time there was so much rhetoric around information. And so when it there was nothing it was like well We would have loved to see, you know, if we just handed them a form Oh, here's the form you should fill out without really interpreting if it mattered That was one of the things we have We've we are trying to get in the vita sites right now to have some offices where it's just assistance without any explanation The hardest part that we're just having in that one is We haven't been able to negotiate to get that in a set of offices right yet. Yeah Um And using economic thinking, um, I'm wondering about demand for college and before that demand for like grade six through twelve school schooling. Yeah Um, because there are all kinds of indicators That for grade six through twelve the average kid is not too thrilled about school going to school And even in college and this process didn't come levels Even in college there reports like a UCLA honors class Somebody teaching that said the hardest the most challenging part of a of a thesis for these honors students was coming up with A topic they didn't have passionate interest. This sounds like a dissertation problem Person was bringing it up to say there's been a cultivation of Interest passions which fits with so much data. So I'm wondering um When when it gets to information if there might be interactions if there were more going on as I've said really robust middle school Um information about you know, the reason you go to college is to be you to really develop your strengths This goes back to also like cherry surveys, you know a UCLA surveys over time showing this kind of flip in What student the reason students say they'd like to go to college and having it be much more about, you know, Career and job and earnings rather than about, you know Some other more ethereal, you know or more self Exactly and so I don't know exactly the one thing that's that though is interesting is When we did that group where all we did was augment it with the assistance It's stuck I mean one of the worries we had was that they were going to show up in college And they were if if they weren't responsible enough to do a FAFSA that they were going to be gone as soon as college started But they stuck So part of it, you know, so so in some sense I agree with you That there there's something else that that needs to be done But for those some students who we gave that information and the assistance That seemed to be enough to kind of get them to actually really change the course of action And I think what you're doing is is fabulous and especially the automated information and so forth And I kind of think of it all as schooling behavior and education like health behavior and education Anything you can do to influence the the behavior and I that's what makes me think interactions Could be Interesting looking at different types of information when you put them together. Is there even more of an effect? Yeah, I agree I agree. I mean the hard part is you know in some sense the the hardest part with information is When do we stop and say we've got all the right? And when do we stop in terms of in their when in their lifetime to give it and what in terms of what to give them Pretty good to know to say is there something missing given that I think there's a lot indicating there's low demand I agree period. No, I think that's that's worthwhile to go down. Monica, yeah I was going to say a related comment which is low income families Might have like a system of beliefs Which is hard to change just with one shot information Yeah, so maybe like providing the same information in several ways more like in a coaching A manner might have that change in the set beliefs instead one shot That's right. I mean then that's one of the weaknesses in this approach the delivery mechanism going back to the earlier question It's just a one shot thing and maybe that's not the right marketing as well as the right Or to there's a randomized evaluation of Europe. There's a much more intensive in high school You know multi-year lots of support zero so Sorry Yeah We contacted turbo tax and turbo tax basically stuck the problem They didn't have a revenue model here and not only that but because the department of education wouldn't let them electronically transmit It actually generated more costs for them to print the information in an organized way for the individual and so turbo tax In 2008 and 2009 had turbo tax with fast for support and they killed they killed it Now it's it's possible that they return to it once the department of ed makes the change But you know in our conversations with them, they're just like look there's nothing there for us We can't make any money off this. It's not bringing in new clients to us. It's it's not clear what to do Take this off So do you have um anyway measuring what people believed the information that they were taught No, I mean that's what I would have like I would have loved to have gone You know kind of like in the netherlands experiment and actually, you know contact a sub sample to just say Hey, we told you this information Can you give us a ballpark of what the costs are and what your eligibility might be? But no we I mean for us we had one touch with these individuals and that's all that h and r block would allow us Once they left the office Nevermore And for a sub sample of You know for the sub sample whose parents had Prior college experience, you know, we didn't find anything of the information whatsoever The ones where they had a sibling who had previously been to college In the full treatment where we did assistance and information There was an impact there But there wasn't any other impact in terms of the information only groups Please So in ours there we didn't because all we knew was the zip code of residents We probably could have put that in there, but we didn't um one that i'm working on right now In terms of trying to provide information and kind of like a matching type strategy Is what we have is the history of the school For by history, I mean the last five years Everywhere that anybody's attended college a national student clearinghouse match And so what we can do is basically if student is for each student basically Based on their college entrance exam score and their gpa We can say look, you know students from your school have previously attended school x And so we're trying to think about some kind of social psychology experiments about the framing of that information And you know some if you know jeff collin greg walton's work You know they their work suggests that framing of that information might actually lead to a difference in kind of the willingness to believe it And internalize it So, you know, we're going to test the theory and see and see if we get the similar results to what they do Yeah So I mean the hard part here is is You know going back to that slide where if information matters even in the smallest amount it might be cost effective um But the hard part is i'm you know like my personal prior is I've yet to see a place where information matters and you know if we started to try to think about you know Some statistical way of combining these repetitive zeros. We're seeing Um, you know, I might be convinced you know more fully convinced that you know if there's an effect It's so small that it's we're never going to be able to actually measure it precisely Um, having said that Um, you know, I'm also open that there's different populations as well as different types of information You know some of the things in our discussion earlier That that might actually be ways to actually find it. So I don't think it's worth kind of giving up on it Um, but more generally, you know, I started to think a little bit that it's a bit of a You know now i'm going to use some stronger words that it's more of a cop out that it's you know It's a way of saying we have to do something politically. We need to be observed doing something So let's do this thing because it's a low cost venture But yet there's no evidence that it actually does anything and so that's you know when when i'm on those calls That's that's a frustration I have Yeah, I mean the cop out story is one I resonate to you, but I think as a as a Just as a whatever person Um, there's just a big puzzle here. Yeah, because in some sense that We we think I mean if impact people think it's quite costly I mean that should have an impact They say They think it's quite ancient that visual So there's something that there's some Something doesn't square here. That's exactly figuring out what doesn't square. It seems to me I don't know what it has as high social return But it seems like it has a high social science return to figuring out what what What I said explains the puzzle No, I think that's exactly right because I mean there is a It's hard to think that all of these individuals are acting irrationally There is something very tangible in what they're thinking and it's a matter of us trying to figure out Do we have the wrong model for capturing it or do we not understand it and and that's why you know I put up the slide about the psychological cost because I think that that's a good candidate to start looking at that Perhaps the the kind of myopia that individuals have or the willingness to procrastinate Um Actually are sizable costs that might actually in a short run actually drown out some of those cost-benefit analysis One is that this guy might whatever it is Myopia, I mean I resonate to that. Yeah Um, the other one is this like not credit those lack of credibility being made Yeah, in just in the sense of you know, I get all this You know, it seems like now online You know, there's more you have to figure out how to click out of those I mean, you know, it's sort of like every ad I just know it's time trying to sell me something I don't want. Yeah, I'm trying to teach my kids I mean, it's it's like one was to don't believe anything I mean because almost anything And I mean by God you could find somewhere on the internet. I mean, maybe it's a What's happened with information too, you know Uh, whatever Olale is Uh, Jewish Islamic terrorists. I mean, I I literally you could find a site that says that you vote Jewish and Islamic Yeah, I mean, I'm sure you know so but then you don't believe anything so you sort of turn everything I mean, so that's that's a different You know, I'm just picking up on the kind of explanation. Yeah, but then figuring out Is it that people just distrust everything which they should or is it they're procrastinators which they are They sort of no scientific this is all but no, but I think that your your bigger point which is there's an intellectual puzzle here That has enormous social impact Is is exactly right and so it's an area where it's it's fertile for people to work in Because if we can figure out how to close that gap We think that there's a real social benefit both in terms of inequality But also in terms of economic growth that could be realized. So no, I think that your your points right on I mean in some sense We're running a very limited experiment on a larger puzzle and we're not making any headway on that puzzle So, you know, we have people you really should be actually, you know, running five miles a week and eating better and black they don't And do we think that sending them another flyer? And we realize we just sort of and I think in health become the conclusion We have to be more sophisticated in our interventions. So when I when I I hear people in education being puzzled that the information isn't doing it I'm like, duh The difference between running five miles a day For many of us not I mean school is as painful for many people. No, no, but that's the point You get the the thing is you get the person to actually you submit the the difference is The puzzle is not the puzzle is that when you actually send in the FASA Some fracture those people not only go They don't just show up the first guy day and find out it sucks and leaves They stay for two years. That's the puzzle. And so you don't tell me though Yes, people feel as if you know, whatever people are in school, but then what are those? It's the puzzle is the different That's the puzzle and I don't think I understand this program I mean, I understand all these things But it's like, you know, the misery that I would get from running every day or I mean Or maybe Is when people avoid getting information about their health I'm getting avoiding getting a diagnostic test that if it revealed something would give them So I think that's the closest analog. No, I disagree. I think all of these things are different Because I don't there's a very good way I don't want to find you know Arlene tells me I should find out whether I you know, I have whatever early on sent dementia So as a point of uh, so so first thing, uh, you know For doctoral students, this is a good strategy when you're giving a talk is get people actually defending or you know To move forward the second thing is At times you have to step in a little bit and just kind of bring you back Because I think the point is because I think the bigger point that you've made Effectively is there's a puzzle here as to why that little bit of assistance has such a big effect And a puzzle as to why this simple change in kind of them calibrating cost has no impact whatsoever And both of those are very open questions that we just we we didn't solve And that bigger question about how to close the gap You know, we've tried something that was seemed to be kind of at least from policy Something that people were thinking about actively as a solution and we just didn't find an interaction with it So these are low income people thinking our assistance can navigate a complex process to taxes, right? So it it seems to me then to give them information and then say go navigate this complex process That and there's four thousand dollars at the end of the rainbow for you is equally You know, it might not just be that these people don't have the capacity to do that So that's I think that's the population issue, right? This is a population with low This is the target population for the Pell for example, right? I mean, you know 75 80 90 percent. Yeah, he's 75 It was something like hr block. So this is and this is the target population for the Pell so So if we want the Pell to hit them, we need to do Let's go here and then we'll come back over You mentioned that there are a lot of different policy initiatives promote the information because even open the same just have like that But just the information that doesn't seem to be an effect is a low-cost thing to do But do you know if there are any other policy initiatives to combine that information with assistance? Because it seems to be the hot to be a turner is a great piece as an example here Because it's an example where data exist So the data exists at the college board Which where we can identify every student's kind of college entrance And we can identify students at particular high schools that seem to have kind of a low rate of kind of going to Challenging themselves to go to a good the best possible institution And so we can target an intervention very narrowly based on information that we have to in some sense Both provide information and provide appropriate assistance. It's it's harder for us to actually Have that information in other sectors And one of the things that's very hard is to walk into a high school And you know sometimes we can walk into a high school and know that it's going to be the whole population But if you have a population that's very diverse in terms of the socioeconomics Sometimes it's hard to go in and figure out which are the students that might actually need the help Now there are you know some people who've discussed for example using free reduced lunch and eligibility for other benefits as a proxy for eligibility here And there's certainly a thing that a case to be made that they're that lowering the barrier might actually improve students take-ups of it I mean one of the hard parts in you know kind of trying to play the other side a little bit Is it's not like we created the fast ones that let's have the most complex form on the face of the earth Right the faster became complex for a number of different reasons And in particular as now we started to move towards an era where the fast has become simpler and simpler Although i'm not quite sure how far we've gotten on that road But what's happened is institutions who are not getting the information they wanted have now started to require You know the css profile and other types of ways of acquiring that information So there's information that that institutions want and if the fast is not going to get it They're going to find another mechanism for identifying that information Yeah That maybe we're not giving the right kind of information or the right information Was that a statement on the accuracy of these estimates or do you think there's a different I think it comes back to the earlier point that there are these domains of information that might be more salient We thought the cost and a personalized estimate of the benefit was a good starting point Maybe that's the wrong place Maybe what we need to do is is provide information that says you know It helps them understand how much effort it's going to take them to succeed in college Maybe that's what we need to do. There's other kinds of information. No domains that we could have looked at These were the two that we chose And we didn't get any traction with them But it doesn't it's it you know, it's not to say that information doesn't matter these two domains don't seem to matter In our sample we actually had a lower rate of of kind of Of One of the things they do is an internal check to see if the data seemed to make sense We had a much lower error rate in the place where we did the assistance Um, you know in the information given that there was no impact on uh filing We didn't go down that road any further Yeah I'm taking in the realization that a lot of kids are now They're verbalizing their motivation for going to college being about economic benefits in the long run So workforce career goals, maybe that's part of how you tailor the information in the future Iterations of where somebody else does is talking about the benefits those economic benefits Not simply the immediate ones about how to pay you for college But like why you kids want to go to college and I try to do that and in our problem The problem was our IRB forces to give the benefits to everyone Uh, and so in some sense, we couldn't actually do any test on that You know the the oriopolis and done that is one of the big parts that they're trying to do there Is really kind of market the benefits in a different way and and They find at least in students kind of stated intent to go to college They don't actually observe college enrollment, but in their stated intent They're finding impacts on the likelihood that they're saying they're going to college Yeah On the basic puzzle By Talking with some community leaders even contacts through standards John Gardner center perhaps Because they have contacts with Love and come communities could be interesting to see just qualitatively what community leaders Who really understand the population might say about the puzzle Well, I mean one of the things in as we've done this I mean we've actually, you know, for example when we were doing Ohio We were highly engaged with the board of regents and highly engaged with the local superintendents around In trying to kind of acquire some of the information about what they thought We should do and whether their populations would would take advantage of it And you know, there was a number of superintendents who were, you know, very gung-ho They didn't fully understand they thought that we could even bring h and r block to all their high schools And so they they viewed this as as really a place where As as being the kinds of information that they thought their students needed And that's why I'm thinking the true community leaders who know these The population might at least kind of Speak to possible that just to try to I don't know just help you think about it The one of the one that we're working on is is going kind of a different direction where You know, there's a number of different advising programs in high schools And then some of those, you know The person who is doing the advising is kind of closer to the phenomenon And so like one that we've done some work with is the advising core Which like teach for america, but they put a you know, kind of a recent college graduate And so trying to see if there's a way to actually change their delivery mechanism To see what they're getting, you know, in some sense because they have that relationship with trust and proximity But you know, that's all very much in its infancy There was another hand From your findings is not We found here's something else. It doesn't work, but that the assistance Worked enormously and it's cheap. Isn't like 79 bucks her participant It was much cheaper than that even I mean the 79 bucks was actually because we were also paying the tax professionals To do that to to ask the questions and we were paying participants Yeah Yeah, so the sales job I think on this is that this is a really cheap intervention that does a lot as social sciences We're like why why does it work? Look it works it's cheap So just do it well Well, I think the other part that the reason I don't I'm not accentuating that as hard Is because you know one of the hard parts here is we've already published paper number one Which said, you know, here's this assistance and information. It really works And now you know this embedded project that we thought would be interesting Just finding information just doesn't matter. You know, that's the paper that's to be written And we're going to come back to this assistance point But you know, that was the real focus of this particular intervention And so that's why at least in a frame, but I take your point wholeheartedly And that's why you know, I try to put those previous results in there to show the contrast Between those information results and the others Okay, the fact that people are leaving Um, how are the red socks doing to get the score? It gets the games not games not up yet Yeah, okay So thank you very much if you have other comments, please feel free to come up and talk to me about them I'd be happy to get those comments and again, thank you very much for for the discussion today