 Our museum is truly an honor to have you here. All the things we grapple with most as museum professionals, I think, and certainly something that always presents one of our biggest challenges is not so much that we're almost always familiar with the topic that we present, but really, rather, how do we present it? How do we really reach people? For many years, I was a curator before I became a band and got into administration and fundraising. And I was always most toward taking the knowledge and excitement I had about the topic and making someone else equally as excited about it. That's not easy, because as much as we think we know and wouldn't look hard to others, we have to understand that chances are people are going to be exposed to what you have to say for the very first time. The good news, though, is that museums have more potential to do this efficiently and effectively. I think another traditional learning venue, such as classrooms or books or photographs or films, because we have the opportunity to directly engage our visitors in the learning process. This is especially true for science. And in this area, Dr. Paul wrote a book. Actually, he's written dozens of books and articles, too, on the subject of free choice learning, which focuses specifically on learning that occurs when visiting museums, zoos, aquariums, or parks. It's toward everything that we're trying to do. A member of the Ballanty of Work on State University, Dr. Paul founded and directed the Institute for Learning Innovation, where for 20 years he oversaw more than 200 research and evaluation projects involving a wide range of free choice learning institutions. He also worked as an early childhood science educator at the University of Maryland and spent 14 years at the Smithsonian Institution, where he held a number of senior positions, including director of the Smithsonian Office of Educational Research. In 2006, Dr. Paul was recognized in the American Association of Museums as one of the 100 most influential museum professionals of the past 100 years. In 2010, he was further recognized at the American Association of Museums Education Committee with its highest award, the John Cotton Dano Award for Leadership. And just last year, the US Council of Science Society presidents gave Dr. Paul their educational research award for his outstanding achievement in research that improved children's learning and understanding. Ladies and gentlemen, it's my honor to welcome Dr. John Paul. Technology work, sir. OK, so thank you, thank you, Charlie. It's truly, can everybody hear me back there? Isn't there a peanut gallery right there? Cheap seats. It truly is a pleasure to be here. I just always feel honored to have folks who are coming from such far distances to hear me speak. Hopefully it'll be a rewarding experience. And actually, I'm going to primarily talk today. I'm going to talk about science. I'm going to talk about informal education. And I'm going to talk about free choice learning. But as I do it, bear in mind, I know that many of you come from other kinds of institutions. And as Sandy said in my introduction, for better or for worse, I work in a lot of different areas. I mean, I work in the area of science education. I work in the area of informal education. I work with museums, but I also work with zoos and aquariums. I work in science institutions, but I've spent a lot of my years working in art and history museums and other cultural institutions as well. And over the last dozen or so years, I've done a lot of work in ecotourism as well. And actually, I have two doctrines. But one is in ecology. So I bring that perspective as well as education. So I'd like to try and think properly about the world. And I'm going to share that perspective with you today. But again, as I said, just every time I use the word science, you know, you can fill in the blanks. Result of that doesn't apply. Here's what I'm going to try and talk about. And I'm going to spend, you know, we've spent almost two hours together. We have two hours together today. I'm going to spend the first half talking at you, hopefully with you. And then the last hour will open up just for discussion, question, and answer. As I said at dinner last night, actually, it's a rare and unusual experience that I get to speak for an hour. Because you know, people invite in and we give you 15 minutes. And I've got a lot to say. So even an hour is putting a lot of stuff together. So hopefully, it'll be an engaging hour. But you know, I'm going to talk about, I'm going to sort of cover a lot of waterfront, particularly quickly, sort of when, why, where do people learn? And then I'm going to focus specifically on this big question, well, that's great people learn all the time in all these places. But what about the stuff that we do, this informal community? Is there any actual evidence that anybody really learns anything in the sense? And that's another one of those words. I'm going to use the word learning. But please know that when I use the word learning, I define it very, very broadly. It's not just facts and concepts, although that's part of it. It's also interest and engagement and behaviors and beliefs, all those things we learn. And so it's appropriate to use the term learning for that. And then I'm going to try and at least give you some brief sense of what I personally and my colleagues at this point at Oregon State University, University of Colorado, and George Mason University who are all involved in this one project, how we have tried to take the understandings that come from this setup here to try and do something within a single community in Oregon to not just talk the talk, but walk the walk. So here we go. OK, so big question. When do people learn science? The short answer is all the time, right? So we're in every day, every day of the week, we're at our lifetime, a little known fact. But we spend less than 3% of our life on any kind of formal instruction. Even school-aged kids spend less than 20% of their waking hours in a classroom. And that's in school, and that doesn't even necessarily mean they're in a classroom. But we live in interesting times, the same least. And one of the things that makes this interesting times is that historically, for most of the last 10,000 years at least, where people have been living in cities, there have been gatekeepers to knowledge. And certainly over the last couple hundred years, those gatekeepers have been places like governments, schools, libraries. And that's just no longer true. There's lots of access to knowledge. In fact, there's so much information out there is actually knowing how to sort through it. But as a consequence, the boundaries of where, when, and why, and the whom one learns are eroding. And so the traditional boundaries between work, leisure, schooling, all that stuff is blurry. As a consequence, learning truly is continuous and cumulative. It always has been. But people are more deeply aware of it now than ever before. Because we live in a time of rapidly changing knowledge and understandings. So even if you did pay attention in school, which most people don't, what you learned in school may not be relevant to you 10 years later. So if we can imagine a life space of 70 plus years and you're awake from 7 in the morning to 11 at night, for example, 12 months out of the year, you know, you go to school for part of that time. And if you're lucky, you go to college. But there's an awful lot of space in there. And this is the space that we generally talk about is informal but free choice learning. Although I, when I, we did the coin, I'm not even calling the term free choice learning, but when we championed that term, part of the reason we picked that term as opposed to informal, which was current at the time still is, is that learning is not about where you are, it's about what you're doing and why you're doing it. And so we want to put the emphasis more on the motivation of individuals. So in theory, schooling can be and is free choice learning for those who get choice of control over their learning. And when you're in a place like the museum, more often than not, it is free choice learning, but it doesn't have to be. If you force somebody to sit in the lecture room and give them a lecture and then send them back to school, that's probably not free choice learning. So why do people learn science? Well, the reality is they learn science for lots and lots of reasons. First and foremost, people learn because they have interests and needs. That's what dictates what most people learn most of the time. But we learn for lots of other reasons too. You know, most of us work, most adults work, and they learn all kinds of stuff and we're gonna need the needs of their jobs. People learn because they have hobbies. People learn to support the needs of others, particularly parents, often. Maybe learning things that they didn't know they wanted to learn, but their kids are interested in them some day, a day ago. We have a real need to learn because of some issue that comes in our lives. Most people in this room probably pick a disease, don't know much about that disease, but should hear significant other, or you get that disease, all of a sudden you become quite knowledgeable in that disease. Is it a matter of science literacy that you should know about that disease? No, what is science literacy is that should you need to know you're capable and take the time to learn about it. And then of course, we also learn because we are told we have to learn in the school and prepare ourselves for careers. We do a lot of that thing. And by the way, even though I'm a great advocate for this thing called free choice learning, there is a place and importance for compulsory learning. For example, talking about medicine, I'm really thrilled that my doctor hasn't had a good medical school and has a boarding thing. I think that's a good thing. I think it's great that we have a driver's license and pass the driving test. Those are compulsory learning opportunities and there truly is a time and a place for people to have to show competency in specific things that they don't define competency for. But those are the exceptions. So they are not the rule in life. Most of us spend most of our life learning things that meet our own needs. We have our own criteria for whether we are sufficiently knowledgeable and those are not. So I did a little sort of grassroots study, sort of a crowdsource study tried to ask people to fill in a little survey, get their friends and neighbors to fill in surveys to ask them, tell me something, tell me an area of science or technology you know something about. And so this is the result of several hundred people who talk about things that they know something about. And as you can see, a lot of hardening, biology, health, cooking, but in there we give them things like physics and school subjects, techs, things, astronomy, anatomy. But some hundred people, hundreds of different domains that people say they know something about in the realm of science and technology. So that's great. So where do people learn all this stuff? Well, you know, if you think about learners, there's lots of ways that you can learn. So people learn a lot from television shows these days. So this is a picture of the investors. People go to places like science museums, science centers. They read books. We learn on the job. We learn from magazines. We learn increasingly kids from gaming situations, hobbies, not hobbies, but vacations, leisure time activities, we get out in the world. We learn an awful lot, which isn't often appreciated, for friends and family. We do learn in school. We learn from after school programs, this is an orange, increasingly from the internet, and then through hobbies. But if we try to quantify that, which I have done over many years, this represents a whole series of random surveys and I'm going to talk more about that random survey. This is all from Los Angeles. Data from Los Angeles. Obviously people could report that they, self-report that they used multiple resources to learn about stuff. So the takeaway message from this is people use a lot of resources in order to get the science knowledge they need and use on a daily basis. There are a couple of really interesting things. Since I started this data collection in 1997, the internet in 1997, 30% of the public said they used the internet to learn about science and technology that increased to 10% in 2000, 24% in 2007, 67% by 2009. It's probably starting to plateau that this is a significant exponential. So this number hasn't changed, basically, in over a decade, but this number obviously has changed. There are two numbers that really have changed. The museum number hasn't changed, the only two numbers that have changed has grown exponentially and this has been linearly dropping. So whereas in 1997, about 50% of the public said they got knowledge from school courses by 2009 it was about a quarter of the public and we can predict that that number will keep dropping too. So a big change is going on in the world. And if we just sort of summarize these in terms of looking at it as formal and formal and work related, what we can see is that, at least on average, about 20% of the learning that people are getting is coming from school, about 20% of the workforce and about two thirds is coming from these other places. These are the places that most of us work in at the moment. Actually, talking about science education, sort of a backing envelope, a little thought experiment idea, that's really curious, we talk about science educators. So typically when politicians and policy makers talk about science educators we talk about school teachers. So I figure how many ins and outs schools, professional science educators are there and my faculty envelope estimation came up with over a million in the US. In fact, there are almost twice as many professional science educators working outside of school as in schools, but that's a little amount of time. But that's all a little good. You know, as people want to say, well that's great, we know that these out of school places are wonderful, they're really nice, but are they a necessity? How much real science learning really happens in these places? You know, they're fun, everybody enjoys them and they're probably good for generating interest, but are they really important contributors to the community? So I've been on a mission for the past 10 to 20 years to try and see if we can generate some evidence in support of the fact that they probably did make a difference. So let's look at some of that data. Let's start specifically with science centers and museums. I actually have been collecting data, as I alluded to in Los Angeles, for almost 20 years now, longitudinally, looking at the impact that that one institution, the California Science Center, I'm not going to share that data with you at the moment because it just completed another study, which I think actually in some ways is more powerful. And this was a study done building off the work that we did in the California Science Center, I was approached by actually the director of the Science Center at that time in Finland, the institution's Hedekai, and the director was Hedek Helsen, who actually is a very interesting dude who was formerly head of the directed, the equivalent of the American Association of Science, but the Finnish version of that. He's a very distinguished scientist, but he took over the Science Center in Finland. He was really impressed when he saw me reporting the results in LA and said, you know, my colleagues and I were always complaining that we don't have enough data, and yet we still sit on our hands and wait for data to be collected. If we generated money, could you do the study internationally, so of course. So between the two of us, and actually also Leslie Lewis and Canada, we ended up getting 17 Science Centers from 13 countries around the world to actually put their own money on the table to me and my colleagues to do an international Science Center event study. And so those are the results that I'm going to talk about. The final report, it's not the final word on the data, but the final report was just finished a couple of months ago. And what we were trying to do was empirically determine whether a Science Center experience actually affects a whole range of significant outcomes in terms of a couple of understanding of science, interest, curiosity. Do people who participate in Science Center experiences do they have more science hobbies? Are they more likely to be engaged in science careers? Those are the kinds of things, the kinds of outcomes we'd like to believe that we make a difference. So let's see if we can do that. And we specifically looked at two samples, a youth sample, 14, 15-year-olds, and adults over the age of 18. There's reasons for all those which I can go into, but I won't go right now. And I just feel compelled to give you to digress and become along slightly more than I already am to talk a little bit about methodology. You know, there are a lot of folks, particularly in the U.S., and you will see there are 17 institutions from 13 countries. There's only one U.S. institution that was willing to put $20,000 in the date because they all plead poverty. And these are many very large institutions with 20 million over-year budgets, by the way. And there's another little site we gave which if time permits I can go into. It was quite a scene at the last aspect meeting that international colleagues just berated the audience who was mostly American and saying, where the hell were you? Why didn't you participate in this? What's your problem? But sort of these randomized control treatment designs are sort of the gold standard. That's how you improve causality. And I very strongly said we cannot do a randomized control study of a science center experience. It's a free-choice environment. What are we going to do? Randomly say, okay, this half the room, we'll only look at this, this half the room, we'll only look at that, or we won't go. And that's going to be valid. I mean, people go, they choose what they want to look at. They invest their time and their energy. What makes these places special is people get to choose. They get to choose with their feet and their minds what they want to pay attention to and why. And the reason they do that is they actually are disproportionately focused on things that they're interested in. And so they find these places very compelling because it satisfies their own needs and interests unlike compulsory education. So we can't do a randomized control study. So I did use an epidemiological study. I'll talk more about some of the implications at that end of the room. But what that means is using that other medical model which basically says we want to understand something complex like, for example, heart disease. It's not as if we can just say, well, you know, heart disease has a single cause at it, causality. Actually, it's because of epidemiological research we know that exercise is important, genetics is important, diet is important, stress is important, where you live is important. There are a lot of factors that contribute to whether or not you have heart disease or not. And so we can't just isolate one variable. We have to deal with all those variables because the world is actually messy. And the nature of learning is such that it is a mess of magnitude. There are a lot of factors that contribute. But if we collect enough data and do our research carefully, we can begin to tease out relationships. And that's just what we did. So I'm going to show you some tables. So first of all, these are the 17 institutions and we had two different sampling strategies. This is the data I'm going to talk about today. So we had roughly 12,000 people from around the world, a minimum of 500 for each of these 17 institutions. And since they've just changed your name, the Frost Museum of Science, which used to be in Miami, used to be... Ah, that's... So, okay. We had lots and lots of variables. Each of these domain, with knowledge and understanding, with interest and curiosity, out-of-school engagement, applications, and confidence in science, essentially science identity, we had actually with 13 variables benefiting this, and as few as three measuring this. We had lots of different ways of trying to slice this. And we did our, you know, good certificate analysis, we did it as constructs. And so what I'm showing you is the construct, basically, of these questions. So these are not just a single question, these are all multiple questions, which collectively give some confidence in the results. The bottom line is that if we look, and I'm just going to show you two data tables. You know, if we look at you, if we look at the number of visits, the more visits, the more differences. And you can see that even one visit makes a difference, but not much, but not much across many of these variables. Where you get a big bump is after two to four visits. I'll come back to that. And then, interestingly enough, for some of these variables, if you have more than four visits, those numbers start to go down. I'll come back to that point. So that's one example of you. Here's one example of adult data. This is how, this is how recently you visited. So never visited. So before 2010, a couple of years ago, a year ago, one year, within the last year. Obviously, the more recently people visit, the more the greater the significance. And what these funny little numbers as Etta thing on the side is, that's called effect sizes. And so something may be statistically significant, but actually may have a very low effect size, which means that it is significant, but that might just be because we've got such a big sample. Maybe that's not much of an effect. So just to help you decode that, an effect size of below 0.1 is probably the most equitable. An effect size of over 0.1 is on low end, but probably still means something. An effect size of 0.2 means that it's basically, that's what you'd expect. That's about average. That's a typical effect size. And if it's over 0.3, which none of these are, then it's a serious effect. Big time. We can really depend on it. But most of the effect sizes we saw on this data were between 0.1 and 0.3. They were in that sort of, sort of on the low end to moderate what you'd expect. So, here's the big conclusion. So, here's where the research gets to slide something in the end. What we found. The first thing is, that for both youth and adults, youth and adults who use science centers are significantly more likely than those who don't. To have high levels of knowledge and understanding of science, interest and curiosity in science and technology, engagement with an interest in science as a school subject, that was just you. So, we talked to, asked all these youth, you know, what is the subject that you're interested in? It was those who visited science centers particularly frequently were significantly more likely to say science. They were not significantly more likely to say art or history or math, which is actually a good thing. You know, basically what it says is this wasn't a function of the population. It wasn't just that those kids who were privileged enough to go to science centers are likely to do well in everything, which we might assume, but good news is if you actually do a science center, you do like science. And the more often you go, the more likely you are. They're more likely to engage with science and technology related activities out of school, hobbies and applications, and they're more likely to pursue careers and have confidence in themselves as a science software. That's all pretty new. Furthermore, the results were strong for both youth and adults, but almost uniformly, the effect sizes were larger for adults than they were for youth, which was an interesting finding. In general, it makes sense, but it's nice to have it reinforced that the more frequently you go, the more recently you go, the more likely to have an effect. Here's another thing that, but for me, I was really interested in this these last few months, and I will digress and tell you a little bit more. So many, many years ago, when I was 19 years old, working my way through college at Berkeley, I had the honor of having a great job. I was a lecture demonstrator at the Lawrence Hall of Science, right after. And, imagine, this was 1970, so I think in 1970 dollars, I was going to pay $50 a lecture, $50 an hour, in 1970. Now, I actually wrote the lectures, there was a front-end work, but basically I waltzed in and did five degrees a week for an hour, and I was living high and high. I was doing great. Remember how I was doing a pretty good job of my crowd? And then the director, I got worried that there was new policy for Lawrence Hall of Science, and they were going to change direction, and rather than invest their dollars in these short one-shot, field trip things, they were going to invest in more intensive programs. And being cautious, as I was then, I made an appointment to see the director, who was a noted physicist. And I said, Professor Portis, I understand completely your assumption that if you put in more time, you will get better results in terms of spending more hours with kids should result in more significant learning for these kids. Makes perfect sense. But do you know what the shape of the curve is? What's the slope? Is it a linear thing, so that if we spend one hour, we get this much going on, if we spend two hours, we get twice as much. If we spend three hours, we get three times as much. If we spend four hours, we get four times as much. Or is it exponential, so that for each hour, we have a geometric increase. Or is there an asymptote? So maybe, actually, you get mostly your bank for the buck in the first hour. Do you know what the slope of the curve is? I mean, how do you spend your money here? And, you know, it's a good use of experiments. You said, John, that's really interesting. You know, you're not going to get your job back, but I'll think about it. And I'm going to tell you that 40 years later, as a community, we still don't have the answer to that question. This data begins to suggest that there is a threshold phenomenon. And that basically, and I was committed to trying to find the sound, that actually, you, by and large, see in the data a significant increase from zero to two to four business. And then it flattens out after that. And likewise, in terms of duration, you see an increase up to four hours, but then it flattens out of effect. In terms of our duration, after five hours, it starts to decrease in terms of effect. So that's good information. That's the kind of information that we can actually use to make our programs better. And it tells us how we should invest our dollars. So, that's great. That's a science center. We have this whole thing in general. So I also collected the data in Los Angeles, again, and it tells conservative about a thousand people. I'll sort of zip through this. I was trying to see, so let's assume that there are four big contributors, I'll say five big contributors, to adult science literacy. Public understanding of science. Schooling. Childhood experiences outside of school. Adult experiences outside of school. Experiences in the workplace. And privilege. Maybe it doesn't matter if you're white, if you're absolute, if you're male, life's good. And as it turns out, I collected, I asked people lots and lots of questions. This is just a sampling of some. There were lots of things that were highly significant contributors to public understanding of science, including books, magazines, and newspapers right outside the school, e-mail, using the internet, income, high level of education completed, work on job, childhood, reading books and magazines, again, an occupational one. So, lots of factors. But when we looked at these, in terms of those five big factors, so we summed up, all created a model again. So, what are all the factors that are related to schooling? What are all the factors that are related to childhood experiences built? What we find is that, to begin with, collectively these explain over 50% of the variance, which anybody who knows social science, that's pretty dang good. So collectively, these aren't things that actually contribute to public understanding of science. Years of schooling contributed 17% of the variance. Childhood, out-of-school experiences contribute the exact same amount, 17% of the variance. Workplace, 20% of the variance. Privilege, 23% of the variance. Helps to be male. Helps to be outward. Helps to be white. Out-of-school adult experiences for choice experiences, 39% of the variance. So, what we can conclude from that is, all of these are highly significant. All of these are important. But you certainly can't say that schooling is the most important. You and, as I provocatively talk about, my apologies to all my school colleagues in my community, certainly. Currently, in the U.S., at the federal, state, and local level, we invest 99 cents on every dollar that we spend on education in this country, on schools. Schools do not contribute 99% of the value. So, as a society, we are not investing in many things that make a significant difference, have a significant impact on public understanding of science, despite the fact that I think we're doing a good job I would certainly argue that this provides some evidence that we might want to think about how we invest our dollars. In fact, in general, although all of this research, again, is correlational, it doesn't exactly prove, it doesn't causally prove that having a science center or all of these experiences creates a more literary public. There is an increasing amount of evidence which is, actually, when you start collecting this kind of data from lots of different places and lots of different ways that you keep getting the same results, and it probably means there's something going on there. And my conclusion is, you know, people talk, well, is it because people who are predisposed to be interested in science go to science centers, that's why they're more, you know, do better? And I say, you know, this is a trick in the next story. So, let me put it in a barnyard context. You're a farmer. You can't decide whether you started having eggs because you had hens, or because you had hens, you started having eggs. Or you got some eggs and then you had hens. But you know you need to sell eggs. And having hens in the barnyard results in a lot of eggs. And so, if you're making money off of eggs, then you probably want to have hens in your barnyard. And if you want to have a society where you value those kinds of outcomes, then it behooves you to have hens like science centers and science museums. This museum too is comparing all these other places because the evidence is overwhelming that communities that have these are significantly more likely to have those kinds of outcomes. And citizens who use them are significantly more likely to have those outcomes. So, if I am a policymaker and I want to have those outcomes, well, I don't know whether a chicken egg but it certainly makes sense to have hens in the barnyard. I think that we can point to data and I have data. I do have some data that is more causal. We have causal studies. We have studies that are comparably causal to what schools have that if you teach X for a year and then you give a test, kids seem to know that. We have a lot of data like that which somehow isn't as valid when we do it in the informal sector that when people go to a science center and there's an exhibit on X we come out knowing about X. We've got lots of data that is just as valid as the standardized test. They're all self-referential but so is the school's data self-referential. What this suggests is that overall you need all of these things to create an ecosystem of learning. So we need to start functioning because this is a complex science learning music learning history learning art learning civics learning community. We need to look at whole pieces. So trying to put my money where my mouth is I have funding to try and see if we can't take what we learn from all this and do something important in a single community as an experiment. This project is called Synergies and it's purposefully named that because what Synergies is is where the whole is greater than some of the parts. So our finding assumptions which should sound familiar from all the data that I just provided is that people learn science across their lifetime from many different sources and disproportionately right? And since learning is this never an impunitive process coming from many different sources it makes sense that the only way we're going to make a significant difference is if we all work together collaboratively within a community. So we have defined the system the public education system of this one community in Oregon as the whole community within the school system the system is all the people who live in this community that's the system and therefore the contributors to public education are all the people who live in this community and therefore they're all teachers they're all learners we have to work collectively to make this a big difference. So what we've been doing is we are we work with a cohort of kids started every 10 year old child who lived within this geographic family whether they were in public school or private school or public school we started trying to track them when they were 10 roughly in fifth grade and we were going to track them and collect data from the next 5 years and meanwhile we are trying to organize and we're collecting data in lots of different ways or even modeling it computer simulation trying to focus on interest and engagement with experiences and our partners are every bloody organization and individual that we can convince to work with us so currently our partners and the Synergies Project are public and private schools library science center the children's museum park district about a half a dozen different kinds of after school providers living in 4-H and Mesa as well as the social service after school provider the faith based number of faith based organizations the community college two universities I'm sure reading some books out but basically we're trying to find anybody who's willing to work with us can be our partner and key to this partnership I've been telling people is we're not saying we're saying all of you are working really hard you're all doing the best you possibly can we're not going to ask you to change your goals your goals are what your goals are all we're asking you is to buy into the idea that there are other people in this community who share some part of your goal and if you work together you're more likely to both achieve your goals so we will help broker those overlaps and maybe even foster some other overlaps that you didn't realize and so we're going to build synergies just to give you a sense of this community it's for historical reasons it has its own school district which makes it easy four elementary school students one middle school it's low income very diverse community academically low achieving all four of the elementary schools fail actually middle school and you know all the science there are no labs there's no hands on any science instruction but you know we're not telling them what they should do we're just saying whatever however you're doing it we can help you be more corrected it's very diverse racially and ethnically and the after care is although it's 40% white 60% of those individuals are recent immigrants from primarily Eastern Europe there are sort of locked off here there are 45 acknowledged languages within the school district but over 100 so it's a relatively small the whole district is about 1500 students 45 acknowledged languages our questionnaires and parent permission stuff was translated into Spanish Chinese Vietnamese Ukrainian and Russian so and what our data over the last couple of years and we spent the first year of the project just talking to people just stakeholder we spent the next two we spent the next two years collecting baseline data because if we're going to make a change in the community we need to know what the community is like to begin with so we've been tracking these kids every year we've been monitoring data we've got case study kids that we talk to every month we've been modeling it and now we're starting to build coalitions we're bringing all these people together and helping them create an education plan for their community but it's based on data so based on the data we've identified three key leverage points where we can see that if these are these conditions are not met we're going to see this precipitous drop off first of all this synergies thing which is obviously key so I'll give you another little window and then my at least my mic my mic so you know we tend to think informal is the place where kids get excited school is the stuff where they learn the content actually I would argue typically it's the other way around actually most people learn most of their in-depth information outside of school because school is sort of this wide it's only if you go to graduate school that you actually get into the depth of stuff and case in point the interest trajectories of these kids and also we get this huge bump in astronomy related interest and it turns out that the schools have units on astronomy and at first we thought we knew that one of the teachers was a astronomy bump you know how these we figured it's probably just this one teacher but in fact no it wasn't just one teacher it was across all three middle school teachers where we got this bump it was the last unit that was taught 6th grade by fall that was all wrong that's because whose job was it to help kids know it's now summer vacation the local community college the local science center has astronomy programs there's an amateur astronomy club in your neighborhood there's workshops and scholarships for kids but how were those kids to know that that was the case nobody told them it was nobody's responsibility the teacher did his job work her job they taught the lesson and I guess they thought it built kids thought not excited but by the time kids went back to school they weren't excited about astronomy anymore by and large because nothing happened in between we gathered together all the players who's the grownups here whose job is this we're not telling you the school teacher that you need to do something in the summer but yeah we are telling you that you should at least tell your kids if you are having fun with this there are these resources out there and we should be communicating to those resources that we're teaching this class in school now this lesson, this topic and you might want to get in touch with the kids who will help you meanwhile you know kids do stuff after school they get excited but the teacher has no idea what they're doing after school and the after school person has no idea what they're doing at home so it's only if we all talk to each other can we actually help each other be better at what we do kids need role models they need support because they get encouraged but they need to know what they do with this so it's not, there are lots of mentor programs this is not about big brothers and sisters it's not just about having a meaningful adult in your life that's a good thing but it's having somebody who shares your interest and passion so if you are interested in astronomy let's match you with somebody who actually is also interested and knows something about and passionate about astronomy is better able to guide you into other additional things you can do I mean if you just have another adult they may or may not know anything about what the resources are how they help you get passionate about astronomy so it's not that we're trying to make every kid an astronomer it's we want to make every kid engage in passionate about some area of science so let's figure out a way to build those mentorships and then finally we all say yeah we know that adolescents we're dealing now with adolescents are really influenced by their peers and son of a gun our data supports that we have kids who were doing things who dropped out of activities because they had no friends but how many educators in school or out of school actually pay attention to that reality who spend as much time supporting peer relationships as they do contact if you want those kids to stay engaged you have to make sure they have friends who are doing things with them and then they'll stay engaged so you need to devote as much time and energy to building cool ways for kids to meet other kids or reinforce that this is a good thing to do with your friends if you want them to stay engaged so we've argued that these are key leverage points and we created committees within the community to figure out how to tackle each of those problems to help us again we're not telling you what to do we're just telling you how to make accomplish what you want to accomplish here are some leverage points as community figured out and we will play brother we'll be happy to be that neutral honest broker who can convene you will provide food and drink if you want to and we'll just make sure you have meetings and do that but we aren't going to tell you what to do and so that's what we're trying to do within this community which ultimately is this again trying to make this a functional ecosystem it is this ecosystem it's just not always a very functional ecosystem and so as we talked about lunch today there's an opportunity here in New Mexico to actually forge a functional ecosystem but it takes work, it takes effort and it takes some kind of higher level goals collaborations don't just happen because you say it's a good idea to collaborate we're all busy so you need to feel that this collaboration will help you solve your problem and so that's what we're trying to do in this one community in subcommunity in Portland is trying to help people in that community solve their problem by being a good broker so anyway that's all I have to say if you want to follow up on any of this you know there's how you get in touch with me and I will now entertain your