 Aloha, everyone, and welcome to Ehana Kako. We're here every week on the ThinkTech Hawaii Broadcast Network. I'm Kili Akeena, president of the Grassroot Institute. Well, whether it's fighting poverty or improving health care or raising the level of education in any part of the world, it is absolutely essential to begin with good, solid research. Very often, methods are not—programs are cut short or fail because they lack the research base necessary in order to set up a program that will truly be effective. And without research, it's not possible really to tell whether results have been achieved. There's somebody who's going around the world empowering people with the tools necessary to use research in order to solve human problems, and she's at the Jamil Poverty Action Lab of the Massachusetts Institute of Technology. She's a researcher, but more than that, she communicates the findings of research as well as the methods by which to do research in order to solve human problems. I'd like you to meet her today. We're going to talk with Mary Ann Bates, the deputy director of the J-PAL project from MIT. Mary Ann, welcome to the program. Thank you so much. It's great to be here. Well, it's good to have you here, and just so our audience can get themselves situated in terms of where you are, although you travel regularly back and forth to MIT in Boston, you actually get to reside most of the time in your hometown, and that's in California. Is that right? That's right. I'm in Davis, California right now. Well, you and I have met in settings in which talking about statistics and research are considered fun, but we have to keep in mind that that could be a rather esoteric community. And I wanted to let our audience know a little bit about the kind of work you do by starting off with an example. Can you tell us somewhere where you've been involved that research has really made a difference in how we solve a human problem? Sure. I'm happy to do that. So, you know, our mission at J-PAL is to reduce poverty by ensuring that policy is informed by scientific evidence. And when I got into this work almost seven years ago, some of the studies that really pulled me into it were the ones where doing a specific experiment and running a study helped to change my view on how to approach a particular topic. So one example to tell you a specific story comes from Kenya. Kenya. So tell me a little bit about the background of the people whom you were trying to reach out to and how they were struggling with the problem. Sure. So this is a study that was done by Jessica Cohen at Harvard and Pasquale and Dupont, Stanford, and they were working in a rural area of Kenya where malaria is a big problem. So malaria, as you know, is transmitted by mosquitoes. And it's particularly risky for very young children and babies and also for women who are pregnant. And one of the key interventions that is used to try to address the spread of malaria are insecticide-treated bed nets. So imagine a bed net that one can hang, you know, over in the bedroom, over the bed, and then at night you can sleep underneath it. And it's insecticide-treated, so it both has the benefit of protecting the person who's sleeping under the net from the mosquitoes. And the insecticide also kills mosquitoes while it's hanging there. So people were using this kind of technology, believing that it actually had effective results. How did it get started, the bed nets? Well, the real question that this project looked to address and the kind of debate that was happening in the international development community at the time was whether or not the bed nets should be given for free or whether one should charge for them. So the key question was that this is a kind of technology that the person who receives it has to use in a certain way day after day after day. So unlike, for example, receiving an immunization, right, where once the child receives immunization there's no further action that the child has to take or that the parents have to take day by day in order for that to be effective, if you give someone a bed net but then they don't remember to use it or they choose not to use it, maybe it's, you know, uncomfortable, it makes it hotter to sleep underneath or what have you, then it's not going to have any effect. Well, I like this because it's very similar to a lot of questions we deal with at different levels here in the United States. For example, this week our state legislature in Hawaii is opening and there is a bill to make college education absolutely free. So the presumption is college education is good but nobody has actually weighed or measured what would happen if it was given without economic value, if it was given free. And so in a sense you're measuring behavior based around the economics of a situation. Exactly. So what we're, what the study looked at was how does people's behavior change when they either receive a bed net for free or if they're asked to pay for it, right? And so what the researchers did is that they worked with a wide range of prenatal clinics in Kenya and they used random assignments, so essentially flipping a coin to see whether, to determine the price at which local women could receive a bed net. So some received a bed net for free, some paid say 50 cents for the bed net. And what the experiment was designed to do was to ensure that you had these two comparable groups of women who on average were identical in all other characteristics but one received a bed net for free and one received a bed net at a price. And then the real question was would they use it or not? And this was a real debate. There were a lot of anecdotes and stories that had been circulating of people, you know, one might worry that either the bed net isn't used, people might sell it if they didn't want one in the first place. And there were also anecdotes of bed nets being used as fishing nets or as bridal veils, right, for different purposes. And those stories were very salient and people can wrap their minds around them and, you know, even as parents we say the same thing, you know, I won't give my kids allowance for free, they have to work for it, right? So these were a lot of the anecdotes and hypotheses that were circulating when this experiment was conducted. And so what they did is they said, well, let's test it, let's do an experiment and see what happens when we give away essentially a preventive health product for free or ask people to pay for it. And so they not only ran the experiment and distributed bed nets in some cases for free and some cases at a price, but then they also followed up and saw were the bed nets being used. So they visited the homes of the people who had received bed nets or who had purchased them, did a survey, asked them questions, and looked to see did they still have the bed net. Was it being used, was it being, was it hung up? Sure. Now, now you've got me, you're holding me in suspense. So don't, don't tell me the results yet. But before you tell me the results, what they discovered, tell me a little bit about why this question was chosen for research. So what was going on in terms of the minds of the people? Was there a question about whether there was real value to providing these bed nets, whether it was a waste of resources or and so forth? What was going on in terms of the doubt that motivated this research? That's a great question. So the question wasn't so much do the bed nets work, right? The question of whether insecticide treated bed nets can prevent mosquitoes was subtle. The question was, are we spending our resources effectively? Are we spending a bunch of money to give away bed nets for free or at a highly subsidized price and hoping that it reduces malaria? But it actually isn't because people aren't using the bed nets. So the question was really, how is people's behavior actually affected by whether or not they receive something for free or whether or not they're charged with price? That was the motivating question. And this was a big debate, not just in Kenya, but in international development more broadly. So the large international organizations that we're working across multiple countries, governments in different countries were all a part of this larger debate. And that's why the researchers chose to do this experiment, because it was driven by this policy question that was really unanswered prior to these evaluations. You talk about anecdotes and stories. I suppose it's easy for people to get an opinion about a certain remedy or a certain intervention, partly because there's an intuitive side to it. You'd think, well, it works and people would want to value and use something that works. But as you're pointing out, that's making quite a bit of assumption about human behavior. And your task is to help test those assumptions. So tell me a little bit more about the results you got. Yes, that's exactly right. We see what we see, right? And if I see one person using a bed net, I'll believe that it's effective. And if I see one person using it for another purpose, I'll believe it isn't. So in this case, when they ran the experiment, they then followed up and saw that there was actually no difference in use between those who received the bed net for free and those who paid for it. So they followed up both in a series of studies. They followed up after two months. They followed up after a year. And use was basically identical for both groups, those who had received the bed net for free and those who had paid for it, which was really interesting. In this case, in Kenya, it turned out to be the case that charging people for bed nets actually did not increase the percentage of those who actively used it. Well, that's interesting. Which changed my mind. That's not what I would necessarily have expected. And so what were the implications? Because we have to also understand a little bit about their culture, a little bit about societal values and so forth. What was the value of learning this? That there was no difference really in their usage, whether the people had been given the bed nets or whether they actually had to purchase them themselves. What were some of the conclusions drawn from that? Well, it really affected the broader policy debate that was happening at the time around whether it's helpful to, when you're weighing the choice between either giving a preventive health product for free or providing it at a very large subsidy, that it might be best to provide it for free. So as another piece of context, the question wasn't whether these bed nets cost about $6 a piece, which was a price that given the typical incomes in this area of Kenya was out of reach for the majority of the families. And so the debate was between providing it at say an 80% or even a 90% subsidy or for free. And this changed a number of international organizations who were working in the area and in multiple countries providing bed nets, changed their policies from moving from a cost-turing model where say they charged $0.50 for the bed net locally to providing it for free because that was found to be a more effective and more efficient way to reach a much larger percentage of the community. So it wasn't just the fact that the prenatal clinics in Kenya where the study was done continued to operate and within Kenya specifically, bed nets have continued to be provided in this manner. But it also changed the debate and large international organizations that had been charging for bed nets decided to move towards a free distribution model as a result. Well, you know, this is a very fascinating finding because decisions are often made from political motives by political actors. So those of us who proceed instead from ideology or philosophy think we're superior because we have better ideologies. And so I would say as a free market thinker, a more libertarian, leaning kind of person that when people have to put their own money and value into something, that they will use it more or treat it better and so forth. And well, that's a fine philosophy to proceed from. But I think what you're doing is showing that we have to test our own presuppositions. So regardless of the prejudices we start with. That's exactly right. And I would also say this is why it's helpful to not just do one experiment, so J-PAL has now been around for almost 14 years. And our network has done over 800 experiments like this across 74 countries. And so this study showed, we learned a lot about what to do with preventive health products in the Kenyan context. But doing other studies, for example, looking at other health products or preventive versus acute care, there might be different answers. So I wouldn't draw from one study and say, therefore we know that things should be given away for free and not charged for like across the board. This is why it's very helpful to do lots of experiments like this in different contexts and building on each other so we can test our assumptions. So as the body of research grows, as multiple contexts and variables are considered, we become a little less imprecise. But I'll leave it at that tentative finding now as we take a break. Marianne, thank you. We're going to be back right after this short break. We're with Marianne Bates of MIT who is showing how research makes a difference in solving human problems. I'm Keely Akina. You're watching Ehana Kako on the Think Tech Hawaii Broadcast Network. Don't go away. We'll be right back after this short break. Hey, has your signal just been taken over or am I supposed to be here? This is Andrew, the security guy, your co-host on Hibachi Talk. Please join us every Friday on Think Tech Hawaii. Aloha. My name is Danelia, D-A-N-E-L-I-A. And I'm the other half of the duo, John Newman. We are the co-hosts of Keys to Success, which is live on Think Tech live streaming network series, weekly on Thursdays at 11 a.m. Aloha. My name is Mark Shklove, and I'm the host of Law Across the Sea. And Law Across the Sea is a program that brings attorneys who have traveled across the sea and live in Hawaii or are staying in Hawaii for a time to talk about their travels, where they're from, where they're going, and bring it all together. Because really, we're all connected some way, although we travel across the sea. So I hope that you'll tune in and watch our program. Thank you very much. Welcome back to Ehana Kako. We're here every week on the Think Tech Hawaii Broadcast Network. I hope you'll tune in. If you're watching live for the live stream, that's at 2 o'clock p.m. Hawaii time every week. But we're broadcast across the world, and you can watch us again on various venues. And you can get the information as to where this program can be seen, and many others, by tuning in or looking up on the web, thinktechhawaii.com. Well, we're talking about something rather fascinating today, and that is the role that rigorous research plays in solving human problems, whether they happen to be poverty or homelessness or other social problems, lack of health care, whatever they may be. Research plays an important role, because oftentimes our decisions made in the public sphere, whether by politicians or by voters, are based upon hunches and based upon feelings. And even when they proceed from rigorous philosophies, we really need to see where the rubber hits the road. And that's what Mary Ann Bates helps us to do at the Jamel Poverty Action Lab of MIT, she is of which she is the deputy director. And so we're going to bring her back onto the program right now. Mary Ann, thank you. That was a fascinating story of what took place in Kenya. But could you give us something perhaps a little closer to home here in the United States? How research helped to solve a problem in society? Sure. So one story that's much closer to home is here in the United States started in Oregon. So in 2008, Oregon expanded a Medicaid program that had previously been closed to enrollment. And they correctly anticipated that many more people would want to sign up for the program than they had slots available. And so they decided that the fairest way to allocate the slots would be to hold the lottery. So they opened a waitlist for the program, and then randomly selected among the 75,000 or so people who signed up for the program, randomly selected some who would then be able to have access to Medicaid insurance. And two researchers in our network, Kate Baker at Harvard and Amy Finkelstein at MIT, used this as an opportunity to test rigorously what is the effect of providing Medicaid to a population who's previously uninsured. And the results of that were published as the Oregon Health Insurance Experiment. And the story that I wanted to tell today centers on the contrast between anecdotes, so individual stories of individual people versus overall research results. Well, you know, that whole area of providing health care and health care insurance is filled with anecdotes and philosophies and values and political posturing. So this is going to be very valuable then to see the results of actual rigorous research and testing our hypotheses. Go ahead and tell us what happened. Exactly. And with the study, you know, the study looked at many different outcomes. One that was of particular interest was visits to the emergency department. So hypotheses had been made in multiple directions on what the effect of providing health insurance could be. So on the one hand, if health insurance helps people visit their primary care doctor more, they might be less inclined to rely on the emergency room. On the other hand, if you reduce the price of going to the emergency room by providing Medicaid health insurance, perhaps people would go more, more frequently. And so the experiment was able to test this. And in addition to measuring rigorously overall emergency department use by looking at administrative records from the emergency departments in Oregon, the researchers also did focus groups and talked with the actual people who gained access to Medicaid insurance has resulted this program. And two anecdotes, our stories came from that. So on the one hand, there was a gentleman who had had kidney stones and told the story of having to frequently visit the emergency department previously before he had access to health insurance. He said, look, in emergency departments, my understanding is they can't turn you away. And so I would always go to the emergency department. But now that I've insurance, I go to my primary care doctor more. Right. So that was one specific and real story that came from someone who gained access to Medicaid in the Oregon experiment. And then another person that they spoke with said, well, you know, I've access to insurance now, but I still don't go to the doctor as much as I should. You know, I typically go to the emergency room when something comes up, because for me, insurance is about not having to worry about that big hospital bill. Right. So those are two opposite stories. And one thing that's important to recognize is both of those stories were real, right? This was a real perspective of a real person who received Medicaid through the Oregon health insurance experiment. And the hard thing is that many times, I think in policy debates, very well intentioned, right? The individual stories of individual people are told, whether through the media or in government forums and through other methods. But it's hard to know which of those stories was more common, right? So were there more people like the first story who ended up going to the emergency department less, or were there more of those who ended up going to the emergency department more? So sifting through stories, it's very difficult to figure out which one was more frequent. That's right. And when the researchers looked at the data, they were able to see. You also have a factor to some extent that when those anecdotes come out, we often use the anecdote that supports our point of view. We use it as evidence to say, look, see people, as a result, went to see their primary care physician, provider more or less in order to support a certain philosophy. But what you're giving here is a means by which we can actually kind of deflate that and tell which case actually happens to be the one more frequent. Tell us what happened. Yes. So in this case, the researchers were very and to what you just mentioned that I think it's a natural human tendency to be drawn towards stories that reinforce our point of view. The thing with an experiment is that you can kind of tie your hands ahead of time, right? So the researchers, professors Baker and Finkelstein laid out before they ever looked at the data, they wrote an analysis plan that said, here's how we're going to look at the data, here's what we're going to do, and here's how we're going to analyze the numbers, and we're very transparent about what they were going to analyze before they ever saw the results. And so then when they crunched the numbers, they found that indeed, emergency department use increased significantly by providing access to Medicaid. So overall, visits increased by about 40% among overall on average, the effect of providing access to Medicaid increased the use of the emergency department. And so that was a very helpful finding that helped us sift through the stories and say which story was more common and on average it provided a bit of clarity on that. Well, I think this is so very important because we take actions as the policy makers or as the general public that incentivize people to some kind of behavior, but it's good to find out what behavior it is that we incentivize them to because in the end we end up pouring in millions and millions of dollars and that affects behavior and drives up economic factors and price costs and so forth. And it's good to be able to deflate some of that. Have you seen your research actually used to change policy? Yes, in a variety of cases. I mean, we overall, if you look on our website, we have a variety of stories internationally as well where research was able to shift policy decisions in both directions. So programs were scaled up that were found to be effective. So we found targeted tutoring programs, for instance, effective in Kenya and in Chicago. And more resources, both public dollars and private dollars, were then unleashed in order to scale up those programs and reach more people. In other cases, sometimes programs have not been found to have the effects that were anticipated. And if we find that in multiple contexts, resources can be allocated to ones that are more effective. And so we've definitely seen that. And I think the most exciting possibilities around this, too, are to build more bridges between government decision makers or practitioners and researchers to proactively test programs. So in Oregon, for instance, the state of Oregon didn't decide to hold that lottery for the purposes of doing research. They did it because they thought it was the most fair way to go about it. But there are many, many programs across our country in all of the different states that are being rolled out where there's some constraint. Maybe there's not enough funding to cover everyone who might be eligible initially. Or perhaps a program is being phased in over time, where there's an opportunity, if you plan for an advance, to say, let's build a research opportunity into this. So at the end of the day, once this program is rolled out, we can know a lot more about whether it actually had the effect that we hope it did. Or perhaps it had effects we didn't even anticipate, positive effects we didn't expect. It can go in both directions. That's the great thing about doing an experiment that can surprise you. You only do an experiment if you don't already know the answer, right? Mary Ann, you come from the thinking world, the research world, the think tank world, the world that looks at data and tries to understand human behavior. Have you found it frustrating to communicate to the policy world or to the public? How persuasive is data? Or is it really persuasive in overall and changing public opinion or policy thinking? I've actually been really inspired by many, many leaders, both within government and outside of government, who have found this kind of evidence very persuasive. And I think in particular, with an experiment, the comparison is relatively straightforward. And people can understand, if you share a bar graph that shows, you know, here's the outcome for those who received the program, here's the outcome for those who didn't. And you can see a difference between the two. It's very salient. And the other is that you can build a lot of transparency in from the start. So, for example, when we partner with the government or with a large nonprofit organization, we discuss what we're going to test and how before anyone ever knows the results. And I think that can open up a lot more openness to say, well, what would we do if we find out that this program is effective? Well, we might want to scale it up. Or what would we do if we find out it's not effective? Well, we might want to try something else. So, we've actually found quite a few innovative leaders in government and beyond who are willing to partner on this. Happy to tell more stories on that, but I know we also don't have that much time. Well, you know, I recall back when I was in college, there was a book called How to Lie with Statistics. And the premise of the book is that you can razzle-dazzle people and persuade them if you put enough charts and numbers in front of them because people don't really understand this and don't really have the capacity to see through bad and shoddy research. You know, as you take a look at national public policy issues as well as state-level ones, how much bad portrayal of research or bad research is out there being used to persuade people to do certain things and to believe certain things? I think there's a wide variety of information that's out there and that the terms data and evidence and research encompass a wide variety of things with varying quality. And I want to be clear, you know, J-PAL focuses on randomized control trials and experiments. We don't think that that's because that's the only way to learn about the world. There are many useful forms of research that aren't experiments that can be complementary to this. But I do think that people can intuitively understand if you share, you know, use analogies like drug trials where there's a drug in a placebo, right? And help them understand why on average the group that won the lottery and lost the lottery on average the same, right? Those two groups in Oregon will on average have the same number of very sick people and on average have the same number of healthy people in them. So if you see a difference, you know, it's due to the program. And I think clarity on that can help a lot to combat some of the cynicism around like, oh, well, anyone can make charts and anyone can make data. So I just won't believe anything anymore. I've actually seen a lot of very positive reaction to this approach where people find it persuasive. Well, Marianne, you're doing a great job in closing. If somebody is out there thinking, well, I'm going to my state legislature to lobby for a certain kind of health care insurance program or so forth or whatever they're dealing with. Could they get ahold of you and your organization, the Jamil Poverty Action Lab and propose that to use your services? Is that available to the public across the country? Yes. So I'd encourage them to go to our website, which is povertyactionlab.org. And we actually have an ongoing initiative right now called the State and Local Innovation Initiative, which provides an opportunity for state and local governments to reach out to J-PAL and say, here are the kinds of experiments that we would like to do. Here are the policy areas that are of interest to us. And we can help link them with researchers and make that more possible. Thank you very much. Marianne Bates of MIT, thank you. And if you want to give your legislator an MIT education, take a look at the J-PAL website. I'm T. Lee Aquina at the Grassroots Institute for Think Tech Hawaii and Connecting Aloha.