 We have Jack Culver, visiting from Berkeley, and just so that I can announce, Peter Houghton is here at 3 o'clock getting a talk and then having a conversation in Global Health at 4 o'clock. And that's in the All-Way 106 building. So there's quite a lot of activities today for some reason. We have a concatenation of stars that are visiting Samford. So my introduction to Jack is Jack is a triple degree person. He has an MD, an MPH and a PhD. His MD is from Houghton's and his MPH and PhD is from Berkeley. He did his training at UCSF, but then he came here to do his chief residency. He's actually a chief resident at Samford and then went back and did his ID training at UCSF. He now is a professor of epidemiology at UC Berkeley where he teaches methods and designs of large field trials. He has done at least four triple blindment randomized studies of water sanitation and hygiene and how it affects child health in Bolivia that has also worked in at least five or six other countries. He's worked for the WHO and the World Bank giving them expert advice on how water quality can actually be improved. Today, I actually looked at his CV and was amazed that he's in the midst of now randomizing 22,000 children in Bolivia, in Bangladesh to a water sanitation intervention to see if it affects their health. He's going to share with us today some of the challenges of doing field studies in large populations, so thank you. Great. Well, it's great to be back and I spent many hours on... I don't know how you came back as chief resident. Okay, sure. I spent many year hours sitting where you're sitting listening to various speakers and wondering if I wanted to be a lab jockey or someone doing procedures all the time or what and kind of stumbled into this career that I have which is sort of a mix of infectious disease. I still practice at the VA in San Francisco in the ID clinic or teaching. I teach quite a lot. I teach intensively in the fall each year and then not the rest of the year when I do the rest of my traveling and teaching and writing. And then I do a lot of field trials of different public health interventions. So if you have interest in international health or infectious diseases or these kinds of things, global health, that's something I'm happy to talk with you outside of today's lecture about. So I'd like to describe, to create a framework today, I'll sort of make a Christmas tree on which we'll hang some ornaments about different issues that come up in study designs. But I want you to think more broadly. I mean, if you're not passionate about diarrhea like I am, which is sort of a key outcome I work with, then you have some other outcome that you'd like to measure rigorously. So the designs we're going to talk about today are transportable to many other both field and hospital type questions. So I think you'll see as we go what I mean and feel free to interrupt me because I'll stop and ask questions. So let me articulate some of these challenges first in my next slide and then I'll illustrate these with a study that we did in Bolivia. I'll go through a bit, talk about some of the design and analysis challenges caused by working with big populations. And then time permitting, I can talk a little bit about some of our group's current work in these other countries. All right, so let me, this slide is just to remind me to bring up a couple of issues that should permeate the whole talk. One is on one of my trips to Bolivia, going out to see one of the farmers whose family was a participant in the trial I'm about to tell you about. Who wanted to show me a new latrine that his family had. An NGO, a non-governmental organization, had come through and built thousands of latrines in this section outside Cochabamba where we were working. So, you know, as is often the case, an NGO can put up on its website, built our latrines, health is better and so forth. So we're walking out to the latrine, the farmer opens the door and it's filled with potatoes from the bottom of the pit to the top of the roof. So it was never going to be used as the way you might think a latrine was going to be used, but it was a great place to store potatoes. But nonetheless, the NGO can come through and build its latrines and leave its mark and people can do what they do. I've had other similar experiences coming on the heels of CDC interventions done a year or two before I arrived to do my work where drinking water storage containers that were built for safe storage of water now serve as the soccer goalposts for the village. So we do lots of things in public health and we assume that kind of the process outcome of having done them and checking them off is enough. And in fact, what I and a number of other people, actually a lot of economists are really getting into are things called impact evaluations where we really try to see, have these big programs made any measurable difference with respect to health. Now, of course, measuring meaningful outcomes, diarrhea, child growth, cognitive development children is much more difficult than counting latrines. I'm going to talk a little bit about the issues that arise from whether or not people participate in your studies because compliance, as you'll see in the study I'll present, is a really important and big issue. You should challenge me, as you should any speaker, on the ethics of what we've done in this trial. Was it ethical to do the trial we did it? What do you think about our comparison group? Was that in violation of a number of guidelines you've probably studied about performing studies in the developing world or not? What's it like to work with multiple disciplines? This work involved engineers, anthropologists, economists, behavioral scientists, and you can just imagine what that's, I mean, you know it's like to work with surgeons and radiologists, right? So this whole mix of people with different disciplines creates its own kind of microculture. Another generic topic that applies to any sort of work that involves cluster, that involves cluster design. So in the work I'm going to describe today we're going to talk about randomizing villages. You might be randomizing wards, or you might in large scale studies be randomizing hospitals. And when you use a design such as this it creates methodologic issues about contamination and spillover. Nothing to do with water contamination, but a spillover of the effect of your intervention and its effect on potential outcomes in the different units that represent the clusters that you're studying. It creates sorts of difficult issues for sample size estimation, sampling, and analysis. What happens when you go on your happy way in your dissertation or your project and you carry out a big intervention and then the government comes along with some other co-intervention that you have nothing to do with, but it totally addresses the outcome you're interested in and sort of swamps the area. You're doing a water sanitation intervention and some other organization comes along and builds its latrines in the same village. How do you take care of that? Lots of behavioral science lurking in what we're doing today. You may or may not have interest in that. We can certainly talk about it. The study I'm going to describe today was not a blinded study, so that should raise some methodologic issues for you. I do do blinded studies as well. Those are more difficult. We can talk about what one might do to aviate the need for intensive blinding in a cluster randomized trial such as this. Another theme I'd like to leave you with today or have you think about is that when you do studies such as this that the people who are measuring the outcomes of the study are different, should be different, from the people who are delivering the intervention. There's just a natural human bias built in. If I tell a health promoter to go and promote solar water disinfection as we did, there's a natural bias if that person is recording the data to want to think that the people who got the solar water disinfection treatment are better. So they're going to shade unintentionally usually, but shade their recording of the data in ways that may influence the truth about what you're trying to find. So separating the implementers from the promoters is an important theme. And then the need for hard outcomes. So although the study I'm going to talk about today was powered and measured on diarrhea, we should talk about other outcomes that are possible in this sort of field and in others. Alright, so let's talk about a specific trial to illustrate some of these points. So the technique or technology I'm going to talk about is called solar water disinfection or sodas. And it's almost achieved status as a religion. So people in the field of water and sanitation believe deeply in solar water disinfection. Because as you're going to see, how could this not be a good thing? It's so sensible, sustainable, all those other S's. Sexy, you know, it just works. It has to work. And it does work. In the laboratory, if you treat water with enough sunlight in clear bottles, it will eliminate most of the pathogens. Now, not all. Can you think of a particular pathogen that's a waterborne pathogen that might be difficult to eliminate? It's really difficult to eliminate in municipal systems too. Cryptosporidine is tough. So cousin of Giardia, really tough cyst, etc. So it's not great for cryptosporidine, which is something we deal with enough, but it's not the key issue. So it does leave a few holes in the coverage, but it's free and it's widely available. And the technique is to expose bottles of water to sunlight for a day. I'll show a little bit more about that in a minute. And then the family usually has two sets of bottles. They put one up on the roof one day and then bring it down and drink it. And then the next day the other bottle goes up. And it's not just the radiation itself, the UV radiation that's treating the water. It's also the heat. So at higher temperatures, it requires less radiation. So this synergy goes on between the two treatment modalities. All right. So here's a study you've probably seen, I think from BMJ, about whether we need to do randomized studies to study everything. So a lot of people in our field have the view that, you know, we know that good drinking water is a good thing. Why do a randomized trial? That's kind of a silly thing to go out and spend, in this case, $3 million doing. So this study was called Parachute Use to Prevent Death and Major Trauma Related to Gravitational Challenge. And it's a systematic review. You're all familiar with systematic reviews. And the expensive conclusion reached was that parachutes reduce the risk of injury after gravitational challenge, but their effectiveness has not been proved with randomized controlled trials. So lots of things in life aren't going to be subjected to randomized controlled trials. We're not going to take residents' post-call and assign half of them to go driving in a dangerous situation and half not and then see whether or not there's more traffic accidents, right? There's just certain things that must be common sense. But the issue here, looking in the solar water disinfection issue, is whether or not people actually use water treatments the way they're supposed to use them. And that results in better health or not. So things can work well in the lab. We all know this from drug trials and everything else. And then you take them out into a population and not so much. Okay. One common problem in international development is this thing I alluded to earlier about this mismatch between implementation outcomes. Some people call these process outcomes. The economists call these first stage outcomes. That in some causal sequence, one has an idea about the different steps that should happen in the intermediate steps of a process. So if I provide solar treatment to water, I should see an improvement in water. And then I should see downstream from that the improvement in health. We're coping with a big World Bank study at the moment where we got some amazing improvements in health in a big campaign in seven million people in six different countries. Big improvements in health, but we couldn't show any improvement in the intermediate steps of the water getting better. And then as we look more deeply into the data, we saw that many outcomes other than diarrhea got better for things that shouldn't have improved with a water intervention like bruising and abrasions and things. Anybody know what those alternate outcomes are called? You ought to think about these in studies you do or read. So you have your main outcome in a study, but you're also interested in other outcomes that might have nothing to do with the intervention. And if those got better in your intervention group, you have a problem. So what are those other outcomes called? Important term to know. Those are called falsification outcomes. Have you heard that term? Some people call that negative control outcomes and so forth, but falsification outcomes is a term I think is most descriptive. So again, not just in the issue of diarrhea, but if you're studying malaria, I work with a lot of people who spend their careers counting bed nets, but the measurement of malaria is really where it's at, but again, it's much more expensive, harder. People who work on HIV infection count condoms distributed, but we really want to see changes in HIV rates. All right, here's the sodas process. So clean bottles which are widely available anywhere in the world, and I know there's a pointer here. Bottles are cleaned out, filled with water, put up on the roof. Where we worked at in Bolivia, we often had people with kind of tin corrugated roofs. So if you have a reflective surface behind the water, all the better, it works faster. It'll work without that as well. Water goes up for four to six hours, again, depending on how warm it is, and then it's ready for consumption. Just to mention that a common and very effective way to treat water in much of the developing world is through boiling. Boiling is extremely effective, and it also gets our friend cryptosporidium. It's really quite a powerful, effective technique. In urban settings, I work with colleagues in DACA. They show me lots of gas lines out in the slum areas of the town where people, you know, pirate gas supplies in order to boil their water. So this is widely known. But out in rural areas, there's often not enough wood or fuel to supply the fuel to boil the water, so it's not such an easy thing to always do, even though it's very effective. But we also have this plague of soft drink bottles around the world, so these soft drink bottles we're all very familiar with can be used for the sodas treatment. So I'm thinking, of course, why not harness these? This is from that church of sodas that I mentioned, this foundation that works worldwide to promote solar water disinfection. And these are just some different kill curves to show what degree of either heat and or sunlight and radiation and heat combined that's necessary to capture some of the more important organisms we care about in contaminated drinking water. And here, just a reflection of this issue about synergy that at higher temperatures things kill off quickly. I'll make the point that it's very expensive to measure all the individual pathogens we care about in drinking water so the field has pretty much resorted to using indicator organisms and then, of course, like any field, there's debates about the best indicator organisms, whether you use fecal coliforms or total coliforms or E. coli. All of them have advantages and disadvantages. Some of the disadvantages, for example, total coliforms, lots of animals shed coliforms, so you're never certain if it's animal waste contamination. It doesn't sound too tasty to drink either, but it's really human waste contamination that causes the main problem for humans developing illness from drinking water. So, again, the promotional campaign, what could be wrong? When we started this study eight or nine years ago, when we first put in the first proposal, the program had spread throughout the world. Many countries were doing it. Based on the results of a couple of small studies, one in particular in Africa among the Maasai that had shown about a third reduction in GI illness in tribal members who use the sodas technique. So, the elders were taught how to teach their people the sodas technique. They went out and taught the technique. Diarrhea went down in young children under five by about 33%. And several facts I just mentioned there are going to be relevant as we come to some sample size and other issues. I think I've hit on most of the technology advantages of sodas so far, so let me skip over to the... I'll just make one other one though is that the big push in much of the water and sanitation field is towards providing people with individual control of household level drinking water treatment because you're familiar with the Millennium Development Goals, for example. We've heard these that by 2015 all these great things were going to happen. One of them was that in the drinking water field that there would be a 50% reduction in the number of people receiving unimproved water in the world. And even that statement has a lot of problem because what counts for the joint monitoring program at the UN that follows water, what counts as improved water is delivery through pipes. So you can deliver sewage water through pipes and that would count towards those MGG goals. So obviously a little silly back to this issue of what are we really measuring, what do we really care about is causing an improvement. Some other limitations of sodas. It doesn't treat chemical problems with the water. I work in a lot of areas, for example, in India and Bangladesh where arsenic is a problem, other areas where fluoride is a problem. So this does not do anything for that. In those settings, sometimes people combine sodas with a physical filter filtration type thing. This isn't useful for treating large volumes of water. It's meant for household level treatment. It requires relatively clear water because the sunlight needs to penetrate the column of water and get to the organisms. And if it's very turbid, the organisms either never feel the sunlight or they cling to the solutes that are in the water and escape treatment essentially. All right, so here's the study I'd like to motivate our discussion with. This was an NIH-funded project we did in Bolivia, in children under the age of five. Obviously choosing one's target population is critical. Obvious question, why would one want to work with children under five in studies such as this? I mean, diarrhea is a problem for lots of people. Yeah, so mortality is higher. Now, one important point is it's really hard, even in large studies, even in my largest studies, it's hard to use mortality as a primary outcome because the study has to be so big to power it properly. So in addition to mortality being higher, incidence is quite high in children under five. And that's why you always go look for an outcome with a really high incidence, regardless of what the study is you're doing. We picked rural over urban Bolivia just because we didn't want to create this kind of stratified sense of if we saw different results in rural versus urban, we wouldn't know what that meant. This is an issue for everybody doing urban studies in urban populations around the world. There's intense migration, so it's really hard to do urban studies and know in a long study that you're going to have the same population after you've enrolled them, you know, a year or two years later and so forth. We had 22 communities with 660 children. They were randomized equally to sodas versus current practice. Who's going to challenge me on the ethics? Current practice is they're going to keep doing what they're doing. You should have a problem with that. Is that an ethical thing to do? You've heard of the HIV trials where pregnant mothers were treated and another group was randomized to current care, which was no treatment. Those trials were stopped, created a huge ethical debate. I hope you've been engaged in at some point. Anybody see an ethical problem? I'm okay with ethical problems. This is my life here. So what's wrong with this study? I can tell when I'm talking to clinical audiences. Yes, in the back. So that's a great side question. After we did the study, another study was published unrelated to ours dealing with the polypropylene phthalate that is the coating of the bottles. It does leach in fact, and in lab rats exposed to 10,000 times of human dose for a year, you can get a mesothelioma kind of thing. That wasn't an issue for us at the point, but that's a good question. Is the treatment itself a risk? Let's assume we're cleared of that because that was nowhere in the literature when we started. That's a great methodologic problem. Current practices could vary in the different clusters. Fortunately in this study they didn't. We weren't too concerned about that, but certainly you do worry when people are just doing whatever as opposed to giving them, you know, we're not giving them a placebo to take. So the ethical challenge that we would often face here, anybody? Yes. That's right. So we have these people doing current practice which is to consume their water, literally. I don't make this up. I mean they go outside their dwelling and they reach into the ditch and they take water out and then take it back and feed it to their children. Hard to believe that that's a safe practice, but that is current practice. That's what they do. So can you think of a way, then how could you do a controlled study? If we agree that that's an unsafe practice and if we agree that investigators shouldn't be promoting or encouraging or enrolling people engaging in unsafe practices, what's the out here? So here's the way we did this dance and I feel okay about it, but it's certainly open for discussion and challenge. The NGO we were working with, Project Concern International, was not able to do all the villages they wanted to do in the year we were going to do the study. So they could do about 10 to 15 villages a year. What we did decide to do, and I think this has application to lots of programs in the developing world, we asked them to introduce randomization into the villages they were going to enroll in their first two years. So they were going to take essentially 10 villages one year and 10 villages the next year in the program. We asked them to do that rather than the way that it's normally done, which is, you know, my cousin is the alcohol day of this district or whatever. We go first to do it with randomization and in fact we found that people actually preferred to get an NGO program through randomization process. We had a big public randomization process. We had bingo balls where we had the communities paired into treatment control and it was randomly drawn who was going to be treatment, who was going to be controlled. So we found that people actually preferred randomization to just assignment in the way it was typically done as these kind of programs would usually come through. The reason I say I think this has broader applicability is a lot of work we do with other organizations that are rolling out big campaigns, there's the potential to introduce randomization into what they're doing with just a little bit of thought and scientific presence. So a large part of what my group does now is try to convince NGOs that are rolling things out that hey, you could introduce randomization in this, cover the same number of people and yet then have a study that has some scientific inference from which you can draw perhaps some valid conclusions. Okay, another methodologic technique we used here that's widely applicable to any kind of work you might do on award if you're doing individual studies or cluster studies is we pair matched the villages on baseline diarrhea. So pair matching is a kind of a whole separate side village or side thought. Let's step over to that thought for a moment. Why might one want to pair match clusters or pair match the units of randomization even in a clinical trial? What's the disadvantage of pair matching? What's randomization supposed to achieve on its own? Decrease in bias of what causes bias when you don't randomize? Sorry? Confounders? Yeah, so here's an extreme example. I do a study of some drug for prostate cancer and I get a group of 1,000 men in their 30s and 1,000 men in their 60s and I give one group the 30-year-olds I give the drug to and just prevent prostate cancer the 60-year-olds I don't and then I report, well, there was no difference and it didn't work, whatever. But what's unbalanced there? Of course it's the age. So if I didn't have randomization to equalize the age and all other factors that I care to be importantly different between the groups then I've created a group that really can't be compared to the other. So randomization is really critical in studies that are going to be doomed if there's an imbalance and an important variable it's critical to try to balance those variables of a baseline even beyond randomization and pair matching is one way to do that. So if you know your key outcome by, you know, if I flip a coin a thousand times no one's going to be surprised and I get 400 heads and 600 tails but that might be too imbalanced even though it's random that might be too imbalanced to do the study I want to do so I add something extra to ensure this balance beyond randomization and that one way to do that is with pair matching we matched on baseline diarrhea could you think of other things one might match on in a study such as this solar water treatment for reducing diarrhea or why match on diarrhea obviously we thought that was important enough covariate to mention what other variables might be of interest here also, okay, so you could match on environmental factors you could match on differences in the village structure differences in temperature whatever however we knew that diarrhea was one of the things that most strongly predicted future diarrhea so by matching on diarrhea it's an attempt to make the baseline conditions which is what you want in the trial make the baseline conditions as equal as possible we chose a sample size that I showed you earlier to detect with 80% power a reduction of 33% in diarrhea between the group so if the rate of diarrhea and the groups before any intervention is 10 cases per child per year we'd be able to see something at 6.7 or lower per year 33% reduction now why 33% reduction where do you think that comes from how do you make sample size estimations in studies before you start prior studies so I had told you earlier that the prior study in the Messiah in Kenya showed a 33% reduction remember that so turns out that's about the the largest reduction smallest reduction we could afford to power the study for too for our budget the NIH has a fixed budget for studies such as this so you have to find and out you know you have to shoehorn your outcome and your sample size and your effect size all together in this magical mix that makes your study fundable so if we couldn't afford to do a 33% reduction we might have had to pick a different outcome with a larger reduction what do you think is it harder or easier to do a study to detect a larger effect so if we had wanted to detect a 50% reduction would that have required more participants or fewer fewer it's easier to see large differences right if I'm looking if I wanted to declare a heavy person different than a light person the greater the discrepancy between the two the easier it is to pick that up if they're you know 16 ounces different pick that up easily it's the same thing in these large studies the larger the difference you're looking for an effect size the easier it is to power the study to see the difference okay so we enrolled the 920 children on average there were about 37 children in each of our village units that's another important point we randomly sampled the children to be in the study in each village we couldn't afford to enroll all the children so the clusters create this sampling frame in which we sample randomly the children to participate in each in each thing I mentioned that we pair matched people on baseline diarrhea that baseline diarrhea was gotten from a eight week study before the study started to measure what diarrhea was in each of the villages and our process created very good you know kind of table one balance of of covariates of the villages you'd like to see in a randomized trial that all the things you can think to measure because you're hoping that all the things you didn't think to measure also are balanced in a randomized trial so we worked in an area called Tatora right outside Cochabamba right smack dab in the middle of the cocaine trail that whole industry was a big part of the area in which we were working so that created all sorts of other secondary issues in conducting the study I talked about spillover and contamination earlier I can tell you these are quite far apart quite isolated we had really no concerns about common markets or common shinnering of water supplies and so forth so we weren't too worried about contamination spillover here's this pair matching process at the beginning these are the community cluster numbers and so for instance these two communities are paired because their baseline diarrhea is the closest so on the intervention group so it's random who becomes the intervention group in each pair so the ramization is done at the pair level lots of different things are collected during the course of the study one important point I'll make is that while sodas as it's being rolled out worldwide now is basically back to the parachute idea sodas is parachuted into these villages in most parts of the world people are trained once left alone we didn't do that we were trying to conduct an efficacy trial here as opposed to an effectiveness trial so we had our promoters going every two weeks to the villages throughout the course of the year that term efficacy effectiveness a common parlance for you what's the difference how is this efficacy not effectiveness or it's a spectrum you know efficacy anybody what's efficacy is like in the lab if I control all the conditions and it works I've shown that something's efficacious whereas if I take it out into the real world and sort of stilted I mean sorry in non stilted or in real conditions we call that effectiveness but there's a spectrum between it so you could argue our study sodas in the lab that was kind of an efficacy study but not of health our study was we think the best you can do for promotion of sodas in the wild so we view it on the study simplified down so diary incidents let me first tell you what these outcome measures meaning diarrhea incidents refers to the number of episodes of diarrhea per child per year and the way WHO defines diarrhea as three loose stools in a 24-hour period and the way we define episode which is common in literature is an episode is a new occurrence of diarrhea that's more than three days separate from the original occurrence in a row our presumption is that's the same pathogen we're not treating that as a different episode so episodes are separated by three days of no symptoms as opposed to prevalence or in this case term would be longitudinal prevalence where we're measuring the number of days a child is ill in the year divided by the number of days they were observed so as expected you would see more the number the absolute number of longitudinal prevalence would be higher because you count up days there's going to be the business of needing to be separated by disease-free days this is a plot of the longitudinal prevalence over time in the study and I'll make a point that in many human health population studies you see this dramatic fall in incidence of disease early in the study in both the control and intervention groups lots is being written about that now our group is working on some of this methodological underpinning of this but we think it's kind of a type of Hawthorne effect so let's look at the results and interpret those together so in terms of episodes of disease the control group went from 4.3 to 3.6 and that ratio of those is a 0.81 with this confidence interval anybody want to interpret that give up on SOTUS continue SOTUS promote it globally sorry but it seems like no effect okay and you say no effect because the confidence because the confidence crosses the null which is right okay so let's work this through so if the confidence interval crosses the null we say that we don't have evidence at the level of evidence we'd like to see that the treatment worked now we're talking about big states here right global childhood diarrhea what's our I mean in our gut what's our best estimate of the effect of SOTUS forget confidence intervals and p-values and if you had to pick a single number to represent its effect its reduction or increase in diarrhea what would you say it is yeah 0.81 now clinically would that be meaningful or not if I could get a 19% reduction in childhood diarrhea in the world do I have to argue with you that that's meaningful that's huge 4% reduction diarrhea would be huge in terms of life saved and growth promoted so we're left with this bit of a dilemma that we went through this big study came to a conclusion we can't prove it with the kind of level of trial certainty that we'd like to have say for a new drug but we're not talking about a new drug so as you can perhaps imagine what came about after article was people who believed in SOTUS believed in it even more strongly and they said at Colford group what they did wrong in their study was what because 0.81 were to be the true estimate let's just in a different parallel universe 0.81 what would turn the confidence interval into a significant finding more power what would I need for more power I need more people you can't but who can guess at how much larger the study and the budget would have had to have been if 0.8 if we had powered the study for 0.81 instead of what we powered for remember click click back the earlier slides what did we power 33% reduction so that would have meant 6.7 here or lower but let's say we went into the NIH and said we're looking for a 0.81 19-20% reduction what does that mean how much larger would have had to have been 2.5 times larger so now it's a 10 million study so this is tricky because if someone else is going to go out and replicate this now they're going to need a study that much larger to do this and let's just look at the consistency or not of the longitudinal prevalence measure that happens interval crossing 1 some reduction so hard to know for sure yes could you argue per child year in 17 prevalence you know on one hand you can argue that's not a lot yeah that's a very that's a very sophisticated question and let me make a point do you remember what we thought the baseline prevalence was going to be when we did our baseline calculations I talked about what did we think diarrhea in the wild was anybody remember it was about 6% so this is a 30% this is already so if you want a 33% reduction what you want to do is get my group to come and do a study because then diarrhea is going to go down by 33% before the study even starts and this happens all the time in lots of human health studies when you go out into a population to do a study all of your assumptions and even pre-measurements often change now whether this is because the people who are in a study are different like the kind of Mr. Fit Lore and this sort of thing or just bad luck that's it that's great let's do it yep that's going to be on a list of what could have gone wrong here that's great that's the next slide good question other questions about the results what's this p-value of .19 are you all sick of interpreting p-values or the statisticians beat this into your heads already anybody what's a p-value of .19 how does that relate to that .81 right it has nothing to do with the .81 it's going to just lead you it's often a good first-year epi question on a test but what's that p-value telling us what's the .20% chance that you see this yeah so I'd say a little differently there's a .19% chance that if we did this study over and over again we could see a result this extreme extremely different but the null would still be true as you started off that's here alright so somebody asked about compliance and whether people were using it and by compliance there's an issue not just with the group that got the solar water treatment there's also concern about whether the group that didn't get the solar water treatment started doing solar water treatment because they were watching people get solar water treatment so luckily I don't have any of that up here about the control group but we have it in the paper and I know that number 20% of the households in the control groups were using sodas at points during the study so quite quite low but now these data all reflect the active group the treatment group so let's talk about human nature so we measured sodas use in a couple of different ways we went to the people right after they had gotten trained all excited lots of community campaign energy you're going to be and a couple of weeks into it we asked them are you using it how's it going you like sodas so here's their response about 78% of the people reported were using sodas so that's a little disappointing but it's not 100 but it's there and just jumping to the end of the study the same amount reported at the end of the study and this was a question framed as did you use it in the last I think two weeks were using sodas but sly investigators that we are as we were in their houses the promoters were trained to do a number of different things they were trained to ask for a cup of water and they watched where that cup of water came from often they declined that cup of water when they saw where it came from because often it didn't come from the sodas bottles they also went and looked for sodas bottles and recorded where they were so there were a number of kind of passive measurements about whether or not sodas was being used compliance let's look at how that compliance compared to self-reported compliance and I'll just make the point I can guarantee you this if you go to more often than not self-reported data are used but if you don't leave the talk today fearful of self-reported data then I have failed so here are the sodas bottles in the kitchen that's down in the 20% range so on average over the whole study over all the weeks of the study it worked out that we had about a 31% compliance rate in terms of weeks that the bottle should have been in use versus weeks that the bottles were in use so one third of what happened so now the story gets even more complicated so if this 19% reduction is true and it comes from such low compliance is this a compliance phenomenon so can you think methodologically or more analytically how could I take the data that you know I have and look at whether or not compliance is related to effect what's the effect reduction in diarrhea so what am I going to correlate here what two variables am I going to divide plot what could I do yeah so I could plot compliance in diarrhea I could do that as a continuous measure or I could divide the groups which is what we did to have enough to analyze I could divide the groups into levels of compliance and then look within each group to see whether or not are the people this is very commonly done are the people with really high compliance was the reduction in diarrhea greater among that group than it was in the people with very low compliance slopes were the same no difference in the mean so does use and the outcome in the different groups based on compliance yeah yes was there a different number of bottles given to each household depending on number of children that's right that's right so that was all calibrated based on the number of children in the house and so forth right okay so any other questions about compliance anybody give people drugs where compliance is an issue yeah so just think about self-report think about compliance very generic okay um lost track of time there so you have 10 minutes I think yeah so I I'm going to pause for a moment just open it for any conversation about the Bolivia study and then I'll I'll talk about some other work we're doing and some of the other challenges that raises yes oh that's a great that's a great question there is variation in the seasonality of the water in that here in the Bay Area maybe much to your surprise when there are lots of rains water quality can change because there's lots of flushing that occurs from sources on the ground that get into the water system so yes in in times of heavy rain there's a change in the water quality luckily in our randomized trial we had controls throughout all the various seasons so that wasn't an issue for us here other other questions about Bolivia compliance ethics yes were the villagers enrolled encouraged to find some the reason I asked yeah why if it's so cheap and easy all the villagers knew about it because of the randomization that didn't go yeah why didn't they just do it have you ever tried to give somebody cholesterol drugs or daily aspirin or pathology oh hopefully not but it's too late when it gets to you human nature is just tough in that there are lots of things that should be easy and should be doable but ours is not the first in fact it's one in a line of ten thousand studies to show that people often don't do what they know to be potentially good for them so the behavioral scientists on our team work on these issues about how do you promote more now what I will say is that our promotion of sodas that I alluded to with that earlier cartoon our promotion of sodas was you know I'm gonna say ten times more than the way the sodas campaign is traditionally done which is to drop off so we could not humanly have done any more than every other week I mean well you couldn't tell I showed you how remote those communities were so getting to those 22 communities in these little valleys and mountains you know this was not an easy task for the promoters to get out there so I don't think we could have done any more on that on that thought but it was certainly something we thought of other questions yeah Michelle it's hand washing so hand washing was not part of this study it's part of other studies we do so with a variable like that in a trial you assume that there isn't an interaction between the intervention you give like clean water isn't changing people's washing with their hands I mean the causal path would go something like that like this what if they got clean water and now they're more attentive to these issues so they also wash their hands or reduction a greater health benefit which we didn't see so either it didn't happen or it didn't matter you wouldn't necessarily think that the people doing what they were currently doing would change their hand washing at all so this is we call this a conservative bias in the sense that we are we are likely underestimating the effect which is what you tend to want to do when you're trying to prove something new works or doesn't yes sure boy this is tough this is a big dilemma in our field because certainly because of the human behavior issues centralized structured systematic I should say systematic interventions generally work better you know for instance in the US there's someone here on the Stanford campus Grant Miller who's tracked how typhoid plunged as sewer systems were built public sewer systems were built across the US you could just and they didn't all come online at the same time so you could see this nice sort of time series so ideally system interventions are better but they're expensive they're hard they're hard to get to that last mile to the people out there so that's why in the past several years this household level focus has been kind of where the action's been at because no one there's really I don't think there's much scientific debate about whether a pipe system can be better this is like Milwaukee where they have a pipe system and someone flips a switch the wrong way and the system gets loaded with cryptosporidium and you have 10,000 cases of cryptosporidium back in the early part of this century so it's a tough dilemma so this is meant to give people who care to do something about it household level individual control are there questions? yes was there anything on how that comes with so this and the other ones so just published recently a group from Ireland that we've now written a letter of discussion with them in the journal about but they are claiming to show an improvement in child growth in a group that got sodas in a completely another study that just finished they did that though it was a randomized trial but they did it with modeled data and they don't show us what the actual results were so this is always kind of for an epidemiologist makes us a little twitchy that if I can't see the raw results I don't really know whether the model is doing something that the real data aren't and it's just kind of bizarre not to show your raw outcome so that's a active debate our group is having in the literature now with this other team that just came out that's Ronan Conroy and Duprize is the first author other questions things we're going to measure in upcoming studies I'll just say a little bit about maybe well let me talk about another set of studies that I'm working on with would I use sodas? I think if I if it personally if it were the only water treatment technology I had available to me in a situation I would use sodas would I recommend it for promulgation around the world I'd have to say honestly the evidence doesn't yet support that because these things come with a cost should sodas be done in preference to boiling should it be done in preference to other you know should be done in preference to vaccine if you're a health policy planner people like to use how do I as a minister in the government make a decision between my water people asking me to do something versus my vaccine people asking me to do something versus my nutrition people asking me to do something and so forth so in other studies we're doing we're combining water interventions with other health interventions as well to see whether there's synergistic effects you know there's a lot written in the HIV literature about there's a fellow at at UNC who works on this who's on the NIH study section for HIV Blinken and his name but big push about drinking water in HIV patients in Africa improving their response to antiretrovirals so that that sort of I think that's the next generation of water and sanitation studies is how does water and sanitation relate to relate to other synergies with other diseases so forth in our work in Kenya in Bangladesh that Michelle alluded to we're actually doing not just diarrhea we're also doing child growth child growth is actually much harder to measure than you might like to think we like to call it a hard outcome but measuring squirmy children with these stadiumeters you have to use to measure them is not an easy task that's really difficult so that's another outcome we're also measuring cognitive development two and three years after the intervention a little side story I have a colleague who's now a professor at Berkeley who has his dissertation work wow 20 years ago did a cluster-randomized deworming trial in Kenya and they had the foresight to gather very good this would be four cell phones to gather very good follow-up information on how to find these people so now with the NIH's help they've gone back and interviewed I don't know like 85% of the people who were in that study so all these people had that we know was different they were randomized to a school program for deworming you know just mass albendazol whatever the anti-parasite dejures versus the school clusters that got nothing and now 20 years later there's a striking difference of economic gain in the group that got treated and they've tracked these people down to the UK and all the places people from Kenya have gone so one of the tough things about these long-term outcomes that we all care about like school attendance and so forth and educational attainment economic attainment they're so hard to measure and they take so long to measure that people's careers have often moved along for the measure even now we're just getting head start data that's quite interesting you know from years ago when head start happened and the educational people are quite excited about in some cases some of the longer term findings were things that didn't work initially with head start that now look like they may have worked later and the only thing that was different in some of the children was the head start intervention other questions I think that's enough time to do two more sites so I'll stop there oh I think so Rebecca is being chief resident here? most do more do more the preceding program is copyrighted by the board of trustees of the Leland Stanford Junior University please visit us at med.stanford.edu