 I generalize the title a little, I'm going to talk about medical research because that's what's wrong with the literature is that the research is not so good. I'll describe to you some of the problems and one of the problems of course has to do with statistics and I'll do a parlor trick, I'll show you how you can be confused by how statistics is stated and then I'll show you how you can get your mind straight if the problem is presented well and then I'll go through a article in the medical literature claiming that red meat causes diabetes and if you need it further proof that that doesn't make sense I'll provide that. Okay so medical research well there's a lot that's wrong with it I'll just list on the screen some of the things that one could say about it. We've heard a few other things today about what's wrong with it. I didn't say this I doubt that I would say it in print in a medical journal this is from Richard Horton who is editor-in-chief of the Lancet. So if things are really that bad he published a famous editorial describing a symposium in which he said that much of the scientific literature maybe half may simply be untrue and it's of course a little surprising since if it's bad he was it was on his watch. The Lancet is very important but he's not the only one probably the most famous critic is Johnny Unidis who similarly said that the PPV exceeding 50% is not right. The PPV stands for positive predictive value and what that means is when you if you come up with a number and you compare it to what is the true number by some independent method that it's true. So what what went wrong and who's to blame. The experts are saying the literature has 50% errors the 50% is not reproducible so nothing's really being done so is this okay why isn't something being done. So I'll tell you a joke a guy goes into the grocery store and says he wants to buy a half a head of lettuce. The clerk says I can't sell you a half a head of lettuce and the customer says sure you can go ask your manager so he goes in the back to see the manager doesn't notice that the customer is following him and he says to the manager there's some jerk out there wants to buy a half a head of lettuce and then he sees the customer and he says and this gentleman would like to buy the other half. The problem is that they're talking in generalities and they're not naming names so of course if you say that the literature is half bad everybody thinks it's the other half and they're doing everything okay. So I'm going to try to describe some of the problems that we have and statistics is at the heart of the matter and the hard idea to understand is that statistics is our servant it's not our master and statistics per se is not directly part of science there is no F equals ma there is no core observation from which everything emanates if you do statistics you're making assumptions you're proposing an idea and if the statistics is good it will line up with your sense of the science or your sense of the biology but by itself it won't tell you the answer. Group statistics in in particular hides information everybody knows that average is mislead my favorite example is still I got this from Paulus's popular book on statistics the average resident of Dade County Florida is born Hispanic and dies Jewish. You know you can't use that as such and relative values are equally misleading so if Alice has 30 percent more money than Bob is she rich well you don't know they may both be on welfare. Okay so my point here is that statistics should be simple in the sense that an idea I want to project to you is that you understand statistics you know you know you're familiar with certain things that you do in your daily life and when they make the statistics too complicated you should be suspicious. So everybody knows what probability is at least once you point it out to them they know the probability of winning a game is the ways to win divided by all the possible ways so the probability of drawing an ace your four aces is a 4 out of 52 or 0.077 or you can describe it as you have a seven percent chance of drawing an ace. The probability of drawing the ace of spades however there's only one of those in the deck and so the probability is 0.019. Odds are similar but slightly different the odds of it an event are the ways of getting that event divided by the ways of not getting the event so it's the ways of winning divided by the ways of losing so that for example the odds of drawing an ace is 4 over 48 because the aces are not counted so it's very similar to the probability. Where I'm going with this is I'm trying to get you I'm gonna use all of the odds and probability measurements as equivalent because in the situations that I'm going to talk about they really are equivalent. Now the odds ratio is the key thing because the odds ratio the odds of drawing an ace compared to drawing it the ace of spades you know is what what is your chance if you needed an ace or if you actually needed the ace of spades is four to one you have four times greater chance of pulling an ace than you do pulling exactly the ace of spades but notice what's what's changed here you lost some real information because I calculated the original odds on the basis of a deck of cards and if you go in with this information to a blackjack game you're going to be in trouble because you don't know the actual odds all you know is the relative odds if the if the casino was playing with four decks the odds ratio is the same but the actual odds are going way down and relative risk is similar that risk is the same as probability and those are similar now what I want to tell you is the hazard ratio in what we're doing is the same thing it's actually the odds within a fixed time period but again for what we're doing it doesn't matter so that's what I'm just telling you so an odds ratio of 1.5 means that you got 60 40 in your favor and but as we said before the odds ratio doesn't tell you about your real odds so good benchmarks that people always use is Bradford Hills study of smoking and lung cancer and your chance of getting lung cancer if you're a smoker compared to a non smoker is 20 to 1 and it's 30 to 1 if you're a heavy smoker so we're going to try to deal with something in literature because what I'm going to tell you is people ask me I'm an editor I was the editor of a journal and I review a lot of papers what do I look for in analyzing a scientific paper well first of all I look for the pictures they're of course called the figures but I think if you can't make a picture something's wrong so the conclusion they got from this is that red meat consumption particularly processed red meat is associated with an increased risk of type 2 diabetes but before you read their case you have to know that statistics is not a science what I said at the beginning is that you have to bring in your knowledge of the biological system and if it if it the statistical result violates your biological experience you better ask for very strong proof if you say you can jump over the chair I'll cut your a lot of slack if you say you can jump over the building I got to see I got to see you do it so the bottom line is that the best statistical test is the eyeball test if it looks like crap it is crap and this is a figure showing that that red meat consumption went down drastically real numbers and the rate of diabetes went up in the last 30 years so it's unlikely that they are correlated so let's try to deal with the paper on their terms let's look at the statistics so this is what you see when you look at these papers and it's a little discouraging that there are three three separate groups called cohorts and they're broken up into each group is broken up into quintiles that means you know they take the people and order them according to red meat and they put them into five groups progressively but you look at this I have the same reaction that you do and that is that they're trying to snow me okay you got to tell me what's going on here so I'm gonna do the parlor trick and there are at least a half a dozen people did this on YouTube and I'm gonna do it again because it's it's great it works pretty well and it's actually the result of the experiment is described here physicians were presented with a problem and they mostly got the answer wrong so I'm gonna give you the present the problem to you so the probably this is a mammogram test and probably the woman has breath breast cancer is 0.8 percent that's the incidence of breast cancer if a woman has breast cancer the probability is 90% that she'll have a positive mammogram mammograms accurate test if she does not have breast cancer the probability is 7% that she still have a positive mammogram anyway so if you want to think about that and see if you can figure out what is the probability if she has a positive mammogram that she actually has cancer and most people including most scientists have pretty much the same reaction to this it hurts your head and you surprised that you can't figure it out even though you did the same problem a week before so now I'm gonna show you that you can actually deal with this problem if it's stated in a more direct way so the same data were presented to the physicians in the following way and then they mostly got the answer right so eight out of every thousand women have breast cancer of those eight women seven will have a positive mammogram this is the same data as previously of the remaining 992 women who don't have breast cancer about 70 will still have a positive mammogram so again if a woman has mammogram what is the probability that she actually has breast cancer well the technique that I'm gonna recommend is widely taught to not universally taught to medical students you make a two by two diagram and is there a pointer anyway the you can look the vertical columns are whether you the number who have cancer under different conditions and or don't have cancer and the horizontal tells you whether they got tests on the mammogram so where that where the each individual subdivision is where that the boxes with where the lines cross so the upper left hand corner will be those who have cancer and also a positive mammogram so let's fill out this thing which says that a thousand women eight have breast cancer so that's the total there now seven will have a positive mammogram of those eight so that goes into the box with positive mammogram and cancer and one is the difference of the remaining 992 that those are the ones who don't have breast cancer so they're in the right hand column 70 will still have a positive mammogram and of course 922 won't so what again I'm gonna ask you what the probability is and I think you can see that you may be able to do it in your head right now positive mammograms well seven true ones and 70 false ones so the total of 77 positive mammograms true positives are seven so the probability with what's called the positive predictive value is just 7077 or 9% so when the in the original case a lot of the positions gets very high incidence of cancer but the actual rate given a positive mammogram is 9% which is surprising the numbers actually are pretty accurate for the real-world case of no no no oncologist would take action on the basis of a mammogram by itself so in doing this what what I'm good what I'm trying to do here is I'm going to show you the diagnostic test method works easily you can make it logical and you can more or less I know it's a little hard in a lecture but you can more logically more or less logically find the answer and I'm going to try to apply that to that cohort study on red meat but you need to know that in addition to the positive predictive value that's the real payoff number you also want to know the sensitivity you know is it a good test and of course the mammogram is a good test it has 88% sensitivity and specificity which refers to the true negatives over those who don't have cancer and that's also very good now high specificity means if you don't have a positive mammogram you're unlikely to have cancer of course even if you do have a positive mammogram you're unlikely to have cancer but it's important to know those numbers so it's a sensitive test but when you have a low incidence of disease you're going to have trouble analyzing the statistics and finding risks so you've got to be very careful so now we're going to ask is high red meat consumption diagnostic of type 2 diabetes so we're going to try to pretend that eating red meat is is like the diagnosis test and that the payoff is is now going to be type 2 diabetes so we're back to this thing now they had we're going to restrict ourselves to one of the cohorts one of the groups that they looked at and they had 37,083 men so we'll put that as the totals in the corner in there now here's that mind-boggling table again and we're just going to look at one of these cohorts which is called the health professionals follow-up study and again what they did is they took all the people in this cohort they determined how much meat they had and they ordered them and then lumped them into five different groups according to how much meat they had already there's some problem there because they've averaged and taken a group statistics so you don't know especially at the high end whether there's somebody who could really chow down a lot of red meat biasing everything but we'll take it as face value so this list three of the cohorts the lowest red meat q1 and the highest red meat q5 and so we're gonna look at the data this table tells you how many people are in it and so we can put those over there now the what we want to know is how many people got sick I mean that many this is actually retrospective how many people got cancer and one of the problems with these kinds of papers is that the data is moved around you have to go to a second table but now I'm telling you what to look at so you can do it if you want to and it lists cases and person years now cases is what you want to know person years refers to a different method of analysis so it's the number of people times the time but it's irrelevant for what we're doing here we're gonna keep it really simple and just look at it as an odds ratio well 655 people in the big red meat study got type 2 diabetes so we and 6592 didn't so we already got what we want the probability which is coincidentally nine percent so this is not compelling I mean this is different than what they were selling us in the abstract in the journal and I won't go through this we just fill in all the other data that we would get these are all in either table one or table two so what does it mean well what it means is the first thing we found is that the positive predictive value is low it's nine percent so that's not much but the question is is it is a zero in other words if you get if you have a cohort group and nine percent of them got sick what they always say you can scale that up to the entire population and you'll save a hundred thousand lives every year that's only true if it's good data and to get a sense of what's what to compare it to if you look at the people who got weight very little red meat is 47 percent now the absolute difference is what you really want to know and that's already 4.3 percent and that's pretty low and then you have to stop and ask where does this come from and it comes from food frequency questionnaires and without going through all the different criticism of food frequency questionnaires how much meat did you eat last year you know that whatever the accuracy of that is it's not in the range of 4% however you could take the ratio and that would be equivalent to an odds ratio of 4.3 the difference between the two over the 9% which is the risk in the target population and that would give you an odds ratio of 0.48 and something you see on in your inbox will say new study shows almost 50% reduction in risk well that just doesn't mean anything and again the positive predictive value is discouraging the specificity is quite good and that means that if you don't eat a lot of red meat you're unlikely to develop type 2 with diabetes of course even if you do eat a lot of red meat you won't so I make my own picture so if you if you just took people and everybody in the group that had diabetes and just lump them at random into five piles you'd get something like this you get something like the dotted line the red is the actual positive predictive value and you see all these positive predictive values are very small and the difference between random and the positive predictive values it may be a real number but it's really like weighing the captain by weighing the ship when he's on board and when he's not it has no accuracy so they adjusted this because they came up with real numbers better than 9% and if you do a study for example showing that carbohydrates cutting out carbohydrates will make you thin you have to correct your data for the total calories make sure that's not the controlling variable so you would take your data and subtract out the effective calories and then then see if it's still held up on carbohydrates well they've corrected for a lot of things family history of diabetes history of hypertension total calories dietary score whatever that is and shirt size no they didn't correct shirt size one point in passing is that mathematically a confounder is symmetrical with the original data so if you correct red meat with smoking you might just as well correct smoking with red meat and that might actually make more sense so let me let me explain something about this I'll tell you joke about confounder so the woman calls the police because the guy across the street is exposing himself and police come the cop says lady that window is too high I can't see anything and she says sure where you are but stand on this chair and you'll see what I'm talking about if you have to do this much work to come up with an odds ratio 1.5 it's not real so this is some of the a couple of things that are wrong with the literature and I refer to them as the four horsemen of the statistical apocalypse and those are group statistics which again averages things that can't be averaged relative risk which I just showed you doesn't make sense and this time I'll run through what's wrong with a meta-analysis and intention to treat I can't talk about here but if you look it up you will be surprised at how bad it is what is a meta-analysis well what it is well I just I described it with the old joke meta-analysis is to analysis what metaphysics is to physics but but basically the I basically the idea is that if you have a group of studies if you think they're really similar and you average them you hope that they'll come out giving something that's more reliable one of the things that I pointed out and we heard this earlier if you look in the literature to see what's going on I'm trained as a chemist and I've worked in a number of fields including behavioral neurobiology I never heard of a meta-analysis until recently until I got into nutrition and one reason is that in 1970 there weren't any in 1980 there were 50 in 1900 there were a hundred this is a logarithmic scale so these are the exponents and then there were 300 then there were a thousand and then there were 3,000 and in 2014 there were 10,000 meta-analysis studies in the literature and if you extrapolate back to 1970 you see it is actually zero this is an example of spontaneous generation the the process grew by itself now I'll just try to quickly show you what's wrong with this and then try to fix things what they're doing here is taking a whole bunch of studies they're looking at the odds ratio and they're averaging them to get that thing in the diamond now what that's showing is that all all of the odds ratios on the right side of the figure in this case they're starting beta blockers are worse with a beta blocker all the things on the left are better with a beta blocker and they say on average it's better but here you need to know it's a technical fact and that is the statistical rule if the error bar those are the horizontal lines if that crosses one there's no statistical difference so all of these things don't cross the line none of these are any good so how you can average a bunch of wrongs and get something statistically significant is hard to understand so a meta-analysis if you plot them out they'll show you whether the different studies are roughly consistent and if they are that's what you know you know what you did before you went to the library but averaging them won't give you any more information than you had before and but if they're different you know if the study as in the previous ones the studies are really different I tell you by analogy the example of an emerging country building a railroad should they use a gauge that matches the country to the north or the country to the south well the parliament votes for a gauge that's the average of the two and the worst is the coconut study and they want to tell you that the most important studies on dietary fat and cardiovascular disease a meta-analysis with two studies that worked and two that didn't so that sort of speaks for itself so how are you gonna fix it well I'm gonna tell you by analogy this about a NYPD officer who was involved in convicting a criminal and later decided that he was actually innocent and was involved in helping get him released and the main idea is he said he had doubts about the work of the NYPD and what he thought was wrong is that the investigators shouldn't stopped at after the arrest and whereas he preferred he later went to work with a federal agency where they kept investigating until there was a trial and that's what we really need we the the bottom line is that we can't stop reviewing the paper after it's published there has to be post publication review and the original publication has to be much more tentative and this is just just have it almost done here these are just a couple examples of responsible people with good credentials who made real criticisms to publish papers and those have to be considered so what we really need is a two different levels of acceptance of published papers they have to be published and then subject to continued review so the bottom line is we're using the wrong model you know we're really talking about physics where you get a number that is a real number and you know who the experts are on the field and although they don't always agree they do agree on methodology so and in in medicine and especially nutrition they're more tentative and more controversial than in the physical sciences post publication review before final acceptance all voices have to be heard until then we have a rule editors have to recognize a conversion controversial manuscript and they have to appoint reviewers on both sides of the controversy they don't do that that constitutes de facto or intentional bias that's it thanks it seems to me men analysis is a lazy way to do research because all you have dudes have an adding machine you don't have to do anything original you just add numbers together and you got a paper and it seems it seems to me the big push to turn out papers is one reason meta analysis is accepted even though it should not be thou has said it and well yeah you don't have to do an experiment well where it really comes from it is that if you have a couple of small studies and they're underpowered which means they don't have a large enough number of subjects to really draw a conclusion you can try to merge them and hope that a pattern will come out but in doing that you have to recognize that it's a last-ditch Hail Mary kind of approach you may see something or you may not but these are all big well-controlled studies and you're not going to get anything by averaging them and this is one of the principles in the medical literature that I object to strongly is they have the idea that the larger the study that that makes it inherently better and because of time restraints I didn't I have a proof that a larger and is not necessarily better had the video of 10,000 Japanese singing Beethoven's ninth over to joy and you can see it on YouTube it's it's pretty good actually but somebody made the comment that all they could think of was all the buses in the parking lot anyway yes yeah that was really interesting I was just wondering if off the top of your head you know of any books I could read that are at a basic level to keep learning about this about the statistics statistics and how to interpret data but nothing too advanced well I'm trying to write stuff myself I have stuff on my blog and okay well the bloggers are good at it I know Rob Wolf well we're publishing a paper along these lines and one of my co-authors said that my talk was a complicated way of doing what Rob Wolf did simply so hard to please everybody but no it's tough I mean the thing about what you have to know going in there is that you can trust your common sense now a good statistics book will say something along the lines of what we do in statistics is we try to put a number on our intuition by intuition they mean our experience with the biology you know we try to ask ourselves I understand this group of people I understand the reliability of the independent variable how much they eat and I understand that they're real different you know the first five people who came in were all completely different what measure can I use that will bring out the differences so for example we published a paper where we said don't do anything till you you're sure that maybe they really are different and so what we did is we took this was a weight loss study and we took the data and we subtracted the weight loss on the well we took the all of the weight loss on all the low-fat diets and we put them across horizontal line and we took all the weight losses on the low carb diet we put them down the side and then we made a matrix we made a grid and each of the grid elements were the difference between the low carb and the low fat and then we color coded it and you could look at it you can eyeball it and you can see that in in the particular experiment that all of the big red ones were drifting in the direction of the low carb now that's not that's making an assumptions to it's always again as we said there's no absolute biology and statistics but what what it was saying is if these people are really different then let's see what the overall effect of this diet is and and it suggested that in this particular group that the low carb diet was way better by the eyeball test it didn't mean that you couldn't do a more sophisticated analysis that the analysis we did emphasized all the differences it brought out all the differences between the people the limitation of more common statistics is that emphasizes it it gives up on differences for reliability so it smooths everything out and so you have to make a balance between those two so the answer to your question is I don't really know the answer the statistics books tend to be too heavy-duty I I can't understand them yeah he's pretty good he's yeah he's written I forgot because I had this the reason I'm hesitating is that I haven't looked at his stuff in a while but he has a lot of stuff on the internet and it does try to get down to earth well actually that's the place to look as you look on YouTube and you look on people that are trying to make it simple and remember it's your money you're the consumer if you're not into it in five minutes the wrong guy you got to go to the next YouTube but well it's hard to shop for that commodity it's you know it's very hard to get people to tell you the simple average down to earth parts of statistics without drowning you in complexity and so yeah actually I did try to we published a paper showing that intention to treat was useless and of course it was turned down by a lot of journals because it was threatening people who were using intention to treat and somebody suggested that I get a statistician on the paper and so I called the law because he was even more critical of the intention to treat but remember somehow we couldn't get along so I just published it myself but it was good to see somebody who was even more annoyed by intention to treat than I was I thank you yes last question thank you for a wonderful talk and for making some excellent points I think another thing that often gets missed in a lot of the nutrition literature is the idea that even if we're trying to isolate a particular variable or you know in the posted the effect of consuming meat we're not considering also the effect of people that aren't eating that meat what are they eating in place of that and even randomized control trials have this problem of testing the effect of eliminating a certain food without necessarily recognizing what people are replacing it with or vice versa and you know given given the complexity of diet and the fact that there's so much variation that happens with what we eat what we're eating it with and interactions within those and then interactions with lifestyle variables and given just the messiness of nutritional data of observational data would you I mean in moving forward and trying to answer these questions of what are the real health effects of eating meat versus other foods would you just throw out all I mean discourage observational studies period or is there a place for observational studies and how would you suggest if so doing a better job at getting at these complex relationships and nuances well the what I would suggest is that whatever study you do you have a certain degree of modesty and analyze what you're really what you're really finding there I mean I think that we need to think about going forward but certainly the idea that we would throw out 80% of the epidemiologic studies is unreliable it's not impossible I would tell you yeah well you'll notice I listed all these compounders this is red meat you didn't find in those compounders the amount of carbohydrate so there's a big difference between having a red a roast beef sandwich and having roast beef lettuce wrap and but the problem is in the science it's it's not just the methodology and I think the only way to get at this because the establishment position is dense on the biology we have to show that this kind of methodology is not good and Big N is not good because the bigger the end the more the variability the more you lose track of the subjects you know we don't know the answer on diet but the extent to which we think that a low carbohydrate diet might be preferable for people with a metabolic syndrome is not from the big studies it's from work done by Jeff Olic for example where he has 40 subjects and the low carb people are way different than the low fat people you know there's just it's not a judgment call and we we also have standards I mean what's wrong with the medical literature is that they don't have clear standards they have arbitrary rules like a big N is better but Bradford Hill Bradford Hill is the guy who did the cigarette smoking lung cancer study and he's published a really excellent list of nine criteria that asked the question when is an observational study really likely to be causal and he gave nine criteria he was very explicit it's worth digging up his presidential address because it's beautifully written it's relatively short and he was explicit that these aren't hard and fast rules that there are extensions of common sense and the first thing that he said was he looks for big changes and that does not mean an odds ratio of 1.5 it means big changes and for example I the particular study that I'm thinking of that Volek did everybody half of the people in the low carb group were better than the average in the low fat group and the no I'm sorry half of the people in the low carb group were better than anybody in the low fat group so half the people were what you might consider outliers and that tells you something I mean that tells you it doesn't tell you that the number that he came up with is an accurate number you don't as you say there are too many variables but but the eyeball test tells you that if it's wrong it's not way wrong and that's got to be the first criterion is it is it a big number and the second criteria is it does it make biological sense it doesn't make sense that red meat is going to cause diabetes it and I would just add one technical point on this which is never brought up in the analysis of these studies is that the statistics in this particular study are what's called two-tailed what that means is that you're asking which of two things is better and therefore in a 6040 outcome you maybe have a 60% chance of getting better but if you don't get better you're going to get worse because you don't know which is the good thing having read me to not having read me a lot of people especially elderly don't get enough red meat so in other words if you think of the lung cancer study not everybody who smokes get lung gets lung cancer and the ones who don't are immune or have other biological features or whatever but nobody thinks that smoking keeps you from getting lung cancer but you don't know that with red meat because we don't know enough about it it may keep you from getting diabetes so that's a very important result and it makes the results even more big