An end to the bizarre fiasco at Food and Chemical Toxicology

12/07/2013 07:58:00 PM
This Sprague-Dawley rat wants to remind you to get health insurance.
I received word recently that the infamous study linking rat tumors to consumption of genetically modified corn has officially been retracted by the journal that published it. Apparently this article has made a big media splash as opponents of GMOs latched onto the study to validate all their preexisting beliefs about the dangers of GMOs. This reminds me a lot of two other incidents: the way anti-vaxers latched onto the fraudulent autism study to validate their anti-vaccine beliefs, and the way anti-gay groups latched onto the dishonest Regnerus study to validate their anti-gay bias. Together, these highlight the need for a robust peer-review process, because a single flawed study can sometimes derail decades worth of science, sometimes with life-or-death consequences.

I hadn't been following this particular GMO study closely, so when I heard that it had been retracted I had only a vague recollection of having heard about that study somewhere before. Aha! I first heard about the study here, in a different controversy over the authors' scandalous attempt to shutdown any media criticism of their methods by forcing journalists to sign a bizarre and totally unethical "non-disclosure agreement." The terms of the agreement said that journalists could write what the study authors told them, but could not consult any other expert to weigh in. The subtext: you can only write what they tell you to write.

The popular press has been surprisingly unhelpful in explaining exactly why the study was retracted. According to the editor-in-chief [emphasis added]:
The journal Food and Chemical Toxicology retracts the article “Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize,”... after a thorough and time-consuming analysis of the published article and the data it reports, along with an investigation into the peer-review behind the article. ... the Editor-in-Chief found no evidence of fraud or intentional misrepresentation of the data. However, there is legitimate cause for concern regarding both the number of animals in each study group and the particular strain selected. The low number of animals had been identified as a cause for concern during the initial review process, but ...A more in-depth look at the raw data revealed that ... Given the known high incidence of tumors in the Sprague-Dawley rat, normal variability cannot be excluded as the cause of the higher mortality and incidence observed in the treated groups.
The Sprague-Dawley rat is a species that is especially prone to unusual tumors. Often, such unique lab animals are used in research as essentially a cost-saving measure, since otherwise we'd need way more rats to observe the same number of tumors in the same amount of time, exponentially increasing the costs of the project. But, the downside is that you have to deal with fat-tailed probability distributions and results that might not be generalizable to other populations.

But even that much is considerably more than I could glean from the articles about this retraction. I have no intention of reading a now-retracted article just to find out why it might have been retracted. What the editor seems to be implying is that the statistical significance the authors' claimed to have was not actually statistically significant, because of a small sample size and fat-tailed probability distribution.

Certainly small sample sizes and crazy probability distributions can cause problems with standard statistical calculations. But is that what happened here? It isn't so obvious to me. To get an idea of what happens with fat tails and small sample sizes I did a little simulation in R. The Pareto distribution is the fattest-tailed probability distribution I know of (no offense)--it is so fat that both the mean and variance are actually infinite. Yet I can't produce a simulation that justifies retracting the paper on these grounds. Here's what I did: I ran 10,000 simulations of t-tests comparing two groups of 10 observations plucked randomly from the same Pareto distribution (location and shape parameters both equal to unity), representing the treatment and control groups of the study. If the fat-tails and low sample size alone were grounds for retraction of the study, then we'd expect a false positive (p-value less than 0.05) more than 5% of the time. I actually got a false positive rate of 1.63%, implying that the t-test was actually biased in the opposite direction. I've attached the R code below the fold. So, I'm very curious exactly what test the authors used, and why the review concluded it was likely generating false positives.

Nevertheless, there are, I think, several lessons here. One is that you really do need to pay close attention to the assumptions under which the t-test or other kinds of statistical tests are valid. Those assumptions were unsound in this case. Another lesson is that peer review is essential--most likely, even though there is no statistically significant difference between rats fed GMO versus non-GMO corn, and it has now actually been retracted, the study will nevertheless remain a mainstay of the anti-GMO movement. Yet, a third lesson is that peer review isn't infallible, and you shouldn't base your beliefs on a single study, especially when there's lots of studies, experts, and evidence disputing it.

But, here's the most important lesson of all: we need way more funding for the sciences. Just take a look at the two reasons that the editor cited for the study's erroneous results:
  1. small sample size
  2. the Sprage-Dawley rat
I mentioned earlier that scientists resort to the Sprague-Dawley rat as a cost-saving measure to produce more tumors with fewer rats. Better funding could give scientists access to much bigger mouse-labs, producing better, more conclusive data on pressing issues of public health.

Here's the simulation I did in R (using the VGAM library for the rpareto function):
for(i in 1:10000){
    tt=t.test(y1,y2, var.equal = TRUE)
    if (tt$p.value<0.05){