Last week, we've already seen that the most prominent way of ranking scholarly journals, Thomson Reuters' Impact Factor (IF), isn't a very good measure for predicting how many citations your scientific paper will attract. Instead, there is evidence that IF is much better at predicting the chance that your paper might get retracted.


Now, I've just been sent a paper (subscription required) which provides evidence that the reliability of some research papers correlates negatively with journal IF. In other words, the higher the journal's IF in which the paper was published, the less reliable the research is. This particular evidence holds only for gene association studies, but given the high correlation between IF and retractions, it likely holds for other research as well. Now what is the data? The paper contains one figure:


On the X-axis you see the impact factor and on the Y-axis a measure for the bias of the gene association data on a logarithmic scale. How did the authors calculate this bias score? In their own words:Thus, the authors plot a value that indicates by how much a single study over- or underestimates the actual effect (estimated by taking many studies into account) of a gene-phenotype association. The size of the circles on the graph indicates the sample size of the study. The overall "R squared" value, or the Coefficient of Determination for this correlation is only 0.13, but at a highly significant P=0.00002. This means that the correlation is pretty weak, but it is statistically significant.
This sort of correlation, low R2 but statistically significant, corresponds roughly to the pattern of correlations one can see with citations: IF is predicting citations of a paper to some degree, but not very reliably. Likewise, IF is predictive of a gene association unreliability, but not very reliably. Or, phrased differently, IF is about as predictive of a paper's unreliability as it is about its citations, which I'd consider quite bad for something that decides about scientific careers.


But another important point can be seen in the graph: IF is also predictive of the sample size of the gene association study: the higher the IF of the journal in which the study was published, the lower the sample size. One could interpret this as evidence that high-IF journals are more likely to publish a large effect, even though it is only backed up by a small sample size, while low-IF journals require a more solid amount of data to back up the authors' claims.


Taken together, this study provides some evidence for one of the potential mechanisms underlying the very strong correlation between IF and retractions we've seen before: authors are more likely to publish unreliable data with predominantly overestimated effect sizes in high-IF journals. Importantly, this constitutes a mechanism which cannot be explained by high-IF journals being more closely scrutinized than low-IF journals. Instead, it suggests that at least a portion of the retractions in high-IF journals is due to the studies published there being more likely to be flawed than studies in low-IF journals.

In the words of the authors:

---

Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77