#143 - John Ioannidis, M.D., D.Sc.: Why most biomedical research is flawed, and how to improve it

Ioannidis uses the tobacco example to anchor the framework: tobacco-lung cancer has effect sizes of 10-30x, consistent across every design, in every population, over decades. That combination — extreme strength, extreme consistency, extreme replications — produces posterior confidence that exceeds anything available from a single p = 0.04 observational study. The practical heuristic: the stronger the prior, the larger the effect size, and the more independent the replications, the more weight to give a positive result.

Ioannidis gives the genetics example: most GWAS findings have odds ratios of 1.01. These are real, replicable effects. They are almost never clinically significant for individual treatment decisions. The flip side is nutritional epidemiology reporting hazard ratios of 1.14 as major findings deserving dietary guideline changes — an effect size that in genetics would be a rounding error and in nutrition almost certainly reflects residual confounding.

Attia walks through tobacco vs. nutrition as the contrast. Tobacco: odds ratios of 10-30x, perfectly consistent across every study design and population, clear dose-response, mechanistic coherence, and replicated experimental evidence. Most nutritional associations: odds ratios of 1.05-1.20, inconsistent across cohorts, no reliable dose-response, and failing to replicate in even modestly rigorous trials. Bradford Hill, Ioannidis argues, would have been appalled at how his criteria have been co-opted to certify associations with effect sizes that would constitute noise in a proper experimental context.

Ioannidis emphasizes that genetics took pre-specification seriously by requiring genome-wide thresholds — effectively pre-registering the significance bar. Nutrition has largely not adopted this. His proposed improvement for observational nutritional research: exposure-wide and outcome-wide association studies that test all nutrients against all outcomes simultaneously, with multiplicity correction, rather than cherry-picking one nutrient-outcome pair for which results happen to be significant.

Ioannidis draws the contrast: in genetics, no one fights for a single SNP because no one has a career staked on it. In nutrition, beliefs are deeply tied to cultural identity, diet books, public health careers, and sometimes religious frameworks. Even well-intentioned meta-analyses can be corrupted by selective inclusion — inviting the 100 teams that have already found a positive association and excluding the 2,900 that have not, then reporting a combined p-value of 10^-30 as definitive proof.

Ioannidis proposes the exposure-wide association study (EWAS) as a more honest approach to nutritional observational data: run all nutrients against all outcomes simultaneously, correct for multiplicity, and report the full matrix. Patterns that are strong and replicable across multiple cohorts have a higher chance of reflecting something real.

Ioannidis positions MR as one of three approaches to better-than-observational evidence on dietary questions: (1) tightly supervised metabolism studies; (2) MR using genetic instruments; (3) exposure-wide observational studies with multiplicity correction. He argues all three should be pursued in parallel rather than continuing to generate thousands of small observational cohort papers.

The model inputs three factors: (1) the prior probability R that a hypothesis is true — which varies enormously by field, from highly targeted pre-specified drug targets to exploratory nutritional associations; (2) the statistical power of studies — underpowered work both misses real effects and, perversely, inflates the effect size when it does find something; and (3) bias u — all the ways a null result gets pushed across the significance threshold through publication bias, selective reporting, multiple-endpoint testing, p-hacking, and flexible analysis. Run these realistic values and the positive predictive value of a published p < 0.05 collapses. The paper was published in PLOS Medicine, a new open-access journal, and has since received close to 10,000 citations — though Ioannidis notes his PRISMA meta-analysis reporting guidelines have received even more.

Ioannidis explains the structural incentive: scientists work with thin resource slices, must demonstrate results to continue funding, and therefore run many small studies rather than one large definitive one. There is even a disincentive to refuting wrong results because refutation does not advance a funding case. Big data creates the opposite problem — overpowered studies produce astronomically small p-values from bias alone, measuring the distribution of confounding rather than true effects. Neither pole is reliable; properly-powered, pre-specified trials are the narrow target.

Ioannidis is careful to note these translated life-expectancy estimates were tongue-in-cheek illustrations of how relative risks are communicated to the public, not epidemiologically valid calculations. But the underlying point is serious: the system generating these numbers is operating in a regime where selective reporting, post-hoc analysis, and correlated confounders produce an absurd density of 'significant' associations. The paper created enemies and friends in equal measure; Ioannidis says he is grateful for the enemies whose constructive criticism pushed the field forward.

The trial randomized participants to low-fat control vs. Mediterranean + olive oil vs. Mediterranean + nuts. Both Mediterranean arms outperformed the low-fat arm on cardiovascular events, and the trial was stopped early at an interim analysis. Post-retraction analysis excluding or adjusting for the non-randomized participants did not dramatically change the results — but Ioannidis now characterizes it as a partly randomized, partly observational trial that has since generated ~300 secondary analysis papers, each adding more fragmented claims of uncertain credibility. His honest conclusion: it no longer has the same credibility as the original clean result he thought he had.

The Santa Clara study was crowdfunded through Stanford's development office; Ioannidis was not even aware who the donors were. A report that an airline executive had contributed $5,000 to Stanford — not to Ioannidis or the study directly — was weaponized to claim the study was industry-funded to reopen air travel. Ioannidis, a scientist who had written extensively about climate urgency and gun violence, was accused of having conservative ideology. A hoax was circulated on social media claiming his 86-year-old mother had died of COVID; her friends called to ask about funeral arrangements, causing a life-threatening hypertensive crisis. Stanford's fact-finding process concluded there was no conflict of interest. The infection fatality rate the study estimated was subsequently validated by hemodialysis patient data in The Lancet.

Ioannidis frames two root causes: (1) Measurement. Genetic platforms have < 0.01% error rates. Dietary recall questionnaires have very high recall bias, low accuracy, and fail to capture the correlated structure of real diets. The signal-to-noise ratio in nutritional epidemiology starts much lower before any analysis decisions are made. (2) Priors and identity. No geneticist has built a career defending polymorphism rs249214. Nutrition researchers have built careers, policy influence, and cultural identities around specific dietary theses — Mediterranean, low-fat, low-carb, vegetarian — which makes abandoning them epistemically much harder, even unconsciously.