The Impact of Underpowered Studies on Clinical Trial Results

To the Editor: The commentary by Marder et al. in the September 2017 issue of the Journal, “Why Are Innovative Drugs Failing in Phase III?” (1), brings up a critical topic: the frequent failure of phase III trials for CNS programs. The authors effectively summarized investigations into this issue. However, we believe that a possible explanation for phase III trial failure was minimized, as noted in the commentary: “that the phase II results were misleading” (1, p. 829).

It is perhaps an underappreciated truism that statistically underpowered studies with small sample sizes (often early-phase studies) are highly likely to both under- and overestimate treatment effect sizes (2, 3). By considering large effects from small studies to be true effects rather than overestimation errors, some researchers have concluded that smaller trials are more advantageous.

Take, for example, the citation of Undurraga and Baldessarini, who suggest limiting phase III antidepressant trials to 30–75 patients (4, p. 860). Testing this suggestion using data from Food and Drug Administration (FDA) antidepressant phase III trials, we found that treatment arms with Ns of ≤75 had both the highest (0.75) and lowest (−0.29) effect sizes of the entire group of 115 arms and that they achieved statistical significance only 50% of the time. In contrast, the largest treatment arms (with Ns of ≥350) had a 100% rate of statistical significance and little variation in effect size (from 0.24 to 0.33).

These smaller trials empirically demonstrate the chance findings of underpowering—they are no more reliable than tossing a coin. These findings are unsurprising, given that a two-arm trial with Ns of ≤75 would be powered only at 50% for an effect size of 0.5 (2). Underreporting negative studies in published research may mask the downside of the underpowering coin, which can be seen in the FDA data; that is, those “unlucky” underpowered trials resulting in failure and underestimations of drug effect.

The increase in size of phase III trials should not be viewed as a design flaw because this rejects the statistical principles of the scientific method. If phase II results are a reflection of the true magnitude of treatment effects, by rule they should be replicated in a larger sample. If they are not, then the phase II results were likely misleading. Seeking to replicate such lucky results from underpowered studies by underpowering future studies only perpetuates scientifically unsound methods and high failure rates.