Sixty Years of Placebo-Controlled Antipsychotic Drug Trials in Acute Schizophrenia: Systematic Review, Bayesian Meta-Analysis, and Meta-Regression of Efficacy Predictors

We searched the Cochrane Schizophrenia Group Controlled Trials Register (compiled by regular systematic hand searches and searches of more than 15 databases, clinical trial registers, the Food and Drug Administration web site, and conference proceedings [21] without language restrictions; available to us until version August 2009) with the term “placebo,” and we searched MEDLINE, EMBASE, PsychINFO, Cochrane CENTRAL, and ClinicalTrials.gov (last search October 2016; search terms presented in Table S3 in the online data supplement). These searches were supplemented by screening previous reviews (9, 11, 22–30).

At least two reviewers (among authors M.H., M.S., and S.L. and contributor M.T. [cited in acknowledgments]) independently selected potentially relevant publications from the abstracts found by our search and made decisions about whether to include studies. At least two reviewers (among authors S.L., C.L., M.H., B.H., M.S., M.R., and S.B. and contributors M.K., P.R., T.A., and N.P. [cited in acknowledgments]) extracted data in duplicate in a spreadsheet. Disagreement was resolved by discussion. Missing data were requested from authors or the sponsoring pharmaceutical companies for all studies published in the last 30 years. When possible, we extracted intention-to-treat data, and we preferred data based on mixed-effect models of repeated measurements over last-observation-carried-forward data. For dichotomous data we assumed that participants lost to follow-up would not have responded (conservative approach). Missing standard deviations were estimated from test statistics or by using the mean standard deviation of the remaining studies (37, 38).

We used a Bayesian hierarchical random-effects model in OpenBUGS 3.2.3 (http://www.openbugs.net/w/FrontPage) to estimate summary effect sizes for each outcome, as heterogeneity was expected. We primarily examined all antipsychotics as a group because efficacy differences between drugs are small (11, 39–41), but results of individual drugs are presented as well. For the primary analysis we merged the different antipsychotic arms within multiarm trials (17) but properly accounted for the inherent correlation in the drug-specific analyses. We estimated standardized mean differences (SMDs) for continuous outcomes and risk ratios for dichotomous outcomes, together with the 95% credible interval (CrI) for each. The numbers needed to treat to benefit and to harm were estimated by using the meta-analytic summary of an outcome in all placebo arms. Heterogeneity was assessed by visual inspection, the between-study standard deviation, and the I-square statistic (values >50% were considered as indicating considerable heterogeneity [42]).

We meta-regressed publication year and the frequency of the moderators to explore which trial characteristics have changed over time. Then, in meta-regressions of the primary outcome we were particularly interested in exploring whether the drug-placebo difference became smaller over the decades, and we systematically examined all possible moderators, reported previously (9, 10, 43–45), that might explain this phenomenon. We categorized the moderators into patient-, drug-, and study-design-related factors, although there were expected overlaps. Moderators that were significant in univariable analyses were included in a multivariable meta-regression model. To identify the most important moderators from this model we used the stochastic search variable selection algorithm to estimate the probability that each variable should be included in the meta-regression model (see protocol, Table S2 in the online data supplement [46]). To measure the strength of a moderator we compared the meta-regression models with the meta-analysis without covariates and estimated the percentage of heterogeneity explained by a moderator. Meta-regressions were not performed on individual drugs, because statistical power would have been insufficient for most of them. Post hoc analyses following recent research (e.g., by Agid et al. [9]) are noted in parentheses.

We applied a fixed-effects instead of a random-effects model, we calculated odds ratios, and we excluded studies based on study completers. We explored whether the effect sizes of haloperidol, the only drug for which both early and recent studies were available, had decreased over the years as well (post hoc analysis).

We used contour-enhanced funnel plots (55), a selection model (software OpenBUGS) (56), and the trim-and-fill method (57) to assess whether eventual small-trial effects were likely due to publication bias (Table S2 in the online data supplement presents details).

Figure S1 in the online data supplement presents the PRISMA (12) flow diagram. Overall, 167 studies published from 1955 to 2016 with 28,102 participants were included (see Table S4 in the online data supplement). The mean duration of illness was 13.4 (SD 4.7) years, the mean age was 38.7 (SD 5.5) years, and the median duration of studies with useable outcomes was 6 weeks (range 3–28 weeks; for the primary outcome all but one study lasted ≤12 weeks; one study without any useable outcomes lasted 156 weeks). There were no studies exclusively examining first-episode patients or treatment-resistant patients. The most frequently used drugs with data for at least one outcome were chlorpromazine (36 studies), haloperidol (28 studies), olanzapine (20 studies), risperidone (15 studies), quetiapine (eight studies), paliperidone (eight studies), aripiprazole (nine studies), thioridazine (seven studies), lurasidone (seven studies), asenapine (six studies), and loxapine (six studies); for all other drugs, fewer than five studies were available. Risk of bias is presented in supplemental Table S5. We included only randomized, double-blind trials, but the reports often did not indicate full details about sequence generation or allocation concealment. Descriptions of the methods and success of blinding were frequently insufficient as well. The data confirmed the high dropout rates in current schizophrenia studies (mean 37.2%, SD 20.5). Older studies were poorly reported, making it often impossible to extract outcome data (50% of the studies had a high risk of selective reporting). Finally, 70 studies (42%) were sponsored by the manufacturers of one antipsychotic included, 72 (43%) were not primarily industry sponsored, and in 25 (15%) of the studies the sponsor was unclear.

The mean effect size of all studies combined was 0.47 (95% CrI 0.42, 0.51; I² 52%; 105 studies with 22,741 participants). Patients treated with antipsychotics were twice as likely to respond as those on placebo when any response criterion was accepted (97 studies, N=20,690; response ratio 1.93, 95% CrI 1.72, 2.19); the number needed to treat to benefit (NNT) was 6 (95% CrI 5, 8; I² 61%). At least a “minimal” response was experienced by 51% (95% CrI 45%–57%) of the antipsychotic-treated patients compared with 30% (27%–34%) on placebo (46 studies, N=8,918; response ratio 1.75, 95% CrI 1.59, 1.97; NNT 5, 95% CrI 4, 5), while 23% (95% CrI 16%–32%) versus 14% (95% CrI 10%–19%) had a “good” response (38 studies, N=8,403; response ratio 1.96, 95% CrI 1.65, 2.44; NNT 8, 95% CrI 6, 11) (Figure 1). Similar results were obtained when responder rates based on the PANSS/BPRS or CGI were analyzed separately (see online Table S6, which also presents odd ratios).

Participants receiving placebo were more likely to discontinue the studies prematurely, both for any reason (38% drug, 56% placebo; 105 studies, N=22,851; risk ratio 1.25, 95% CrI 1.20, 1.31; NNT 11, 95% CrI 9, 14; I² 19%) and for inefficacy of treatment (13% drug, 26% placebo; 94 studies, N=23,017; risk ratio 2.09, 95% CrI 1.90, 2.32; NNT 7, 95% CrI 6, 9; I² 46%).

The effect size for positive symptoms (64 studies, N=18,174; SMD 0.45, 95% CrI 0.40, 0.50; I² 56%) was similar to that for overall symptoms, while effects on negative symptoms (69 studies, N=18,632; SMD 0.35, 95% CrI 0.31, 0.40; I² 42%) and depression (33 studies, N=9,658; SMD=0.27, 95% CrI 0.20, 0.34; I² 50%) were smaller.

As shown by six trials, the quality of life of participants in the antipsychotic groups was better than that in the placebo group (N=1,900; SMD 0.35, 95% CrI 0.16, 0.51; I² 43%), and so were improvements in social functioning, as shown in 10 trials (N=3,077; SMD 0.34, 95% CrI 0.21, 0.47; I² 46%) (Figure 2).

Antipsychotic drugs produced more movement disorders, as judged by use of antiparkinson medications (19% drug, 10% placebo; 63 studies, N=14,942; risk ratio 1.93, 95% CrI 1.65, 2.29; number needed to treat to harm [NNH] 12, 95% CrI 9, 16; I² 51%); they were more sedating (14% drug versus 6% placebo; 86 studies, N=18,574; risk ratio 2.80, 95% CrI 2.30, 3.55; I² 54%), led to more weight gain (59 studies, N=17,076; SMD –0.40, 95% CrI −0.47, –0.33; I² 73%), led to more prolactin increase (51 studies; N=15,219; SMD –0.43, 95% CrI −0.55, –0.30; I² 91%), and led to more QTc prolongation (29 studies, N=9,883; SMD –0.19, 95% CrI −0.29, –0.08; I² 80%) than placebo. Results for individual drugs are presented in Figures 3 and 4 and in Figure S2 in the online data supplement.

Table 1 shows that several trial characteristics changed significantly over the years: the numbers of participants and sites, the use of minimum baseline severity as an inclusion criterion, fixed-dose designs, use of operationalized criteria, use of the PANSS, percentage of men, studies outside the United States, and placebo response increased, while the duration of the washout period, use of dopamine D₂ antagonists (50), study duration, risk of bias in terms of incomplete outcome data, mean doses, and number of academic sites decreased.

Effect sizes have become smaller over the years. The coefficient of –0.08 in Table 2 indicates that a study published 10 years later than another one had, on average, a 0.08-unit lower effect size. Figure 5 demonstrates this effect not only for all antipsychotics as a class (Figure 5A) but also for haloperidol, the only antipsychotic for which both early and recent studies were available (Figure 5B). Moreover, Figures 5C and 5D show that the decrease of effect size was paralleled by an increase in placebo response, while drug response remained quite stable, which contradicts a previous analysis (10).

Significant factors involving study design were larger sample size (total numbers of participants and sites), number of drugs, PANSS rather than BPRS, a minimum symptom level as an entry criterion, and industry sponsorship. With the exception of the number of drugs, all these factors were associated with smaller effect sizes (Table 2).

Significant patient- or drug-related factors were operationalized rather than clinical diagnostic criteria, higher placebo response rates, lower dose, and D₂ and 5-HT_1A receptor partial agonists versus D₂ antagonists (mainly haloperidol), all of which were associated with smaller effect sizes (Table 2).

As several significant predictors are related by nature, we made the following choices for the multivariable model: 1) we chose sample size as representative of the number of sites, number of drugs, and number of arms, and 2) we chose publication year to represent the use of operationalized criteria (such criteria did not exist for early studies), use of the PANSS (introduced in 1987) versus BPRS, and drug mechanisms according to the Neuroscience-Based Nomenclature (50), which also changed over the years. Only pharmaceutical sponsorship and the degree of placebo response remained significant, and they resulted in large probabilities from the variable selection algorithm (82.8% and 81.6%, respectively), implying that they are probably the most important moderators. Studies with a 10-PANSS-point larger placebo response, on average, had a 0.13 smaller effect size, and, surprisingly, industry-sponsored studies, on average, had a 0.16 smaller effect size compared with non-industry-sponsored trials (Table 3).

Both predictors remained the only significant ones in a post hoc sensitivity analysis where all significant moderators were entered. When pharmaceutical sponsorship—which is probably a composite of various factors—was removed from the model in another sensitivity analysis, only degree of placebo response remained significant, demonstrating the strength of this factor. Both sensitivity analyses had less explanatory power than the primary analysis, however (31.3% heterogeneity explained in the primary model versus 18.8% in both sensitivity analyses; see Table S7 in the online data supplement).

Contour-enhanced funnel plots revealed small-trial effects. As studies were missing in the area of nonsignificant effect sizes (Figure 6) and as the selection model showed a strong correlation between probability of publication and magnitude of effect in various scenarios (range of correlation coefficients, R=0.66–0.85), part of the small-trial effects is likely a result of publication bias. The publication bias “adjusted” SMD ranged between 0.36 and 0.41 in various scenarios of the selection model, corroborated by the trim-and-fill method (adjusted SMD 0.38, 95% CrI 0.33, 0.43; see online Table S2).

The use of a fixed-effects rather than a random-effects model (105 studies, N=22,741; SMD=0.44, 95% CrI 0.42, 0.47) and the exclusion of completer analyses (95 studies, N=22,352; SMD=0.46, 95% CrI 0.42, 0.51) did not significantly change the primary outcome, nor did the use of odds ratios (Table S6 in online data supplement).

We examined two response criteria—“any” response and a “good” response to antipsychotics. This was important because previous systematic reviews (23–29) analyzed whatever response criterion was presented in the individual studies, leading to a difficult-to-interpret criteria mix. Several analyses showed that a 20% PANSS/BPRS reduction roughly corresponded to minimal improvement on the CGI and a 50% PANSS/BPRS reduction corresponded to much improvement, and they justified analyzing them together (35, 58–60) (results of individual scales were similar, as shown in Table S6). Antipsychotic drugs clearly increased the number of patients with “any” response (51% versus 30%), but, importantly, few patients (23% versus 14%) reached a “good” response within the confines of short-term double-blind trials. The mean effect size for overall symptoms (0.47) was only medium according to Cohen (61), and it translates to a difference of 9.6 PANSS points. This contrasts with the large (N=463) early NIMH study from 1964 that has been used frequently as a benchmark for antipsychotic drug efficacy (7). Its impressive difference in response rates (61% under drug versus 22% under placebo showed much improvement) can be explained by the fact that approximately 50% of patients suffered from their first episode or were antipsychotic naive (7). In the current review not a single study was restricted to first-episode patients. Thus, its results are representative only for chronically ill, often previously treated patients, who respond less well to antipsychotics (62). In the future, first-episode trials could provide better signal detection. Moreover, the trials in a meta-analysis are weighted by their sample size. Thus, the mean effect size of 0.47 largely represents the effects of the avalanche of trials after the reintroduction of clozapine in 1990. In this context we caution that the efficacy effect sizes of the single drugs in Figure 3 and Figure S2 have not been adjusted for publication year.

A recent meta-analysis found an almost two times higher superiority compared with placebo (SMD 0.58 versus 0.35 here), but, mistakenly, standard errors rather than standard deviations were often used in the calculation of effect sizes, which artificially inflated them (63). In our analysis of antipsychotics, the effect size for negative symptoms was smaller, but the range was similar to that for positive symptoms. However, as studies with primary negative symptoms were excluded, the effect size might mainly reflect reductions of secondary negative symptoms (64). Similarly, the small effect of antipsychotics on depressive symptoms might also be a consequence of the reduction of positive symptoms and associated psychological distress. Nevertheless, some second-generation antipsychotics have proven efficacy in major depressive disorder (65).

The only purpose of our side effect analysis was to present a brief measure of the efficacy-tolerability trade-off in this class review. Effect sizes across drugs were medium with sometimes very large heterogeneity. This heterogeneity reflects the enormous differences of single antipsychotics in their side effects, and it suggests that careful choices can minimize the side effect burden for individual patients. But we caution that sometimes small participant numbers, which make results unreliable (reflected by large 95% credible intervals), must be considered in interpreting Figure 4 and supplemental Figure S2.

The results suggest a small to medium benefit of antipsychotics in quality of life even in the short term. As in our systematic review on maintenance treatment with antipsychotics compared with placebo (14), only six recent trials have investigated this crucial outcome, which combines efficacy and safety and which might be more relevant for patients that the mere reduction of hallucinations and delusions, but with a sample size of 1,900 patients the results are robust (66). The same holds true for social functioning. More time may be needed until antipsychotics develop their full effects, but patients may want to know whether after approximately 6 weeks they are already doing better in this regard. Outcomes that help to understand whether antipsychotics also help social reintegration should become a standard (Figure 2).

Table 1 shows that several study characteristics changed over the decades, and some of them were also significant predictors of drug-placebo differences.

In univariable analyses, drug-placebo differences decreased over time, with an average rate of 0.08 effect size units per decade, signifying that a study from 1970 would have an effect size of approximately 0.74 and a study from 2015 an effect size of 0.38, a trend that has not been stopped by the most recent antipsychotics brexpiprazole and cariprazine (Figure 3 and supplemental Figure S2). Publication year can only be a surrogate for other factors, but it is not surprising that trials without standardized diagnostic criteria or using the BPRS had larger effect sizes than trials with standardized criteria or using the PANSS, nor that the old D₂ antagonists had larger effect sizes than the recent D₂/5-HT_1a partial agonists. In the early studies, standardized criteria and the PANSS were simply not available, and D₂ antagonists were primarily examined in the older, smaller trials. When we analyzed haloperidol separately, its superiority compared with placebo had become smaller over the years as well (Figure 5B), demonstrating that decreasing effect sizes over time cannot be explained solely by more recent drugs being less efficacious.

Larger sample sizes and a related moderator, number of sites, were associated with smaller effect sizes, which is consistent with the funnel plot suggesting substantial small-trial effects, which are well known from other medical fields (54, 67). The patients in small trials might be better selected than those in large trials. In contrast, the methodology of the often older, small trials was less stringent. For example, independent monitoring is a relatively recent requirement. However, our specific tests suggest that studies were missing, at least in part, because of publication bias.

The purpose of minimum baseline severity thresholds is to have drug-responsive populations (49), but, counterintuitively, studies with severity thresholds had lower effect sizes, possibly because such criteria invited artificial baseline inflation. Although the direction of the effect changed in the multivariable analysis, suggesting that this moderator might be confounded by another one, alternative ways to have severely ill populations should be considered.

However, in the multivariable meta-regression, the only moderators that remained significant were the degree of placebo response and industry sponsorship, which, contrary to criticisms and our expectations (6), was associated with smaller effect sizes. Industry studies are often large and involve multiple countries and sites, leading to problems such as cultural differences in the interpretation of psychopathology, which may increase variability and decrease effect sizes. The “patent clock” is running down, thus patients are recruited quickly by professional centers. As multiple effective antipsychotics are available, patients think twice before consenting to a placebo-controlled trial and those who do consent can create negative selection; examples include (partial) nonresponders to previous drugs or so-called “professional patients” who benefit from a free trial of medication by answering a newspaper advertisement. These factors may also contribute to high placebo response.

Agid and colleagues (9) focused on predictors of placebo response, while our research question was about the drug-placebo difference. To explain placebo response was important from a methodological point of view, but for patients and psychiatrists it is the drug-placebo difference that counts. In this context it was by no means self-explanatory that—together with pharmaceutical sponsorship—placebo response was the strongest predictor of efficacy effect sizes in our analysis. Some other factor could have well been more important, and a parallel increase of drug response, which would have attenuated placebo response, was a priori likely. Only a few significant predictors of placebo response in the study of Agid et al. (9) were also significant predictors of drug-placebo differences here, at least in univariable analyses (publication year and the number of sites). In addition to the different research question, another possible explanation is that our database was two times larger (105 randomized controlled trials versus 50 in the study by Agid et al.), which made our results more robust.

Rutherford and colleagues (10) addressed drug-placebo differences, but four out of six of the findings that they emphasized in their abstract were not confirmed by our multivariable analysis. They used multilevel meta-analysis and hierarchical modeling, which is another valid method.

It is unclear how much this difference in methods accounted for trial duration and severity, findings emphasized in the abstract by Rutherford et al. (10), not being significant moderators in our analysis. But the fact that we had 2.5 times more placebo-controlled studies available could have changed many findings. The major difference in results was that in the Rutherford et al. analysis (10) increasing response in the placebo arms was paralleled by decreasing response in the drug arms. In our analysis, drug response remained stable over the years (Figure 5D). In the study by Rutherford et al. (10), many of the data on drug response were from trials that compared drugs head to head (208 arms) and were not placebo-controlled trials (39 arms) (68). Although trial type was statistically controlled for, this is a quite different population of trials. For example, dropout rates are much higher in placebo-controlled trials than in active-controlled trials (69). When we reanalyzed the drug arms in the 39 placebo-controlled studies of Rutherford et al., drug response remained stable (online supplemental Table S8). Thus, drug response has decreased only in active-controlled studies, not in placebo-controlled studies. This has major implications for drug development: To improve signal detection in placebo-controlled trials, researchers need to focus on reducing placebo response rather than on increasing drug response. Finally, Agid et al. (9) did not detect publication bias and Rutherford et al. (10) did not explore it.

The major limitation is that all antipsychotics were analyzed as a class, because efficacy differences between individual drugs are thought to be small (except clozapine, for which one trial with only nine patients was excluded) (11, 70). The number of drugs involved rendered it impossible to fully control for the resulting heterogeneity. Additionally, many older studies were so poorly reported that it was impossible to extract outcome data. For example, two early large Veterans Affairs studies (312 and 692 patients, respectively) showed a significant superiority of antipsychotics compared with placebo, but an effect size could not be calculated (71, 72). In one of them the difference in response rates was as high as in the NIMH study (7) (51% responded to antipsychotics versus 8% to placebo) (71). Only 46 and 38 studies, respectively, reported on the number of participants with at least “minimal response” and “good response.” It is possible that some authors presented response data based on the cutoff showing the best result. Finally, conventional meta-analyses cannot detect subtle moderators of treatment effects. The main reason is ecological fallacy, i.e., conclusions about individuals that are based on analyses of group data can be biased. Another one is limited variability in the observed means, which could be overcome by meta-analyses of individual patient data, which capture large interindividual variability.

Our results are important on several levels.

First, clinicians can expect that approximately two times more patients improve when treated with antipsychotics compared with placebo, but only a minority will experience a good response in the short term. We need to document better whether antipsychotics only suppress positive symptoms or whether they also help social reintegration, reflected by improvements in social functioning and quality of life.

Second, network meta-analyses need to consider possible temporal changes. If placebo-controlled trials on one drug developed in the 1970s are combined with those for a drug developed in the 2010s, the older drug might artificially turn out better owing to higher effect sizes in that period. In a previous report we therefore excluded placebo-controlled trials in a sensitivity analysis and examined publication year as a moderator (11).

Third, industry sponsorship has not inflated effect sizes. But there was publication bias, because companies do not always publish inconclusive studies. Increasing placebo response, but not decreasing drug response, contributed to the decreasing effect sizes over time. Finally, sample size and related measures arose several times as significant moderators, and these are modifiable design features for drug development. There could be a vicious circle. Sample sizes have increased continually over the years (see online Figure S4). Companies conduct large trials to assure statistical significance. The inclusion of many patients and sites leads to more recruitment pressure and variability, which, by definition, reduces effect sizes (SMD=mean difference/standard deviation). The next sample size estimation will suggest an even larger sample. We recommend somewhat smaller studies, but with better selected patients, to reverse this trend.