Response to Amos: Addressing Purported Methodological Flaws in RAISE-ETP Results

To the Editor: We appreciate Dr. Amos’s comments on our study on the Recovery After an Initial Schizophrenia Episode Early Treatment Program (RAISE-ETP) (1), but we do not believe that our conclusions were the result of methodological flaws. Dr. Amos specifically contends that the reason that participants in NAVIGATE, the comprehensive approach to first-episode psychosis care, did better across the 2-year observation period was that participants in usual, community care had a worse prognosis at entry.

As presented in our prespecified data analysis plan (http://raiseetp.org/studymanuals/RAISE_Statistical_Plan.pdf), our assessment was that a cluster randomized design offered substantial advantages for our study. The rationale is presented in the article (1). Our approach considered the possibility that there would be differences in participants at the NAVIGATE and community care sites. We specified the baseline demographic and clinical variables that would be compared between groups, and in the event of baseline differences, we specified criteria for including variables as covariates in the outcome analyses (briefly, variables needed to differ significantly and be correlated with outcome). Of the prespecified variables, NAVIGATE participants differed significantly from community care participants on four measures. NAVIGATE had significantly more males (77.6% compared with 66.2%; p=0.05); NAVIGATE had a smaller proportion of participants with prior hospitalization (76.3% compared with 81.6%; p<0.05); NAVIGATE participants had worse total scores on the Positive and Negative Syndrome Scale (p<0.02); and NAVIGATE had fewer participants attending school at baseline (16.0% compared with 26.0%; p<0.02). Three out of the four differences would suggest a worse prognosis in the experimental NAVIGATE group, contrary to Dr. Amos’s contention of a worse prognosis among community care participants. Table 2 of the article shows which variables met the statistical analysis plan criteria for inclusion as a covariate in each analysis.

Furthermore, we do not agree with the additional suggestion that differences in the number of potential participants screened at NAVIGATE sites and at community care sites resulted in selection bias. Screening results can vary between community sites for any number of reasons, including different referral patterns or the personalities of the screeners. However, consistent and systematic application of inclusion and exclusion criteria is key. RAISE-ETP utilized a cadre of well-trained clinical professionals who were blinded to site and cluster assignment in order to select participants on the basis of standardized diagnostic and assessment interviews. As noted above, this strategy minimized potentially confounding differences between treatment and control groups. In addition, for clustered randomization at site level, we used site-level random effects to account for any potentially unmeasured differences between the two treatments.

Regarding the duration of untreated psychosis, as Dr. Amos suggests, our study sample size was chosen based upon power to detect differences across conditions in quality of life and not in duration of untreated psychosis.

As Dr. Amos noted, the difference in mean duration of untreated psychosis between conditions was small in comparison with the standard deviations in this measure. Unlike Dr. Amos, we believe that the selection of statistical tests was appropriate. These analyses showed no significant difference with two tests of duration of untreated psychosis (the p value of the comparison of the mean duration of untreated psychosis was 0.35 unadjusted and was 0.33 adjusted, and the p value for the median duration of untreated psychosis was 0.35). These results did not meet the statistical plan criteria for the inclusion of duration of untreated psychosis as a covariate, and thus we believe that our exclusion of duration of untreated psychosis from the variables included as covariates was warranted.

Although the duration of untreated psychosis was not significantly different between groups at baseline, it did emerge as a significant moderator of several outcomes. We did not comment on the Treatment and Intervention in Psychosis study because a discussion of whether reducing the duration of untreated psychosis would change outcomes was beyond the scope of our article. We found that stratifying comparisons by duration of untreated psychosis was associated with significant differences in outcomes. We did not seek to explain the mechanism behind this difference.

The statement in our earlier article (2) that none of the sites withdrew after randomization referred to the potential risk that sites would initiate withdrawal because of disappointment in their cluster-randomized assignment. We did discontinue two sites due to poor enrollment.

We appreciate the opportunity to address Dr. Amos’s concerns, but we believe that given the nature of this study, we have done everything appropriate to demonstrate that our conclusions are valid and follow the data as closely as possible.