Selection Bias Effects on RAISE-ETP Results

To the Editor: The Recovery After an Initial Schizophrenia Episode Early Treatment Program (RAISE-ETP), described by Kane and colleagues in the April 2016 issue of the Journal (1), represents a serious effort to address a significant area of unmet need in the treatment of patients with first-episode psychosis. However, equally serious methodological flaws mean that the authors’ conclusion that RAISE-ETP improved outcomes cannot be relied upon.

The simplest explanation for the results is that there were substantial differences in the patients entering treatment and control groups. Supplemental material shows that 512 patients were screened at the treatment sites (272 eligible, 240 excluded), while 795 patients were screened at control sites (231 eligible, 564 excluded). The ratio of eligible to excluded patients was 113% at treatment sites and was 40% at control sites. Either the populations from which the treatment and control groups were selected were radically different, or the selection procedures were radically different. It is surprising the article does not consider the possibility of selection bias.

Given the importance of duration of untreated psychosis in their rationale and analysis, it is also surprising the authors do not refer to the Treatment and Intervention in Psychosis study (TIPS) (2), the gold standard. TIPS was shown to successfully reduce the duration of untreated psychosis of Scandinavian patients from 26 weeks to 5 weeks by early response teams and public health interventions (2). At 10-year follow-up, there was no difference on original outcome measures, suggesting that associations between long durations of untreated psychosis and poor outcomes are not because delays in treatment predict poorer outcomes, but because patients with more severe forms of illness take longer to access care.

While Kane et al. note that there was no statistical difference in the duration of untreated psychosis (1), the mean duration of untreated psychosis in the control group was 33 weeks longer than it was in the treatment group. Given the large standard deviation, as well as the large difference in the mean duration of untreated psychosis (211 weeks for the control group compared with 178 weeks for the treatment group) and in the median duration of untreated psychosis (88 weeks for the control group compared with 66 weeks for the treatment group), the lack of statistical significance is likely due to a combination of questionable test selection and inadequate power. The combination of clear selection bias demonstrated by participant flowcharts and greater duration of untreated psychosis in the control group suggests that the reason the treatment group did relatively better over 2 years was because there was a bias toward selecting patients with a worse prognosis in the control areas.

With respect to possible sources of bias, it is interesting that while Kane et al. state unequivocally that no clinics withdrew from the study after randomization (1), in their study design article they report that two sites were dropped after randomization and initial training, with another site added (3).