Comparison of Clozapine and Risperidone

To the Editor: In their interesting comparison of clozapine and risperidone in partially responding schizophrenic patients, Alan F. Breier, M.D., and colleagues (1) made several statements for which interpretation may be problematic.

First, this extremely small group, with subgroups, clozapine or risperidone, and two measurement time points, pretreatment and 6 weeks into treatment, was correctly analyzed as a two-group analysis of covariance (ANCOVA) by using the baseline score for each subject as the covariate. However, the small group size (N=29, with 14 subjects assigned to risperidone and 15 assigned to clozapine treatment) makes interpretation of any nonsignificant results problematic.

For example, in their analysis of the response rates, the authors found no difference between the two treatments (χ²=0.9, df=1, p<0.34). With the small group size, a change in the response status of one subject in each group changes the p value from the 0.34 they found to 0.07, which is a lower value than that found in the results they called “a trend” later in the article. Failure to reject a null hypothesis should not be taken as tantamount to deciding the null hypothesis is true. In several places in the article, there were statements such as “We found no significant differences between the two agents for negative symptoms” (p. 296) and “Moreover, neither drug demonstrated significant reductions in negative symptoms” (pp. 296–297). Although the language was very carefully and correctly used to state that no significant differences were found, we believe a typical reader might infer from the language that the authors intended to communicate evidence that no difference exists. Had the authors placed a statement in the article noting the small group size, and had they done a post hoc power analysis, they might have helped readers understand that with the size of the group, they had little chance of finding any except a very large difference to be statistically significant.

Second, they made errors in the degrees of freedom in their chi-square tests in several places. On page 296, the chi-square tests in the first two paragraphs are stated to have 28 (N–1) degrees of freedom. Both tests used 2×2 analysis, and each test has only one degree of freedom. The value of the chi-squares calculated in both cases are correct, as are the p values. It was simply the degrees of freedom that were incorrect.

Third, in several places, the authors used a test of parallel slopes in the ANCOVA to decide whether to use a single-slope ANCOVA model. When they rejected the null hypothesis of single slopes, they moved to an analysis of change scores. This is illogical. Using an ANCOVA assuming equal slopes makes one assumption: that the slopes are equal. Using change or difference scores not only assumes that the slopes are equal, but it assumes they both equal 1. Thus, an analysis of variance (ANOVA) on change scores is simply an ANCOVA with the slopes not only constrained to be equal but to both equal 1. (The two-group, unequal-slope, ANOVA model may be stated as Y_ij=µ_i + β_iX_ij + s_ij, where the Y_ij’s are the endpoint scores, the X_ij’s are the baseline scores, the µ_i’s—with appropriate constraints—are the two group population means, and the β_i’s are the two slopes. If the authors are unwilling to assume that the β_i’s are equal, they should be even more unwilling to assume that they both equal 1. [If we substitute 1 into the model for β, we obtain Y_ij=µ_i + (1)X_ij + s_ij, and if we use algebra to move that term to the left side of the equals sign, we have Y_ij – X_ij=µ_i + s_ij, which is the analysis the authors chose—an ANOVA on change scores.])

Fourth, in table 2 in the article, change is reported as a percent. If the percent change is really of interest, then it should have been analyzed. It is not correct to analyze the data, without transformation, by t tests of differences or even by an ANCOVA. The outcome of interest should be what is used in the analysis. This could be done by using percent change or by possibly, more appropriately, using a log transformation with an ANCOVA. A partial attempt to look at percent change was made by the investigators when they looked at a 20% change in the Brief Psychiatric Rating Scale scores.

Fifth, no justification for the 20% cut point was given. Typically, a 50% change in scores indicates a response. Had the authors used 50% improvement as their cut point, fewer subjects than the already small number would have met the criteria for response, and it would have been more obvious that the study had failed to detect group differences.

Sixth, the analysis of neuroendocrine levels is problematic. Typically, these levels can fluctuate considerably during the day. Not only is there a diurnal variation with many neuroendocrine parameters, but they fluctuate widely within a relatively short period of time. The wide variance found for the groups indicates such a fluctuation. It also makes the use of a t test inapplicable with such a small group.

It is probable that the major conclusions of this article, particularly with respect to the differences found between clozapine and risperidone, will hold. It may be less plausible that the differences Dr. Breier and colleagues failed to find indicate more than a small group size and lack of power. Our criticisms revolve around about the analysis and the presentation of results.