Limitations of Retrospective Research

To the Editor: There are several issues with the article by Tami L. Mark and colleagues (1) published in the October 2010 issue of the Journal that deserve attention. The first issue is that of validity. The statistical analyses performed do not solve the concerns with the validity of the fundamental question in this paper: Does it make clinical sense that the branded antidepressant medications available between 2003 and 2006 would outperform the generic antidepressants in effectiveness? Or, even if that argument was valid for that period of time, does it make clinical sense that the currently available branded medications would be able to outperform the currently available generic antidepressants in effectiveness, to the point of affecting outcomes?

The article has other limitations. The most concerning is the significant difference in baseline characteristics between the two groups. The group who received step therapy had a lower average income, higher levels of comorbidity, and a higher burden of chronic illness, indicating that the group that received step therapy might have been at higher risk for worse outcomes and higher utilization. Although the study used statistical models to adjust for the differences between the two groups, it is likely that baseline differences in the two groups explain the difference in outcomes. The currently available models for risk adjustment in psychiatry perform poorly and usually do not explain more than one-third of the variance (2).

Another piece of information missing is the baseline utilization of the two groups prior to the implementation of step therapy. Prior year expenditures have been shown to outperform any other risk-adjustment model when analyzing utilization and expenditure (3, 4). It would be important to know if the group in which step therapy was implemented already had higher levels of utilization.

The article did not consider the complexity of the treatment of depression, including the use of polypharmacy. Patients treated for depression commonly receive prescriptions for several other psychotropic medications (5). Any attempt to understand the effects of change in pharmacy benefits, including step therapy, should include a broader analysis of all classes of psychotropic medications, especially antipsychotics, which have been increasingly used in combination with antidepressants.

Hopefully, the issues raised above will help readers better understand the complexity of this kind of study.