Pitfalls of Meta-Analyses

To the Editor: In recent years, psychiatry has been relying to an ever-increasing extent on meta-analyses to infer the efficacy of psychiatric treatments. Meta-analysis is a useful tool for integrating data from different studies using similar treatments on similar patients. Consequently, meta-analyses provide the field with an estimate of the effect size of a treatment. Many assumptions are made in meta-analysis; among the most important are that studies of comparable quality be included in the meta-analysis, that study quality be weighed in the computation, and that sample sizes be roughly comparable. Besides statistical issues, there are assumptions that in order to be accurate a meta-analysis must include all relevant research studies and be based on studies that have comparable patient populations.

Although Leichsenring and Leibing (1) attempted to do a conscientious job of comparing the efficacy of dynamic therapy to cognitive therapy in patients with personality disorders by using meta-analytic techniques, they failed to include several studies: two of avoidant and obsessive-compulsive personality disorder (2, 3) and one of borderline personality disorder (4). In addition, they did not include any studies involving group therapy. Omitting relevant studies when so few are available raises questions not only about the generalizability of this meta-analysis but also about the technology and methods used in selecting studies for inclusion. Thus, we strongly recommend that meta-analysts describe the exact terms used in searches and that searches using similar terms be conducted across different databases. There are limitations to blind computerized searches; therefore, searches must be supplemented by, for example, looking at references of relevant articles and by contacting experts.

Although we sympathize with the wish to examine the effectiveness of different psychotherapy systems for patients with personality disorder, there is a clear limitation of lumping together studies looking at the efficacy of treatment for so many different kinds of patients. It could be that instead of having found support for the dodo bird verdict (psychotherapeutic equivalence), these studies could show that different therapies are better at helping patients with different personality disorders. Therefore, the lack of apparent differences between treatments might, for example, be due to the possibility that cognitive therapies are more effective with patients with avoidant personality disorder, while interpersonal/dynamic therapies might be more effective with obsessive-compulsive personality disorder.