The Evolution of Meta-Analysis in Psychotherapy Research
Publication: American Journal of Psychiatry
Psychodynamic psychotherapies have a smaller evidence base than do cognitive-behavioral treatments (1). However, their study quality is not worse, and as with all psychotherapy research, quality improves with year of publication (1, 2). Nonetheless, the relative lack of research into outcomes of psychodynamic therapy, along with politicization of psychotherapy guilds (3, 4), has led to its increasing marginalization and endangerment (3). It could conceivably become difficult for patients to find dynamic treatment in the relatively near future if this problem is not addressed.
Christiane Steinert and colleagues (5) have contributed a ground-breaking meta-analysis evaluating the equivalence of efficacy of psychodynamic psychotherapies with other efficacious treatments “for mental disorders.” In as well-controlled a meta-analysis as any to date, the authors demonstrate equivalence of psychodynamic psychotherapy with other active treatments, a finding with important public health implications. This brings potential hope for improved patient access to dynamic therapy (6). As the authors note in their conclusion, however, all treatments for mental disorders have a wide margin for improvement, as rates of response and remission remain relatively modest.
At first blush, it might seem that addressing such a broad question—efficacy of psychodynamic psychotherapy for “mental disorders”—which these authors have been attempting to do for years (7), is too far reaching. Indeed, it is: the 23 randomized controlled trials in the meta-analysis cover six diagnostic categories: mood, anxiety, posttraumatic stress disorder, eating, personality, and substance use disorders, but not psychosis or the fuller range of psychopathologies. Although the study has limitations, it nevertheless represents an important advance for two reasons. The primary reason is that, for the first time, a meta-analysis of psychotherapies has incorporated into its analytic approach measures of psychotherapy study quality and researcher allegiance, two design flaws in past meta-analyses of psychodynamic psychotherapy (8). Secondly, this is the first well-enough conducted meta-analysis of psychodynamic psychotherapy efficacy for multiple mental disorders to generate credible conclusions about its equivalence with other efficacious treatments: cognitive-behavioral therapy (CBT) and selective serotonin reuptake inhibitors. The study findings lend additional support to the growing body of randomized controlled trial evidence demonstrating efficacy equivalence of dynamic therapies with CBT, particularly for depression (9, 10).
Meta-Analyses and Their Limitations
Research into psychotherapy outcomes is a cottage industry that depends on scarce, hotly contested, and increasingly vanishing federal and international funds. Studies are expensive and require meticulous stewardship. They are perforce relatively small, even when supported by grants from the National Institutes of Health or the European Union. These and other operational challenges make it difficult to draw broad conclusions about the effectiveness of various psychotherapies on the basis of even the largest individual studies. Hence, meta-analysis is an approach that can apparently address the problem of small samples: Why not combine outcomes from a large group of studies so as to have a larger N to evaluate? This seemingly sensible approach hides a world of compromise. Few psychotherapy studies, even for the same target disorder, boast identical study entrance criteria or research benchmarks monitoring “response” or “remission,” so combining cross-study data can lead to inaccuracies. When meta-analyses include studies addressing outcomes for a range of psychiatric disorders, tracking outcomes across various disorders and outcome measures can become a free-for-all, a progressively messy endeavor with the addition of each target illness (7). The Steinert study suffers from this drawback, despite the study’s relative strengths.
It is even more problematic that a meta-analysis, or any large-scale lumping of data derived from various individual studies, has heretofore not incorporated formal psychotherapy-specific measures of study quality, thereby diluting data from less biased studies so they carry the same weight as heavily biased, poor-quality studies (6). It is worth noting that Cuijpers et al. (2), who have conducted more than 100 meta-analyses of the effects of various treatment interventions on a wide range of psychiatric and psychological problems, employ the Cochrane study quality measure (11). However, this measure is not specifically designed to track a range of potential bias encountered in psychotherapy outcome studies as accurately as the Randomized Controlled Trial Psychotherapy Quality Rating Scale (12) does, which Steinert et al. (5) employ. Thus, meta-analytic findings are less reliable as to true efficacy of specific interventions than are individual, well-conducted randomized controlled trials of very good quality. The incorporating of investigator bias into the relative weight that a study carries also has not been done before in meta-analyses evaluating the efficacy of psychodynamic psychotherapy. It is an important improvement.
Additional advances in the Steinert study include considering only psychotherapy studies with active comparison interventions (either CBT or medication) and focusing primary analyses only on primary, standard, accepted measures of symptomatic improvement, albeit across disorders. These advances minimize the usual meta-analytic “noise.”
A chief problem of psychotherapy research meta-analyses is study inclusion criteria. Such choices, presumably determined from an a priori rationale, profoundly influence findings (13). This study, like all meta-analyses, suffers from this problem, as the authors note. Specifically, inclusion of several older studies (five published before the late 1990s) means that untracked medications and psychotherapy interventions during the intervention and follow-up periods were likely to have had a diluting, unreported effect on outcomes, a ubiquitous problem in older psychotherapy research literature before study quality improved (2, 12, 14). Including outcome studies across “mental disorders” is an overly broad target and is not ideal for a meta-analytic approach. Nonetheless, in light of dwindling funding around the world for psychotherapy outcome studies in the foreseeable future, the authors can be forgiven this.
Conclusions
It is heartening that in this first truly well-conducted meta-analysis designed to limit poor study quality and allegiance bias, dynamic psychotherapies were found equivalent to other efficacious treatments across a range of mental disorders. This study, by design, cannot engage specific treatment approaches within the varied “psychodynamic” domain. What we can do to improve our outcomes must become a research focus. In the future, improvements in these outcomes may well be achieved through better incorporation of recent mediation and mechanism findings about specific dynamic therapy approaches and targets, thereby helping clinicians to hone even more active, useful patient interventions (4).
Acknowledgments
Supported by a fund in the New York Community Trust established by DeWitt Wallace.
References
1.
Thoma NC, McKay D, Gerber AJ, et al: A quality-based review of randomized controlled trials of cognitive-behavioral therapy for depression: an assessment and metaregression. Am J Psychiatry 2012; 169:22–30
2.
Cuijpers P, Cristea IA, Karyotaki E, et al: How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry 2016; 15:245–258
3.
Milrod B: The IOM framework for developing evidence-based standards in the field of psychosocial interventions for mental illness and substance abuse: a dynamic researcher’s perspective: cause for concern. Depress Anxiety 2015; 32:796–798
4.
Markowitz JC, Lipsitz J, Milrod BL: Critical review of outcome research on interpersonal psychotherapy for anxiety disorders. Depress Anxiety 2014; 31:316–325
5.
Steinert C, Munder T, Rabung S, et al: Psychodynamic therapy: as efficacious as other empirically supported treatments? A meta-analysis testing equivalence of outcomes. Am J Psychiatry 2017;174:943–953
6.
Markowitz JC, Milrod BL: What to do when a psychotherapy fails. Lancet Psychiatry 2015; 2:186–190
7.
Leichsenring F, Rabung S: Effectiveness of long-term psychodynamic psychotherapy: a meta-analysis. JAMA 2008; 300:1551–1565
8.
Wampold BE, Flückiger C, Del Re AC, et al: In pursuit of truth: a critical examination of meta-analyses of cognitive behavior therapy. Psychother Res 2017; 27:14–32
9.
Driessen E, Van HL, Don FJ, et al: The efficacy of cognitive-behavioral therapy and psychodynamic therapy in the outpatient treatment of major depression: a randomized clinical trial. Am J Psychiatry 2013; 170:1041–1050
10.
Connolly Gibbons MB, Gallop R, Thompson D, et al: Comparative effectiveness of cognitive therapy and dynamic psychotherapy for major depressive disorder in a community mental health setting: a randomized clinical noninferiority trial. JAMA Psychiatry 2016; 73:904–911
11.
Higgins JPT, Altman DG, Gøtzsche PC, et al: The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343:d5928
12.
Kocsis JH, Gerber AJ, Milrod B, et al: A new scale for assessing the quality of randomized clinical trials of psychotherapy. Compr Psychiatry 2010; 51:319–324
13.
Barber JP, Milrod B: Pitfalls of meta-analyses (letter). Am J Psychiatry 2004; 161:1131
14.
Milrod B, Busch F: Long-term outcome of panic disorder treatment: a review of the literature. J Nerv Ment Dis 1996; 184:723–730
Information & Authors
Information
Published In
History
Accepted: May 2017
Published online: 1 October 2017
Published in print: October 01, 2017
Keywords
Authors
Funding Information
Weill Cornell Medical College10.13039/100007273: CTSC Translation Award UL1 TR000457
A Fund in the New York Community Trust established by DeWitt Wallace
The author reports no financial relationships with commercial interests.Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).