Validity of Controlled Clinical Trials of Psychotherapy: Findings From the NIMH Treatment of Depression Collaborative Research Program

Participants were outpatients between the ages of 21 and 60 who met Research Diagnostic Criteria (16) for a current episode of major depressive disorder and who scored 14 or higher on a revised 17-item Hamilton Depression Rating Scale (17). Exclusion criteria included certain other psychiatric disorders (bipolar and psychotic disorders), concurrent psychiatric treatment, medical conditions that contraindicated use of imipramine, and the need for immediate intervention (e.g., active suicide potential). Two hundred fifty potential participants met these criteria and were randomly assigned to one of four treatment conditions. The 239 participants who entered treatment were primarily women (70%) and Caucasian (89%). The average age was 35. Seventy-seven participants terminated prematurely (before completing at least 12 sessions of therapy and 15 weeks of treatment). There were no significant differences among the psychosocial treatments in the number of premature terminations (11). The remaining 162 participants are referred to as the “completer group.” The mean number of sessions for subjects in the completer group was 16.2, the maximum was 20.

Participants were randomly assigned to one of four brief treatments for depression: interpersonal psychotherapy, cognitive behavior therapy, imipramine plus clinical management as a standard reference treatment, and pill placebo plus clinical management as a double-blind control group. Patients improved in all treatment conditions, and the effect sizes of the psychotherapeutic treatments were found to be consistent with those reported in the literature (11, 18). When differences between the treatments outcomes were observed (19, 20), they were inconsistent, not robust, and appeared only in specific areas. It is generally accepted that the treatments in the NIMH Treatment of Depression Collaborative Research Program were effective and equivalent overall.

Interpersonal psychotherapy and cognitive behavior therapy were carried out in accordance with detailed manuals (21, 22). Ten therapists delivered interpersonal psychotherapy, eight delivered cognitive behavior therapy. The therapists, 72% (N=13) of whom were men, averaged over 11 years of clinical experience. They were all trained in the treatment they provided, and tapes of sessions were reviewed to monitor adherence to treatment protocols (15).

Treatment outcome was measured from a variety of perspectives and with an array of different measures. For example, the clinical evaluator-rated Hamilton Depression Rating Scale (17) and the client-rated Beck Depression Inventory (23) measured depressive symptoms; overall functioning was measured by the clinical evaluator-rated Global Assessment Scale (GAS) (24) and the client-rated HSCL-90 (25). Information on the procedures and methods of the NIMH Treatment of Depression Collaborative Research Program beyond what is presented here is available in Elkin et al. (11, 26).

This 100-item instrument furnishes a language and rating procedure for the comprehensive description, in clinically relevant terms, of the therapist-patient interaction in a form suitable for quantitative comparison and analysis (15). The Psychotherapy Process Q-Set uses an entire hour as the unit of observation, allowing a greater opportunity to capture events of importance. Clinical judges read the verbatim transcript of a therapy session and then sort the 100 items in the Q-set on a continuum from least characteristic or negatively salient (category 1) to most characteristic or salient (category 9). The middle pile (category 5) is used for items deemed either neutral or irrelevant to the particular hour being rated. Most items have specific instructions that provide examples of the distinction between uncharacteristic and neutral ratings. For example, one item describes the therapist as “distant or aloof” when rated in the characteristic range. However, when rated in the uncharacteristic range, the item indicates that the therapist was “genuinely responsive or affectively involved” (the opposite of “distant or aloof”). Only if the item were irrelevant to the description of the hour would it be placed in the neutral range. The number of cards sorted into each category of the Q-sort (from five at the extremes to 18 in the middle or neutral category) conforms to a normal distribution, requiring judges to make multiple evaluations among items, and thereby avoiding halo effects and response sets (27). Judges rate the frequency, intensity, and estimated importance of each of the 100 statements. A detailed coding manual provides the Q-items and their descriptions as well as operational examples. When rating, judges are asked to take the position of a “generalized other,” i.e., an observer who stands midway between patient and therapist and who views the interaction from the outside. In placing each item, judges are instructed to ask themselves: Is this attitude, behavior, or experience clearly present (or absent)? If the evidence is not compelling, the judge is asked to search for specific evidence of the extent to which it present or absent. The rating procedure allows judges to form impressions and hypotheses and study the material for confirmation or alternative conceptualizations. The Psychotherapy Process Q-Set was developed pantheoretically to assess therapist actions in different types of therapy, so it is especially useful for comparing the process of different forms of therapy (28, 29). The Psychotherapy Process Q-Set has been used to rate hours of psychodynamic, cognitive behavior, client-centered, Gestalt, and rational-emotive therapies (29). The Psychotherapy Process Q-Set has demonstrated both reliability and strong discriminant validity across a variety of studies and treatment samples (10, 29–31).

Responses by expert therapists of different orientations to a questionnaire form of the Psychotherapy Process Q-Set were used to develop prototypes of ideal treatments. Therapist panels of experts in cognitive behavior therapy (N=10) and interpersonal psychotherapy (N=11) were formed; the panels comprised leading theoreticians and practitioners of each perspective. All of the therapists were highly experienced and internationally recognized for their expertise. Each member of the expert panels had trained therapists in their orientation, and most had published work concerning their approach to psychotherapy. Many of the members of the expert panels were involved in the creation of the treatment modality.

Members of the expert therapist panels were asked simply to rate each of the 100 items of the Q-set questionnaire, on a scale from 1 to 9, according to how characteristic each item was of their understanding of an ideally conducted course of therapy that adheres to the principles of their theoretical perspective. Each questionnaire yielded one score for each of the 100 items of the Q-set. Each expert was also asked to suggest additional items if they felt that the item set was not sufficient to capture adequately the important aspects of their approach to treatment. There were no consistent suggestions for additional items.

Coefficient alpha reliabilities demonstrated that the level of agreement of the expert therapists’ ratings was high for both orientations (cognitive behavior therapy=0.95; interpersonal psychotherapy=0.96). The prototypes were created by using a small sample statistical method for studying points of view, sometimes called Q-technique (32–35) (see our earlier article [7] for a detailed description). The responses of the therapist experts in interpersonal psychotherapy and cognitive behavior therapy to the Psychotherapy Process Q-Set were subjected to a principal-components factor analysis. As expected, the factor analysis yielded distinct theoretical orientation factors with eigenvalues above 1.0 after varimax rotation, which together explained 70.9% of the variation in the correlations among the expert therapists. In other words, the interpersonal and cognitive behavior therapists had distinct points of view about an ideal treatment process.

Linear regressions were calculated to determine the contribution of each Q-item to the experts’ factors. Factor scores represent the weighted sum of each Q-item. The items with the highest factor scores are most defining of the factor, and the items with the lowest factor scores are least defining of the factor. Table 1 and Table 2 present the 20 items with the highest factor scores for the cognitive behavior and interpersonal therapy techniques, respectively. In the interest of space, only these 20 items are displayed, even though the entire array of factor scores on all 100 items of the Psychotherapy Process Q-Set was included in the Q-prototypes. In other words, each prototype contains all of the 100 items of the Psychotherapy Process Q-Set (not only those appearing in the tables). All 100 items were included in the prototype because it is important that the description of an ideal treatment capture not only those items that are characteristic, but also those that are uncharacteristic and not relevant. That is, a prototype should reflect what is absent from an ideal treatment as well as what is prominent. This is important to consider when evaluating the face validity of the prototypes, since items contributing significantly to a prototype may not be listed among the 20 items with the highest factor loadings. For example, one psychotherapy process item that describes the therapist as challenging the patient’s view or core beliefs received a high factor loading (0.66), indicating that the experts felt it was an important component of cognitive behavior therapy. However, it does not appear in Table 1 because it was not among the 20 items with the highest factor loadings. Because the prototypes consist of all 100 items of the Psychotherapy Process Q-Set, they account for the relative contribution of all items in the item set.

As seen in Table 1 and Table 2, both the cognitive behavior therapy and the interpersonal psychotherapy experts rated six of the same items as being among the 20 most defining aspects of their respective treatments. These items have different factor loadings, so they each contribute to the prototype to a varying degree. For example, the item “therapist adopts supportive stance” has a higher factor loading on the cognitive behavior therapy scale, indicating that the experts judged the item to be more characteristic of ideal cognitive behavior therapy than ideal interpersonal psychotherapy.

A pool of nine research-oriented psychologists and master’s-level graduate students in clinical psychology, all trained in the application of the Q-technique, completed Q-sort ratings of interpersonal psychotherapy and cognitive behavior sessions from the NIMH Treatment of Depression Collaborative Research Program. The raters represented a range of theoretical perspectives, although most were eclectic in their clinical orientations. Two verbatim transcripts, a session early in treatment (session 4) and a session late in treatment (session 12) were selected for each patient in the interpersonal psychotherapy and cognitive behavior therapy groups. When transcripts of these sessions were not available, the next closest session was selected. The length of treatments for these patients ranged from 12 to 20 sessions, with an average length of 16.2 sessions. Transcripts were only available from the archive for two of the three sites that participated in the NIMH Treatment of Depression Collaborative Research Program. Therefore, transcripts were available for only 35 of the 47 participants who received interpersonal psychotherapy and 29 of the 37 participants who received cognitive behavior therapy. All of the transcripts (N=128) were randomly assigned, and independent ratings were completed by at least two judges who were blind to treatment type and session number. Average interrater reliability for Q-sorts achieved 0.82. There were no significant differences in reliability between treatment modalities. When agreement was below r=0.50, the author of the Psychotherapy Process Q-Set, a licensed and practicing clinical psychologist (E.E.J.), was added as a third rater. Periodic calibration meetings were conducted to correct rater drift. Statistical analyses comparing Q-sort ratings from sessions 4 and 12 revealed little change in process over time. The independent Q-sort ratings (sessions 4 and 12) of the judges for each transcript were therefore composited as they have been in previous studies that used the Psychotherapy Process Q-Set. The composite Q-sort ratings were averaged across both sessions to obtain one score per Q-item for each patient.

The Q-prototypes of ideal interpersonal psychotherapy and cognitive behavior therapy were then correlated with the Q-sort ratings of the actual psychotherapy sessions from the NIMH Treatment of Depression Collaborative Research Program in order to determine how close the treatments were to the ideal prototypes. That is, for each patient the composite Q-rating of each of the 100 items was correlated with the factor score for that item from the prototype. These correlations with the Q-prototypes then were transformed into z scores by using the Fisher r-to-z transformation. The z scores represent the degree to which an hour of psychotherapy was correlated with the Q-prototypes. The z scores were then correlated with outcome to determine whether those aspects of the therapy process that in theory should promote patient change were in fact responsible for predicting positive outcome. Partial correlations of the z scores and the outcome measures were calculated in order to control for pretreatment scores.

A remarkable finding of this study was that the nature of the process fostered by therapists administering interpersonal psychotherapy in the NIMH Treatment of Depression Collaborative Research Program was so similar to that fostered by therapists administering cognitive behavior therapy. In fact, interpersonal psychotherapy conformed more to the expert-generated prototype of cognitive behavior therapy than it did the prototype of interpersonal psychotherapy. Interpersonal psychotherapy included a more diverse set of therapeutic processes, including significant aspects of the kind found in cognitive behavior therapy. Cognitive behavior therapy, on the other hand, was more pure in its emphasis relative to its prototype. An earlier study of the NIMH Treatment of Depression Collaborative Research Program showed that the two treatments could be reliably distinguished on a rating scale meant to assess features of therapist activity specific to the respective treatments (13). That study also showed that both treatments contained an equivalent degree of what has been termed common “facilitative conditions.” Our own results suggest that the two treatments were by no means pure and distinct despite adherence checks. The overall process of interpersonal psychotherapy was startlingly similar to cognitive behavior therapy in actual practice, challenging the presumption of large differences across these two brand names of therapy.

Since the interpersonal psychotherapy and cognitive behavior therapy processes can be quite similar, it is not at all surprising that the two forms of therapy could achieve such strikingly similar outcomes in the Treatment of Depression Collaborative Research Program (11). Because the treatments were manualized, it was generally concluded, without having examined the relationship between process and outcome, that the techniques of interpersonal psychotherapy and cognitive behavior therapy were at the same time very different but equally effective. Our results suggest that both interpersonal psychotherapy and cognitive behavior therapy were characterized by a significant degree of overlap, particularly in what experts described as cognitive behavior therapy processes. It was these shared cognitive behavior processes that most robustly predicted positive outcome in both the interpersonal psychotherapy and the cognitive behavior therapy groups. Nevertheless, the interpersonal psychotherapy process also contributed to positive outcome in both treatment modalities. Similarities in process across treatments help explain the lack of differential outcomes frequently observed in trials of psychotherapy. What are often referred to as nonspecific effects in the literature may actually represent the specific effects of shared processes.

One possible explanation for how the process of treatments with different brand names and different prescribed techniques can be so similar lies in the theoretical language used by proponents of different forms of psychotherapy. Clinicians from different orientations may use very different terminology to describe psychological constructs and processes that are actually quite similar. Because of these differences in language and ways of conceptualizing, treatments may appear quite different on the surface. However, once we move beyond theory-specific language and examine what actually occurs during treatment sessions, these two treatments in fact share many common features.

Another likely explanation for the similarity between the two forms of therapy, at least as they were actually conducted, concerns the contribution of the patient to the treatment process. Previous reports on the NIMH Treatment of Depression Collaborative Research Program documenting distinctions between interpersonal psychotherapy and cognitive behavior therapy focused on therapist technique (13). Indeed, most observations of distinctions between treatments tend to focus only on the therapists’ interventions and adherence to prescribed techniques in isolation from the larger context in which they occur. It is usually assumed that influence flows in one direction, i.e., therapist interventions cause patient change. However, the nature of the therapeutic process is inherently interactional. Patient and therapist mutually affect one another and the nature of what occurs in treatment sessions. Every clinician knows that beyond patients’ specific diagnoses, their personal qualities, affect states, and style of relating to others enormously influence what may or may not occur in a treatment session. Our research suggests that the patient’s contribution to the process in interpersonal psychotherapy and cognitive behavior therapy was remarkably similar (14). That is, patients in both brief treatment forms fostered a similar overall process. The overlap in overall process of interpersonal psychotherapy and cognitive behavior therapy is likely the result of strong similarities in patients’ contribution to the process coupled with important areas of overlap in therapists’ stance toward patients.

Contributing to the appearance of large differences across these treatments is a focus on content rather than process. The content of what the cognitive behavior therapist takes up with his patient (dysfunctional attitudes or irrational beliefs) is often quite different from the content on which the interpersonal psychotherapy therapist might focus (e.g., disruptions in personal relationships). However, the nature of the interaction between therapist and patient might be quite similar. For example, a cognitive behavior therapist may coach or educate a patient about more effective ways of thinking (e.g., to not “catastrophize”), while an interpersonal psychotherapy therapist might teach a patient how to manage relationships with co-workers more effectively. The content is different, but the nature of the interaction is similar. In both treatments, the therapist assumes an authoritative role and coaches patients to think or conduct themselves differently and encourages them to test out these new ways of thinking and behavior in everyday life. A narrow focus on the different content of these manualized treatments makes it easy to overlook the high degree of correspondence in process. The Psychotherapy Process Q-Set, which was used to describe treatment processes in the current study, largely avoids the use of theory-specific language and focuses on process rather than content (30). Although the Psychotherapy Process Q-Set has strong discriminant validity (10), these features of the measure may have helped to reveal the similarities across the study treatments.

Our previous study of the process of psychotherapy in the NIMH Treatment of Depression Collaborative Research Program (14), which used individual Q-items as the unit of analysis, suggested more specifically how these treatments helped patients with depressive symptoms. The majority of the individual Q-items that predicted positive outcome in both interpersonal psychotherapy and cognitive behavior therapy described the patient’s contribution to the process rather than the therapists’ actions and interventions. The Q-items associated with change across both interpersonal psychotherapy and cognitive behavior therapy described patients’ experience of a positive sense of self and an unambivalent, deferential, even idealized view of their therapist. This constellation of Q-items captured the presence of a positive attachment to a benevolent, supportive, and reassuring authority figure that apparently facilitated improvement in both forms of brief treatment. Therapists for both the interpersonal psychotherapy and the cognitive behavior therapy groups relied heavily on interpersonal influence; they offered advice and guidance, suggested patients change their behavior, reassured patients that they would feel better soon, and did not, as a rule, discuss features of their interaction with the patient. In spite of their different theoretical orientations, our results suggest that the two short-term treatments shared these common characteristics, essentially supportive in nature, and it is these features that are associated with positive outcome at termination. Taken together, the results of our two studies suggest that what was shared between the two forms of therapy in the NIMH Treatment of Depression Collaborative Research Program was more salient and defining of the treatments than what was different.

The NIMH Treatment of Depression Collaborative Research Program represents a state-of-the-art controlled clinical trial of psychotherapy in which therapists were trained in treatments that were prescribed by manuals and then checked for adherence (13). The controlled clinical trial paradigm enforces all of these experimental controls on the therapy process in an effort to maintain pure and distinct treatments. With such controls in place, improvement during the course of treatment is presumed to be the result of the techniques described in the treatment manual. The results of this study and others suggest that this can be a flawed assumption. It may in fact not be possible to control, to a sufficient degree, how treatments are actually conducted. If it is impossible to fully control the process of psychotherapy, then a basic premise of a randomized, controlled clinical trial is not met. Such studies can only reveal whether a treatment is effective. It is not possible to draw conclusions about what promotes patient change, let alone establish the validity of a treatment’s underlying theory of therapy.

This has, of course, implications for the movement to empirically validate treatments. The primary means for establishing the validity of therapies is the comparative clinical trial. However, if the nature of the treatment cannot be controlled despite the use of manuals and adherence checks, the usefulness of the clinical trials model for validating particular psychological therapies must be questioned. The controlled clinical trial method was initially designed by medical science for use in studies of medications. A physician administers a specific medication knowing it is the only medication being administered in order to compare the results to a placebo condition. Unlike medication, psychotherapy cannot be administered in such pure form, and adherence is much more difficult to measure. The controlled clinical trial method is effective in investigating medical interventions for comparing psychotherapy to pharmacotherapy or their combination. It is limited, however, when imposed on the studies of psychotherapy alone. The reality of clinical practice is that therapists do not exclusively rely on certain techniques or adhere strictly to a particular treatment as manualized. In actuality, therapists often consider disparate intervention strategies to be compatible with their own theoretical orientation.

Our findings must be considered in the context of the limitations of our study design and methods. As in all studies, it is important to consider issues of replicability and generalizability. Because this study relied on a relatively small number of subjects, it is possible that some bias exists in the sample despite random assignment to treatment conditions. It is possible that with a different set of therapists and a different set of patients, less overlap may have been found between the process components of interpersonal psychotherapy and cognitive behavior therapy. Future process studies with other treatment groups are needed to address the generalizability of our findings.

The external validity that is sacrificed in controlled clinical trials of psychotherapy does not seem warranted if the primary goal of establishing causal relationships cannot be attained because of the shortcomings we have described. Our findings raise questions about the usefulness of clinical trials for establishing the validity of psychological treatments. Alternate models are needed to establish which therapies might earn an imprimatur as an “empirically validated treatment.” Many prominent psychotherapy researchers have called for a shift in focus from the study of comparative effectiveness of treatments to the empirical validation of change processes (5, 36, 37). Psychotherapy research would profit from the study of change processes as they occur naturalistically, rather than focusing on the empirical validation of brand names of therapy. Comparative treatment studies emphasize treatment procedures, and it is important to include more fully the role of the individual patient in a treatment’s outcome. It is assumed that if the patient sample is homogeneous in terms of diagnosis (e.g., major depression in the NIMH Treatment of Depression Collaborative Research Program), the most important source of patient variance has been controlled. Our studies have demonstrated that patient in-session characteristics were far more important correlates of outcome than treatment type. This conclusion is consistent with the long-standing finding (38, 39) that patient qualities, including personality characteristics, are far more robust predictors of outcome than are treatment techniques. Naturalistic studies that assess relevant patient characteristics, and their interaction with therapists and intervention strategies, would be a fruitful complement to randomized clinical trials in therapy research.