Continued Use of Evidence-Based Treatments After a Randomized Controlled Effectiveness Trial: A Qualitative Study
Abstract
Objective
This study examined the extent to which therapists who participated in a randomized controlled trial (RCT) of evidence-based treatments continued to use them with nonstudy clients after the trial as well as the types of treatment used and the reasons for their continued use.
Methods
Semistructured interviews and focus groups were conducted with 38 therapists, three clinical supervisors, and eight clinic directors three months after an RCT of evidence-based treatments for depression, anxiety, and conduct disorders among children and adolescents. The therapists had been assigned randomly to one of three conditions: modular (N=15), allowing flexible use and informed adaptations of treatment components; standard (N=13), using full treatment manuals; and usual care (N=10). Grounded-theory analytic methods were used to analyze interview transcripts.
Results
Twenty-six therapists (93%) assigned to the modular or standard condition used the treatments with nonstudy cases. Of those, 24 (92%) therapists, including all but two assigned to the standard condition, reported making some adaptation or modification, including using only some modules with all clients or all modules with some clients; changing the order or presentation of the modules to improve the flow or to work around more immediate issues; and using the modules with others, including youths with co-occurring disorders, youths who did not meet the age criteria, and adults.
Conclusions
The results provide insight into the likely sustainability of evidence-based treatments, help to explain why the outcomes of the RCT favored a modular approach, and highlight the strengths and limitations of use of evidence-based treatments.
In recent years, there have been increased calls for the development and use of hybrid designs to examine simultaneously the effectiveness and the implementation of evidence-based treatments in real-world settings (1,2). An extension of the concept of “practical clinical trials” (3), effectiveness-implementation hybrid designs provide more rapid translational gains in clinical intervention uptake, more effective implementation strategies, and more useful information for researchers and decision makers, among other benefits (1,4–6). Such designs may give equal priority to the testing of clinical treatments and implementation strategies or give priority to the testing of the treatment effectiveness or the implementation strategy.
Increasingly, studies of implementation are embedded within randomized controlled effectiveness trials of specific treatments and practices (7–12). However, it is unclear whether such treatments are sustained once the effectiveness trial is concluded or whether they are abandoned because of lack of funding and external support or undergo adaptation to fit specific community needs, preferences, and support resources. Such information is critical to evaluating on the one hand, the value of hybrid designs, and on the other hand, the factors that promote or inhibit sustainment of evidence-based treatments.
Sustainment, defined as continued use of an innovation in practice (13), is included as the final stage in most models of evidence-based treatment implementation (5,13–15); yet, comprehensive models of factors that support maintenance or sustainment of evidence-based practices in public service sectors are lacking (15), and the factors that facilitate or impede sustainment—such as organizational culture, leadership, funding, and staffing—are poorly understood (15,16).
To address this lack of information, we examined the use of two alternative approaches to delivering evidence-based treatments subsequent to a randomized controlled trial (RCT) designed to evaluate their comparative effectiveness. The study sample was youths ages eight to 13 years being treated for anxiety, depression, and conduct problems. Embedded in the RCT was a qualitative study of the process of dissemination and implementation of the evidence-based practices. In a previous article (6), we reported that therapists anticipated they would continue using the evidence-based treatments upon conclusion of the RCT, but in a selective fashion—in other words, using some elements with all clients and all elements with some clients. In this article, we report on therapists’ continued use of the evidence-based treatments with nonstudy clients upon conclusion of the clinical trial. Our aim was to examine therapists’ reported patterns of use of these treatments for nonstudy clients, reasons for continued use, and reasons for treatment adaptation or modification.
Methods
The Child STEPS Effectiveness Trial
The qualitative study was part of the Child System and Treatment Enhancement Projects (STEPS) Effectiveness Trial (CSET), carried out by the Research Network on Youth Mental Health, which is funded by the MacArthur Foundation. The CSET focused on children ages eight to 13 who had been referred for treatment of problems involving disruptive conduct, depression, anxiety, or any combination of these. Ten clinical service organizations in Honolulu and Boston, 84 therapists, and 174 youths participated in the project. Youth participants were treated at one of these settings with the usual treatment procedures or with three selected evidence-based treatments: cognitive-behavioral therapy (CBT) for anxiety (17), CBT for depression (18), and behavioral parent training for conduct problems (19).
These evidence-based treatments were tested in two forms: standard manual treatment (standard treatment), in which therapists use full treatment manuals in the manner tested in previous research trials; and modular treatment, in which therapists learn all the component practices of the treatments but individualize their use for each child, guided by a clinical algorithm (20) and feedback on practices and clinical progress (21). Therapists assigned to the modular condition used a collection of modules that correspond to treatment procedures included in the standard treatment manuals (22). This Modular Approach to Therapy for Children, called MATCH, prioritizes a focus on the initial problem area identified by using a default sequence of modules outlined in a protocol flowchart. If interference arises—for example, a comorbid condition or stressor impedes use of the default sequence—the sequence is altered, with other modules used systematically to address the interference. For example, if treatment begins with a focus on depression but disruptive behavior interferes, the therapist may use modules from the disruptive-behavior section of the protocol to help parents manage that behavior, returning to depression treatment when the interference is resolved.
Therapists who consented to participate were randomly assigned to one of the three conditions: standard treatment, modular treatment, or usual care. We used a cluster randomization design (23), with therapists assigned to condition by using blocked randomization stratified by therapist educational level (doctoral versus master’s). Youths who met study criteria were randomly assigned to treatment delivered by one of these three groups of therapists (24). Therapists randomly assigned to standard or modular conditions trained together for specific treatment procedures at the same workshops. However, they were given different reference materials (MATCH instructions or the standard manual) and were separated to talk specifically about issues unique to their study condition. For example, MATCH therapists would discuss common structured adaptations and use of the flowcharts, whereas the standard group would discuss strategies to maintain fidelity of the planned sequence of sessions in the face of challenges. Both groups received weekly case consultation from project supervisors familiar with the protocols to help them apply the treatment procedures to their study caseload. Therapists providing usual care received no instructions or feedback about their practice. The usual-care sessions were audio taped, and a coded sample of the content revealed that only 8% involved procedures that were represented in either the modular or the standard protocol (24).
Mixed-effects regression analyses of weekly measures of a standardized brief problem checklist and a patient-generated top-problems assessment showed significantly superior outcome trajectories for modular treatment relative to usual care. After treatment, youths receiving modular treatment had significantly fewer diagnoses than youths who received usual care (24). In contrast, none of these outcomes were significantly different among youths in standard treatment versus usual care. Follow-up analyses of the weekly trajectory measures also showed significantly better outcomes for modular treatment than standard treatment. In general, the modular approach outperformed usual care and the standard approach on the clinical outcome measures, and the standard approach did not outperform usual care. Additional details of the RCT protocol and study results are available in the article by Weisz and colleagues (24).
Dissemination and implementation study
Participants.
Participants included 38 therapists (45% of the RCT participants), all six project supervisors from the RCT, and eight clinical organization directors or senior administrators (80% of the RCT participants). All of the therapists had master’s degrees or higher. A total of 76% (N=29) of therapists were female, and 79% (N=30) were Caucasian; their mean±SD age was 40.6±10.5 years, and they had 11.1±8.4 years of clinical experience. Among them were 14 (37%) social workers, 11 (30%) psychologists, and 13 (34%) other providers, such as licensed mental health counselor. There were no significant differences with respect to treatment assignment, demographic characteristics, or clinical training of the therapists who did or did not participate in the qualitative study. The protocol was approved by the institutional review boards of the University of Southern California, the University of Hawaii, and the Judge Baker Children’s Center, Harvard Medical School. After complete description of the study to participants, informed consent was obtained.
Data collection.
A series of semistructured interviews was conducted approximately three months after the conclusion of participation in the RCT. Interviews focused on the participants’ involvement with the CSET, their experiences of using either standard or modular approaches (depending on their assigned condition), and the continued use of the treatments after conclusion of the trial. Three focus groups, one with project clinical supervisors (N=3), one with therapists in the standard condition (N=3), and one with therapists in the modular condition (N=3), were conducted as a “member checking” procedure (25) to review validity of preliminary findings from the semistructured interviews and obtain additional information on practice patterns considered relevant by participants. Copies of the interview guides are available from the first author on request.
Data management and analysis.
All interviews and focus groups were digitally recorded and transcribed. All data were analyzed by “coding consensus, co-occurrence, and comparison,” a methodology outlined by Willms and others (26) and rooted in grounded theory (27). This analysis included a review of all data to develop a broad understanding of content and to identify topics of discussion and observation, open coding by two investigators to condense the data into analyzable units, axial coding to describe connections between categories and between categories and subcategories, discussion of codes to arrive at consensus regarding their use and comparison of the codes applied to sample texts (kappa=.89), use of NVivo 9 to generate a series of categories arranged in a treelike structure connecting text segments grouped by categories of code, or “nodes,” to further the process of using axial or pattern coding to examine the association between different a priori and emergent categories, and condensing of categories into broad themes through the process of constant comparison.
Results
Continued use of evidence-based treatments
During the clinical trial, all of the therapists assigned to the standard condition (N=13) and the modular condition (N=15) had attended training sessions in each of the three evidence-based treatments. Five of the ten therapists assigned to the usual-care condition received training in MATCH at the conclusion of the trial, but without follow-up clinical supervision of its use, and none had yet incorporated it into their practice.
Twenty-six of the 28 therapists (93%) who had been assigned to the standard or the modular condition reported using the techniques, for example, the fear ladder with clients with anxiety and homework with clients with disruptive conduct, with nonstudy clients subsequent to the conclusion of the trial. One therapist assigned to the standard condition reported that only one of his clients had been a study participant. Once the study ended, he returned to delivering services as usual, in part, because he felt unsure about whether he was applying the treatment correctly. Another therapist, assigned to the modular condition, reported never being assigned a client who was a participant in the study and thus, lacking the clinical supervision provided to participating therapists, did not feel competent to deliver the treatment as instructed.
Reasons for continued use
Reasons for the continued use of the three evidence-based treatments are illustrated in Table 1 with quotes from therapists. The primary reason for continued use among all 26 therapists was their personal experience with the effectiveness of the treatments, as demonstrated by improvements in their clients’ behaviors. Initial skepticism about the efficacy of CBT among therapists whose training reflected a more psychodynamic tradition and concerns about a lack of control over treatment were dispelled. Therapists also reported an improvement in morale because they were learning something new. Seven (27%) therapists also appreciated that the behavioral-problems modules were effective because they engaged parents in the treatment process, even though many therapists had no prior experience with eliciting such parental involvement in a child’s treatment.
| Reason | Example |
|---|---|
| Accepted treatments after using them | “Then at the first training, you could definitely feel . . . I wouldn’t say skepticism, that’s too strong a word, but old habits die hard. And a lot of us are psychodynamically trained, and so there were definitely some rumblings of . . . ‘We’re gonna, you know, jump full force with this, and we’re on board, but it’s gonna be hard.’ And then once we started using it, we, in general, we found that it was, you know, we all really embraced it and really liked it. I mean, I think that we were given a lot of freedom to . . . make it our own.” (modular condition) |
| Valued interactions and support from researchers | “The supervision was excellent. I can’t say enough about how valuable it was. And it was . . . unlike the blocks of counseling time. The supervisors really provided ample time for discussions. So I never felt like . . . they were looking at a watch and saying ‘Okay, well our time is up.’ That was really helpful because sometimes I really wanted to talk things over. Sometimes I really wanted ideas about . . . where to go next.” (standard condition) |
| Valued the structure of the treatments | “I really enjoyed using that structure with this, with the child who was rather vague otherwise, and this way, you know, was able to focus treatment with her.” (modular condition) |
| Valued the evidence base of the treatments | “I think there’s a lot to be said about evidence-based practice, because if there’s evidence backing it . . . as a practitioner, I’d buy into it more . . . knowing that it’s been tried and it’s shown some success over time, and that it has . . . all the trials, and that makes me comfortable as a practitioner . . . to know that what I’m doing is something that . . . has some weight to it.” (standard condition) |
a
For each evidence-based treatment, therapists were assigned to a modular condition, allowing flexible use and informed adaptations of treatment components, or a standard condition, using full treatment manuals.
All 26 therapists also cited the positive interactions and relationships with members of the research team as a reason for their continued use of the treatments. The research team was perceived as respectful and accommodating to the needs of the therapist and the organization. All 26 therapists in the standard and modular conditions stated during the interview that they valued the training and supervision and thought the researchers were helpful and accessible. However, the therapists assigned to the modular approach were more likely than therapists assigned to the standard approach to report that the approach they were assigned to allows for more accommodation and negotiation. Both therapists and supervisors felt that the modular approach gave them more “license” to negotiate with researchers with respect to circumstances in which the modules could themselves be modified or, more often than not, supplemented with additional materials and techniques acquired through experience with working with similar clients. As one therapist assigned to the modular condition observed, “I felt they gave me a lot of freedom within the protocols to kind of present things in a style that felt comfortable to me.”
A third reason for continued use, cited by 18 (69%) therapists in the modular and the standard conditions, was the structure of the treatments. Most therapists found the manuals and the training to be very concrete and easy to follow and considered the structure to be a useful tool for organizing their own treatment plans. Finally, 11 (42%) therapists reported that the treatment increased their confidence in treating clients because it is evidence based.
Patterns of use
Of the 26 therapists who reported using the treatments with nonstudy clients, 24 (92%) reported making some form of adaptation or modification. This group included all 14 therapists assigned to the modular condition, which included a coordinating framework for making informed adaptations of the protocols, and ten of the 12 therapists assigned to the standard condition, who were provided manuals with explicit instruction in how to use the treatments. Only two of the therapists assigned to the standard condition (17%) indicated they had continued using the treatments as they had been trained. The most typical pattern was to use selected components or modules of a treatment with all clients in need of that treatment (consistent with the modular approach) or all modules with some clients (consistent with the standard approach), but not the entire protocol with every client in need of treatment for a particular disorder, such as depression. A second pattern of adaptation, reported by 22 (85%) study participants, was to use the protocols with clients who did not meet the criteria specified in the clinical trial itself. A third pattern of use, reported by 19 (73%) therapists, was to make changes in the presentation of the materials, either by rearranging the order in which the components were delivered or by making changes in the tools used to facilitate the presentation of the modules. As explained by one therapist assigned to the standard condition, “I definitely loved the tools that were there. I just don’t necessarily use them in that order or necessarily all of them.”
Reasons for adaptations and modifications
Our analyses of the qualitative data revealed several reasons for adapting or modifying the protocols for nonstudy clients. These reasons were grouped into three broad categories: client centered, therapist centered, and organization centered. Illustrative quotations reflecting the reasons for adapting or modifying the protocols are provided in Table 2.
| Reason | Example |
|---|---|
| Client centered | |
| Client needs | “For me, I’m using it whenever it’s appropriate. I’m not following it ‘to a T.’ If a youngster needs a contract with their parents in their home or school, I will use that contract and have the school or parent try to work on their behavior using the program for the youngster at home that works, but not with all the kids. It depends on why they’re coming in also. Are they coming in for some trauma? Are they coming in for academic issues?” (standard condition) |
| Competition with more immediate or important issues | “I think it would be hard for me to stick to the regimen of going through the manualized treatment step by step, [because] crisis happens on a daily basis. So, it’s not easy to go through and be that regimented . . . as these other things are happening in between . . . random crises, random family issues, [and] random school issues. So I think it would be difficult to kind of work that in there.” (usual care) |
| Youth’s ability to participate | “I have modified it for different cases because some kids could move faster through . . . these strategies. And so, I’ve sort of adapted them [based on] whether a kid can move faster or slower. . . . So depending on the cognitive ability of the kid I could either move more quickly through the workbook or I would have to work more slowly.” (modular condition) |
| Youth’s unwillingness to participate | “I went to the training, some of the techniques . . . for the kids that I work with, they would probably disconnect. . . . I mean, I could see some of my students . . . ‘checking out’ or thinking like the conversation would be too hard for them to relate to.” (usual care) |
| Parent’s inability to participate | “With the parents, I definitely don’t deliver it in the same order as I was trained to do. You have to start with one thing and . . . go through all the different modules in a certain order. So I don’t do it in that order. I mean there’s some that I don’t sort of mention, because parents are so busy and it’s . . . rare to have so many sessions with them in a row. So you just kind of pick out the most important, and I find that the most important are the rewards.” (modular condition) |
| Parent’s unwillingness to participate | “I do believe that focusing on the parents can make a big difference. And I think the training convinced me that these were good strategies, but . . . some of the time, I might even say half of the time, I’m running into parents that have that ‘just want my child to have somebody to talk to’ idea and then that really seems to be a stumbling block.” (modular condition) |
| Desire to make treatment “client centered” | “I think for me, and I think it’s a reflection of my professional attitude, I would allow my patient or client to drive more of the module or the intervention.” (standard condition) |
| Need to make treatment culturally appropriate | “I’m in a rural area in Hawaii and . . . just interacting with the kids and using a lot of slang or their phrases or their wording just to help relate to them and . . . the language gets too technical or too over their head. I just think it hinders the process.” (usual care) |
| Therapist centered | |
| Flexibility consistent with approach to treatment | “And rather than just putting myself into a bind and saying, ‘I’m only going to use these three protocols all the time,’ well we don’t do that. I mean . . . as a family therapist I use different techniques from different schools of family therapy. There are about 12 different ones. I use different techniques from the different conferences I go to.” (standard condition) |
| Desire to integrate with usual care | “I also feel philosophically, I do a lot of Constructivist Narrative work, so whenever I feel stuck, I’m kind of going back to trying to draw [information] out from somebody, ‘When did they succeed? What does that say about them? How did they accomplish it?’ And that’s different than evidence-based protocol, but it really connects you to the person.” (modular condition) |
| Concern that treatment would interfere with building therapeutic alliance | “There is always a challenge when you’re working with people . . . to try and connect with them and develop trust. And I think sometimes people are going to feel put off by you saying, ‘Well, we’re gonna do these steps . . . like they are sort of looking for . . . wanting to vent, or wanting to connect, or they have their own ideas about what they think would be helpful for them.” (modular condition) |
| Concern for art of treatment | “Well, I mean, I think there’s a little bit of . . . the idea, the classic idea of mental health and therapy of art and science. So yeah, there’s a sense of, uh, individualized and, to each client, individualized in their own style versus . . . I mean, I think they’ve enjoyed doing it, but I have heard that, you know.” (agency director) |
| Not consistent with therapist personality | “I also think that . . . in terms of our various personalities and styles . . . for example, I mean I’m not a very rules-based kind of person. . . . the modular condition is exactly not my style. . . . It wasn’t fun.” (modular condition) |
| Organization centered | |
| Agency priorities | “The protocols were consistent with the goals of the clinic director, but not with the goals of the actual whole agency, cuz the whole agency, there’s a lot of pressure from the agency just to meet productivity.” (modular condition) |
| Timing of treatment | “It is hard in the school setting when the kids are in class and to have to . . . work around their school schedules.” (modular condition) |
| Location of treatment | “To be honest with you, I don’t think I made a lot of modifications. The only way it was somewhat modified was to practice things here at school. For instance, in one of the modules it recommends having a home visit or something like that. . . . We didn’t do any of the home visits.” (modular condition) |
| Length of treatment | “In my environment, manual-based interventions were a little difficult, given the school situation that I’m in because I often get called in to work with a student . . . in the moment . . . I also have to say that there are time constraints at school as well. We don’t have the luxury of being in a clinic where you have . . . a set hour every week.” (standard condition) |
a
For each evidence-based treatment, therapists were assigned to a modular condition (N=15), allowing flexible use and informed adaptations of treatment components, or a standard condition (N=13), using full treatment manuals. Five therapists assigned to provide usual care were trained in the modular condition after the randomized controlled trial but had not used it.
Client-centered reasons.
All of the therapists who reported the selective use of the treatment modules or components indicated that they did so because specific needs or circumstances of their clients limited therapists’ ability to use the treatments, especially the standard approach, as instructed. For instance, nine (35%) therapists reported difficulty in using the modules as instructed while attempting to respond to their clients’ frequent crises. Five (19%) therapists reported not using the treatments as instructed because either the client or the client’s parents were unwilling or unable to participate in treatment as directed by the protocols or because the clients took a particular interest in some specific protocol element. Four (15%) therapists used the treatments with clients who did not meet the age criterion of the clinical trial by making adjustments to the modules to fit the developmental stage of the child. Three therapists expressed a preference for making the treatment “client centered” by allowing clients to decide which elements of the protocol to use. Two (8%) therapists in Hawaii reported having to “translate” the modules to make them culturally appropriate to their clients by framing elements of the parent-training intervention in terms of traditional parent-child relations, avoiding technical jargon, and using cultural idioms of behavior and roles.
Therapist-centered reasons.
Almost all (N=35, 92%) therapists believed that flexibility in using the treatments was consistent with their approach to treatment in general. Eight (31%) therapists sought to integrate the new treatments with more familiar and time-tested therapeutic modalities. Four (15%) therapists reported concerns that using the protocols as instructed would interfere with the development of a therapeutic alliance with the client because the structure interfered with the normal flow of conversation and development of client trust in the therapist. Three (12%) therapists reported that exclusive use of scientifically validated treatments detracted from the “art” of treatment. Finally, two (8%) therapists reported that rigid adherence to strict protocols was inconsistent with their personality.
Organization-centered reasons.
Ten of the 24 therapists (42%) who reported making modifications to the standard or modular intervention were required to do so as a condition imposed by the work setting. These conditions included the priorities assigned to services delivery and related measures of performance. For instance, in school-based settings, academic performance is the primary indicator of performance—mental health treatments are viewed as important only to the degree that they help to improve academic performance. Thus use of both standard and modular approaches would be terminated as soon as there is evidence of improvement in academic performance, regardless of mental health status. In community clinic settings, where the primary indicator of performance is the number of clients seen, the standard approach of completing every module for every client was not as desirable as a modular approach that reduces the amount of time required for each client, thus enabling the therapist to see more clients. School-based policies determine when, where, and for how long therapists can work with students and parents. Community clinic policies prevent some therapists from engaging with clients outside the office, resulting in some modifications to the anxiety treatment module, which requires spending time with the client outdoors in the community.
Discussion
The results of this study suggest that engagement of therapists in a hybrid RCT of effectiveness and implementation of evidence-based treatment can and does lead to some form of sustainment in community-based settings. Almost all of the therapists interviewed in this study reported use of some or all of the three treatments for nonstudy clients after the trial.
There were four primary reasons for continued use: therapists came to appreciate the utility of the treatments after seeing for themselves the positive outcomes associated with their use; they valued the interaction and support from the researchers and treatment developers and trainers; they valued the structure and organization of the treatment protocols; and they valued the fact that the treatments are evidence based. Successful outcomes, positive interactions with the purveyors of evidence-based treatments, structure of treatment, and a foundation grounded in research appeared to be important to the continued use or sustainment of the treatment protocols.
Our results also suggested that continued use involved some form of modification or adaptation of the standard, manualized versions of the treatments. There were three patterns of modification or adaptation. The first and most common pattern was the use of some of the modules in each protocol with all of the clients or all of the modules with some of the clients, but rarely all of the modules with every client who met criteria for use as specified by the treatment developers. This was true for both the therapists assigned to the standard condition, which encourages completion of all modules for treatment of a specific mental health problem, and the therapists assigned to the modular condition, which explicitly encourages adaptation or modification of the model depending on the client’s needs or circumstances.
The second pattern was to use the modules with other types of clients, including youths with co-occurring disorders other than anxiety, depression, or conduct-related disorders, youths who did not meet the age criteria, and even some adults. The third pattern was to change the order or presentation of the modules to improve the flow or to work around more immediate issues. These patterns were not mutually exclusive.
Our results also indicated three predominant reasons for selective use, modification, or adaptation. The first reason was related to the reported willingness and ability of clients to work with therapists under the guidance of the protocols. Therapists made modifications to the protocols because they perceived that the protocols did not meet the needs of their clients; because they were compelled to deal with more immediate issues or crises; because of a desire to make the treatment client centered by allowing clients to choose the elements of the treatment with which to engage; and because the youths or their parents were either unable or unwilling to perform many of the activities as prescribed, usually for developmental (youths), logistical (parents), or cultural (both youths and parents) reasons.
The second reason for modification was the preferences and priorities of the therapist. Therapists deviated from the protocols as presented by the treatment developers and trainers because of an overriding conviction in the importance of flexibility, even with the modular approach; because of a desire to integrate the new techniques with usual and more familiar approaches to treatment; because of concerns that the protocols would interfere with the therapeutic alliance with the client; because of concerns that therapy should be as much an “art” as a “science”; and because the structure was inconsistent with their own personality.
The third reason was the resources and constraints imposed on treatment by the organizational context. Therapists in school-based settings, for instance, could not use the protocols in the recommended order of presentation or with all modules because of limitations imposed on them by the school schedule, the inability to visit the home, and the priority given to academic performance over mental health. Therapists in community-based settings were under pressure to see more clients, thereby reducing the time available to use the entire protocol of each treatment with individual clients.
The results of this study are consistent with previous studies that have documented organizational (28–31), provider-based (9,32–34), and client-based (9,35) barriers or constraints to the implementation of evidence-based treatments and practices. They are also consistent with previous studies that have focused on adaptations of evidence-based treatments to accommodate the needs of clients belonging to specific age (36) or racial-ethnic (37) groups, provider preferences (38), or organizational context (39).
This study also revealed distinctions between the adaptation of the content of the treatments and the adaptation of the process of using them. Content adaptations were made because organizational constraints limited the opportunities to use every module, therapists concluded that specific modules were not appropriate for the client, or the client (youth or parent) was unable or unwilling to use certain modules. Similarly, process adaptations were made because organizational constraints limited full implementation—for example, if teachers could not or would not visit parents in their homes, they met with them at school; therapists believed that altering the flow of module presentation or delivery would help build the therapeutic alliance, which some regarded as a greater priority than fidelity to the treatment; treatment was interrupted by crises; efforts were made to integrate the new protocols with usual care; and clients were unwilling or unable to use certain modules. Adaptations of both content and process were made as a result of a process of trial and error by therapists, therapists’ desire to make the treatment “client centered” by giving the client a choice or role in deciding in which modules to engage, and therapists’ view that treatment was both an “art” and a “science.”
There are several additional important implications of the findings of this study. First, the naturally occurring adaptations of the evidence-based treatments for nonstudy clients were consistent with the modular approach to treatment use and may help explain why this approach was found by the RCT to produce significantly better outcomes than the standard approach or usual care (24). Such an approach gives the therapist greater flexibility in using the treatment and greater control over the treatment process and is consistent with usual practice.
Second, the findings point to limitations as well as strengths of evidence-based treatments. Variations in organizational and system contexts, therapist skills and preferences, and client needs and characteristics make it difficult if not impossible to use evidence-based treatments that provide little latitude for modification or adaptation. For these types of treatments, giving priority to fidelity may be inconsistent with the increased emphasis on a client-centered approach to treatment (40). Health care changes consequent to the Affordable Care Act are leading states to promote personalized medicine, patient-centered approaches, and outcome monitoring. Use of a modularized approach to treatment and of approaches that allow flexibility is consonant with these larger system changes, even though such use may be inconsistent with the expectations of treatment developers regarding fidelity. States are creating performance-monitoring metrics to encourage use of specific health care indicators to assess, track, and hold accountable health providers, including behavioral health providers, for client outcomes (41). Flexible and adaptable therapies that target specific outcome indicators, such as improvement in functioning and reduction of symptoms, will be used increasingly within this rapidly changing health care context. Similar approaches to using evidence-based treatments within systems of care have been illustrated elsewhere in the literature (42,43). Specifically, use of modular design principles and clinical models that allow structured adaptation of treatment in response to real-time feedback have shown promising results (44).
Several limitations to our study deserve mention. First, this study focused on the factors contributing to sustainment of the evidence-based treatments from the perspective of the therapists, who represented only one group of stakeholders involved in the process of implementation. Other factors known to influence implementation processes and outcomes, such as the availability of funding or broader sociopolitical support for such treatments—for example, consumer demand and government legislation—and the culture and climate of mental health services agencies (11–13), will be presented in a subsequent study. Second, this investigation relied on self-reports of use of evidence-based treatments for nonstudy clients. There are a variety of methodological challenges in assessing practice, and there are potential limitations of relying solely on therapist self-report. However, these reports were verified by both clinic directors and CSET clinical supervisors, who stated during their interviews that study therapists as a group were continuing to use the treatments as indicated in the therapist interviews. Third, the therapists participating in this study may not represent the broader population of therapists who participated in the CSET or otherwise engage in child and adolescent mental health services. Although we found no differences between CSET participants who did and did not participate in this study with respect to demographic characteristics, condition assignment, clinical training, and years of experience, there may have been other characteristics of therapists, such as attitudes toward evidence-based treatments and theoretical orientation, that distinguished the participants from the nonparticipants. Fourth, as a qualitative study, both the collection and the interpretation of data were susceptible to subjective bias and preconceived ideas of the investigators. However, the use of multiple groups of participants (therapists, CSET supervisors, and clinic directors) to achieve “triangulation” (45) was designed to minimize such bias.
Conclusions
Despite these limitations, the results of this study suggest that once trained in the use of a specific evidence-based treatment, community-based therapists were highly likely to continue using it and likely to use it selectively or to make modifications or adaptations by either using some parts with most or all clients or all parts with some clients, by reorganizing the order or presentation of modules, or by using the treatment with groups of clients who did not satisfy evidence-based criteria for use. These patterns of use were determined by the therapist to meet specific needs of individual clients and their families, to integrate them with their own clinical priorities and experience, and to address constraints imposed by the systems or organizations that employ them. Consideration of these factors would seem to be important both in the design of effective evidence-based treatments and in their successful implementation in community-based settings. Future research efforts are recommended to determine whether such consideration results in more effective and sustainable treatments.
Acknowledgments and disclosures
This study was funded through grants 83423 and 90231 from the John D. and Catherine T. MacArthur Foundation and P30 MH074678 and P30 MH090322 from the National Institute of Mental Health. This study was conducted under the auspices of the MacArthur Research Network on Youth Mental Health (www.childsteps.org). During the time of this study, the network included Dr. Chorpita, Dr. Garland, Robert Gibbons, Ph.D., Charles Glisson, Ph.D., Evelyn Polk Green, Dr. Hoagwood, Kelly Kelleher, M.D., Dr. Landsverk, Stephen Mayberg, Ph.D., Jeanne Miranda, Ph.D., Dr. Palinkas, Sonja Schoenwald, Ph.D., and Dr. Weisz (network director).
Dr. Weisz and Dr. Chorpita receive income from a manual describing the modular approach to treatment of children with anxiety, depression, or conduct problems. The other authors report no competing interests.
References
1.
Curran GM, Bauer M, Mittman B, et al.: Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care 50:217–226, 2012
2.
Landsverk J, Brown CH, Chamberlain P, et al.: Design and analysis in dissemination and implementation research; in Dissemination and Implementation Research in Health: Translating Science to Practice. Edited by, Brownson RC, Colditz GA, Proctor EK. New York, Oxford University Press, 2012
3.
Tunis SR, Stryer DB, Clancy CM: Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 290:1624–1632, 2003
4.
Lagomasino IT, Zatzick DF, Chambers DA: Efficiency in mental health practice and research. General Hospital Psychiatry 32:477–483, 2010
5.
Glasgow RE, Chambers D: Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science. Clinical and Translational Science 5:48–55, 2012
6.
Brown CH, ten Have TR, Jo B, et al.: Adaptive designs for randomized trials in public health. Annual Review of Public Health 30:1–25, 2009
7.
Palinkas LA, Schoenwald SK, Hoagwood K, et al.: An ethnographic study of implementation of evidence-based treatments in child mental health: first steps. Psychiatric Services 59:738–746, 2008
8.
Brown AH, Cohen AN, Chinman MJ, et al.: EQUIP: implementing chronic care principles and applying formative evaluation methods to improve care for schizophrenia: QUERI series. Implementation Science 3:9, 2008
9.
Aarons GA, Palinkas LA: Implementation of evidence-based practice in child welfare: service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research 34:411–419, 2007
10.
Blasinsky M, Goldman HH, Unützer J: Project IMPACT: a report on barriers and facilitators to sustainability. Administration and Policy in Mental Health and Mental Health Services Research 33:718–729, 2006
11.
Whitley R, Gingerich S, Lutz WJ, et al.: Implementing the illness management and recovery program in community mental health settings: facilitators and barriers. Psychiatric Services 60:202–209, 2009
12.
Glisson C, Schoenwald SK, Hemmelgarn A, et al.: Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology 78:537–550, 2010
13.
Fixsen DL, Naoom SF, Blase KA, et al: Implementation Research: A Synthesis of the Literature. Florida Mental Health Institute (FMHI) pub no 231. Tampa, University of South Florida, Louis de la Parte FMHI, National Implementation Research Network, 2005
14.
Greenhalgh T, Robert G, Macfarlane F, et al.: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly 82:581–629, 2004
15.
Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research 38:4–23, 2011
16.
Chambers DA: Advancing sustainability research: challenging existing paradigms. Journal of Public Health Dentistry 71(suppl 1):S99–S100, 2011
17.
Kendall PC: The Coping Cat Workbook. Ardmore, Pa, Workbook Publishing, 1990
18.
Weisz JR, Thurber CA, Sweeney L, et al.: Brief treatment of mild-to-moderate child depression using primary and secondary control enhancement training. Journal of Consulting and Clinical Psychology 65:703–707, 1997
19.
Barkley RA: Defiant Children: A Clinician’s Manual for Assessment and Parent Training, 2nd ed. New York, Guilford, 1997
20.
Chorpita BF, Daleiden E, Weisz JR: Modularity in the design and application of therapeutic interventions. Applied and Preventive Psychology 11:141–156, 2005
21.
Chorpita BF, Bernstein AD, Daleiden EL, et al.: Driving with roadmaps and dashboards: using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research 35:114–123, 2008
22.
Chorpita BF, Weisz JR: Modular Approach to Therapy for Children With Anxiety, Depression, or Conduct Problems. Honolulu and Boston, University of Hawaii at Manoa and Harvard Medical School, Judge Baker Children's Center, 2005
23.
Campbell MK, Elbourne DR, Altman DG: CONSORT statement: extension to cluster randomized trials. British Medical Journal 328:702–708, 2004
24.
Weisz JR, Chorpita BF, Palinkas LA, et al.: Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: a randomized effectiveness trial. Archives of General Psychiatry 69:274–282, 2012
25.
Patton MQ: Qualitative Research and Evaluation Methods, 3rd ed. Thousand Oaks, Calif, Sage, 2002
26.
Willms DG, Best JA, Taylor DW, et al.: A systematic approach for using qualitative methods in primary prevention research. Medical Anthropology Quarterly 4:391–409, 1990
27.
Glaser BG, Strauss AL: The Discovery of Grounded Theory: Strategies for Qualitative Research. New York, Aldine de Gruyter, 1967
28.
Aarons GA: Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research 6:61–74, 2004
29.
Brunette MF, Asher D, Whitley R, et al.: Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatric Services 59:989–995, 2008
30.
Glisson C, Schoenwald SK: The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research 7:243–259, 2005
31.
Glisson C, Schoenwald SK, Kelleher K, et al.: Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Administration and Policy in Mental Health and Mental Health Services Research 35:124–133, 2008
32.
Henke RM, Chou AF, Chanin JC, et al.: Physician attitude toward depression care interventions: implications for implementation of quality improvement initiatives. Implementation Science 3:40, 2008
33.
Nelson TD, Steele RG: Predictors of practitioner self-reported use of evidence-based practices: practitioner training, clinical setting, and attitudes toward research. Administration and Policy in Mental Health and Mental Health Services Research 34:319–330, 2007
34.
Palinkas LA, Aarons GA: A view from the top: executive and management challenges in a statewide implementation of an evidence-based practice to reduce child neglect. International Journal of Child Health and Human Development 2:47–55, 2009
35.
Hoagwood KE, Vogel JM, Levitt JM, et al.: Implementing an evidence-based trauma treatment in a state system after September 11: the CATS project. Journal of the American Academy of Child and Adolescent Psychiatry 46:773–779, 2007
36.
Murray MM, Southerland D, Farmer EM, et al.: Enhancing and adapting treatment foster care: lessons learning in trying to change practice. Journal of Child and Family Studies 19:393–403, 2010
37.
Kumpfer KL, Alvarado R, Smith P, et al.: Cultural sensitivity and adaptation in family-based prevention interventions. Prevention Science 3:241–246, 2002
38.
Lundgren L, Amodeo M, Cohen A, et al.: Modifications of evidence-based practices in community-based addiction treatment organizations: a qualitative research study. Addictive Behaviors 36:630–635, 2011
39.
Barreira PJ, Tepper MC, Gold PB, et al.: Adapting evidence-based interventions to fit usual practice: staff roles and consumer choice in psychiatric rehabilitation. Psychiatric Quarterly 81:139–155, 2010
40.
Amiel JM, Pincus HA: The medical home model: new opportunities for psychiatric services in the United States. Current Opinion in Psychiatry 24:562–568, 2011
41.
Pincus HA, Spaeth-Rublee B, Watkins KE: The case for measuring quality in mental health and substance abuse care. Health Affairs 30:730–736, 2011
42.
Chorpita BF, Daleiden EL: Mapping evidence-based treatments for children and adolescents: application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology 77:566–579, 2009
43.
Daleiden EL, Chorpita BF: From data to wisdom: quality improvement strategies supporting large-scale implementation of evidence-based services. Child and Adolescent Psychiatric Clinics of North America 14:329–349, 2005
44.
Daleiden EL, Chorpita BF, Donkervoet CM, et al.: Getting better at getting them better: health outcomes and evidence-based practice within a system of care. Journal of the American Academy of Child and Adolescent Psychiatry 45:749–756, 2006
45.
Denzin NK: The logic of naturalistic inquiry; in Sociological Methods. Edited by, Denzin NK. Thousand Oaks, Calif, Sage, 1978
Information & Authors
Information
Published In
Cover: Paul Cadmus, by Luigi Lucioni, 1928. Oil on canvas, 16 × 12⅛ inches. Brooklyn Museum, Dick S. Ramsay Fund, 2007.28.
History
Published online: 1 November 2013
Published in print: November 2013
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).