Training and Consultation to Promote Implementation of an Empirically Supported Treatment: A Randomized Trial
Abstract
Objective:
The study evaluated the efficacy of three training modalities and the impact of ongoing consultation after training. Cognitive-behavioral therapy (CBT) for anxiety among youths, an empirically supported treatment, was used as the exemplar. Participants were randomly assigned to one of three one-day workshops to examine the efficacy of training modality: routine training (training as usual), computer training (computerized version of training as usual), and augmented training (training that emphasized active learning). After training, all participants received three months of ongoing consultation that included case consultation, didactics, and problem solving.
Methods:
Participants were 115 community therapists (mean age of 35.9 years; 90% were women). Outcome measures included the Adherence and Skill Checklist, used to rate a performance-based role-play; a knowledge test; and the Training Satisfaction Rating Scale.
Results:
All three training modalities resulted in limited gains in therapist adherence, skill, and knowledge. There was no significant effect of modality on adherence, skill, or knowledge from pretraining to posttraining. Participants were more satisfied with augmented and routine training than with computer training. Most important, number of consultation hours after training significantly predicted higher therapist adherence and skill at the three-month follow-up.
Conclusions:
The findings suggest that training alone did not result in therapist behavior change. The inclusion of ongoing consultation was critical to influencing therapist adherence and skill. Implications for implementation science and mental health services research are discussed. (Psychiatric Services 63:660–665, 2012; doi: 10.1176/appi.ps.201100401)
Implementation science is an emerging field and is the “scientific study of methods to promote the systematic uptake of clinical research findings and other evidence-based practices into routine practice” (1). Implementation science addresses challenging questions concerning how to most effectively transport empirically supported treatments into community settings. Implementation of empirically supported treatments in the children's mental health system has only recently gained prominence and holds promise (2). Barriers to implementation of clinical innovation in the children's mental health system include poor attitudes toward innovation, difficulty embedding innovation into existing systems, and untrained providers (3). Perhaps the most daunting challenge to implementation of empirically supported treatments for youths in community settings is training clinicians to administer these complex treatments with fidelity (3). Thus one frequently used implementation strategy is training providers (2).
The most commonly used training strategy is a brief workshop (4). Workshops improve therapist knowledge and attitudes but are not effective in changing therapist behavior in providing patient care (5,6). This outcome is due to the one-time nature of training, or “training and hoping” (7), which does not allow for integration into existing practices. Two methods hypothesized to improve brief workshops are to construct workshops that are informed by the adult learning literature (8) and to follow workshops with ongoing consultation that allows providers to integrate clinical innovation into existing practices (6).
The creation of brief workshops informed by adult learning principles may result in improved therapist behavior outcomes (8). Most workshops are passively delivered despite research demonstrating that active learning, a process in which an individual acquires knowledge and skills through interaction and reflection, is preferable (9). Furthermore, it has been recommended that training focus on principles of a treatment rather than on one protocol because principles provide a foundational understanding and reduce the burden of learning multiple protocols (10). Finally, computer-assisted training holds promise (11,12) because it is well suited for limited-resource settings, such as community mental health clinics.
Improving the efficacy of brief workshops is desirable but may not be sufficient to change therapist behavior. Ongoing consultation after training may be the critical element in increasing effectiveness of training (6). For example, one study demonstrated that coaching and performance feedback on cases posttraining changed therapist behavior (13). The mechanism by which consultation works is unknown, but consultation likely provides therapists with a venue for clarification, and practice of concepts, learning concepts and practicing over time, case consultation, and using problem solving to overcome implementation barriers. The impact of adding consultation to brief training has not been examined as an implementation strategy in child mental health services and holds promise for effective provider behavior change.
This study evaluated the efficacy of three modalities of brief training and the impact of including ongoing consultation after this training. Cognitive-behavioral therapy (CBT) for treating anxiety among youths, an empirically supported treatment, was used as the exemplar. Participants were randomly assigned to a one-day workshop to examine routine training (training as usual), computer training (computerized version of training as usual), or augmented training (training that emphasized active learning and principles of CBT). After training, all participants received three months of ongoing consultation that included case consultation, didactics, and problem solving of implementation barriers specific to the local context. Outcomes included therapist adherence to CBT for child anxiety, skill in CBT for child anxiety, knowledge about CBT for child anxiety, and satisfaction with training. The study design was prospective and experimental.
Methods
Recruitment and screening
All procedures were approved by the Temple University Institutional Review Board. Participants were recruited from the community via professional Listservs, training programs, and word of mouth. After complete description of the study to participants, written informed consent was obtained. Recruitment began in May 2009 and was completed by September 2009. Assessments, training, and consultation began in June 2009 and were completed by January 2010.
Inclusion criteria.
Criteria for inclusion in the study were working in the community with children ages eight to 17 with anxiety disorders, having training in mental health, reading and speaking English, having access to a computer or telephone for consultation, and participating voluntarily. Exclusion criteria were having eight or more hours of prior training in CBT for child anxiety (13).
Power analysis.
Before recruitment, we conducted a power analysis, which indicated that a sample size of 38 participants in each group (N=114) would result in 80% power to detect a moderate effect (Cohen's d=.6) at an alpha level of .05.
Randomization.
We recruited 115 participants, who were randomly assigned to one of three training modalities. Using equal allocation concealment, random assignment occurred at the level of training date. An unpredictable randomization sequence was generated that determined random assignment of training dates to condition. Participants were not informed of which training condition they received.
Training methods
Study design.
Participants were randomly assigned to one of the three training conditions, all of which were provided in person. After training, all participants received weekly consultation by telephone or over the Internet for three months.
Routine training.
This condition, involving 41 participants, included a six-hour workshop that focused on session-by-session administration of the Coping Cat program (14). The workshop was designed to follow the format of continuing-education workshops and did not emphasize active learning. Didactic instruction included a PowerPoint presentation and viewing of videotapes of representative youths receiving treatment.
Computer training.
This condition included a six-hour computer-based, self-guided training for 34 participants (15). It closely resembled routine training, with the exception that it was delivered via computer and was therefore self-paced. The training provided step-by-step instructions for each session of the Coping Cat program and examples and materials for the therapist to peruse (14). Active learning in this condition included videos of treatment sessions, therapist tips, and links to research articles. Although training was via computer, participants received training in a computer lab to ensure that they completed an amount of training equivalent to that provided in the other two modalities.
Augmented training.
Forty participants attended a six-hour experiential workshop focusing on core CBT principles: coping modeling, identifying somatic cues and relaxation, identifying and challenging anxious self-talk, problem solving, and behavioral exposure (16). This training emphasized active learning through the use of behavioral role-play (17) and interteaching (18). Behavioral role-play is a technique where therapists role-play implementing a treatment and receive feedback on their performance (11), whereas interteaching is a behaviorally based instructional technique in which individuals engage in small-group discussion with guided questions (18). Core principles were introduced didactically, modeled by the instructor, and then followed by small-group activities.
Consultation
Participants from all three conditions were provided weekly consultation via the WebEx virtual conferencing platform for three months after training. Participants could call in via telephone or computer to attend the one-hour weekly virtual group meeting. Those who opted to consult via computer were able to view a whiteboard and the consultant via Web camera. The consultation curriculum was designed with participant input and included case consultation, didactic topics (such as treating a client with comorbid depression), practice with concepts (such as relaxation), and assistance in implementation of the treatment within context (in a psychiatric clinic or school, for example).
Measures
Clinician Demographics and Attitudes Questionnaire.
This 15-item questionnaire assesses demographic characteristics and prior experience with CBT for youth anxiety (19).
Training Satisfaction Rating Scale.
This 12-item questionnaire assesses training satisfaction on a Likert scale from 1 to 5 (20). The score was the total of the ratings on the 12 items. Possible scores range from 12 to 60, with higher scores indicating higher satisfaction.
Knowledge test.
Adherence and Skill Checklist.
This checklist was developed to measure therapist fidelity to CBT for child anxiety (19). Adherence, defined as the utilization of the procedures of a protocol in the treatment of a client (22), was assessed by coding the presence of six CBT competencies: identification of somatic symptoms, identification of anxious cognitions, relaxation, coping thoughts, problem solving, and positive reinforcement. The score could range from 0 to 6, with higher scores indicating better adherence. Skill, the level of competence shown by the therapist in the delivery of treatment (22), was evaluated on a Likert scale that ranged from 1 to 7. Coders were blind to hypotheses, training condition, and time point of the assessment. Rated adherence (intraclass correlation=.98) and skill (intraclass correlation=.92) demonstrated outstanding interrater reliability.
Performance-based role-play
A structured role-play was used to assess participants' adherence and skill in a simulated clinical setting. Research assistants blind to condition were trained to portray anxious youths. Therapists were asked to prepare the client for a treatment session that included an exposure task, a competency central to CBT. The role-plays were digitally recorded and later independently coded with the Adherence and Skill Checklist. Role-plays were conducted pre- and posttraining and at three-month follow-up.
Data analysis
All analyses were conducted with SPSS, version 17.0. Four outcomes were examined: adherence, skill, knowledge, and satisfaction. Analyses were conducted to isolate the impact of training modality from pre- to posttraining on the outcome variables and the impact of consultation at the three-month follow-up on the outcome variables. To test the main effects (training and time) and the interaction (training × time) on the outcome variables (adherence, skill, and knowledge), repeated-measures analysis of variance was used. Satisfaction was analyzed with one-way analysis of variance. To test the impact of consultation on outcome variables, linear regression was used to model the effect of number of consultation hours on adherence, skill, and knowledge.
Adherence, skill, and knowledge were analyzed with reference to a clinically significant criterion (12). A 70% cutoff score was used to indicate being trained to criterion for adherence and knowledge. This value was selected to be consistent with the 80% training criterion used in other evaluations of child anxiety CBT (21) but is less conservative because the participants were community clinicians naïve to CBT as a treatment for child anxiety. A skill score of 3.5 was used as the cutoff for being acceptably trained to criterion and was consistent with past trials evaluating the efficacy of CBT (5,12,23). Chi square analyses were used to investigate differences in training to criterion by condition.
Results
Demographic characteristics and retention of participants
Participants were primarily female (90% of sample), and the mean±SD age was 35.9±11.4 years. Table 1 provides additional information about demographic characteristics and professional background.
Individuals who completed the pretraining assessment composed the intent-to-train sample. Of the 115 intent-to-train participants, 113 (98%) completed the posttraining assessment and 100 (87%) completed the follow-up assessment.
Adherence
Mean adherence scores significantly improved over time from pretraining (1.6±1.2) to posttraining (3.0±1.4) (F=83.21, df=1 and 111, p<.001), but there was no significant effect of training or interaction of time and training (Table 2).
Skill
Participant skill significantly improved over time from pretraining (2.9±1.2) to posttraining (3.8±1.5) (F=69.06, df=1 and 111, p<.001), but there was no significant effect of training. A significant interaction of time and training was identified (F=4.55, df=2 and 111, p<.01) such that participants in the augmented training (pretraining, 2.7±1.2; posttraining, 3.7±1.5) improved their skill more from pre- to posttraining compared with participants in the computer training (pretraining, 2.9±1.3; posttraining, 3.7±1.5) (Table 2).
Knowledge
Participant knowledge scores significantly improved over time from pretraining (15.1±2.2) to posttraining (17.2±1.9) (F=96.74, df=1 and 112,p<.001), but there was no significant effect of training or interaction of time and training (Table 2).
Training to criterion
There were no significant differences across training modalities in percentage of participants trained to criterion (Table 3). Note that these analyses were based on data from only the participants who completed the study, not the intent-to-train sample. Overall, from pretraining to posttraining, the number of participants trained to criterion increased on adherence (32%), skill (37%), and knowledge (15%). From posttraining to follow-up, the number of participants trained to criterion increased on adherence (23%), skill (20%), and knowledge (1%). On average, participants received 13 hours of training (six hours of training and seven hours of consultation). After receiving that dosage, approximately 96% of participants were trained to criterion in knowledge, 61% were trained to criterion in adherence to CBT for child anxiety, and 85% of participants were trained to criterion in skill.
Satisfaction
Differences in satisfaction across training modalities were observed (F=7.22, df=2 and 112, p<.001). Planned comparisons identified that participants in routine training (53.7±5.4, t=−3.78, df=112, p<.001) and augmented training (55.5±4.7, t=2.40, df=112, p<.05) were more satisfied than participants in computer training (50.8±5.9).
Consultation
On average, the 108 consultation sessions lasted 52.6±10.8 minutes and had 7.8±4.5 participants per session. The mean number of cases discussed per call was 2.7±1.9. Each participant used a mean of 7.2±3.2 consultation hours. Number of consultation hours significantly predicted participant adherence (F=20.67, df=1 and 113, p<.001, β=.39, R2=.16) and skill (F=11.46, df=1 and 113, p<.001, β=.30, R2=.09) at the three-month follow-up but did not significantly predict participant knowledge.
Discussion
Two main findings relevant to the implementation of empirically supported treatments in child mental health services emerged from this study. First, a brief workshop was not effective in changing therapist behavior to implement CBT into practice. All three training modalities (routine, computer, and augmented training) resulted in limited gains in therapist adherence and skill—even brief training emphasizing active learning. Second, ongoing consultation after a brief workshop was effective; it significantly improved therapist adherence and skill and allowed therapists to solidify concepts and skills learned in the brief workshop for embedding CBT within their clinical practices.
There was no effect of training modality on therapist adherence, skill, or knowledge, although satisfaction varied across modalities. The null finding with regard to therapist adherence and skill is consistent with previous research showing that, even at its best, training by itself is not a sufficient implementation strategy (12,13). Consistent with previous findings, knowledge of CBT for youth anxiety increased for all participants from pretraining to posttraining (11,24). Inconsistent with previous studies (11,12), participants were more satisfied with the face-to-face training conditions than with computer training. However, satisfaction was high across training modalities. Given limited resources in community mental health settings and high rates of therapist turnover, computer training may be the most viable training strategy given that it is as effective as instructor-led training.
Ongoing consultation after a brief workshop significantly improved therapist fidelity to an empirically supported treatment. This important finding corroborates theory (6) and suggests that consultation should be included as a core element in implementation. Each hour of consultation after training improved therapist adherence by .4 point and skill by .3 point. These increases suggest a sizable return on investment. The virtual delivery of consultation was especially innovative because a diverse set of clinicians could interact at times that suited their schedules. Virtual consultation seems ideal for limited-resource settings, such as community mental health clinics (25). Furthermore, consultation provided an experiential laboratory where therapists could learn from peers, receive feedback from an expert consultant on actual child cases, practice skills, tailor the treatment for context, and resolve implementation difficulties. Ongoing support made it possible for therapists to embed the new clinical practice into their existing repertoire of skills.
With regard to clinical significance, after a six-hour workshop, approximately 38% of participants were trained to criterion in adherence and 65% were trained to criterion in skill. Ongoing consultation of approximately seven hours over three months brought those numbers up further, to 61% and 85%, respectively. This finding is particularly noteworthy given that other studies have demonstrated only minimal to moderate gains in adherence and skill after training and often a decline at follow-up when training was not followed by consultation (11–13,17). These findings suggest that a relatively limited dosage of training, approximately 13 hours, can result in meaningful behavior change for therapists when consultation is included.
Despite several strengths of this study, there were limitations. The focus on an empirically supported treatment for child anxiety may not be generalizable to training in other treatments. However, a number of child mental health service settings were represented (including schools, outpatient clinics, hospitals, and community mental health clinics) and resulted in high generalizability across settings. The sample was self-referred, which may not be generalizable to therapists mandated to receive training. In addition, the therapists identified highly with CBT, had moderate knowledge of CBT, were primarily female and Caucasian, and had a master's-level education, which may not be generalizable to other samples. Finally, because our study endeavored to control the number of training hours in each workshop, it was necessary for participants to receive the computer training in a computer lab. This was a limitation because in a real-world setting, computer training would occur in the home or office. Further, the computer training was not highly interactive; thus these findings may not generalize to computer training that includes more interactive design.
Conclusions
Although a one-day workshop can be effective in changing provider knowledge, it is not effective in changing provider behavior. We recommend that training no longer be used as a stand-alone implementation strategy. However, with the combined implementation strategies of brief training and ongoing consultation, community clinicians can be effectively trained in clinical innovations that are complex and multistep (26). Brief training may be best provided through computer delivery given that it is a permanent product that an agency can purchase that is effective, accessible, and cost-efficient and can be used in situations where there is high therapist turnover (8). Most important, consultation merits as much emphasis as brief training, and we suggest that whatever limited resources are available be used to provide consultation. Unfortunately, little empirical work has been completed on consultation as an implementation strategy. Research on supervision (27) and coaching teachers (28) can inform this emerging and exciting area. Investigation of how to train consultants, what type of consultation is most effective (individual versus group), and the merit of internal versus external consultation is needed.
Acknowledgments and disclosures
This study was funded by grant F31-MH083333 from the National Institute of Mental Health and by the Shipley Research Prize awarded to Dr. Beidas. The authors thank Robert Fauber, Ph.D., Philip Hineline, Ph.D., and David Mandell, Sc.D., for comments on drafts of this article. The authors thank the following individuals for their help as consultants: Kathleen Carroll, Ph.D., Wendi Cross, Ph.D., and Kimberly Hoagwood, Ph.D. The authors thank the following research assistants: Stacey Austin, Kayla Brehm, Mamatha Chary, Margaret Downey, Philip Fizur, Kyle Peer, and Anna Slavachevskaya.
Dr. Marcus has received grant support from Ortho-McNeil Janssen and Forest Research Institute and has served as a consultant to AstraZeneca. Dr. Kendall reports receiving royalties from sales of materials related to the treatment of anxiety among youths. The other authors report no competing interests.
References
1.
Graham ID, Logan J, Harrison MB, et al.: Lost in knowledge translation: time for a map? Journal of Continuing Education in the Health Professions 26:13–24, 2006
2.
Powell BJ, McMillen JC, Proctor EK, et al.: A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, ePub Dec 26, 2011
3.
McHugh RK, Barlow DH: The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. American Psychologist 65:73–84, 2010
4.
Herschell AD, McNeil CB, McNeil D: Clinical child psychology's progress in disseminating empirically supported treatments. Clinical Psychology: Science and Practice 11:267–288, 2004
5.
Beidas RS, Kendall PC: Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice 17:1–30, 2010
6.
Herschell AD, Kolko DJ, Baumann BL, et al.: The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clinical Psychology Review 30:448–466, 2010
7.
Fixsen D, Naoom S, Blase K, et al.: Implementation Research: A Synthesis of the Literature. Tampa, University of South Florida, Louis de la Parte Florida Mental Health Institute, 2005
8.
Beidas RS, Koerner K, Weingardt KR, et al.: Training research: practical recommendations for maximum impact. Administration and Policy in Mental Health 38:223–237, 2011
9.
El-Tannir AD: The corporate university model for continuous learning, training and development. Education and Training 44:76–81, 2002
10.
Chorpita BF, Daleiden EL: Mapping evidence-based treatments for children and adolescents: application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology 77:566–579, 2009
11.
Dimeff LA, Koerner K, Woodcock EA, et al.: Which training method works best? A randomized controlled trial comparing three methods of training clinicians in dialectical behavior therapy skills. Behaviour Research and Therapy 47:921–930, 2009
12.
Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, et al.: We don't train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology 73:106–115, 2005
13.
Miller WR, Yahne CE, Moyers TB, et al.: A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology 72:1050–1062, 2004
14.
Kendall PC, Hedtke K: Cognitive-Behavioral Therapy for Anxious Children: Therapist Manual. Ardmore, Penn, Workbook Publishing, 2006
15.
Kendall PC, Khanna MS: CBT4CBT: Computer-Based Training to Become a Cognitive-Behavioral Therapist for Anxious Youth. Ardmore, Penn, Workbook Publishing, 2008
16.
Sburlati ES, Schniering CA, Lyneham HJ, et al.: A model of therapist competencies for the empirically supported cognitive behavioral treatment of child and adolescent anxiety and depressive disorders. Clinical Child and Family Psychology Review 14:89–109, 2011
17.
Cross WF, Seaburn D, Gibbs D, et al.: Does practice make perfect? A randomized control trial of behavioral rehearsal on suicide prevention gatekeeper skills. Journal of Primary Prevention 32:195–211, 2011
18.
Boyce T, Hineline P: A strategy for enhancing the user-friendliness of behavioral arrangements in the college classroom. Behavior Analyst 215–226, 2002
19.
Beidas RS, Barmish AJ, Kendall PC: Training as usual: can therapist behavior change after reading a manual and attending a brief workshop on cognitive behavioral therapy for youth anxiety? Behavior Therapist 32:97–101, 2009
20.
Tello FP, Moscoso S, Garcia I, et al.: Training Satisfaction Rating Scale. European Journal of Psychological Assessment 22:268–279, 2006
21.
Walkup JT, Albano AM, Piacentini J, et al.: Cognitive behavioral therapy, sertraline, or a combination in childhood anxiety. New England Journal of Medicine 359:2753–2766, 2008
22.
Perepletchikova F, Treat TA, Kazdin AE: Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology 75:829–841, 2007
23.
Carroll KM, Nich C, Sifry R, et al.: A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence 57:225–238, 2000
24.
Rubel EC, Sobell LC, Miller WR: Do continuing education workshops improve participants' skills? Effects of a motivational interviewing workshop on substance-abuse counselors' skills and knowledge. Behavior Therapist 23:73–77, 2000
25.
Weingardt KR, Cucciare MA, Bellotti C, et al.: A randomized trial comparing two models of web-based training in cognitive-behavioral therapy for substance abuse counselors. Journal of Substance Abuse Treatment 37:219–227, 2009
26.
Hawe P, Shiell A, Riley T: Complex interventions: how “out of control” can a randomised controlled trial be? BMJ 328:1561–1563, 2004
27.
Milne D: Can we enhance the training of clinical supervisors? A national pilot study of an evidence-based approach. Clinical Psychology and Psychotherapy 17:321–328, 2010
28.
Noell G, Witt J, Slider N, et al.: Increased intervention implementation in general education following consultation: a comparison of two follow-up strategies. Journal of Applied Behavior Analysis 33:271–284, 2005
Figures and Tables
Information & Authors
Information
Published In
Cover: Flag on Orange Field, by Jasper Johns, 1967. Museum Ludwig, Cologne, Germany. © VAGA, New York. Photo © Erich Lessing/Art Resource, New York.
History
Published online: 1 July 2012
Published in print: July 2012
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).