Maintenance of Training Effects of Two Models for Implementing Evidence-Based Psychological Treatment

Complete details of the study have been previously reported in an article describing the primary outcome (6), but key aspects are outlined here. The sample included 24 colleges with a counseling center. Centers were randomly assigned to either the expert or train-the-trainer approach. Each center had at least three therapists interested in the study and a staff member interested in serving as study director. Therapists who treated students at least 25% of the time provided informed consent and were enrolled in the study. Participating therapists obtained informed consent from up to two student patients of their choosing, who were eligible if they were age ≥18 years with symptoms of depression or eating disorders (excluding anorexia nervosa, for which IPT is not an EBT). Therapists decided whether the patient met these criteria by using their usual assessment procedures during each study phase: baseline (i.e., before IPT training), posttraining (but during the expert consultation period), and maintenance (after the consultation in the expert condition had ceased). The therapists were asked to submit audio recordings for all sessions (up to eight) with each student patient. Two audio recordings were rated for each patient during each study phase: one from session 1 and one from a randomly chosen later session. On average, maintenance data were collected 12.8 months after the last submitted posttraining session (range 5.7–29.9 months), which was 19.1 months after training (range 13.5–33.3 months); that is, maintenance data were collected about 7 months after the end of the expert consultation period. The timing of collection of maintenance data did not statistically significantly differ across conditions (the mean±SD was 15.6±8.5 months for the expert training group and 10.7±3.6 months for the train-the-trainer group). At each site in the train-the-trainer condition, one trainer was selected by the study’s director at that center. The study received institutional review board approval at the coordinating sites and participating colleges.

Regarding the training conditions, sites in the expert condition were provided with a 2-day workshop in IPT, conducted by a study team member with IPT expertise. Participating therapists were provided a manual, and the workshop included a review of key IPT principles and procedures, role-playing, and case examples to demonstrate IPT treatment phases. Therapists at each site in the expert condition had the opportunity to engage in a 1-hour consultation call with the study team member who conducted the workshop every month for up to 12 months after the workshop.

In the train-the-trainer condition, trainers attended two workshops; the first 2-day workshop was identical in content to the workshop provided in the expert condition and was designed to teach participants to conduct IPT. The second workshop provided training in teaching IPT to others. After the first workshop, each trainer was encouraged to use IPT to treat up to two patients. The study team member who conducted the IPT training reviewed a selection of recorded sessions and provided feedback to the trainers. Once the trainers had trained their colleagues in IPT, they were encouraged to meet with them weekly for 1-hour group consultation sessions. Trainers were also encouraged to join monthly group review calls with the study team member who had conducted their training and with other trainers.

Three measures were used to assess outcomes during each study phase: therapist adherence to, competence in, and knowledge of IPT. Adherence and competence were assessed through listening to audio recordings of therapy sessions, by raters blinded to condition, using the IPT Fidelity Rating Scale (adapted from the measure developed for the Veterans Health Administration IPT Training Program) (12). Details are available in the primary outcomes article (6). Knowledge of IPT was assessed from therapist responses to 20 multiple-choice questions.

Therapist characteristics assessed at baseline included age, gender, race-ethnicity, professional degree, years in present position, attendance at a previous IPT workshop or class (yes or no), experience using IPT with patients in past year (yes or no), and job satisfaction. The Evidence-Based Practice Attitude Scale (EBPAS) (13) was used to ascertain degree of acceptance of EBTs. Finally, job satisfaction was assessed with the 14-item job satisfaction scale (14).

We used standard linear mixed-effects modeling to estimate changes in outcomes from baseline to maintenance. In line with the intention-to-treat principle, we included all participating therapists in the analyses as long as data were available from at least one repeated assessment. For all model estimations, we used maximum likelihood implemented in R, version 3.5 (lme4 package) (15). For model specification, we used a random intercept model assuming linear change over time. Effect sizes (ESs) are reported as Cohen’s d.

From a total of 184 enrolled therapists in the study, we received audio-recorded sessions from 105 therapists at baseline (48 in the expert condition and 57 in the train-the-trainer condition), from 87 therapists at posttraining (41 in the expert condition and 46 in the train-the-trainer condition), and from 23 therapists during the maintenance period (10 in the expert condition [21% of those providing baseline recordings] and 13 in the trainer condition [23% of those providing baseline recordings]; there was no significant difference between the two groups in the percentage of therapists who provided recordings during the maintenance period. However, one therapist in the train-the-trainer condition provided a tape at both posttraining and maintenance, but not at baseline). Therapists providing data during the maintenance period did not significantly differ from therapists not providing data in terms of age, gender, race-ethnicity, degree, and years in present position and in whether they had taken an IPT class or used IPT during the past year; EBPAS or job satisfaction score; or adherence, competence, or IPT knowledge at baseline or posttraining. There was no significant difference between the two groups in self-reported consultation hours used by therapists during the posttraining phase (9.0±8.8 hours for train-the-trainer group and 12.4±8.1 hours for the expert group).

Both training groups showed significant within-group improvement over the three phases (from baseline to maintenance period) for adherence (train the trainer: slope=0.22, ES=0.92, 95% confidence interval [CI]=0.80–1.04, p<0.001; expert: slope=0.16, ES=0.66, 95% CI=0.53–0.80, p<0.001), competence (train the trainer: slope=0.19, ES=0.78, 95% CI=0.65–0.91, p<0.001; expert: slope=0.11, ES=0.47, 95% CI=0.33–0.61, p<0.001), and IPT knowledge (train the trainer: slope=0.90, ES=0.41, 95% CI=0.19–0.63, p<0.001; expert: slope=1.03, ES=0.47, 95% CI=0.23–0.71, p<0.001). Importantly, there were statistically significant between-group effects, indicating greater improvement over time from baseline to maintenance, in the train-the-trainer group compared with the expert group for adherence (slope difference=0.06, ES=0.26, 95% CI=0.10–0.42, p=0.002) and competence (slope difference=0.08, ES=0.32, 95% CI=0.15–0.48, p<0.001). No significant between-group differences were detected in the change in IPT knowledge. Figure 1 depicts the longitudinal models of change for adherence, competence, and IPT knowledge for the two groups.

The train-the-trainer and expert groups showed significant within-group improvement in IPT adherence and competence from baseline through maintenance, with large ESs, but these improvements were significantly greater in the train-the-trainer condition. Both groups also displayed significant within-group improvement in IPT knowledge over time, with no difference between the two conditions. Overall, the train-the-trainer condition resulted in greater maintenance of training effects in terms of adherence and competence, relative to the expert condition.

These findings are important, because they provide empirical support for the theoretical notion that the train-the-trainer approach may be a better long-term training strategy. It is possible that trainers could foster a culture supportive of IPT implementation and that they were able to continue to provide consultation to their colleagues well into the maintenance period; however, future research will be needed to explicitly explore mechanisms that may underlie greater long-term effects. Overall, these results expand previous findings supporting the effectiveness of the train-the-trainer model (3, 6) and suggest that not only may this approach result in comparable or even superior outcomes in the immediate posttraining period, including for the key implementation outcome of fidelity, but that these effects may extend over time—over 1.5 years, on average, from initial training.

Strengths of this study included the large initial sample of therapists from a wide range of colleges, increasing generalizability, as well as the rigorous fidelity assessment. Our findings also should be interpreted in the context of the study’s limitations, which included the relatively small number of therapists who provided maintenance data (possibly because of high rates of staff turnover). Notably, we observed no significant differences between therapists who did and did not provide maintenance data, and our analytic approach could capitalize on the data available. Future research should assess additional indicators of sustainability (i.e., whether the treatment is formally adopted by the institution) as well as cost-effectiveness.

Our results indicate that the train-the-trainer approach produced improved implementation outcomes (i.e., in adherence and competence) from baseline through the maintenance period that were superior to those of the expert training approach. Given not only its more robust long-term effects but also its potential to train more therapists over time relative to traditional approaches, the train-the-trainer model may be a viable option for increasing dissemination of EBT.