Implementing a Web-Based Intervention to Train Community Clinicians in an Evidence-Based Psychotherapy: A Pilot Study

During the study, conducted from September 2011 through February 2014, we randomly assigned five community mental health outpatient clinics to an Internet-supported e-learning intervention arm (three clinics assigned to e-learning) or a comparison arm (two clinics assigned to in-person training). We invited all clinicians who treated patients with bipolar disorder (N=37) at the clinics to participate in IPSRT training as part of a study examining alternative approaches to implementing IPSRT. Of those invited, 36 (97%) agreed to participate, including 19 social workers (53% of 36), 14 licensed professional counselors (39%), two clinical psychologists (6%), and one nurse (3%). The clinicians had a mean±SD of 12.9±9.9 years of experience as therapists. The clinics were not routinely implementing any evidence-based treatment for bipolar disorder; however, many clinicians had been independently trained in other evidence-based psychotherapies, such as cognitive-behavioral therapy (6).

We invited clinicians and clinical supervisors in the e-learning arm to receive IPSRT training at their convenience via an interactive 12-hour online program. We considered clinicians to have finished the training when they completed all training modules and passed a final exam. For the first three to six months after starting to use IPSRT with clients, most of the e-learning clinicians also participated monthly in a 60-minute group telephone supervision with an IPSRT expert.

Each e-learning clinic also formed an implementation team consisting of three to four clinicians and administrators. These small teams participated in a learning collaborative comprising all three e-learning clinics. The learning collaborative included the fundamental parts of collaborative quality improvement efforts (7), as conveyed through Webinars conducted by research team faculty. Implementation teams set goals for their clinics related to implementing IPSRT, made changes designed to achieve aims, and chose simple metrics to determine whether changes helped to achieve the aims. Implementation teams collected data for their chosen metrics (for example, percentage of individuals with bipolar disorder who were receiving IPSRT) and shared results with the other teams.

Clinicians and clinical supervisors from clinics in the comparison condition participated in a traditional, two-day 12-hour workshop conducted by IPSRT developers. Clinicians also received onsite supervision of IPSRT cases as part of routine weekly supervision. They were encouraged to contact IPRST experts by phone with questions. The University of Pittsburgh and RAND Corporation institutional review boards approved the study.

We subsequently obtained information about clinicians’ use of various therapeutic techniques from 136 individuals with bipolar disorder who were receiving treatment from IPSRT-trained clinicians (mean=4.1±3.8 patients per clinician). Patients completed the Psychotherapy Practice Scale adapted for IPSRT (PPS-IPSRT) on a monthly basis after their therapy sessions with IPSRT-trained clinicians. The PPS is a validated measure of clinicians’ use of core psychotherapy elements (8). The PPS-IPSRT includes subscales for interpersonal therapy (14 items) and social rhythm therapy (five items). Patients rate the frequency of therapists’ use of techniques on a Likert scale from 1, never, to 4, very often. For each IPSRT-trained clinician, we obtained a mean of 13.0±15.9 therapy session reports from patients who completed the PPS-IPSRT (mean=4.1±3.8 session reports completed per patient). Subscale internal consistency was computed by using Cronbach’s alpha (for IPSRT, 92.1; for interpersonal therapy, 90.7; and for social rhythm therapy, 82.8). Our outcome variable, use of IPSRT, was the percentage of 19 IPSRT techniques that patients reported clinicians using “often” or “very often” in a session.

To characterize clinicians’ use of IPSRT techniques after training, we compared mean scale scores of patients by arm. To investigate differences in mean scale scores over time, we estimated multivariable models of clinicians’ use of IPSRT techniques by arm with time elapsed since a clinician completed IPSRT training. Because clinicians completed IPSRT training at different times, we estimated these models as a function of IPSRT training method, time from IPSRT training completion to patient report (zero to 180 days, 181 to 365 days, and >365 days), and an interaction term of training method and time since completing training. Models included patient-level random effects via random intercepts, and we found significant random effects for all models (p<0.001).

Sixteen clinicians were trained in e-learning clinics, and 20 were trained in in-person training clinics. In the first six months after IPSRT training, patients’ PPS-IPSRT scores indicated that the trained clinicians were using 61%±28% of the IPSRT techniques often or very often. Use of IPSRT techniques increased over time. In the second six months after training (days 181–365), patients’ PPS-IPSRT scores indicated that clinicians were using 66%±22% of the techniques often or very often, and this increased to 71%±24% at 12 months after training. The 12-month scores indicated significantly greater use of IPSRT techniques than used immediately after training (p<.01). Clinicians’ use of IPSRT techniques was comparable for the in-person training and e-learning groups in all three periods (Figure 1). In the multivariate regression, we found no significant difference in use of IPSRT techniques between the e-learning and in-person training groups, overall or in any time period.

Among the techniques most commonly used by the clinicians were exploring ways to improve relationships and how moods might be related to those relationships. Among the least commonly used were techniques such as keeping a record of sleep, wake, and meal times and a schedule of activities. Compared with use of techniques related to social rhythm therapy, a greater percentage of interpersonal therapy techniques were used often or very often in sessions in the first six months after training (49% versus 64%), in the second six months after training (56% versus 69%), and more than a year after training (59% versus 75%). Compared with the six months immediately after training, both interpersonal therapy techniques (p<.01) and social rhythm therapy techniques (p<.01) were used significantly more often a year after training. Across the entire posttraining period, patients reported that clinicians were more likely to use interpersonal therapy techniques (71% of the techniques used often or very often), compared with social rhythm therapy techniques (56% used often or very often)

In this pilot study, patients reported increasing use of IPSRT techniques over time among trained clinicians, with no statistically significant differences between clinicians who participated in Internet-supported e-learning and those who received in-person training and supervision. This finding suggests that relatively efficient training and implementation support efforts can significantly change the behavior of mental health clinicians.

Efficiently disseminating evidence-based psychotherapies to the frontline workforce is a key challenge facing the mental health field. Few clinical training program graduates are trained in evidence-based psychotherapies (4). Training barriers include high rates of workforce turnover (9), clinical demands that prevent staff from attending in-person trainings, and poor access to intervention experts and other implementation supports (10,11). Our finding that clinicians’ increased use of IPSRT techniques was comparable between the Internet-supported e-learning and traditional training arms suggests ways to reduce some of these barriers. E-learning can provide a more scalable and efficient approach to training large numbers of clinicians. It can also give busy community mental health clinicians greater flexibility in terms of when they participate in training and can ensure that newly hired clinicians receive training of the same quality as the originally trained clinicians.

We also found that clinicians made less use of patient homework, consistent with what has been documented with cognitive-behavioral therapy (12). Clinicians may find assigning patients homework challenging, because it deviates from commonly used in-session therapy techniques. Nevertheless, we observed a tendency for homework use to increase (albeit from a relatively low baseline) as clinicians became more comfortable with IPSRT. Further research is needed to understand how to enhance clinicians’ and patients’ use of homework in evidence-based psychotherapies.

Learning collaboratives can support the effective implementation of health care interventions to improve quality of care (13), and we found that the increase in use of IPSRT techniques by clinicians in clinics that participated in the e-learning collaborative was comparable to the increase among clinicians who received in-person training. However, our design did not allow us to assess to what extent e-learning clinicians would have increased their skills in the absence of a learning collaborative or to what extent various components of the learning collaborative contributed to enhanced use of IPSRT techniques.

This study had several limitations. The clinics are in a state with a relatively robust public mental health system. In addition, by agreeing to participate in the study, the organizations, clinics, and clinicians showed a heightened level of interest in using IPSRT. We believe IPSRT use would be lower in settings where interest was lower. We do not know to what extent our findings would generalize to clinicians in different treatment settings or regions or to other psychotherapies. However, the study’s IPSRT Web site (www.ipsrt.org) and associated e-learning collaborative activities were modeled closely on other commonly used psychotherapy e-learning Web sites (14) and learning collaborative approaches (15). We also do not know how variations in the amount or quality of supervision were associated with use of IPSRT techniques.

This study was not designed to test whether the two training approaches were equivalent. However, the relatively tight confidence intervals around the differences at all three time points provide some reassurance that our negative finding did not stem primarily from a lack of statistical power. Further research is needed to test whether the two approaches lead to comparable use by clinicians of specific therapeutic techniques, as well as whether the two approaches produce comparable patient outcomes.

We did not have information to assess whether clinicians’ use of the techniques improved patient outcomes. In the absence of videotaped recordings of sessions, we cannot be sure that patients’ reports of session content accurately reflected the interventions delivered. Finally, the absence of an inactive comparison condition raises the possibility that increases in clinicians’ use of techniques were unrelated to the training received.

Despite the limitations, the results of our pilot study could have compelling implications for the mental health field, which has long struggled to efficiently and effectively increase clinicians’ use of evidence-based practices. Our finding of no significant difference between Internet-supported e-learning and in-person training as measured by patients’ reports of IPSRT techniques by clinicians is an important step toward this goal. Future studies must examine whether Internet-supported e-learning by clinicians can achieve equivalent patient outcomes and explore how emerging interactive technologies, such as virtual simulations and game-based role plays, might also be used to enhance clinician skills and patient outcomes.

Dr. Stein has served on an advisory board for Otsuka Pharmaceuticals. Dr. Frank owns stock in Psychiatric Assessments, Inc., and Health Rhythms; has served on an advisory board for Servier International; and has received an honorarium from Lundbeck. The other authors report no financial relationships with commercial interests.