Scaling Up the Dissemination of Evidence-Based Mental Health Practice to Large Systems and Long-Term Time Frames

In the first phase of the implementation, the sole national supported employment educator (supported by a national team of administrators and evaluators) focused on 21 "mentor-trainer" sites ( 6 ); in the second phase the focus shifted to the process of supporting the 21 mentor-trainer sites as they worked to disseminate the program to the remaining 145 sites throughout the country. The unique challenge of this project was to institutionalize a sustained process of training trainers to maintain program fidelity for the foreseeable future. Time-limited external demonstrations conducted thus far have not been designed to meet this challenge.

The supported employment implementation and dissemination initiative began in late 2004, after the allocation of $16 million in recurring funds through the VA Strategic Mental Health Plan for the implementation of the individual placement and support model of supported employment ( 7 ) within VA's existing Compensated Work Therapy program ( 6, 8 ). The training program began with a series of national conferences and conference calls in which the principles of supported employment were reviewed and the rationale and structure of the dissemination plan were outlined. Written materials on supported employment were also distributed to each site.

At the heart of the dissemination effort was the assumption that periodic, on-site, face-to-face performance review with frank and specific feedback at multiple organizational levels is essential to effective dissemination. As mentioned above, the plan relied upon a two-stage cascading training strategy, in which 21 sites, one in each VA geographic region (the Veterans Integrated Service Networks, or VISNs), were selected as the mentor-trainer sites that would be the focus of the initial training effort. Once trained, mentors would themselves be responsible, in the second phase, for training personnel in other programs within their VISN. Conference calls, e-mail groups, and performance data were used to foster continuous communication among all involved. The mentor-trainer programs typically have three or four supported employment specialists and a supervisor, with one of these employment specialists designated as the mentor trainer. The 145 other programs are divided among 21 VISNs with mentor-trainer sites responsible for between four and ten programs in their region ( 6 ).

A central approach to fostering successful dissemination has been the use of fidelity measures that provide a methodical accounting of adherence to the principles and specific activities that define each practice ( 9 ). Fidelity scales were originally developed for use in single-site or small multiple-site clinical trials to demonstrate that the intervention being tested was being implemented as intended by its originators. More recently they have also been adapted for use as a tool to provide systematic, concrete feedback to practitioners on program areas that need adjustment in order to reach a high level of adherence to the practice. Documentation of the process and outcome of treatment at the level of each client, collected through an electronic data collection system designed specifically for this purpose, has been a central informational foundation for the national dissemination of supported employment in the VA ( 10 ). Regularly distributed reports summarizing outcome data allow each site team to compare its data with VISN and national averages ( 11 ).

For the first two to three years of the dissemination, each of the 21 sites received a series of visits every six months from an expert trainer in supported employment who communicated the goals of the program to facility leadership, provided didactic training on supported employment to the entire mental health service, and provided ongoing technical assistance for the supported employment team. At each visit, the trainer conducted a fidelity assessment and used the fidelity scale results as the foundation for providing feedback to the supported employment team as well as to the higher administrative levels of each facility. In addition, twice-monthly national conference calls and e-mail groups provided a forum for ongoing dialogue, case reviews, and exchanges of experiences. During this phase, the primary emphasis was on fostering support from facility leaders and guiding the mentor-trainer sites to reach full implementation of supported employment, although all 166 sites participated in most activities.

Once this goal was largely achieved, the energies of the expert trainer and national program staff shifted to the second, more challenging phase: the dissemination of supported employment to the 145 nonmentor sites and development of self-sustaining institutional structures. In this phase, guidance, instruction, and support were provided to the 21 mentor trainers in their efforts to offer training and technical assistance to the programs under their purview, with the eventual goal of developing the skills to provide fidelity assessments, with constructive feedback, to all sites. A more elaborate dissemination framework was developed to provide structure for this effort and to articulate the principles and values that shape supported employment mentorship.

The first goal in phase 2 was simply to provide a structural framework for the interaction of mentor trainers with supported employment participants at the 145 other VA sites. There were five structural elements in this phase.

First, a national conference was held on phase 2 of the implementation. Participants from the mentor-trainer sites were introduced to the goals and the proposed structures of this phase. The conference also established a sentinel group that was charged with undertaking this effort.

Second, mentor trainers were advised to establish a monthly conference call involving all sites in their VISN and to make two to four face-to-face site visits per year.

Third, every six months, corresponding to the mentor-trainer site's fidelity assessment, the national trainer accompanied the mentor trainer to a different site in that mentor trainer's region to perform a fidelity assessment. The mentor trainer conducted as much of the assessment as she or he was able, with the national trainer providing training and backup during the visit. This allowed the mentor trainer to receive important hands-on training in fidelity assessment and in delivery of feedback.

Fourth, mentor trainers were encouraged to meet regularly with overall VISN mental health leaders in their area and with medical center directors at each supported employment site to garner top-level support for their work with each site.

Fifth, an e-mail group and periodic conference call for mentor trainers was established. This provided mentor trainers and their supervisors with ongoing guidance and support from national supported employment program leadership, as well as opportunities for sharing experiences and discussing challenges. The conference calls were also intended to provide opportunities for skill building, problem solving, and fostering a learning community among the mentor-trainer sites.

In addition to establishing a structural framework for this large-scale mentorship effort, a training initiative in the art of supported employment mentorship was also initiated. Although it is common to utilize subject matter experts as trainers, the skill set necessary to master a skill is not likely to be the same skill set needed to teach the skills to someone else ( 12 ). There is little consensus in the literature as to what makes a good trainer and even less guidance on how to teach someone to be an effective trainer ( 13 ). Some characteristics hypothesized to be related to being an effective trainer include being approachable, supportive, willing to answer questions ( 14 ), and willing and able to provide helpful feedback. Model trainers also are respectful of trainees and encourage their self-reflection ( 15 ).

There is a similar paucity of information about the key components of effective technical assistance that lead to high-fidelity implementation ( 16 ). To address these subtler issues, plans were developed for each mentor trainer to participate in intensive two-day training with a small number of their peers at a centrally located site. No more than five mentor trainers and approximately three or four faculty participated in each workshop. Interactive training was intended to develop interviewing techniques, assessment of specific fidelity scale items, methods of teaching the philosophy and nuances of supported employment, and approaches to giving both positive and negative feedback to encourage change.

A more specific goal of this effort was to develop the mentor trainer's skills in the use of the Supported Employment Fidelity Scale for both assessment and providing feedback to guide program development. Frequent fidelity ratings are viewed as an invaluable training tool, a way to document performance for national VA administrators, and a process for supporting sustainability ( 5, 6 ).

Fidelity scales were originally created as a means of systematizing expert knowledge, allowing individuals already expert in supported employment to document their fidelity to the supported employment model, often with minimal additional instruction or support. Most if not all supported employment fidelity assessments described in the research literature have been performed by a relatively small number of experts who have been hand-picked for the purpose because they already have demonstrated expertise in assessing and fostering fidelity through supportive feedback. Intensive training methods have not yet been needed, and thus they have not been fully developed or tested ( 17 ). Although manuals and instructional videos are available for training people in supported employment and use of the Supported Employment Fidelity Scale ( 18 ) ( dms.dartmouth.edu/prc/employment ), handing out such materials is insufficient for teaching clinicians who have limited teaching and training experience in conducting constructive and accurate fidelity assessments and providing useful feedback. Assessors need to have specialized skills, such as how to ask open, nonleading questions; how to understand and identify common implementation challenges; how to examine data critically; how to interact effectively with all levels of the organization, including leadership; and how to conduct a site visit in a supportive and nonjudgmental manner. If interventions such as supported employment are ever to become widely available, train-the-trainer procedures such as those we have outlined will need to be deployed on a large scale.

Although there have been challenges, we consider this to be a successful implementation. As of this writing, all 21 sites have received five fidelity visits, and all but one have received six over the course of the three-year implementation period. Nine (42.9%) sites are at full implementation (>65 on the Supported Employment Fidelity Scale), and the mean±SD fidelity score at the end of three years is 64.9±2.94. Mentor trainers from all 21 sites are conducting VISN-wide conference calls and conducting site visits based on the needs of the VISN, and they are receiving ongoing training on fidelity assessment by the national consultant.

Our experience with this project thus far suggests that successful implementation is facilitated by supportive leadership at the local level, familiarity of mentoring staff with rehabilitation values, and possession of both skill and experience in leadership and teaching roles. A major challenge for the mentors has been dealing supportively with skeptical administrators and clinicians who have not "bought in" to the supported employment model and genuinely do not feel a need to try a new approach. Although the implementation of supported employment is a national priority, mentors derive limited direct authority from their selection for this task and must reach out to construct alliances with local mental health and medical center leaders in support of the effort. This demands skills in the art of persuasion and alliance building as much as or more than it requires knowledge of and experience in supported employment. Staff turnover in this effort is perhaps the most serious impediment because it sometimes results in a nearly complete loss of acquired skills. At all levels, interactive communication—be it in person or via conference calls and e-mail groups—is the key vehicle for promoting the effort, because formal written materials have little impact. We have been continuously challenged to clarify goals of the program and to provide concrete guidance and support to mentors with a wide range of skills and relevant past experience.

Although great progress has been made in the dissemination of evidence-based practices in recent years, if we are ever to reach the ambitious goal of rebuilding the mental health system as outlined in the New Freedom Commission report ( 4 ), such efforts will eventually need to be carried out on a far larger scale than has been described in the literature thus far and with the explicit goal of establishing self-sustaining internal institutions. We have shown previously that by using one national trainer and a strong support team it has been possible to achieve nearly acceptable fidelity levels at almost two dozen sites ( 6 ). It will take years to collect and analyze the data generated through this effort. However, we believe that if we are to bring about the wholesale broad national change envisioned in the New Freedom Commission report, we must now begin the important dialogue on how best to scale up these efforts. We hope that this Open Forum will stimulate this discussion.

This material was based on work supported by grant MNT 05-098 from the Department of Veterans Affairs Health Services Research and Development.

The authors report no competing interests.

Dr. Resnick is with the Northeast Program Evaluation Center (182), Department of Veterans Affairs Connecticut Healthcare System, 950 Campbell Ave., West Haven, CT 06516 (e-mail: [email protected]). Dr. Rosenheck is with the Department of Psychiatry, Yale University School of Medicine, New Haven, Connecticut.