Implementation of Integrated Dual Disorders Treatment: A Qualitative Analysis of Facilitators and Barriers

The unit of analysis was each community mental health center that participated in the National Implementing Evidence-Based Practices Project. Thirteen agencies in three states volunteered and were chosen to participate in the study of integrated dual disorders treatment implementation. State mental health authority directors or their designees were instructed to independently select agencies that represented the array of agencies in their state, that were of average competency, and that were interested in implementing integrated dual disorders treatment and willing to do so. Thirteen agencies were selected. Two agencies dropped out immediately after the baseline assessment when they learned more about implementation of the practice, leaving 11 agencies that completed the two-year project. Agency employees gave written informed consent before participation. The Dartmouth Committee for the Protection of Human Subjects reviewed and approved the project.

The model of implementation for this project was developed by implementation experts who used the available research on organizational change and technology transfer ( 15, 16 ). This research on implementation suggests that implementation efforts tend to be more successful when educational materials and training are supplemented by consultation, implementation support, and feedback and that organizations implementing more complex practices may benefit from more implementation support.

The model in this study involved a lead consultant-trainer who used a standardized set of procedures and materials that were packaged into an implementation resource kit ( 17 ). The kit included written information developed for stakeholders, tips for leaders, training tools (training slide sets and videos), and learning tools (demonstration videos and workbooks). The resource kit cued implementation in stages (preparation, startup, and maintenance) and recommended engaging community, agency, and consumer stakeholders. The consultant-trainer facilitated implementation by assisting each site with choosing a steering committee, advising implementation leaders and the steering committee over two years, conducting staff training for the practice, and being available to answer questions throughout the implementation. Program quality was monitored with a formal fidelity rating and report every six months. The report was provided to the steering committee and leaders as feedback and guidance about the quality of the developing service and to help them modify and further implement the service.

The main quantitative outcome was the degree to which the new service adhered to established principles for integrated dual disorders treatment. Overall fidelity methods and results were previously reported ( 11 ). To contextualize the facilitators and barriers to implementation, integrated dual disorders treatment fidelity scores are reported here in more detail to identify organizations that implemented the practice to a greater or lesser degree.

The main qualitative outcomes of interest were the facilitators and barriers to implementation. Twenty-six dimensions (see Table 1 ) that were hypothesized to be important to the implementation process, nested within five broad domains, were designated a priori through consultation of relevant literature ( 8 ) and discussion among implementation experts.

Research staff, trained as implementation monitors, functioned as independent observers. They documented implementation efforts over two years with standardized and systematic methods to ensure rigor and comparability across sites and monitors. Monitors visited sites monthly in the first year and every other month in the second year to observe implementation planning meetings, clinical team meetings, and other activities related to implementation, as well as to talk with clinicians, administrators, and consumers. The ethnographic observations from these visits were documented in field notes. In addition, at six-month intervals, the implementation monitors (one at each site) conducted and taped semistructured interviews with the consultant-trainer and the program leader. All interviewers were trained to follow a standardized topic guide designed to elicit opinions and experiences regarding facilitators and barriers to implementation. These interviews included a series of open-ended questions, beginning with "How is the implementation going?" and ending with "What are the barriers to implementation here?" Interviewer reliability was facilitated by regular supervision. Interview tapes were transcribed and imported into Atlas.ti, a software program for qualitative analysis, as were field notes and other sources of information.

Paired t tests were used to assess change in mean fidelity scores from baseline to two years.

Because this was a large study with multiple sites and a large data collection staff, a combined deductive-inductive model for analyzing the qualitative data ( 18 ) was undertaken in a three-step process. First, the implementation monitor at each site coded the qualitative interviews and the field notes from site visits according to the 26-dimension scheme in Table 1 . Once the data from the two-year period had been collected and coded, the implementation monitor rated each theme within each dimension on the degree to which it served as a facilitator or barrier to implementation; possible ratings ranged from 2 (strong facilitator) to 0 (neither facilitator nor barrier) to -2 (strong barrier). At the end of implementation, the monitors wrote a report for each site, including a detailed table of facilitators and barriers categorized into the five domains and 26 dimensions described above. These reports followed a standardized format and provided the basis for the cross-site analysis reported herein. Implementation monitors were trained and supervised by the lead research team. To maintain consistency of the coding and ratings, implementation monitors independently coded and extracted themes from common documents and then discussed their choices during supervision meetings throughout the study.

The second step of analysis distilled the three most prominent facilitators and barriers among the 26 dimensions at each site. A theme was defined as prominent when it had a rating of 2 or -2 in the dimension display, was emphasized in the report synopsis, and was spoken of in strong or persuasive language in interviews and field notes recorded in the Atlas.ti database. Using a standard qualitative data analysis technique ( 19 ), we used several layers of rating of facilitators and barriers to distill out the most prominent factors. Two lead analysts and four supporting analysts carefully reviewed each site report. The two lead analysts independently extracted the main three facilitators and barriers for each site, reported and reviewed the ratings with the analyst group, and then came to consensus on the prominent facilitators and barriers for that site. We found a strong degree of agreement between the two lead analysts. Although codes were designated and grouped a priori, the analytic team allowed the data to drive renaming or regrouping of themes, which occurred in some cases. For example, although attitude is listed under the domain "priority," the analysts found that the attitudes of the leaders were frequently referred to as a facilitator or barrier. Attitude was therefore considered part of leadership.

The third step was to distill the prominent common facilitators and barriers across all 11 sites. After all site facilitators and barriers had been chosen, the analyst group compared and contrasted the main facilitators and barriers for all 11 sites, discussed them, and examined examples and supporting quotes to establish common and predominant implementation facilitators and barriers. Broad themes, if present, were sought. Facilitators from higher-fidelity sites and barriers from lower-fidelity sites were emphasized. This form of multistep coding is considered a robust check and balance on rigor in qualitative research ( 19 ).

Administrative leadership was noted as a significant factor in implementation at every site. Successful sites tended to have a committed midlevel leader who had sufficient authority to make changes within the organization. Three components of leadership emerged: attitude, priority, and action.

Administrative leaders' attitudes set the tone and modeled appropriate staff responses to the challenges of starting a new service. Attitude alone, however, was insufficient. Leaders at successful (high-fidelity) sites made implementation a priority among their myriad, competing responsibilities, and they took action by making administrative and policy changes to implement the new practice. Action steps varied, including hiring, changing, or firing staff; changing the structure and focus of supervision; changing the treatment plan formats; and implementing new policies and practices for screening for and assessing substance use. Leadership action overcame barriers, such as staff turnover, in the implementation process. Notably, many of the successful agencies assigned implementation leadership to two administrators, one with administrative skills and one with advanced clinical expertise, who shared leadership tasks.

Organizations that implemented the service with low fidelity experienced problems with leadership. Some had enthusiastic leaders who, amid competing demands, were unable to prioritize implementation and take facilitative action. As one steering committee member put it, "The agency takes on these projects without allocating internal resources to make them succeed." Other organizational leaders appeared unable or unwilling to take action to implement the practice, despite the agency's agreement with the state authority or training center.

Successful organizations utilized the consultant-trainer for two key functions: consultation for initial and ongoing implementation plans, including the type and timing of implementation activities; and training staff and leaders in the information and skills necessary to deliver the service. Consultation efforts varied; some efforts focused on initiating facilitators, and others focused on overcoming a variety of barriers. At some sites, consultation focused on policies and organizational structures to support the practice, while at other sites consultation focused on helping administrators learn leadership and supervisory skills.

Because the delivery of integrated dual disorders treatment requires skills in assessment and counseling that many public mental health clinicians may not have—for example, motivational interviewing and substance abuse counseling—the agencies either hired staff who possessed these skills or, in most cases, trained existing staff. Successful sites supplemented classroom training with regular supervision (see next section). At some centers, the consultant-trainer attended team meetings to provide supervision between monthly classroom-style trainings; at other centers, agency leaders were able to function in this key capacity.

Although many informants reported that the toolkit materials were helpful, most stated that the materials were less important than interactions with the consultant-trainer. Many project leaders made such comments as, "It's easier to call (or e-mail) someone than to look it up." The presence of the consultant-trainer was viewed as encouraging and appeared to boost morale during the challenges of organizational change.

Although skilled consultation to the agencies was helpful, unskilled consultation was harmful. In some cases, a consultant-trainer did not work well with leaders and appeared to confuse and hinder the implementation process. In addition, some low-fidelity sites received poor-quality training or simply did not allow employees to attend training.

Many observers noted that clinical leaders require mastery of service-specific knowledge and skills in order to provide ongoing staff training and supervision. Most leaders developed these skills during the study. As one implementation monitor stated, "Supervision played a key role in the success of the integrated dual disorder treatment team."

The absence of high-quality clinical supervision was a common barrier observed in organizations with moderate or low fidelity.

Many of the 11 organizations experienced significant staff turnover. In some, change in staffing facilitated implementation by moving unwilling staff to other positions, terminating incompetent staff, or hiring new staff who were willing and competent. Abrupt turnover of multiple staff was perceived as a barrier, although several organizations overcame this problem and attained reasonably high fidelity by immediate rehiring and training. Chronic staff turnover, however, appeared to be a much more difficult barrier, creating heavier training and supervisory challenges that some leaders were unable to meet. In addition, some of these teams were short-staffed for long periods, resulting in high caseloads. One case manager stated, "We are so overwhelmed we don't have time to practice the [integrated dual disorders treatment] techniques." It appeared in some cases that staff turnover was related to leadership problems with hiring, supervision, and other implementation support. One organization with chronic turnover was able to accept advice from the consultant-trainer to change hiring practices to reduce turnover, including utilizing team members in the hiring process, allowing job candidates to shadow experienced practitioners, and providing additional supervision to new practitioners.

In organizations that implemented the model with low fidelity, staff and leaders often expressed concern about the financial implications of making organizational changes, including allotting time for staff training, providing supervision, and reducing caseload size. One project leader stated, "Every time I've got a clinician sitting in a meeting … it's costing me money." These concerns occurred despite some funding (less than $10,000) to defray implementation costs and the provision of free training and consultation. In one state a simultaneous initiative to implement assertive community treatment competed with the implementation of integrated dual disorders treatment, in part because additional funds were offered for assertive community treatment implementation. This situation led one organization to abandon integrated dual disorders treatment implementation entirely. The program leader stated, "We are going to do whatever they … pay for."

At least one financially strapped organization was nevertheless able to overcome this barrier with leadership priority and consultation, implementing the service model with excellent fidelity. Successful organizations reported that their treatment teams were financially viable because Medicaid and Medicare payments for the clinical services covered the cost of running the services in those agencies.

Several other themes arose that deserve mention. One facilitator embedded in this model of implementation was the use of regular fidelity measures to assess service quality and to provide feedback to leaders. Leaders at successful sites used these reports to develop action plans to further their implementation efforts. Another important facilitator was the integration of the agency's quality improvement actions with the implementation efforts. A third was each organization's relationship with the state or county mental health authority, which could help or hinder the process.