Implementing the Illness Management and Recovery Program in Community Mental Health Settings: Facilitators and Barriers

Four states volunteered to implement illness management and recovery. More detailed information is given on participating states, relevant state-level data, and the role of state mental health authorities in other articles ( 11, 12, 13 ). To summarize, various mechanisms were used among these states to recruit public-sector community mental health agencies as study sites. Some chose among solicited proposals; others used less formal procedures. At each site, illness management and recovery was implemented in programs for people with severe mental illness. Implementation efforts occurred over a two-year period, from 2002–2003 to 2004–2005 (sites had different start dates).

Standardized instructions (described below) regarding systematic observation of implementation efforts were designed and distributed by a central coordinating center (Dartmouth Psychiatric Research Center) to ensure rigor and comparability across sites. Each site had a supervised researcher (an implementation monitor) who functioned as an independent observer of implementation, documenting the process both qualitatively and quantitatively. Following the instructions given by Dartmouth Psychiatric Research Center, the implementation monitor made monthly visits (occasionally by phone) that were written up as field notes. The implementation monitor also conducted interviews every six months with the illness management and recovery program leader and the consultant trainer. These were taped and later transcribed. The implementation monitor also wrote down his or her own responses to the same questions posed to the program leader and the consultant trainer. These interviews followed a standardized topic guide designed to elicit opinions and experiences regarding barriers to and facilitators of implementation. Topics covered in the interview included "reasons for success," "reasons for shortcomings," and "assessment of the implementation package." Examples of questions include, "For those elements that are not high-fidelity, what has gotten in the way?" The implementation monitor also interviewed the agency director and the clinical director at baseline and 24 months; all these interviews were taped and transcribed. Transcriptions and field notes were imported into Atlas.ti qualitative software.

The quantitative measurement of implementation performance was based on assessments of fidelity to the practice model, which were conducted at baseline and every six months thereafter. Further information on these processes is given in another article ( 10 ). Consultant trainers and implementation monitors conducted these assessments by using a standard illness management and recovery fidelity scale with 13 items, such as the use of motivational interviewing, the use of cognitive-behavioral techniques, follow-up on personal recovery goals, and involvement of significant others. Interrater agreement was evaluated across pairs of independent raters and found to be high for this fidelity scale (intraclass correlation=.97, N=50). The Dartmouth Medical School Institutional Review Board approved the overall study, and similar approval for each state was obtained through the local principal investigator's affiliated university.

Twenty-six dimensions hypothesized to be important to the implementation process, nested within five broad domains, were designated through expert consensus a priori after review of relevant literature ( Table 1 ). The five domains were prioritization, leadership, workforce, workflow, and reinforcement. This deductive a priori approach to data analysis is recommended as optimal for large multisite, multi-investigator studies. It creates standard categories that can be compared across sites with ease ( 14 ).

Implementation monitors were instructed to code the transcribed qualitative interviews and the site-visit field notes in Atlas.ti according to this dimensional schema. Once final interviews and site visits were completed and coded, implementation monitors generated 26 Atlas.ti output files (a list of all text allotted under a code) for each dimension. Through careful content analysis of this list, implementation monitors then judged the bearing that each dimension had as a facilitator of or barrier to overall implementation, summarizing their judgment in a dimensional display table. Each dimension was also given a summary score from 2 (high facilitator) to -2 (high barrier). This summary score was created in order to facilitate cross-site comparison and also to allow analysts a means of quickly assessing and categorizing the impact of a dimension.

At the end of the two-year implementation period, implementation monitors wrote summary site reports in which the dimensional display table was included, along with a brief synopsis discussing and analyzing the implementation process over the 24 months in narrative format. These reports followed a standardized template.

For this article, the site reports were assembled in rank order from highest fidelity to lowest fidelity. The first and second authors (hereafter referred to as evaluators) then independently reviewed and evaluated each site report. This form of multiple evaluation is considered a robust check and balance on rigor in qualitative research ( 15 ). The overall aim was to distill prominent facilitators and barriers to implementation. Prominence was defined as dimensions that were labeled with a 2 or a -2 in the dimension display, that co-occurred in the synopsis, and that were described in strong, forceful language in both the dimension display and the synopsis.

Evaluators listed prominent dimensions for each site report, sometimes synthesizing dimensions into broader themes where the site reports implied dimensional overlap. This process resulted in two lists, one for each evaluator, documenting evaluator-identified barriers and facilitators for each site. Once this process was finished, the evaluators compared their lists of overall dimensional and thematic extractions, formulating the final schema presented in the results. At this stage, the two last authors read a subsample of site reports in order to examine the verisimilitude of the first two authors' final schema. All authors agreed on four broad emergent themes as strongly influencing the success (or lack thereof) of implementation.

As seen in Table 1, leadership was preidentified at the start of the study as a broad domain potentially influencing implementation. Strong and active leadership appeared to characterize high-fidelity sites, with both evaluators identifying leadership as the strongest facilitator at the four highest-fidelity sites. Leadership was exercised at various organizational strata. We identified four strata of leadership that had an influence on implementation. These were leadership by the state mental health authority, the agency director, the community support program director, and the illness management and recovery program leader. Implementation appeared to be most effective when there was creative synergy between leadership at these different levels. This was often the case, because strong leadership at higher levels generally permeated down to leaders at lower levels. Similarly, we found that low-fidelity sites were marked by a leadership void. Both evaluators identified this as the strongest barrier at the four lowest-fidelity sites. Some sites appeared to lack state mental health authority leadership, whereas others lacked agency or program leadership. Again, this operated in a synergistic manner, with low-fidelity sites lacking leadership across the organizational spectrum. For example, in one state, the state mental health authority was underfunded and appeared to lack the time to champion new programs, such as illness management and recovery. This was mirrored by a lack of enthusiasm displayed by leaders at community mental health centers in the state.

Organizational culture was not one of the preidentified domains or dimensions listed in Table 1 . However we created this superordinate theme after noticing that preidentified dimensions such as "attitude," "reward," "policies," and "change culture" all reflected different components of what could broadly be labeled "organizational culture." The highest-fidelity sites were marked by a strong culture of innovation accompanied by a positive attitude to new practice in general and to illness management and recovery in particular. This may be a corollary of previously described state mental health authority and agency leadership, policies, and attitudes that encourage innovation and cutting-edge practice. Both evaluators identified this theme as one of the top three facilitators in four of the five highest-fidelity sites. These sites had successfully implemented evidence-based practices in the recent past and considered themselves models of vanguard agencies. These agencies thus embraced implementing the illness management and recovery program as an opportunity to develop expertise in a new, recovery-oriented, and consumer-centered approach. In contrast, low-fidelity sites appeared much more conservative and marked by organizational inertia. These sites had not embraced new practices in the recent past and expressed a strong preference toward the status quo. Attitudes toward practice change were generally negative. Implementing illness management and recovery was thus seen as a burden or a threat rather than an opportunity. For some agencies, maintaining the status quo meant staying afloat in troubled waters, marked by diminishing budgets and state mental health authority instability, with the view that implementing a new practice would consume precious resources and energy.

Training was one of the preidentified dimensions hypothesized as important in the implementation process. We found that the provision of high-quality training in illness management and recovery by competent and respected trainers appeared to be a key factor in some of the high-fidelity sites, with one evaluator noting training as an important facilitator at three of the top six sites and the other evaluator noting its import at two of the top six sites. In low-fidelity sites, the picture was more complex. In a few lower-fidelity sites, program leaders stated that training was insufficient and of low quality. However in the four lowest-fidelity sites, neither evaluator identified training as a significant barrier. When taking the wider data set into consideration, we cautiously posit that the few complaints about trainers by program leaders were due more to interpersonal conflict or to an effort to shift blame. In fact, the quality of training was generally extolled at the four lowest-fidelity sites by the implementation monitor, the program leader, and other agency staff. This suggests that training may act synergistically in the presence of other facilitating variables, such as strong leadership. However, strong training may be insufficient to overcome the tide of wider macro-level barriers, such as lack of leadership and an inert organizational culture. This may especially be the case where intensity of training is dependent upon leadership request. Leaders in inert organizational cultures were not proactive in engaging trainers or incorporating their suggestions. Training therefore appeared to have some influence on successful implementation, but it appeared dependent upon strong leadership and an organizational culture that was open to change.

Supervision was one of the 26 preidentified dimensions posited as important to implementation. There was no dimension labeled "staff," although many comments related to staff skills, attitudes, and abilities were coded under dimensions such as "staffing," "attitude," and "mastery." The data suggested a strong link between quality and commitment of supervisors and their staff. The highest-fidelity sites were characterized by staff that was strongly committed to consumer well-being and recovery, displaying a positive attitude to the new practice. Many of these staff members also had previous experience with interventions involving similar clinical skills (for example, cognitive-behavioral therapy) and were eager to learn and practice something considered innovative and cutting edge. Supervisors in high-fidelity sites seemed equally convinced of the desirability and efficacy of illness management and recovery. This intervention was seen as an opportunity for staff and supervisors to expand their skills and experience. One evaluator identified staff and supervision in four of the six highest-fidelity sites, the other in five of them.

In contrast, low-fidelity sites had serious staffing problems across many dimensions. Many had high turnover of frontline practitioners, seemingly as a result of low pay, demoralization, and unappreciative or unsupportive leadership. Staff and supervisors at these low-fidelity sites seemed uninterested in learning new practices, and they seemed to perceive illness management and recovery implementation as adding to an already heavy workload. Some were also unconvinced about illness management and recovery as an intervention. For example, staff at some low-fidelity sites complained that the written materials were dense and difficult for clients, especially those who had literacy problems, whereas staff at high-fidelity sites did not express this concern. Others did not see how illness management and recovery differed from extant "wellness" programs. This negative attitude among staff and supervisors could be an inevitable outcome of lack of leadership and negative organizational culture. Both evaluators identified this as a significant barrier in four of the bottom six sites.

The following case studies are included in this article to embody the themes outlined in the Results section. The first is from the highest-fidelity community mental health center; the second, from the center scoring second lowest in fidelity. The latter was chosen because the lowest-fidelity site had missing data.

High-fidelity case study . This site had a baseline fidelity of 2.2, which rose to 4.2 at six months and reached 4.5 at two years. (Fidelity was measured with the 13-item fidelity scale mentioned in the Methods section. Possible scores range from 1 to 5, with higher scores indicating better fidelity.) All four factors identified above coalesced at this site. The implementation monitor noted that "the local mental health authority is very pro evidence-based practice—[it] had representatives who has [sic] historically provided forward-thinking and fiscal guidance and support for the agencies' new initiatives." Indeed the state mental health authority provided generous resources to support the project throughout the two-year implementation period. This seemed to encourage clinical leadership, which was also strong.

The implementation monitor noted that "The team leader and the program leader were able to provide excellent supervision and mentoring in house. … Clinicians take pride in their programs." The agency has a well-utilized open-door policy, where staff can seek advice from any available clinical professional. This suggests a strong agency commitment to mentoring, as well as proactive staff willing to initiate supervisory contact. There is a strong culture of innovation at this agency, which is strongly recovery oriented. The implementation monitor noted that the "agency is committed to recovery … [it] has [a] history of striving for excellence and innovation." It was regularly chosen as a site for pilot studies and had successfully implemented other innovative practices already. The community support program leader stated that this was done "because it fit with our mission of balancing practicality with high standards. It was very successful, beginning three years ago and still going strong." Good leadership manifested itself in hiring and allocating decisions. The consultant trainer noted that the staff were "enthusiastic and energized … filled with high-achievers" who were a model of success. From this pool, illness management and recovery clinicians were chosen on the basis of their being "eager" or "positive." The implementation monitor noted that "a major strength is the level of education and the training of staff selected for illness management and recovery. Most have master's degrees and experience with evidence-based practices." Leaders, staff, and consumers utilized regular feedback from the research team to improve performance. News of success further motivated involvement, and action steps were taken to ensure continued success. All this intertwined to ensure smooth implementation at this site.

Low-fidelity case study. This site had baseline fidelity of 1.6, which rose to 3.6 at six and 18 months but dropped to 2.8 at 24 months. There was a leadership void at all levels of implementation. The implementation monitor noted that "A great deal of financial and leadership upheaval at the state mental health authority in early implementation led to decreased involvement at the state level. Competing commitment led agency leaders to become less interested … the evidence-based practice program leader was viewed as lacking in initiative and not successful at identifying and removing barriers." These problems in leadership appeared to demoralize staff, to the extent that by month 12 all illness management and recovery therapists had resigned. The implementation monitor noted that "Several clinicians stated that their reason for leaving was because of disagreement with the evidence-based practice program leader's leadership. The clinicians felt the program leader was not responsive to their needs and was too controlling." For several months, the program leader was the only illness management and recovery practitioner and had reduced ability to provide illness management and recovery services because of time spent recruiting new clinicians. These newly hired clinicians were very inexperienced. They reported trouble understanding illness management and recovery and were skeptical about its benefits. The agency was not marked by a culture of innovation. It had never attempted to implement an evidence-based practice in the past. The implementation monitor noted that its recent past was "a period of great instability and lack of funding." The implementation monitor reported that the consultant trainer attempted various strategies to energize the leaders and clinicians at this agency, such as providing additional training in illness management and recovery for new clinicians. However, these met with very limited success, because agency inertia and indifference appeared to work against the consultant trainer's efforts to advance implementation.