Strengths Model Case Management Fidelity Scores and Client Outcomes

The fidelity scale for SMCM was developed to ensure adherence to key components of the model for implementation purposes (1). The scale consists of 11 items in three core areas: structure, supervision, and service and clinical issues. Items are rated on a 1–5 scale with 5 signifying higher fidelity. For example, “What percentage of consumer contact time occurs in the community?” is rated as 1, less than 40%; 2, 50%–64%; 3, 65%–74%; 4, 75%–84%; and 5, 85% or more. Fidelity reviews were conducted at baseline and then every six months during the first two years of implementation. Annual reviews occurred thereafter. A baseline fidelity review was conducted after an agency had committed to implementing SMCM but before any formal training and technical assistance had started. Total fidelity scores can range between 11 and 55, with a score of 45 and higher considered to indicate high fidelity to the model. Data sources used to rate fidelity included observation of team meetings, chart review, review of SMCM tools used in practice, and interviews with program staff and people receiving services.

Secondary data analysis was conducted. Data were collected over an 18-month period between May 2004 and October 2010 during regularly scheduled fidelity reviews for each of the 14 case management teams representing ten agencies that served an average of 953 clients who met criteria for serious mental illness. There were 1,195 clients at baseline, 965 at six months, 849 at 12 months, and 802 at 18 months. The intervention time varied among the teams.

Fidelity scores were added to a database once the review was conducted. Each review was conducted by at least two consultant-trainers who had extensive experience in SMCM implementation at the University of Kansas School of Social Welfare. Interrater reliability (intraclass correlation) between the two raters of the fidelity scale was .97, representing a high level of agreement.

In addition to rating fidelity, all case managers on the team completed reports regarding the current status of each client on their caseload in the areas of psychiatric hospitalization, competitive employment, postsecondary education, and independent living. These outcomes were provided for the ratio of successful events (such as being competitively employed) over total number of clients in a team. Demographic characteristics of case managers and their clients were not available in this data set. In order to assess the relationship between the fidelity scores and client outcomes, time-varying covariate linear growth modeling was applied via SAS version 9.2 with PROC MIXED. The study was approved by the institutional review board of the University of Kansas.

Marginal means and standard deviations for fidelity scores over an 18-month period were 27.4±9.7 at baseline, 39.2±6.9 at six months, 44.7±4.8 at 12 months, and 47.0±4.4 at 18 months. Competitive employment rates at baseline and respective follow-ups were 15.5%±6.5%, 17.6%±5.6%, 18.1%±3.4%, and 20.2%±5.1%. Psychiatric hospitalization rates (percentage hospitalized) were 10.8%±5.7%, 9.9%±4.8%, 5.4%±3.2%, and 4.2%±3.1% at baseline, six, 12, and 18 months, respectively. Postsecondary education rates (percentage attending school) were 3.6%±3.0%, 4.5%±4.0%, 5.4%±4.0%, and 5.7%±3.1%, respectively. Percentage of persons living independently were 89.1%±5.0%, 90.3%±6.4%, 95.0%±3.7%, and 95.7%±3.2%, respectively.

The fidelity score (β=6.43, p<.001) and outcomes, including psychiatric hospitalization (β=–2.48, p<.001), postsecondary education (β=.65, p<.05), and independent living (β=2.42, p<.001) significantly improved over time, and the change in the rate of competitive employment was marginally significant (β=1.43, p=.06). However, once a time-varying covariate (fidelity score) was introduced, the time effect disappeared for employment, hospitalization, and education, and the time-varying covariate effect of the fidelity score was significant (p<.05) for all but independent living (p=.85). That is, the improvement of outcomes was explained by the improvement of the fidelity score (Table 1).

Concurrent correlations between the fidelity score and outcomes at each time period showed expected directions (that is, the correlation coefficients ranged from .2 to .4 for employment, from −.3 to −.6 for hospitalization, and from .4 to .9 for education) except for independent living (that is, the correlation coefficients were .2 for baseline, −.3 for six months, and 0 for both 12 and 18 months).

To our knowledge, this is the first published study to examine the impact of SMCM fidelity on case management teams' performance in terms of clients' outcomes, including competitive employment, psychiatric hospitalization, postsecondary education, and independent living rates. Our preliminary findings showed that client outcomes improved over time, but improvements depended on the fidelity scores, except for independent living. The nonsignificant association for independent living may be explained by the fact that the state average in Kansas for independent living rates was already high at 90%. This may have contributed to the minimal effect of fidelity on this particular outcome. However, the effort made for improving SMCM fidelity seemed to enhance the clients' outcomes over time. Most teams were able to achieve high fidelity within 12 months, with some doing so within six months, making SMCM a feasible practice for agencies to implement.

The study had some limitations. First, secondary data with a convenience sample were used. A lack of demographic information for clients as well as case managers limits the generalizability of the study results. Second, individual client outcome was aggregated at the team level as a proxy of the team level achievement of SMCM. That is, client outcomes were calculated as rates restricted to the cohort active on the caseload at each time point. This would restrict the direct linkage of the improvement during 18 months for the observed outcomes. Third, a number of clients declined over time. Information about the reasons for dropout is not available in this data set. Under naturally occurring circumstances, case managers and clients might come and go from the teams as a result of turnover and disengagement. Future experimental designs will be needed to ensure internal validity. Finally, the overall improvements of the outcomes were modest. For a sizable portion of the 18 months, teams had yet to achieve high fidelity, meaning that clients did not receive 18 months of high-fidelity SMCM. It may take a longer period of time for clients to receive the full benefits of SMCM services. Continued effort to examine the effect on client outcomes with more time points will be needed.

Despite the limitations, our findings add to the existing literature on SMCM. The strengths perspective, along with the recovery movement, has gained popularity over recent decades. Empirical evidence linking fidelity and client outcomes has been needed. Future studies are needed that focus on intraindividual growth with SMCM and on attributes of potential clients who receive greater benefit from SMCM, such as demographic characteristics, psychiatric symptoms, and interactions with other mental health services use. How specific items on the fidelity scale contribute to outcomes will also need to be examined.

This is the first study to investigate the association between SMCM fidelity and client outcomes. The results suggest that SMCM had a significant positive effect on the client outcomes of competitive employment, psychiatric hospitalization, and postsecondary education rates over an 18-month period. The fact that all 14 teams reached high fidelity suggests that SMCM is an implementable practice. Given the dismal performance for these outcomes by many mental health systems, further research that avoids the limitations extant in this study is both warranted and necessary.

This study was developed by the University of Kansas School of Social Welfare, Office of Mental Health Research and Training, through contract DBHS 0710 0611 0330 with the Kansas Department of Social and Rehabilitation Services.

The authors report no competing interests.