Implementing the Collaborative Chronic Care Model in Mental Health Clinics: Achieving and Sustaining Clinical Effects

The study was approved by the VA Central Institutional Review Board as a combined quality improvement evaluation and research project. Detailed methods are found in the primary outcomes report (8) and protocol design paper (9).

Briefly, in collaboration with the VA Office of Mental Health and Suicide Prevention, we recruited nine VA medical centers that expressed interest in implementing the CCM in one of their general mental health outpatient clinics. These were identified through top-down publicity on national and regional conference calls. The first sites to complete the project letter of agreement were enrolled. Control clinics comprised all other general mental health clinics in these medical centers. The implementation strategy used was evidence-based blended internal-external facilitation, as detailed in the protocol design paper (9). Medical centers provided protected time of up to 4 hours per week for an internal facilitator, who was a mental health clinician or administrator with process redesign experience. Medical centers also supported 1 hour per week for process redesign meetings with clinic providers. These staff worked for 1 year with an offsite external facilitator, predominantly through videoconference, telephone, and e-mail (approximately 2.6 hours per week), to review and revise their existing clinical processes to better align with CCM principles, following the rubric laid out in a self-assessment CCM workbook (available upon request from authors C.J.M. and B.K.). In this stepped-wedge design, all sites received implementation support, and the start time of support was randomly assigned in three waves of three sites each.

Both implementation and clinical outcome measures focused on changes from baseline to the end of 1 year of facilitation support. Clinical outcome measures included mental health hospitalization rate based on admissions to mental health units. Rates during 1 year of preimplementation and 1 year of implementation support were compared among veterans treated in the CCM clinics (N=5,596) versus veterans treated in non-CCM clinics (N=46,755); dementia was the only exclusion criterion. In the analysis of the primary trial data, fitting a linear spline model demonstrated reductions in hospitalization rate in both groups during the implementation year, but rates were lower in the CCM clinic populations compared with the non-CCM clinic populations in the same medical centers (0.81% per quarter versus 0.94% per quarter, respectively; p<0.001).

To investigate whether the beneficial effect of CCM on hospitalization rate was maintained after the cessation of implementation support, we conducted two sets of similar analyses. In analysis 1, we compared hospitalization rates for the CCM versus non-CCM clinics in the year after the cessation of implementation support by using a repeated measures logistic regression model. The populations utilized in this analysis were the same populations as in the original protocol, followed for an additional year after the cessation of implementation support. That is, veterans were included if they were active in the clinic by virtue of having had two visits to the clinic in the year before implementation, including one visit in the quarter immediately preceding implementation support (9). This analysis was designed to determine whether there were enduring effects on hospitalization rate in the populations being treated at the time the clinics were receiving implementation support. In analysis 2, we conducted identical analyses on veterans active in the clinics as the postimplementation year was beginning, defined as those active in the clinic at the time that implementation support ended (i.e., they had two clinic visits in the year during implementation support, including one visit in the quarter before the cessation of implementation support). This analysis was designed to determine whether the effects were maintained in populations active in the clinic at the time implementation support was ending (i.e., later in the process of establishing CCM principles in the clinic). Veterans were included in each population regardless of whether they continued to receive care in those clinics (intention to treat).

We did not maintain formal contact with the sites after the 1 year of implementation support ceased. Thus, we were unable to assess the extent to which CCM processes themselves continued beyond the year of implementation support. However, we do know that there was clinician staff turnover at each site and that internal facilitators at three sites left their Behavioral Health Interdisciplinary Program (BHIP) team roles late in the implementation support year or during the follow-up year. On the other hand, six of the internal facilitators had supervisory roles over BHIP teams during the second year, so continuity of structure (if not processes) was potentially present to some degree in a majority of sites.

Demographic and clinical characteristics comparing veterans treated in CCM clinics with those treated in non-CCM clinics were analyzed by using an unpaired, two-tailed t test or chi-square test. Knowing that most variables would show statistically significant differences because of very large sample sizes, we assessed potential clinical impact by determining effect size. Effect size was obtained from Cramer’s V for categorical variables (medium effect size, 0.3–0.6; large effect size, >0.6) or η² for continuous variables (medium effect size, 0.06–0.14; large effect size, >0.14) (10).

The primary outcomes paper (8) addressed the question of whether hospitalization rates in the CCM clinics dropped more than those in non-CCM clinics from the same medical centers during the year of implementation support. The present analyses directly compared hospitalization rates in the year after the cessation of implementation support. Following the approach of the primary outcome analyses, longitudinal comparisons of quarterly hospitalization rates in CCM versus non-CCM clinics during the postimplementation support year were analyzed by using a repeated measures logistic regression model that accounted for correlation between the four adjacent times (quarters) to detect whether many of the same participants were repeatedly hospitalized (the PROC GLIMMIX procedure). Because covariate adjustment minimally perturbed the estimate of the CCM effect in the primary paper, these subsequent analyses were unadjusted for the demographic and clinical characteristics used in the primary outcomes paper (8). Analyses were conducted with the SAS statistical package, version 9.4.

A table showing population characteristics for CCM and non-CCM clinics is available as an online supplement to this report. No differences reached medium or large effect sizes.

The two main analyses compared the mental health hospitalization rates in the postimplementation support year for the CCM versus non-CCM populations. The null hypothesis (H₀) assumed equal rates (i.e., odds ratio [OR]=1). Analysis 1 (Figure 1A) failed to reject H₀, comparing the original CCM and non-CCM populations (estimated log[OR]=−0.06 0.10, p=0.52; estimated OR=0.94, 95% confidence interval [CI]=0.78–1.14). Analysis 2 (Figure 1B) also failed to reject H₀, comparing CCM and non-CCM clinic data for veterans who were active in the clinics as the implementation support year was ending (estimated log[OR]=−0.11 0.12, p=0.37; estimated OR=0.90, 95% CI=0.71–1.14).

Although we saw significant differences in the mental health hospitalization rate between CCM and non-CCM clinics during the 1-year period of active implementation support, those differences did not continue in the year after implementation support ended. We are aware of no other CCM-related studies on the maintenance of clinical effects after the cessation of implementation support.

There are several possible reasons why the clinical effects seen during implementation support waned after its cessation. It is possible that CCM clinical care processes were not maintained or that the CCM clinical care processes were maintained but, for some reason, stopped working. The latter is highly unlikely, because one would have to posit that added experience with the model led to the loss of CCM skills. It is also unlikely that there was a wholesale shift in some population or system factor across each of the nine sites that rendered the CCM ineffective.

Alternatively, we speculate that the lack of maintenance of clinical effects may have been due to the lack of sustainment of CCM clinical processes (11–13). For example, although our implementation strategy contained some attention to maintaining clinical processes, this attention may have been of insufficient intensity or duration, or it may not have started early enough in the implementation support process. Additionally, external facilitators had no continuing formal contact with the sites after the initial 1-year period of implementation support. There was also staff turnover in the clinics; although part of the implementation process included developing a procedure manual to assist with training, it may not have been used effectively with new staff. Furthermore, there were changes at some sites in health care leadership, which may have affected support. Moreover, changes in organizational priorities at a national level were evident across the system (e.g., growing emphasis on opiate and suicide reduction initiatives in VA national strategic plans), which made maintaining the CCM—although clearly relevant to these challenges—less of an explicit priority for leadership and clinic staff.

There are several limitations to these analyses. We were not able to collect longitudinal data on clinical processes to determine whether these processes drifted from adherence to CCM principles, as would be the case in a formal implementation sustainment study (11–13). Similarly, we were not able to extend the collection of the clinical status outcome measures that were reported in the primary study to determine whether those clinical effects were maintained. Finally, data derive from experience within a single integrated, capitated U.S. health care system that treats a predominantly male population; findings may not apply to fee-for-service or other health care delivery systems.

These results underscore the need not to assume that, once an intervention is shown to have clinical impact, it will continue to have clinical impact in the absence of ongoing implementation support. Rather, attention to factors that will support continuation of the processes responsible for clinical effects is needed throughout the period of implementation support as well as during initial intervention development. Investigators, administrators, and clinicians should keep in mind that achieving enduring clinical impact requires the long view: although it may be difficult to get health care systems to change, it is even more difficult to keep them changed.