The Learning Curve After Implementation of Collaborative Care in a State Mental Health Integration Program

The institutional review boards at the University of Washington and Weill Cornell Medical College approved this study, with a waiver of informed consent for individual patients. Data were from Washington State’s Mental Health Integration Program (MHIP), a systematic implementation of collaborative care in nine CHCs in two counties (King and Pierce) in western Washington from January 2008 to March 2015. We excluded one of the nine organizations because of insufficient patient enrollment during the study period. All patients with a baseline Patient Health Questionnaire–9 (PHQ-9) score of 10 or higher and at least one follow-up PHQ-9 score were included in the analyses.

Depression improvement was defined as having a PHQ-9 score of less than 10 or a reduction of 50% or greater in PHQ-9 score compared with baseline (referred to as the depression outcome measure) at any point within 6 months of treatment initiation. We also examined whether patients had at least one follow-up contact with the care manager and PHQ-9 assessment in the month following enrollment (referred to as the process-of-care outcome measure). This particular process-of-care metric was selected because it could be readily measured with available registry data for every enrolled patient and had strong support in the literature (5).

We estimated a logistic model of depression improvement at the patient level as a function of the year in which a patient was enrolled (a dichotomous [0/1] indicator for each of years 2 to 8, with year 1 as the reference), a dichotomous [0/1] indicator for each CHC (to capture differences in patient outcomes between CHCs that did not change over time), and interactions between the two groups of dichotomous indicators—years since MHIP implementation and CHCs (to capture potentially varying changes in outcomes over time across CHCs). The model controlled for patients’ demographic characteristics (age and gender), behavioral health conditions or symptoms comorbid with depression at baseline (including anxiety disorders, bipolar disorder, neurocognitive disorders, psychotic disorders, posttraumatic stress disorder, substance use disorders, and suicidal ideation), baseline PHQ-9 score, reasons for MHIP program eligibility, and a dichotomous indicator (0/1) for the presence or absence of a value-based payment program that went into effect in January 2009 (6, 7). The model yielded yearly predicted rates of depression improvement at the CHC level (along with 95% confidence intervals [CIs]) and yearly average rates of improvement across all CHCs. A similar logistic model was estimated at the patient level for the process-of-care measure.

Aggregated data from all 13,362 patients within the eight CHCs included in the analyses demonstrated that depression outcomes improved most rapidly over the first 2 years of MHIP implementation. As indicated by the adjusted analysis, the mean overall rate of improvement increased from 39.5% (CI=33.0–40.5) to 43.1% (CI=40.5–45.8) over that time period. After the first 2 years, depression improvement continued, albeit at a slower rate, until reaching a peak of 45.6% (CI=43.6–47.7) in year 5 (Figure 1). After year 5, the percentage of patients with depression improvement decreased, reaching its final value of 41.8% (CI=35.8–47.7) in year 8. [A table summarizing depression outcomes by CHC is available as an online supplement to this report.]

Individual CHC-level analyses demonstrated substantial variability in depression outcome improvement over time. Two organizations (G and H) consistently performed above the average rate across all CHCs for most of the study period, whereas organization D performed largely below average. Outcomes for organizations A, B, C, E, and F varied between above-average and below-average performance. At two of these five organizations (B and E), performance started considerably below average but improved between years 2 and 3. Both organizations remained high performers for the duration of the study. One organization (A) displayed the opposite trend—starting as a high performer but dipping below average between years 2 and 3. The remaining two organizations (C and F) were inconsistent, moving above and below the average rate several times throughout the study period.

Patient-level process-of-care data also demonstrated, on average, an improvement over the first 3 years of implementation. The adjusted rate of having at least one follow-up contact and PHQ-9 assessment in the month following enrollment increased from 67.4% (CI=61.5–73.3) to a peak of 77.0% (CI=75.4–78.5) over that time period. After the first 3 years, however, improvement leveled off considerably. In years 7 and 8, the overall rate declined, ending at 73.0% (CI=67.7–78.4) in year 8 [see online supplement for CHC-level data on the process-of-care outcome].

We also observed substantial variability across CHCs in the process-of-care outcomes. Organization H performed consistently above average throughout the study period, whereas organization F remained consistently below average. The remaining six organizations (A, B, C, D, E, and G) demonstrated substantial variability.

The above results suggest that average improvement in depression outcomes across the eight MHIP organizations generally followed improvement in process-of-care outcomes. The curves for both types of outcomes (depression and process of care) demonstrated a similar pattern of initial improvement, subsequent plateau, and eventual regression. Time to peak performance differed notably between the two metrics, requiring 3 years for the process-of-care outcome and 5 years for the depression outcome. Both outcome curves showed setbacks in years 7 and 8.

At the level of the individual organization, there was less observed concordance between the process-of-care and depression outcome curves, although a number of similarities were identified. Organization H, for example, was consistently a high performer on both outcomes for the duration of the study. In addition, organizations A, B, and C demonstrated a comparable degree of inconsistency for both outcomes. Organization G, on the other hand, was a consistently high performer on depression outcomes, but its performance with regard to process-of-care outcomes was variable.

Data from this investigation principally suggest that successful implementation of CC may take as many as 2 to 3 years, with additional—but smaller—gains achieved in years 3 to 5. That is not surprising given the complexity of practice change that is required to implement CC. In addition, the results demonstrate that it took organizations longer to reach peak performance in the depression outcome measure than in the process-of-care measure, suggesting that organizations achieve process-of-care gains before realizing maximum depression outcome improvement. Finally, after initial improvements in program outcomes, all organizations encountered setbacks in the later years as they worked to sustain CC and its related clinical practices (8).

Although we do not have definitive data on the challenges in sustaining initial gains, possible explanations include a loss of clinical champions or key program staff over time, competing organizational priorities for quality improvement efforts, and a lack of resource investment in program sustainment (9–11). Regardless of the reasons, the data suggest that organizations should carefully evaluate their progress roughly 2 years after implementing CC and consider implementing organizational learning strategies to optimize performance. These strategies could include single-loop evaluations, which aim to improve the efficiency of current practices, or higher-order analyses, which can be used to reevaluate initial workflow and process goals (12, 13).

Notably, the overall depression improvement observed during this study was modest, with less than half of the patients showing substantial clinical improvement. However, we were encouraged by the improvement in aggregate depression outcomes, which increased by 6.1 percentage points (15.4%) from baseline to peak performance during year 5. These findings are consistent with the delivery of CC among vulnerable, low-income, disabled, or unstably housed individuals such as those served by the eight CHCs in this sample, although they differ from the results of randomized controlled trials of CC. Similar results have been reported from a recent CC implementation in a safety-net health system serving a comparable population in New York City (14).

The results also demonstrate that organizational learning varies considerably for CC, although the study’s observational design limited our capacity to identify specific explanations for this deviation. However, the variations in process-of-care outcomes were generally similar to the variations in depression outcomes, suggesting that sustained attention to core practices, such as timely follow-up after an initial visit (5–7), is important for program success. This result is consistent with previously published literature, which suggests that efficacious dissemination of health services interventions is often impeded by suboptimal adherence to evidence-based processes of care (15). The result also supports the use of pay-for-performance incentives for CC that are based on indicators of operational quality (6, 7).

Limitations of this study included the selection of only one process-of-care variable (rate of timely follow-up) as well as the relatively small number and limited geographic distribution of organizations included in the analysis. These factors attenuate the findings’ generalizability, despite the comparatively large patient-level sample size (>13,000). The small organizational sample size also limits the ability to draw conclusions about organization-level associations between process-of-care and depression outcome variables. Finally, the exclusion of one of the nine participating organizations (because of inadequate patient enrollment) could have introduced bias, given that organizations with limited enrollment may have had more difficulty implementing or sustaining CC.

Quantitative studies with a large sample of organizations may further elucidate associations between process-of-care and depression outcomes in CC. Additionally, qualitative studies could help explain potential factors associated with trends in depression and process-of-care outcomes, as well as variability across organizations.

Participating CHCs experienced the most rapid improvements in depression outcomes in the initial 2 years following CC implementation. Improvement continued at an attenuated rate until peak performance was reached around year 5, after which all observed organizations encountered setbacks. On average, depression outcomes tended to follow process-of-care outcomes. In addition, there was considerable variability between organizations in depression outcomes over time, suggesting that successful implementation of CC requires significant up-front practice change. An ongoing focus on key process-of-care measures (such as patient engagement and follow-up) may be important for organizations to sustain outcome improvement.

The authors thank CHPW and Public Health–Seattle and King County for sponsorship and funding of MHIP and for data on quality of care and clinical outcomes collected in the context of ongoing quality improvement.