Assessing the Impact of Different Depression Treatment Success Metrics on Organizational Performance

For years, the AIMS Center has supported practices implementing the collaborative care model (CoCM), an evidence-based practice for the treatment of common behavioral health problems in medical settings. Part of this support has included development and dissemination of the Care Management Tracking System (CMTS) (13), a specialized treatment registry that records contact details with patients and facilitates measurement-based care (e.g., tracks PHQ-9 depression scores over time). With the written consent of participating organizations, we compiled a data set of 36,887 adult patients with depressive symptoms who were treated in one of 145 primary care clinics (across 33 organizations) and who had depression outcomes tracked using CMTS between 2008 and 2018. Health care organizations and clinics were located across nine states; approximately 83% (N=120) of the clinics were in urban areas (as defined by the Federal Office of Rural Health Policy), and 64% (N=93) were federally qualified health centers (FQHCs). Analysis of this deidentified data set was granted exemption status by the University of Washington Institutional Review Board (ID STUDY00005907).

Our analysis, which was conducted at the organization and clinic levels, included all patients ages ≥18 who had at least two documented PHQ-9 assessment scores: one at baseline and one or more within the following 12 months. PHQ-9 scores closest to and within 30 days of 3, 6, and 12 months from baseline were extracted and noted as the scores for those respective time points. Incorporating these criteria, we created three overlapping and nonexclusive time cohorts (3, 6, and 12 months), each including patients who had follow-up scores at that time point. Of note, these time cohorts were not mutually exclusive and were not a single cohort being followed longitudinally over time. For example, an included patient could have baseline PHQ-9 and follow-up scores at both 3 and 12 months. Such a patient would be included in the 3- and 12-month time cohorts but not the 6-month time cohort.

Further inclusion and exclusion criteria were applied at the clinic level. In the 3-, 6-, and 12-month time cohorts, a clinic’s data were included if at least one of its patients had a recorded PHQ-9 score. At 3, 6, and 12 months, 135, 130, and 113 clinics met inclusion criteria, respectively. This finding corresponded to 33, 33, and 32 organizations as well as 19,862, 11,303, and 3,308 patients, respectively. Missing race and gender data were imputed at the clinic level. (For baseline characteristics of organizations and clinics in each of the three time cohorts, see the online supplement to this report.)

First, mean improvement rates defined by the eight depression response and remission metrics (and weighted by clinic size) were calculated for the 3-, 6-, and 12-month cohorts. Next, to analyze the impact of metric choice on comparative organization-level performance, all 33 organizations in the 6-month time cohort were ranked according to their improvement rates across the eight metrics. We calculated ranks by using empirical Bayes predictions from a random-intercept logistic regression model with reliability adjustment, a strategy that has been used with other health outcome rankings (14). We chose to calculate rankings using this random-effects, model-based approach (as opposed to using raw values or direct sample means) for two reasons: it reduced the impact of chance-driven uncertainty from small samples in certain groups because the between-groups variability was estimated by using data from all groups, and it made the rankings more reproducible over time (14). Finally, Spearman’s rank-order correlation coefficients of predicted ranks from different metrics for the 6-month time cohort were calculated.

Across all eight metrics, the 12-month cohort had higher rates of metrics indicating treatment success than the 3- and 6-month cohorts. Additionally, rates within time cohorts varied substantially by response or remission definition. In the 3-month cohort, for example, depression response rates ranged from 32% (metric 7) to 51% (metric 4), whereas the remission rate was 22% (metric 8). Additionally, response rates appeared to form two clusters: metrics 2, 3, and 4 were similar (ranging from 48% to 51% in the 3-month cohort), as were metrics 1, 5, 6, and 7 (ranging from 32% to 34% in the 3-month cohort). Similar ranges were observed for the 6- and 12-month time cohorts. All treatment response and remission rates, in addition to mean initial and time cohort follow-up PHQ-9 scores, are presented in Table 1. (For the 6-month time cohort correlation matrix that was calculated with Spearman’s rank-order correlation coefficient, see the online supplement.) All pairwise rank-order correlation coefficients were positive, with mean=0.86, and only three of the 28 <0.75. Metric 2 was least correlated with the others. These results broadly demonstrate that across metrics, organization-level rank orders were highly correlated.

In this analysis of PHQ-9 scores from 33 organizations and three time cohorts across nine states, we found that choice of depression treatment success metric led to markedly different rates of improvement. Response rates for the 3-month time cohort ranged from 32% to 51% depending on choice of metric, whereas remission was 22%. At the same time, we found that organization-level rank orders defined by performance on eight different metrics were uniformly positively correlated (and largely >0.75). These findings lead to two primary conclusions.

First, it is of paramount importance for organizations to be compared with benchmarks or with one another using the same depression response or remission metric. For example, if organization A uses metric 4 (≥50% decrease from baseline or score <10) and organization B uses metric 1 (≥50% decrease from baseline), their improvement rates cannot be meaningfully compared. We would expect organization A, with a compound metric including an “or” logical operator, to have the higher response rate even if the true rates of improvement were equivalent. This finding also has similar implications for research, in which consistent PHQ-9 definitions for depression treatment response, in particular, are lacking.

Second, our Spearman’s rank-order correlation coefficient findings suggest that the eight metrics all largely appear to tell the same story with regard to depression treatment response and remission. Of note, metric 2, although still positively correlated with the other metrics, was positively correlated to a lesser extent. One possible explanation is that it is the only assessed metric defined solely by an absolute change in PHQ-9 score over time. Depending on the initial PHQ-9 score, treatment success metrics defined in this way can be more or less challenging to achieve relative to those defined multiplicatively (i.e., ≥50% decrease from baseline). Citing similar reasoning, a recent study advocated for the use of multiplicative PHQ-9 metrics over commonly used threshold metrics (i.e., score <5) (15).

Regardless of which metric is chosen, however, similar organizations in this study tended to be ranked favorably and were deemed to be high performing. This finding suggests that inter- and intraorganization consistency of metric use may be more important than which specific metric is chosen. In other words, organization-level depression outcome measurement and research efforts should, above all else, strive to compare the metric equivalent of “apples with apples.”

At the same time, our result show that differently defined response rates tend to cluster across metrics: metrics 2, 3, and 4 were similar, as were metrics 1, 5, 6, and 7. This finding suggests that when comparing organizations using different metrics within each of these clusters, there is perhaps less concern about the impact of metric choice. Furthermore, given that the eight metrics in this investigation are highly correlated, one possible application of our findings could be to provide a theoretical “conversion factor” for different metrics. For example, on the basis of the 3-month cohort data in Table 1, one could expect that a response rate defined by metric 2 (48%) would be roughly 1.5 times greater than that of metric 5 (32%) with no true differences among patients’ clinical statuses.

The findings in this investigation are limited by the real-world nature of this AIMS Center data set, which had missing information, including follow-up PHQ-9 scores and patient demographic characteristics. This limitation is evidenced by the comparative sizes of the time cohorts, with the 3-, 6-, and 12-month cohorts including roughly 50%, 30%, and 10% of the full sample, respectively. Although these findings were not surprising, they highlight the challenges associated with consistent longitudinal outcome ascertainment in real-world outpatient settings. Of note, we were able to reduce the impact of missing gender and race data in this investigation through imputation methods. Additionally, our sample of 33 organizations across multiple states remains one of the largest and most diverse real-world CoCM implementation data sets to date. Approximately half of included patients were persons of color, and two-thirds of clinics were FQHCs. Additionally, our relative lack of patient-level exclusion criteria makes our results externally valid and generalizable.

Our findings demonstrate that the choice of PHQ-9 response or remission metric substantially affects observed treatment improvement rates. Furthermore, organization-level rankings for depression response and remission vary depending on choice of metric, but their rankings are highly correlated. We therefore conclude that in organization-level PHQ-9 response or remission rate comparisons, metric consistency is imperative, whereas the specific metric chosen is of secondary importance.

The authors thank the 33 organizations that agreed to share their deidentified Care Management Tracking System registry data for this work. Through their generosity, a collaborative care implementation data set of more than 35,000 adult patients was created, providing a tremendous opportunity to help study and improve care for depression.

The authors report no financial relationships with commercial interests.