A Community-Partnered, Participatory, Cluster-Randomized Study of Depression Care Quality Improvement: Three-Year Outcomes

Depressive disorders are prevalent and a leading cause of adult disability (1), and there are ethnic and racial disparities in depression care quality and outcomes (1–3). In underresourced communities with limited access to services and high stigma in regard to help seeking, individuals often seek support for depression in community-based settings (3,4).

Few data exist on effects of multisector, coalition approaches to implementing depression care quality improvement (QI) across health care and community-based programs in underresourced communities (5,6). Community Partners in Care (CPIC) used a community-partnered participatory research (CPPR) (7,8) framework to examine the added value of a community coalition approach (Community Engagement and Planning [CEP]) versus individual program technical assistance (Resources for Services [RS]) for depression care QI across multiple health, social, and community sectors in underresourced communities (4,8–11). At six-month client follow-up, CEP was more effective than RS in reducing probabilities of poor mental health–related quality of life, behavioral health hospitalizations, and multiple risk factors for homelessness; increasing physical activity; reducing use of specialty medication visits; and increasing use of primary care and community-based depression services (8). At 12 months, primary longitudinal analyses demonstrated evidence of reductions in poor mental health–related quality of life and behavioral health hospitalizations, but significance levels for these findings were sensitive to statistical modeling techniques (10).

Prior depression care QI studies based on collaborative care identified persistent improvements (12–14); this extension study tested the hypothesis that CEP would show continuation of the 12-month improvements at three-year follow-up for the overall CPIC client sample, two years after study intervention support ceased. Overall, we considered the analyses as exploratory but of potential policy interest, given initiatives promoting collaboration and coordination of patient care between health care and non–health care settings under expanded Medicaid (15) and this study’s relatively unique focus on coalition and noncoalition approaches to depression intervention (6).

Methods

Study Design

Data are from the three-year client follow-up extension study for CPIC (4,8–10). CPIC was a cluster-randomized trial implemented by using CPPR (7,8), which supports community and academic partners in research coleadership through two-way knowledge exchange. CPIC was fielded in South Los Angeles and Hollywood–Metro Los Angeles, with a population of two million and high representation of racial-ethnic minority groups (8). Study design and procedures have been described elsewhere (4,8–11). Funded in 2007, prior to required inclusion of delivery interventions in trial registries, CPIC was not considered a clinical trial by the National Institutes of Health. After participants were enrolled but before the study’s three-year continuation, the study was registered (ClinicalTrials.gov NCT01699789). Procedures were approved by institutional review boards of RAND and participating agencies, and written consent was obtained from participants.

Participants and Randomization

Using county lists and partner nominations, we identified agencies offering services identified by community members as relevant to depression (mental health specialty services, primary care, public health services, substance abuse treatment services, social services, faith-based services, park community centers, hair salons, and exercise clubs). Eligible agencies offered services to adults or parents of children and were financially stable—that is, expected to operate for one to three years—and were selected to oversample four subgroups prioritized by participating communities (homeless persons, seniors, African Americans, and persons served in substance abuse treatment programs). Agency and program enrollment occurred from November 6, 2008, through August 17, 2010. In 60 potentially eligible agencies that had 194 programs, 133 programs were confirmed as potentially eligible and randomly assigned (RS, N=65; CEP, N=68). Site visits after randomization to confirm eligibility and finalize enrollment were conducted by staff blinded to assignment; 20 programs were ineligible, and 18 refused. Therefore, 95 programs from 50 consenting agencies enrolled (RS, N=46; CEP, N=49). Zip code–level census tract data indicated that participating and nonparticipating programs were comparable in neighborhood demographic characteristics (p>.10) (8,16).

Within programs, staff blinded to the intervention screened clients for eligibility in waiting rooms or at events over two to three days for each program. Eligibility was based on being at least 18 years of age, speaking English or Spanish, providing contact information, and having depression symptoms (as measured by a score of ≥10 on the eight-item Patient Health Questionnaire, modified [PHQ-8]). Between March 25 and November 18, 2010, staff approached 4,649 adults to assess their eligibility; 4,440 (96%) in 93 programs agreed. Of these, 1,322 (30%) were eligible, and 1,246 (94%) of the eligible adults consented. Of these, 981 (79%) completed baseline telephone surveys conducted by staff blinded to the intervention (April 27, 2010, to January 2, 2011). Enrolled participants were invited to complete six- and 12-month follow-ups, except for those whose previous survey status was in one of the following categories: ill or incarcerated, unable to contact, or deceased at previous survey participation. These results were reported elsewhere (8,10).

For the extension study, between January 14, 2014 and October 14, 2014, we attempted to contact 1,004 participants from 89 programs who were eligible for three-year surveys—that is, enrolled; completed baseline, six-month, or 12-month follow-ups; and neither refused follow-up nor were reported deceased at prior surveys. Of the 1,004 clients, 600 (60%) participated (RS, N=293; CEP, N=307), 24 (2%) were deceased (RS, N=13; CEP, N=11), ten (1%) refused (RS, N=7, CEP, N=3), three (<1%) were ill or incapable (RS, N=2; CEP, N=1), and 367 (37%) were not reached (RS, N=181, CEP, N=186). [A figure showing study recruitment is included in an online supplement to this article.] The mean postbaseline follow-up interval was 1,321 days (median=1,314).

Interventions

Both CEP and RS interventions encouraged use of depression care QI tool kits, based on collaborative care, including manuals for cognitive-behavioral therapy for depression (CBT), clinician assessment and medication management, care management (that is, depression screening, care coordination, patient education, and outcomes tracking), lay health worker support, and team leadership (12,17–20). Tool kit materials were designed so that all staff with direct client contact (paid, volunteer, licensed, and nonlicensed) could benefit, depending on their role. Tool kits were provided in hard copy, on flash drives, and via a Web site (http://www.communitypartnersincare.org/community-engagement/cep) and introduced in kickoff conferences prior to participant enrollment. Providers received an orientation and review of modules relevant to their discipline. [A table in the online supplement presents more information about the interventions and the training.]

RS used a train-the-trainer implementation approach between December 2009 and July 2010, which offered site visits and 12 webinars of 90 to 120 minutes each covering core principles and skills, along with access to all versions of the tool kits. Expert trainers included psychiatrists, a nurse care manager, a CBT trainer, and QI and community engagement specialists with support staff.

CEP used a coalition implementation approach between December 2009 and July 2011, and invited program administrators across all sectors within a given community to attend two-hour meetings biweekly for four months. The coalitions followed a workbook that outlined intervention goals, principles, and potential sessions and provided information on resources. Planning meetings were co-led by academic and community members. Main activities included reviewing and adapting tool kits to community priorities and culture; developing plans for staff training as a network; training local staff to co-lead QI training sessions with experts; and developing a written plan for QI training, implementation, and maintenance. Each CEP council was provided $15,000 and tool kits (equivalent value of RS resources) to support planning. Final plans featured half- or full-day conferences, follow-up training sessions at sites, telephone and webinar supervision for CBT and case management, and innovations such as alternative medicine training, provider self-care, and resiliency psychoeducation classes led by laypersons to introduce CBT concepts. CEP plans as implemented, compared with RS expert training, led to higher rates of participation in depression QI training by eligible staff (11). Except for one agency with a common waiting room for RS and CEP, lists of enrolled clients were provided to CEP but not RS program administrators for safekeeping in a locked file, consistent with the design for CPIC (12).

All enrolled CEP and RS clients were instructed that they were free to discuss study participation with their provider. All screened participants were given a health and community services resource guide. Clients were free to access any services or programs they wished, regardless of intervention status.

Data Sources and Outcomes

Staff blinded to randomization status administered surveys by telephone to clients at baseline, six months, 12 months, and three years.

Primary outcomes were poor mental health–related quality of life, defined by a 12-item mental composite score (MCS-12) ≤40 and, according to the original protocol, depression as indicated by a score of ≥10 on the PHQ-8 (21). The two community-prioritized outcomes, developed under a prespecified community input process, were physical functioning, defined by a 12-item physical composite score (PCS-12) and behavioral health hospitalization nights—that is, hospitalization for a mental or substance use disorder as measured by total nights hospitalized in the prior six months. This measure differs from prior hospitalization measures (any, any with ≥ nights) from six- and 12-month analyses, due to low hospitalization frequency at three years. We developed counts of emergency room visits, outpatient visits to primary care providers (PCPs), outpatient services by PCPs for depression, mental health outpatient visits, outpatient visits to a substance abuse treatment agency or self-help group, social services for depression, calls to a hotline for a substance abuse or mental health problem, and days on which a visit to a self-help group for mental health were made. We also measured in the prior six months any outpatient visits to services for depression in the health care sector (primary care or public health, mental health, or substance abuse treatment program) and in the social and community sector (social services program, faith-based program, park community center, exercise program, hotline calls, and other), as well as visits to faith-based services for depression. We developed indicators of use of any antidepressant, any mood stabilizer, any antipsychotic, and any visit for depression (8,12). We developed an indicator of having at least minimally adequate depression treatment, defined as having at least two months of antidepressant use or at least four depression visits across mental health specialty and primary care sectors in six months (12).

Analyses

Models of longitudinal outcomes models using baseline, six-, and 12-month data were previously published (10). For this study, we focused the primary analysis on client status at three-year follow-up, or 2.5 years after the end of the active intervention period, adjusting for baseline covariates. This approach permitted use of multiple imputation as well as use of response weights to account for attrition.

We conducted intent-to-treat analyses with intervention status as the main independent variable, using linear regression for continuous variables, logistic regression for binary variables, or Poisson regression for count variables, adjusted for age, sex, three or more chronic general medical conditions, education, race-ethnicity, family income below the federal poverty level, 12-month alcohol abuse or use of illicit drugs, 12-month depressive disorder, community, and baseline status of outcome. We used item- and wave-level imputation for missing data (22,23) to adjust findings to the three-year eligible sample (1,004 eligible minus 24 deceased=980). We used weights to account for nonenrollment (24,25) and attrition. [Additional information about the analytic procedure is available in the online supplement.]

All analyses used Taylor series linearization with SUDAAN release 11.1 to estimate variability, accounting for clustering (clients within programs), weighting, and multiple imputation. Significance of comparisons was assessed by using contrasts among regression coefficients. Results of regression models are presented as between-group differences for linear regression, odds ratios (ORs) for logistic regression, and incidence-rate ratios (IRR) for Poisson regression with 95% confidence intervals. We illustrated average results adjusted for covariates by using standardized predictions generated from fitted models (25). We used two-sided tests with p<.05 for statistical significance. In the proposal, we designed the extension study anticipating a sample of 650 for group differences of ≥11%−12% in comparing proportions and standardized effects of ≥.23– .25 for comparing means, to achieve 80% power with alpha of .05 (two-sided) and intraclass correlation coefficient of .01–.04.

Given multiple secondary outcomes (26), we built on the false-discovery-rate (FDR) framework (27) as extended by Yekutieli and Benjamini (28) and used both standard and FDR-adjusted p values (pFDR) in interpreting results across a large number of regression analyses (26–28). Results with pFDR <.05 are viewed as convincing evidence of a difference, and higher pFDR thresholds are considered as suggestive evidence. We separately calculated pFDR for the two primary outcomes, the two community-prioritized outcomes, service use in the health care sector, service use in the social and community sector, medication use, and summary utilization indicators.

We conducted a sensitivity analysis using all waves of data (baseline, six months, 12 months, and three years) without response weights, adjusting for the same set of baseline covariates as in the main analysis. We specified a spline model, with a linear segment between baseline and the first follow-up for initial improvement and another linear segment for the subsequent follow-ups; the two linear segments were specified to join at the first follow-up. In analyzing continuously scaled PCS-12 as the dependent variable, we employed a three-level, mixed-effects regression model by using SAS PROC MIXED. To account for the intraclass correlation due to the multilevel structure, we specified random effects at the clinic level, including random intercepts at the program level and a spatial power covariance structure at the client level to account for the unequal spacing of waves (29). Initial explorations of three-level, random-effects logistic models by using SAS PROC GLIMMIX for binary outcomes yielded unstable estimates for program-specific random effects. We utilized a generalized estimating equation framework with logistic regression models for binary outcomes and Poisson models for count data by using SAS PROC GENMOD, specifying exchangeable correlation at the program level. From the estimated spline model, we developed a contrast involving a linear combination of coefficients to test intervention effects at each end point (baseline, six months, 12 months, and three years) and tested differences between intervention groups in change from baseline to six months, 12 months, and three years.

To provide context for anticipated dampening of intervention effects over time, we report information from the main implementation periods (baseline, six months, and 12 months) on the extent to which clients were exposed to elements of their assigned intervention as well as to potential cross-intervention contamination (that is, exposure to the intervention not assigned).

Results

Baseline Characteristics

Clients had similar characteristics across both intervention groups (Table 1). Most clients were from racial-ethnic minority groups, with family incomes below the federal poverty level. Most had a 12-month depressive disorder and multiple chronic general medical conditions.

Three-Year Outcomes

There were no significant differences between the effects of CEP and RS on poor mental health–related quality of life and depression as measured by the PHQ-8 (Table 2). For the community-prioritized outcomes, which were identified by a prespecified participatory process, statistically significant, modest effects were noted for CEP versus RS in improving mean PCS-12 scores (difference in scores=1.2, p=.022) and reducing the number of nights hospitalized for behavioral health reasons (IRR=.2, p=.020); the differences remained significant with a pFDR <.05. For other utilization outcomes, CEP participants were significantly more likely than RS participants to use faith-based depression services (p=.006, pFDR=.023). The results suggested that CEP participants were also more likely than RS participants to use community depression services (p=.042, pFDR=.127) and mood stabilizers (p=.049, pFDR=.147), but no significant differences were found by intervention status for other utilization outcomes.

Sensitivity analyses found that the intervention effect on end status (three years) was significant at p<.05 for the PCS-12 score, number of nights hospitalized for a behavioral health reason, use of any faith-based services for depression, and use of mood stabilizers; however, the differences between intervention groups in change from baseline to three years were either borderline significant (for example, p=.052 for hospital nights) or not significant. For use of any community services for depression, neither end status nor change from baseline was significant in the three-year longitudinal analysis. [Tables presenting results of the sensitivity analyses are included in the online supplement.]

Intervention Exposure and Contamination

Table 3 presents data on the distribution of use of depression services stratified by intervention group and based on the service location reported by the client for depression services at baseline and at six- and 12-month follow-up. Across survey periods for both CEP and RS, the percentage of clients with any exposure in that period to intervention elements associated with their screening site was about 50% at baseline, 40% at six months, and 30% at 12 months. Levels of exposure to services at a site assigned to the other intervention were somewhat higher for six months before baseline for CEP than for RS (19% versus 12%) mostly before client intervention exposure, but levels were modest across interventions at six and 12 months (about 10%).

Discussion and Conclusions

This is the first long-term evaluation of outcomes for depressed clients from health- and community-sector programs that either participated in a community coalition–based approach to depression care QI or received technical assistance to individual agencies for depression care QI. To examine three-year client outcomes while accounting for attrition (a Cochrane Collaborative review noted that failure to account for attrition is a design limitation [6]), we used covariate-adjusted end status analyses incorporating attrition weights and multiple imputation. Using this approach, we found no main intervention effect on primary outcomes (depressive symptoms and mental health–related quality of life). We found statistically significant but modest effect sizes on two community-prioritized outcomes (physical health–related quality of life and behavioral health hospitalization nights), both favoring CEP. Improvements in physical health with CEP might be attributable to earlier CEP effects on mental health–related quality of life, physical activity, or social risk factors (8). A reduction in behavioral health hospitalization nights with CEP is consistent with findings at six months and 12 months (8,10). This long-term modest effect on hospitalization nights could reflect effects of CEP on improving physical health–related quality of life or effects of increasing alternative community supports, given the increased use of faith-based depression services by CEP participants at three years. Given the clinical complexity of participants’ conditions, evidence of greater use of mood stabilizers could also be a factor in reduction of hospitalization nights, but this finding was sensitive to analytic approach and less significant with FDR adjustment for multiple secondary outcomes. Whether such increases in medication use may be a factor in reduced hospitalization with CEP is a potential area for future research. The findings reflect outcomes two-and-a-half years after the active intervention period ended, reflecting the extent to which the systems sustained intervention elements or clients learned from their initial intervention exposure.

Our analyses of intervention exposure did not yield strong signs of cross-intervention contamination during the active intervention period postbaseline, suggesting that diminishing of intervention differences over time might be better explained by there having been only moderate levels of sustained exposure to assigned intervention elements. During the active intervention period, about 20%−25% of clients did not use any depression services. This would suggest that expectations for intervention effects at three-year follow-up would be for modest differences at best. Given the social and clinical vulnerability of participants, some subpopulations may show more robust long-term intervention effects, an issue for future research.

Limitations include inclusion of only two urban communities, use of self-report measures, moderate follow-up rates and community-sector sample size, and absence of a usual-care control group, which was considered unethical by the partners given known access disparities (2,3). Also, the study identified CPIC clients to administrators in the CEP but not RS conditions (for locked files). The identification of enrolled participants to CEP programs could be part of why CEP had a stronger initial effect on mental health–related quality of life.

Limitations include inclusion of only two urban communities, use of self-report measures, moderate follow-up rates and community-sector sample size, and absence of a usual-care control group, which was considered unethical by the partners given known access disparities (2,3). Also, the study identified CPIC clients to providers in the CEP but not RS conditions. The identification of enrolled participants to CEP programs could be part of why CEP had a stronger initial effect on mental health–related quality of life.

Overall, the primary analysis suggested a potential for modest longer-term effects with CEP versus RS—not on the primary outcomes but on the secondary, community-prioritized outcomes of physical health–related quality of life and behavioral health hospitalization nights. It is likely that sustained differential gains from CEP would require continued active intervention support through more sustained system change across whole communities, which is what might occur with whole-community assignment rather than assignment that splits community organizations from their natural partnerships, or policy support, such as accountable community initiatives for underresourced populations. Given that this is one of the first rigorous studies in the international literature of the added value of a coalition approach to health for racial-ethnic minority communities (6), replication of the study would be valuable, as would efforts to sustain intervention support and delivery, which might more robustly improve outcomes. We note that a similar coalition model was used to support QI in depression services and outcome recovery following Hurricane Katrina in New Orleans (20,30), which supports the feasibility of achieving coalition building in practice, with the equitable inclusion of patients, families, community members, and providers as coleaders.