Assessing the Long-Term Effectiveness of a Behavioral Health Home for Adults With Bipolar and Psychotic Disorders

A BHH was established in September 2015 at a community-based behavioral health clinic in an urban, safety-net health system caring for about 140,000 patients annually. Patients with primary psychotic or bipolar disorders (plus any antipsychotic prescription) receiving care at the clinic were automatically enrolled in the BHH on September 1, 2015. Referrals within the health system were accepted on the basis of set criteria for psychiatric diagnosis, medical risk, and care coordination needs. The BHH sought to increase access to primary care through intensive referrals and service colocation, but neither primary care visits, nor any other kind of visit for GMH care, were required for participants. (For key BHH components, see part 1 in an online supplement to this article.)

The BHH implementation start date was September 1, 2015; our observation period included service encounters 1 year preintervention (September 1, 2014–August 31, 2015) and 3.5 years postintervention (September 1, 2015–February 28, 2019).

Our study sample consisted of all patients receiving care in the health system during the observation period before October 31, 2015, with diagnoses of either primary psychotic disorders or bipolar disorder accompanied by an antipsychotic medication. Data from patients whose BHH enrollment was documented in the first 2 months of enrollment were included in this analysis (N=413). Those enrolled after this date (N=304) were included in a sensitivity analysis (see part 2 in the online supplement). Our control group consisted of patients from the same health system diagnosed as having serious mental illness who never enrolled in the BHH and who had at least one specialty behavioral health visit in the pre- and postintervention periods.

Electronic health record data were accessed for individual-level demographic characteristics (age, sex, race, language spoken at home, marital status), medical information (diagnosis, prescriptions, health insurance type), and health encounter information (date and type, lab values). Area-level variables (percentage foreign born, percentage living in poverty, percentage of female-headed households, and percentage with less than a high school education) were sourced from census data on individual zip codes or were imputed for patients with missing zip codes by using the “mi impute” command in Stata (35).

We employed multigroup interrupted time series analysis (ITSA), which compares pre- and postintervention trends from longitudinal data between the treatment sample and a representative control sample. The control sample adjusted for exogenous shocks that could influence outcomes of interest.

To adjust for differences between samples, we used inverse propensity weighting (IPW) to weight individuals from the control sample based on how they resembled the BHH sample (36). Propensity scores were calculated by using individual and area-level characteristics. We did not include service use variables in the propensity score to avoid potential bias due to regression to the mean (37). Control group members with scores lower than the lowest value in the BHH sample were dropped. To estimate average treatment effect on the treated sample, standardized mortality ratio weights (38) were generated from scores and applied to individual observations for both BHH patients and control group members.

Autocorrelation was assessed with the Cumby-Huizinga test. When first-order autocorrelation was detected, Newey-West standard errors were replaced with Prais-Winsten standard errors. Newey-West standard errors were used to account for heteroskedasticity and autocorrelation. Monthly indicators were added to our models to control for seasonal variation. All analyses were conducted in StataMP, version 15.1 (39).

Our ITSA followed Linden (40). We applied individual propensity scores to weight the number of monthly service encounters for each individual. The sample average of these values (the sum of individual weighted values divided by the sample size) was the dependent variable in the ITSA regression model.

We considered six visit types as dependent variables: primary care, ED, GMH inpatient, GMH outpatient (i.e., other types of specialty GMH care), behavioral health inpatient, and behavioral health outpatient. Our primary outcomes of interest were, first, the difference in predicted monthly visits immediately after the intervention start date (“level shift”). Level shifts represent abrupt changes in patient behavior that occurred as soon as the intervention began. Second, we considered how the rate of monthly visit counts changed (i.e., whether counts were falling or rising, and by how much) over the 3.5-year follow-up period compared with the preintervention period (“trend shift”). A negative trend shift would occur, for instance, if utilization declined at a faster rate than in the preintervention period. Level shifts represent immediate intervention impacts, whereas trend shifts capture the intervention’s impact on outcome trajectories over time. Our ITSA model analyzed primary outcomes through a difference-in-difference (DID) framework, comparing shifts between BHH enrollees versus control group members.

Secondary outcomes included the average individual-level change in cardiometabolic measures between the pre- and postintervention period for the two samples. We conducted this analysis for all individuals with at least one value recorded in the pre- and postintervention period and, as a sensitivity analysis, among only those who had a categorically high value recorded in the preintervention period (greater than 5.6% for HbA1c and greater than 190 mg/dL for low-density lipoprotein [LDL] cholesterol).

Subgroup analyses assessed how the intervention’s impact on the assessed variables varied by race-ethnicity and language spoken at home, both between the BHH and control groups and within the BHH group (no IPW applied). This study was approved by the institutional review board of the Cambridge Health Alliance.

The demographic and clinical characteristics of 413 patients who enrolled in the BHH between September 1 and October 31, 2015, were compared with those of 1,929 control group members. Table 1 shows differences by sample before IPW. More BHH patients than control group members had primary psychotic diagnoses (86% vs. 51%), and fewer had bipolar diagnoses (31% vs. 61%). BHH patients were also more likely to identify as non-Hispanic Black (23% vs. 16%) and more likely to be a native English speaker (91% vs. 80%).

After we conducted IPW, all characteristics were balanced across the two samples (Figure 1 and part 3 of the online supplement). For the primary analyses, 61 control group members were excluded because their propensity scores were outside the range of that for BHH patients (i.e., scores were not good for comparison).

Figure 2 displays data from the ITSA for our primary outcomes. Table 2 presents the multigroup ITSA estimates for utilization shifts between the pre- and postintervention period among the treatment and control groups. (All preintervention values are available in part 4 of the online supplement.)

In the year before BHH enrollment, average primary care utilization was 0.36 visits per month (falling at rate of 0.0057 fewer visits per month) for eventual BHH enrollees and 0.29 visits per month (falling at a rate of 0.0022 fewer visits per month) for the control sample. Postimplementation, BHH patients experienced an immediate level shift of 0.20 more primary care visits per month, compared with 0.0037 more visits per month among the control sample, resulting in a level shift DID of 0.18 more primary care visits per month (p<0.01) for those enrolled in BHH. BHH enrollment was also associated with an immediate decrease in ED visits (DID level shift=–0.031, p<0.01) and increase in specialty GMH outpatient visits (DID level shift=+0.055, p<0.01). Although both BHH and non-BHH participants showed a drop in behavioral health inpatient visits after BHH program implementation, BHH participants showed a shallower drop, suggesting a relative increase (DID level shift=+0.018, p<0.05). If level shifts were applied equally across the first year (not just the first month, as estimated by our model), BHH participation would be associated with 2.16 more primary care, 0.36 fewer ED, 0.66 more GMH outpatient, and 0.036 more behavioral health inpatient visits during that year.

When considering the intervention’s impact on the trajectory of outcomes (trend shifts), the rate of change among BHH patients for behavioral health outpatient visits was 0.024 fewer during the postintervention period compared with the preintervention period; this measure was estimated at 0.0090 fewer in the control sample. Within the DID model, BHH enrollment was associated with 0.016 fewer behavioral health outpatient visits every month after the intervention began (DID trend shift, p<0.01). Although GMH inpatient and outpatient visits were decreasing over time for both groups, relative to the control group, BHH enrollment was also associated with small increases in the monthly rate of GMH inpatient visits (DID trend shift=+0.00074, p<0.01) and GMH outpatient visits (DID trend shift=+0.0044, p<0.01). If trend shifts were applied equally across the 3-year postintervention period, BHH participation would be associated with 0.19 fewer behavioral health outpatient, 0.009 more GMH inpatient, and 0.05 more GMH outpatient visits for every year of observation. Shifts not listed here were not statistically significant and are presented in Table 2.

When restricting the sample to non-White patients and those with English as a second language, intervention effects were generally smaller or nonsignificant. The DID level shift in primary care visits for BHH enrollees after the intervention was 0.184 in the entire sample, but this value decreased to 0.167 when the sample was restricted to non-White patients (BHH patients, N=139; control group members, N=698); no associated level shift in primary care visits was detected when the entire sample was restricted to non-English speakers (BHH patients, N=39; control group members, N=382).

Compared with non-White BHH enrollees (N=139), White enrollees (N=274) experienced minor differences only for behavioral health outpatient visits. Among BHH enrollees, native English speakers (N=374) experienced a greater immediate increase in primary care visits and fewer ED visits (both immediately and over time) than did nonnative English speakers (N=39). Results for the subgroup analyses were exploratory and should be interpreted with caution because of small sample sizes (see part 5 of the online supplement).

To assess how BHH enrollment affected cardiometabolic measures, we restricted our sample to participants who had at least one laboratory test value in the pre- and postintervention period (N=186 BHH patients and 498 control group members for HbA1c; N=174 BHH patients and 475 control group members for LDL cholesterol). When comparing average laboratory test values from the pre- and postintervention periods between BHH patients and control group members, we detected no statistically significant change in LDL levels, but BHH participation was associated with 0.29 fewer percentage points for HbA1c (p<0.05). When restricted to those with high values in the preintervention period (504 control group members and 148 BHH patients for HbA1c; 10 control group members and six BHH patients for LDL), BHH participation was associated with 0.47 fewer percentage points for HbA1c (p<0.05). Data from all 10 control group members in the subsample with high LDL values were dropped because the propensity scores of these members were too low. An exploratory analysis suggested higher cardiometabolic screening rates among BHH participants compared with the control population (see part 6 of the online supplement).

The follow-up period of our analysis (3.5 years) was longer than other analyses on BHHs and allowed us to distinguish between immediate and long-term effects of the intervention (level vs. trend shifts). In relation to a prior analysis of this BHH based on a 1-year follow-up (34), the present analysis confirmed that the increase in primary care visits was sustained for the 3.5-year follow-up period and that behavioral health outpatient visits began to decrease over time. The estimates produced from our study benefited from the quasi-experimental techniques of IPW and multigroup ITSA. We incorporated area-level socioeconomic characteristics into our propensity score model, which helps estimate the BHH program impact by accounting for variation in outcomes that may be due to area-level factors.

This study had several limitations. First, all individuals needed at least one encounter in the health system in the pre- and postintervention periods. For the control group, this encounter was for a behavioral health appointment. Individuals may have recorded this encounter early in the postintervention period before leaving the health system’s catchment area. However, results of qualitative sensitivity analyses in which individuals in the control groups required an encounter in 3 separate years postintervention did not influence our primary conclusions (see part 7 of the online supplement). Results were similar in a sensitivity analysis that included later BHH enrollees, although BHH participation for later enrollees was associated with smaller shifts in primary care and behavioral health outpatient utilization compared with shifts among early enrollees (see part 2 of the online supplement). Second, our study mimicked an intent-to-treat analysis. BHH participants were included only if enrolled when implementation began. We did not capture differences for patients enrolled later, and we included in our analyses patients who might have disengaged from the program. Third, subgroup analyses were exploratory because of smaller sample sizes. Fourth, our propensity score model did not control for potential selection bias based on preintervention levels of behavioral health inpatient and outpatient visits. However, not including preintervention levels in the model prevents bias from regression to the mean, which may be strong precisely because of preintervention level differences. Finally, we acknowledge an abrupt decrease in behavioral health inpatient visits for the control population when the intervention began. This decrease may reflect general efforts across the health system to reduce behavioral health hospitalizations in the population. Unexpected changes in real-world data highlight the need for multigroup comparisons when evaluating complex interventions in real-world settings.