Trends in Results of HBIPS National Performance Measures and Association With Year of Adoption

Specifications for the HBIPS measures were developed by using the Joint Commission’s standard methodology (7). A technical advisory panel was established to develop draft measures, which were posted for public comment and revised on the basis of the comments. The revised measure specifications were pilot-tested in hospitals to evaluate implementation feasibility and data collection reliability. On the basis of test results, the final measure specifications were refined and published.

The measures address the following topic areas: percentage of patients screened at admission for risk of violence, substance use, psychological trauma, and psychological strengths (HBIPS 1); ratio of restraint hours per 1,000 patient hours (HBIPS 2); ratio of seclusion hours per 1,000 patient hours (HBIPS 3); percentage of patients discharged with documented treatment justification for two or more antipsychotic medications (HBIPS 5); percentage of patients discharged with a continuing care plan (HBIPS 6); and percentage of patients discharged with a continuing care plan that was transmitted to the next level of care (HBIPS 7).

HBIPS data were collected quarterly from hospitals that treat psychiatric inpatients (freestanding psychiatric hospitals and acute care hospitals with psychiatric units) and that are accredited by the Joint Commission. Clinical data were abstracted from each eligible patient’s medical record by specialists using data element definitions and collection protocols (8). For the percentage measures, record sampling was permitted for hospitals with greater than 44 eligible patients in a given quarter (62% [N=552] of hospitals submitted sampled records). Sampling was not permitted for the ratio measures (restraint and seclusion) because they require the calculation of total inpatient psychiatric days for each facility as well as total seclusion and restraint time. Sampling details are available in the Specifications Manual for Joint Commission National Quality Measures (7). HBIPS data were submitted for each quarter through an approved performance measurement system vendor.

Characteristics of hospitals that reported HBIPS measures are reported in Table 1. The majority of organizations were freestanding psychiatric hospitals located in urban areas. A plurality were nonteaching, not-for-profit, and midsize (100–299 beds) facilities. There were significant differences among the four cohorts by hospital characteristic. As a group, for-profit and freestanding hospitals tended to report early, and acute care hospitals with psychiatric units tended to report later (p<.001).

Table 2 shows results of regression analyses for HBIPS 1, 5, 6, and 7. After an adjustment was made for covariates, the 2011 cohort had higher average performance on the admission screening measure (p=.04 [HBIPS 1]) and the two discharge planning measures (p=.01 [HBIPS 6], p=.02 [HBIPS 7]), compared with the 2009 cohort. The 2014 cohort had lower performance on admission screening (p<.001) than the 2009 cohort, but it performed better than the 2009 cohort on justification for multiple antipsychotics (p=.01 [HBIPS 5]) and transmission at discharge of the continuing care plan (p=.02 [HBIPS 7]). The 2015 cohort performed better than the 2009 cohort on providing justification for multiple antipsychotics (p<.001 [HBIPS 5]), preparing written continuing care plans at discharge (p<.001 [HBIPS 6]), and transmitting the plans to the next care provider (p<.001 [HBIPS 7]).

Differences in measure performance over the first few reporting quarters may be partly explained by differences that existed at the initial reporting point. After adjusting for covariates, we found that at the initial reporting point, the 2011 and 2015 cohorts had significantly higher performance on each of the four measures compared with the 2009 cohort (p<.001). The 2014 cohort showed lower initial performance than the 2009 cohort on admission screening (p<.001) but higher initial performance for antipsychotic medication justification at discharge (p=.02) and the continuing care plan measures (p<.001). In addition, the reporting cohort coefficients were positive and significant for each measure (p<.001 for all comparisons; Table 2), indicating overall progress by each cohort.

The results for the two-part regression model of restraint and seclusion are reported in Table 3. The coefficients for occurrence indicate the percentage of positive (nonzero) hours per 1,000 patient hours for each cohort. The nonsignificant slope × cohort coefficients indicate that the percentage of nonzero hours of restraint did not change during the initial six quarters.

Table 3 also examines the intensity of use of restraint and seclusion, reported as the average number of hours per 1,000 patient hours (excluding zero hours) spent in restraint or seclusion. Only the 2011 cohort had significantly fewer overall hours of restraint (p<.001) and seclusion (p<.03) per 1,000 patient hours compared with the 2009 cohort. The slopes for hours of restraint and hours of seclusion per 1,000 patient hours indicate that their use decreased significantly over the initial six reporting quarters in the 2009 (restraint, p=.002; seclusion, p=.017) and the 2015 (restraint, p<.001; seclusion, p=.005) cohorts.

For HBIPS 1, 5, 6, and 7, each cohort improved its performance from its initial level to the last measured point (Figure 2). For these four measures, the weighted average percentage of clients who were treated at baseline in a manner consistent with the guideline was 76.4%. For the last reporting quarter, the weighted average gave a value of 89.2%. The difference of 12.9 percentage points provides a rough estimate of overall improvement. The geometric mean of hours per 1,000 patient hours for the first and last reporting quarters was calculated separately for restraint and seclusion and was compared across cohorts. Mean decreases (improvements) in these measures were less than 2 percentage points.

When each cohort’s performance for each measure was examined within the context of the entire time period, the percentage measures appeared to show consistent improvement. For cohorts with already high levels of performance, there was evidence of a ceiling effect. For the ratio of restraint hours to 1,000 patient hours, the 2014 and 2015 cohorts showed consistent improvement. The 2009 cohort appeared not to change much across quarters, although the level of restraint and seclusion for this cohort began and remained low.

Measure performance differed significantly by some hospital characteristics. The most consistent finding was related to hospital ownership type. Compared with not-for-profit hospitals, for-profit hospitals had significantly higher performance on writing a continuing care plan (p=.01) and transmitting the plan to the next treatment facility (p<.001) (Table 2). Somewhat less consistently, for-profit hospitals had a significantly higher ratio of restraint hours (p<.001) but a lower ratio of seclusion hours (p=.012) (Table 3).

This research was designed primarily to examine performance trends across time for psychiatric treatment facilities that report on the HBIPS measures. The first cohort (2009) consisted of hospitals that were early adopters of the measures and voluntarily reported their data. This cohort was the most varied of the four cohorts in terms of facility type and other hospital characteristics. For this reason, this cohort provides an interesting point of comparison with the other three cohorts.

It is worth noting that the 2009 early adopters did not report consistently better or worse performance at baseline compared with hospitals in later cohorts, nor did the 2009 cohort stand out in terms of its overall rate of improvement. That is not altogether surprising, given that results from previous studies on health care organizations that voluntarily report data have been mixed. A study among managed care organizations of women’s health care quality showed that higher-performing organizations were more likely to voluntarily report their data (10). In contrast, a study comparing quality rankings of hospitals that voluntarily participated in the 2008 Leapfrog Survey found little upward bias in the volunteer group and some evidence of downward bias (11). Similarly, hospitals that voluntarily reported surgical outcomes to the CMS, which then posted the information on a Web site, did not vary on most quality indicators compared with hospitals that chose not to volunteer to have their information presented (12).

One finding related to measure performance, however, was remarkably consistent across the cohorts. Hospitals in the earlier reporting cohorts were outperforming hospitals in later reporting cohorts by the time each new cohort began reporting. Examination of the longitudinal data presented in Figure 2 clearly supports the benefits of earlier measurement and reporting. This finding suggests that hospitals in the early adopter group had the most time to utilize the feedback from reporting to improve performance. Although this conclusion should be interpreted with some caution, it is worth noting that the specifications for the HBIPS measures were available to all hospitals simultaneously. The freestanding psychiatric hospitals in the 2011 cohort would have likely been aware that the measures would focus on specific evidence-based practices and that reporting on these measures would soon be required. It is possible that many of the hospitals in the 2011 cohort chose to implement the measures earlier but elected not to report them to the Joint Commission until 2011. If that is the case, it would suggest that performance measurement initiatives may be enhanced by public reporting and accountability and not just by measurement and feedback.

Examination of the cohort × quarter interactions for HBIPS 1, 5, 6, and 7 showed that not every cohort improved at the same rate on each measure over the initial reporting quarters. However, for these measures, results for the last reporting quarter showed improvement over the initial reporting quarter for every cohort. The greatest variability in cohort performance was observed in the restraint and seclusion measures. This variation may be related to a combination of factors associated with ownership type and hospital type.

The 2011 and 2014 cohorts, which had significantly higher proportions of government-owned hospitals, had higher restraint intensity compared with 2009. The 2011 cohort, however, had seclusion ratios that were similar to those of freestanding psychiatric hospitals in the 2009 cohort and that were significantly lower than those of the 2014 and 2015 cohorts, which largely comprised hospitals with psychiatric units. It is possible that differences in restraint and seclusion ratios may be due to differences in patient mix associated with ownership type and hospital type. Although differences by hospital type were mixed and are difficult to interpret, one set of findings—for-profit hospitals performed better on writing and communicating discharge planning—was consistent with recent research by Shields and Rosenthal (13).

This study had a number of limitations. First, although performance differences between cohorts, hospital types, and ownership types were observed, the study was not designed to explain such differences. Second, the magnitude of differences across some of the measures may be affected by ceiling or floor effects. Hospitals with high starting points on performance measures (or low starting points, in the case of restraint and seclusion hours) have little room to demonstrate improvement, reducing the value of slope comparisons across cohorts. In addition, performance measure rates may also be affected by changes in specifications, although the impact of such changes is likely to be small.

Third, limited data were available for the hospitals in the 2014 and 2015 cohorts, so a longer reporting period may have revealed different patterns compared with the 2009 and 2011 cohorts. It is also important to note that quality measures, such as the HBIPS measures, were developed by experts to help health care facilities provide better treatment. Although they address important aspects of care, they do not represent every dimension underlying good psychiatric treatment. Moreover, although the measures were selected and constructed on the premise that improved treatment processes lead to better treatment outcomes, they are not outcome measures. Finally, it is important to note that the study included only organizations that were accredited by the Joint Commission. This group represents the vast majority of freestanding psychiatric hospitals across the United States, but results cannot be generalized to the facilities that are unaccredited or accredited by some other organization.

This study found significant differences in performance among cohorts of hospitals that began reporting psychiatric measure data at various points in time. The cohorts were marked by significant variations in hospital type, ownership type, and other hospital characteristics, and their participation in measure reporting was likely influenced by changes in external reporting requirements. Examining measures separately by cohort provides an interesting perspective that is not generally included in reports on trends in measurement. The assumption that hospitals that chose early adoption of measurement—electing to voluntarily report data before being required to do so—were motivated by confidence that they were performing at a higher level than other hospitals was not supported.

Perhaps the most interesting finding from the study, however, was that previously observed associations between measure reporting and improvement appear to be quite robust. A trend toward improvement was found across nearly all measures and nearly all cohorts, despite the fact that the hospital cohorts had various initial starting points and improved at different rates and that various types of hospitals may have elected to start reporting at different times and for different reasons. Furthermore, the consistent association between measurement and reporting with steady improvement appears to have a corollary—hospitals that begin reporting early have a comparative advantage in performance over hospitals that delay.