Sustained Effectiveness of the Mental Health Environment of Care Checklist to Decrease Inpatient Suicide

In an effort to decrease inpatient suicide, a multidisciplinary group of VA employees with expertise in design, engineering, architecture, nursing, medicine, psychology, and law developed the MHEOCC (5). The MHEOCC is focused on architectural changes, because prior work suggested that environmental hazards, such as anchor points for hanging, were implicated in most attempted and completed suicides. The MHEOCC, which includes specific recommendations for use and suggested abatements for potential hazards, was designed for implementation by mental health staff. Usability testing was performed at a subset of VA inpatient psychiatry units, starting a continuous and ongoing process of iterative improvements to the MHEOCC. After a national VA mandate in November 2007 to implement the MHEOCC, each VA hospital with a mental health unit treating actively suicidal patients began using the checklist every three months to report the hazards found and abated.

The VA National Center for Patient Safety (NCPS) was established in 1999 to facilitate the development of formal patient safety activities across the VA health system (11). One of the first NCPS activities was the institution of an RCA program to individually and collectively analyze adverse events (11). Each Veterans Health Administration (VHA) facility supports a full-time patient safety manager responsible for investigating all adverse events at the facility. The NCPS requires that an RCA be done in all cases of sentinel events, including inpatient suicide.

To identify all completed inpatient suicides in the database, we identified all RCA reports conducted between January 1999 (known date of the first RCA) and October 30, 2015, that listed suicide or suicide attempt as the event in the incident field. We also used PolyAnalyst 6.0 to search the entire text of the remaining RCA reports for the terms “suicide,” “suicide attempt,” and “self-harm.” We then manually reviewed each of the identified RCA reports to find cases of completed suicides on inpatient mental health units.

To determine the rate of suicides on inpatient mental health units, we accessed 2000–2015 data from VHA administrative data sets regarding the number of admissions per quarter and bed-days of care per quarter provided on VHA acute inpatient mental health units. These numbers were used as the denominator to determine inpatient suicide rates per admission and per bed-days of care.

Because suicide on inpatient mental health units is a very rare event, we adopted several approaches in our statistical analysis. We studied these events as two rates: the rate of inpatient mental health suicide per 100,000 inpatient mental health admissions and the rate of suicide per one million bed-days of inpatient mental health care. We modeled the count data of the number of suicides on inpatient mental health units by using the Poisson distribution. The link function was the logarithm, and either total inpatient mental health admissions or total inpatient mental health bed-days was the offset.

To evaluate whether the effect of the MHEOCC on inpatient suicides on mental health units was sustained, we used the rate of inpatient suicide during the implementation phase of MHEOCC (2008–2010) as our reference rate. Then, using the Poisson maximized sequential probability ratio test (maxSPRT) approach, we repeatedly tested whether the inpatient suicide rates during the continuation phase (2011–2015) were significantly higher than this reference rate. Poisson maxSPRT was selected because it is the most appropriate sequential testing method to detect signals for rare events that follow a Poisson distribution. It has been commonly used to detect rare events, such as adverse effects related to vaccinations. The statistical power for suicides after 2010 was calculated by using the stringent selection of α=.01 and predefined upper limits of two (the lowest number of suicides observed in a single year before the implementation of MHEOCC). Critical values of the log-likelihood ratio, above which the null hypothesis would be rejected, were obtained for Poisson for a relative risk of 1 (that is, no change) from published tables. Analyses were conducted using SAS, version 9.2.

Our review of the RCA database found that 29 completed suicides occurred on mental health units from 2000 to 2015. Twenty-four occurred prior to the MHEOCC, and five occurred afterward. Across all VHA medical centers, the mean±SD number of acute mental health admissions per year was 77,893±2,332 from 1999 to 2015. During the same period, the VHA provided a mean of 1,143,890±171,006 bed-days of care on acute mental health units. The rate of suicide on acute mental health units prior to the MHEOCC was 4.2 suicides per 100,000 admissions or 2.72 suicides per million bed-days of care. After the MHEOCC, the rates were .74 suicides per 100,000 admissions or .69 suicides per million bed-days of care.

In earlier work, we found an initial decrease in inpatient suicides, as measured either by the suicide rate per admission or per bed-days of care (6). However, that report included only the first two years after implementation (up to 2011). The results reported here extend the earlier findings, with an additional five years of data. When 2008–2010 (initial implementation of the MHEOCC) is compared with 2011–2015 (continuation of the MHEOCC), it appears that the effect on suicides on inpatient mental health units was not only sustained but perhaps even enhanced (Table 1).

Except for 2012, when one inpatient suicide occurred, there were no suicides during the continuation phase, and thus the upper limit of two was never reached. The critical value of the log-likelihood ratio was never crossed during this phase, and the null hypothesis could not be rejected. In other words, inpatient suicide rates remained at a level that was equal to or lower than the rate during the implementation period.

Use of the MHEOCC was associated not only with an initial reduction in the number of suicides on inpatient mental health units but also with a sustained reduction over more than seven years. The trend suggests that the rate of suicide continues to decline since implementation of the checklist.

The finding of a sustained trend is in contrast to much of the literature involving the sustainability of various clinical improvements, which has demonstrated the difficulty in both sustaining and maintaining fidelity after the adoption of new practices (12). Wiltsey Stirman and colleagues (10) reviewed the sustainability of mental health interventions and concluded that there was “little evidence for a sustained effect.” Their findings are consistent with a body of evidence suggesting that mental health interventions are not sustained (12). There are multiple examples showing the difficulty of sustaining mental health interventions (8).

This study found evidence of the sustained effectiveness of a mental health intervention at more than 150 VHA hospitals across the United States. The key to sustainability may lie in the intervention itself. Most mental health interventions involve training mental health staff to engage in a new practice. The MHEOCC primarily involves physical changes to the care environment or architecture. Past studies have shown that changes to the architecture and physical environment are more likely to be sustained (13). The NCPS has proposed a rubric to suggest which types of interventions are most likely to be sustained when responding to RCA data (11). That guide suggests that interventions and improvements based on staff education and training are unlikely to be sustained, whereas changes that are related to physical changes are more likely to have a sustained effect. Thus it is possible that the lack of evidence for sustained effectiveness described by Wiltsey Stirman and colleagues (10) reflects the fact that most mental health interventions involve training staff to provide a new intervention. This is in contrast to interventions aimed at improving other health care domains, which are less likely to rely exclusively on staff training and education and appear more likely to be sustained (14). This assertion, if valid, would have broad implications for future interventions to improve mental health; it suggests that interventions that are not entirely based on staff training and education are preferable. However, application of such interventions in typical situations involving improvements in mental health may not be possible.

This work is not without limitations. It is possible that some inpatient suicides were not reported. The intervention was administered only at VA hospitals, and thus it may have different effects at non-VA sites. The period of sustained effect was limited to seven years, and the effect might be lost over a longer period. It is also possible that a change in the acuity or other characteristics of patients admitted to VA mental health units resulted in the observed decrease in the suicide rate. Finally, we cannot rule out that suicide was not prevented but rather delayed until after discharge.

The findings of this study provide additional evidence of the usefulness of the MHEOCC and suggest ways to increase the sustainability of mental health interventions.