The Veterans Health Administration REACH VET Program: Suicide Predictive Modeling in Practice
Abstract
The U.S. Veterans Health Administration developed a suicide prediction statistical model and implemented a novel clinical program, Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET). This high-value suicide prevention program aims to efficiently identify patients at risk and connect them with care. Starting in April 2017, national REACH VET metric data were collected from electronic health records to evaluate required task completion. By October 2020, 98% of veterans identified (N=6,579) were contacted by providers and had their care evaluated. In the nation’s largest health care system, it was feasible to implement a clinical program based on a suicide prediction model.
HIGHLIGHTS
•
Using a suicide prediction model to identify patients at statistically elevated risk of suicide, the U.S. Veterans Health Administration has implemented the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment, a clinical program to contact and engage veterans who might benefit from enhanced care.
•
In the nation’s largest health care system, performance data suggested that it is feasible to implement a clinical program based on a suicide prediction model.
Accurate risk identification is a component of high-value suicide prevention care. It can result in efficiently connecting at-risk patients to appropriate treatment. Traditionally, methods of risk identification have included screening and assessment to detect those with risk factors and warning signs. However, authors of a large meta-analysis concluded that the ability to predict suicide solely on the basis of epidemiological risk factors is only slightly better than chance (1). Given the critical public health concern suicide represents (2), researchers have developed predictive models to improve suicide prevention capabilities. An initial proof-of-concept model developed by U.S. Veterans Health Administration (VHA) and National Institute of Mental Health researchers documented that a suicide mortality risk prediction algorithm could be developed from VHA electronic health record (EHR) data (3). To facilitate implementation by the VHA, developers used machine-learning analyses to generate a model that performed similarly when compared with the original model but used 84% fewer variables (4).
Work has been done to conceptualize how to incorporate such models into clinical workflows; however, some researchers have suggested that clinical implementation may not be appropriate given potential risks (5). For example, concerns have been noted regarding health care systems’ ability to manage identified risk effectively and safely with limited resources, especially because many people identified as being at risk do not die by suicide (5). On the basis of these considerations, VHA developed and implemented a novel clinical program based on the VHA predictive model, the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET). VHA policy (6) requires that each veteran identified to be in the top 0.1% tier of statistical risk of suicide have their care evaluated and be contacted by a VHA provider (6). REACH VET identification was associated with a 5% reduction in the probability of a documented suicide attempt, with adjustment for patient age and sex, as well as with more outpatient encounters, increased documentation of new suicide prevention safety plans, and fewer inpatient mental health admissions and emergency department visits. Analyses did not identify an effect on suicide, but they may have been underpowered because death data were not available for the entire cohort (7). This column complements the evaluation work and describes the REACH VET clinical program, implementation strategies, and an evaluation of implementation feasibility.
Clinical Program Overview
The suicide risk prediction model was run monthly for patients who had received care in the prior 24 months at each of the 140 VHA facilities. The statistical model is composed of 61 risk variables derived from the EHR. Each patient received a risk score, and those who fell in the top 0.1% risk stratum were identified to receive intervention. Model development and validation efforts found individuals in this tier of risk to have suicide rates 39 times higher than that of the general VHA patient population (3, 4). An estimate in excess of 89 suicide deaths over 12 months is anticipated among the approximately 6,500 veterans identified each month.
Within 2 weeks of identification of a veteran at risk of suicide, the VHA provider who worked most closely with the patient was required to make contact and evaluate the patient’s clinical risk, as well as to collaboratively discuss the patient’s care needs and to offer suggestions for care enhancement. Unless contraindicated, it was recommended that providers inform patients of their identification by the predictive model and provide education regarding the REACH VET program. If the veteran had been previously identified, then the provider determined whether additional outreach was indicated.
Implementation
Rapid phased national implementation of REACH VET started in November 2016, when the predictive model identified eight veterans with the highest risk scores at each facility. By February 2017, facilities received the names of all veterans in the top 0.1% risk tier at their facility.
A wide range of strategies were used to support REACH VET implementation. Exemplars of implementation strategies (8) from a four-phase (exploration, preparation, implementation, sustainment) framework (9) are provided in Table 1. Critical program development decisions were made during the preparation phase. For example, because of the limited evidence base regarding interventions for those at statistically significantly elevated risk of suicide, it was decided that specific interventions (e.g., safety planning) should be recommended but not required.
| Implementation strategy | Description of strategy | Phase of implementation |
|---|---|---|
| Mandate change | Policy memorandum | Preparation |
| Conduct educational meetings | Informational presentations tailored to the audience (e.g., leaders, providers) | Preparation, implementation |
| Distribute educational materials | Distributed via intranet site, e-mail, and coordinator calls | Preparation, implementation, sustainment |
| Centralize technical assistance | Developed and sustained a national implementation team | Preparation, implementation, sustainment |
| Identify and prepare champions | Identification of a REACH VET coordinator at each facility | Preparation, implementation, sustainment |
| Audit and provide feedback | Quality data and benchmarks were made available via online reports and distributed monthly to national and regional leadership | Implementation, sustainment |
a
REACH VET, Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment.
Implementation strategies were deployed to address barriers. For example, some providers were hesitant to implement REACH VET without having program effectiveness data. In response, the implementation team identified facility-level leaders to serve as internal champions and disseminated patient feedback in support of the program. Additionally, best practices used by high-performing facilities were identified and disseminated to address barriers to integrating required tasks into existing workflows.
Feasibility Evaluation
The period of observation (April 2017–October 2020) began 2 months after full national implementation. Data were sourced from the U.S. Department of Veterans Affairs Corporate Data Warehouse. Data elements generated by completion of standardized note templates were transmitted from the EHR into the Corporate Data Warehouse. Four monthly REACH VET performance metrics were calculated with these data: “coordinator accepted” (coordinator documentation of receiving the name of the identified veteran and assigning a provider), “provider accepted” (provider documentation of receiving the name and required tasks), “care evaluation performed” (provider documentation of care reevaluation), and “outreach attempted” (provider documentation of an attempt to contact the veteran).
During the period of observation, an average of 6,593 (range 6,368–6,662) veterans were identified monthly across VHA facilities. Among the veterans identified in the final observation month (N=6,579), most were male (95.3%, N=6,268) and White (86.3%, N=5,675). More than 90% (N=6,214) of veterans had major depression or a substance use disorder, and more than 75% (N=5,055) had an anxiety disorder diagnosis. Almost half (N=3,063) had experienced homelessness in the prior 2 years.
Performance on all four metrics improved during the 3.5-year observation period. Coordinator accepted was consistently the task completed at the highest rate, with the lowest rate in October 2017 (89%) and the highest in April 2020 (100%). Outreach attempted consistently had the lowest rate, ranging from 50% to 98%. From April 2017 through October 2020, completion of the coordinator accepted task increased from 90% to 100%, provider accepted increased from 58% to 99%, care evaluation performed increased from 54% to 98%, and outreach attempted increased from 50% to 98%. All metrics met or exceeded the 95% benchmark by April 2020. The variation between facilities with respect to performance decreased over time. For example, in April 2017, performance on the outreach attempted metric ranged from 0% to 100%, but by October 2020, it ranged from 81% to 100%.
Discussion
REACH VET is a clinical intervention based on a suicide prediction statistical model. A wide range of implementation strategies were used to facilitate program uptake. Performance on all metrics related to completion of program requirements improved throughout implementation. Targets were met and sustained until the end of the observation period. That is, 98% of veterans identified were contacted by providers and had their care reevaluated. These findings suggest that it is feasible to implement a clinical program based on a suicide prediction model.
Several factors likely contributed to successful implementation of REACH VET. Suicide prevention is a top priority among senior VHA leaders. Consistent senior leadership engagement was evidenced by regular briefings between VHA national suicide prevention program staff and the Under Secretary for Health, as well as Congressional leaders. These briefings generated inquiries regarding implementation at underperforming facilities. National suicide prevention leaders also provided Veterans Integrated Service Network (VISN; i.e., regional) leaders with monthly performance metrics, which they disseminated to facilities in their region. Tailored outreach efforts, supported by VISN leaders, were provided to underperforming facilities.
Critical tasks completed during the preparation phase likely contributed to program feasibility. The implementation team made multiple ethical, clinical, and operational decisions while developing the intervention. Outer and inner contextual factors were considered throughout this process to maximize innovation-values fit, which has been shown to enhance implementation (9). For example, the value of transparency was incorporated into the decision to inform providers of patient-level clinical variables contributing to suicide risk. Additional key decisions were made in regard to identification of potential false positives. The decision not to prescribe specific interventions facilitated patient-centered care and resource allocation, because time-intensive interventions were not required for all identified patients.
The development of a diverse strategy for implementation also likely affected feasibility (8). The approach was flexibly modified during the implementation and sustainment phases, via collaboration with leadership and the field. The choice of specific implementation strategies likely enhanced feasibility. For example, REACH VET coordinators were internal, local champions who served as liaisons between the national implementation team and the local facility, enhancing local tailoring. Additionally, audit and feedback, paired with proactive technical assistance, allowed for the identification and support of underperforming facilities. The implementation team worked with stakeholders to identify and address specific barriers, provide practical solutions, and build trust.
This study did not include a formal evaluation of contextual factors, such as organizational climate, that could inform subsequent implementation efforts. Moreover, it was unknown which factors were associated with facility performance. Little is known about the patient experience of REACH VET and how coordinators and providers use the program to enhance care. One study, however, reported that veterans on an inpatient psychiatric unit found the idea of REACH VET to be generally acceptable (10). Analyses are under way to evaluate the cost-effectiveness of REACH VET.
The findings noted in this column highlight the feasibility of implementing a suicide prevention clinical program that is based on predictive modeling. Although predictive modeling is complex, it can facilitate relatively simple outreach efforts associated with decreased suicidal behavior (7). We recommend that health care systems interested in implementing a similar program utilize a multifaceted and flexible approach to implementation. Such an approach can facilitate adaptation to meet unique system needs. Additionally, this approach allows health care systems to ensure ethical implementation of such a program by empowering their leaders to consider factors such as which types of care patients at risk of suicide are connected with and how this connection occurs.
References
1.
Franklin JC, Ribeiro JD, Fox KR, et al: Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull 2017; 143:187–232
2.
The Surgeon General's Call to Action to Implement the National Strategy for Suicide Prevention: A Report of the US Surgeon General and of the National Action Alliance for Suicide Prevention. Washington, DC, US Public Health Service Commissioned Corps, 2021. www.hhs.gov/sites/default/files/sprc-call-to-action.pdf
3.
McCarthy JF, Bossarte RM, Katz IR, et al: Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs. Am J Public Health 2015; 105:1935–1942
4.
Kessler RC, Hwang I, Hoffmire CA, et al: Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration. Int J Methods Psychiatr Res 2017; 26:e1575
5.
Belsher BE, Smolenski DJ, Pruitt LD, et al: Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry 2019; 76:642–651
6.
Memorandum to VHA VISN Directors: Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (VAIQ 7755251). Washington, DC, Department of Veterans Affairs, 2016
7.
McCarthy JF, Cooper SA, Dent KR, et al: Evaluation of the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment suicide risk modeling clinical program in the Veterans Health Administration. JAMA Netw Open 2021; 4:e2129900
8.
Powell BJ, Waltz TJ, Chinman MJ, et al: A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015; 10:21
9.
Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health 2011; 38:4–23
10.
Reger MA, Ammerman BA, Carter SP, et al: Patient feedback on the use of predictive analytics for suicide prevention. Psychiatr Serv 2021; 72:129–135
Information & Authors
Information
Published In
History
Received: 1 November 2021
Revision received: 21 December 2021
Revision received: 7 March 2022
Accepted: 26 April 2022
Published online: 30 August 2022
Published in print: February 01, 2023
Keywords
Authors
Competing Interests
Dr. Brenner reports editorial remuneration from Wolters Kluwer and royalties from the American Psychological Association and Oxford University Press. In addition, she consults with sports leagues via her university affiliation. The other authors report no financial relationships with commercial interests.
Funding Information
This work was supported in part by the VA OMHSP.This column does not necessarily represent the views of the VA or the U.S. government.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).