Effectiveness of a Self-Administered Computerized Mental Health Screening Tool in the Emergency Department

The Computerized Assessment and Referral System (CARS) is a self-administered computerized screener adapted from the Composite International Diagnostic Interview, a validated tool developed by the WHO for assessing psychiatric diagnoses (11). CARS identifies up to 16 major psychiatric disorders and was initially developed by the Foundation for Advancing Alcohol Responsibility for use among individuals who were cited for driving under the influence (DUI), because these individuals have shown increased rates of certain types of psychiatric illness (12, 13). After completion, the screener generates a printout that identifies both positive lifetime and past-year diagnoses as well as more detailed information about each disorder identified. This printout includes a risk assessment report, which details an individual’s biopsychosocial factors that can increase the potential for committing a DUI offense (10), and treatment referrals based on the diagnosis and patient zip code. CARS has undergone usability trials and implementation pilots in settings outside of the ED (12).

This study was conducted at the University of Arkansas for Medical Sciences (UAMS), the only adult level-one trauma center in the state, in Little Rock; it was approved by the UAMS Institutional Review Board. At the time of the study, the UAMS ED served approximately 65,000 patients a year. Patients who presented to the ED for reasons unrelated to mental health conditions at the current visit and who had no previously documented psychiatric diagnosis were eligible to complete CARS if they were at least age 18 years, not incarcerated or in police custody, and English speaking. Participants were excluded if they were unable or unwilling to complete the screener; were unable or unwilling to provide a medical or psychiatric history; were currently presenting with a chief concern involving mental health or substance use; had a history of any psychiatric condition, excluding substance use disorders; or had a history of any use of a psychiatric medication.

Patients were enrolled 7 days a week from 10 a.m. to 10 p.m. during the period of December 1, 2017, to December 21, 2019. After eligible patients provided consent and completed the screener, research assistants reviewed the CARS report for any past-year or current diagnoses. Using features within the Research Electronic Data Capture (REDCap) software, the research assistants randomly assigned participants with one or more CARS diagnoses in a 1:1 fashion to a condition in which the participant’s ED physician either received the risk assessment report or did not receive the report. This randomization sequence was generated by the software, and so allocation was concealed until the time of randomization. Physicians were informed of the study’s purpose and randomization scheme before study initiation. All participants received their CARS report, as well as a list of mental health resources tailored to the regions in which they resided, before being discharged from the ED. Although the time at which patients received their respective reports was not standardized, all patients had ample time to bring up their screening results with their ED physicians (a CONSORT flow diagram is available in the online supplement to this report).

Three months after discharge, the patient’s electronic medical record (EMR) was searched for any return visits or any of the primary outcomes according to best practices for retrospective chart review (two independent reviewers who were blind to the data collected by the other prespecified data abstraction procedures, ongoing progress and performance monitoring, and disagreement resolution by consensus) (14). The primary outcome was the composite measure of change in physician care from study admission to ED discharge (psychiatric consultation, ED diagnosis of a previously unidentified psychiatric disorder, mental health referral, and transfer to a psychiatric facility). The secondary outcome was the change in patient behavior (ED visits 3 months after discharge vs. ED visits 3 months before study admission). Demographic characteristics included age and gender. Although race-ethnicity is standardly collected in psychiatric settings, it is almost never collected in the ED because self-report is thought to be inferior to more objective testing and may cause patient harm and therefore was not included in analyses.

This study was statistically powered to find at least the clinical difference noted by Schriger et al. (15). Consequently, 476 participants (238 per group) with a CARS diagnosis were needed for 80% power to detect this difference. On the basis of previous research, we estimated that 40% of the enrolled participants would have a CARS diagnosis. Therefore, we planned to enroll 1,190 patients. Welch’s t test was used to evaluate baseline differences between the two groups. Pearson’s chi-square test was used to analyze the two groups with respect to the primary composite outcome. All UAMS ED visits documented in participants’ EMRs were captured, including instances in which participants had left without having been seen by an ED physician. Differences in ED utilization between patients who received a CARS diagnosis and those who did not were measured longitudinally with an F1-LD-F1 design (an experimental scheme with one whole-plot factor and one subplot factor, with the whole-plot factor referring to a factor effective for each subject at all times and the subplot factor referring to a factor effective at a single time point for all time curves and all subjects); the Wald-type statistic was used for categorical variables. All analyses were performed in R Studio, version 1.0.136.

Of the 985 enrolled patients, 451 (46%; mean±SD age=40.5±15.8 years; 59% female) received a new CARS psychiatric diagnosis and were included in the study; 207 were assigned to physicians who received the patients’ CARS reports, and 244 to physicians who did not receive the reports (Table 1). No significant differences were detected between the two groups in terms of demographic characteristics. Newly identified past-year diagnoses included panic disorder (15%), explosive disorder (4%), major depressive disorder (14%), bipolar disorder (7%), generalized anxiety disorder (18%), social phobia (7%), gambling disorder (1%), psychotic disorder (11%), and posttraumatic stress disorder (10%). Suicidal ideation was reported by 7% of the patients. Patients whose ED physicians received and reviewed their CARS report and those whose ED physicians did not receive the report did not significantly differ in rates of identified past-year psychiatric diagnoses or suicidal ideation, with the exception of generalized anxiety disorder, whose rate was significantly higher among those whose physician did not receive the CARS report (p=0.005).

CARS reports did not result in a change in care by ED physicians. Seven participants whose ED physicians were given the CARS report had a change in care (five were formally diagnosed as having a psychiatric condition, and two were referred to outpatient treatment). Four patients whose ED physicians were not given the CARS report had a change in care (all were formally diagnosed as having a psychiatric condition, and two were referred to outpatient treatment). The number of ED visits in the 3 months before the CARS screening did not significantly differ from the number of visits in the 3 months after the screening (Table 1). Furthermore, having ED physicians review the CARS report did not affect ED return visits.

The findings of this study indicate that computerized mental health screening effectively identified previously undiagnosed psychiatric disorders known to have poor outcomes if left untreated. Of note, consistent with other studies of universal mental health screening in the ED (computerized or not), the detection of mental health problems did not necessarily result in a formal patient psychiatric diagnosis, a psychiatric consultation, referral to outpatient mental health treatment, admission to a psychiatric facility, or decreased patient ED return visits.

Physician behaviors did not change regardless of whether they were provided screening information, including detailed risk assessment summaries and lists of treatment options available in specific patient zip codes. This finding is counterintuitive yet not unexpected given studies on ED physician attitudes, beliefs, and perceived barriers to conducting mental health screening in the ED (e.g., time limitations, inadequate training, insufficient and fragmented mental health resources, and concerns about breach of patients’ expectations for the ED visit). Furthermore, physicians were informed of study purpose and randomization scheme before study initiation, potentially introducing a Hawthorne effect. However, if such an effect had occurred, it would likely have resulted in changes in physicians’ behaviors. Perhaps the physicians believed that giving patients their screening reports, particularly reports that also provide community resources for follow-up mental health treatment, is appropriate and adequate care or that doing further assessment, diagnoses, and referral is beyond the physician’s scope of practice; another possibility is that the physicians were too busy providing care to other patients with emergency care needs.

We note that the CARS screening tool is diagnostically broad and identifies symptoms of an array of diagnoses (e.g., social phobia, explosive disorder, and gambling disorder) that are not necessarily relevant to a patient’s presentation in the ED. Thus, this relatively unfocused approach to screening for psychiatric diagnoses might not be the most appropriate for ED settings, given that the identification of past-year disorders did not affect physician or patient behaviors. Moreover, mental health screening, especially when a patient presents to the ED for a purpose unrelated to a mental health condition, previously identified or not, has the potential to be stigmatizing and damaging.

Although the patients in this study adhered to the screening protocol, other patients have not always found mental health screening to be appropriate for the ED (10, 15), which might affect their historical objectivity of responses to the screening questions. Patients might become resistant to improving their knowledge of, and considering treatment options for, an often unexpectedly identified psychiatric condition. Moreover, it is unethical to identify previously undiagnosed psychiatric disorders, particularly among patients in crisis, and not provide or ensure treatment for patients screening positive for a mental disorder. Despite these ethical issues, ED physicians do not usually have the capacity to follow up with positive screening results. For a successful adoption of universal screening for mental health problems, the impact of the screening on the clinical workflow must be recognized.

This study was strengthened by its rigorous research design (randomized controlled trial, large sample, consecutive screening and enrollment procedures, and adequate statistical power) and use of a validated tool for identifying mental health problems. The study was limited by the fact that patients were given screening results in an unstandardized time frame and that it was unknown whether patients spontaneously discussed screening results with physicians (regardless of whether the physician received the screening results).

A common recommendation to improve physicians’ attitudes, self-efficacy, and empathy has been to provide them with information on available evidence-based interventions, including follow-up procedures. Another suggestion has been to add more staff designated to conducting universal mental health screenings, formally diagnosing psychiatric disorders, linking patients to treatment, and providing follow-up contacts to ensure continuum of care for patients. These recommendations are not always economically viable, and the effects of following these recommendations on physician treatment and patient outcomes have not been systematically examined. To implement universal mental health screening in EDs, processes are needed that do not solely involve adding to ED staff workload or placing more staff in an already overcrowded ED. Strategies to facilitate an effective screening workflow could entail incorporating mental health screening tools (computerized or not) into the EMR and establishing protocols to systematically address positive mental health screens.

ED physicians did not use information from patients’ computerized mental health screening to prompt further psychiatric diagnoses or guide further care when a mental health condition was unrelated to a patient’s ED visit. Furthermore, patients who completed computerized mental health screening did not change their use of ED services over the following 3 months. Collaboration among EDs and mental health treatment agencies, organizations, and researchers is needed to facilitate appropriate patient referrals and linkage to follow-up treatment.