Syndromic Surveillance of Mental and Substance Use Disorders: A Validation Study Using Emergency Department Chief Complaints
Abstract
Objective:
This study evaluated whether emergency department (ED) patient presentations for problems related to mental and substance use disorders could be validly monitored by a syndromic surveillance system that uses chief complaints to identify mental disorders.
Methods:
The study used syndromic surveillance data on 146,315 ED visits to participating Fresno County, California, hospitals between January 1 and December 31, 2013. Free-text patient chief complaints are automatically classified into syndromes based on the developer’s algorithms. Agreement was assessed between the algorithm’s syndrome classification of mental health and substance abuse (MHSA) disorders and ICD-9-CM discharge diagnostic codes. Diagnosis and ED utilization patterns among patients with at least one visit with an MHSA syndrome classification were also examined.
Results:
Approximately 8% of ED visits during the study period received an MHSA syndrome classification. Overall agreement between MHSA syndrome classification and psychiatric- or substance use–related ICD-9 discharge diagnoses was high (κ=.92, 95% confidence interval=.91–.92). Sensitivity (100%) and specificity (98.6%) of the MHSA syndrome classification were also very high. MHSA syndrome–classified patients exhibited high levels of health care and morbidity burden compared with other patients.
Conclusions:
ED chief complaints can be utilized to reliably and validly ascertain the incidence of patient presentations for mental and substance use disorders in contexts in which discharge diagnoses are not routinely available. Wider adoption of MHSA-related syndrome algorithms by syndromic surveillance systems could be valuable for public mental health surveillance, service delivery, and resource planning efforts.
There is increasing recognition that public health surveillance activities should include surveillance of mental and substance use disorders (1,2). Such disorders—for example, depression, anxiety, and alcohol and drug abuse—affect approximately half of all Americans at some point during their lifetime (3) and as many as 25% of adults during any given year (4). They also account for growing portions of the U.S. population’s total burden of disability (5,6) and economic costs (7,8). Ongoing surveillance of mental and substance use disorders is key to assessing community mental and behavioral health needs and to monitoring long-term trends in the disorders’ occurrence (2). Such surveillance is also important for documenting shorter-term trends, given that the occurrence of and care seeking for mental and substance use disorders can fluctuate rapidly in response to community stressors, such as a natural disaster (9,10), economic downturn (11,12), and terrorist attack (13,14), and to changes in health care practices (15).
Although most current efforts to conduct surveillance of mental and substance use disorders take place at the national or state level (1), such surveillance may be most useful at the local level. County and city agencies are typically responsible for administering and evaluating mental health services and for responding to psychiatric crisis situations. However, surveillance of mental and substance use disorders at local levels presents several challenges. First, although conducting population-representative health surveys is the most rigorous approach to ascertaining burden of prevalent mental health problems (16,17), the resources required for such surveys are prohibitive for most local jurisdictions. Second, it can be difficult to establish case definitions of mental and substance use disorders that are usable in surveillance contexts because of evolving psychiatric definitions and interpractitioner variability in diagnosis (2). Third, data on mental and substance use disorders at the local level are rarely available on a timely basis. For example, frequency counts of patients obtaining care from a county department of behavioral health may be reported just once per year, with little opportunity to monitor or respond to shorter-term fluctuations (18).
Syndromic surveillance of presentations to emergency departments (EDs) that are related to mental and substance use disorders may be one promising approach to addressing these challenges. Syndromic surveillance is a widely used, though underrecognized, functionality of many local public health agencies in the United States. In their most typical structure, syndromic surveillance systems utilize specialized software that automatically collects, processes, and transmits prediagnostic patient data (chief complaints and basic demographic characteristics) in real time from local EDs to a city or county public health department (19,20). A crucial aspect of this process is the automatic classification of each ED patient’s chief complaint into one or more syndromes—such as respiratory, neurological, rash, or injury—based on keyword searches of the free-text chief complaints. Such data have primarily been used to identify emerging outbreaks of infectious disease, and no research has used syndromic surveillance system data to track mental and substance use disorders. However, it has been proposed that syndromic surveillance system data might also provide a useful indicator of clinically relevant occurrences or exacerbations of mental and substance use disorders (2).
To address this question, this study made use of data from Fresno County, California, to investigate whether chief complaints and their associated syndrome classifications can be used to validly and reliably ascertain ED visits for mental and substance use disorders. In 2008, the Fresno County Department of Public Health (Fresno DPH) launched a syndromic surveillance program that collects visit data from local emergency departments that are recorded prior to diagnosis. The visit data include chief complaints that are then automatically classified into syndromes. In January 2012, this data stream began incorporating ICD-9-CM (21) discharge diagnoses of ED patients who were evaluated by a clinician. This change provided the opportunity to assess the validity of using chief complaints as the basis of syndrome classification of mental and substance use disorders by using the associated discharge diagnosis as the criterion standard. ICD-9-CM diagnosis patterns and overall ED utilization among patients who received a syndrome classification related to mental and substance use disorders were also examined.
Methods
Data Source
Data for the study were obtained from the Fresno DPH syndromic surveillance system. This system utilizes the EpiCenter software package developed by Health Monitoring Systems (HMS). EpiCenter electronically captures ED visit data elements collected during routine administrative and patient care activities at linked hospital facilities and securely transmits those data to HMS in real time for preprocessing. This preprocessing includes automatic classification of patient chief complaints into one or more syndrome classifications, developed by HMS. The preprocessed data are then transmitted to Fresno DPH for further analysis and investigation.
The analytic sample for this study comprised all ED visits by Fresno residents to participating facilities from January 1 to December 31, 2013. During this period, two of the county’s four acute care hospitals participated in the syndromic surveillance system; these two facilities together accounted for approximately 170,000 ED visits, or 60% of annual countywide visits. The larger of the facilities is the region’s largest health care provider as well as the sole level 1 trauma center and safety-net hospital in the county.
ED visit data used for this study included patients’ age and sex, visit disposition, chief complaint at registration, up to seven automated syndrome classifications, and up to 10 patient discharge diagnoses. Discharge diagnoses typically become available in the syndromic surveillance software system at the time of patient discharge, but this process can be delayed or obstructed by billing procedures, electronic transmission errors, or other problems. Therefore, not all ED visits in the study database had associated discharge diagnoses.
The Fresno syndromic surveillance system captures encrypted unique patient identifiers that allow patient tracking between various EDs within Fresno County. Because the primary goal was to examine overall ED use, in the main analyses all ED visits were counted as independent. However, the analysis also examined ED utilization patterns for the subset of unique patients who made at least one visit for a mental disorder during the study period. Duplicate visits for the same date, time, and unique patient identifier were deleted from the analytic sample.
Measures
Syndrome classification.
ED visits were defined as having a mental health/substance abuse (MHSA) syndrome classification if the automated syndrome classification of “mental disorder” was present in any of the seven syndrome positions. EpiCenter’s “mental disorder” syndrome classification uses keywords found in the free-text chief complaints describing the patient’s main symptoms. Relevant keywords include “sad,” “anxiety,” “depression,” “chemical dependency,” and “attempted suicide”; also included are common misspellings (such as “anxeity” and “anxity”) and abbreviations (“chem dp”). A sample of the chief complaints classified under the MHSA syndrome category, along with their primary syndrome classifications, is shown in Table 1.
| Chief complaint text | Associated syndrome classifications |
|---|---|
| “Shortness of breath, chest tightness. Anxiety state, unspecified/unspecified essential hypertension” | Respiratory; congestion; mental disorders; diseases of the circulatory system |
| “Hallucinations. Unspecified schizophrenia, unspecified condition/other abnormal blood chemistry” | Mental disorders; neurological |
| “5150 DTS. Unspecified psychosis/personal history of traumatic fracture” | Mental disorders; injury |
| “Psych eval” | Mental disorders; symptoms, signs and ill-defined conditions |
| “Ingestion. Suicidal ideation” | Mental disorders |
| “5150 overdose” | Drugs; mental disorders |
| “AMS. Alcohol abuse, unspecified” | Mental disorders; neurological |
| “Medical clearance. Amphetamine and other psychostimulant dependence, unspecified/unspecified gingival and periodontal disease” | Mental disorders; diseases of the digestive system; drugs |
| “BIBA 5150; anxiety. Paranoid schizophrenia, unspecified cond” | Mental disorders |
a
Includes emergency department (ED) visits to facilities participating in the Fresno County, California, syndromic surveillance system between January 1 and December 31, 2013. AMS, altered mental status; BIBA, brought in by another; DTO, danger to others; DTS, danger to self; ETOH, ethyl alcohol; MHSA, mental health and substance abuse; 5150, refers to Section 5150, the California legal code authorizing involuntary psychiatric holds, which allows a qualified physician or police officer to confine a person in a hospital for up to 72 hours if he or she is deemed to pose a danger to him- or herself or others, is gravely disabled, or both.
Discharge diagnosis.
Visit variables.
Patient age was examined as a continuous variable. Visit disposition was categorized as discharged home, left against medical advice/without being seen by a clinician, transferred to psychiatric/rehabilitation facility, transferred to other facility, transferred to law enforcement, or other. Approximately one in five ED visits (20.8%) were missing disposition data; on the recommendation of the EpiCenter developers and Fresno DPH staff, these were assumed to reflect patients who left the ED without being seen.
Statistical Analyses
Descriptive statistics were calculated to characterize the ED visits in the entire study sample. Statistical calculations for the study’s central analyses used an analytic sample restricted to ED visits with a valid discharge diagnosis. Kappa statistics were used to assess overall agreement between MHSA syndrome classifications and MHSA discharge diagnoses. Sensitivity, specificity, positive predictive value, and negative predictive value of the MHSA syndrome classification measure were also calculated, with the MHSA discharge diagnoses used as the reference standard. All analyses were conducted by using Stata, version 14.1 (StataCorp LP). The institutional review boards of the University of California–Merced and the Fresno DPH approved this study.
Results
During the 12-month study period, there were a total of 146,315 ED visits (mean±SD=12,193±461.7 visits per month) to the participating Fresno facilities. Of these visits, 11,941 (8.2%) had an MHSA syndrome classification. Nearly three-quarters of all ED visits during the study period (N=104,513; 71.4%) had a valid discharge diagnosis. Most (95.7%) visits without a valid discharge diagnosis had a disposition indicating the patient had left without being seen. Of the visits with a valid discharge diagnosis, 9,669 (9.3%) had an MHSA discharge diagnosis. Figure 1 shows the monthly percentage of all ED visits with an MHSA syndrome classification and ED visits with an MHSA discharge diagnosis during the study period.
FIGURE 1. Percentages of all emergency department (ED) visits per month with an MHSA syndrome classification or an MHSA discharge diagnosisa
aIncludes ED visits in 2013 to facilities participating in the Fresno County, California, syndromic surveillance system. MHSA, mental health and substance abuse
Descriptive statistics for the overall sample of ED visits, for ED visits with an MHSA syndrome classification, and for ED visits with an MHSA discharge diagnosis are shown in Table 2. Patient and visit characteristics for the subsamples of visits with an MHSA syndrome classification and with an MHSA discharge diagnosis (hereafter referred to jointly as “MHSA-related visits”) were very similar. Compared with other patients, those with MHSA-related visits were significantly more likely to be male (p<.001) and were significantly younger (p<.05), according to chi-square tests. The visit dispositions of patients with MHSA-related visits also differed. Whereas 18.1% of patients with an MHSA syndrome classification and 17.4% of patients with an MHSA discharge diagnosis were transferred upon discharge from the ED to a psychiatric or rehabilitation facility, less than 1% of other patients were transferred to such facilities (odds ratio=37.7 and 24.1, respectively; p<.001 for both).
| Characteristic | Total (N=146,315) | Valid discharge diagnosis (N=104,513) | MHSA syndrome classification (N=11,941)b | MHSA discharge diagnosis (N=9,669)c |
|---|---|---|---|---|
| Patient gender (male) | 46.5 | 54.1 | 57.0 | 57.9 |
| Patient age (M±SD) | 40.6±20.7 | 37.5±19.5 | 38.6±15.9 | 37.9±14.9 |
| Visit disposition | ||||
| Discharged home | 68.0 | 93.0 | 66.9 | 72.5 |
| Left without being seen/AMAd | 26.1 | 1.6 | 6.2 | 2.2 |
| Transferred to psychiatric or rehabilitation facility | 2.0 | 2.4 | 18.1 | 17.4 |
| Transferred to inpatient/other facility | 1.7 | 1.9 | 6.2 | 5.8 |
| Transferred to law enforcement | .7 | .9 | 2.0 | 2.0 |
| Other | 1.6 | .2 | .6 | .1 |
a
Includes emergency department (ED) visits between January 1 and December 31, 2013, to facilities participating in the Fresno County, California, syndromic surveillance system
b
Denominator is the total number of ED visits (N=146,315).
c
Denominator is the number of ED visits with a valid discharge diagnosis (N=104,513).
d
Against medical advice
Concordance statistics between MHSA syndrome classifications and MHSA discharge diagnoses for the 104,513 visits with a valid discharge diagnosis are presented in Table 3. Overall agreement between the MHSA syndrome classification and discharge diagnoses was very high (κ=.92, 95% confidence interval [CI]=.91–.92); there were only 1,509 cases in which syndrome classification and discharge diagnosis were discordant. All visits receiving an MHSA diagnosis (N=9,668) also received an MHSA syndrome classification (sensitivity=100%; CI=99.9%−100.0%). Of the 94,845 visits receiving a diagnosis unrelated to an MHSA, 93,336 received a non-MHSA syndrome classification (specificity=98.4%; CI=98.3%−98.5%). The positive predictive value of the MHSA syndrome classification, indicating the proportion of syndrome-classified patients who ultimately received an MHSA discharge diagnosis from a clinician, was 86.5% (CI=85.9%−87.1%). The syndrome classification’s negative predictive value was 100% (CI=99.9%−100.0%), indicating that all patients with other kinds of chief complaints ultimately received a discharge diagnosis unrelated to an MHSA.
| Measure | Point estimate | 95% CI |
|---|---|---|
| Kappa | .92 | .91–.92 |
| Sensitivity | 100.0% | 99.9%–100.0% |
| Specificity | 98.4% | 98.3%–98.5% |
| Positive predictive value | 86.5% | 85.9%–87.1% |
| Negative predictive value | 100.0% | 99.9%–100.0% |
a
MHSA, mental health and substance abuse. ED, emergency department. Visits with an MHSA discharge diagnosis were those with an ICD-9-CM diagnosis of 290.0–316.x in the principal or first two secondary positions in the syndromic surveillance system used by Fresno County, California, from January 1 to December 31, 2013.
The most common ICD-9-CM principal diagnosis codes among ED visits with an MHSA syndrome classification were 305.x (nondependent drug or alcohol abuse, 13.9% of visits), 300 (anxiety states, 13.8%), 298.9 (psychosis, 8.4%), and 296.2 (major depressive episode, 5.1%). The most common ICD-9-CM code for secondary and tertiary diagnoses among these visits was V62.84 (suicidal ideation).
Data on the 84,526 unique patients in the sample were used to examine ED utilization patterns among Fresno patients who presented with at least one MHSA-related visit in 2013. Patients who ever presented with an MHSA syndrome classification (N=7,469) or an MHSA discharge diagnosis (N=6,324) made significantly more ED visits in 2013 compared with other patients. The mean number of total ED visits for patients with a visit with an MHSA syndrome classification (3.38±4.99) and patients with a visit with an MHSA discharge diagnosis (3.53±5.23) were significantly higher than for patients with no MHSA-related visit (1.57±1.56) (t=–70.85 and –70.46, respectively; df=84,524 and p<.001 for both).
Discussion
To our knowledge, this study is the first to assess the utility of emergency department–based syndromic surveillance data for public health surveillance of mental and substance use disorders. The findings reveal that the chief complaints associated with ED visits can be utilized to reliably and validly ascertain the incidence of patient presentations for mental and substance use disorders. Notably, there was near-perfect agreement between the MHSA syndrome classification and the group of ICD-9-CM discharge diagnoses indicating psychiatric or substance use disorders.
Demonstrating high concordance between syndrome classifications based on chief complaint and ICD discharge diagnoses, as do these analyses, is an important step in public mental health surveillance efforts. Discharge diagnoses are frequently not available in syndromic surveillance system data. Some software systems do not currently collect this information; in those that do, diagnoses may be missing for a large subset of ED visits, for example, those that involve patients who leave without being seen or those that involve delays in billing and transmission of the final diagnosis or patient admissions from EDs to inpatient facilities with different electronic health record systems. Therefore, diagnosis codes cannot be relied upon for comprehensive, real-time monitoring of MHSA-related (or other) ED visits. Syndrome-classified chief complaints—which are universally and promptly collected by ED-based syndromic surveillance systems—thus offer a valuable alternative.
The study findings also suggest high levels of health care burden and morbidity among patients whose visit was associated with an MHSA syndrome classification. These patients were far more likely than other patients to be transferred to a psychiatric, rehabilitation, or inpatient facility (overall transfer rates of 27% versus 6%), and they were more frequent utilizers of the Fresno County ED system. This study’s findings lend support to the argument that patients with visits receiving an MHSA syndrome classification constitute a population with genuinely impaired functioning and service needs, not simply elevated levels of distress. However, future research should correlate trends detected by syndromic surveillance of MHSA-related visits with additional claims or behavioral health systems data to better understand service utilization among these high-acuity cases.
This study had several limitations. First, data quality depended on participating providers capturing maximally complete and valid patient information. As with most syndromic surveillance data sets, discharge diagnoses were not available for all ED visits, which could have affected calculations of the agreement between visits with MHSA syndrome classification and discharge diagnoses if there were substantial differences in the presenting complaints of patients who did and did not receive a diagnosis (24). However, of the visits with a missing discharge diagnosis, only 1.8% had an MHSA syndrome classification. Given high levels of agreement between the MHSA syndrome classification and discharge diagnoses, any bias in the results is likely to be small in magnitude (25). Second, although the algorithm used in this study to determine MHSA syndrome classification has already been commercially developed and is immediately applicable to other public health agencies using the EpiCenter system, it cannot be assumed that the results are generalizable to other systems or geographic areas (26).
Third, the EpiCenter “mental disorder” syndrome classification does not distinguish between mental disorder–related (such as psychosis) and substance-related (such as drug overdose) ED presentations; instead, both types of presentation can be included in this single category. The development of new syndrome classification algorithms that more precisely differentiate between mental and substance use disorder chief complaints would thus be of significant value. Any new algorithms, however, would need to be informed by local context. For example, California ED triage staff use the chief complaint phrase “5150” to denote patients under an involuntary psychiatric hold, but this phrase is relevant only in California.
The findings of this study have implications for public mental health surveillance efforts and behavioral health care delivery. Although syndromic surveillance data on MHSA-related ED presentations cannot be used to assess overall prevalence or new cases of disorder, our results suggest that such data can reliably indicate acute disorder occurrences or exacerbations. Prior research shows that the rate of such presentations responds precipitously to natural disasters (27) and changes in the local mental health system, among other things (28). Mental health data from syndromic surveillance systems—the only real-time, automated indicator of service utilization currently available—may thus be particularly useful for behavioral health service coordination and delivery in emergency situations (for example, incidents of mass violence), where timely cooperation between behavioral health, public health, and law enforcement agencies is vital.
Such data could also serve as a useful basis for ongoing communication and planning between local public health and behavioral health agencies, which frequently serve the same populations but which may not routinely share resources or information (29). For example, syndromic surveillance data could be feasibly used for tracking ED boarding times and disposition outcomes for patients with psychiatric emergencies, targeting substance abuse intervention and outreach efforts, and evaluating the effectiveness of locally implemented mental health–related programs. All of these prospective uses have direct implications for client-level service delivery. Moreover, because of the highly automated nature of syndromic surveillance systems, health agencies typically do not need additional staff in order to effectively make use of syndromic data. However, training of local emergency department staff to ensure the accuracy of the original data transmissions will improve the utility of these systems for public health practice.
Conclusions
This study demonstrates the potential for algorithms based on chief complaints to validly detect ED presentations for disorders related to mental health and substance abuse. Wider adoption of such algorithms into syndromic surveillance systems, and implementation of a standardized software system across agencies, could be a valuable tool for both local and national public mental health surveillance efforts.
Acknowledgments
The authors gratefully acknowledge the assistance of Joseph Prado, B.A., Monique DeVilla, B.A., and Marissa Henry, B.A., in this research.
References
1.
Freeman EJ, Colpe LJ, Strine TW, et al: Public health surveillance for mental health. Preventing Chronic Disease 7:A17, 2010
2.
Galea S, Norris F: Public mental health surveillance and monitoring; in Methods for Disaster Mental Health Research. Edited by Norrs F, Galea S, Friedman M, et al. New York, Guilford Press, 2006
3.
Kessler RC, Berglund P, Demler O, et al: Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry 62:593–602, 2005
4.
Kessler RC, Chiu WT, Demler O, et al: Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry 62:617–627, 2005
5.
Murray CJL, Atkinson C, Bhalla K, et al: The state of US health, 1990–2010: burden of diseases, injuries, and risk factors. JAMA 310:591–608, 2013
6.
Murray CJ, Barber RM, Foreman KJ, et al: Global, regional, and national disability-adjusted life years (DALYs) for 306 diseases and injuries and healthy life expectancy (HALE) for 188 countries, 1990–2013: quantifying the epidemiological transition. Lancet 386:2145–2191, 2015
7.
Greenberg PE, Fournier A-A, Sisitsky T, et al: The economic burden of adults with major depressive disorder in the United States (2005 and 2010). Journal of Clinical Psychiatry 76:155–162, 2015
8.
Rehm J, Mathers C, Popova S, et al: Global burden of disease and injury and economic cost attributable to alcohol use and alcohol-use disorders. Lancet 373:2223–2233, 2009
9.
Galea S, Brewin CR, Gruber M, et al: Exposure to hurricane-related stressors and mental illness after Hurricane Katrina. Archives of General Psychiatry 64:1427–1434, 2007
10.
Krug EG, Kresnow M, Peddicord JP, et al: Retraction: suicide after natural disasters. New England Journal of Medicine 340:148–149, 1999
11.
Goldman-Mellor SJ, Saxton KB, Catalano RC: Economic contraction and mental health. International Journal of Mental Health 39:6–31, 2010
12.
Bruckner T, Kim Y, Snowden L: Racial/ethnic disparities in children’s emergency mental health after economic downturns. Administration and Policy in Mental Health and Mental Health Services Research 41:334–342, 2014
13.
Galea S, Ahern J, Resnick H, et al: Psychological sequelae of the September 11 terrorist attacks in New York City. New England Journal of Medicine 346:982–987, 2002
14.
Jordan NN, Hoge CW, Tobler SK, et al: Mental health impact of 9/11 Pentagon attack: validation of a rapid assessment tool. American Journal of Preventive Medicine 26:284–293, 2004
15.
Nesper AC, Morris BA, Scher LM, et al: Effect of decreasing county mental health services on the emergency department. Annals of Emergency Medicine 64:S82, 2014
16.
Moffitt TE, Caspi A, Taylor A, et al: How common are common mental disorders? Evidence that lifetime prevalence rates are doubled by prospective versus retrospective ascertainment. Psychological Medicine 40:899–909, 2010
17.
Insel TR, Fenton WS: Psychiatric epidemiology: it’s not just about counting anymore. Archives of General Psychiatry 62:590–592, 2005
18.
Vulnerable Communities in Los Angeles County: Key Indicators of Mental Health, 2008. Los Angeles, County of Los Angeles Department of Mental Health, 2008
19.
Buehler JW, Sonricker A, Paladini M, et al: Syndromic surveillance practice in the United States: findings from a survey of state, territorial, and selected local health departments. International Society for Disease Surveillance 6:1–20, 2008
20.
Ising A: The Current State of Syndromic Surveillance in the US. Atlanta, Council of State and Territorial Epidemiologists webinar library, 2015. http://www.cste.org/?page=WebinarLibrary/
21.
International Classification of Diseases, 9th Revision, Clinical Modification. Los Angeles, PMIC, 1995
22.
Mahajan P, Alpern ER, Grupp-Phelan J, et al: Epidemiology of psychiatric-related visits to emergency departments in a multicenter collaborative research pediatric network. Pediatric Emergency Care 25:715–720, 2009
23.
Brennan JJ, Chan TC, Hsia RY, et al: Emergency department utilization among frequent users with psychiatric visits. Academic Emergency Medicine 21:1015–1022, 2014
24.
Whiting P, Rutjes AWS, Reitsma JB, et al: Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Annals of Internal Medicine 140:189–202, 2004
25.
Reitsma JB, Rutjes AWS, Khan KS, et al: A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. Journal of Clinical Epidemiology 62:797–806, 2009
26.
Conway M, Dowling JN, Chapman WW: Using chief complaints for syndromic surveillance: a review of chief complaint based classifiers in North America. Journal of Biomedical Informatics 46:734–743, 2013
27.
Lauper U, Pantea C, Chen J-H, et al: Using hospital emergency department data to identify mental illness trends after Hurricane Sandy. International Society for Disease Surveillance Annual Conference Proceedings, 2014. www.researchgate.net/publication/285663307_Using_Hospital_ED_Data_to_Identify_Mental_Illness_Trends_After_Hurricane_Sandy
28.
Catalano R, McConnell W, Forster P, et al: Psychiatric emergency services and the system of care. Psychiatric Services 54:351–355, 2003
29.
Disaster Mental Health Surveillance at State Health Agencies: Results From a 2013 CSTE Assessment. Atlanta, Council of State and Territorial Epidemiologists, 2013. http://www.cste.org/
Information & Authors
Information
Published In
Cover: Understanding, by Michael Olszewski, 1984. Dyed and pleated silk plain weave with silk embroidery. Philadelphia Museum of Art, gift of Nancy and David Bergman, 2015.
History
Received: 19 January 2017
Revision received: 17 April 2017
Revision received: 23 May 2017
Accepted: 29 June 2017
Published online: 15 September 2017
Published in print: January 01, 2018
Keywords
Authors
Competing Interests
The authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).