Interrater Agreement Among Psychiatrists Regarding Emergency Psychiatric Assessments

To the Editor: We read with great interest the article by Bruce B. Way, Ph.D., and colleagues (1) regarding poor assessment reliability among psychiatrists in the emergency room setting. In the discussion, the authors suggested that the development of a screening tool with structured clinical interview characteristics that might be used more reliably in this setting would be useful. We have recently published data on just such a brief scale that is designed to be used in emergent psychiatric intake settings to determine medical necessity (2).

This brief, 11-item scale covers most of the items examined in Dr. Way et al.’s study (psychosis, depression, suicidality or homocidality, hostility or aggression, uncooperativeness, treatment noncompliance, substance abuse, physical dysfunction, role dysfunction, and social support), and it is interesting that it rates them on a similar 0–6 Likert scale. Reliabilities were originally reported for only the three subscales that emerged from factor analysis in the published article (0.96, 0.92, 0.79), but we reanalyzed the items individually, and they showed interrater reliabilities that, except for homicidality (0.28), ranged from 0.68 to 0.88. Reliability was facilitated by the use of behavioral descriptor anchor points for each pair of 0–6 ratings (mild=1–2, moderate=3–4, severe=5–6). This type of scaling method, originally used by Bigelow and Berthot (3) with the Psychiatric System Assessment Scale (3), has been used for a larger inpatient scale at our facility (4), developed as a modification of the Psychiatric System Assessment Scale, and has facilitated accurate and reliable ratings in our clinical setting. We also demonstrated that this medical necessity scale was valid on the basis of correlations with subsequent inpatient ratings that were carried out independently by different clinicians, including inpatient length of stay, and on the basis of its ability to discriminate between patients requiring and not requiring hospitalization.

We would like to make an important point regarding implementation of such a scale: while a scale with 11 items may seem rather brief, it has been an onerous task for our emergency staff over the past year of implementation. However, we have been successful and have shown an internal consistency reliability of 0.78 with our first 168 subjects. The authors’ proposal to use “a small number of questions for each dimension” would result in a scale several times larger than ours. While such a scale would certainly be more comprehensive, it would be unlikely to work well and be hard to implement in a busy clinical setting.