Interrater Agreement Among Psychiatrists in Psychiatric Emergency Assessments

Objective:The authors’ purpose in this study was to investigate the interrater agreement among psychiatrists in psychiatric emergency service settings. The interrater reliability of many of the key concepts in psychiatric emergency service settings has not been studied.Method:Videotapes of 30 psychiatric emergency service patient assessment interviews conducted by psychiatrists were shown to eight experienced psychiatric emergency service psychiatrists. The eight psychiatrists rated each videotape on dimensions such as severity of depression and psychosis and recommended a disposition for each patient. Interrater reliability was then explored. Results:The level of agreement (intraclass correlation coefficient) among the reviewing psychiatrists was higher for psychosis and substance abuse but lower for psychopathology, impulse control problems, danger to self, and disposition. The reviewers’ disposition recommendations did not match well with the assessing psychiatrist’s actual disposition, but comparisons with actual practice should be considered only suggestive.Conclusions:Psychiatric emergency service assessments need improvement. This may be accomplished by exploring the underlying structure of psychiatric emergency service concepts, the creation and validation of structured assessment tools, and the creation of practice guidelines. Am J Psychiatry 1998; 155: 1423-1428