Reliability of the Part II Board Certification Examination in Psychiatry: interexaminer consistency

OBJECTIVE: The aim of the study was to examine the reliability (interexaminer consistency) of the American Board of Psychiatry and Neurology (ABPN) Part II (oral) examination in psychiatry. METHOD: Grades were assigned independently by two examiners who observed the same examination in a 1-year cycle (1,422 candidates, two examinations each). The consistency between these pairs of grades (pass, condition, fail) was analyzed using a weighted kappa statistic. RESULTS: There was perfect agreement between examiners in 67% of examinations, minor disagreement in 26%, and major disagreement in 7% (weighted kappa = 0.54-0.56). CONCLUSIONS: The Part II ABPN examination demonstrates fair to good reliability as measured by interexaminer consistency. Development of more explicit grading criteria should further improve examiner agreement in future examinations.