The DSM-5 Field Trials and Reliability of Alcohol Use Disorder

To the Editor: The DSM-5 criteria incorporate many changes to improve psychiatric diagnosis compared with DSM-IV. While test-retest reliability was not a major DSM-5 focus, valid measurement requires adequate reliability (1). For alcohol use disorders, a 2006 review (2) indicated good to excellent reliability of the DSM-IV criteria for alcohol dependence (kappa values, 0.66–0.82). Reliability of DSM-IV alcohol abuse was lower, but the criteria were more reliable when analyzed independently of dependence (2). In the DSM-5 field trial, alcohol use disorder reliability was much lower (kappa=0.40) (1). Why the precipitous drop in reliability?

We considered three explanations. First, compared with DSM-IV, DSM-5 field trial cases could have been more complex because of comorbidity that reduced reliability. This appears unlikely, however, because the DSM-IV studies did not exclude comorbid cases, and they recruited participants from settings in which comorbidity was common. Second, DSM-IV kappas could have been inflated because multisite studies did not adjust for data pooled across sites. The reanalysis of data from one of the studies (3) after making this adjustment (4) suggests that this is also unlikely. After pooling data from sites with more than 50 patients, the unadjusted and adjusted kappas did not differ dramatically (Table 1).

Third, error variance could have differed between the DSM-IV studies and the DSM-5 field trial. Two main types of error variance reduce reliability (5–7): criterion variance and information variance. Criterion variance is minimized by having clear guidelines to evaluate symptoms. Information variance is minimized by standardizing 1) content, through fully or semistructured interviews, and 2) procedures, such as investigator training in the use of the interviews. While information variance can never be eliminated completely, studies using standard methodology (structured interviews and training) (7) to control information variance provide more information on criterion variance than studies that do not.

In terms of criterion variance, did the DSM-5 criteria for alcohol use disorder become less clear than those in DSM-IV? This appears unlikely, since the biggest change in alcohol use disorder from DSM-IV to DSM-5 was the combination of dependence and abuse criteria into a single disorder. Except for the addition of craving (new to DSM-5) and the deletion of legal problems, the alcohol use disorder criteria are identical in DSM-IV and DSM-5.

Did information variance increase in the DSM-5 field trial compared with studies in the 2006 review (2)? Here, the answer appears to be yes. All DSM-IV reliability studies reviewed in 2006 used standard methodology to minimize information variance. The DSM-5 field trials were very different (1, 4). After a brief introduction to DSM-5 criteria, clinicians conducted unstructured diagnostic interviews, rating criteria checklists for any disorder they deemed appropriate. Because clinician training was minimal and diagnostic assessment and checklist selection were unstructured, information variance in the DSM-5 alcohol use disorder field trial was not controlled.

Given the design differences in the DSM-IV studies and the DSM-5 field trial, the reliability of DSM-5 alcohol use disorder cannot be directly compared with DSM-IV; studies of DSM-5 alcohol use disorder that minimize information variance are needed. We think such a study should be conducted to accurately make comparisons between the DSM-IV and DSM-5 criteria for alcohol use disorder.