DSM-5 Field Trials in the United States and Canada, Part II: Test-Retest Reliability of Selected Categorical Diagnoses

In order to understand the difference between the results of DSM-5 and previous DSM field trials, it is critical to note the difference in conceptual approach and design of these trials. With DSM-III (17), the major focus was on demonstrating that the introduction of explicit diagnostic criteria, as recommended by Stengel (4), could demonstrably improve reliability—the degree to which two or more clinicians could agree on a diagnostic assessment. The focus on reliability has been a highly successful innovation for psychiatry and has clearly advanced clinical communication and the feasibility of more productive research investigations.

Critical to assessment of the reliability of DSM-III diagnoses was the newly developed intraclass kappa, a statistical measure of agreement between two raters that would adjust for chance agreement. The intraclass kappa statistical approach had recently been developed by Fleiss, Spitzer, Endicott, and Cohen specifically to address measurement of the reliability of mental disorder diagnoses (18). Given the state of the science in 1980, limited attention was given to the precision of the kappa statistic—that is, the standard error of any kappa statistic estimate, which was the major focus of the DSM-5 Field Trials (19).

It is noteworthy that both the DSM-5 and DSM-III Field Trials attempted to obtain representative patient samples in general clinical settings, used usual clinical interview approaches (instead of structured research interviews), and had test-retest interviews with two clinicians assessing the same patient on separate occasions—although one-third of the DSM-III patients had interrater assessments with two clinicians rating the same interview conducted by one of them (17).

In order to obtain more precise kappa estimates, the sampling strategy for DSM-5 was designed to ensure the availability for assessment of a greater number of patients with relatively rare disorders in clinical settings than could be obtained from the random sampling strategy used for the DSM-III Field Trials. The stratified sampling approach for DSM-5, described in the accompanying article by Clarke et al. (20), increased the availability of patients with rare diagnoses, although there were still some diagnoses for which it was not possible to obtain an adequate number of patients for analysis. In the DSM-III trials, which used a random sampling strategy, many of the diagnoses had fewer than six patients in any diagnostic group—with others combined into larger groups of schizophrenic, substance use, anxiety, somatoform, sexual, and personality disorders when estimating kappa—with no confidence intervals to assess the precision of the estimates. The ability to assess the precision of the kappa estimates depends on an adequate sample size, which is very difficult to achieve with a random sample (20).

In contrast to the DSM-III Field Trials focus on obtaining a random sample of patients, DSM-IV Field Trials focused on select clinical populations in specialty settings in which diagnoses were made with structured research interviews. The major objective of the DSM-IV Field Trials was to evaluate the changes in prevalence that would occur for these highly selected patients if DSM-III, DSM-III-R, ICD-10, or the proposed DSM-IV criteria were used, with reliability assessments considered less important (21–23). When reliability trials were conducted, many of the interviews were videotaped and interrater assessments of the interview were conducted by two clinicians viewing the identical interview (24).

Our expectation in the DSM-5 Field Trials (20, 25) was that the levels of reliability would be lower than those seen in the DSM-IV Field Trials (which were conducted with carefully selected patients and highly trained clinicians) and more like the reliabilities seen for medical diagnoses assessed in clinical settings. The DSM-5 Field Trials design can be compared to NIH-supported clinical treatment effectiveness trials that focused on representative samples of patients with comorbid diagnoses in clinical settings—in contrast to industry-supported treatment efficacy trials of “pure” disorders used to establish medication indications (26). Inclusion criteria for “pure” disorder groups may be successful in generating good internal validity for the patients studied, but they lack external validity when treatment outcome or kappa reliability estimates are applied to patients with comorbid disorders in usual clinical settings (27).

The DSM-5 Field Trials were designed to evaluate the test-retest reliability, convergent validity, clinical utility, and feasibility of selected DSM-5 diagnoses that were considered to be of high public health importance or were proposed for new additions to the manual. The nature of the clinical population in which the field trials took place, the clinical assessment approach, and the primary objective of obtaining statistically precise estimates of the kappa measure of reliability were all important components of this approach. The latter focus on obtaining kappa estimates for disorders with a standard error of less than 0.1 (a kappa confidence interval size of 0.5 or less) required the use of a stratified sampling design in general clinical populations of consecutive patients. No attempt was made to exclude comorbidities at entry or at either clinical interview or to exclude patients with unclear presentations, which were often excluded from past field trials. This approach was intended to enhance the possibility of examining patients for all expressions of psychopathology as they appear in typical clinical settings.

The objective of this analysis is to document the reliability of the categorical diagnoses obtained by two independent clinicians using their usual clinical interviews followed by use of a computer-assisted checklist to document the presence or absence of the symptomatic criteria needed to support their clinical diagnosis. Detailed information on the site characteristics, study design, patient characteristics, clinician characteristics, diagnostic distributions of eligible clinical populations, and analytic approach is provided in an accompanying article (20). In addition to the focus in this article on the reliability of categorical diagnoses, the DSM-5 Field Trials included dimensional assessments of cross-cutting symptom domains obtained from patients who were able to fill out this “review of mental systems” on a laptop computer or tablet in the waiting room. Results of this review were summarized and made available electronically to the clinician before the clinical interview. The reliability of the patient cross-cutting dimensional ratings in two separate interviews is described in an accompanying article (28).

To illustrate the stratified sample design at each site that enabled calculation of the intraclass kappa reliability results, a summary from the combined Houston VA/Menninger Clinic is provided in Table 1. Column 1 identifies the target DSM-5 diagnoses for this site: PTSD, alcohol use disorder, major depressive disorder, mild traumatic brain injury, borderline personality disorder, and an “other diagnosis” stratum. As seen in column 2, patients could be assigned to these six strata based on their DSM-IV diagnosis or qualifying symptom profile. Column 3 identifies the proportion of the clinic population that qualifies for each stratum; note that these proportions are the same for each target diagnosis row and, because of comorbidity, total >1.0. Column 4 identifies the percentage of the sample assigned for sampling into each stratum (those classified into more than one stratum were assigned for sampling to the rarest stratum at that site). These provide the sampling weights for computations.

Looking at the first bolded row for PTSD, note that 47% of the clinic population qualified for PTSD but only 18% of the sample was assigned to this stratum. At the Houston site, PTSD was the most common stratum and was often comorbid with other target diagnoses. Consequently patients with, for example, PTSD and mild traumatic brain injury were assigned to mild traumatic brain injury for sampling, since it was rarer than PTSD (15% compared with 47%). Columns 5 and 6 identify the total number of patients in each stratum who had one visit and two visits, respectively, and the two-visit number is the number evaluated for the reliability of diagnoses. Column 7 identifies the number identified by one or both clinicians with the target DSM-5 diagnosis from each stratum—e.g., 34 of the 43 patients in the PTSD stratum with two visits were diagnosed with DSM-5 PTSD (79% as indicated in column 8) by one or both clinicians, as were 17 from the alcohol use disorder stratum, 16 from the major depressive disorder stratum, 37 from the mild traumatic brain injury stratum, 14 from the borderline personality disorder stratum, and nine from the other diagnosis stratum.

Clinicians participating in the reliability study were encouraged to identify comorbid diagnoses where they existed. Thus the same patient may be counted multiple times in columns 7 and 8. Column 9 provides the intraclass kappa statistic for each disorder (a kappa of 0.69 for PTSD in this setting). Column 10 gives the 95% two-tailed confidence interval (CI) and column 11 suggests the interpretation of the kappa reliability results, as previously documented (25). Finally, column 12 provides the clinic prevalence of the DSM-5 disorder, which can be contrasted in most cases with the DSM-IV prevalence or symptomatic presentation in column 3. Both the intraclass kappa and the prevalence are based on the use of sampling weights. Comparable tables from each site provide a wealth of information about the performance of the target diagnoses and will be provided in more detailed forthcoming analyses of individual sites and diagnoses.

Table 2 provides data for all field trials for adult disorders assessed at the seven sites that were successful in obtaining kappa estimates with a two-tailed 95% CI size ≤0.5. Included in this table for each disorder are the intraclass kappa statistic, 95% CI, interpretation of the kappa, the weighted DSM-IV/symptomatic screening prevalence, and the DSM-5 prevalence (with CI) in the site-specific clinical samples. For disorders that were studied at multiple sites, a pooled kappa is provided with these combined results being emphasized with a notation if the CIs at the multiple sites failed to overlap—indicating the need for caution in the interpretation of the pooled kappa. The disorders are arranged in the order that they will appear in the revised organizational structure of DSM-5.

A total of 25 trials in these seven adult field trial sites were successful in obtaining adequate samples that permitted reliable kappa estimates for 15 separate diagnoses. Three diagnoses were in the very good (kappa 0.60–0.79) range: PTSD, complex somatic symptom disorder, and major neurocognitive disorder. Seven were in the good (kappa 0.40–0.59) range: schizophrenia, schizoaffective disorder, bipolar I disorder, binge eating disorder, alcohol use disorder, mild neurocognitive disorder, and borderline personality disorder. Four diagnoses were in the questionable (kappa 0.20–0.39) range: major depressive disorder, generalized anxiety disorder, mild traumatic brain injury, and antisocial personality disorder. Finally, one newly proposed diagnosis, mixed anxiety-depressive disorder, was in the unacceptable (kappa <0.20) range.

Table 3 contains results from seven adult trials in which sample sizes were insufficient to obtain a kappa estimate with adequate precision to provide a stable population estimate—i.e., either kappa CI size ≤0.5 or fewer than seven patients recruited into the target stratum. These results are presented in the interest of complete reporting. Even though some of these kappas are in the good range, inadequate precision for the kappa estimates limits their interpretability.

In general, there was a small decrease in prevalence between the DSM-IV clinical screening diagnoses for entry into the study and the DSM-5 prevalence estimates in these weighted stratified samples. A small but significant decrease was noted for single sites studying schizophrenia, major depression, generalized anxiety disorder, PTSD, major neurocognitive disorder, borderline personality disorder, and bipolar II disorder. Since general population epidemiological studies are never conducted until after a DSM revision is completed, it remains to be seen whether the changes in diagnostic criteria for these disorders will result in any changes in population prevalence in contrast to the shifts in clinical site prevalence in these field trial settings.

Table 4 contains results from 14 trials in four child/adolescent sites that were successful in obtaining adequate samples that permitted precise kappa estimates for eight separate diagnoses. Two were in the very good (kappa=0.60–0.79) range: autism spectrum disorder and ADHD. There were two in the good (0.40–0.59) range: avoidant/restrictive food intake disorder (a newly proposed disorder in DSM-5) and oppositional defiant disorder. Two diagnoses were in the questionable (0.20–0.39) range: major depressive disorder (as in the adult studies) and disruptive mood dysregulation disorder, a new diagnosis proposed for DSM-5 (the latter pooled estimate was heavily influenced by site variability). Finally, two disorders were in the unacceptable (kappa values <0.20) range: mixed anxiety-depressive disorder (as in the adult studies) and nonsuicidal self-injury, a new diagnosis that includes patients with frequent self-inflicted cutting.

Table 5 contains results from four child/adolescent trials in which sample sizes were insufficient to obtain a kappa estimate with adequate precision to provide a stable population estimate. As noted previously, the kappa statistics are reported for completeness, but without further comment because of their lack of precision.

The DSM-5 Field Trials are the first to directly include patients with diagnostic comorbidity when the explicit diagnostic hierarchies of DSM-III and DSM-IV are removed. However, most of the DSM disorders studied with their proposed revisions have demonstrated that multiple clinicians can communicate effectively about the nature of their patients’ diagnoses with adequate precision to carry out the clinical treatment goals first predicted by Stengel in the early recommendations to develop explicit diagnostic criteria (4).

For some common disorders, such as major depressive disorder and generalized anxiety disorder, the marked heterogeneity of patients who meet these diagnostic criteria and their comorbidity with other disorders are associated with lower reliability levels. Greater attempts to improve both the reliability and validity of these diagnoses are called for. The initial steps taken to address these issues for major depressive disorder in DSM-5 include adding the mixed specifier (to identify closer links with bipolar disorder) and an anxious distress specifier (to identify a subset with anxious depression).

Some disorders that have moved to a more inclusive or dimensional approach (e.g., major neurocognitive disorder, complex somatic symptom disorder, autism spectrum disorder, borderline personality disorder, and alcohol use disorder) demonstrated good to very good reliability and offer a new paradigm for future disorder revisions. However, maximizing the reliability of our current categorical diagnostic conventions is not the only or ultimate goal. As with all of medicine, the goal is to move beyond reliability to a better assessment of the validity of disorders identified by our diagnostic criteria. The DSM-5 proposal to obtain “cross-cutting” measures of 13 psychological symptom domains, described by Narrow et al. (28), is intended to provide a more dimensional description of patient presentations than can be captured by existing DSM-IV diagnostic criteria and boundaries. This approach is also consistent with the NIMH Research Domain Criteria (RDoC) project, which is attempting to identify both biological and symptomatic dimensional measures of psychopathology that correlate with genetic, neuroimaging, and neuropsychological factors irrespective of current diagnostic boundaries (40).

Emil Kraepelin, who pioneered the separation of schizophrenic and affective psychoses into separate diagnostic groups in 1898 (41), noted later in a 1920 publication—prescient in its anticipation of a current polygenetic-environmental interaction model of mental disorders—that the strict separation of these categorical diagnoses was not supported (42). We are now coming to the end of the neo-Kraepelinian era initiated in the U.S. by Robins and Guze (15) with a renewed appreciation of both the benefits and limitations of a strict categorical approach to mental disorder diagnosis (43).

The ultimate goal is to build on the progress achieved with categorical diagnoses by continuing with longitudinal follow-up of patients with these diagnoses, incorporating cross-cutting dimensional measures judiciously into the diagnoses where they prove useful, and in some cases recommending simple external tests (such as a cognitive test for mild neurocognitive disorder) that might improve the reliability and move toward a more mature scientific understanding of mental disorders. A noted philosopher of science, Carl Hempel, observed that “although most sciences start with a categorical classification of their subject matter, they often replace this with dimensions as more accurate measurements become possible” (44).

Clinicians think dimensionally and adjust treatments to target different symptom expressions in patients who may have the same categorical diagnosis. The intent of DSM-5 is to provide a diagnostic structure that will more fully support such dimensional assessments with diagnostic criteria revisions, specifiers, and cross-cutting symptom domain assessments. The goal is to support better measurement-based care and treatment outcome assessment in an era when quality measurement and personalized medicine will require new diagnostic approaches.

The authors wish to acknowledge the extensive efforts of the participating clinicians at each of the DSM-5 Field Trial sites. Principal Investigators: Bruce Pollock, M.D., Ph.D., F.R.C.P.C., Michael Bagby, Ph.D., C. Psych., and Kwame McKenzie, M.D. (Centre for Addiction and Mental Health, Toronto, Ont., Canada); Carol North, M.D., M.P.E., and Alina Suris, Ph.D., A.B.P.P. (Dallas VA Medical Center, Dallas, Tex.); Laura Marsh, M.D., and Efrain Bleiberg, M.D. (Michael E. DeBakey VA Medical Center and the Menninger Clinic, Houston, Tex.); Mark Frye, M.D., Jeffrey Staab, M.D., M.S., and Glenn Smith, Ph.D., L.P. (Integrated Mood Clinic & Unit and the Behavioral Medicine Program at Mayo Clinic, Rochester, Minn.); Helen Lavretsky, M.D., M.S. (The Semel Institute for Neuroscience and Human Behavior, Geffen School of Medicine, University of California Los Angeles, Los Angeles, Calif.); Mahendra Bhati, M.D. (University of Pennsylvania School of Medicine, Philadelphia, Pa.); Mauricio Tohen, M.D., Dr.P.H., M.B.A. (University of Texas San Antonio School of Medicine, San Antonio, Tex.); Bruce Waslick, M.D. (Child Behavioral Health, Baystate Medical Center, Springfield, Mass.); Marianne Wamboldt, M.D. (The Children’s Hospital, Aurora, Colo.); Prudence Fisher, Ph.D. (New York State Psychiatric Institute at Columbia University, New York, N.Y.; Weill Cornell Department of Psychiatry at Payne Whitney Manhattan Division, New York, N.Y.; North Shore Child and Family Guidance Center, Roslyn Heights, N.Y.; and Weill Cornell Department of Psychiatry at Payne Whitney Westchester Division, Westchester, N.Y.); Carl Feinstein, M.D., and Debra Safer, M.D. (Stanford University Child & Adolescent Psychiatry Clinic and the Behavioral Medicine Clinic, Palo Alto, Calif.).

The authors also wish to acknowledge the contributions of the DSM-5 work group members who provided the revised diagnostic criteria for DSM-5. Chairs for each of the DSM-5 work groups and study groups: Dan Blazer, M.D., Ph.D., M.P.H. (Chair, Neurocognitive Disorders); William T. Carpenter, Jr., M.D. (Psychotic Disorders); Joel E. Dimsdale, M.D. (Somatic Symptom and Related Disorders); Jan A. Fawcett, M.D. (Mood Disorders); Dilip V. Jeste, M.D. (Chair Emeritus, Neurocognitive Disorders); Charles O’Brien, M.D., Ph.D. (Substance Use and Addictive Disorders); Ronald Petersen, M.D., Ph.D. (Co-Chair, Neurocognitive Disorders); Daniel Pine, M.D. (Child and Adolescent Disorders); Katharine A. Phillips, M.D. (Anxiety, Obsessive-Compulsive and Related, Trauma and Stress-Related, and Dissociative Disorders); Charles F. Reynolds III, M.D. (Sleep-Wake Disorders); David Shaffer, M.D. (ADHD and Disruptive Behavior Disorders); Andrew E. Skodol, M.D. (Personality Disorders); Susan Swedo, M.D. (Neurodevelopmental Disorders); B. Timothy Walsh, M.D. (Eating Disorders); Kenneth J. Zucker, Ph.D. (Sexual and Gender Identity Disorders); Jack D. Burke, Jr., M.D., M.P.H. (Diagnostic Instruments); Steven E. Hyman, M.D. (Diagnostic Spectra); Jane S. Paulsen, Ph.D. (Impairment and Disability Assessment); Susan K. Schultz, M.D. (Lifespan and Development); Lawson Wulsin, M.D. (Psychiatric/General Medical Interface); and Kimberly Yonkers, M.D. (Gender and Cross-Culture).

Finally, the authors wish to acknowledge the ongoing contributions of APA staff, whose extensive support efforts for the DSM-5 work groups made the field trials of proposed DSM-5 diagnostic criteria possible: Jennifer Shupinka, Seung-Hee Hong, Anne Hiller, Alison Beale, and Spencer Case.