Clinical Utility as a Criterion for Revising Psychiatric Diagnoses

Although in this article we discuss diagnostic validity and clinical utility as if they were distinct concepts, there is, in fact, considerable debate about the overlap between these two constructs, largely because of the lack of widely accepted definitions of these constructs as they apply to psychiatric diagnoses. At one extreme are those that use the terms interchangeably. For example, Spitzer (15) defines diagnostic validity as “the extent [to which] the defining features of a disorder provide useful information not contained in the definition of the disorder.” At the other extreme, Kendell and Jablensky (7) see validity and utility as completely distinct constructs, arguing that a disorder can be considered valid only if it can be demonstrated to have natural boundaries with other disorders.

Our position is to take the middle ground. Although we view validity and utility as separate constructs, there is considerable overlap between them. Diagnostic validity is a complex multifaceted construct that has historically been adapted from the field of psychological testing and includes a number of different types of validity. These include the following:

When evaluating whether a proposed change improves or detracts from the clinical utility of DSM, it is important to first identify the various components of clinical utility that might be affected by that change. Based on the definition of clinical utility that we proposed, we suggest that the merits of a proposed change to DSM can be evaluated by considering 1) its impact on the use of DSM, 2) whether it enhances clinical decision making, and 3) whether it improves clinical outcomes. These three components of clinical utility imperfectly resemble the classic “structure/process/outcome” framework for evaluating quality of care set forth by Donabedian (18). Each of these will be discussed in more detail.

This aspect of clinical utility focuses on whether the diagnostic system is used at all by its intended end population (user acceptability) and whether it is used correctly (accuracy in the application of the diagnostic criteria). Achieving adequate levels of user acceptability is critically important since nonutilization of the diagnostic system eliminates any potential benefits that might result from DSM changes that target either clinical decision making or patient outcome. Similarly, the correct application of the DSM diagnostic criteria in clinical settings is also a prerequisite for assigning the proper psychiatric diagnosis. Selecting the wrong diagnosis from the get-go may subsequently lead to improper treatment selection and ultimately to poor patient outcomes.

Before one can determine whether a proposed change to DSM positively affects its use, it is important to establish baseline information about how the current version of DSM is being used and to identify potential sources of poor user acceptability. Surprisingly, relatively little research has focused on the clinical use of DSM. Two methods have been used: 1) surveys of clinicians’ and researchers’ self-reported use of—and attitudes toward—various editions of the DSM and ICD classifications and 2) direct measurement of use through examination of recorded chart diagnoses. Although a number of surveys have been conducted over the past 20 years (19–26), their focus has been almost exclusively on general system-wide issues rather than on discerning attitudes toward specific diagnostic categories, Such restrictions have limited conclusions about the user acceptability of specific diagnostic categories.

One exception to this approach in survey questions concerns the DSM multiaxial system. A major impetus for overhauling DSM-III-R axis IV was the concern that it was “used infrequently in clinical settings and research studies” (27). As part of the literature review of articles examining the use, reliability, and validity of axis IV, Skodol examined the surveys of DSM-III and DSM-III-R users that included questions regarding their use of axis IV. For example, in the 1986 survey of practicing psychiatrists and 1984 graduates of residency programs (21), only 44% of the psychiatrists and 56% of the residents found axis IV to be useful. Partly based on the surveyed underuse of axis IV, the DSM-IV work group ultimately recommended that axis IV be replaced by a psychosocial and environmental problem list. It should be noted that although the executive summary of changes in the DSM-IV Sourcebook (28) noted that the psychosocial and environmental checklist was added because of “its simplicity and the fact that it allows clinicians to note the specific problems of concern,” no field trials were conducted to empirically determine whether this assertion was, in fact, true.

An alternative to assessing use by surveying clinicians is to measure it by reviewing clinical records for evidence of use. This method formed the basis of the recommendation by the Psychiatric Systems Interface Disorders Work Group to delete three of the subtypes for adjustment disorder (i.e., adjustment disorder with physical complaints, adjustment disorder with withdrawal, and adjustment disorder with work/academic inhibition) because they were “rarely used” (29). As part of the MacArthur data reanalysis conducted by Strain and colleagues (30), an analysis of the frequency distribution of DSM-III subtypes of adjustment disorder from a National Institute of Mental Health (NIMH) sample (6,654 patients diagnosed with adjustment disorder) and a database of patients seen at the Adult Psychiatry Crisis Clinic at Western Psychiatric Institute and Clinic (2,916 diagnosed with adjustment disorder) indicated that “with withdrawal” and “with work/academic inhibition” were almost never used. These findings led the work group to recommend deletion of these subtypes for simplification.

An important limitation in measuring use by looking at chart diagnoses is that they may bear little relationship to actual use of DSM or to the potential clinical utility of a diagnosis or subtype. Some clinicians may record a particular chart diagnosis to receive payment for the treatment they provide, to minimize stigmatization, or because it fits the “gestalt” of the patient, rather than because the diagnostic criteria were carefully evaluated and determined to have been met. More extensive evaluations of diagnostic procedures may be needed to draw accurate conclusions about DSM use. Such evaluations might entail requiring a much more extensive documentation of whether the diagnostic criteria have been met or conducting post hoc “debriefing sessions” in which the clinician’s diagnostic decision making is carefully examined by using cognitive analysis techniques (31).

User acceptability of a proposed change can be divided into two components, each of which can be assessed separately: 1) confidence in its diagnostic validity and 2) its ease of use. For example, the user acceptability of the DSM-IV proposal to introduce the diagnosis of Asperger’s disorder to the section on pervasive developmental disorders depends both on whether the user believes that the DSM-IV diagnosis is a valid construct and whether the criteria set is easy enough to use. However, any given clinician’s perception that a proposed diagnostic construct is valid will depend on a number of personal factors, such as familiarity with the scientific literature (relevant for deciding whether a proposed change has external validity), clinical experience (relevant for deciding about predictive validity and face validity), and practice setting (since case mix can affect a clinician’s perception of what is descriptively valid). In light of these limitations, we are certainly not arguing that diagnostic changes should be based primarily on their popularity with clinicians. Nonetheless, to ensure that proposed changes are in line with overall clinical sensibilities, some assessment of clinicians’ attitudes about the credibility of proposed changes is advisable, as long as the inherently subjective nature of such assessments is acknowledged. Furthermore, surveys should attempt to explore the reasons behind clinicians’ judgments regarding the acceptability or unacceptability (use or nonuse) of a particular disorder.

The ease of use or “user friendliness” of a proposed diagnostic change is also crucial in determining its ultimate user acceptability. This parameter covers the practical issues of applying the diagnostic system and includes the length of time it takes to assess a particular criteria set, the number and complexity of criteria that affect the likelihood that the user will be able to recall from memory a criteria set (thus obviating the need to have the DSM manual always on hand), and the ease in locating the appropriate disorder in the classification.

User acceptability of a proposed change can be measured in a number of different ways, including the following:

In addition to potentially improving user acceptability, proposed changes that make the criteria sets easier to use can improve users’ ability to correctly apply the diagnostic criteria in practice. These include proposals to simplify criteria sets or diagnostic algorithms, clarifications in the wording of an ambiguously written diagnostic criterion, changes in criteria to make them easier to assess, and modifications in exclusion criteria to more clearly delineate the differential diagnosis. For example, the diagnostic criteria for schizoaffective disorder in DSM-III-R were ambiguous in that it was unclear whether the user was supposed to focus on the current episode of illness or on the lifetime pattern when determining the temporal relationship of mood and psychotic symptoms (34). For DSM-IV, the wording of the diagnostic criteria was changed to clarify that all of these criteria concern an uninterrupted period of illness rather than the lifetime pattern of symptoms, with the expectation that clinicians using the revised diagnostic criteria will be more likely to apply them correctly from a procedural standpoint. It should be noted that the goal of these wording changes was not to improve the diagnostic validity of the schizoaffective disorder construct per se. That is, these changes were not intended to indicate that a definition of schizoaffective disorder that focuses on the relationship of mood and psychotic symptoms within a continuous episode is more valid than a definition that concentrates on the lifetime pattern of mood and psychotic symptoms. Instead, these changes to the criteria set were made to ensure that the operationalization of the schizoaffective disorder construct was consistent across different diagnostic criteria.

Assessment of whether clinicians are “correctly” applying the diagnostic criteria usually entails comparing the clinician’s application of the diagnostic criteria with an expert’s. This method has been used in a number of studies to confirm diagnostic accuracy (e.g., reference 35, in which the accuracy of the clinician’s diagnosis is measured by virtue of its agreement with the best estimate diagnosis by using a structured interview that promotes rigorous application of the diagnostic criteria). Whether the proposed change in the diagnostic criteria for schizoaffective disorder actually improves diagnostic accuracy can be assessed by comparing clinicians’ ability to agree with a Longitudinal Expert All Data (LEAD) diagnosis (36) using the DSM-III-R versus DSM-IV diagnostic criteria.

Clinicians do not use DSM simply to assign a diagnostic label (or labels) to a patient’s clinical presentation. Rather, the DSM diagnosis informs a variety of clinical decisions by both the clinician and the patient. For the clinician, these decisions include selection of a particular setting for treatment (e.g., inpatient versus outpatient), mode of treatment (e.g., somatic and/or psychosocial), and expected duration of treatment (e.g., brief versus long-term). Patients may also alter their behavior once their psychiatric diagnosis and its implications are explained to them (e.g., avoiding known triggers of acute episodes of illness). Most changes that entail the addition of a new disorder or new subtype are intended to highlight a homogeneous (and previously unidentified) group of patients that require special clinical attention. For example, the addition in DSM-IV of the rapid cycling subtype for bipolar disorder was intended to alert clinicians, patients, and their families to this important subgroup of bipolar patients and to encourage clinicians, patients, and their families to follow certain management guidelines (e.g., for clinicians, measurement of thyroid hormone, use of valproate in lieu of lithium, and exercise of extra caution in the use of antidepressants, and for patients, the avoidance of potential triggers such as substance use or sleep deprivation).

Assessments of whether clinicians are making the appropriate clinical decisions can be done by using methods developed for evaluating the degree of adherence to practice guidelines. Such methods involve, first, the delineation of “quality indicators,” which consist of specific accessible data elements that indicate a reasonable likelihood of concordance with a particular guideline (37, 38). These indicators usually consist of a denominator specifying the population to which the guideline applies (e.g., all individuals with an insurance claim for bipolar disorder) and a numerator suggesting conformance with the guidelines (e.g., all individuals in the denominator who have been given a prescription for a mood stabilizer). Data can be collected from clinician or patient surveys, systematic chart abstraction, or secondary analyses of electronic medical records or claims data.

There are a number of ways to design studies to examine the impact of a proposed change on clinical decision making. The ideal design would document an increased incidence of the desired clinical decision in a randomized controlled trial in which patients are randomly assigned to one of two groups: patients diagnosed according to the existing diagnostic classification and patients diagnosed according to the proposed changes. This design has the advantage of controlling for other factors that might affect the clinical decision-making process. Other designs that are less ideal but still useful could examine claims forms, encounter data, or patient charts to document increased use of the proposed diagnostic specifier in association with the desired clinical decision (e.g., use of valproate associated with use of the rapid-cycling specifier).

For example, the “atypical features” specifier was added to DSM-IV because it identified a subgroup of depressed patients that respond less well to tricyclic antidepressants (39). In order to determine whether this addition to DSM-IV is clinically useful (in regard to the effect of the change on clinical decision making), one could conduct a randomized controlled trial that would examine the rates of use of tricyclic antidepressants in groups of depressed patients with a mix of subtypes who were randomly assigned to groups according to whether they are diagnosed according to DSM-III-R or DSM-IV. Improvement in clinical utility could be demonstrated if the rates of use of tricyclic antidepressants in patients diagnosed according to DSM-IV criteria (some of whom would have been identified as having atypical features and thus not given tricyclic antidepressants) were shown to be lower compared to patients diagnosed according to DSM-III-R criteria (none of whom would have been labeled as atypical). A study examining chart data for an association between the atypical features specifier and less frequent use of tricyclic antidepressants would be problematic since it would not be able to account for other reasons for lower use of tricyclic antidepressants (e.g., the potential for lethality in overdose and increased rates of problematic side effects).

The “holy grail” of clinical utility is the positive effect of a change in the diagnostic system on outcome. Improvement in clinical outcome can be assessed by measuring reduction in symptom severity, measuring improvement in functioning, or documenting the prevention of a future negative outcome (e.g., reduction in relapse rates over a period of time). Given the number and complexity of the determinants of outcome, it has traditionally not been possible to empirically demonstrate that a change in diagnostic practices results in an improvement in outcome because of the very large group sizes that would be required to demonstrate a difference among groups. However, in response to severe limitations in the utility and generalizability of typical clinical trials, the NIMH has initiated several large-scale, public-health-oriented clinical trials that have a more naturalistic design (40). In such trials, diagnostic algorithms are typically tested in a wide variety of clinical settings, comparing a “treatment-as-usual” arm with various permutations of the diagnostic algorithm. Trials of this type may potentially provide an opportunity to relate clinical outcome to diagnostic innovations (e.g., treatment-oriented specifiers such as rapid cycling, seasonal pattern, and atypical features). Studies that employ treatment algorithms that incorporate existing or proposed DSM diagnostic considerations in their decision points (i.e., if the patient has a particular DSM subtype, then institute a particular treatment) may allow for an examination of the effects of a diagnostic change on outcome by allowing comparison with a treatment-(and diagnosis)-as-usual arm.

To illustrate how the impact of a particular change on clinical utility might be measured, we will use as examples two changes proposed for, and ultimately included in, DSM-IV.

In comparison to DSMs of the past, the DSM-IV revision process moved from an almost sole reliance on expert consensus to a greater reliance on reviews of published literature, data set reanalyses, and field trials focused on diagnostic issues. Although many of the changes in DSM-IV aimed to improve its clinical utility, the focus of the DSM-IV literature reviews and empirical studies were almost exclusively on diagnostic validity. Given the important role that DSM serves in facilitating clinical practice, an equally crucial target for evaluating the advantages and disadvantages of a particular change is its effect on clinical utility. Improvements in clinical utility can be measured in terms of 1) their impact on the use of DSM, 2) their enhancement of clinical decision making, and 3) whether they lead to improvement in clinical outcomes. Given the current lack of standardized instruments to measure user acceptability and ease of use, efforts should be made to develop psychometrically sound assessment tools that can be used in future studies. Only by applying empirical methods for assessing improvement in clinical utility can it be demonstrated that purported improvements in clinical utility are, in fact, achievable and that the magnitude of these improvements in clinical utility justifies the inevitable disruption that comes with changing the diagnostic criteria.

Received Jan. 22, 2003; revision received April 2, 2003; accepted July 15, 2003. From the Department of Psychiatry, Columbia University, New York; Western Psychiatric Institute and Clinic, University of Pittsburgh, Pittsburgh; Rand Corporation, Pittsburgh; the Department of Psychiatry, Harvard Medical School, Boston; Classification, Assessment Surveys, and Terminology, World Health Organization, Geneva, Switzerland, and the Psychiatry Department, George Washington University, Washington, DC. Address reprint requests to Dr. First, New York State Psychiatric Institute, 1051 Riverside Dr., Unit 60, New York, NY 10032; [email protected] (e-mail). Dr. Pincus is supported in part by the NIMH-funded Mental Health Intervention Research Center on Mood and Anxiety Disorders, the University of Pittsburgh, and the Robert Wood Johnson Foundation. The authors comprise an ad hoc subcommittee of the American Psychiatric Association’s Committee on Psychiatric Diagnosis and Assessment. This subcommittee was given the charge of considering the role of clinical utility and approaches to developing an associated evidence base in future revisions of DSM. Although the views expressed in this article have been endorsed by the Committee on Psychiatric Diagnosis and Assessment, they do not necessarily represent the policies and opinions of the American Psychiatric Association.