Reliability and Validity of Proposed DSM-5 ADHD Symptoms in a Clinical Sample of Adults
Publication: The Journal of Neuropsychiatry and Clinical Neurosciences
Abstract
The DSM-5 ADHD and Disruptive Behaviors Work Group proposed two major changes for diagnosis of attention deficit hyperactivity disorder (ADHD) in adults: 1) inclusion of four new impulsivity symptoms and 2) reduction in the number of symptoms required for assigning an ADHD diagnosis. In this case-control study, the performance of these modifications was assessed in a clinical sample of 133 adult subjects (68 ADHD cases and 65 non-ADHD control subjects). The proposed new impulsivity symptoms for adults do not improve ADHD diagnosis enough to overcome potential negative effects of changing the criteria. However, fewer symptoms than the six-of-nine threshold required by DSM-IV provided the best cutoff point for identifying adults who are impaired.
APA launched DSM-5 in May 2013.1 In 2010, the APA ADHD and Disruptive Behavior Disorders Work Group (referred to hereafter as the DSM-5 Work Group) proposed changes to the criteria for attention deficit hyperactivity disorder (ADHD).
DSM-IV ADHD criteria and the proposed DSM-5 criteria were not tested in field trials with adults.2,3 We focus here on two of the 2010 proposed changes that substantially affect adults: 1) reducing the number of required symptoms from six to four in both ADHD dimensions (inattention and hyperactivity/impulsivity) for adult patients and 2) the addition of four new impulsivity symptoms.
Previous investigations documented a normative decrease in ADHD symptoms across the life span in both clinical and community samples. Nevertheless, impairment resulting from the symptoms tends to persist,4–6 even when the number of remaining symptoms is fewer than required to assign the diagnosis. Reducing the number of required symptoms to four (rather than six) in both ADHD dimensions has been proposed as more appropriate for adult patients, capturing a significant proportion of individuals with clinically relevant impairment from ADHD symptoms.7–9 Considering this, the DSM-5 Work Group initially suggested a four-symptom cutoff on either list for assigning ADHD as a diagnosis for adults. Although this proposal was supported by previous research, it raised concerns about the possibility of artificially increasing false positives and the prevalence of the disorder.10
Another area of debate concerning the diagnosis of ADHD in adults is which set of symptoms best captures the clinical presentation of ADHD after childhood. It is not clear which DSM-IV ADHD symptoms are more robustly associated with the ADHD diagnosis or with clinical impairment. Concerns have been raised about whether relevant symptom groups that cause impairment for adults (e.g., executive dysfunction, emotional dysregulation, and impulsivity) are adequately captured in the existing symptom set.11,12 To address such concerns, the DSM-5 Work Group initially suggested including four additional impulsivity symptoms to the already large list of ADHD symptoms to be tested in field trials. These symptoms were as follows: acts without thinking, is often impatient, is uncomfortable doing things slowly and systematically and often rushes through activities or tasks, and finds it difficult to resist temptations or opportunities, even if it means taking risks. The proposal of adding these new symptoms would raise the question of whether it would be better to 1) replace old with new symptoms to maintain a total of nine and thus avoid having more symptoms of hyperactivity/impulsivity than of inattention or 2) increase the total hyperactive/impulsive symptoms set length from nine to 13 items. In either event, the effects on internal and external validity would need evaluation.
In this study, we aimed to evaluate the reliability and validity of the 2010 DSM-5 Work Group proposed modifications for ADHD criteria in adults in a clinical sample. Because reliability is a prerequisite to validity, we first examined test–retest reliability. We then addressed the following research questions. First, does the inclusion of four new impulsivity symptoms make a three-factor solution (allowing a better representation of impulsivity) a better fit for data than the traditional two-factor solution found with DSM-IV criteria? Second, are the proposed DSM-5 ADHD symptoms more specific for ADHD (versus comorbid psychiatric conditions)? Third, what set of DSM-5 ADHD symptoms best captures the clinical diagnosis of ADHD and/or impairment, and do the four new proposed impulsivity symptoms add value? Finally, what is the best number (cutoff point) of DSM-5 ADHD symptoms to identify impaired adults?
It is important to bear in mind that the DSM-5 Work Group proposed other modifications in ADHD criteria in 2010 that might affect adults, such as raising the age of onset criterion13–15 and rewording the original 18 DSM-IV symptoms to make them more adequate for adults.2 Thus, when using 2010 DSM-5 ADHD proposed criteria, we always required an age of onset before age 12 years and used the reworded symptoms. In addition, because of the rewording, we assessed the performance of all proposed DSM-5 symptoms, rather than only the four new ones.
Methods
Participants
The sample for this case-control study includes 133 participants: 68 patients seen in an ADHD outpatient clinic and assessed as having ADHD by DSM-IV criteria and 65 comparison participants (10 of which were outpatients without ADHD and 55 were non-ADHD control subjects).
A consecutive sample of 78 adults seeking treatment was evaluated for ADHD in the ADHD Outpatient Program at the Hospital de Clínicas de Porto Alegre (Porto Alegre, Brazil), from July to December 2011. Of these individuals, 10 did not meet DSM-IV diagnostic criteria for ADHD, leaving 68 subjects who were included in this study as the ADHD group. The inclusion criteria were as follows: 1) native Brazilians of European descent, 2) age ≥18 years, and 3) fulfillment of DSM-IV diagnostic criteria for ADHD, both currently and during childhood. The exclusion criteria were as follows: 1) evidence of clinically significant neurologic diseases (e.g., delirium, dementia, epilepsy, head trauma, or multiple sclerosis) and 2) current or past history of psychosis. All measurements were performed after recruitment and before the initiation of treatment for ADHD.
The control sample comprised 65 subjects: 55 adults who were recruited at the blood bank of the same hospital and the 10 above-described ADHD-negative subjects. The inclusion and exclusion criteria were the same for controls as for cases, except for the absence of DSM-IV ADHD diagnosis. The University Hospital Institutional Review Board approved the project, and all subjects (ADHD cases and controls) provided written informed consent.
Assessment Procedures
All interviewers in this study were psychiatrists who had extensive training in the application of all of the instruments of the research protocol. The ADHD and control subjects were assessed using the same protocol.
The information about DSM-5 ADHD criteria was collected through a clinical interview, during which all symptoms were formulated exactly as written in the 2010 DSM-5 Work Group proposal and rated by the clinician as present or absent. To evaluate criteria stability over time, a reevaluation of DSM-5 ADHD criteria was performed in 18 ADHD cases 15 days after the first interview by the same psychiatrist who had previously interviewed the patient.
DSM-IV ADHD and oppositional-defiant disorder criteria were assessed using the Portuguese version of the Schedule for Affective Disorders and Schizophrenia for School-Age Children–Epidemiological Version,16 as usually applied for adolescents.17 The severity of ADHD and oppositional-defiant disorder symptoms was assessed by the Brazilian version of the SNAP-IV Rating Scale,18 in which the frequency of each DSM-IV symptom is rated from zero (not at all) to three (very much).
The diagnoses of conduct and antisocial personality disorders were assessed using the Portuguese version of the Mini-International Neuropsychiatric Interview, a short semistructured diagnostic interview for DSM-IV and ICD-10 psychiatric disorders.19 All other lifetime psychiatric disorders were assessed using the Portuguese version of the Structured Clinical Interview for DSM-IV (Research Version).20
Clinical impairment was measured by the Sheehan Disability Inventory,21 a clinician-rated Likert scale from zero (not at all) to 10 (extremely), with items addressing the effect of symptoms on three areas of functioning: work, social, and family life. Impairment was defined as a score of ≥5 in any of the three dimensions, as previously suggested.22
All instrument-derived diagnoses were rechecked and confirmed by a clinical committee composed by the psychiatrist interviewers and the head of the staff (E.H.G.), as previously described.23 The research protocol also included the assessment of demographic and educational data, medical history, and social problems.
Data Analytic Strategies
To test whether the ADHD sample in this study adequately represents our general ADHD outpatient clinic population, the ADHD cases in this study were compared with all other ADHD cases from the ADHD outpatient clinic in terms of demographic and clinical profiles using the two-proportion z test for categorical variables and the independent-samples t test for quantitative variables.
The test–retest reliability of DSM-5 ADHD symptoms was assessed with Cohen’s kappa coefficient calculation.24
To evaluate the change in clinical and demographic profiles with DSM-5 proposed criteria, we compared cases fulfilling DSM-IV criteria with additional cases that would only be identified after DSM-5 proposed criteria application (with a threshold of four of nine inattention symptoms and four of 13 hyperactivity/impulsivity symptoms for the diagnosis). These independent groups were compared using the two-proportion z test for categorical variables and the two-sample t test for quantitative variables.
The factor structure of the ADHD symptoms was tested with confirmatory factor analysis solutions with two versus three correlated factors; this was done for the 18 DSM-IV symptoms as well as the 22 DSM-5 symptoms. This strategy examines whether the factor structure would change with the addition of the four new symptoms and whether a three-factor solution (with impulsivity emerging as a separate factor from hyperactivity) has a better fit when the four new symptoms are included than when they are not.
The performance of the proposed DSM-5 ADHD symptoms in predicting clinical DSM-IV ADHD diagnosis was tested to evaluate how well the symptoms conform to the DSM-IV definition of ADHD. This was done in a three-step approach. First, the bivariate association between individual DSM-5 symptoms and DSM-IV ADHD diagnosis (clinician derived) was assessed with odd ratios estimates, which allowed ranking the DSM-5 symptoms according to their unadjusted odds ratio related to DSM-IV ADHD diagnosis. At this point, the association between DSM-5 symptoms and other psychiatric disorders was also tested with the same statistics. Thus, for each DSM-5 item, the odds ratios of the associations with ADHD and with comorbidity could be compared.
After the initial bivariate step, a binary stepwise logistic regression model with forward entrance was performed, considering DSM-IV ADHD diagnosis as the dependent variable and all DSM-5 ADHD proposed symptoms as independent variables. This analysis allowed us to find how many and which DSM-5 symptoms are independently associated with DSM-IV ADHD diagnosis after controlling for the other DSM-5 symptoms.
In the third step, all possible subsets (APS) logistic regression analysis was used to confirm the set of DSM-5 symptoms that best predicted DSM-IV ADHD diagnosis. The APS analysis helps to select the best subset from a larger set of predictors. In such situations, different subsets might have almost equivalent associations with the outcome, and conventional stepwise regression analysis might select a suboptimal subset owing to minor differences in bivariate associations. The APS analysis protects against this problem because it generates results for a large number of different models with a fixed number of predictors, which was determined from the earlier stepwise logistic regression analysis. The APS analysis also ranks the best subsets according to their association with the outcome (using the chi-square as the ranking criterion). Once the ranking of subsets is known, the researcher can select the predictors that are more consistent across the top-ranked subsets. In our APS analyses, we strictly followed the procedures by Kessler et al.12 to allow comparability of the findings.
The same three-step statistical approach was used to test the performance of the DSM-5 proposed symptoms in predicting clinical impairment.
Finally, receiver-operating characteristic curves were built for testing the best cutoff (best balance between sensitivity and specificity) for the number of DSM-5 ADHD symptoms to predict clinical impairment (considering the above-described impairment cutoff). The following strategy was used: 1) initial comparison between DSM-IV and DSM-5 criteria in terms of general performance in predicting impairment (area under the curve) and 2) determination of the best cutoffs for predicting impairment separately considering DSM-5 inattention and hyperactivity/impulsivity symptoms. These models all assumed equal weighting of false positive and false negative errors in selecting the optimal cutpoint.
For all analyses, a 5% significance level for a two-tailed test was adopted (p<0.05). The analyses were performed in SAS (version 9.0; SAS Institute, Cary, NC), Signal Detection Software ROC4, Mplus (version 6.11; Muthén and Muthén, Los Angeles, CA), and SPSS (version 20; SPSS, an IBM Company, Chicago, IL) software for APS analysis, receiver-operating characteristic curves, factor analysis, and other analyses, respectively.
Results
Clinical Representativeness of the Sample
The 68 ADHD cases in this study did not differ clinically or statistically from the general 440 ADHD outpatient clinic cases in terms of gender distribution, age, or marital, economic, or academic status. There were no significant differences between these groups in terms of ADHD clinical presentation (both type distribution and ADHD severity) and comorbidity profile (data not shown, but available on request).
Test–Retest Reliability of DSM-5 Proposed Criteria
Table 1 displays the results for test–retest reliability of DSM-5 criteria. The test–retest reliability was perfect (100% agreement) for the diagnosis of ADHD. Reliability varied widely for individual symptoms. Among inattention symptoms, all but three (fails to give close attention to details, reluctant to engage in mental tasks, and easily distracted) reached a moderate test–retest reliability with kappa coefficients >0.45 (p<0.05). In general, hyperactivity/impulsivity symptoms tended to have more substantial test–retest reliability than inattentive symptoms. However, three of the four new impulsivity symptoms failed to reach significant test–retest reliability; as a group, their kappa coefficient was smaller (0.12–0.44) than the kappa coefficient of the hyperactivity/impulsivity symptoms that were already present in DSM-IV (0.51–0.77).
| Symptom no. | DSM-5 symptom | Test–retest reliability, kappa coefficient (N=18) | Performance in predicting impairment, DSM-IV ADHD, and comorbidities (N=133) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Symptom versus impairment | Symptom versus DSM-IV ADHD | Symptom versus comorbidities, chi-squareb | |||||||||||||
| Chi-squareb | Stepwise regressionc | APS regressiond | Chi-squareb | Stepwise regressionc | APS regressione | ODD | ASPD | Bipolar | Depression | Anxiety | Alcohol | Other SUD | |||
| 1 | Fails to give close attention to details | 0.308 | 33.1 (4th) | 2 | 40.8 (5th) | 0 | 5.7* | NA | 4.4* | 1.6 | 2 | 9.8* | 5.1 | ||
| 2 | Difficulty sustaining attention | 1.000* | 34.3 (2nd) | 0 | 58.3 (3rd) | 1 | 5.3* | NA | 2.1 | 1.9 | 3.1* | 7 | 3.7 | ||
| 3 | Does not seem to listen | 0.766* | 16.6 (7th) | 1 | 20.6 (10th) | 0 | 1.9 | 1.8 | 2.2 | 1.3 | 2.3* | 8.3* | 9.8* | ||
| 4 | Does not follow through | 0.455* | 30.1 (5th) | 3 | 62.1 (2nd) | 29.8 (4.1–218.1) | 10 | 2.9* | 3 | 4.6* | 1.4 | 1.4 | 10.1* | 1.1* | |
| 5 | Difficulty organizing tasks | 0.600* | 16.5 (8th) | 3.4 (1.1–11.1) | 1 | 18.9 (11th) | 0 | 3.1* | NA | 2.3 | 1.15 | 1.3 | 4.6 | 5.6 | |
| 6 | Reluctant to engage in “mental” tasks | 0.438 | 34.3 (2nd) | 6.3 (1.8–21.9) | 8 | 42 (4th) | 1 | 5.3* | NA | 5.3* | 1.9 | 1.3 | NA | NA | |
| 7 | Loses objects | 0.557* | 9.7 (12th) | 0 | 18.6 (12th) | 6.7 (1.3–33.8) | 8 | 2 | 3.8 | 1.5 | 1.1 | 1.9 | 5.4* | 2.5* | |
| 8 | Easily distracted | 0.341 | 50.0 (1st) | 10.7 (3.0–38.1) | 10 | 95.7 (1st) | 11.4 (1.5–83.6) | 10 | 8* | NA | 2.2 | 2 | 2 | 7.3 | NA |
| 9 | Forgetful | 0.455* | 15.1 (10th) | 0 | 24.9 (8th) | 2 | 3.3* | NA | 4.2* | 1.6 | 1.6 | 13.1* | NA | ||
| 10 | Fidgets | 0.507* | 4.2 (19th) | 0 | 7.3 (19th) | 2 | 1.2 | NA | 2.2 | 0.8 | 2.5* | 2.5 | 5.4 | ||
| 11 | Gets up | 0.775* | 12.7 (11th) | 1 | 15.2 (13th) | 0 | 1.6 | 8.1 | 1.8 | 1.4 | 3.7* | 2.7 | 2.6 | ||
| 12 | Runs about | 0.667* | 16.8 (6th) | 2 | 29.5 (6th) | 14.7 (1.9–115.3) | 5 | 3.5* | 6.7 | 3.7* | 1.1 | 3.1* | 3.5 | 11.7* | |
| 13 | Excessively loud | 0.723* | 2.4 (22nd) | 0 | 3.4 (22nd) | 0 | 0.9 | 2.5 | 1.2 | 1.1 | 2.1 | 1 | 2.5 | ||
| 14 | On the go | 0.649* | 4.6 (17th) | 0 | 5.5 (21st) | 0 | 2.2 | NA | 2.6 | 0.9 | 2.1* | 1.4 | 4.3 | ||
| 15 | Talks excessively | 0.750* | 3.6 (21st) | 0 | 6.3 (20th) | 0 | 2.3 | 2.4 | 1.2 | 0.9 | 1.3 | 0.6 | 2.4 | ||
| 16 | Blurts out answers | 0.556* | 9.1 (13th) | 0 | 8.4 (17th) | 0 | 3.2* | NA | 4* | 1.1 | 1.5 | 5.6* | 6.8 | ||
| 17 | Difficulty waiting his turn | 0.675* | 9.0 (14th) | 0 | 14.4 (14th) | 0 | 5.9* | 2.7 | 2.6 | 0.9 | 2.8* | 2.8 | 2.7 | ||
| 18 | Interrupts or intrudes | 0.557* | 7.3 (16th) | 0 | 22 (9th) | 4 | 3* | NA | 2.5 | 1.4 | 1.2 | 2.7 | 5.5 | ||
| 19 | Acts without thinking | 0.348 | 8.6 (15th) | 0 | 10.1 (15th) | 0 | 3.4* | 4.9 | 3.6* | 1.3 | 1.9 | 2.5 | 8.5 | ||
| 20 | Impatient | 0.118 | 4.4 (18th) | 0 | 7.8 (18th) | 0 | 2.3 | NA | 2.1 | 1.2 | 2.4* | 4.3 | 5.2 | ||
| 21 | Uncomfortable doing things slowly | 0.438 | 15.3 (9th) | 1 | 26.3 (7th) | 0 | 4* | NA | 5.1* | 1.4 | 1.8 | 2.7 | NA | ||
| 22 | Difficulty to resist temptations | 0.444* | 4.0 (20th) | 0 | 9.5 (16th) | 18.4 (2.0–116.1) | 7 | 1.9 | 9.1 | 2 | 1.6 | 1.7 | 2 | 3 | |
a
ASPD, antisocial personality disorder; NA, odds ratio not available (no subject in the sample had the comorbidity without the DSM-5 ADHD symptom); ODD, oppositional-defiant disorder; SUD, substance use disorder (abuse or dependence).
b
Unadjusted odds ratios derived from chi-square statistics are shown with odds ratios ranking in parentheses. All are significant at p<0.05.
c
Stepwise regression values are shown as adjusted odds ratios with 95% confidence intervals. All are significant at p<0.05.
d
APS regression is shown as the number of appearances among the 10 top-ranked three-item subsets.
e
APS regression is shown as the number of appearances among the 10 top-ranked five-item subsets.
*
p<0.05.
Factor Structure of DSM-5 Proposed Criteria
Table 2 displays the factor structure of DSM-5 proposed criteria. For the list of 18 DSM-IV ADHD symptoms only, a three-factor solution provided a trend for a better fit for the data than a two-factor solution. For the list of 22 proposed DSM-5 ADHD symptoms, a three-factor solution provided a significantly better fit for the data than a two-factor solution. Nonetheless, both a two-factor solution and a three-factor solution yielded a satisfactory fit for the 18 DSM-IV ADHD symptoms as well as for the 22 DSM-5 ADHD symptoms.
| Analysis | Chi-square/df | RMSEA(90% confidence interval) | CFI | TLI | Chi-square for three- versus two-factor solution |
|---|---|---|---|---|---|
| DSM-IV symptoms (N=18) | 5.90 (p=0.05) | ||||
| Two-factor solution | 1.37 | 0.05 (0.03–0.07) | 0.99 | 0.99 | |
| Three-factor solution | 1.35 | 0.05 (0.03–0.07) | 0.99 | 0.99 | |
| DSM-5 symptoms (N=22) | 13.31 (p=0.001) | ||||
| Two-factor solution | 1.31 | 0.05 (0.03–0.07) | 0.99 | 0.99 | |
| Three-factor solution | 1.23 | 0.04 (0.02–0.06) | 0.99 | 0.99 |
a
CFI, comparative fit index (range, 0–1; values >0.90 indicate acceptable model fit); RMSEA, root mean square error of approximation (range, 0–1; values <0.06 indicate acceptable model fit); TLI, Tucker-Lewis index (values >0.90 indicate acceptable model fit).
Agreement Between DSM-IV and DSM-5 ADHD Diagnoses
Overall, the agreement between DSM-IV and DSM-5 ADHD diagnoses was high (kappa=0.82, p<0.001). The number of ADHD cases rose from 68 with DSM-IV criteria (51.1% of the total sample) to 80 with DSM-5 criteria (60.2% of the total sample), because 12 subjects that were not cases according to DSM-IV were defined as cases with DSM-5 criteria application.
Change in Clinical and Demographic Profiles With DSM-5 Proposed Criteria
Table 3 presents the clinical and demographic profile of the 68 ADHD cases according to DSM-IV and the 12 additional cases using DSM-5 criteria. Both groups had generally a similar demographic profile. However, additional DSM-5–derived cases had lower ADHD severity and disability scores. As expected, the 12 additional DSM-5 ADHD cases had a higher proportion of hyperactive/impulsive presentation and a lower (although not statistically significant) proportion of inattentive presentation (25% for inattentive plus restrictive inattentive versus 50%, p=0.10). The comorbidity pattern was generally similar between the two groups, except for conduct disorder, which was absent in the additional DSM-5 cases.
| Characteristic | Cases according to DSM-IV | Additional cases using DSM-5 |
|---|---|---|
| No. of cases | 68 | 12 |
| Male gender | 47.1 | 25 |
| Age (years) | 32.4±1 | 34.8±13.3 |
| Marital status (single) | 51.5 | 41.7 |
| Economic status | ||
| Upper middle class | 30.9 | 16.7 |
| Middle class | 51.5 | 50 |
| Lower middle class | 16.2 | 33.3 |
| Poor | 1.5 | 0 |
| Academic status | ||
| Incomplete elementary school | 1.5 | 0 |
| Elementary school | 4.4 | 8.3 |
| High school | 55.9 | 66.6 |
| University | 38.2 | 25 |
| ADHD presentation | ||
| Restrictive inattentive | 16.7 | |
| Inattentive | 50 | 8.3 |
| Hyperactive/impulsive | 2.9 | 25* |
| Combined | 47.1 | 50 |
| ADHD severity (SNAP score) | 1.51±0.53 | 0.99±0.47* |
| Disability (Sheehan score) | 5.85±2.04 | 2.58±1.99* |
| Comorbidities | ||
| ODD | 30.9 | 33.3 |
| Conduct disorder (aged <15 years) | 26.5 | 0* |
| ASPD | 5.9 | 0 |
| Depressive disorders | 33.8 | 41.7 |
| Bipolar disorders | 16.2 | 16.7 |
| Anxiety disorders | 42.6 | 41.7 |
| Alcohol (abuse or dependence) | 13.2 | 8.3 |
| Other substance (abuse or dependence) | 8.8 | 0 |
a
Data are expressed as percentages or mean ± standard deviation unless otherwise specified. ASPD, antisocial personality disorder; ODD, oppositional-defiant disorder.
*
p<0.05 for the comparison between DSM-IV and DSM-5 cases.
After documenting lower symptom severity and clinical impairment scores for DSM-5 additional ADHD cases, we performed secondary analyses addressing whether these cases are more severe and impaired than the controls. Mean SNAP scores were 1.51 (SD=0.53), 0.99 (SD=0.47), and 0.28 (SD=0.24) for DSM-IV ADHD cases (N=68), DSM-5 additional ADHD cases (N=12), and control subjects (N=53), respectively. For the same groups, mean Sheehan Disability scores were 5.85 (SD=2.04), 2.58 (SD=1.99), and 0.86 (SD=1), respectively. In analyses of variance, there were significant differences among the groups in both ADHD severity (F=119.5, df=130, 2, p<0.001) and clinical impairment (F=130.3, df=130, 2, p=0.001). In post hoc analyses with Bonferroni tests, means for all three groups remained significantly different from each other for both severity and impairment.
DSM-5 Proposed Symptom Performance in Predicting Clinical Impairment
Table 1 displays results of the performance of the 22 DSM-5 symptoms in predicting clinical impairment. Among the best 10 symptoms in bivariate tests (according to unadjusted odds ratio ranking), there were eight inattention symptoms (all inattention symptoms, except loses objects), one hyperactivity symptom (runs about), and one of the new impulsivity symptoms (uncomfortable doing things slowly).
In the conventional logistic regression analysis, a final model with three symptoms had the highest goodness of fit. This solution had a generalized R2 of 0.694 and a chi-square of 82.177. Three inattention symptoms (easily distracted, reluctant to engage in mental tasks, and difficulty organizing tasks) remained as independent predictors of clinical impairment.
In the APS regression, among the 10 top-ranked subsets of three symptoms, the three most recurrent predictors of impairment were also inattention symptoms (easily distracted, reluctant to engage in mental tasks, and does not follow through).
DSM-5 Proposed Symptom Performance in Predicting DSM-IV ADHD Diagnosis and Comorbidities
Table 1 displays results for the DSM-5 symptom performance in predicting DSM-IV ADHD. Among the best 10 symptoms in bivariate tests (according to unadjusted odds ratio ranking), there were seven inattention symptoms (all inattention symptoms, except loses objects and difficulty organizing tasks), one hyperactivity symptom (runs about), one of the impulsivity symptoms already present in DSM-IV (interrupts or intrudes), and one of the new impulsivity symptoms (uncomfortable doing things slowly).
In the conventional logistic regression analysis, a final model with five symptoms had the highest goodness of fit. This solution had a generalized R2 of 0.872. Three inattention symptoms (loses objects, easily distracted, and does not follow through), one hyperactivity symptom (runs about), and one new impulsivity symptom (difficulty to resist temptations) remained as independent predictors of DSM-IV ADHD.
In the APS regression, among the 10 top-ranked subsets of five symptoms, the five most recurrent predictors of DSM-IV ADHD were three inattention symptoms (does not follow through, easily distracted, and loses objects), one hyperactivity symptom (runs about), and one new impulsivity symptom (difficulty to resist temptations).
DSM-5 symptoms were also tested in their association with other psychiatric disorders (Table 1). The odds ratios for these associations were much smaller than for the association with ADHD, and <30% of the associations between individual DSM-5 symptoms and comorbidity were statistically significant (even without any adjustment for multiple tests). The new impulsivity symptoms performed similarly to the other DSM-5 symptoms in this issue.
Best Cutoff of ADHD Symptoms to Predict Impairment
Figure 1 displays these results. The general performance in predicting clinical impairment was very similar using all 22 DSM-5 symptoms or only 18 DSM-5 symptoms (excluding the four new symptoms). Adding four new impulsivity symptoms did not improve the overall accuracy for identifying impaired individuals. Therefore, we performed additional receiver-operating characteristic curves including only the 18 DSM-5 symptoms that were already present in DSM-IV (but were reworded with new examples in the proposed DSM-5 criteria).
FIGURE 1. Receiver-Operating Characteristic Curves for DSM-IV and DSM-5 Symptoms in Predicting Impairmenta
aAUC, area under the curve; ROC, receiver-operating characteristic; Sens, sensitivity; Spec, specificity.
Considering the inattentive dimension, the best cutoff point for identifying impairment was five symptoms. For the hyperactivity/impulsivity dimension (again, using only nine symptoms total and omitting the four proposed new impulsivity symptoms), the best cutoff point was three symptoms.
Discussion
This seems to be the first comprehensive assessment of the two main modifications proposed by the DSM-5 Working Group affecting ADHD criteria in adults. Ghanizadeh25 recently documented an 11% increase in children and adolescents with ADHD with DSM-5 criteria, a 0.75 kappa coefficient agreement between DSM-IV and DSM-5 diagnoses, and a low specificity of the four new impulsivity symptoms for the diagnosis of ADHD.
In our study, most DSM-5 proposed symptoms had test–retest reliability with kappa coefficients in the moderate range (the majority of them in the range of 0.4–0.7). Recently, a kappa coefficient between 0.4 and 0.6 was considered a realistic goal and between 0.2 and 0.4 was considered still acceptable for DSM-5 diagnoses, based on similar test–retest reliability reported in the medical literature.26 As a group, the four new impulsivity items had worse test–retest reliability compared with other hyperactivity/impulsivity items, and the “impatient” symptom presented very low reliability.
Our findings regarding the factor structure of DSM symptoms confirm those from previous studies in adult samples.27 A three-factor solution provided a slightly better fit to the data than the two-factor solution for the DSM-IV original set of 18 symptoms and a more pronounced better fit for the DSM-5 set of 22 symptoms. However, the standard two-factor solution fits sufficiently well in both cases. Thus, there is little practical advantage for a three-factor model.
Using the 2010 APA proposal for DSM-5 criteria for ADHD, we identified 12 additional subjects as ADHD cases, which was a natural consequence of a lower threshold of symptoms (decrease from six to four for both dimensions) and a higher number of potential impulsive symptoms (e.g., positive diagnoses more easily achievable in the hyperactive/impulsive dimension with the threshold of four of 13 than with six of nine proposed in DSM-IV). The proportion of women was higher among these 12 additional ADHD cases. Applying both proposed changes in ADHD criteria for adults at the same time captured new cases that had less severe ADHD and were less clinically impaired. Overall, these findings suggest that using the proposed changes together might change the ADHD case profile. However, these new ADHD cases are significantly more symptomatic and impaired than control subjects. Our findings concur with previous studies suggesting that DSM-IV criteria with a six-symptom cutoff may capture only the most extreme portion of adult ADHD cases.8,11
The proposal of including four new impulsivity symptoms may have negative consequences for both clinical and research settings (e.g., difficulty in interpreting data from former classificatory systems, need to change symptomatic scales, prevalence changes, or deriving ADHD cases with a different response pattern to treatment strategies). Therefore, it is important to test whether these symptoms might have stronger associations with ADHD diagnosis and/or impairment compared with traditional ADHD symptoms. In our study, the three-step analysis revealed that inattentive symptoms were in general better predictors of clinical impairment and ADHD diagnosis than hyperactivity/impulsivity symptoms. No consistent pattern of association was detected with either ADHD diagnosis or with clinical impairment for the four new impulsivity symptoms as a group.
It is also important to note that none of the 22 DSM-5 symptoms had a weaker association with ADHD diagnosis than with comorbidities, concurring with findings from Kessler et al.,12 who reported that their set of best symptoms was more related to ADHD than to comorbidity. That study addressed the best predictors of ADHD diagnosis among DSM-IV ADHD symptoms plus 14 non-DSM symptoms (mostly related to executive functions). All inattention symptoms had a stronger association with ADHD diagnosis than hyperactivity/impulsivity symptoms (odds ratio, 24.1–694.6 versus 1.8–16.8). In an APS regression including all items, three non-DSM symptoms (difficulty prioritizing work, trouble planning ahead, and cannot work unless under a deadline) and one DSM inattention item (difficulty sustaining attention) were the best predictors of narrowly defined ADHD (meeting full childhood and adulthood criteria). None of the hyperactivity or impulsivity symptoms were among the best predictors of ADHD.
Barkley et al.11 reported similar findings. Among DSM-IV ADHD symptoms, one single inattention symptom (easily distracted) was sufficient for discriminating adults with ADHD from community controls in a study conducted at the University of Massachusetts. A set of three inattentive symptoms (fails to give close attention to details, difficulty sustaining attention, and does not follow through) and one hyperactivity item (excessively loud) was sufficient for discriminating ADHD cases from clinical controls. In the same study, the five ADHD symptoms with strongest associations with the presence of any impairment were inattentive symptoms. A population-based study also recently documented that inattentive symptoms were the most strongly associated with clinical impairment.5 Overall, the specific list of best predictors of ADHD diagnosis or clinical impairment has some variation across the studies, but these results and our findings are consistent in reporting that inattention symptoms are usually better predictors than hyperactivity/impulsivity symptoms.
Clinicians will be asked to decide whether ADHD is present based on the number of symptoms in both inattention and hyperactivity/impulsivity. In DSM-IV, this cutoff point in both instances is six of nine symptoms. It has not been clear whether this cutoff is adequate in adults. In our study, a lower threshold than the one in DSM-IV was the best cutoff for capturing clinical impairment (five of nine inattentive symptoms and three of nine hyperactivity/impulsivity symptoms and not using the four new impulsivity items). These findings concur with previous studies suggesting that a lower threshold of symptoms would work better in adult ADHD samples.7,9 Another study addressed this issue in a large population-based sample (aged 18–75 years) in The Netherlands, reporting that subjects with ≥4 DSM-IV symptoms in any ADHD dimension were significantly more impaired than subjects with fewer symptoms.8
This study has some limitations. Our findings should be extrapolated with caution for nonclinical populations and other clinical samples with different cultural backgrounds. It is not possible to exclude that some of the analyses might have been underpowered, because the sample size was moderate. Particularly, our test–retest sample was small and only included cases. The same clinician applied DSM-IV and DSM-5 ADHD criteria and an impairment scale. Although this nonblind assessment might have determined a higher agreement between DSM-5 and DSM-IV results, there is no reason to expect a substantial bias in assessing individual symptom performance and determining the best cutoff point. Finally, the cutoff point determined for both dimensions was obtained allowing free variation of symptoms in the other dimension. Although we could have run conditional analyses for searching different cutoff points for ADHD-I and ADHD-H than those detected for ADHD-C, this was neither the approach used in previous studies nor the one used in current diagnostic classifications. Nonetheless, there is a scarcity of field trials of DSM-5 criteria for ADHD in adults, and this study adds important information on the reliability and validity of the 2010 DSM-5 ADHD proposed criteria.
Conclusions
Overall, our findings suggest that the proposed four new impulsivity symptoms do not improve ADHD diagnosis sufficiently to surpass potential negative effects of changing diagnostic criteria, such as modifying ADHD prevalence and correlates, needing to reconstruct ADHD scales, and losing comparability with research from the last 20 years.
On the other hand, our study reinforces that a lower cutoff point for the number of inattentive and/or hyperactive/impulsive symptoms might be more adequate for adult patients. However, this result must be balanced against the risk of potential increase in ADHD prevalence using a lower threshold of symptoms. This issue might have even more significance in the general population. The population consequences of using a lower cutoff point for ADHD diagnosis on prevalence rates and cases profile are largely unknown. Therefore, future studies testing the effect of this criterion change in large population samples are needed.
Acknowledgments
The authors thank Helena Kraemer for statistical advice, Paulo Abreu for support in data collection planning and coordination of the adult ADHD outpatient program, and Giovanni Salum Jr for support in APS analyses.
Footnote
This article was partially funded by grants from the Brazilian National Council of Technological and Scientific Development (159782/2011-2) and the National Institute of Mental Health (R01MH05150 to Drs. Fisher and Turner), as well as a contract from APA. The ADHD outpatient program is supported by grants from the Fundo de Incentivo à Pesquisa e Eventos do Hospital de Clínicas de Porto Alegre, the Brazilian National Council of Technological and Scientific Development, the Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul, and PRONEX. Dr. Grevet receives a postdoctoral scholarship from the Brazilian National Council of Technological and Scientific Development.
References
1.
APA: Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Arlington, VA, APA, 2013
2.
Matte B, Rohde LA, Grevet EH: ADHD in adults: a concept in evolution. Atten Defic Hyperact Disord 2012; 4:53–62
3.
Clarke DE, Narrow WE, Regier DA, et al: DSM-5 field trials in the United States and Canada, part I: study design, sampling strategy, implementation, and analytic approaches. Am J Psychiatry 2013; 170:43–58
4.
Barkley R: Attention Deficit Hyperactivity Disorder in Adults. Sudbury, MA, Jones and Bartlett Publishers, 2010
5.
Das D, Cherbuin N, Butterworth P, et al: A population-based study of attention deficit/hyperactivity disorder symptoms and associated impairment in middle-aged adults. PLoS One 2012; 7:e31500
6.
Faraone SV, Biederman J, Spencer T, et al: Diagnosing adult attention deficit hyperactivity disorder: are late onset and subthreshold diagnoses valid? Am J Psychiatry 2006; 163:1720–1729, quiz 1859
7.
Murphy K, Barkley RA: Prevalence of DSM-IV symptoms of ADHD in adult licensed drivers: implications for clinical diagnosis. J Atten Disord 1996; 1:147–161
8.
Kooij JJ, Buitelaar JK, van den Oord EJ, et al: Internal and external validity of attention-deficit hyperactivity disorder in a population-based sample of adults. Psychol Med 2005; 35:817–827
9.
Solanto MV, Wasserstein J, Marks DJ, et al: Diagnosis of ADHD in adults: what is the appropriate DSM-5 symptom threshold for hyperactivity-impulsivity? J Atten Disord 2012; 16:631–634
10.
Batstra L, Frances A: DSM-5 further inflates attention deficit hyperactivity disorder. J Nerv Ment Dis 2012; 200:486–488
11.
Barkley RA, Murphy KR, Fischer M: ADHD in Adults: What the Science Says. New York, Guilford Press, 2008
12.
Kessler RC, Green JG, Adler LA, et al: Structure and diagnosis of adult attention-deficit/hyperactivity disorder: analysis of expanded symptom criteria from the Adult ADHD Clinical Diagnostic Scale. Arch Gen Psychiatry 2010; 67:1168–1178
13.
Kieling C, Kieling RR, Rohde LA, et al: The age at onset of attention deficit hyperactivity disorder. Am J Psychiatry 2010; 167:14–16
14.
Polanczyk G, Caspi A, Houts R, et al: Implications of extending the ADHD age-of-onset criterion to age 12: results from a prospectively studied birth cohort. J Am Acad Child Adolesc Psychiatry 2010; 49:210–216
15.
Willcutt EG, Nigg JT, Pennington BF, et al: Validity of DSM-IV attention deficit/hyperactivity disorder symptom dimensions and subtypes. J Abnorm Psychol 2012; 121:991–1010
16.
Mercadante MT, Asbarh F, Rosário MC, et al: K-SADS, Entrevista Semi-Estruturada Para Diagnóstico em Psiquiatria da Infância, Versão Epidemiológica, 1st ed. São Paulo, Brazil, PROTOC–Hospital das Clínicas da FMUSP, 1995
17.
Ambrosini PJ: Historical development and present status of the schedule for affective disorders and schizophrenia for school-age children (K-SADS). J Am Acad Child Adolesc Psychiatry 2000; 39:49–58
18.
Mattos P, Serra-Pinheiro MA, Rohde LA, et al: A Brazilian version of the MTA-SNAP-IV for evaluation of symptoms of attention-deficit/hyperactivity disorder and oppositional-defiant disorder. Rev Psiquiatr Rio Gd Sul 2006; 28:290–297
19.
Amorim P: The Mini Neuropsychiatric Interview (MINI): validation of a short structured diagnostic psychiatric interview. Rev Bras Psiquiatr 2003; 22:26–39
20.
First MB, Spitzer RL, Gibbon M, et al: Structured Clinical Interview for DSM-IV Axis I Disorders, Patient Edition (SCID-I/P, Version 2.0, 8/98 revision). New York, New York State Psychiatric Institute, 1998
21.
Leon AC, Shear MK, Portera L, et al: Assessing impairment in patients with panic disorder: the Sheehan Disability Scale. Soc Psychiatry Psychiatr Epidemiol 1992; 27:78–82
22.
Williams JBW: Mental health status, functioning, and disabilities measures, in Handbook of Psychiatric Measures. Edited by Rush AJ, First MB, Blacker D. Washington, DC, American Psychiatric Publishing, Inc., 2008
23.
Grevet EH, Bau CH, Salgado CA, et al: Lack of gender effects on subtype outcomes in adults with attention-deficit/hyperactivity disorder: support for the validity of subtypes. Eur Arch Psychiatry Clin Neurosci 2006; 256:311–319
24.
Kundel HL, Polansky M: Measurement of observer agreement. Radiology 2003; 228:303–308
25.
Ghanizadeh A: Agreement between Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, and the proposed DSM-5 attention deficit hyperactivity disorder diagnostic criteria: an exploratory study. Compr Psychiatry 2013; 54:7–10
26.
Kraemer HC, Kupfer DJ, Clarke DE, et al: DSM-5: how reliable is reliable enough? Am J Psychiatry 2012; 169:13–15
27.
Bauermeister JJ, Canino G, Polanczyk G, et al: ADHD across cultures: is there evidence for a bidimensional organization of symptoms? J Clin Child Adolesc Psychol 2010; 39:362–372
Information & Authors
Information
Published In
History
Received: 21 June 2013
Accepted: 1 November 2013
Published online: 12 June 2015
Published in print: Summer 2015
Authors
Funding Information
Dr. Rohde was on the speakers’ bureau/advisory board and/or acted as a consultant for Eli Lilly, Janssen-Cilag, Novartis, and Shire in the last 3 years. He also received travel awards (airline tickets and hotel) from Janssen-Cilag and Novartis in 2010 to take part in two child psychiatric meetings. The ADHD and Juvenile Bipolar Disorder Outpatient Programs chaired by Dr. Rohde received unrestricted educational and research support from the following pharmaceutical companies in the last 3 years: Abbott, Eli Lilly, Janssen-Cilag, Novartis, and Shire. All other authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).