ICD-11 Guidelines for Mental and Behavioral Disorders of Children and Adolescents: Reliability and Clinical Utility

The ICD-11 classification has not been significantly modified. The description of depression in the section on developmental presentations highlights the fact that among children, depressive mood may occur in the form of somatic complaints, increased anxiety, or excessive crying; among children, depressive mood may present as general irritability; among children and adolescents, reduction in the ability to concentrate and sustain attention may be manifested in a decline in academic performance; and decreased appetite may be expressed in a lack of weight gain expected for a given age and level of development, rather than weight loss.

Generalized anxiety disorder (GAD), specific phobia, separation anxiety disorder, social anxiety, and selective mutism are the most frequent anxiety disorders among children and adolescents. The GAD description in the section on developmental presentations highlights the fact that among children, expressions of concern may include a particular interest in and adherence to rules, with a strong desire to please others, or seeking others’ approval excessively to reassure themselves; and among young people, excessive irritability and somatic and depressive symptoms are common (8).

Four disorders are included: posttraumatic stress disorder (PTSD), complex PTSD, prolonged grieving disorder, and adjustment disorder. The PTSD description underlines the following: among children, both the presence of reexperiencing and active avoidance of internal states and the perception of an intensified current threat are usually expressed through behavioral manifestations; adolescents may be more reluctant to report their reactions to traumatic events, and when reexperiencing occurs, adolescents communicate a lack of affection or other emotions. As for adjustment disorders among children, these tend to be expressed through behavior such as tantrums, bedwetting, and sleep disorders. Somatic symptoms, excessive irritability, or psychoactive substance use may be reported, and adolescents may not explicitly verbalize a connection between stressful events and their own symptoms, meaning that the clinician must consider the relationship between the time of stressor and the attendant symptoms.

Hyperkinetic disorder was removed from ICD-11 and renamed attention-deficit hyperactivity disorder (ADHD). ADHD may present with or without impulsivity and hyperactivity. Inattentive presentation is manifested in a diversity of contexts, with a combination of difficulty concentrating, tendency toward distraction, and organizational problems; losing things frequently; and inattention to details of the task at hand. ADHD combined presentation is diagnosed when both inattention and hyperactivity or impulsivity are present. The detection of ADHD is complicated by the fact that it is often not identified or is attributed to other causes, even though it is a common cause of academic problems (9).

Oppositional defiant disorder and conduct disorder (CD) were incorporated into disruptive behavioral disorders (DBD). Oppositional defiant disorder may be present with or without chronic irritability-anger; the subtype with chronic irritability-anger was incorporated as the ICD-11 equivalent of disruptive mood dysregulation disorder (DMDD) in DSM-5 (10, 11). Field studies and secondary analysis of their diagnostic criteria showed that DMDD has limited interrater and temporal reliability, a lack of consensus between psychiatrists, and high comorbidity rates, especially with ADHD and oppositional defiant disorder (12–14). Both oppositional defiant disorder and CD could present as “limited prosocial emotions.” This specifier is linked to more severe, persistent violent behavior (15) and could be identified in internalizing and externalizing disorders (16).

Given the need to evaluate whether the diagnostic guidelines work in Mexico, the aims of this research were to establish the interrater reliability and clinical utility of ICD-11 guidelines for mood, anxiety, and fear-related disorders and ADHD and DBD among children and adolescents.

Children and adolescents ages 6–17 participated. In the first consultation, the clinician explained the symptoms of all the disorders to the participant and the family in the two venues that participated. Youngsters with one of their parents received a comprehensive explanation of the nature and aims of the research and agreed to sign the assent/consent forms. Exclusion criteria included the presence of communication difficulties, clinically evident cognitive dysfunctions, and physical disabilities that could interfere with the participation of patients in the diagnostic interview. Patients at imminent risk of self-harm or injury to others were excluded, and these patients were managed in the psychiatric emergency units available at both institutions. All participants, both children and adolescents, were interviewed in the presence of one of the parents.

All clinical raters at both institutions were psychiatrists; some were also child and adolescent psychiatrists and others were 6th-year residents in the specialty. All clinical raters received a 4-hour training session on the most important changes in ICD-11 and the use of diagnostic guidelines. The research thereby emulated what will happen in specialized scenarios once ICD-11 is implemented. Rater training was provided by a clinical expert in child and adolescent psychiatry (FRP) with over 20 years’ experience in the field, who was a member of the ICD-11 guideline development team. As part of the training, clinician raters practiced applying the diagnostic guidelines to case vignettes and discussed the issues that arose during this process.

Two clinicians participated in each interview, one as an observer and the other as an interviewer, without communicating with each other. In addition to establishing a diagnosis for each participant (a main diagnosis and up to two secondary diagnoses when appropriate), they evaluated the clinical utility of the guidelines as used with each patient. Clinical rater pairs were assigned according to a systematic sampling procedure by using a list of clinicians available daily and considering their most recent participation as an observer or interviewer to maximize the variability of dyads and roles.

The instrument for assessing the clinical utility of the ICD-11 diagnostic guidelines for mental and behavioral disorders was specifically developed for this purpose (18). It comprises 15 questions on the different domains of the construct and is answered by using a 4-point Likert scale. According to an exploratory and confirmatory factor analysis of data from a study of Mexican clinicians who evaluated the guidelines for common mental and behavioral disorders in adulthood (3), all the items were grouped congruently in two general dimensions: implementation characteristics, and identification and management. Cronbach’s alpha coefficients for the subtotals and total scale were 0.90, 0.90, and 0.93, respectively.

The ethics review boards at both institutions approved all the procedures used in the research, including the assent/consent forms for the children and adolescents and their parents. Clinicians responsible for the initial evaluations were asked to refer all participants who met the selection criteria to a research assistant, who oversaw the informed assent/consent process. Only participants and parents who had agreed to participate voluntarily in the research signed the assent/consent form, and all were invited at their convenience to a 2-hour interview with a dyad of clinical raters. After the interviewer had finished the questions, the observer had an opportunity to ask any questions that the observer considered necessary.

At the very least, each clinical interview covered the entire diagnostic evaluation, including symptoms present, history of the current episode, and an examination of mental status and personal, family, medical, and psychiatric history. If studies or other information existed, it was agreed that both raters would have access to it. The main objective of these measures was for clinicians to establish a diagnosis on the basis of exposure to the same information.

After the interview, both clinicians recorded their diagnostic impression of the patient without consulting each other, completed a step-by-step evaluation of the essential ICD-11 characteristics of the selected diagnoses for each patient, and answered the clinical utility evaluation questionnaire of the ICD-11 diagnostic guidelines for each case. In addition to the main diagnosis, each clinician could include up to two secondary diagnoses. It was also possible to register a “nondiagnosis,” if the clinician considered that the participant failed to present the diagnostic characteristics required to determine the presence of a mental disorder (none of the patients in the sample received a nondiagnosis).

Clinicians were not allowed to discuss the case with each other until they had independently entered their diagnostic evaluation into the database. In addition, the electronic data collection system allowed the two clinicians to complete the same patient’s information independently to ensure that they were blind to their peer’s opinions. Data were collected by using the Qualtrics online survey platform, making it possible to ensure the confidentiality and security of information.

A statistical analysis was conducted with SPSS, version 20. General characteristics were described based on means and standard deviations or frequencies and percentages and were compared by using Student’s t tests of independent samples or chi-square tests. The frequencies and percentages of each of the diagnoses assigned by both the rater and the observer were also calculated. Each specific diagnosis was subsequently grouped into one of five general categories: mood disorders, anxiety and fear-related disorders, disorders specifically related to stress, ADHD, and DBD. The corresponding kappa values were calculated to determine the level of general diagnostic agreement between both clinicians. In all cases, the preestablished level of significance was p≤0.05. Finally, to summarize information on the clinical utility of the guidelines, the frequencies and percentages of responses to each item in the corresponding questionnaire were calculated for both the interviewers and the observers.

Eight diagnoses were the most frequently assigned main diagnoses (Table 2). ADHD combined presentation and oppositional defiant disorder with chronic irritability-anger were the most frequent disorders identified by interviewers and observers.

Table 3 shows the frequencies and percentages of participants included in each diagnostic group. No significant differences between the interviewer and the observer were found in the diagnostic assignments for any group. Kappa values between raters were subsequently calculated, taking into account the number of interviewees whom they both assigned to the same diagnostic group (Table 4).

Clinicians’ responses to the questions on the instrument assessing clinical utility of the guidelines are shown in Table 5. Over 80% of clinicians reported that the guidelines were clear, easy to use, and accurate; that the guidelines and qualifiers were useful in helping to communicate with patients, select treatment, and determine patients’ prognosis; and that descriptions of developmental presentations were useful as applied to the patient.

More specifically, interrater reliability based on ICD-11 guidelines was strong for DBD, particularly for oppositional defiant disorder with chronic irritability-anger. This finding is especially relevant, because one of the most important debates regarding the classification of mental disorders in childhood and adolescence focuses precisely on the evaluation, diagnosis, and treatment of children with severe chronic irritability and anger. In the past, classification systems have found it difficult to identify this phenomenon, which has delayed its diagnosis and treatment. Fortunately, the formulation of irritability and oppositionality put forth in ICD-11 may more accurately identify chronic irritability than do the ICD-10 or DSM-5 proposals (11, 27).

Conversely, when contrasting our kappa values with those of ICD-11 adult guidelines for internalizing disorders (3), we found similar moderate values, perhaps because of the high comorbidity of the anxiety-depression dimension. The high comorbidity of anxiety and depressive disorders may also account for the fact that not all disorders or groups of disorders obtained strong interrater values.

In general terms, good clinical utility was the most frequently chosen answer by both interviewers and observers for almost all the questions. Moreover, the clinical utility of qualifiers, as well as the usefulness of the description of developmental presentations in the guidelines (developmental presentations section), was good to extremely good, which could facilitate the dissemination of guidelines among child and adolescent psychiatrists.

In other words, clinicians perceived that these guidelines improve communication with patients, are detailed enough, and constitute a useful tool for clinical management decisions (2), suggesting that they might be helpful in reducing the global burden of such diseases through early identification and treatment. This could be particularly relevant for countries with a shortage of economic and specialized human resources.

The study had some limitations. The group of raters was diverse and included clinicians with several years’ experience and others with fewer years of practice. Moreover, it should be emphasized that kappa values and clinical utility perceptions were obtained from clinicians who had received a 4-hour training session that sought to harmonize criteria and decision making when disorders were diagnosed, highlighting the changes proposed in the diagnostic guidelines for each category. This process may have sensitized clinicians to the diagnostic criteria and rationale of the guidelines, specifically in the developmental presentation sections, which constitutes an innovative way to identify mental disorders across the lifespan (28).

Additionally, the sample was relatively small in that we did not have enough children (ages 6–11) to determine whether interrater reliability indices for clinicians evaluating children are different from those for evaluating adolescents. Moreover, the use of a convenience sampling strategy at specialized centers limited the generalization of results, particularly to patients in other health care settings (such as primary care centers).