Reliability and Factor Structure of the Short Problem Behaviors Assessment for Huntington’s Disease (PBA-s) in the TRACK-HD and REGISTRY studies

A total of 366 individuals participated in the TRACK-HD study, at four sites: Leiden (The Netherlands), London (UK), Paris (France), and Vancouver (Canada). Subjects were divided into three groups: 123 early manifest HD patients (disease stage I and II, equating to a Total Functional Capacity Score¹² of 7–13), 120 premanifest HD gene carriers, and 123 controls. Premanifest participants all had a total UHDRS motor score of less than 5 and a disease burden score greater than 250; the latter was calculated from the formula [age × (CAG-35.5)] and gives an estimation of lifetime exposure to the disease, without predicting its course.¹³ The 123 control participants were age- and gender- matched to the combined gene-positive groups, and were either spouses, partners, or gene-negative siblings of the premanifest and early HD patients. Subjects attended an annual assessment day, which included a cognitive battery, MRI scan, motor, functional and quality of life assessments, and a behavioral interview (the PBA-s). Baseline assessments were conducted between January and August 2008, with follow-up at 12, 24, and 36 months. Written informed consent was obtained from all participants, including consent to video record parts of the assessment for quality control, research, and training purposes. The study was described in detail by Tabrizi et al.¹¹

A total of 732 TRACK-HD PBA-s interviews were conducted during the first 2 years, of which 329 were selected for rescoring. In addition, follow-up assessments from subsequent years were screened for severity scores of ≥2 for the least-commonly reported symptoms (paranoid thinking, hallucinations, disorientation), to improve reliability assessment of these items; eight such interviews were identified.

A total of 230 manifest HD patients attending the Manchester REGISTRY site for baseline assessment were included in the factor analysis. Recorded PBA-s interviews were available from 73 individuals and included in the reliability study because they encompassed a wider range of disease severity and PBA-s scores. Written informed consent to have their interviews video recorded for research and training purposes was obtained from all patients. The reliability study, thus, included data from 410 PBA-s interviews, conducted with a total of 308 participants (Table 1).

The version of the PBA-s used initially at the baseline assessments comprised 10 items taken from the original 40-item PBA-HD: depression, suicidal ideation, anxiety, irritability, aggressive behavior, apathy, perseverative thinking, paranoid thinking, hallucinations, and behavior suggestive of disorientation. A list of all the items included in the original PBA-HD is shown in Table 2 for comparison. An additional PBA-HD item measuring obsessive-compulsive symptoms was added to the PBA-s at the 12-month follow-up interview, to help raters differentiate between perseverative phenomena and obsessive-compulsive behaviors. Interviews were conducted jointly with the subject and a knowledgeable informant (usually the spouse or partner) who was then briefly re-interviewed afterward to elicit any additional information which (s)he had felt unable to discuss openly during the joint interview; raters were instructed to make an overall judgment based on all available information, including their own observations as well as the answers provided by subject and informant. Suggested prompt questions were provided for each of the 11 items, and raters were instructed to ask whatever additional questions they deemed necessary until they had enough information to determine the correct scores. Severity and frequency during the previous 4 weeks were rated separately for each symptom on a 5 point (0–4) scale; these were then multiplied to yield an overall symptom score for each item. The criteria for rating frequency were the same for all items, and ranged from 0 (not present) to 4 (present all the time). Detailed criteria for rating severity were provided for each symptom, based on the following general principles: 0=symptom absent; 1=trivial, or rater is not completely convinced that the symptom is present; 2=symptom definitely present, but not causing a practical problem or interfering with daily life; 3=symptom is causing problems or distress; 4=symptom causing severe problems or making normal life impossible.

All the TRACK-HD interviewers were trained in person during a site investigator meeting at the outset of the study, which involved watching and scoring videos of PBA-s interviews and discussing scores. PBA-s interviews were video recorded and uploaded to the TRACK-HD web portal for rescoring by an independent expert rater (JC) who was familiar with the long-version PBA-HD. Any discrepancies were resolved by a third expert rater (DC) and feedback provided by telephone or e-mail to the original interviewer. Once a site rater was judged to have achieved an acceptable performance, the proportion selected for rescoring was reduced to 25%, and eventually 10%; site raters were blind to which interviews would be selected. In the case of interviews conducted in languages other than English, rescoring was carried out by expert raters (CB, M-FB, EvD) who were native French or Dutch speakers respectively, and later on by an alternate PBA-s rater at the Leiden site. In all cases, the rater doing the rescoring was blind to the scores assigned during the original interview, and to the group status (affected, premanifest, or control) of the patient. The Manchester interviewers were trained in a similar fashion, and rescored by the same expert rater (JC) as the English TRACK-HD videos.

Inter-rater reliability was assessed using the Cohen’s kappa (k) statistic to correct for the possibility of chance agreement.¹⁴ Weighted kappa (k_w) was devised as a way of describing inter-rater reliability in situations where some disagreements between raters are of greater gravity than others¹⁵; for example, in the case of an interview like the PBA-s, a symptom close to the borderline between a ‘mild’ or ‘moderate’ score could be rated as ‘2’ or ‘3’, but should not receive a score of ‘1’ or ‘4’. The ‘clinically significant’ k_w used here accepts cases where two raters differ by ±1 point as agreement, whereas k requires both raters to give exactly the same score. Both k and k_w were calculated for the present study.

To investigate the factor structure of the PBA-s, we conducted a principal components analysis with Varimax rotation, using REGISTRY 3 data from Manchester rather than TRACK-HD because the range of disease severity more closely resembled the population used for the original PBA-HD factor analysis. The symptoms ‘paranoid thinking’ and ‘hallucinations’ were excluded because fewer than 3% of subjects scored more than zero on these items, leaving nine remaining PBA-s variables for inclusion, and a ratio of>25 subjects per variable. There are no clearly established rules to determine the number of subjects required for a satisfactory factor analysis, but a ratio of at least 20 subjects per variable appears to be a conservative estimate from the published literature.¹⁶

The percentage of severity and frequency ratings where both the interviewer and rescoring rater agreed was calculated for each of the 16 pairs of raters. On average, raters agreed completely on 87.7% of severity scores (range 76.9%‒100%) and 88.6% of frequency scores (range 80.2%‒96.1%). Using the more realistic criterion of agreement within ±1 point, the average percentage agreement for severity scores was 98.9% (range 97.4%‒100%), and for frequency scores 98.3% (range 94%‒100%). The corresponding figures for the English-speaking sites only (London, Manchester, and Vancouver) were 88.4% for severity scores (99.3% within ±1 point) and 89.8% for frequency scores (99.2% within ±1e point). There was very little difference in the percentage of ‘clinically significant’ agreement between symptoms, or between languages.

The mean k was 0.74 for severity and 0.76 for frequency ratings. Mean k_w was 0.94 for severity and 0.92 for frequency ratings. Table 4 shows the k and k_w values for each of the 16 pairs of raters, and the mean k and k_w values for each language. When the analysis was restricted to English-speaking sites only, the mean values of k were 0.77 for severity and 0.80 for frequency scores, while the mean k_w values were 0.96 for severity and 0.96 for frequency.

After excluding the ‘paranoid thinking’ and ‘hallucinations’ items, the remaining data satisfied conventional requirements for factor analysis (Kaiser-Meyer-Olkin measure of sampling adequacy=0.60; Bartlett’s test of sphericity p <0.001). Principal components analysis extracted three factors with Eigenvalues greater than one, collectively explaining 57.4% of total variance. The factor loadings after Varimax rotation are shown in Table 6.