Safety Reporting in Randomized Trials of Mental Health Interventions

We used a random sample of trials derived from the PsiTri registry. PsiTri is a comprehensive registry of controlled trials across the field of mental health (13). It has been developed as a concerted action funded by the European Commission Quality of Life Programme, and it combines the specialist registries of five mental-health-related collaborative review groups of the Cochrane Collaboration that are working on depression, anxiety, and neurosis; developmental, psychosocial, and learning problems; dementia and cognitive improvement; schizophrenia; and abuse of drugs and alcohol. The PsiTri registry is based on trials rather than references; i.e., references of different reports that pertain to the same controlled trial are grouped under the same trial entry. In December 2002, when sampling was performed, the registry contained 16,504 controlled trials. The registry is evolving, with the continuous addition of more trials, but it is estimated that over 90% of trials in the field are already included. We randomly selected 200 controlled trial entries from the registry. We stratified the random selection by collaborative review group (corresponding to the respective mental health diagnoses) to ensure that the proportion of diseases in the selected sample is representative of the overall registry. We then excluded trials that were controlled but not randomized and trials without peer-reviewed journal publications because it is impossible to access the coverage of safety information simply from abstracts. We included all trials, regardless of sample size. Whenever there were several peer-reviewed publications on the same trial, we decided which was the main trial publication (the one presenting the most complete version of data on the primary outcomes of the trial) and examined whether the other reports offered any additional information on safety parameters. Both the main report and any additional published information were considered in the appraisal of the adequacy of safety reporting. Whenever a single report presented two or more trials with distinct, independent randomizations, we considered each trial separately.

We used both qualitative and quantitative components of adverse event reporting that have been previously validated (7, 8). Qualitative components include 1) whether the number of withdrawals and discontinuations of study treatment due to toxicity is reported, 2) whether the number is given for each specific type of side effect leading to withdrawal, and 3) whether the severity of the described clinical adverse events and abnormalities of laboratory tests (laboratory-determined toxicity) is adequately defined, only partially defined, or inadequately defined. Adequate definition requires either detailed description of the severity or reference to a known scale of toxicity severity (typically with grades defined as 1=mild, 2=moderate, 3=severe, 4=life-threatening), with at least separate reporting of severe or higher-grade events; at least two adverse events (clinical or laboratory) have to be defined in this way, with numbers or rates given per study arm. Partial definition means that the reported numbers or rates mixed grade 2 with higher toxicity counts or the number of severe or higher grades of toxicity cases are separately specified only for one of many reported adverse events and laboratory abnormalities. Inadequate definition includes protocols lumping numbers of various types of adverse events; those lumping numbers for all grades of toxicity without separating any grades for any specific adverse events; those providing only generic statements (e.g., “the medication was not well tolerated”); and those not reporting anything regarding harm.

Quantitative measures assess the relative emphasis given to safety in the results of published trials (7). These measures are 1) the extent of space (absolute numbers of printed pages and proportion) devoted to safety in the results sections; 2) pages devoted to safety compared with pages devoted to the names and affiliations of authors, participants, and contributors in the same trial report; and 3) the use of tables and/or figures containing safety information. The space for each section was measured with a 0.05-page resolution (7).

Data were extracted by one investigator (P.N.P.) and discussed with a second investigator (J.P.A.I.) whenever any information was deemed unclear. For the assessment of the adequacy of reporting of clinical adverse effects and laboratory-determined toxicity, the kappa coefficients between independent data extractors have been previously estimated to be 0.72 and 0.85, respectively (8). In a sample of 60 trials, there were no instances in which one extractor considered the reporting adequate and the other inadequate. Moreover, we observed no important discrepancies in data extraction of quantitative parameters.

Trials were categorized first according to whether drugs were involved in the randomized comparisons or not. Nondrug interventions (e.g., psychological, behavioral, and social) may need different approaches to safety. We also separated trials that had at least 100 randomized subjects and over 50 subjects in a study arm. This group has been specifically targeted in previous evaluations of safety reporting (7, 8). Furthermore, for each eligible trial, we noted whether the trial showed significant differences (p<0.05) for any of the main efficacy endpoints considered, whether different doses/schedules of the same therapy were compared, and whether the study design compared only different doses/schedules; the exact trial sample size; masking (double blind or not); the population (predominantly adult versus pediatric); the mean duration of follow-up (long-term being more than 1 year); the main financial sponsor of the trial (government versus other/unreported); the location where most patients were recruited (United States versus other); and the year of publication.

Analyses included descriptive assessments of the qualitative and quantitative parameters of safety reporting. Furthermore, regression analyses were conducted to assess parameters influencing safety reporting.

Descriptive data are reported for trials in which drugs were involved; for all studies; for trials with a sample size of at least 100 patients and at least 50 patients allocated to a study arm; and for trials published in 1990 or later.

All of the variables mentioned were used as predictors in least-squares regression models by using the percentage of space devoted to safety reporting in the results section as the dependent variable. Other regression analyses using logistic regression addressed the effect of these parameters on adequate reporting of clinical adverse events, laboratory-defined toxicity, and reasons for discontinuations due to toxicity per study arm. Multivariate models started from all variables with p<0.25 in univariate analyses and used backward elimination procedures. Only drug trials were included in the regression analyses. Regression analyses also examined whether any of the reporting parameters improved over time.

Statistical analyses were performed in SPSS (SPSS, Inc., Chicago). P values were two-tailed.

Of 200 randomly selected Psi-Tri trial entries, we excluded 22 in which none of the references had been published in journals, 44 that did not pertain to randomized trials, and two that could not be retrieved in full text. Thus, 132 reports with 142 separate randomized trials (N=11,939 subjects) were eligible for further analysis. Of those, 11 had more than one journal publication (N=48 secondary publications). Only two secondary publications provided additional safety data, on one trial each. Trials belonged to the depression, anxiety, and neurosis (N=79); developmental, psychosocial, and learning problems (N=5); dementia and cognitive improvement (N=13); schizophrenia (N=33); and drug and alcohol (N=12) domains, respectively. Of the 142 trials, 25 had at least 100 subjects and at least 50 subjects in a study arm. Drugs were involved in the randomized comparisons in 103 (72.5%) of the 142 trials (N=7,963 subjects). Of the 103 drug trials, 62 contained an arm that received only placebo or no treatment. Nondrug trials addressed other biological (N=6), behavioral (N=13), social or health services (N=11), and psychological (N=9) interventions.

Overall, studies were of small group size (Table 1). About two-thirds of the trials were double blind, the percentages were higher among drug trials, and about a quarter involved some dose comparisons. Most trials showed statistically significant results for efficacy. About a third of the trials had received government funding. Few trials had long-term follow-up, and few involved children. Approximately half had been conducted in the United States. Trial reports covered a wide range of publication years, with half being published after 1990.

Overall, less than one out of six trials had adequate reporting of clinical adverse effects and even fewer trials had adequate reporting of laboratory-determined toxicity (Table 2). Studies with at least 100 subjects and at least 50 subjects in an arm had somewhat worse documentation of clinical adverse events and somewhat better documentation of laboratory-determined toxicity. Even among drug trials, the rates of adequate reporting were very low (21.4% and 16.5%, respectively). Drug trials with placebo/no treatment arms also had very low rates of adequate reporting (17.7% and 12.9%, respectively). No nondrug trials reported adequately on clinical adverse events or laboratory-determined toxicity.

Discontinuations due to toxicity per study arm were mentioned in slightly less than half of the trials, and the percentage was not much better in trials with at least 100 subjects or drug trials. Specific reasons for these discontinuations per arm were given less frequently, and this was recorded in only 8% of the trials with at least 100 subjects and at least 50 subjects in an arm. Nine nondrug trials simply stated clearly that no subjects discontinued the trial or mentioned reasons for discontinuations that were unrelated to toxicity. Otherwise, none of the nondrug trials provided data on discontinuations occurring due to toxicity.

There was no statistically significant heterogeneity in the rates of adequate reporting across collaborative review group domains for clinical adverse events (p=0.09), laboratory-determined toxicity (p=0.28), or the two withdrawal outcomes (p=0.72 and p=0.52) (all by Fisher’s exact test).

Little space was devoted to safety (Table 3). Even when nondrug trials were excluded, this was on average about 1/10 of a page or even less for trials with a placebo/no treatment arm (median=0.05 page). Large trials with at least 100 subjects and at least 50 subjects in an arm used only slightly more space (median=0.15 page). Overall, the space given to safety information was less than the space given for the names of authors and their affiliations (p<0.001). The difference remained significant when analyses were limited to drug trials, while for trials with at least 100 subjects and at least 50 subjects in an arm, the space for safety was similar to the space devoted to authors and affiliations. Tables and/or figures for safety were used infrequently, and even for exclusively drug trials.

The percentage of space in the results sections dedicated to safety was significantly and independently larger in trials of schizophrenia, and it was independently smaller in trials conducted in the United States and in trials with long-term follow-up (Table 4). Adequate reporting for clinical adverse events was more common in trials of pediatric populations and less common in trials conducted in the United States (multivariate odds ratio=8.6, 95% confidence interval [CI]=1.2–62.0, p<0.04, and multivariate odds ratio=0.3, 95% CI=0.1–0.9, p<0.04, respectively). Adequate reporting of laboratory-determined toxicity was more common in larger trials (odds ratio=3.7, 95% CI=1.1–12.5 per 10-fold increase in sample size, p<0.04), but it was not significantly affected by any other parameters. Conversely, reporting of the reasons for discontinuations due to toxicity per study arm was less common in larger trials, and it was also independently less common in studies where dose comparisons were involved (multivariate odds ratio=0.2, 95% CI=0.1–0.6, p=0.004, and multivariate odds ratio=0.3, 95% CI=0.1–0.9, p<0.04, respectively).

The quality of safety reporting did not improve in recent trials (Table 2 and Table 3). In logistic regression, the odds of adequate reporting of clinical adverse events, laboratory-determined toxicity, and reasons for withdrawals due to toxicity nonsignificantly changed by 0.78-fold (p=0.29), 0.86-fold (p=0.54), and 0.94-fold (p=0.76) per 10 years, respectively, while the pages devoted to safety nonsignificantly decreased by 0.05 per 10 years (p=0.29).