The Case for Practical Clinical Trials in Psychiatry

Over the past several decades, the pressing need to understand the effect of widely practiced treatments on the major outcomes for common illnesses such as heart attack and stroke gave rise to increasingly large and sophisticated practical clinical trial networks in cardiovascular medicine (9). This process was greatly facilitated by network development and maintenance activities funded by NIH (32). As a result, practical clinical trials are now the norm in areas such as acute coronary syndrome and heart failure (10). For example, the Virtual Coordinating Center for Global Cardiovascular Research network has so far conducted 15 trials of thrombolytic agents with more than 150,000 enrolled patients at 3,000 sites in 50 countries (33). These practice-changing practical clinical trials have led to convincing improvements in coronary reperfusion and, more important, survival in patients with acute myocardial infarction (24).

In pediatric oncology, practical clinical trials conducted in the Children’s Oncology Group network have revolutionized the care of children with cancer (34). Illustrating the fact that practical clinical trials can and do take place in specialty clinics if that is where the patients are located, more than 95% of children with cancer in the United States are treated at the approximately 500 participating centers in the Children’s Oncology Group network, which is funded by the National Cancer Institute. The Children’s Oncology Group provides a national network of communication for researchers, care providers, and families of pediatric patients with malignant disease and conducts laboratory investigations and clinical trials of new treatments for cancer in infants, children, adolescents, and young adults. Cases seen in the network constitute, in effect, a national registry of nearly all childhood cancers in the United States. Because of the network’s broad coverage, more than half of American children with cancer are entered into at least one randomized Children’s Oncology Group trial, which in turn has led to dramatic improvement in pediatric cancer outcomes.

Practical clinical trials sometimes reveal that widely used, seemingly sensible clinical strategies that have been shown to work in small industry-funded registration trials can, in actual fact, be harmful when applied in real clinical settings. In a sobering and heuristically valuable example, the Cardiac Arrhythmia Suppression Trial examined Class I antiarrhythmic drugs that had been shown in small efficacy trials to suppress ventricular extrasystoles in patients at increased risk of death after a heart attack (35). Intuitively, it seemed reasonable that a reduction in irregular heartbeats also should result in reduced mortality. However, the Cardiac Arrhythmia Suppression Trial revealed that these drugs were associated with a 250% increase in short-term mortality, whereas newer Class III antiarrhythmic agents appeared to reduce mortality (36). The Cardiac Arrhythmia Suppression Trial nicely illustrates how practical clinical trials can be useful in confirming that the results of small efficacy trials will translate (or will not translate, as the case may be) to real-world practice (37), especially when the initial trials involved surrogate markers for relatively rare outcomes such as mortality.

Practical clinical trials are just beginning to occur in psychiatry (1, 38). Projects illustrating recent progress in implementing practical clinical trials in the mental health field include the British Bipolar Affective Disorder: Lithium Anticonvulsant Evaluation trial involving adults with bipolar disorder (2); the industry-funded International Suicide Prevention Trial in schizophrenia (39); the NIMH-funded practical clinical trials in adults with depression, bipolar disorder, psychosis, and Alzheimer’s disease (40–43); and the NIMH-funded Child and Adolescent Psychiatry Trials Network (38).

Funded by the Stanley Foundation and the British Medical Research Council, the Bipolar Affective Disorder: Lithium Anticonvulsant Evaluation trial addresses the question, “In adult patients with bipolar illness, is lithium, sodium valproate, or a combination of the two more effective in reducing the risk of further episodes of mania or depression and in improving long-term effects on mood and more tolerable in terms of side effects?” The trial will recruit thousands of patients at 54 centers who will be randomly assigned to best-practice active treatments and assessed with simple, clinically relevant categorical and dimensional outcomes measures (2).

The recently completed International Suicide Prevention Trial compared the effects of clozapine and olanzapine in 980 adults with schizophrenia at high risk for suicide at 56 sites in 11 countries (44). As is typical of a practical clinical trial, the study included freedom to augment these treatments as needed, blind ratings in which suicide was the primary outcome, and equivalent clinical contact for the two treatment groups. Clozapine proved superior to olanzapine in reducing suicidality. Although it is too soon to evaluate the impact of the International Suicide Prevention Trial on clinical practice, the investigators suggested that clozapine should be considered for all patients with schizophrenia at high risk for suicide (44).

NIMH recently funded three large practical clinical trials designed to address questions of major public health relevance that emerged from but were not answered by earlier industry-funded efficacy trials: 1) Sequenced Treatment Alternatives to Relieve Depression, which compares active treatments and treatment strategies in adults with depression (40); 2) Systematic Treatment Enhancement Program for Bipolar Disorder, which focuses on the pharmacological and psychosocial treatment of bipolar illness (41); and 3) Clinical Antipsychotic Trials of Intervention Effectiveness, which contrasts various atypical antipsychotics as treatments for psychosis or for Alzheimer’s disease (42, 43). These trials have innovative experimental designs—including an equipoise-stratified design in Sequenced Treatment Alternatives to Relieve Depression (45) and an outcomes-based rerandomization scheme in Clinical Antipsychotic Trials of Intervention Effectiveness (46)—and are much larger than any study previously attempted in psychiatry. They pioneer the construction of the kind of networks that are essential for making practical clinical trials a routine part of psychiatric practice.

A partnership between researchers at Duke University and the American Academy of Child and Adolescent Psychiatry, the NIMH-funded Child and Adolescent Psychiatry Trials Network is an investigator-initiated proof-of-concept effort to establish a practical clinical trial network in pediatric psychiatry (38). Guided in the selection of specific questions by network members and expert advisory panels, the Child and Adolescent Psychiatry Trials Network will focus on two critically important clinical issues: 1) obtaining evidence from randomized trials of the effectiveness of widely used but understudied combined drug treatments and 2) investigating the short- and long-term safety of pharmacotherapy. With 200–400 child and adolescent psychiatrists each participating in two or three practical clinical trials at any one time, the Child and Adolescent Psychiatry Trials Network promises to advance both the evidence base and the research capacity of child and adolescent psychiatry.

Practical clinical trials typically flourish in networks that are developed and maintained by research centers in productive partnership with clinical and administrative decision makers in federal, community, and academic settings (5, 37, 47). No stably funded practical clinical trial infrastructure currently exists in psychiatry—the current practical clinical trials in psychiatry (Sequenced Treatment Alternatives to Relieve Depression, Systematic Treatment Enhancement Program for Bipolar Disorder, Clinical Antipsychotic Trials of Intervention Effectiveness, and Child and Adolescent Psychiatry Trials Network) will sunset if their funding is not renewed. Thus, the development of a stable practical clinical trial infrastructure, including establishing a coordinating center or centers, remains a necessary step toward the implementation of the practical clinical trials model in mental health care. A coordinating infrastructure with access to researchers and other individuals with the skills and equipment required to conduct a practical clinical trial is necessary 1) to recruit and sustain a broad-based network of clinical sites representative of current practice and service systems (site development); 2) to institute the mix of financial, practical (e.g., continuing medical education), and altruistic incentives that makes ongoing network participation reasonable and desirable; 3) to develop trial-independent and trial-dependent clinical research training for site investigators; and 4) to characterize the practice patterns and factors that affect clinical decision making, treatment provision, and conformance with evidence-based treatment recommendations within each site. It is important to note that to cover the practice landscape in which mentally ill patients are found and to model rational triage strategies, it will be necessary to extend practical clinical trial networks to specialty clinics (which, by analogy to specialty cancer clinics, might be appropriate for rare conditions such as schizophrenia or complex tic disorders) and to community practice settings (for more common illnesses, such as attention deficit hyperactivity disorder or major depression). Other important aspects of network coordination include the development of systems for data management and the provision of analytic and dissemination infrastructures. Finally, initiatives to conduct practical clinical trials in other areas of medicine have been successful because network participants were trained in network protocols during residency and continued to participate after they began clinical practice. Hence, seamless integration of practical clinical trials into training programs (as will be done in the Child and Adolescent Psychiatry Trials Network [38]) is of paramount importance if practical clinical trials are to be widely implemented in mental health delivery systems.

Although much progress has been made, clinical practice in psychiatry remains far from being evidence based (48), with heterogeneity in practice predicting heterogeneity in quality of care (49, 50). In treatment of children, for example, there has been a rapid shift from a single-drug standard toward complex multidrug treatment regimens (51) for which scientific support is completely lacking (52). These gaps in the research literature, which can be identified by examining postmarketing pharmacoepidemiologic studies (53) and findings from practice research (30, 31) and by perusing expert-driven treatment guidelines (17, 54), contribute both to the rationale and, by guiding the choice of questions, to the method for prioritizing a practical clinical trial research agenda. In addition, because physicians participating in practical clinical trials are themselves interested in securing straightforward answers to a large number of highly relevant clinical questions, vetting questions and protocols through the network members is an effective way to prioritize practical clinical trials and also serves a network maintenance function by enhancing recruitment of patients into clinical trials.

As with efficacy studies, the choice of an experimental design in a practical clinical trial depends on the theoretical and practical aims of the trial (55). When the primary purpose of a trial is to secure an unbiased evaluation of whether a treatment signal is present, an inactive control condition—typically pill placebo in pharmacotherapy trials—maximizes the so-called assay sensitivity of the experiment. Two settings in which a placebo condition would be considered for a practical clinical trial are: 1) if one or more treatments have never been shown to be superior to an inactive control treatment or 2) if it is unreasonable to infer that the results of prior randomized clinical trials of an active treatment, compared with placebo, will be replicated if tested in the same patient population and if a true tie is unacceptable for this setting. Although the optimal time for a placebo-controlled trial is early in the treatment development cycle, before widespread belief in the clinical effectiveness of a treatment accrues, a great deal of the evidence clinicians need to practice effectively involves comparisons of active treatments and treatment addition strategies targeting partial response. This requirement makes placebo-controlled practical clinical trials appropriate long after early-phase efficacy studies have been completed.

Despite the fact that some investigators believe that a null result in a trial with two active treatments cannot be interpreted in the absence of an inactive control condition, practical clinical trials without a concurrent placebo arm are relatively common. For example, if both active treatments have shown unequivocal superiority against placebo, then a practical clinical trial without a concurrent placebo control condition is generally thought reasonable, especially if use of placebo can be said to violate the principle of therapeutic uncertainty (56). Even when this standard is not met, some advocates of practical clinical trials would argue that, unless a newer treatment has substantial advantages with respect to adverse events, user friendliness, or cost, if it cannot beat a proven treatment in a practical clinical trial powered for a small effect size, “Who needs it?”

Finally, in a practical clinical trial with an active comparator or comparators but no placebo group, the investigator must decide whether masking (blinding treatments to eliminate observer bias) is essential to the study aims. Observer bias occurs when information held by the observer influences the way he or she interprets and scores a primary outcome. In clinical practice, knowledge of treatment assignment is part of the ecological validity of the outcome (because patients know the nature of their treatment assignment) and, moreover, is consistent with the uncertainty principle for the study treatments. Thus, openly assigning patients to treatment does not necessarily detract from (and, in fact, may enhance) the robustness of a practical clinical trial.

As in efficacy trials, treatment protocols in a practical clinical trial must specify the treatment(s) to be used; titration of medications for subjects through the use of fixed, flexible, or a combination of fixed and flexible dosing methods; range of starting doses; administration schedule; selection of brand-name or generic drugs; choice of rating forms and symptoms rated during titration and how this information will be used; adverse event monitoring and its effect on dosing strategy; and procedures for managing emergencies. In this regard, tensions exist between adherence (how well the standardized treatment protocol is followed) and competence (how well the treatment is done clinically); between tailoring treatment to the needs of the patient and using uniform, manualized, explicitly specified intervention(s); and between the desire for site flexibility and efficiency and the need for a common protocol that minimizes the potential for between-site drift in protocol implementation. To facilitate protocol acceptance and compliance, practical clinical trials evaluate these tensions through the perspective of the practicing clinician, especially when a standard of care is widely acknowledged. Thus, practical clinical trial researchers must work collaboratively with network clinicians to develop short, user-friendly protocols and manuals that are empirically grounded and clinically meaningful in order to maximize the generalizability of best practices to the community of psychiatrists caring for patients.

Although well-justified exceptions exist (see reference 57 for an explanatory aim linked to an important functional outcome), practical clinical trials generally eschew complex and time-consuming assessments designed to satisfy explanatory aims in favor of assessment strategies that are intended to be widely applicable to real-world practice settings (4). Specifically, practical clinical trials use unambiguous, readily detectable, ecologically valid diagnostic strategies and endpoints that reduce misclassification errors and simplify data collection, thereby easing the burden of participation on patients and clinicians (Table 2). For example, a practical clinical trial end-of-treatment assessment might use a DSM-IV diagnostic checklist, one or two short self-report scales, a checklist of functional outcomes, and summary clinician-rated measures, such as a Clinical Global Impression score, that together might take all of 30 minutes to complete in the context of a routine office visit. For practical clinical trials, the central question is whether these simplified assessments, which by definition should transfer readily to clinical practice, yield reliable and valid inferences regarding the outcome of treatment.

The most vexing objection aimed at the practical clinical trials model as applied to psychiatry is whether endpoints can be found that match this ideal (1, 4, 58). Practical clinical trials in psychiatry have been criticized because, unlike cancer and cardiovascular randomized clinical trials, which use “hard,” dramatic, and unmistakable events (such as mortality) as endpoints, psychiatry trials rely on “soft” behavioral/symptomatic outcomes (4). Although it is true that psychiatric outcomes are subject to greater measurement error, relative to, for example, mortality as an outcome, psychiatric research does have some readily available hard endpoints, such as predefined study rescue procedures, suicide attempts, treatment switching, hospitalization, school failure or truancy, job loss, or even dropping out of the trial itself. Furthermore, it is possible to overestimate the “hardness” of both medical and psychiatric endpoints—consider, for example, the misclassification of myocardial infarction in cardiology trials (59) or the difficulty of suicide adjudication in the International Suicide Prevention Trial (44)—and to underestimate the proven utility of “softer” endpoints, such as quality of life (60), global outcomes, or scores on psychometrically robust rating scales (see, for example, references 61, 62).

Even granting that measures used in a practical clinical trial are reliable and valid, critics argue that site differences and clinician-to-clinician variability in diagnostic and endpoint assessments could conceivably attenuate the ability to identify between-group differences (63). Fortunately, one of the strengths of the practical clinical trials model is the small standard errors associated with the large sample sizes (2). Put differently, the estimate of the true population mean almost certainly will be more precise in a 2,000-subject parallel-group practical clinical trial than in a meta-analysis of ten underpowered 200-subject efficacy trials. This feature is one of the reasons why most designers of practical clinical trials would prefer to collect a clinical global improvement score in hundreds or thousands of patients rather than to measure 100 variables in a small group of patients. Likewise, the ecological validity, and thus generalizability, of the result as it would translate to clinical practice depends on reproducing the variability present in best clinical practice. In this context, a large study “N,” coupled with assessment strategies that capture the normal outcome of treatment, can be seen as an essential ingredient, not a weakness, of the practical clinical trial approach.

The generalizability of a practical clinical trial is determined in large part by the extent to which the trial patients and the practice settings in which they are found resemble the patients and practice settings to which the results of the study are intended to apply. Filters—whether related to inclusion or exclusion criteria or to some aspect of subject or investigator burden associated with quality assurance procedures—attenuate this relationship by selectively limiting study enrollment or encouraging dropout. Complex quality assurance procedures, such as intensive face-to-face training procedures, site visits, and real-time reliability checks, are impossible in a practical clinical trial. Instead, quality assurance in a practical clinical trial typically focuses on consistent implementation and documentation of compliance with assessment and treatment protocols by using adherence checklists. In this feature, practical clinical trials follow the approach of Peto and Baigent (9) with respect to the implementation of quality assurance:

For the simple reason that practical clinical trials are new to psychiatry, it will be necessary to develop novel methods of sample recruitment and retention, new measures to broaden assessment of the effect of interventions at the individual and system levels, innovative research designs, innovative data entry and database management techniques, and new statistical analytic methods. Hence, practical clinical trials in psychiatry inevitably will encourage collaboration of methodologists from diverse academic backgrounds, including epidemiology, statistics, behavioral and social science, engineering, computer science, and public policy. Because longitudinal data are common in psychiatric research, particular attention must be paid to the application of long-time-frame random regression, pattern mixture, and propensity scoring techniques for handling missing data and other sources of loss of randomization; survival analysis models; power and sample size considerations; and concerns regarding multiple comparisons and multiple outcomes across many publications. In turn, attendant methodological innovation will increase the relevance of research findings for community stakeholders such as payers and public policy makers, as well as drive changes in methods for efficacy and hybrid efficacy/effectiveness trials.