Community-Based Intervention Research: Coping With the “Noise” of Real Life in Study Design

The ultimate goal of clinical intervention research—whether based in academic medical centers or in real-world community settings—is to find a way to improve the care and lives of people suffering from specific illnesses, disorders, and/or disabilities. Although the goal is the same in each setting and the methods are in many cases similar, the questions guiding research are different. Neither type of research, by itself, is sufficient to reach the goal of improving patients’ care and lives; both are necessary.

The research process begins when a promising treatment approach is identified and/or developed and the decision is made to test it. Traditional, academically based, randomized clinical trials, sometimes called “efficacy” research, test the experimental intervention—whether biological, pharmacological, and/or psychosocial—against a placebo or alternate control treatment, focusing on a single specific main outcome. This approach follows a regulatory or Food and Drug Administration model (1) and is designed to test whether the experimental intervention has a statistically significant effect greater than that of placebo or an alternative treatment. To make this determination, investigators must control, as much as possible, all the variables extraneous to the intervention. They must ensure the internal validity of the study. As a result, subject inclusion criteria are defined as narrowly as possible, procedures are standardized, a single main outcome is defined, and subjects are randomly assigned to receive an active or an inactive, alternate, or control treatment. These procedures reduce the error variance, maximize the signal-to-noise ratio, and decrease the chances of type I and II errors. The data from such a trial are used to determine whether the experimental treatment is active—that is, whether there are statistically and clinically significant differences between the treatment and control conditions.

Community-based intervention trials, sometimes called “effectiveness” research, also test a treatment intervention but occur within the context of the community environment. Conducting a trial in a community setting requires a shift in focus from the mindset of an academically based, regulatory-model, randomized clinical trial. A community-based trial is not a simple replication of a randomized clinical trial with more subjects and more diverse outcome measures conducted in a naturalistic setting. It is not a regulatory-model trial with special attention to power and intervention fidelity. To conduct a community-based trial and to target the need for external validity, a whole new set of issues needs to be considered by the investigator designing the trial, including

A community-based intervention study typically begins with a treatment or components of treatments that have already been shown to be active in a randomized clinical trial, such as medication plus cognitive behavior therapy for treatment of depression. In developing the experimental design and choosing the variables to be measured, an investigator needs to consider how this treatment might be affected by the various contexts (social, cultural, economic, and psychological) in which the trial is conducted. For example, how will people who screen positive for the disorder in question be convinced that they can benefit from treatment and should join the study? Will it matter to recruitment, retention, or study results if some of the people receiving this treatment live in substandard housing and do not have access to transportation? If some of the people who are participating in the study refuse medication but are willing to accept psychotherapy, can the treatment protocol be modified? How will co-occurring substance abuse affect treatment compliance? How will community clinicians be convinced to follow the treatment protocol? In other words, in community-based research, investigators must somehow cope with the “noise” introduced by real life without compromising their ability to draw conclusions about the effect of treatment.

For someone accustomed to traditional, academically based, randomized clinical trials, it might be helpful to think of community-based intervention trials as experiments that are designed to understand how the noise or error variance that exists in community settings (that would be controlled in a randomized clinical trial) affects the intervention, those providing the intervention, and the potency of its effect. The variables to be considered in a community-based trial can include many specific factors, including additional clinical illnesses and disabilities, the treatment setting, the attitudes and training of community clinicians, the system of financing and the organization of care, family, friends, and patients’ and clinicians’ life stresses, personalities, motivations, perceptions, culture or ethnicity, gender, and socioeconomic status.

Both academically based and community-based randomized clinical trials are critically important in reaching the goal of improving mental health care and the lives of those suffering from mental disorders. However, making the transition from conducting research in an academic setting to designing and conducting a community-based trial is a difficult task. What follows is one way to think about community-based intervention research that could facilitate the transition from the academic setting to the community.

The initial goal in moving beyond clinic-based trials is to improve the generalizability or external validity of the study design and to maximize the public health significance of the intervention (see, for example, the web site of the Association of Schools of Public Health for an explanation of the distinction between a public health approach and a medical approach: http://www.asph.org/aa_section.cfm/3/87). However, this entails a large number of choices that the investigator must make. Each choice can have a direct effect on both the study’s total cost and the degree of external validity. Many challenging questions arise in adapting a randomized clinical trial to a community setting. For example,

There is no one way to proceed. But a public health perspective is a useful guide in making choices. It encourages investigators to look beyond the DSM disorder categories to issues of community health and the social context of illness and functioning. It also provides a framework in which to think through the answers to the questions. What follows are public-health-oriented questions related to the sample, the site/community, and the outcome measures that can help frame the decision-making process.

Most of the studies in the literature that are characterized as clinical effectiveness research are expanded versions of efficacy research. Typically, these studies have a large sample, the site is in the community, and the outcomes assessment includes nonsymptom variables such as quality of life. While the generalizability afforded by the inclusion of more diverse samples, settings, and measures does expand the knowledge necessary for moving efficacious treatments into community settings and addressing the problems of individuals, more complex issues need to be considered.

For example, in every clinical trial, particularly in community settings, the results can be somewhat ambiguous and perhaps confusing, particularly when there is little difference between outcomes in the control and experimental groups. Some in the treatment group will fail to improve or will even get worse. Some in the control group will improve dramatically, while others will seem to stabilize. This was the case in a Fort Bragg study that was designed to test whether comprehensive mental health services for children would improve outcomes (3, 4). Even after 5 years, there were no differences between the children receiving a substantial amount of care and those who received little. However, because of the way the study was designed, the investigators did not know what accounted for the lack of difference.

To be able to make sense of ambiguous outcomes, investigators need to design studies that can answer more than, “Does treatment A work better than treatment B?” More complex questions and more diverse methods are necessary. Community-based studies need to be designed to help us understand how the treatment works or does not work for this particular population and under these circumstances and why.

Another way to think about the design of the study is to ask, “What will clinicians in practice settings like this need to know?” The question for a clinician in a practice setting is not, “What on average works best for this disorder?” The question for each clinician is, “What should I do for this specific patient given what I know about the person, his or her life, background, and preferences, the disorder, and the known treatments, and what are the likely outcomes?”

To address these clinical-practice questions, methods and perspectives that have not typically been a part of clinical treatment trials are useful. They come from the social and behavioral sciences. To understand these, a change of focus is needed. The focus needs to be on the situation in which that treatment is being administered, not on the experimental treatment itself. The act of providing treatment is a behavior. It takes place in a social, cultural, economic, and political context. This context affects the behaviors of both the person providing the treatment and the one receiving it. Treatment process research, which aims to identify mechanisms of action in behavioral treatments, is one step in this direction. But to be able to provide information the clinician in practice needs, a much deeper linkage and collaboration with the social and behavioral sciences is necessary.

The theory, methods, and empirical findings of the social and behavioral science literatures have not typically been integrated into clinical research (5). But they have the potential to help us understand many important aspects of the treatment situation. For example, these literatures can help inform our understanding of the relationship between the patient and the clinician and how individual psychological characteristics might affect that relationship and treatment outcomes (6, 7); how people outside the treatment relationship might affect the treatment approach, patient participation in treatment, and treatment outcomes (8); how organizational rules, incentive structures, and morale can affect how clinicians feel about their jobs and treat their patients (9); and how ethnic, gender, age cohort, and religious cultures affect a whole host of human attitudes and behaviors (10, 11), including whether a treatment or clinician will be accepted. However, thus far, little of the information from these and other studies has been incorporated into clinical trials. It is our hope that a new coalition of scientists can change that.

To understand how, why, when, and for whom treatments work and to provide information for practicing clinicians, the teams designing and conducting research need to be more diverse than in the past. A collaborating team requires a partnership among clinical treatment researchers, mental health services researchers, and social and behavioral scientists. Each has an important role.

Clinical treatment researchers have unique experience that nonclinician researchers do not have. This experience is critical to the design of a community-based trial. Their understanding of the disorders and the treatment process make them essential 1) for identifying key places where the behavioral sciences might improve our understanding, 2) for ensuring the integrity of clinical intervention protocols, and 3) for ensuring the safety of human subjects participating in the research.

Mental health services researchers frequently lack sufficient clinical expertise and/or the theoretical background to do this type of research on their own. What they do bring is experience in service and community settings. They know how to implement a study in the field, retain a focus on public health problems, and analyze and interpret complex data from these community settings. They understand the larger context in which treatments are provided, including issues related to the financing of care, the organizational structure of practice settings, and community politics.

Social and behavioral scientists have typically not been included in either clinical or services research. But they have a powerful collection of perspectives, theories, and methodological tools for understanding behavior in social and cultural contexts. However, they do not understand clinical issues and frequently have a difficult time understanding how their research could be applicable to clinical research or issues of public health (5).

Collaboration among these groups of researchers needs to begin even before a design or the choice of a community site is finalized. To be able to design and test meaningful interventions in community settings that are likely to succeed, investigators need to understand the sociocultural context of the people and their community. In order to achieve this understanding, much more than a pilot test of the intervention is necessary. Social scientists who understand in-depth qualitative methods, such as cultural anthropologists, need to be brought into teams. While conducting informal focus groups is a way to begin this process, experts in qualitative methods are critical for identifying elements of organizational, social, and cultural contexts that need to be considered in the design of the study.

The approach described here is obviously a more time- and resource-intensive process than simply transporting an efficacious treatment to a community setting. It includes

But if each of these steps is followed, we can perhaps avoid or at least reduce the number of trials that show no effect. This must be our goal if we hope to answer the fundamental questions faced by clinicians in everyday practice: what works for whom, under what circumstances, and why?

Received Nov. 20, 2000; revision received Aug. 21, 2001; accepted Aug. 29, 2001. From the Division of Services and Intervention Research, NIMH; and the Department of Psychiatry, Western Psychiatric Institute and Clinics, University of Pittsburgh, Pittsburgh. Address reprint requests to Dr. Hohmann, Services Research and Clinical Epidemiology Branch, Division of Services and Intervention Research, National Institute of Mental Health, 6001 Executive Blvd., Rm. 7135, MSC 9631, Bethesda, MD 20892-9631; [email protected] (e-mail).The views expressed in this article are those of the authors and should not be construed as the official position of NIMH.