Examining the Effects of Prevention Programs on the Incidence of New Cases of Mental Disorders: The Lack of Statistical Power

The problem of statistical power is most prominent in studies examining universal prevention programs for mental disorders, and this is especially true for disorders with low incidence rates. For example, the incidence rate of anorexia nervosa, a severe and often fatal mental disorder, is estimated to be about 6.3 cases in 100,000 subjects per year (15). A simple power calculation shows that if we were to have a universal intervention that could reduce this incidence by one-third, we would need about 1.5 million subjects in an experimental condition plus another 1.5 million subjects in a control condition to be able to show that this intervention has indeed reduced the incidence (if we assume statistical power to be 0.80 and alpha 0.05, one-sided). In absolute numbers, a prevention study involving 1,565,749 subjects would result in a reduction from 95 new cases of anorexia nervosa in the control group to 63 new cases in the experimental group. Of course, we could drastically reduce the number of needed subjects for a prevention study by concentrating on women, as anorexia nervosa in men is almost nonexistent, or by assuming that a preventive program has effects that reach beyond 1 year. But, whatever we do, the number of needed subjects is enormous and adds up to hundreds of thousands. By way of comparison, a review of school programs aimed at the prevention of eating disorders included six studies, with a total of 1,905 subjects (16). These programs may show effects on knowledge about dieting, on skills, or on eating habits, but a study giving evidence of a reduction in the incidence of eating disorders because of a universal prevention program is clearly not feasible.

Research examining universal prevention of mental disorders with higher incidence rates, such as mood disorders, anxiety disorders, and substance use disorders, has basically the same power problem as prevention of low-incidence disorders, except that the numbers of subjects needed to show a significant reduction are somewhat smaller. Figure 1 shows how many subjects are needed in studies of prevention programs to be able to show a reduction of 15% in the new cases of a mental disorder and the numbers needed to show a reduction of 30%. For example, the incidence of major depression, one of the most common mental disorders, was recently estimated to be 1.7% per year (17). In order to be able to show that a universal prevention program has reduced this to 1.4% (a reduction of 15%), both the experimental group and the control group need to consist of 30,211 subjects. This is considerably smaller than the numbers needed in research for prevention of anorexia nervosa, but it is still very large. For example, the National Comorbidity Survey, one of the most recent and influential epidemiological studies of mental disorders in the United States, consisted of 8,098 subjects (18).

Accordingly, the sheer number of subjects needed to examine the effects of universal prevention programs on the incidence of mental disorders makes this type of research almost impossible to realize. And this is complicated by the ethical problem of imposing an intervention on a population of whom only a fraction will develop the disorder. This does not mean, of course, that universal prevention has no effects on the incidence of mental disorders, but it does mean that it is almost impossible to demonstrate these possible effects in scientific research.

There are more problems in universal prevention that complicate this research even further. One problem is related to the fact that prevention programs typically try to reduce the influence of existing risk factors for mental disorders (14). This is true for universal, selective, and indicated prevention. A large body of research has shown that no single risk factor for mental disorders explains more than 15% of the onsets (19). If a universal intervention is 100% effective in eliminating one risk factor or all consequences of one risk factor, the incidence of new cases of mental disorders cannot be reduced by more than 15%. And an effectiveness of 50% (which would be more realistic) instead of 100% would again dramatically increase the number of subjects needed per experimental condition.

Other problems are that many important risk factors—such as hereditary conditions, life events, gender, and age—cannot be influenced and that the ability of prevention programs to influence risk factors is limited. It may be possible to influence some risk factors, such as socioeconomic status or education level, but this would need political or social actions instead of health programs. Therefore, prevention programs necessarily focus on psychological and psychosocial risk factors and on the consequences of risk factors that cannot be influenced themselves.

Research examining the effects of selective prevention programs is also hampered by power problems. As indicated earlier, these problems can be traced back to the fact that the exact causes of mental disorders are not yet known. Most theoretical models that try to explain the development of mental disorders are based on the assumption that these disorders are caused by a complex interplay of physical, social, and psychological factors. Mental disorders are never caused by one of these factors alone, as no single risk factor explains more than about 15% of the onsets (19). And which combination of factors causes the disorder in a specific individual at a specific time is not known. Moreover, the specificity of most risk factors is low, implying that most subjects who are exposed to the risk factor do not develop the disorder and that one such risk factor by itself is not sufficient to produce the disorder (13).

For example, in a study of subjects who lost their partners, it was found that about 16% met criteria for major depression 1 year after the death, compared to 4% of comparable subjects who did not lose their partners (20). Suppose we could develop an intervention that is capable of reducing this incidence rate by 30%. In this case we would need 671 subjects in each experimental condition of an effect study in order to show this result. This would mean that 671 subjects would be submitted to an intervention in order to reduce the number of new depressive cases from 107 to 75.

Although the feasibility of such a study is better than for a study of universal prevention, the numbers needed for such a trial are still considerable and the costs would be very high. As the presence of a mental disorder has to be determined with a personal diagnostic interview, a study of about 1,350 subjects with repeated measurements would be expensive and would need a strong commitment from financers and researchers.

Only a few randomized trials have examined the effects of selective prevention (for subjects belonging to high-risk groups without meeting criteria for current mental disorders) on the incidence of mental disorder. One of these is a study on the prevention of depression in ethnic minorities with chronic illness by Muñoz and colleagues (21, 22). Unfortunately, the statistical power in this study was too small to show significant effects on the incidence of mental disorder.

In another study, the effects of a preventive intervention for pregnant women on the incidence of postpartum depression were examined (23). It was found that 33% of the subjects in the control group (N=18) developed a postpartum depression, compared to none in the experimental group (N=17). The power problem clearly does not exist in a trial with such a high incidence rate and such large contrast between the experimental and control groups. Other studies in this area, however, did not show such positive outcomes (24). Several more studies have examined the effects of debriefing on the prevention of posttraumatic stress disorder. In a meta-analysis of these studies (25), it was concluded that there is no evidence that debriefing prevents the incidence of posttraumatic stress disorder and that it possibly even has negative results.

Three major randomized, controlled studies of indicated prevention programs (aimed at subjects who have symptoms of a mental disorder but do not meet criteria for the full-blown disorder) that examined the effects of prevention on the incidence of mental disorders are summarized for illustrative purposes in Table 1. These studies do not give a comprehensive overview of studies examining the effects of prevention on the incidence of new cases of mental disorders. Other pilot studies have examined the effects of preventive interventions on psychosis (29) and generalized anxiety disorder (30). But the three studies in Table 1 are among the best trials in the field and can illustrate the power problem well. Two of the three studies are aimed at the prevention of depression, and one targets anxiety disorders. In all three studies a significant reduction of the incidence of new cases of mental disorders was found.

These studies have clearly solved the power problem. Where the incidence of mental disorders is never higher than 15% in high-risk groups, the incidence in subjects who already have some symptoms of the disorder without meeting full criteria can be much higher. The control group in the 2001 study by Clarke and colleagues (26) had an incidence of major depressive disorder of 28.8%, and the intervention reduced this rate to 9.3%. In this study, adolescent children of depressed parents, who themselves had depressive symptoms but no major depressive disorder, received a psychoeducational cognitive behavior training program for mood management to prevent the onset of major depression. The study by Dadds and colleagues (28) even showed an incidence rate of 54% in the control group and 16% in the experimental group (at 6-month follow-up). In this study, primary school children with subclinical anxiety symptoms, but no anxiety disorder, received a cognitive behavior intervention to prevent the onset of an anxiety disorder.

These studies illustrate that relatively small numbers of study subjects are needed when the incidence rate in the target population is high enough. Such studies use a relatively simple method of identifying a high-risk group. They screen a population for the presence of symptoms of a mental disorder. Potential participants who score high on the screening test and meet diagnostic criteria for the mental disorder are excluded from the study, and those who do not meet diagnostic criteria enter the trial. They are randomly assigned to the prevention program or to a control condition. If the incidence in this population is high enough, the number of subjects needed to yield sufficient power is relatively small. And if the intervention is very successful in reducing the incidence, such as in the study by Dadds and colleagues (28), the number of needed subjects can be reduced to less than 50 for the total study group.

There are several ways to increase the statistical power in prevention studies. I will discuss the most important ones.

The lack of statistical power and the related large numbers of subjects needed for prevention studies are important reasons why so few studies have examined the effects of preventions programs on the incidence of mental disorders. This “power problem” makes it almost impossible to design feasible studies examining the effects of universal prevention on the incidence of mental disorders. Studies examining the effects of selective prevention on new cases are feasible but very expensive because of the high number of needed subjects. Indicated prevention can without any doubt be examined best, without the staggering numbers of subjects needed for universal and, to a lesser extent, selective prevention. Several studies examining indicated prevention have shown that this type of research is possible and feasible.

The indicated prevention studies have shown that well-designed interventions are capable of reducing the incidence of mental disorders. This is an important finding, from both scientific and public health perspectives. Many researchers assume that prevention of mental disorders is not possible because the exact causes and pathways that lead to these disorders are not known (36). The indicated prevention studies show that a reduction of the incidence of mental disorders actually is possible. However, critics could argue that subjects with subclinical symptoms of anxiety, depression, or substance use disorders are in fact in the prodromal phase of the disorder (37) and that the indicated intervention is in fact an early intervention, keeping the disorder from developing further instead of actually preventing it. It would therefore be interesting from a scientific point of view to examine whether selective prevention can also reduce incidence. Unfortunately, there is currently insufficient scientific evidence available to answer the question of whether real prevention (universal or selective) of mental disorders is possible.

A major challenge for future researchers in this area is to identify target populations with high incidence rates of mental disorders. Epidemiological research is needed for the identification of these populations. Much research in psychiatric epidemiology defines risk status only in terms of relative risk or odds ratio. Although these measures are informative in illustrating the strengths of risk factors, they are not sufficient for the design of prevention programs. As illustrated in this article, the incidence rate in a high-risk group has to be clear, but in order to compare the importance of different risk factors, other measures are also needed, such as attributable risk (37) and the numbers needed to be treated.

In this article, I have described the power problem of prevention studies, using the prospective, randomized prevention trial as the gold standard. Because of the enormous study group sizes needed for trials examining universal prevention and, to a lesser extent, selective prevention, the use of other designs, such as time-series designs and case-control studies, may be considered in some cases.

This article should not be interpreted as a recommendation to discard universal and selective prevention and focus on indicated prevention and treatment. Universal prevention may be effective in informing individuals about risk factors for mental disorders, reducing stigma related to mental disorder, or stimulating awareness of mental problems in general. Selective prevention programs have been shown to strengthen protective factors and reduce risk factors, and they may reduce the incidence of mental disorders. And combining universal, selective, and indicated prevention programs may result in larger effects than can be realized with each of these interventions alone.

The power problem described in this article is also a relative one. With nonpsychiatric conditions, huge study groups have been studied in prevention research, as was shown in a review of studies of newborn screening for cystic fibrosis (38). In this review two well-designed trials in this area were found, with a total of 1,124,483 included neonates. So there is no doubt that it is partly because of stigma that insufficient resources are available for this type of mental health research.

Five of the “top ten” causes of disability have been identified as mental disorders, and unipolar depression is expected to be the second leading cause of disability worldwide in 2020 (39). Reducing the burden of mental disorders by preventing the onset of new cases is possible, as was shown by the indicated prevention studies described in this article . The further development and study of preventive interventions in this area constitute major challenges for prevention science.

Received May 6, 2002; revision received Oct. 15, 2002; accepted Dec. 17, 2002. From the Trimbos Institute, Netherlands Institute of Mental Health and Addiction. Address reprint requests to Dr. Cuijpers, Trimbos Institute, P.O. Box 725, 3500 AS Utrecht, Netherlands; [email protected] (e-mail).