The most valuable research integrates information from three levels of investigation: clinical efficacy studies; "real life" effectiveness research, including studies of cost-effectiveness; and health policy research. Although basing decisions on evidence can only improve health care, the method for obtaining good evidence is different for each level. Evidence-based medicine rests on a foundation of multiple randomized controlled efficacy trials summarized in systematic reviews, which are ill suited to answer policy questions.
Why do policy questions require different information? The United States has a decentralized health care system in which health care policy determines the institutional framework but not the direct provision of health care. Examples of health policy include legislation that regulates health insurance, establishes mandates for employers (mental health parity), defines liability (such as in malpractice suits), and reforms Medicaid regulations. Data on treatment efficacy and effectiveness provide important background material but do not represent the type of information policy makers need when they are considering legislation or implementing regulations. Instead, policy makers want evidence about the likely consequences of health care proposals, especially the distributional consequences of proposals—that is, who has to pay, how much, and who benefits.
This information is difficult to obtain through randomized trials. First, policy typically affects a large number of individuals indirectly and has relatively minor effects. Therefore, randomized trials for health policy research would require huge samples and randomization at the level of larger units—cities, counties, and states. Second, the relatively short window of opportunity to inform policy, often just a few months, leaves no time to conduct complex randomized studies. Third, the policy and health care environments change much more quickly than human biology. Data collected five years ago may already be obsolete, whereas data from clinical efficacy studies can be cumulative over many years. Thus we are lucky to find any current data to evaluate a policy proposal, whereas the frequent problem in clinical medicine is the large number of trials, some with conflicting findings. Systematic reviews and meta-analyses can sort out conflicting messages, but they are of no help if there are few data to begin with.
In summary, the most relevant data for informing policy will never come from randomized trials but from observational studies or, occasionally, natural experiments. Imposing the same methodologic rules that guide evidence-based medicine would remove the best policy research and make recommendations even less "evidence based." Let's make sure that this does not become an unintended side effect of the current enthusiasm about evidence-based medicine.
