Toward Evaluation in Public Services: Getting Past the Barriers

In public human services, it is easier to start programs than to evaluate them. Programs are often started by legislators and administrators who know little about what previous initiatives had achieved. Evaluation science, an invention of the 20th century, has a prominent and evolving role in health care of individuals (1,2), but its application to public human services, even before the current change in federal policy (3), has lagged.

What accounts for the omission of such a crucial component of public service? Such evaluation is difficult; recall the decades-long controversy over the benefits of Head Start. The Community Mental Health Center Act of 1963 specifies evaluation as an essential service, but most programs do not follow through. With regard to health services in general, the Center for Medicare and Medicaid Innovation (CMMI), part of the Centers for Medicare and Medicaid Services (CMS; https://innovation.cms.gov), has a charge to do robust evaluation of all payment and service delivery models. In particular, evidence about health outcomes, as opposed to participation in the health care system, has started to emerge^, on several CMMI models, including Medicare Pioneer Accountable Care Organizations (4).

With regard to another significant change in public health care, the privatization of Medicaid, more is known about access and cost than about health outcomes. This gap may begin to be filled as CMS supports measurement science and data collection strategies and as states test their own payment and service delivery models under Medicaid, moving from fee-for-service reimbursement to accountable care or bundled payment arrangements (5).

In public mental health services, process often gets more attention than outcomes. For instance, a Canadian group applied a Cochrane analysis to the development of practice guidelines in child and youth mental health. They found that most guidelines used to guide care do not meet internationally established criteria for guideline development (6). But the role of practice guidelines in measuring outcomes played only a small part in their review. In another review, the U.S.-based advocacy organization Building Bridges (www.buildingbridges4youth.org), which looked at residential treatment, generated “Recommendations for Outcome and Performance Measures” (7). The organization found no consensus in measuring outcomes; the search for outcomes easily got lost, the investigators found, amid many “performance indicators” at payer and provider levels.

Politics plays a role. Voters want to see something get done. The politician who raises questions about effectiveness may lose popular support. Elected officials want to show results to voters while still in office, not afterward, and they do not want to see unfavorable data emerge.

Such political realities are unavoidable. Translating public concern into political will and legislation is not easy. But the relevant barriers can easily take the form of “reasons not to.” These reasons may or may not be stated explicitly.

Some reasons not to evaluate programs can be given names: “HIPAA,” “IRB,” “not my job,” “it’s already evidence based,” “we know what’s right,” “we don’t know enough,” “we don’t have baseline data,” and “there’s too much to do.” Naming and describing the reasons for not evaluating programs allow us to recognize them, identify the core values on which they are felt to be based, and identify a response that can promote evaluation. [The eight “reasons not to” are summarized in a table available as an online supplement to this Open Forum.]