The Limits of Using Outcomes as Quality Measures

As value-based systems of payment for health care have become more widespread, so have criticisms of such systems. One complaint is that they can hold a provider accountable for issues that fall outside the provider’s control or usual practice. Another is that such systems potentially penalize providers who care for patients with complex conditions and serious illness. Efforts to address quality issues assessed by value-based systems may involve significant expense. Such efforts also may distort usual practices by emphasizing improvement in some areas while ignoring others that may be more important.

Despite concerns such as these, every indication suggests that the use of value-based payment models will continue to expand. Accordingly, critical attention is better placed on the design of such systems, particularly the quality measures that form their basis. Quality measures used in value-based systems are generally classified as structural, process, or outcomes based. Structural measures assess static characteristics such as staffing ratios or the use of an electronic prescription system. Process measures focus on the performance of certain actions, such as whether patients with a particular diagnosis receive specific treatments or further tests. Finally, outcome measures determine whether treatment or other actions improve patients’ health or other treatment-associated effects. Hospital readmissions and reductions in mortality rates are examples of outcome measures.

Quality measurement programs commonly use process measures more than either structural or outcome measures because process measures are often easier to develop, test, and collect. Yet, some stakeholders believe that quality measurement programs should consist only of outcome measures. For example, the past administrator of the Centers for Medicare and Medicaid Services stated that the agency’s quality measurement programs needed to focus on outcomes rather than processes (1).

Basing quality measurement solely on outcome measures has obvious intuitive appeal. After all, are improvements in health and health-related outcomes not the primary goal of health care? But outcome measures have features that limit their efficacy as part of a value-based payment program. These limitations mean that basing quality reporting primarily on the use of such measures is ill-advised.

The simplest objection to an outcomes-based measurement system is that many important behavioral health quality issues lack outcome measures. All quality measures need to meet certain criteria to be useful. Even when an outcome measure can be conceptualized, its operationalization may fail to meet such criteria. One fundamental requirement is that values for a measure be available for most providers whose performance is assessed. Outcome events that occur at low frequency, then, will not be useful as quality measures. Suicide after psychiatric hospitalization, for example, is often mentioned as an important quality concern. However, because of the low base rate of suicide, only the largest psychiatric inpatient facilities have sufficiently high numbers to generate a reliable, reportable measure. Measurable changes in other types of outcomes may occur only after a significant period after the treating event. As the time to measurable change increases, attributing the change to the previous treating event becomes more questionable.

The use of outcome measures also implicitly assumes that providers understand how to improve outcomes but lack the motivation to do so. However, providers may not know what to do differently to change the outcomes of their treatment or may lack the resources to discover how to do so. A further problem with outcome measures is that they are more vulnerable to gaming or unanticipated adverse effects than are either process or structural measures. A recent illustration of this concerned an outcome measure that assessed the frequency of falls in institutional settings. To improve their scores on this measure, some hospitals were confining patients to their beds, compromising their postdischarge recovery (2). Meanwhile, outcome measures in behavioral health often rely on patient report and are therefore vulnerable to ways in which providers may influence responses.

Another potential form of gaming is through patient selection. If they wish, providers can avoid treating patients who have refractory or high-risk conditions and thereby show better outcomes with no change in practice. One approach to reduce the potential for such gaming is to use statistical methods to adjust for differences in social or other applicable risk factors. Indeed, such an approach is recommended to correct for population differences in factors known to mediate treatment outcomes (3). However, the use of risk adjustment does not fully correct for these shortcomings and introduces problems of its own. One problem is that available data may be inaccurate or omit some variables known to affect health status or outcomes. Efforts to minimize this problem still may not guard against manipulation by providers with information about outcome-related variables not captured by the risk adjustment model.

A further problem is that a risk-adjusted measure only truly incentivizes improvement for the worst-performing providers. Such measures produce results where a large number of providers fall within the confidence interval for the risk adjustment and are identified as not performing differently from the typical provider. The only way for such providers to increase their quality-related reimbursement would be by significantly exceeding the performance of a typical provider. Accordingly, for most providers the business case for improving raw measure outcomes under these circumstances is weak, because only a small percentage of such efforts will be sufficient to affect reimbursement.

For all of these reasons, a quality measurement program based solely on outcomes will likely comprise only a limited number of measures that leave important quality issues unaddressed and pose problems for providers who may not know what to do to change their measure results. Further, such a program will provide the potential for unanticipated adverse effects and gaming by providers. Finally, risk-adjusting such measures reduces incentives for increasing quality.

Value-based payment has the ability to improve provider performance in a variety of areas. But choosing to exclusively base such programs on the use of outcome measures is a highly questionable strategy. The limitations of outcome measures do not mean that process or structural measures should be preferred in a quality measurement program. These types of measures have their own set of advantages and disadvantages that may suit them for assessing some quality issues but not others. Accordingly, an effective quality measurement program will likely have some mixture of all three types. Rather than make a priori judgments about the characteristics of any such program, it would be better to first determine the desired goals of the program. Only then should the number and types of measures likely to achieve these goals be considered. In this way, the benefits of value-based payment programs have the best chance of being maximized.