When Quality Indicators Undermine Quality: Bias in a Quality Indicator of Follow-Up for Alcohol Misuse

Valid quality indicators are needed to monitor and encourage identification and management of mental health and substance use conditions (behavioral conditions). Because behavioral conditions are frequently underidentified, quality indicators often evaluate the proportion of patients who screen positive for a condition who also have appropriate follow-up care documented. However, these “positive-screen–based” quality indicators of follow-up for behavioral conditions could be biased by differences in the denominator due to differential screening quality (“denominator bias”) and could reward identification of fewer patients with the behavioral conditions of interest. This study evaluated denominator bias in the performance of Veterans Health Administration (VHA) networks on a quality indicator of follow-up for alcohol misuse that used the number of patients with positive alcohol screens as the denominator.

Two quality indicators of follow-up for alcohol misuse—a positive-screen–based quality indicator and a population-based quality indicator—were compared among 21 VHA networks by review of 219,119 medical records.

Results for the two quality indicators were inconsistent. For example, two networks performed similarly on the quality indicators (64.7% and 65.4% follow-up) even though one network identified and documented follow-up for almost twice as many patients (5,411 and 2,899 per 100,000 eligible, respectively). Networks that performed better on the positive-screen–based quality indicator identified fewer patients with alcohol misuse than networks that performed better on the population-based quality indicator (mean 4.1% versus 7.4%, respectively).

A positive-screen–based quality indicator of follow-up for alcohol misuse preferentially rewarded networks that identified fewer patients with alcohol misuse.