Common Ground: A Framework for Selecting Core Quality Measures for Mental Health and Substance Abuse Care

We present a framework, summarized in Figure 1, that integrates considerations of the attributes of measures—in a sense, the qualities of quality measures—and dimensions of the mental health system. In doing so, we highlight two challenges to the development of a core set of measures. First, there are fundamental tensions between maximizing the quality of measures and broadly representing diverse features of the mental health system. Second, competing priorities among stakeholders become manifest in the process of selecting what to measure.

In presenting a framework, we seek to make more explicit these measure attributes and system dimensions—and trade-offs between them—in order to educate stakeholders and facilitate the process of developing a core set of measures. Our framework is built on the work of others. Most groups that have developed measures have based their initiatives on a framework, typically some variation of Donabedian's triad of structure, process, and outcome (18). For example, the American Psychiatric Association categorized measures into four groups: access, quality, perceptions of care, and outcomes (19). Several organizations have also described desirable attributes of measures, such as meaningfulness or basis in evidence (9).

We focus on process measures, which examine interactions between patients and the health care system. Many of the concepts we discuss could be applied to other methods of assessing quality of care, including measurement of outcomes, patients' perspectives, and the fidelity of treatment programs to evidence-based protocols. Our focus is not intended to imply that process measures are superior; we have previously written about the importance of varied approaches to quality assessment (20,21). Our focus here reflects the fact that process measures are already widely used in mental health care (6) and could be used more effectively.

An example of a simple process measure is the proportion of outpatients for whom an antidepressant is prescribed for major depression who remain on the medication for the 12-week acute treatment phase (an NCQA measure) (22). With adjustment for differences among patients—not a small matter—this measure could be applied to caseloads of individual providers, to clinics and group practices, and to facilities, health plans, or beneficiaries of public health insurance programs.

In the sections that follow we ask three questions related to principles of core measure selection. What characteristics of process measures are desirable? What features of the mental health system should these measures evaluate? What trade-offs are necessary to achieve balance between the attributes of high-quality measures and the diverse priorities of stakeholders?

Selecting core measures to assess the quality of mental health care is an ambitious undertaking. The mental health care "system," which was described as a "de facto system" first by Regier and colleagues (26) and then more broadly by Fox and colleagues (27), includes a sprawling array of services, settings, populations, and modalities. Each of these dimensions, elaborated on below and depicted in Figure 1, is a potential area for core measures. Which dimensions, and which areas within them, should be given priority? As with decisions about attributes of measures, these decisions can be informed by data but may also differ by stakeholder perspective.

The challenges inherent in reaching consensus among stakeholders on core measures can be seen by viewing Figure 1 not as a two-dimensional diagram but as a multidimensional matrix. Each cell in the matrix represents the intersection of measurement principles, potential subjects of measures, and conflicting stakeholder perspectives that must be prioritized internally and relative to other cells in the selection of measures for common use. Prioritizing and integrating these components was an implicit goal at policy-making forums such as the stakeholder meetings held by the American College of Mental Health Administration and the recent summit at the Carter Center. The prioritization process itself can also be made explicit, as it was in a consensus development process conducted by the Center for Quality Assessment and Improvement in Mental Health at Harvard Medical School. A diverse panel of stakeholders used a modified Delphi process to rate the attributes of measures; rating scores were then used to select a dimensionally balanced set of candidate core measures (Hermann R, Palmer R, Shwartz M, et al, unpublished manuscript, 2001).

Some fundamental tensions have emerged in both implicit and explicit processes. In many cases, assigning priority to one measure comes at the expense of another. Highly meaningful measures of evidence-based practices often require data from medical records and other sources rich in clinical information. However, these data are more costly to collect than more commonly used claims data, and therefore the measures may be less feasible. Detailed specifications produce an accurate and reliable measure that has high feasibility, but potential users may find results from complex measures difficult to comprehend, resulting in low actionability.

Fortunately, the attributes of measures can sometimes be improved. Upcoming federal standards under HIPAA may improve the accuracy and comparability of claims data by specifying data elements and definitions (48). Influential payers such as Medicare are beginning to add clinically important variables to the administrative data that providers submit for reimbursement, facilitating evidence-based quality measurement without chart review (49,50). Even chart review itself—opposed by some groups because of its cost—can become more efficient. Nearly every health care facility does some chart review in response to external requirements. Developing a consensus on the most important chart-based variables would improve the use of limited resources for chart review.

A basic tension arises in the development of a core set of measures between seeking to maximize the quality of measures, depicted by the vertical arrow in Figure 1, and representing the breadth and diversity of the mental health care system, depicted by the horizontal arrow. For example, a principle of measure selection is a strong basis in research evidence. However, selecting measures that are supported by well-controlled research leads to a preponderance of medication measures that pertain to relatively few conditions (6)—hardly a broad representation of mental health care. Although some psychosocial interventions, including assertive community treatment, have a rigorous research foundation, the complexities of these interventions necessitate a more extensive evaluation of fidelity to the empirically tested model. A simple process measurement does not suffice. In contrast, the results of drug trials can be more easily assembled into simple measures of diagnosis, drug selection, dosage, and duration. Well-controlled trials exist for specific types of psychotherapy, such as interpersonal or cognitive-behavioral therapy, but neither administrative claims nor medical records typically document the type of therapy provided.

Understanding these tensions helps to bring into focus some of the trade-offs involved in the production of a set of core measures. Differences in priorities among stakeholders complicate the selection process. There has been little systematic study of the attitudes of mental health stakeholders toward quality measures. However, a review of stakeholders' published reports (6) and observations at national meetings suggests that differences are marked, rational, and deeply held.

Representatives of groups who carry out the data collection for quality measurement—managed behavioral health care organizations, delivery systems, and accreditors—have frequently expressed concern about measurement burden. AMBHA's Performance Measures for Managed Behavioral Healthcare Programs (PERMS 2.0) (14) contains a preponderance of utilization measures, appropriate for a trade association for managed behavioral health care organizations, which principally manage resource utilization. Physicians are trained intensively in technical clinical processes, such as diagnosis and treatment interventions. Thus it is not surprising that the American Psychiatric Association's set of measures emphasizes technical quality of care. Lacking such clinical training, consumers and families often rely on personal experience to identify problems with quality; the consumer-focused measures of the Mental Health Statistics Improvement Project (MHSIP) focus on interpersonal experiences (51). More subtle differences emerge even among groups allied in many areas. Family advocates have encouraged evidence-based measures of medication use and assertive community treatment outreach, while consumers have advocated for measures that promote autonomy, generally by emphasizing such topic areas as recovery, peer support, and housing assistance.

Several opportunities to identify preliminary core measures for mental health care will present themselves over the next few years. The multistakeholder process begun last year at the Carter Center continues to pursue common process and outcome measures for adults and children with mental health and substance use disorders. The American Medical Association's Physician Consortium for Performance Improvement plans to join with other organizations to produce a consensus set of measures for depression. The Agency for Healthcare Research and Quality has funded a National Quality Measure Clearinghouse to disseminate detailed information about selected measures in each area of medicine, including mental health.

The challenge to policy makers, stakeholder group leaders, and measurement methodologists is to work together to select a set of measures that reflects each of their priorities enough to warrant individual participation but that also covers enough common ground to justify broad use. There is widespread anxiety among stakeholder groups that, once selected, these measures—and the underlying treatment processes—will rapidly receive disproportionate attention. Policy leaders might gain stakeholder confidence by emphasizing gradual implementation of preliminary measures designed initially to provide individuals and institutions with confidential feedback for internal improvement activities. Experience with such measures will allow users to improve their feasibility and will generate data to evaluate validity and develop benchmarks. After iterative cycles of measure development, use of the measures for oversight activities and public disclosure may be better received. Fears of undue focus can be assuaged in part by rotating some of the measures in a core set over successive measurement cycles. However, continuous use of some measures allows for an examination of trends over time.

Process measures are most useful when combined with complementary methods of quality assessment. Clinicians can use outcome measures to compare the progress of their patients with that of patients treated at similar facilities. Adding process measures can then highlight areas for improvement when outcomes are lagging (20,21). Simple population-based process measures can indicate whether a patient is receiving an appropriate form of treatment. For example, such a measure will reveal whether an individual with unstable symptoms of schizophrenia and multiple hospitalizations is enrolled in an assertive community treatment program. More complex fidelity measures can be used to "drill down" to evaluate the quality of an individual assertive community treatment program in greater detail (52).

Assessment of patients' perceptions of care provides information about interpersonal process to complement the technical detail of process measurement (53). Another potential linkage is between measures that identify gaps in quality and the availability of effective models for improvement. For example, health plans that score poorly on the NCQA's evidence-based depression measures may be motivated to consider quality improvement models for depression such as those developed by Wells and associates (54) and Katon and colleagues (55).

Further experience with core measures will in turn allow for further refinement of the concept of a single core set from a societal—that is, a broadly based—perspective. It may be that a "core menu" of measures is a more useful vision, with subsets used for individual purposes (quality improvement versus purchasing), settings (specialty versus primary care), and populations (competitively employed versus psychiatrically disabled groups). Despite the obstacles inherent in selecting core measures, we should not lose sight of the advantages: lowering measurement burden, focusing resources for development and testing, and improving the interpretability of results. Each of these steps will advance the mission of identifying quality problems and improving care for mental and substance use disorders.

This work was supported by grant K08-MH001477 from the National Institute of Mental Health, by grant R01-HS-10303 from the Agency for Healthcare Research and Quality, and by the Substance Abuse and Mental Health Services Administration.

Dr. Hermann is assistant professor of psychiatry, health policy, and management at Harvard Medical School and School of Public Health and director of the Center for Quality Assessment and Improvement in Mental Health at the Cambridge Health Alliance. Dr. Palmer is professor of health policy and management and director of the Center for Quality of Care Research and Education at the Harvard School of Public Health. Send correspondence to Dr. Hermann at the Department of Psychiatry, Cambridge Hospital, 1493 Cambridge Street, Cambridge, Massachusetts 02139 (e-mail, [email protected]).