Challenges in the Operationalization of Mental Health Quality Measures: An Assessment of Alternatives

The International Initiative for Mental Health Leadership (IIMHL) has sought to create an international framework for quality measurement. IIMHL is an international organization of mental health leaders seeking to exchange information and collaborate in developing best practices. The IIMHL Clinical Leaders Group has led a multiphase process to create and pilot such a framework. This work began with identifying available quality measures and measurement concepts through a review of existing research and “gray” literature and a survey of IIMHL Clinical Leaders Group members (6,7). The 656 identified indicators were collapsed into 36 broader concepts that were assessed by using a modified Delphi process to collaboratively identify a quality measurement framework; details are described in a previous publication (8). This process was similar to an effort by the Organization for Economic Co-operation and Development that used a Delphi panel of academic and administrative experts as part of the Health Care Quality Indicators project (9).

The next step for the IIMHL Clinical Leaders Group was to attempt to move from the 36 broader concepts to specific, operationalizable quality measures. We returned to the original list of 656 indicators and identified one base indicator and a number of alternatives for each concept, organized around key operationalization issues. The Delphi panel members rated and discussed these alternatives to develop a consensus on these operationalization issues. Table 1 provides examples of some alternatives identified and highlights the decisions needed for successful operationalization.

Among the issues weighed by panel members were decisions balancing data source with accountability for the quality measure. In some cases, choosing data sources that support more feasible implementation shifted accountability. For example, using filled prescriptions from pharmacy databases to assess medication adherence instead of more labor-intensive chart reviews of prescribed or offered medications shifts accountability away from providers and onto consumers. Similarly, use of administrative data to determine wait times can improve feasibility of data collection, but it comes at the expense of not using more granular data (such as direct surveys) collected more proximal to providers and consumers directly involved and affected by the measure.

In many cases, panel members made choices about segmentation (that is, “splitting” rather than “lumping”) of measures, either by patient or service characteristics. Segmentation results in a greater administrative burden in terms of data collection and analysis, but the improved granularity can provide a better “fit for purpose” to drive quality improvement. For example, segmentation by type of injury on inpatient psychiatric units by separating the clinically distinct phenomena of falls and self-injury can drive more specific quality improvement interventions for each mechanism of injury. However, in cases in which more quantification was required, there was no evidence base on which to set specific thresholds (for example, four versus two medication visits a year). It is unlikely that valid evidence beyond clinical consensus will be forthcoming.

These efforts by the IIMHL Clinical Leaders Group are an important first step in operationalizing a consensus international set of quality measures and addressing key issues regarding data source, frequency, period of assessment, thresholds, segmentation, and breadth of the denominator. However, there are still large gaps in data on the operationalization of mental health quality measures. Empirical data are needed to determine which of the many alternatives can be successfully implemented, how the choice of specific alternatives affects provider and system behavior, and how choices affect health care provision and outcomes. It is unclear who will shepherd this empirical evaluation, given a high level of fragmentation of responsibility and variation in the organization of mental health quality measurement, both in the United States and internationally (7). Data system incompatibility will also pose a significant barrier to international cooperation on quality measure comparison and benchmarking (10). Further complications for mental health services include diversity in loci, funding, and methods of care and administrative separation between mental health and general medical care (1).

Nevertheless, empirical data alone will not provide all the answers needed. Policy makers and mental health leaders will need to make qualitative decisions on how to balance the ideal of evidence-based measures with strong validity with the feasibility of data access, the people or systems to be held accountable for performance on the measure, and the ultimate purposes served by the measures. In particular, stakeholder input will be required to tailor the specific choice of measures to the intended use, whether it is targeted quality improvement, benchmarking within or across systems, accreditation or maintenance of standards, or public reporting to facilitate consumers’ health care choices. The work thus far by the IIMHL Clinical Leaders Group has included only clinical leaders of public mental health systems; input is needed from a broader array of providers, administrators, and especially consumers. However, this work can provide a window into the process of decision making that will be required for the successful operationalization of mental health quality measurement.

The authors thank the following members of IIMHL Clinical Leaders Group for their assistance: Peggy Brown, Alan Rosen, Ruth Vine, Tom Callaly, Peter McGeorge (Australia); David Goldbloom, Rohan Ganguli, Catherine Zahn, Paul Kurdyak (Canada); Hugh Griffiths, Susan O’Connor (England); Wolfgang Gaebel, Jürgen Zielasek (Germany); Martin Rogan, Ian Daly (Ireland); Hiroto Ito (Japan); Jan Tromp, Paul Spronken (Netherlands); Memo Musa, David Chaplow, Lyndy Matthews (New Zealand); Torleif Ruud (Norway); Moira Connolly (Scotland); Joseph J. Cheng (Taiwan); and Joe Parks, Ken Thompson, Kevin Hennessy, and Pete Delaney (United States).