Comparison of National Mental Health Quality Assessment Programs Across the Globe

A total of 38 qualifying programs were identified in the 12 participating countries. [A list of the programs is available online as a data supplement to this article.] Seventy-nine percent of programs (N=30) were active and provided data for collection, and 16% of programs (N=6) were being planned for future implementation but were at least in the data-collection phase. Sixty-six percent of programs (N=25) were administered by governmental organizations, and 29% (N=11) were administered by nongovernmental organizations, including independent regulators, accrediting or licensing organizations, health insurance providers, research institutes, and professional organizations. The mental health care entities most commonly assessed were hospitals and outpatient mental health care clinics, with fewer programs or countries measuring private health care plans or insurers (Table 1).

Methods to identify and establish quality assessment indicators included literature reviews, forums or consultations with key stakeholders and experts, or adoption from preexisting quality measurement programs. Many programs had specific inclusion criteria for identified measures and included feasibility studies and field testing of determined measures prior to full implementation. In most cases, the organization responsible for the operation of the program was the organization involved in deriving or identifying the quality indicators used (such as the Council of Australian Governments’ National Action Plan on Mental Health and England’s Care Quality Commission’s Periodic Review).

The data collected for the reported programs most commonly came from mental health care utilization databases or registries or from client survey data, and few programs included data directly obtained from physician surveys or insurance claims (Table 2). Results were predominantly reported on a national or regional level (74%, 28 of 38) or on a clinic or organization level (50%, 19 of 38) and were less frequently reported on the level of consumer demographic cohort (34%, 13 of 38), diagnosis (21%, eight of 38), or individual provider (13%, five of 38). Seventy-three percent of programs (28 of 38) presented their data publicly, compared with 13% that did not (five of 38). Sixty-four percent of programs presenting data publicly (18 of 28) presented individualized quality or performance data about the participating care providers or organizations being measured.

The data collected by the programs were used in various ways, often related to the structure and intent of the responsible organization. At the time of our survey, many programs were still in the process of collecting and generating data and had not become active. Programs variably used quality data to identify national or regional targets for quality improvement, to compare existing services with identified benchmarks, to create interventions to improve quality performance, and to track changes over time. In particular, the Department of Health in England and the Scottish Integrated Care Pathways for Mental Health used quality data both on the national level to drive countrywide health care policy and on the local level to drive quality improvement and redesign. Programs such as the National Psychiatric In-Patient Reporting System in Ireland and the Uniform Reporting System in the United States were used to compare regions within the respective country, whereas programs such as the Taiwan Quality Indicator Project were used to compare individual hospitals. Several programs were geared toward accreditation procedures for health care organizations, such as Scotland’s Electro-Convulsive Therapy Accreditation Network and the ORYX(R) Performance Measurement Initiative in the United States. Programs reporting data publicly were also used by stakeholders such as consumers, providers, and employers to allow for comparison of providers or health plans (such as the GGZ Transparency Steering Group Performance Indicators in the Netherlands, the National Quality Indicators in Norway, and the Healthcare Effectiveness Data and Information Set in the United States). Some programs, such as Germany’s Ambulatory Quality Indicators and Key Measures, Taiwan’s National Health Insurance Bureau quality measurement program, and the Physician Quality Reporting Initiative in the United States, were used to determine financial incentives for systems or providers based on quality benchmarks.

Table 3 presents data for the domains and subdomains of quality assessment measured by the indicators included in the reported programs. The presence of indicators within the listed domains was highly variable among the programs. No domain was assessed by every participating country. Subdomains measured by two-thirds or more of participating countries included symptom assessment domains (such as bipolar or depressive disorder, substance abuse, and suicide risk), efficiency and continuity measures (including duration of hospitalization, utilization of outpatient services, and inpatient readmission), safety and legal issues (involuntary hospitalization and use of seclusion or restraints), total population mental health care expenditures, and access to emergency care. The subdomain most commonly measured was duration of hospitalization, assessed by 42% of programs and 83% of participating countries. Domains measured by half or fewer countries were recovery, cultural competence, evidence-based pharmacotherapy, nonpharmacological somatic treatment, and substance abuse.

Participants provided examples of subdomains not included in the survey prompts, such as adequate duration of antidepressant treatment, use of measurement-based care, self-injurious behavior, elopement, assessment of dementia, attention-deficit hyperactivity disorder or borderline personality disorder, use of supported housing or occupational therapy, access to crisis resolution or home health services, and screening for patient strengths and wellness.

This study builds on prior work by the IIMHL Clinical Leaders Project and other groups (the OECD, STAKES, and WHO) in identifying mental health quality indicators and quality assessment programs and in helping to provide a framework to understand the state of international mental health quality assessment. In particular, this study provides a more in-depth and direct comparison of program-level features of quality assessment, such as methods of deriving quality indicators and uses of quality data. The findings of this study are consistent with prior studies, showing a wide variation among established quality assessment programs in indicator derivation, program administration, and utilization of generated data. This may indicate that programs are focused on local priorities rather than on building a consensus framework for mental health quality assessment. This variability may also be attributed to the significant variation in the organization of mental health care systems and quality assessment infrastructure in the participating countries; data collected on these topics will be reported separately. Further study is required to better understand how individual national priorities may be reflected in the choice of quality assessment methodologies, uses, or domains of focus.

Although prior studies by the IIMHL group have broadly demonstrated variety in the domains covered by quality measurement programs, this study extended beyond the prior work by providing an updated and quantified comparison of specific quality indicator subdomains measured by these programs. The domains and subdomains most commonly assessed by the identified programs in this study were largely related to high-acuity mental health care, such as involuntary hospitalization, inpatient readmission, access to emergency care, and use of seclusion and restraints. These areas are consistent with the finding that a high proportion of the identified programs measure hospital-based care and may reflect either a general international consensus regarding the critical domains of mental health quality measurement or a greater availability of data for indicators measured in these domains. The low proportion of recovery-oriented quality assessment measures, despite the high proportion of programs and countries collecting consumer-level data, is notable given the recent emphasis, including from the IOM, on recovery-oriented, patient-centered services (5). The impact of this misbalance of represented quality domains requires further exploration, but it is concerning because of a potential lack of focus of national policies and resources on recovery-oriented, outpatient, and preventive services. These findings may also indicate the need for a more balanced range of mental health care quality indicators and provide further impetus for the development of a consensus framework for mental health quality assessment.

There are a number of important limitations to this study. The surveys and responses were generated in English only because we had insufficient resources to provide translation; respondents were responsible for translation from native languages. This may have placed a larger burden on respondents from non–English-speaking countries and influenced both the interpretation of the survey prompts and generated responses. Furthermore, the study focused on only a subgroup of developed countries; further study is required to determine how these data relate to mental health systems in low- and middle-income countries. Some reported programs did not have distinct elements separating data collection, measurement development, and data reporting, which may be attributable to countries with more centralized mental health quality structures. The analysis included programs that have been implemented and programs that are still in the implementation phase. The yet-to-be implemented programs were included in the analysis because they are still able to reflect national methods and priorities of mental health quality assessment, such as domains of interest, data sources, and uses of data. We did not identify the feasibility of full implementation of these programs, and therefore the results may be skewed away from truly feasible quality assessment. Also, the reported percentages are overrepresentative of countries without a unified mental health quality assessment infrastructure, with countries such as the United States and Ireland having multiple independent groups measuring quality data and thus more reported programs. In addition, participants from countries with decentralized systems may not have fully identified quality measurement programs in their country, partly due to the nature of their positions in the public mental health system, and therefore may have contributed to a skew in program-level reporting. The study was reliant on participants to report only programs and quality domains that met the established criteria for quality assessment, and independent verification of validity could not be reliably established. Additional important domains and subdomains of mental health quality assessment may exist that were not included in this survey and may require further exploration.

Both this study and prior IIMHL Clinical Leaders Project studies have provided mental health leaders the opportunity to exchange information and to obtain peer-to-peer consultations for the assessment and implementation of quality improvement initiatives. Ultimately, the goal of this project is to establish a consensus international framework for mental health quality assessment and thus a means to compare key measures of performance across countries. The second phase of this project involves examining all of the collected mental health quality indicators and creating a set of core quality measurement concepts based on their validity, importance, and feasibility. Furthermore, we intend to create an international network linking quality measurement groups or organizations within each country to help facilitate quality measurement framework development and assist implementation of the core measures.

Further steps in research and practice will be required to improve quality assessment internationally, including establishing tighter links between process and outcome measurements, increasing use of standardized assessments, expanding use of information technology, delineating benchmarks for comparison across settings, increasing investment in quality research, and integrating mental health quality assessment into the broader framework of health care quality (11). The development of a common framework of mental health quality assessment may be an instrumental first step in the process of refining mental health quality assessment. The results of this study will help provide data to better understand shared priorities for mental health quality assessment and may help identify barriers to development of a common framework.

This study sought to assess existing or soon-to-be implemented mental health quality assessment programs among participating members of the IIMHL Clinical Leaders Project. Thirty-eight qualifying programs from 12 participating countries were identified. Most of the programs assessed were active, were administered by governmental organizations, were focused on hospital care, and used encounter or utilization databases as sources of information. Different methods were used by the programs to identify and establish quality indicators. Program data were generally publicly reported and were used for various purposes by their respective organizations or countries. A wide range of domains of quality were represented in the reported programs, although most commonality was seen in domains associated with high-acuity mental health care, with fewer programs assessing domains related to recovery. These data will help future work intended to establish an international framework for mental health quality assessment.

Funding was provided by grant R25 MH086466 from the National Institute of Mental Health. Additional funding came from government and nongovernment organizations of the countries participating in the IIMHL project (Australia, Canada, England, Germany, Ireland, Japan, the Netherlands, New Zealand, Norway, Scotland, Taiwan, and the United States). The authors thank the members of the IIMHL for their assistance with this project.

Dr. Parameswaran is the recipient of an unrestricted educational grant from the American Psychiatric Institute for Research and Education and Janssen Pharmaceuticals. Dr. Pincus receives travel support as a consultant for Value Options and publication royalties from the American Psychiatric Press and from Lippincott, Williams and Wilkins. The other authors report no competing interests.