Surveying the Landscape of Quality-of-Care Measures for Mental and Substance Use Disorders
Abstract
Objective:
Quality measures that are used to evaluate health care services have a central role in monitoring and incentivizing quality improvement and the provision of evidence-based treatment. This systematic scan aimed to catalog quality-of-care measures for mental and substance use disorders and assess gaps and redundancies to inform efforts to develop and retire measures.
Methods:
Quality measure inventories were analyzed from six organizations that evaluate health care quality in the United States. Measures were included if they were defined via symptoms or diagnoses of mental and substance use disorders or specialty treatments or treatment settings for adults.
Results:
Of 4,420 measures analyzed, 635 (14%) met inclusion criteria, and 376 unique quality-of-care measure constructs were cataloged and characterized. Symptoms or diagnoses of disorders were most commonly used to define measures (46%, N=172). Few measures were available for certain disorders (e.g., anxiety disorders), evidence-based treatments (e.g., psychotherapy), and quality domains (e.g., equity). Only one in four measures was endorsed by the National Quality Forum, which independently and critically evaluates quality measures. Among measures that were actively in use for national quality improvement initiatives (N=319), process measures (57%) were most common, followed by outcome measures (30%), the latter of which focused most often on experience of care.
Conclusions:
A vast landscape of mental and substance use disorder quality-of-care measures currently exists, and continued efforts to harmonize duplicative measures and to develop measures for underrepresented evidence-based treatments and quality domains are warranted. The authors recommend reinvesting in a national, centralized system for measure curation, with a stakeholder-centered process for independent measure review and endorsement.
HIGHLIGHTS
•
Quality measures used to evaluate health care services have a central role in tracking and incentivizing improvement of care quality and provision of evidence-based treatment.
•
This systematic environmental scan of 4,420 quality measures identified 376 unique measure constructs that can be used to evaluate the quality of mental health or substance use disorder treatments.
•
Hundreds of similar measures indicated a need for ongoing efforts to harmonize or retire measures, and gaps identified suggested a need to develop measures for select disorders, evidence-based treatments, and quality domains.
•
Recommendations include reinvesting in a national centralized system for curation of care quality measures with a cross-agency, stakeholder-centered process for independent review and endorsement of measures.
Measures used to evaluate the quality of health care services are commonly tied to compensation, reimbursement, and reputation; are used to incentivize care quality; and hold service providers accountable (1, 2). The results from evaluations that involve use of quality measures are often publicly reported so that patients and purchasers can decide where to seek or buy health care (3). Thus, the measures on which evaluations of quality and performance are based serve a central role in policy making, health care administration and delivery, and the quality of health care that patients receive.
A reasonable question is whether there are too few or too many quality measures of mental and substance use disorder care (4). Such measures are numerous and vary in how rigorously their measurement properties (e.g., reliability and validity) have been evaluated, in the strength of evidence supporting their core concepts or use to improve quality, and how much they overlap with similar measures (relevant for harmonization efforts) (5–8). In 2015, Patel et al. (7) found in a systematic environmental scan of 510 mental health quality measures that 10% had received National Quality Forum (NQF) endorsement and that 5% were used for national quality reporting initiatives in the United States. In 2016, Goldman and colleagues (9) found in a systematic environmental scan of 730 quality measures of integrated medical-behavioral health a heavy focus on care during and after hospitalization and an inadequate representation of the full range of evidence-based treatment options (e.g., in outpatient settings). The field appears to have both too many quality-of-care measures overall and not enough good measures in specific areas. For the field to move forward, it is essential to identify the measures with the most potential to promote implementation of evidence-based mental health services, pinpoint where developers should focus on filling measurement gaps, and find redundant measures or those with undesirable characteristics that should no longer be used (10).
The landscape and definitions of quality measures change rapidly. Measure inventories have changed since previous publication of measure reviews. The National Quality Measures Clearinghouse (NQMC), the primary catalog curator of measures in the United States for 17 years, was defunded in 2018 (11). Past studies did not include measures developed at the Veterans Health Administration (VHA), the largest integrated health care system in the United States and a leader in mental health performance management (12). Therefore, an updated and restructured study on the landscape of clinical quality measures for mental and substance use disorder treatments is warranted. The primary purpose of this study was to assess measurement gaps (e.g., measure characteristics and clinical focus) and redundancies to inform quality-of-care measure development and retirement efforts. We also aimed to provide a catalog containing a snapshot of quality measures that can be sorted by key measure attributes, because no single authoritative source currently exists for stakeholders to consult on the large number of available measures. We also make recommendations for strategies to enhance and curate the catalog into the future.
Methods
We searched public information on quality measures from six organizations involved in nationally standardized evaluation and comparison of mental or substance use disorder care quality. Measures were identified and cataloged from March 1, 2019, to October 31, 2020. We scanned the entire inventory of quality measures from the Office of Mental Health and Suicide Prevention and Office of Performance Measurement of the U.S. Department of Veterans Affairs (VA) (13–15), the Measure Inventory Tool of the Centers for Medicare and Medicaid Services (CMS) (16), the NQF Quality Positioning System (17), the 2019–2020 Healthcare Effectiveness Data and Information Set of the National Committee for Quality Assurance (NCQA) (18), the National Healthcare Quality and Disparities Report of the Agency for Healthcare Research and Quality (19), and the strategic plan for fiscal years 2019–2023 of the Substance Abuse and Mental Health Services Administration (SAMHSA) (20).
Measures were included in our catalog if they had been defined through the use of symptoms or diagnoses for substance-related and addictive disorders (excluding tobacco use disorder), depressive disorders, trauma and stressor-related disorders, anxiety disorders, schizophrenia spectrum and other psychotic disorders, bipolar and related disorders, or suicide and related behaviors (21). Measures not defined with disorders were included if they were defined with the use of data elements relevant to psychotropic medications or in any care delivery setting for managing specialty mental or substance use disorders or if the measure was in use by a national mental or substance use disorder quality improvement initiative (e.g., the CMS Inpatient Psychiatric Facilities Quality Reporting program [22]). We included measures that evaluated care for adults ages ≥18 years; measures that evaluated only pediatric services were not included.
Measures deemed to tap the same quality construct, or with an identical definition, were reconciled to create a set of unique constructs of quality measures to evaluate care for patients with mental or substance use disorders. We abstracted information about attributes of each measure construct that might be of interest to evaluators, purchasers, and other stakeholders who must choose among available measures (Box 1). Measures defined via the use of disorder symptoms or diagnoses were coded according to specific mental or substance use disorders, or medical disorders (multiple disorders when applicable), because evaluations commonly focus on specific disorders. We coded each measure’s NQF endorsement status as an indicator of critical, independent, and transparent review of measure properties (e.g., reliability and validity) (23). We coded each measure’s “current use” in national quality improvement initiatives as an indicator of its potential for cross-system comparisons. For example, for CMS measures current use was defined as having an “active” status associated with any of the agency’s programs listed in the CMS Measure Inventory Tool (16). Additionally, because measures are used by different types of health care organizations (provider vs. payer) and for different goals (internal quality improvement vs. pay-for-performance), we categorized measures by attributes of interest to stakeholders: modality of treatment, type of quality measure, domain of quality, level of analysis, and the data source from which a measure can be calculated (23–29). See Box 1 and the online supplement to this article for detailed definitions of coded measure characteristics.
BOX 1. Cataloged attributes of mental or substance use disorder quality-of-care measures
Symptoms or diagnoses in measure definition
Substance use disorder
Alcohol use disorder
Opioid use disorder
Other drug use disorders
Mental health conditions
Depressive disorders
Trauma and stressor-related disorders
Anxiety disorders
Schizophrenia spectrum and other psychotic disorders
Bipolar and related disorders
Suicide and related behaviors
General medical health and medical conditions
Diabetes mellitus
Cardiovascular disease
Kidney disease
Neurological diseases
Hepatitis C
HIV/AIDS
Other: pain, pregnancy, obesity
Treatment modality
Pharmacotherapy
Nonmedication somatic treatments
Psychotherapy by a licensed independent provider
Psychosocial services, including services by nonlicensed independent providers
Case or care management
Non–face-to-face modalities (telehealth)
Symptom screening and treatment planning
Environmentally managed settings (residential, inpatient)
Type of measure
Structure
Access
Process
Outcome
Cost and resource use
Experience of care
Person reported (patient or provider reported)
Composite
Use for national quality improvement
National Quality Forum endorsement status
Centers for Medicare and Medicaid Services (CMS) evaluation program
National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set
Veterans Affairs evaluation program
National Healthcare Quality and Disparities Reports
Key measure of strategic plan of the Substance Abuse and Mental Health Services Administration
Quality domain
Meaningful measurement area of the CMS
Make care safer by reducing harm caused in the delivery of care
Strengthen person and family engagement as partnership in care
Promote effective communication and coordination of care
Promote effective prevention and treatment of chronic disease
Work with communities to promote best practices of healthy living
Make care affordable
Institute of Medicine quality domain
Timely
Efficient
Effective
Safe
Patient centered
Equitable
Analysis level
Single clinician or provider
Clinician or provider group or practice level
Facility (e.g., inpatient psychiatric facility)
Health plan
Health care or integrated delivery system
Population or geographic area
Data source
Electronic medical records
Health records (e.g., paper)
Claims or other administrative data
Electronic clinical data
Registry
We summarized the landscape of measure constructs by their attributes, presented as counts and percentages of unique measure constructs. The complete measure catalog is available in the online supplement. This study was approved by the Stanford University Human Subjects Research and Institutional Review Board.
Results
Of 4,420 measures reviewed, 14% (N=635) met our inclusion criteria. After we had reconciled measures with the same definition or construct, we included 376 unique constructs of measures of quality of care for mental or substance use disorders in our catalog (Figure 1). Among these cataloged measures, symptoms and diagnoses of specific disorders were used most often in defining the measures (46%, N=172). Among the 54% (N=204 of 376) of measures not defined by disorders, experience-of-care measures were most common (27%, N=54), followed by measures defined by using inpatient psychiatric stays (18%, N=37) (Table 1).
FIGURE 1. Identification and inclusion of quality-of-care measures for treatments of mental or substance use disordersa
aMeasures from sources (N=635) that were deemed to tap the same quality construct, or that had identical definitions, were reconciled to create a set of unique calculable quality measure constructs (N=376).
| Attribute used to define measure | N | % |
|---|---|---|
| Based on treatment modality or setting (all) | 110 | 54 |
| Psychiatric inpatienta | 37 | 18 |
| Residentiala | 13 | 6 |
| Outpatient- or population-based utilization | 25 | 12 |
| Pharmacotherapy | 24 | 12 |
| Antipsychotic medications | 5 | 3 |
| Antidepressants | 1 | 1 |
| Anxiolytics | 7 | 3 |
| Antimanic agents | 3 | 2 |
| Opioid agonists or antagonists | 2 | 1 |
| Other classes | 6 | 3 |
| Psychotherapy | 4 | 2 |
| Other subspecialty treatmentsb | 9 | 4 |
| Experience-of-care measures (all)c | 54 | 27 |
| Patient reported | 52 | 26 |
| Clinician or provider reported | 3 | 2 |
| Based on clinical risk (all) | 23 | 11 |
| Risk related to suicide | 21 | 10 |
| Risk based on patient-level predictive modeling | 2 | 1 |
| Other concepts (e.g., psychosocial functioning) | 12 | 6 |
| Other health care system concepts (e.g., staffing) | 3 | 2 |
| Not categorized | 2 | 1 |
a
Two measure definitions included both inpatient and residential settings and are duplicated across rows. The category total for treatment modality or setting was adjusted to remove the duplicates.
b
Other subspecialty treatments included assertive community treatment, psychiatric rehabilitation programs, supported employment, and behavioral health care programs with community reintegration components.
c
One composite measure was defined by using both patient- and provider-reported data and is duplicated across rows. The category total for experience-of-care measures was adjusted to remove the duplicate.
Table 2 cross-classifies whether a measure was disorder based, NQF endorsed, or used in a national quality improvement initiative with other measure attributes. Note that many of these classifications were not mutually exclusive. Ninety-five measures (25%) were endorsed by the NQF, indicating that only one in four met independent review criteria (e.g., reliability and validity). The CMS and NCQA inventories overlapped more with NQF’s inventory of endorsed measures (81% and 28%, respectively) than with the VA or SAMHSA inventories (10% and 1%, respectively). We identified 319 quality measures actively in use for national evaluations, our indicator of potential for use in cross-system comparisons. The VA inventory had the most measures used in national quality improvement efforts (N=193), followed by the CMS (N=102) and NQF (N=98) inventories. Of measures in use for national quality improvement efforts, process measures were more commonly used (57%) than outcome measures (30%).
| Quality measure source or attribute | Symptom- or diagnosis-based measure (N=172) | Not symptom- or diagnosis-based measure (N=204) | National Quality Forum endorsed (N=95) | Used in national quality improvement initiative (N=319) | ||||
|---|---|---|---|---|---|---|---|---|
| N | % | N | % | N | % | N | % | |
| Organizationa | ||||||||
| Veterans Health Administration | 79 | 46 | 114 | 56 | 9 | 10 | 193 | 61 |
| Centers for Medicare and Medicaid Services (CMS) | 62 | 36 | 85 | 42 | 77 | 81 | 102 | 32 |
| National Quality Forum | 52 | 30 | 61 | 30 | 89 | 94 | 98 | 31 |
| National Committee for Quality Assurance | 33 | 19 | 0 | — | 27 | 28 | 32 | 10 |
| National Healthcare Quality and Disparities Reports | 6 | 4 | 20 | 10 | 19 | 20 | 26 | 8 |
| Substance Abuse and Mental Health Services Administration | 19 | 11 | 2 | 1 | 1 | 1 | 21 | 7 |
| Quality measure type | ||||||||
| Structure | 3 | 2 | 31 | 15 | 0 | — | 32 | 10 |
| Access | 1 | 1 | 0 | — | 1 | 1 | 1 | <1 |
| Process | 134 | 78 | 91 | 45 | 36 | 38 | 183 | 57 |
| Outcome | 30 | 17 | 76 | 37 | 58 | 61 | 94 | 30 |
| Cost and resource use | 1 | 1 | 2 | 1 | 0 | — | 2 | 1 |
| Patient or provider reporteda | 17 | 10 | 67 | 33 | 50 | 53 | 70 | 22 |
| Compositea | 6 | 4 | 44 | 22 | 27 | 28 | 38 | 12 |
| Treatment modalitya | ||||||||
| Screening and treatment planning | 39 | 23 | 77 | 38 | 46 | 48 | 87 | 27 |
| Pharmacotherapy | 80 | 47 | 113 | 55 | 73 | 77 | 185 | 58 |
| Nonmedication somatic treatment | 40 | 23 | 83 | 41 | 62 | 65 | 118 | 37 |
| Psychotherapy by licensed independent provider (LIP) | 63 | 37 | 98 | 48 | 61 | 64 | 155 | 49 |
| Psychosocial services by non-LIP | 59 | 34 | 94 | 46 | 61 | 64 | 147 | 46 |
| Population, case, or care management | 53 | 31 | 89 | 44 | 52 | 55 | 137 | 43 |
| Environmentally managed setting | 60 | 35 | 102 | 50 | 64 | 67 | 140 | 44 |
| Tele-mental health | 58 | 34 | 85 | 42 | 54 | 57 | 139 | 44 |
| Not categorized | 37 | 22 | 24 | 12 | 11 | 12 | 43 | 14 |
| Single treatment modality measured | 56 | 33 | 68 | 33 | 11 | 12 | 108 | 34 |
| CMS meaningful measures area | ||||||||
| Make care safer by reducing harm caused in the delivery of care | 7 | 4 | 15 | 7 | 8 | 8 | 14 | 4 |
| Strengthen person and family engagement as partners in care | 1 | 1 | 57 | 28 | 46 | 48 | 56 | 18 |
| Promote effective communication and coordination of care | 27 | 16 | 46 | 23 | 8 | 8 | 61 | 19 |
| Promote effective prevention and treatment of chronic disease | 132 | 77 | 64 | 31 | 33 | 35 | 163 | 51 |
| Work with communities to promote best practices of healthy living | 5 | 3 | 16 | 8 | 0 | — | 19 | 6 |
| Make care affordable | 1 | 1 | 6 | 3 | 0 | — | 6 | 2 |
| Institute of Medicine quality domain | ||||||||
| Timely | 0 | — | 5 | 3 | 0 | — | 5 | 2 |
| Efficient | 3 | 2 | 8 | 4 | 1 | 1 | 10 | 3 |
| Effective | 149 | 87 | 76 | 37 | 42 | 44 | 189 | 59 |
| Safe | 9 | 5 | 33 | 16 | 6 | 6 | 33 | 10 |
| Patient centered | 0 | — | 64 | 31 | 46 | 48 | 53 | 17 |
| Equitable | 11 | 6 | 19 | 9 | 0 | — | 30 | 9 |
| Analysis levela | ||||||||
| Individual clinician or provider | 38 | 22 | 16 | 8 | 13 | 14 | 23 | 7 |
| Clinician or provider group or practice | 40 | 23 | 16 | 8 | 15 | 16 | 25 | 8 |
| Facility | 33 | 19 | 65 | 32 | 48 | 51 | 63 | 20 |
| Health plan | 42 | 24 | 39 | 19 | 57 | 60 | 66 | 21 |
| Health care system or integrated delivery system | 103 | 60 | 143 | 70 | 40 | 42 | 223 | 70 |
| Population or geographic area | 26 | 15 | 42 | 21 | 41 | 43 | 61 | 19 |
| Not categorized | 14 | 8 | 0 | — | 0 | — | 14 | 4 |
| Data sourcea | ||||||||
| Electronic medical record | 98 | 57 | 120 | 59 | 29 | 31 | 208 | 65 |
| Other health record (e.g., paper) | 34 | 20 | 26 | 13 | 27 | 28 | 44 | 14 |
| Administrative data | 53 | 31 | 19 | 9 | 34 | 36 | 52 | 16 |
| Standardized electronic clinical data | 17 | 10 | 7 | 3 | 12 | 13 | 21 | 7 |
| Registry | 18 | 11 | 7 | 3 | 3 | 3 | 19 | 6 |
| Other (population-based sampling) | 28 | 16 | 74 | 36 | 47 | 50 | 79 | 25 |
a
Assignment to these categories was not mutually exclusive, and summed cell totals may exceed the total number of unique measure constructs reported in the column header.
Regarding other measure attributes, we observed substantial overlap (or lack of specificity) regarding treatment modalities; two-thirds of the measures (67%, N=252) each tapped multiple treatment modalities. Pharmacotherapy measures were included most often among the treatment modalities we characterized (58%, N=185) (Table 2). In terms of quality domains, measures assessing clinical effectiveness were well represented, with focus on other quality domains such as patient-centeredness, and communication being more prominent among measures not defined by a disorder. Whether categorized by CMS Meaningful Measures Area (N=26) or Institute of Medicine (IOM) Quality Domains (N=24), few measures were represented across domains such as “make care safer,” “work with communities,” “make care affordable,” “timely,” “efficient,” and “equitable.” Regarding analysis level, 77 measures (20%) were specified for more than one analysis level, and two measures (1%) were specified for all six analysis levels. We identified 261 measures (69%) that could be calculated from routinely collected electronic data sources such as electronic medical records (EMRs) or standardized electronic clinical data.
Figure 2 displays a more detailed analysis of cross-classifying measures defined by disorder, measure type, and treatment modality. Small black circles in the figure represent the number of measures defined with symptoms or diagnosis of only a single disorder; larger gray circles represent the number of measures that were defined with any of multiple disorders. Of 72 quality-of-care measures for substance use disorder treatment, 51 would evaluate quality of care for all substance use disorders combined; the other 21 measures would evaluate symptom screening, treatment planning, or pharmacotherapy specifically for alcohol or opioid use disorders. Depressive disorders were represented most often (60%, N=73) among the 121 mental disorder quality measures. Measures for schizophrenia- and bipolar-related disorders (50%, N=60) were often grouped into a “serious mental illness” construct; only 13 of these measures would evaluate care for one of these disorders separately. Fewer measures have been defined to evaluate care quality for trauma- and stressor-related disorders (12%, N=15) or anxiety disorders (1%, N=1) independent of other mental disorders (Figure 2).
FIGURE 2. Distribution of quality-of-care measures for mental or substance use disorder treatment, defined by disorder, measure type, and treatment modalitya
aN=172, based on unique, calculable mental or substance use disorder quality-of-care measures or measure constructs that had symptoms or diagnoses of mental or substance use disorders in their definition. LIP, licensed independent provider.
Discussion
The landscape of quality-of-care measures to evaluate mental or substance use disorder treatments is vast and rapidly changing. Comprehensive inventories such as the NQMC have been defunded or have not been recently updated (11, 30). In the absence of a centralized curator, measures are siloed within disparate agency repositories. We sought to identify gaps and redundancies in the current landscape of quality-of-care measures for mental and substance use disorder treatments among adults, culled from six different organizations, and to make available a snapshot of measures organized by their attributes. We hope that measure developers can use this information to focus on current gaps and not add to the already significant redundancies in this corpus. We also hope that evaluators, purchasers, and other stakeholders can use the results of this study, along with data available from agency repositories, to select and sort measures by important attributes: clinical focus (e.g., disorders or treatment modalities), units to be analyzed (e.g., provider or payer), NQF endorsement status, data source, and potential for cross-system comparison (31). Ideally, measure selection would also be informed by evidence indicating a measure’s psychometrics and likelihood to improve patient outcomes. However, we found no such systematically graded evidence in the inventories scanned, a long-standing issue (6, 32) that all stakeholders should work to remedy.
Major Measure Gaps
Any new measure should complement the already vast and expanding landscape of quality-of-care measures for mental or substance use disorder treatments (6–9). We found more outcome measures than did previous studies (7, 9), of which experience-of-care measures were most numerous. We found no outcome measures based on symptom improvement or remission for most disorders, other than depressive disorders, and few options to evaluate outcomes such as functioning or health-related quality of life. Process measures still dominate the landscape. Researchers have called for developing measures of evidence-based psychosocial treatments (vs. generic codes) (6)—a gap that persists. For example, we cataloged 161 measures that evaluate provision of treatments including psychotherapy (33), but only four of those measures would enable a distinct evaluation of psychotherapy. Psychotherapy is part of 40%−50% of outpatient treatments of mental health or substance use disorders and is the only treatment modality for 10%−16% of patients who receive outpatient treatment (34). Devoting four of the 376 total measures in the landscape to distinct evaluations of psychotherapy quality is not representative of the types of mental health care provided to patients.
Quality domains underrepresented in previous studies—equity, safety, and patient and family engagement in care (7, 9, 35)—appeared to be slightly better represented in our catalog, but still with significant room for increasing representation of measures within these domains. Other measure features that lacked coverage were those that assess structural aspects of quality (e.g., night and weekend hours of services and waivered buprenorphine prescribers), population identification and access, and efficiency. We found more measures calculated from EMR and patient-reported data than were reported in previous studies (7), suggesting a promising trend toward use of data sources that may be required to address gaps in available measures. However, measures specified and validated for use at multiple levels of analysis remain scarce, potentially limiting efforts to align quality improvement across stakeholders (clinicians, purchasers, and others). Evidence of a measure’s reliability and validity at one level does not necessarily generalize to other levels. Developers should analyze cross-level applications of their measures.
The Need for Independent Evaluation of Measures Before Use
Before measures are used, they should be independently and transparently evaluated. We note that 81% of CMS measures were NQF endorsed. Although CMS is not required to subject measures to NQF evaluation, it typically does so to ensure that only the highest-quality measures are used in CMS programs. In NQF’s review process, each measure is evaluated for importance (evidence and gap in performance), scientific acceptability (reliability and validity); feasibility, usability, and use; and harmonization (36). One of the most important gaps revealed by our analysis was the relative rarity of NQF-endorsed measures or of other independent and transparent evaluations, especially for measures in the VA and SAMHSA inventories. A policy intervention that could improve the landscape of quality-of-care measures for mental and substance use disorder treatments would be to more strongly encourage or require independent and transparent evaluation of measures before they are used in federal or high-stakes programs (e.g., national pay-for-performance initiatives).
Lump or Split?
Important trade-offs exist between measures that combine diagnoses (e.g., all substance use disorders) and measures targeting specific diagnoses. Combining diagnoses into a single measure is more efficient, reduces measurement burden, and has face validity where care delivery processes or patient outcomes are transdiagnostically relevant (e.g., access to care). However, combined measures risk missing clinically important details about specific conditions and masking poor performance for low-prevalence diagnostic groups. Patel et al. (7) reported that 32% of measures were defined with more than one disorder. More than half of the measures defined by disorder in our study were defined with more than one disorder. Combining diagnostic groups might reflect efforts to harmonize measures and reduce measurement burden (6–9). However, our results provided no clear evidence of greater overall harmonization since publication of previous studies that systematically scanned for and examined measures used to evaluate the quality of mental and substance use disorder treatment. The development of more diagnostically focused measures may be warranted where measures are lacking (e.g., anxiety disorders and drug use disorders other than alcohol and opioids). Before developing such measures, it would be informative to analyze how well performance on combined measures reflects performance for diagnostic subgroups and how tightly diagnosis-specific processes and outcomes are linked with more general measures.
Essential Investment in Curation
A significant limitation of this study was that it will soon be out of date. No research program can sustain the continual curation needed, and the status of measures will need to be verified with agency repositories. Given the resources spent by the federal government on developing and evaluating new measures, and the significant potential for waste by developing redundant measures and failing to identify measurement gaps, investment in a national curation program is essential. CMS alone spent $1.3 billion between 2008 and 2018 to develop nearly 2,300 measures, 35% of which are being used in CMS programs (37). This massive investment in infrastructure is just one among federal, state, and private organizations’ measurement enterprises, each of which is largely siloed from the other. The NQMC was a national resource (11) and strong curation program that should be revived. It is time to fulfill committee recommendations from the IOM to establish a comprehensive and dynamic curation system (32, 38), perhaps supported by a consortium of federal (e.g., VA and CMS) and state partners who would directly benefit from measure curation. Such a consortium could focus on overall measure harmonization and specification, systematically grade evidence about measures, and extend essential work already in motion such as independent reviews conducted by the NQF.
Reducing and Harmonizing the Measure Landscape
In this study, we distilled 635 individual measure results into 376 unique measure constructs. For example, we found seven nearly identical versions of “follow up after hospitalization for mental illness,” which is fewer than the 25 versions of this measure construct reported by Patel et al. (7). However, our recommendations to independently evaluate measures, including overlap with existing measures, could reduce the overall number of measures. Furthermore, our recommendation to revive and fund a centralized curator of quality measures could facilitate awareness of existing measures so developers can explicitly and routinely assess measure harmonization and retirement issues.
Limitations
A limitation of this study was that we did not include measures related to certain public health issues and disorders (e.g., tobacco use and cognitive disorders) or measures used in pediatric care. Nonetheless, our study reflects core services commonly evaluated in national quality initiatives aimed at improving treatments for mental and substance use disorders. Our study also was not designed to systemically review the underlying evidence base across the hundreds of quality measures cataloged, nor was information on graded evidence available for most measures. Although we could not verify whether a sufficient evidence base existed for each measure, we cataloged whether each measure was endorsed by the NQF, an indicator that a measure’s properties, including underlying evidence, have been independently and critically reviewed. Another possible limitation of our study was that we relied on public inventories; we therefore may have missed internal measures used in organizations. Nonetheless, we reviewed >4,400 measures housed in public inventories. Moreover, this is the first study reporting on measures developed at the VA, rather than measures defined by other organizations for evaluating VA care (39). Including VA measures is important because they are used to evaluate the largest integrated health care system in the United States (12).
Conclusions
The measures on which we base evaluations of the quality of treatment for mental or substance use disorders represent a crucial aspect of performance management and, ultimately, the quality of care that patients receive. A vast number of measures exist to evaluate the quality of health care services for mental or substance use disorders, hundreds of which are in use for national quality improvement initiatives in the United States. Paring back and harmonizing the landscape of quality measures to meet stakeholder needs should be emphasized over expanding measurement options. We recommend a stakeholder-centered process with independent review and endorsement to accomplish these goals. Until that time, this study provides an updated and restructured analysis of the landscape of health care quality measures for managing mental and substance use disorders.
Acknowledgments
The authors thank Todd Wagner, Ph.D., for consultation and input into their presentation and discussion of results.
Supplementary Material
File (appi.ps.202000913.ds001.xlsx)
- Download
- 206.07 KB
References
1.
Quality Measures. Baltimore, Centers for Medicare and Medicaid Services, 2020. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures. Accessed Sep 14, 2020
2.
Hayford TB, Maeda JL: Issues and Challenges in Measuring and Improving the Quality of Health Care. Washington, DC, Congressional Budget Office, 2017. https://www.cbo.gov/system/files/115th-congress-2017-2018/workingpaper/53387-workingpaper.pdf
3.
Dunlap NE, Ballard DJ, Cherry RA, et al: Observations From the Field: Reporting Quality Metrics in Health Care. Washington, DC, National Academy of Medicine, 2016. https://nam.edu/wp-content/uploads/2016/07/Observations-from-the-Field-Reporting-Quality-Metrics-in-Health-Care.pdf
4.
Pincus HA, Li M, Scharf DM, et al: Prioritizing quality measure concepts at the interface of behavioral and physical healthcare. Int J Qual Health Care 2017; 29:557–563
5.
Iyer SP, Spaeth-Rublee B, Pincus HA: Challenges in the operationalization of mental health quality measures: an assessment of alternatives. Psychiatr Serv 2016; 67:1057–1059
6.
Kilbourne AM, Beck K, Spaeth-Rublee B, et al: Measuring and improving the quality of mental health care: a global perspective. World Psychiatry 2018; 17:30–38
7.
Patel MM, Brown JD, Croake S, et al: The current state of behavioral health quality measures: where are the gaps? Psychiatr Serv 2015; 66:865–871
8.
Pincus HA, Scholle SH, Spaeth-Rublee B, et al: Quality measures for mental health and substance use: gaps, opportunities, and challenges. Health Aff 2016; 35:1000–1008
9.
Goldman ML, Spaeth-Rublee B, Nowels AD, et al: Quality measures at the interface of behavioral health and primary care. Curr Psychiatry Rep 2016; 18:39
10.
Williams AR, Nunes EV, Bisaga A, et al: Development of a cascade of care for responding to the opioid epidemic. Am J Drug Alcohol Abuse 2019; 45:1–10
11.
About NGC and NQMC. Rockville, MD, Agency for Healthcare Research and Quality, 2018. https://www.ahrq.gov/gam/about/index.html. Accessed Sep 14, 2020
12.
National Academies of Sciences Engineering and Medicine: Evaluation of the Department of Veterans Affairs Mental Health Services. Washington, DC, National Academies Press, 2018
13.
Schmidt EM, Krahn DD, McGuire MH, et al: Using organizational and clinical performance data to increase the value of mental health care. Psychol Serv 2017; 14:13–22
14.
Lemke S, Boden MT, Kearney LK, et al: Measurement-based management of mental health quality and access in VHA: SAIL mental health domain. Psychol Serv 2017; 14:1–12
15.
Trafton JA, Greenberg G, Harris AH, et al: VHA mental health information system: applying health information technology to monitor and facilitate implementation of VHA Uniform Mental Health Services Handbook requirements. Med Care 2013; 51(suppl 1):S29–S36
16.
Measures Inventory Tool. Baltimore, Centers for Medicare and Medicaid Services, 2020. https://cmit.cms.gov/CMIT_public/ListMeasures. Accessed Sep 14, 2020
17.
Quality Positioning System. Washington, DC, National Quality Forum. http://www.qualityforum.org/Qps/QpsTool.aspx. Accessed Oct 10, 2021
18.
Healthcare Effectiveness Data and Information Set. Washington, DC, National Committee for Quality Assurance, 2019
19.
2018 National Healthcare Quality and Disparities Report. Rockville, MD, Agency for Healthcare Research and Quality, 2019. https://www.ahrq.gov/research/findings/nhqrdr/nhqdr18/index.html. Accessed Sept 14, 2020
20.
SAMHSA Strategic Plan FY2019–FY2023. Rockville, MD, Substance Abuse and Mental Health Services Administration, 2020. https://www.samhsa.gov/about-us/strategic-plan-fy2019-fy2023. Accessed Sep 14, 2020
21.
Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Arlington, VA, American Psychiatric Association, 2013
22.
Inpatient Psychiatric Facility Quality Reporting (IPFQR) Program. Baltimore, Centers for Medicare and Medicaid Services, 2018. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/IPFQR. Accessed Sep 14, 2020
23.
NQF Glossary of Terms. Washington, DC, National Quality Forum. https://www.qualityforum.org/Field_Guide/Definitions_and_Descriptions.aspx. Accessed Sep 14, 2020
24.
Institute of Medicine: Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academies Press, 2001
25.
Types of Health Care Quality Measures. Rockville, MD, Agency for Healthcare Research and Quality, 2015. https://www.ahrq.gov/talkingquality/measures/types.html. Accessed Sep 14, 2020
26.
Six Domains of Health Care Quality. Rockville, MD, Agency for Healthcare Research and Quality, 2018. https://www.ahrq.gov/talkingquality/measures/six-domains.html. Accessed Sep 14, 2020
27.
Report to Congress: Identification of Quality Measurement Priorities—Strategic Plan, Initiatives, and Activities. Baltimore, Centers for Medicare and Medicaid Services, 2019. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures/Downloads/CMS-RTC-Quality-Measurement-March-1-2019_508.pdf
28.
CMS Quality Strategy 2016. Baltimore, Centers for Medicare and Medicaid Services, 2016. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityInitiativesGenInfo/Downloads/CMS-Quality-Strategy.pdf
29.
NPDB Guidebook October 2018. Rockville, MD, US Department of Health and Human Services, 2018. https://www.npdb.hrsa.gov/resources/NPDBGuidebook.pdf
30.
Mental Health Quality Measures. Rockville, MD, Agency for Healthcare Research and Quality, 2017. https://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/chtoolbx/measures/measure9.html. Accessed Oct 19, 2020
31.
Choosing Quality Measures. Rockville, MD, Agency for Healthcare Research and Quality. 2020. https://www.ahrq.gov/patient-safety/quality-resources/tools/chtoolbx/choosing/index.html. Accessed Oct 9, 2021
32.
Institute of Medicine: Improving the Quality of Health Care for Mental and Substance-Use Conditions: Quality Chasm Series. Washington, DC, National Academies Press, 2006
33.
VA/DoD Clinical Practice Guidelines—Management of Posttraumatic Stress Disorder and Acute Stress Reaction 2017. Washington, DC, US Department of Veterans Affairs and Department of Defense, 2017. https://www.healthquality.va.gov/guidelines/mh/ptsd/index.asp. Accessed Sep 14, 2020
34.
Olfson M, Marcus SC: National trends in outpatient psychotherapy. Am J Psychiatry 2010; 167:1456–1463
35.
Fisher CE, Spaeth-Rublee B, Alan Pincus H: Developing mental health-care quality indicators: toward a common framework. Int J Qual Health Care 2013; 25:75–80
36.
National Quality Forum Measure Endorsement Review Process. Baltimore, Centers for Medicare and Medicaid Services. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/Downloads/NQF-Measure-Endorsement-Review-Process.pdf
37.
Wadhera RK, Figueroa JF, Joynt Maddox KE, et al: Quality measure development and associated spending by the Centers for Medicare & Medicaid Services. JAMA 2020; 323:1614–1616
38.
Institute of Medicine: Psychosocial Interventions for Mental and Substance Use Disorders: A Framework for Establishing Evidence-Based Standards. Edited by England MJ, Butler AS, Gonzalez ML. Washington, DC, National Academies Press, 2015.
39.
Watkins KE, Pincus HA, Smith B, et al: Veterans Health Administration Mental Health Program Evaluation: Capstone Report. Santa Monica, CA, RAND, 2011. https://www.rand.org/pubs/technical_reports/TR956.html. Accessed Aug 5, 2021
Information & Authors
Information
Published In
History
Received: 16 December 2020
Revision received: 17 August 2021
Revision received: 10 October 2021
Accepted: 5 November 2021
Published online: 17 February 2022
Published in print: August 01, 2022
Keywords
Authors
Competing Interests
The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the VA, the U.S. government, or Stanford University. The VHA had no role in the design of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
Funding Information
This study is based on work supported by the VHA Office of Research and Development, Health Services Research and Development (RVR 19-480). Dr. Harris is supported by a research career scientist award (RCS-14-232) from the VHA Office of Research and Development, Health Services Research and Development.The authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).