Evaluating the Implementation of Integrated Mental Health Care: A Systematic Review to Guide the Development of Quality Measures

This study was conducted from May 2014 to July 2016. Institutional review board approval was not required. We initially planned a scoping review to comprehensively inventory relevant measures of integrated care implementation (39,40). We developed a study protocol to guide the review process (PROSPERO registration number CRD42016038387) (41). After data extraction and critical appraisal of the measures and prior to data analysis, we modified our analytic strategy. In light of the extensive and heterogeneous scope of available data, we conducted a qualitative synthesis that forms the beginning of a logic model regarding integrated care program functioning.

We identified 3,761 literature sources, of which 197 met inclusion criteria. We achieved substantial agreement for abstract screening (kappa scores, 75%–98%) and for full-text screening (kappa scores, 90%–99%). Included literature was heavily weighted toward disease-specific studies, especially randomized controlled trials, published in the United States (Table 1). For sources that were literature reviews (N=24), we extracted data from the primary studies only, and for one source there was insufficient data. From the remaining 172 sources, we extracted 1,255 implementation and outcome measures, which we grouped into 148 unique measures. [A spreadsheet available in Online Supplement 2 presents the 148 unique measures and validated scales used to measure them, when available. Full details about the 1,255 found measures and bibliographic references are available from the authors.]

Literature frequently reported on the evaluation of individual clinical outcomes, such as depression symptom severity, health status, and level of functioning; cost-effectiveness, such as the incremental cost of reducing depression symptoms or increasing quality-adjusted life years; and evidence-based care processes, such as appropriateness and adequacy of pharmacotherapy for a specific illness condition. Therefore, a very strong emphasis on measuring clinical effectiveness was evident, along with some emphasis on efficiency (IoM domains); emphasis on process and outcome measures was roughly equal (Donabedian framework) (Tables 2 and 3).

Apart from patient-reported outcome measures of level of functioning and quality of life, which we categorized as effectiveness measures, client centeredness was represented through unidimensional scales of satisfaction with care, economic impact on clients (for example, direct and indirect costs of care, financial or housing status, and employability), and, rarely, clients’ engagement in their care or in program design or quality improvement.

We did not locate any measures of patient safety and found few measures of equitability or of accessibility or timeliness of care (for example, measures addressing vulnerable populations, stigma, and wait times). However, we identified several measures that did not fit with the IoM domains but rather reflected provider experience and the culture of health care delivery, such as health care provider buy-in and engagement in integrated care delivery, confidence in providing care within the integrated care model, and satisfaction with services. Most measures have been implemented, although some, predominantly from the gray literature, were recommended but not implemented.

With respect to critical appraisal, the found measures were highly variable in their quality. It was necessary to examine the original 1,255 found measures. Although they were aggregated into 148 unique measures, each of the unique measures may include examples that were defined and measured slightly differently, leading to variable quality scores for each unique measure. Furthermore, in this study we focused on measures that were actually implemented, which were more thoroughly described. For the 841 measures implemented, 30 (4%) were assigned critical appraisal scores of 1 to 3, 404 (48%) were assigned scores of 4 to 6, 385 (46%) were assigned scores of 7 to 9, and 22 (3%) were not assigned an overall score for quality because of the lack of reporting and missing data in the original citation. Generally, the highest-quality measures were those that evaluated individual outcomes of effectiveness by using validated measurement scales (for example, of psychiatric symptoms, physical symptoms, level of functioning, and quality of life). For other measures, common limitations included imprecise specification, lack of evidence of reliability or validity, lack of risk adjustment (for outcome measures), and high burden of measurement.

The thematic analysis of the quality measures yielded broad domains and specific dimensions of integrated care program functioning and impact that may be important to measure (Figure 1).

In this study, we comprehensively reviewed and analyzed existing measures by which to evaluate the implementation of integrated care programs. We identified key elements of integrated care and specific examples of quality measures that can be used to inform program development and evaluation, quality improvement, and the design of future research studies. Our thematic analysis and visualization invite users to consider the comprehensiveness and complementarity of measures that they may use.

This study had several strengths. First, we considered the broad range of integrated care models that have been implemented in primary care settings in English-speaking countries globally. Second, we included indicators regardless of implementation status and whether they were applied in trials or real-world settings. Third, we employed a rigorous search strategy to exhaustively locate the aforementioned types of indicators. Thus our study provides a unique and far-reaching summary of existing and proposed quality measures of integrated care.

In contrast with Goldman and colleagues’ (36) search of the National Quality Forum and National Quality Measures Clearinghouse databases, our approach captured measures that have been used or proposed to be used specifically to measure integrated care implementation or outcomes from a wide variety of sources, rather than measures that could be repurposed toward this goal. Similarly, our scope incorporated measures from client and family, provider, program, and population and system perspectives, expanding beyond those currently included in the AHRQ Atlas of Integrated Behavioral Health Care Quality Measures (34). However, the drawback of our comprehensive sweep was the inclusion of lower-quality measures (for example, measures that were ill defined or infeasible) that were identified during the critical appraisal process.

Several limitations of our study should also be considered. The concepts explored in our study—primary care, integrated care, and quality of care—are all somewhat complex; our choices in defining the boundaries and search terms of these concepts influenced the literature and measures we located. Our analysis was driven by found implementation and outcome measures in the literature and aimed to summarize these measures into accepted categories. Even though our analysis explored emerging themes and areas of overlap, our approach was not theory building and does not address causality. Other authors of systematic reviews have also identified a knowledge gap regarding the active ingredients of integrated care interventions, and different methods may be needed to delineate these (for example, realist reviews and innovative trial designs) (8). Researchers seeking to test hypotheses regarding the necessary structures and processes to achieve intended outcomes of integrated care can refer to the visualization (Figure 1) and database [Online Supplement 2] to inform theory development and investigation.

The body of literature we reviewed yielded a multitude of measures and captured diverse perspectives on integrated care program implementation and outcomes, and at least some measures were of high quality. Clinical and administrative leaders could use measures from our database to understand how their integrated care program is functioning and to monitor the results of quality improvement and program development efforts. For a comprehensive program evaluation, we recommend using multiple measures capturing diverse perspectives (as depicted in Figure 1) and several different IoM and Donabedian domains of quality. For a focused quality improvement project, a small number of measures could suffice, with one to two process measures, a balancing measure, and, importantly, an outcome measure. However, although certain types of indicators are frequently represented in the literature, frequency should not be interpreted as confirmation that they are the optimal or only targets for measurement. Indeed, some measures may be prevalent because they are easier to implement (looking under the proverbial lamppost) or because of funders’ emphasis on cost control. For example, many measures are disease specific, which may not translate well into real-world implementation of integrated care programs targeting multiple (and often comorbid) conditions. Furthermore, caution is warranted regarding the potential unintended consequences of focusing on the domains of effectiveness and efficiency at the expense of other domains. Aspects of quality that are underrepresented in the existing literature (for example, equity, accessibility and timeliness, and client centeredness of care) are vital aims of integrated care; some successes have been demonstrated, and further evaluation is merited (50–54).

Some components of integrated care functioning that are key to implementation—such as scaling up and sustainability in real-world settings—may be overlooked. Many of the indicators we found were implemented in randomized controlled trials of efficacy, whereas local contextual factors and specific unmeasured processes embedded within trials are often underrecognized and underreported (21–23,25). For example, it is possible that intricate processes of communication, collaboration, and coordination that make integrated care work and that require role changes and different types of working relationships may be missed. Our proposed new quality domain—the “culture of health care”—highlights and begins to address this gap. This concept is also reflected in other overarching quality frameworks, such as the quadruple aim, which extends the Institute for Healthcare Improvement’s triple aim of improving patient experience and population health and reducing health care costs by incorporating provider experience (55). Notably, such factors may not only be ignored but may in fact be subverted when performance measures are used for external accountability and funding, as other authors have proposed for integrated care (32,36,56).

These concerns speak to the importance of exploring the experiences and perspectives of clinicians and clients in the field. Qualitative studies may yield further insights into the aspects of care that matter to clients, as well as the indicators relevant to implementation success or failure that may not be measured at present. Consultation with experts and key stakeholders could also help identify areas for further indicator development, as well as prioritize from among the many available avenues for measurement those that are postulated to be most influential on overall quality of care. As next steps, we plan to interview health care providers for their perspectives on implementing integrated care in real-world contexts and clients for their perspectives on experience and outcome measures that matter to them. Finally, we will engage diverse experts and stakeholders in forming a consensus on a quality framework for integrated care to guide the future development of relevant, actionable, high-quality measures.

Meaningful and valid quality measurement has been identified as key to promoting uptake of robustly evidenced models that improve outcomes and to encouraging more cost-effective care (32). To realize the potential for quality measures to guide and effect improvements in integrated care implementation, we will need to broaden the scope of measures to incorporate key domains of quality that largely remain unaddressed (for example, equity, access, timeliness, and safety); microprocesses of care that influence effectiveness, sustainability, and transferability of integrated care models; and client perspectives on important domains and dimensions of quality. We concur with other authors regarding the importance of measuring care processes that affect outcomes (highlighting the need to test these hypothesized relationships), improving information systems, developing health care provider capacity, and engaging clients in health care system design and improvement (33). Finally, measures will need to be developed and tested for feasibility, applicability, and ability to drive improvements in real-world settings.

The authors are grateful to Sydney Dy, M.D., M.Sc., and Elizabeth Lin, Ph.D., for feedback on drafts of the article. The authors also thank Anjana Aery for contributions to the literature searching and data collection.