Full access

Articles

Published Online: 1 September 2014

A Literature Review of Learning Collaboratives in Mental Health Care: Used but Untested

Erum Nadeem, Ph.D., S. Serene Olin, Ph.D., Laura Campbell Hill, B.A., Kimberly Eaton Hoagwood, Ph.D., and Sarah McCue Horwitz, Ph.D.Authors Info & Affiliations

Publication: Psychiatric Services

Volume 65, Number 9

https://doi.org/10.1176/appi.ps.201300229

PDF/EPUB

Abstract

In recent years, the mental health field has made learning collaboratives the centerpiece of efforts to implement training in evidence-based practices. Lasting an average of 14 months, these interventions typically begin with in-person training for two days and can cost more than $100,000 per site. In this literature review of 16 peer-reviewed studies, the authors found minimal evidence of the effectiveness of LCs in mental health settings. They called for more rigorous research to evaluate the utility of these costly endeavors.

Abstract

Objective

Policy makers have increasingly turned to learning collaboratives (LCs) as a strategy for improving usual care through the dissemination of evidence-based practices. The purpose of this review was to characterize the state of the evidence for use of LCs in mental health care.

Methods

A systematic search of major academic databases for peer-reviewed articles on LCs in mental health care generated 421 unique articles across a range of disciplines; 28 mental health articles were selected for full-text review, and 20 articles representing 16 distinct studies met criteria for final inclusion. Articles were coded to identify the LC components reported, the focus of the research, and key findings.

Results

Most of the articles included assessments of provider- or patient-level variables at baseline and post-LC. Only one study included a comparison condition. LC targets ranged widely, from use of a depression screening tool to implementation of evidence-based treatments. Fourteen crosscutting LC components (for example, in-person learning sessions, phone meetings, data reporting, leadership involvement, and training in quality improvement methods) were identified. The LCs reviewed reported including, on average, seven components, most commonly in-person learning sessions, plan-do-study-act cycles, multidisciplinary quality improvement teams, and data collection for quality improvement.

Conclusions

LCs are being used widely in mental health care, although there is minimal evidence of their effectiveness and unclear reporting in regard to specific components. Rigorous observational and controlled research studies on the impact of LCs on targeted provider- and patient-level outcomes are greatly needed.

Recently, a tremendous emphasis has been placed on the integration of evidence-based practices into routine mental health care. Substantial budget cuts to mental health funding at the state and national levels have forced policy makers to seek out efficient and effective ways to scale up training in evidence-based practices (1). States, counties, and national organizations have turned to learning collaboratives (LCs) as a method for large-scale training with ongoing support.

This collaborative approach has clearly become a priority in the field. The Substance Abuse and Mental Health Services Administration recently issued a call for applications for State Adolescent Treatment Enhancement and Dissemination grants totaling $30 million over three years to help states develop “learning laboratories” focused on shared provider experiences during the implementation of new evidence-based practices (2). Similarly, through the National Council for Community Behavioral Healthcare, 35 states are now using LCs to change health care provider practices (personal communication, Salerno A, July 2012). LCs represent a significant investment in the field as a potentially viable approach to large-scale implementation and dissemination of new treatment practices. However, there has been little research on the effectiveness of LCs for evidence-based practices in mental health care.

LCs as they are implemented in mental health care are adapted from quality improvement collaborative (QIC) models used in health care. One of the most widely cited and adapted QIC models is the Breakthrough Series (BTS) collaboratives of the Institute for Healthcare Improvement (IHI) (3–9). The quality improvement processes at the core of the IHI and other approaches are rooted in industrial improvement practices and the work of W. Edwards Deming and Joseph Juran, statisticians who advocated for process improvement driven by ongoing data collection and analysis and an assumption of workers’ interest in learning and improvement (10–12).

Although some evidence exists for the effectiveness of QICs in health care, there is a need for rigorous research in this area. A systematic literature review by Schouten and colleagues (13) identified nine controlled studies of health care QICs and concluded that the QICs showed promise in changing provider practices. However, the authors found less evidence in support of an impact on patient-level outcomes. Although the review included two randomized controlled trials (RCTs), a majority of the studies used matched control sites or compared administrative data from similar sites in a larger provider network.

Building on these findings, a more recent review included 24 articles, with the goal of updating the original literature review and developing a deeper understanding of the core components of QICs as they are reported in the literature (14). This review included additional RCTs (five distinct studies); however, as with the earlier review (13), a vast majority of studies used matched controls. Of the 14 crosscutting components identified as common ingredients, the QICs reported including, on average, six or seven components—most commonly, in-person learning sessions, plan-do-study-act (PDSA) cycles, multidisciplinary quality improvement teams, and data collection for quality improvement. As in the earlier review (13), outcome data suggested that the greatest impact of the QICs was on provider-level process-of-care variables; patient-level findings were less robust. Because of the imprecise reporting on specific components, it was not possible to link any specific components with improved care.

Of note, neither of these systematic reviews included collaboratives focused on mental health issues because when they were undertaken there had been no controlled studies targeting mental health care. LCs have been applied in mental health care to a wide range of practices, including the process of care (for example, engagement in services, care integration, and use of a screening tool) (15–18) and implementation of complex evidence-based practices (7,19). The focus on evidence-based practices is notable given the complexity of the patient outcomes and the substantial skill development required of providers.

This systematic literature review focused on peer-reviewed studies of mental health LCs that included any patient or provider pre-to-post outcome data. Given the differences between mental health and general health care settings in terms of their structure, types of interventions and patient issues addressed, and data systems available, there is a critical need for a better understanding of how LCs are implemented in mental health care. The primary goal of this review was to identify the components of LCs as reported in mental health studies and to characterize the existing data on LCs (for example, patient-level data, reports of changed provider practices, and analyses of feasibility or acceptability in real-world care settings).

Methods

This literature search on LCs focused on individual empirical articles published from January 1995 to October 2013. The database search included Ovid MEDLINE, ProQuest, PsycINFO, and PubMed. Search terms included “learning collaborative,” “quality improvement collaborative,” “Breakthrough Series,” and “NIATx.” These terms were refined after several preliminary searches and are similar to those used in earlier reviews (13,14). NIATx (Network for the Improvement of Addiction Treatment) was included in order to capture the NIATx process improvement approach used in the substance abuse literature, which draws on conceptual models similar to the predominant approach to collaboratives, specifically the IHI’s BTS (20).

Articles that met inclusion criteria were peer reviewed, written in English, and included a pre- and postintervention comparison of the impact of an LC. To define LCs in mental health care, we searched the theoretical literature on QICs (3–5,9,21–24) and reviewed the definition used by Schouten and colleagues (13). We then conducted informational interviews with a subset of LC purveyors to elicit more detail. This review defined LCs as organized, structured group learning initiatives in which organizers took the following steps: convened multidisciplinary teams representative of different levels of the organization; focused on improving specific provider practices or patient outcomes; included training from experts in a particular practice or in quality improvement methods; included a model for improvement with measurable targets, data collection, and feedback; engaged multidisciplinary teams in active improvement processes wherein they implemented “small tests of change” or engaged in PDSA activities; and employed structured activities and opportunities for learning and cross-site communication (for example, in-person learning sessions, phone calls, and e-mail Listservs) (3,5–7,9,25,26). We assessed the ways in which the 14 components identified by Nadeem and colleagues (14), including in-person learning sessions, phone meetings, data reporting, feedback, training in quality improvement methods, and use of process improvement methods, were reported in these studies.

Two authors (EN and LCH) reviewed all abstracts generated by the initial search to select articles that merited a full-text review. These two authors also reviewed each article retrieved to determine whether it met final inclusion criteria. In the event of a discrepancy or if inclusion was unclear, the two authors conferred with the other authors to make a final determination. Once article selection was finalized, each article was coded by using a standardized table to summarize study details (for example, targets for improvement, study design, setting, study sample, and LC components). A primary coder was assigned to each article, and a secondary coder reviewed the primary coder’s work. Disagreements were resolved by consensus.

The initial search generated 421 unique articles across several disciplines (primarily mental health, education, and health care). From a review of the 421 abstracts, 52 were determined to be related to mental health or substance abuse, 28 of which met criteria for full-text article review (that is, they appeared to be focused on learning collaboratives). Articles were excluded after the full-text review if they did not report any pre-post LC quantitative data. After a review of those articles and their references, 20 articles were selected for final inclusion (7,15–18,27–41). [A figure illustrating article selection is included in an online data supplement to this article.]

Results

The 20 articles selected for inclusion encompass 16 distinct studies. Table 1 provides a summary of the study type, LC model, and LC components reported in each study. Table 2 provides definitions of the study characteristics and LC components tracked in this review. The LC features were categorized into components, quality improvement processes, and organizational involvement. LC components refer to LC features that constituted the structure of the model. Quality improvement processes include available details about PDSAs and other quality improvement activities. The organizational involvement section includes indicators of the ways in which the LC penetrated various levels of the organization.

Table 1 Twenty articles representing 16 studies of learning collaboratives (LCs) included in the review^a

Article	Target	Model^b	Sample	Length	Expert panel	ODC	In-person learning sessions	PDSAs	Multidisciplinary QITs	QIT phone calls	E-mail or Web support	New QI data	QI data review	External support with data review	Involved leaders	External training for non-QIT staff	QIT training for non-QIT staff
				LC component^c								Quality improvement^d			Organizational penetration^e
	Article information				Prework
Cavaleri et al., 2006 (15)^f	Mental health service use and evidence-based engagement strategies	BTS	12 mental health agencies	9 months	Unclear	Yes	Yes	Unclear	Yes	Yes	No	Unclear	Unclear	No	Yes	Unclear	No
Cavaleri et al., 2007 (27)^f	Mental health service use and evidence-based engagement strategies	BTS	9 mental health agencies (9 of the 12 mental health agencies from Cavaleri et al. [15])	9 months	Unclear	Yes	Yes	Unclear	Yes	Yes	No	Unclear	Unclear	No	Yes	Unclear	No
Cavaleri et al., 2010 (16)	Mental health service use and evidence-based engagement strategies	BTS	5 mental health agencies (4 experimental, 1 did not implement any engagement strategies)	9 months	Unclear	Yes	Yes	Unclear	Yes	Yes	No	Unclear	Unclear	No	Yes	No	Yes
Duffy et al., 2008 (32)	Use of a depression assessment in psychiatric practices	BTS, CCM	19 psychiatric practices (2 practices dropped out without completing data collection)	12 months	No	Yes	Yes	Yes	Yes	Yes	No	Yes	No	No	Unclear	No	Yes
Ebert et al., 2012 (7)	Use of Trauma-Focused Cognitive Behavioral Therapy in community practice settings	BTS	11 mental health agencies (2 agencies did not complete)	18 months	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	No	Yes	No	No
Epstein et al., 2008 (28)^g	Adherence to guidelines for evidence-based assessment and treatment of attention-deficit hyperactivity disorder (ADHD)	CCM	19 practices: 65 pediatricians and 19 family physicians	Unclear	Unclear	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	No
Epstein et al., 2010 (29)^g	Adherence to guidelines for evidence-based assessment and treatment of ADHD	CCM	47 practices: 142 pediatricians and11 family physicians	Unclear	Unclear	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	No
Epstein et al., 2010 (30)^g	Adherence to guidelines for evidence-based assessment and treatment of ADHD	CCM	31 pediatric practices: 123 pediatricians (data from family physicians were excluded)	Unclear	Unclear	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	No
Gustafson et al., 2013 (34)	Time to treatment, client retention, and new patient recruitment in addiction centers	NIATx	201 addiction treatment centers	18 months	Unclear	No	Yes	Yes	Unclear	Yes	Yes	Yes	Yes	Yes	Yes	No	No
Haine-Schlagel et al., 2013 (37)	Attendance engagement in community-based early childhood intervention programs	BTS	4 developmental services programs within a children’s hospital: 29 providers (2 providers did not complete, and 1 was added after initiation)	9 months	Yes	No	Yes	Yes	Yes	Yes	Yes	No	No	No	Yes	No	No
Katzelnick et al., 2005 (17)	Implementation of the chronic care model for depression treatment in primary health care	BTS, CCM	20 health care organizations (3 teams did not complete)	13 months	No	No	Yes	Yes	Unclear	Yes	Yes	Yes	Unclear	Yes	Yes	No	Yes
Hoffman et al., 2008 (36)^h	Time to treatment and client retention in outpatient, intensive outpatient, or residential addiction treatment units	NIATx	Second cohort: 11 addiction treatment agencies (10 outpatient, 4 intensive outpatient units)	18 months	Yes	Yes	Yes	Yes	Unclear	Yes	Yes	Yes	Yes	Yes	Yes	No	No
McCarty et al., 2007 (35)^h	Time to treatment and client retention in outpatient, intensive outpatient, or residential addiction treatment units	NIATx	First cohort: 13 addiction treatment agencies (7 outpatient, 4 intensive outpatient, 4 residential units)	18 months	Yes	Yes	Yes	Yes	Unclear	Yes	Yes	Yes	Yes	Yes	Yes	No	No
Meredith et al., 2006 (31)	Depression treatment in primary care	BTS, CCM	17 mental health agencies	13 months	No	No	Yes	Yes	Yes	Yes	No	Unclear	No	No	No	No	Yes
Roosa et al., 2011 (19)	Client retention in chemical dependency treatment and client access to mental health services	NIATx	Chemical dependency collaborative: 4 treatment agencies (1 did not complete); mental health collaborative: 6 treatment agencies	27 months	No	No	Yes	Yes	Yes	No	No	No	No	No	Yes	Unclear	No
Rutkowski et al., 2010 (38)	Time to treatment, no-show rates, admissions, or continuation in treatment for addiction treatment services	NIATx	Phase I: 6 treatment agencies, 7 change teams; phase II: 8 treatment agencies, 13 change teams	Phase I, 11 months; phase II, 12 months	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	No	No
Stephan et al., 2011 (39)	Mental health service quality and collaborative care in school-based mental health centers	NA SBHC	19 school-based health centers	15 months	Yes	Yes	Yes	No	Yes	Yes	No	Yes	Yes	Yes	Unclear	No	Yes
Strating et al., 2012 (41)	Four distinct collaboratives focused on social psychiatric care, recovery-oriented care, social participation, and somatic comorbidity of psychiatric clients	BTS	94 distinct teams of mental health care providers; collaborative 1: 25 teams; collaborative 2: 25 teams; collaborative 3: 26 teams; collaborative 4: 18 teams	12 months	No	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	No	No	No
Vannoy et al., 2011 (18)	Integration of services between community health centers (CHCs) and community mental health centers (CMHCs) to improve treatment of depression and bipolar disorder in CHCs and improve care of patients at risk of metabolic syndrome in CMHCs	BTS	15 CHC and CMHC pairs (1 pair dropped out because of staff turnover)	12 months; 3 cohorts	No	Yes	Yes	Yes	Unclear	Yes	No	Yes	Yes	Yes	No	No	No
Versteeg et al., 2012 (33)	Implementation of multidisciplinary practice guidelines in mental health care organizations (specific domains: anxiety disorders, dual diagnosis, and schizophrenia)	BTS	19 mental health care organizations; 26 distinct LC teams	12 months	Unclear	No	Yes	No	Yes	No	Yes	Yes	Yes	Yes	Unclear	No	No

Articles are in alphabetical order by author name and grouped by study.

BTS, Breakthrough Series; CCM, chronic care model; NIATx, Network for the Improvement of Addiction Treatment; NASBHC, National Assembly on School-Based Health Care

ODC, organizations required to demonstrate commitment; PDSAs, plan-do-study-act cycles; QITs, quality improvement teams

New QI data, sites collected new data for quality improvement purposes; QI data review, sites reviewed quality improvement data and used feedback

Involved leaders, involvement or outreach (or both) to organizational leadership

^f,g,h Articles that share a footnote are based on data from a single study.

Table 2 Definitions of domains and components of learning collaboratives (LCs) in the studies reviewed^a

Domain	Definition
Article information^b
Target for improvement	Focus area for the LC
Model	LC alignment with existing collaborative models
Study sample	Focus population
LC component^c
Length of collaborative	Standard LC length
Prework: convened expert panel	The Institute for Healthcare Improvement’s Breakthrough Series (BTS) model calls for a planning group that identifies targets for improvement change and plans the collaborative.
Prework: organizations required to demonstrate commitment	The BTS model recommends requiring formal commitments, application criteria, or “readiness” activities for LC sites.
In-person learning sessions	Teams are traditionally trained in clinical approaches and quality improvement (QI) approaches during in-person sessions.
Plan-do-study-act (PDSA) cycles	PDSA cycles are a key component of the rapid cycle approach to change recommended in QI collaborative models.
Multidisciplinary QI team	LCs typically involve staff members at various levels of the organization.
QI team calls	Group phone calls among QI team members or between members in other participating organizations are used as an approach for providing ongoing support.
E-mail or Web support	E-mail, Listservs, or other forms of Web support used as an approach for providing ongoing support
QI processes^d
Sites collected new data for QI	During the LC, did sites collect new data for QI purposes?
Sites reviewed data and used feedback	Did the LC sites review new data, receive feedback, and adjust their practices according to findings?
External support for data synthesis and feedback	Did LC faculty or other experts provide support with data synthesis and feedback?
Organizational involvement^e
Leadership involvement and outreach	Did members of the LC involve or otherwise reach out to organization leadership?
Training for non–QI team staff members by experts	Did LC faculty or other experts provide training for staff members who were not part of the QI Team?
Training for non–QI team staff members by the QI team	After the LC, did newly trained QI team members provide training for staff members who were not part of the QI team?

Table adapted from Nadeem et al. (14) with permission

Basic study details highlighted by the published article

LC components were compiled through the literature review and explicitly referenced by study authors.

Beyond the basic components of the LC, which QI techniques were included in the LC?

In theory, LCs enable an organization to enact change at multiple levels within their organizational structure. Did the LC take steps to train or otherwise involve members of the organization who were not directly included in the collaborative?

Description of LC components

Ten of the 16 distinct studies were explicitly based on the IHI BTS model, three of which also noted using the chronic care model, a model originally used as part of a joint effort by the IHI and the Robert Wood Johnson Foundation (42). One additional study cited the chronic care model without the BTS model, four studies reported using the NIATx model for process improvement (43), and one study reported using the National Assembly on School-Based Health Care’s (NASBHC) QIC model, which is based on nationally recognized models for quality improvement (39). On average, each of the 16 studies reported implementing seven LC components. The most commonly reported components included in-person learning sessions (16 of 16), multidisciplinary quality improvement teams (12 of 16), PDSAs (12 of 16), and quality improvement team calls (12 of 16). In addition, 11 of the 16 studies reported doing some leadership outreach or engagement. Across articles, there was great variability in the level of detail provided in descriptions of the components of each LC.

Overall LC structure.

The LCs lasted an average of 14 months (range nine to 27 months), with a modal length of 12 months. LCs typically began with an in-person learning session; LC faculty hosted the sessions, and multidisciplinary quality improvement teams attended. Follow-up occurred via additional in-person learning sessions, regular phone meetings for the quality improvement teams, and e-mail or Web-based support. Sites conducted quality improvement projects between quality improvement team calls and in-person learning sessions. All in-person learning sessions and most phone meetings involved multiple sites.

Content of in-person learning sessions.

All studies reported including in-person learning sessions throughout the course of the LC. The most common number of sessions was three (range one to four). In-person sessions were typically two days long, ranging from a half-day to three day-long sessions. One of the studies was an RCT; the four conditions compared were interest circle calls (group teleconference calls), individual site coaching (that is, in-person, telephone, and e-mail support provided to each site), in-person learning sessions, and a combination of all three (34). All studies appear to have included in their sessions some didactic training in a particular care process or specific practice. One study, which focused on care for attention-deficit hyperactivity disorder in primary care clinics, used a combination of shorter in-person sessions (four 90-minute sessions focused on didactic lectures and quality improvement methods) and office visits (28–30).

In the National Child Traumatic Stress Network model, all LC participants had already received standard training by the treatment developer in trauma-focused cognitive behavioral therapy before the LC began (7). Participants in a NIATx collaborative took part in a two-day workshop on an evidence-based practice, Seeking Safety (19), in addition to LC activities. Similarly, participants in the NASBHC collaborative learned core components from evidence-based treatment elements for depression, anxiety, disruptive behavior disorders, and substance abuse, along with selected manualized interventions (39). Participants in an LC on engagement strategies received training for agency staff in addition to the standard learning sessions (37).

All of the studies that included descriptions of the in-person sessions also reported that the LC faculty provided training in quality improvement techniques, such as engaging in PDSA cycles or improvement projects. Very few details were provided on the techniques that were taught. In some studies, the LC purveyors had already identified potential areas for improvements that sites should consider for their quality improvement projects (for example, domains in the chronic care model, system improvements, and known implementation barriers) (7,18,31,33). In addition to didactic training related to practices and quality improvement methods, four of the studies reported that individual sites presented information to other participating quality improvement teams during the in-person sessions (7,15,27,37). Few specific details were included about the structure of these cross-site collaborative efforts. Some studies reported having individual site presentations, breakout sessions among “affinity groups,” or the use of “storyboards” (7).

PDSAs.

Twelve studies reported use of PDSAs between in-person sessions during “action periods” (7,17–19,28–31,34–38,41). However, it was largely unclear what occurred during the PDSA cycles, how they were used, or how the ongoing data collection informed the quality improvement process. However, a few studies provided some detail about use of quality improvement methods. In those LCs, the faculty set forth possible improvement areas from which a site could develop its PDSAs or provided hands-on coaching and support (7,17,18,29–31,34,37,38). One study did not include PDSAs but instead provided teams with a template to develop “work plans” to facilitate the integration of mental health and primary care in school-based health centers (39).

Quality improvement team calls.

Twelve studies reported that there were calls between in-person sessions for the quality improvement teams (7,15–18,27,31–39). The calls were typically held monthly with the goal of allowing sites to share progress and solve problems together. Few details were provided on the content or structure of the calls. Two studies reported holding “all collaborative” calls to facilitate sharing and problem solving (7,37). Others described “affinity group” calls targeted toward clinical supervisors, change leaders, or executive leadership or calls focused on specific clinical issues and other special topics (7,38). Studies using the NIATx model also described holding individual site coaching calls focused on the use of process improvement methods (34,38).

E-mail or Web support.

Six studies reported e-mail or Web-based support for the LC participants (7,17,33–37). Articles did not provide information about the extent to which LC participants used e-mail Listservs or Web-based support to communicate with other LC participants or LC faculty.

Quality improvement processes.

Eleven studies reported some type of ongoing data collection for the purposes of the LC (for example, performance indicators and ongoing reporting on target outcomes) (7,17,18,28–30,32–36,38,39,41); eight reported that the LC faculty provided sites with data-based feedback (18,28–30,33–36,38,39,41). Nine studies reported external support with data collection and feedback (17,18,28–30,33–36,38,39,41). With a few exceptions (7,30,33,34,38), most articles provided very little information about the data collected, how data were used, or how the data informed quality improvement activities.

Organizational involvement.

Ten studies reported that the organization’s leadership was involved in the LC (7,15–17,19,27–30,34–38). However, it was unclear whether the organizational leadership was included as a part of the quality improvement team or was engaged through other outreach efforts. We also examined indicators of the LCs’ penetration into the broader organization by tracking the training provided to staff who were not members of the quality improvement team, either by LC faculty or by local quality improvement team members themselves. No studies reported providing expert training (conducted by LC faculty or treatment developers) for frontline staff members who were not already on the quality improvement team. Five studies reported that quality improvement team members trained additional staff in the organization (16,17,31,32,39).

Pre-LC activities.

Finally, we tracked “prework” activities, which we defined as planning activities delineated in the original IHI BTS model (8,9). Only five studies reported that the LC used an expert panel during this prework phase—that is, a planning group that identifies targets for improvement and that plans the LC (7,35–39). Eight studies reported requiring formal commitments, application criteria, or readiness activities before the start of the LC (7,15,16,18,27,32,35,36,38,39).

Study goals and findings

Study goals.

The primary intent of 19 of the 20 articles was either to explore general feasibility and acceptability of the LC model or to examine pre-post LC changes at the patient and provider level. The only RCT was designed to test various components of the LC to determine which were most related to change (34). In this study, sites were randomly assigned to receive interest circle calls (group teleconference), individual site coaching, in-person learning sessions, or a combination of all three components with the intent of examining which components were related to study outcomes. The study’s use of individual site coaching is somewhat unique. Individual site coaching was described in some studies of the NIATx model (35,36), but most articles did not specify the use of coaching.

Across the studies, ten examined provider-level variables (7,17–19,30–32,37,39,41), 11 examined patient-level variables (15–17,19,28,29,34–36,38,41), nine examined acceptability of the LC model to providers (7,17–19,31,32,37,39,41), and eight examined sustainability of the changes achieved (7,19,27,30,31,34,36,39). One study examined the relation between LC components and study outcomes in an RCT (34). Three studies examined how elements of the LC process may have contributed to the findings from the LC by exploring issues such as the relation between reported barriers and facilitators (31), social networks (31), and theoretically or empirically derived attitudinal and contextual factors (for example, team effectiveness) (33,40) and changes in outcomes. In addition, two articles provided cost estimates for participation in the collaborative (31,34) (Table 3).

Table 3 Variables examined by 20 articles on learning collaboratives (LCs) included in the review^a

Article	Provider-level variables	Patient-level variables	Acceptability of the LC model to providers	Sustainability of changes	Relationship between LC components and LC outcomes	Relationship between aspects of implementation and LC outcomes	Cost estimates
Cavaleri et al., 2006 (15)^b		✓
Cavaleri et al., 2007 (27)^b				✓
Cavaleri et al., 2010 (16)		✓
Duffy et al., 2008 (32)	✓		✓
Ebert et al., 2012 (7)	✓		✓	✓
Epstein et al., 2008 (28)^c		✓
Epstein et al., 2010 (29)^c		✓
Epstein et al., 2010 (30)^c	✓			✓
Gustafson et al., 2013 (34)		✓		✓	✓		✓
Haine-Schlagel et al., 2013 (37)	✓		✓
Katzelnick et al., 2005 (17)	✓	✓	✓
Hoffman et al., 2008 (36)^d		✓		✓
McCarty et al., 2007 (35)^d		✓
Meredith et al., 2006 (31)	✓		✓	✓		✓	✓
Roosa et al., 2011 (19)	✓	✓	✓	✓
Rutkowski et al., 2010 (38)		✓
Stephan et al., 2011 (39)	✓		✓	✓
Strating et al., 2012 (41)	✓	✓	✓			✓
Vannoy et al., 2011 (18)	✓		✓
Versteeg et al., 2012 (33)						✓

Articles are organized by author name and grouped by study.

^b,c,d Articles that share a footnote are based on data from a single study.

Study findings.

There was wide variability in study designs and methods, quality of the methodology, and methodological details provided in the articles. Moreover, with the exception of one RCT (34), the strength of the outcomes was difficult to judge across studies because of the lack of control groups and the variability in the reporting of the LC elements. Therefore, we were unable to draw conclusions about the overall effectiveness of the LC within the mental health context.

However, the study by Gustafson and colleagues (34) suggested that certain LC elements may be more potent in predicting patient outcomes. Specifically, the authors found that waiting times declined for clinics in the individual site coaching, in-person learning sessions, and a combination of three LC components (in-person learning sessions, individual site coaching, and group calls). They also found that the number of new patients increased for the combination and coaching-only groups and that interest circle group teleconferences had no impact on outcomes. Although individual site coaching and the combination intervention were considered to be similarly effective, individual site coaching was more cost-effective over the 18-month study period ($2,878 per clinic versus $7,930) (34).

Of the 19 other articles that were not RCTs, most reported positive findings with respect to patient, provider, or sustainability variables. Each of the ten articles that reported on provider-level variables reported positive trends from pre- to post-LC, suggesting improvements in areas such as process of care and uptake of new practices (7,17–19,30–32,37,39,41). Similarly, although there were some mixed findings, each of the 11 articles that reported on patient-level variables reported positive pre- to post-LC changes in areas such as symptoms and engagement in services (15–17,19,28,29,34–36,38,41). Six of the eight articles that reported on sustainability reported sustained use of new practices or procedures after the conclusion of the LC (7,27,30,31,36,39). In addition, the LC model was reported to be feasible and acceptable to providers in each of the nine articles that assessed these variables (7,17–19,31,32,37,39,41).

Discussion

The use of LCs in the mental health context is an important area for research as policy makers seek to scale up evidence-based practices and improve the quality of care. LCs are being widely used as an attractive alternative to traditional developer training models because they hold promise for achieving sustained change in a way that typical treatment developer trainings may not (7,44–46). LCs can help sites build local capacity and address organization- and provider-level implementation barriers (44,47,48). They have the potential to foster local ownership of the implementation process, promote transparency and accountability, create a culture of continuous learning, provide an infrastructure for addressing barriers, and cultivate support networks (7,44).

The major challenge for the mental health field is the lack of rigorous studies of LCs. In our previous review, we found 20 studies of LCs in other areas of health care that used comparison groups (14), but only one study in mental health care was an RCT (34). In the review reported here, we identified 20 articles that reported data on LC outcomes. Although we can be encouraged by the positive trends reported in these studies with respect to provider, patient, and sustainability outcomes, the findings must be interpreted with caution given the lack of comparison data. In addition, because of the variability in methods and rigor used in these studies, it was not possible to come to any broad conclusions about the effects of LCs on provider- or patient-level outcomes.

It is critical that future research on LCs include more studies with comparison conditions—ideally with randomized designs that can examine the impact of different implementation strategies. There are a number of quality improvement approaches to implementation of new practices that could be tested against LCs. Evidence has been found for several approaches in terms of improving the quality of care: audit and feedback methods from health care (49); individual site–focused quality improvement initiatives that involve training of local quality improvement teams, leadership support, coaching, and audit and feedback (50,51); and the availability, responsiveness, and continuity model, an organizational-level quality improvement intervention (52). In addition, a review of Six Sigma and Lean continuous improvement approaches borrowed from industry and applied in health care suggest that these are promising strategies that could be further tested (53). Of particular importance are studies such as the one conducted by Gustafson and colleagues (34) that can identify which structural and theoretical components of LCs contribute to favorable outcomes.

Recent studies provide insights into active components that could be directly tested. These include cross-site and local learning activities (for example, staff education, PDSAs, and team effectiveness) (31,41,48,54,55), local leadership support, sites’ ability to address common implementation barriers, expert support, ongoing data collection, and the visibility of local changes achieved through quality improvement methods (3,33,48,56–59). In addition, there is a great need to continue to examine the costs associated with LCs and the incremental cost-benefit of using this approach, compared with traditional developer trainings and other quality improvement methods. This type of information is critical for decision makers because LCs can be costly. One study of an LC for depression care reported that the average cost of participation was more than $100,000 per site (31). Another study suggested that the added cost of in-person learning sessions may not bring much incremental cost-benefit with respect to patient outcomes, compared with individual site coaching (34).

With respect to the reporting of LC components, we found patterns similar to those found in previous research. Prior reviews have highlighted the variation in implementation of the LC model and inconsistent reporting of components (4,13,14,25). Across studies, the LCs in this review had a similar structure. However, insufficient detail was provided with respect to the presence of LC components and how they were implemented in most studies. Moreover, because the original QIC models in health care were based on management theory (10–12), the lack of specificity on how process improvement was conducted, how quality improvement data were collected, and how data were used is striking. It is essential to carefully describe how quality improvement methods are being used in mental health care because previous studies have suggested that LC participants perceive instruction in quality improvement methods to be useful (31,48,59) and because the innovations implemented in mental health are often complex evidence-based treatments that may require adaptations from the original QIC models in health care. This review provides one potential template for the reporting of specific LC components, each of which should be reported in sufficient detail that others could replicate the activities and processes (that is, “dosage” provided, engagement of participants, details on how quality improvement was taught, how data were used, and how teams and leadership were engaged). In addition, it will be important for future research to report on and explore theoretically driven active ingredients of LC by examining not only structure but also LC processes.

Some limitations should be considered in interpreting these findings. As with any systematic review, it is possible that relevant studies were omitted. By searching multiple databases, reviewing the reference lists of key articles, and cross-checking with free-text search terms, we minimized the possibility of such omissions. In addition, negative findings are generally not published, potentially biasing our results. Despite these potential limitations, our review provides an important assessment of the state of the evidence for use of LCs in mental health care. Uses of LCs that focus on processes of care (for example, implementation of engagement practices and depression guidelines) align more closely with the targets of collaboratives that have been applied in other areas of health care. The applicability of LCs for disseminating and implementing more complex mental health evidence-based practices remains unknown; in the mental health field, such efforts often require additional specialized trainings to develop provider skills in implementing these evidence-based practices. The cost-effectiveness or added value of such an approach must thus be carefully assessed.

Conclusions

As LCs continue to grow in popularity among policy makers and national organizations, there is great need for rigorous research that evaluates the utility of these costly endeavors. Moreover, research focused on active components of LCs is vital to the replication of successful LCs, ensuring quality and fidelity to the model, guiding future adaptations, and identifying the types of innovations and improvements for which the model is most appropriate.

Acknowledgments and disclosures

Writing of this article was supported by grants from the National Institutes of Health: K01MH083694 to Dr. Nadeem and P30 MH090322 to Dr. Hoagwood.

The authors report no competing interests.

Supplementary Material

Supplemental Material (1088_ds001.pdf)

View/Download
164.70 KB

References

Honberg R, Kimball A, Diehl S, et al.: State Mental Health Cuts: The Continuing Crisis. Alexandria, Va, National Alliance on Mental Illness, 2011

Format	RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks
Direct importt
Citation style
Style

Copy to clipboard
Tips for downloading citations

Abstract

Abstract

Objective

Methods

Results

Conclusions

Methods

Results

Description of LC components

Overall LC structure.

Content of in-person learning sessions.

PDSAs.

Quality improvement team calls.

E-mail or Web support.

Quality improvement processes.

Organizational involvement.

Pre-LC activities.

Study goals and findings

Study goals.

Study findings.

Discussion

Conclusions

Acknowledgments and disclosures

Supplementary Material

References

Information

Published In

History

Authors

Details

Funding Information

Metrics

Citations

Export Citations

View options

PDF/EPUB

Get Access

Login options

Purchase Options

Not a subscriber?

Figures

Other

Share

Share article link

Share