Sustainment of Integrated Care in Addiction Treatment Settings: Primary Outcomes From a Cluster-Randomized Controlled Trial
Abstract
Objective:
Integrated treatment services are the gold standard for addressing co-occurring mental and substance use disorders, yet they are not readily available. The Network for the Improvement of Addiction Treatment (NIATx) was hypothesized to be an effective strategy to implement and sustain integrated mental health and substance use care in addiction treatment programs. This study examined sustainment of integrated services for up to 2 years after the active implementation phase.
Methods:
The effectiveness of NIATx strategies to implement and sustain integrated services was evaluated by using a cluster-randomized, waitlist control group design. Forty-nine addiction treatment organizations were randomly assigned to either NIATx1 (active implementation strategy) or NIATx2 (waitlist control). The Dual Diagnosis Capability in Addiction Treatment Index was used to evaluate organizations’ capability to provide integrated care. The NIATx Stages of Implementation Completion scale was used to assess participation in and adherence to the NIATx implementation process. Linear mixed-effects modeling was used to evaluate changes from baseline to end of the sustainment period.
Results:
Both cohorts sustained their capability to provide integrated treatment services. Both groups achieved successful implementation and sustained integrated services to a similar degree, regardless of sustainment year. Sustainment did not vary as a function of NIATx adherence.
Conclusions:
The delivery of integrated treatment services was sustained for 2 years after receipt of active implementation support. Future research should consider how contextual factors may predict, mediate, and moderate sustainment outcomes.
HIGHLIGHTS
•
Insufficient access to integrated treatment services for individuals with co-occurring mental health and substance use disorders is a persistent health service delivery gap.
•
Prior results indicate that the Network for the Improvement of Addiction Treatment (NIATx) implementation strategies can successfully be used to integrate mental health and addiction treatment services.
•
Delivery of integrated treatment services was sustained for 2 years after active implementation support.
Implementing evidence-based practices (EBPs) is an investment requiring time, effort, and expenditures. These investments are worthwhile insofar as the implemented EBPs are sustained. Sustainment, or the state of continued evidence-based intervention, service, or program delivery, is the final stage of the implementation process and is ongoing (1–3). It is a stage during which most organizations fail to continue to deliver an EBP as originally planned or intended (4, 5). Therefore, one could argue that the true measure of a successful implementation process and the effectiveness of any implementation endeavor lies in the sustainment phase. Maintenance or sustainment of outcomes of interest can serve to demonstrate and validate the value of investments and expenditures, ultimately translating to better care for patients. In this report, we present the sustainment outcomes of a study designed to evaluate the effectiveness of the Network for the Improvement of Addiction Treatment (NIATx)—a multifaceted implementation strategy—to implement integrated mental health and addiction treatment services for persons with such comorbidity (6, 7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020).
In 2018, in the United States, an estimated 9.5 million adults had co-occurring disorders (8). This is a significant public health concern, given the associated societal burden (9). Often under- or misdiagnosed due to their inherent complexity, co-occurring disorders are also often not treated effectively (10). Integrated care, in which treatment for mental health and substance use problems is delivered conjointly, at the same time, and by the same provider or providers, is the most effective form of treatment, yet it is rarely provided (11–17). Data from the 2018 National Survey on Drug Use and Health confirm that less than 8% of individuals with co-occurring disorders reported receiving integrated services (8).
Although persistent and significant efforts have been made to integrate substance use and mental health treatment services, barriers to delivery persist. Deterrents, such as financial cost, administrative obstacles, or provider resistance obstructing integrated care delivery, are situated at multiple contextual levels: system, organizational, and individual (18–20). It is thus imperative to use a multilevel, multifaceted approach, such as NIATx, that targets barriers at every level to enable and sustain delivery of integrated care. Additionally, because NIATx is effective for making practice change in behavioral health settings, promotes sustainment of change, and combines process improvement tools and techniques with quality improvement interventions, we hypothesized it to be potentially effective to implement and sustain integrated services (6, 21–28).
Outcomes from our longitudinal study confirmed NIATx as an effective implementation approach to conjointly deliver mental health and substance use treatment (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Previous results indicate that most participating agencies, regardless of study arm, successfully transitioned from offering addiction-only treatment services at baseline to providing integrated services by the end of the implementation period (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Furthermore, we demonstrated that organizations’ capability to provide integrated treatment services varied by level of NIATx adherence (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020).
Here, we present sustainment outcomes associated with implementation of integrated care for co-occurring mental and substance use disorders by using NIATx. Sustainment was considered when improved capability for integrated services was maintained up to 2 years beyond the active implementation phase.
Methods
Design
This cluster-randomized, waitlist control group trial was designed to evaluate the effectiveness of NIATx strategies for implementing and sustaining integrated services for co-occurring disorders. Recruitment began in 2016, and the sustainment period of interest concluded in 2019. Following baseline assessments, organizations enrolled in the study were randomly assigned to one of two cohorts: NIATx1 (N=25) or NIATx2 (N=24). NIATx1 received active NIATx for the first 12 months (year 1, 2017), followed by 24 months of sustainment (year 2, 2018; and year 3, 2019). NIATx2 organizations were delayed in starting NIATx for 12 months (year 1), followed by active NIATx for 12 months (year 2) and sustainment for 12 months (year 3) (a figure in an online supplement to this article depicts the design). In this article, we report on results from the sustainment period. Given the study design, the length of the sustainment period varied between groups (2 years for NIATx1 and 1 year for NIATx2), thus providing the opportunity to examine whether gains in integrated service capability were differentially durable. The study protocol has been previously described in depth (6, 7).
Participants
To participate, organizations had to be state licensed to provide addiction treatment services at outpatient, intensive outpatient, or residential levels of care; be tax exempt; have government status or meet a 50% minimum of public funding; and have no previous NIATx training or participation in a NIATx project. Participating organizations were recruited via letter by state behavioral health authorities. Forty-nine of the initially enrolled 53 (92%) community-based addiction treatment programs from the State of Washington were randomly assigned at baseline to either the NIATx1 (N=25) or NIATx2 (N=24) study arms.
Measures
Dual Diagnosis Capability in Addiction Treatment Index.
The Dual Diagnosis Capability in Addiction Treatment (DDCAT) Index is a quantitative organizational measure used to evaluate the current capability of addiction treatment programs to deliver integrated treatment services for persons with co-occurring disorders (29). It comprises 35 items across seven dimensions, including program structure; program milieu; clinical process, assessment; clinical process, treatment; continuity of care; staffing; and training. The 35 items are rated on a 5-point Likert scale, which includes anchor scores of 1, addiction-only services (AOS); 3, dual diagnosis capable (DDC); and 5, dual diagnosis enhanced (DDE). Ratings of items within a dimension are averaged to generate a subscale score. A DDCAT total score is derived by averaging the 35 items.
The DDCAT also produces a categorization of the organization’s integrated service capacity by using the American Society of Addiction Medicine taxonomy (30, 31). If less than 80% of scores are rated 3 or higher, an organization is categorized as AOS. If at least 80% of scores are 3 or higher, an organization is categorized as DDC, which is the adequate and acceptable standard for all specialty addiction treatment organizations (30, 32). If at least 80% of scores are rated 5, then an organization is categorized as DDE. The reliability and validity of the DDCAT have been demonstrated in numerous psychometric studies, and public and private health care systems have adopted it as a measure of behavioral health service integration (29, 32–35). The DDCAT Index (version 4.1) and DDCAT Toolkit (version 4.0) are in the public domain, free and available from the authors (31).
Evaluators, blind to the study arm, were rigorously trained in data collection, specifically pertaining to the DDCAT for this study. Evaluators conducted site visits in pairs to the addiction treatment organizations every 12 months for data collection. Data were collected from key informant interviews with staff and patients, site observations, and chart or program manual review over a 2- to 3-hour period. The information collected was then synthesized and summarized across all data sources by the evaluators to complete the DDCAT Index measure.
NIATx Stages of Implementation Completion.
The Stages of Implementation Completion (SIC) was adapted to track each organization’s participation and adherence to the NIATx implementation process (36, 37). Similar to the original SIC, the NIATx-SIC measure captures the proportion of completed activities, the duration of activities, and the total time frame from first to last activity (7, 36–38). Organizations with values above the average across all three components are deemed to have full NIATx adherence. NIATx activities completed during the preimplementation phase and active implementation phase were recorded by our research team and entered into an online data collection and reporting system.
Ethics
The institutional review boards at Stanford University, the University of Wisconsin–Madison, and the Washington State Health Care Authority reviewed the study and determined it to be exempt.
Statistical Analysis
McNemar’s test was used to compare the proportion of agencies categorized as AOS versus DDC and DDE between baseline and study end. Longitudinal outcomes were analyzed both with and without consideration for NIATx adherence. Standard linear mixed-effects modeling was employed to estimate changes from baseline to end of sustainment in the organization’s capability for providing dual diagnosis in addiction treatment (39, 40). For all model estimation, we used maximum likelihood embedded in the Mplus program (41). Specifically, we used a random-intercept piecewise model, assuming three segments of linear change over time, from baseline to year 1, year 1 to year 2, and year 2 to year 3.
Per the intention-to-treat principle, all randomized agencies were included in the analyses as long as their data were available from at least one assessment. Unavailable data resulting from attrition or missed assessments were treated as missing-at-random, conditional-on-available (observed) information, a standard strategy of handling missing data in mixed-effects modeling. In the per-protocol analysis, we examined the two randomized arms after excluding organizations that were not fully adherent to NIATx, as determined by the NIATx-SIC. In both the intent-to-treat and per-protocol analyses, Cohen’s d, a measure of effect size, was calculated based on observed standard deviation pooled across the two conditions at each time point.
Results
Baseline Characteristics
Forty-nine of the 53 organizations enrolled in the study were randomly assigned to either NIATx1 (N=25) or NIATx2 (N=24). By study end, 40 organizations (82%) remained enrolled. A majority of organizations that discontinued participation did so because their facility closed or was sold (see online supplement for an extended CONSORT diagram [42]).
Baseline characteristics of organizations have been extensively presented in previous reports (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Participants were community addiction treatment organizations located in Washington State. Participating organizations were primarily from the public sector and located in regions with shortages in health care services (43). Organizations in the NIATx1 group had more annual admissions on average (487.9±777.6), compared with the NIATx2 group (276.5±243.1). NIATx2 programs had longer lengths of stay on average (137.1±109.6), compared with NIATx1 programs (113.2±109.4).
Primary Outcome: Changes in Dual Diagnosis Capability
Across both study arms, dual diagnosis capability improved and was maintained throughout the study period. Both groups started with a substantial number of organizations categorized as AOS (78%, N=31). At study end, in a complete reversal, the preponderance of organizations (78%) was at least DDC across both groups (p<0.05). Specifically, 35% (N=7) of NIATx1 organizations were at least DDC at baseline, compared with 80% (N=16) at year 3 (p<0.001). For NIATx2, 10% (N=2) were DDC or DDE at baseline, compared with 75% (N=15) at year 3 (p<0.001). Although most gains were made between baseline and the end of the active implementation period, both cohorts continued to experience growth in DDC organizations between the end of the intervention and the end of the first sustainment period (NIATx1, 50% [N=10] to 65% [N=13]; NIATx2, 60% [N=12] to 70% [N=14]). NIATx1 saw further accrual in DDC organizations in the second sustainment year (from 65% [N=13] to 75% [N=15] in year 3).
Evidence for Sustained EBP
Intent to treat.
Intent-to-treat analyses, including all randomly assigned agencies (N=49), were conducted by using mixed-effects modeling (Table 1; see also online supplement). No significant changes occurred in the NIATx1 group in its first year after active implementation. During the second sustainment year, the NIATx1 group showed significant improvement in the DDCAT subscale metrics clinical process, assessment; clinical process, treatment; and continuity of care. No significant changes occurred for the NIATx2 group in the sustainment year. Effect sizes for changes from baseline to study end ranged from small to large (Cohen’s d=0.25 to 0.88). The only significant between-group difference in terms of the change from baseline to study end was for the training subscale: NIATx2 outperformed NIATx1 (Cohen’s d=0.88, p=0.02).
| Change from year 1 to year 2 | Change from year 2 to year 3 | Change from baseline to year 3 | |||||
|---|---|---|---|---|---|---|---|
| NIATx1 | NIATx2 | NIATx1 | NIATx2 | NIATx1 vs. | |||
| DDCAT dimension | SY1 | Active year | SY2 | SY1 | NIATx1 | NIATx2 | NIATx2 |
| Overall | –.14 | .38* | .21 | .05 | .83** | 1.07** | –.24 |
| Program structure | –.08 | .49* | .19 | .09 | .87** | 1.19** | –.33 |
| Program milieu | –.25 | .33 | .31 | .04 | 1.10** | .86** | .24 |
| Clinical process, assessment | –.10 | .28 | .36* | .11 | .68** | .87** | –.20 |
| Clinical process, treatment | –.17 | .28 | .26* | .04 | .83** | 1.05** | –.21 |
| Continuity of care | –.22 | .45** | .30* | .03 | .85** | 1.08** | –.23 |
| Staffing | –.23 | .35* | .27 | –.03 | .79** | 1.05** | –.26 |
| Training | .03 | .44 | –.20 | .03 | .68** | 1.35** | –.68* |
a
NIATx, Network for the Improvement of Addiction Treatment. DDCAT, Dual Diagnosis Capability in Addiction Treatment Index. Each DDCAT dimensions is rated on a scale from 1 to 5, with 1 indicating that the addiction treatment provider offers addiction-only services, 3 indicating that the provider offers services that are dual diagnosis capable, and 5 indicating that the provider offers enhanced dual diagnosis services. The overall DDCAT score is the average of all seven dimensions. Values presented in the table reflect the change in DDCAT score between each time point, not the DDCAT score itself.
*
p≤.05, **p≤.001.
Per protocol.
Twenty-three agencies were fully adherent to NIATx per their SIC scores (NIATx1, N=11; NIATx2, N=12) and were included in per-protocol analyses (Table 2; see also online supplement). Neither NIATx1 or NIATx2 agencies showed any significant changes in the first year after active implementation. No significant changes were noted for the NIATx1 group in its second sustainment year. However, the NIATx1 group slightly improved on the subscale for clinical process, assessment, whereas NIATx2 slightly deteriorated on that subscale. Effect sizes for changes from baseline to study end were small and ranged from 0.04 to 0.34. There were no significant between-group differences at study end in terms of change from baseline.
| Change from year 1 to year 2 | Change from year 2 to year 3 | Change from baseline to year 3 | |||||
|---|---|---|---|---|---|---|---|
| NIATx1 | NIATx2 | NIATx1 | NIATx2 | NIATx1 vs. | |||
| DDCAT dimension | SY1 | active year | SY2 | SY1 | NIATx1 | NIATx2 | NIATx2 |
| Overall | −.12 | .57** | .06 | −.09 | .98** | 1.06** | −.08 |
| Program structure | −.14 | .75* | −.03 | .04 | .98** | 1.20** | −.23 |
| Program milieu | −.35 | .63* | .15 | −.19 | 1.16** | .90** | .26 |
| Clinical process, assessment | −.14 | .38 | .37 | −.23 | .93** | .85** | .08 |
| Clinical process, treatment | −.18 | .33 | .21 | −.19 | 1.00** | .94** | .06 |
| Continuity of care | −.12 | .65** | .22 | −.16 | 1.12** | 1.16** | −.04 |
| Staffing | −.05 | .73** | −.14 | −.02 | .77* | 1.11** | −.34 |
| Training | .15 | .54** | −.35 | .08 | .89* | 1.16** | −.27 |
a
NIATx, Network for the Improvement of Addiction Treatment. DDCAT, Dual Diagnosis Capability in Addiction Treatment Index. Each DDCAT dimensions is rated on a scale from 1 to 5, with 1 indicating that the addiction treatment provider offers addiction-only services, 3 indicating that the provider offers services that are dual diagnosis capable, and 5 indicating that the provider offers enhanced dual diagnosis services. The overall DDCAT score is the average of all seven dimensions. Values presented in the table reflect the change in DDCAT score between each time point, not the DDCAT score itself.
*
p≤.05, **p≤.001.
Discussion
Gains in capability for integrated treatment services were sustained after the active implementation year. Both groups achieved successful implementation and sustained the implementation to a similar degree regardless of the sustainment year. At baseline, 78% of participating organizations were categorized as AOS. By the end of the trial, the ratio was reversed: 78% of participating agencies across both groups were at least DDC. These 31 agencies significantly improved and maintained care and services for their patients with dual diagnoses of mental and substance use disorders. Attainment of DDC or DDE status mainly was achieved during the active implementation phase, and these changes were sustained until study end. Additionally, NIATx1 organizations continued to improve on DDCAT subscales of clinical process, assessment; clinical process, treatment; and continuity of care throughout the sustainment period.
This study had some limitations. There was a volunteer bias for organizations that agreed to participate in the study, those that completed the implementation strategy, and those that maintained participation through study end. Also, baseline differences existed between cohorts. Although statistical analyses accounted for these differences, it would have been preferable to balance groups on key variables. Further, an additional measure of integrated treatment delivery would validate our results and add richness to the interpretation of findings, including patient-level or service-level outcomes. Nonetheless, previous studies with the DDCAT Index have found this measure to have good validity and to be associated with integrated service delivery and receipt and with patient outcomes (6, 7, 29, 34, 35, 44, 45).
These results build on, solidify, and extend our previous findings. We previously established the importance of using a proven manualized but adaptive strategy, such as NIATx, to enable change, as well as the importance of documenting participation during the active implementation stage and of measuring adherence to participation when examining outcomes (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Here, we have demonstrated that increased capability to deliver integrated treatment services for individuals with co-occurring mental and substance use disorders was sustained. This finding is significant because individuals with comorbid disorders in Washington State now have increased access to integrated care, the gold standard for co-occurring disorders.
Results from both intent-to-treat and per-protocol analyses converged, and minor differences in sustainment outcomes emerged between both sets of analyses. Thus our results indicated sustained changes in behavioral health settings when these settings were evaluated under both actual and ideal circumstances (46).
It is, however, challenging to determine what accounted for the lack of difference between both sets of analyses. Following our year 1 and year 2 results, we would expect adherent organizations to outperform nonadherent organizations during the sustainment period. We suggest that this lack of differences was attributable to enhanced monitoring and feedback. Performance data were compiled and provided to participants for their agencies, along with data from all participating agencies combined. All organizations received this enhanced monitoring and feedback from baseline to study end, regardless of adherence and participation status. Our study design did not account for enhanced monitoring and feedback, and thus its role cannot be statistically confirmed. However, the results from the intent-to-treat analyses in sustainment make a convincing argument that enhanced monitoring and feedback is a viable and effective implementation strategy.
Conclusions
This article presents results from a study of the sustainment of integrated services after active implementation support was withdrawn. We observed wide variation in adherence to the study implementation strategy (NIATx) yet good overall sustainment of the target outcome. To further confirm these findings and advance the field of addiction treatment health services dissemination and implementation science, a better understanding of what and how an organization’s contextual factors may predict, moderate, and mediate sustainment outcomes is needed (47). This need is further strengthened by findings from Swain et al. (48), who reported differences in sustainability of mental health services for people with mental illness on the basis of agency characteristics, such as financing, staffing, training, fidelity, and agency leadership. Future studies will also be needed to determine whether NIATx is effective over time, compared with groups that did not receive the intervention.
In implementation practice and research, it is typical to offer a fixed protocol for expected strategy participation. A one-size-fits-all implementation strategy is, however, neither effective or efficient. It creates a high demand and burden on organizations to engage in all activities, whether they need to or not, to increase access to an EBP. Much like a stepped-care approach in a health care situation, a similar approach to implementation might apply whereby low-intensity strategies, such as monitoring and feedback, might be tried first and then evaluated for desired outcomes. If not effective, then increasingly resource-intensive strategies might be deployed. This stepped measurement–based implementation approach lends itself to an adaptive trial design and may inform future implementation endeavors with greater specificity and cost-efficiency. In addition, it would, decrease program burden and make better use of limited resources to support program implementation. Intensive supports that would have been spent on programs needing only a light touch could be redirected toward reaching other programs.
Footnotes
The authors express their gratitude to all partners on this project: Washington State Department of Social and Health Services, Michael Langer, Thomas Fuchs, Leslie Carey, and the leadership, staff, and patients from all participating community addiction treatment programs. They are also grateful for the work of Theresa Sharin, Amy McIlvaine, Eric Osborne, Ahney King, and Kevin Campbell.
NIDA was not involved in data collection, data analysis, or writing of the report. The statements made here are those of the authors.
Supplementary Material
File (appi.ps.202000293.ds001.pdf)
- View/Download
- 522.38 KB
References
1.
Aarons GA, Green AE, Trott E, et al: The roles of system and organizational leadership in system-wide evidence-based intervention sustainment: a mixed-method study. Adm Policy Ment Health Ment Health Serv Res 2016; 43:991–1008
2.
Moullin JC, Dickson KS, Stadnick NA, et al: Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci 2019; 14:1
3.
Palinkas LA, Spear SE, Mendon SJ, et al: Measuring sustainment of prevention programs and initiatives: a study protocol. Implement Sci 2016; 11:95
4.
Wright C, Catty J, Watt H, et al: A systematic review of home treatment services—classification and sustainability. Soc Psychiatry Psychiatr Epidemiol 2004; 39:789–796
5.
Moore JE, Mascarenhas A, Bain J, et al: Developing a comprehensive definition of sustainability. Implement Sci 2017; 12:110
6.
Ford JH 2nd, Osborne EL, Assefa MT, et al: Using NIATx strategies to implement integrated services in routine care: a study protocol. BMC Health Serv Res 2018; 18:431
7.
Assefa MT, Ford JH 2nd, Osborne E, et al: Implementing integrated services in routine behavioral health care: primary outcomes from a cluster randomized controlled trial. BMC Health Serv Res 2019; 19:749
8.
Key Substance Use and Mental Health Indicators in the United States: Results From the 2019 National Survey on Drug Use and Health. HHS pub no PEP20-07-01-001, NSDUH Series H-55. Rockville, MD, Substance Abuse and Mental Health Services Administration, 2020
9.
Drake RE, Mueser KT, Brunette MF, et al: A review of treatments for people with severe mental illnesses and co-occurring substance use disorders. Psychiatr Rehabil J 2004; 27:360–374
10.
Temmingh HS, Williams T, Siegfried N, et al: Risperidone versus other antipsychotics for people with severe mental illness and co-occurring substance misuse. Cochrane Database Syst Rev 2018; 1:CD011057
11.
Karapareddy V: A review of integrated care for concurrent disorders: cost effectiveness and clinical outcomes. J Dual Diagn 2019; 15:56–66
12.
Baigent M: Managing patients with dual diagnosis in psychiatric practice. Curr Opin Psychiatry 2012; 25:201–205
13.
Baker AL, Kavanagh DJ, Kay-Lambkin FJ, et al: Randomized controlled trial of cognitive-behavioural therapy for coexisting depression and alcohol problems: short-term outcome. Addiction 2010; 105:87–99
14.
Hawkins EH: A tale of two systems: co-occurring mental health and substance abuse disorders treatment for adolescents. Annu Rev Psychol 2009; 60:197–227
15.
Hides LM, Elkins KS, Scaffidi A, et al: Does the addition of integrated cognitive behaviour therapy and motivational interviewing improve the outcomes of standard care for young people with comorbid depression and substance misuse? Med J Aust 2011; 195(suppl 3):S31–S37
16.
Kelly TM, Daley DC: Integrated treatment of substance use and psychiatric disorders. Soc Work Public Health 2013; 28:388–406
17.
Steinberg KL, Roffman RA, Carroll KM, et al: Brief Counseling for Marijuana Dependence: A Manual for Treating Adults. HHS pub no (SMA) 12-4211. Rockville, MD, Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment, 2005
18.
Damschroder LJ, Aron DC, Keith RE, et al: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4:50
19.
Lewis CC, Scott K, Marriott BR: A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Implement Sci 2018; 13:68
20.
Powell BJ, Beidas RS, Lewis CC, et al: Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res 2017; 44:177–194
21.
Ford JH 2nd, Stumbo SP, Robinson JM: Assessing long-term sustainment of clinic participation in NIATx200: results and a new methodological approach. J Subst Abuse Treat 2018; 92:51–63
22.
Stumbo SP, Ford JH 2nd, Green CA: Factors influencing the long-term sustainment of quality improvements made in addiction treatment facilities: a qualitative study. Addict Sci Clin Pract 2017; 12:26
23.
Ford JH 2nd, Abraham AJ, Lupulescu-Mann N, et al: Promoting adoption of medication for opioid and alcohol use disorders through system change. J Stud Alcohol Drugs 2017; 78:735–744
24.
Gustafson DH, Quanbeck AR, Robinson JM, et al: Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction 2013; 108:1145–1157
25.
Hoffman KA, Ford JH 2nd, Choi D, et al: Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment. Drug Alcohol Depend 2008; 98:63–69
26.
McCarty D, Gustafson DH, Wisdom JP, et al: The Network for the Improvement of Addiction Treatment (NIATx): enhancing access and retention. Drug Alcohol Depend 2007; 88:138–145
27.
Quanbeck A, Wheelock A, Ford JH 2nd, et al: Examining access to addiction treatment: scheduling processes and barriers. J Subst Abuse Treat 2013; 44:343–348
28.
Schmidt LA, Rieckmann T, Abraham A, et al: Advancing recovery: implementing evidence-based treatment for substance use disorders at the systems level. J Stud Alcohol Drugs 2012; 73:413–422
29.
McGovern MP, Matzkin AL, Giard J: Assessing the dual diagnosis capability of addiction treatment services: the Dual Diagnosis Capability in Addiction Treatment (DDCAT) Index. J Dual Diagn 2007; 3:111–123
30.
Miller SC, Fiellin DA, Rosenthal RN, et al. (eds): The ASAM Principles of Addiction Medicine, 6th ed. Philadelphia, Wolters Kluwer, 2019
31.
DDCAT Dual Diagnosis Capability in Addiction Treatment (DDCAT). Rockville, MD, Substance Abuse and Mental Health Services Administration, 2011
32.
Sacks S, Chaple M, Sirikantraporn J, et al: Improving the capability to provide integrated mental health and substance abuse services in a state system of outpatient care. J Subst Abuse Treat 2013; 44:488–493
33.
Gotham HJ, Claus RE, Selig K, et al: Increasing program capability to provide treatment for co-occurring substance use and mental disorders: organizational characteristics. J Subst Abuse Treat 2010; 38:160–169
34.
Lambert-Harris C, Saunders EC, McGovern MP, et al: Organizational capacity to address co-occurring substance use and psychiatric disorders: assessing variation by level of care. J Addict Med 2013; 7:25–32
35.
McGovern MP, Lambert-Harris C, McHugo GJ, et al: Improving the dual diagnosis capability of addiction and mental health treatment services: implementation factors associated with program level changes. J Dual Diagn 2010; 6:237–250
36.
Chamberlain P, Brown CH, Saldana L: Observational measure of implementation progress in community-based settings: the Stages of Implementation Completion (SIC). Implement Sci 2011; 6:116
37.
Saldana L: The Stages of Implementation Completion for evidence-based practice: protocol for a mixed methods study. Implement Sci 2014; 9:43
38.
Saldana L, Chamberlain P: Supporting implementation: the role of community development teams to build infrastructure. Am J Community Psychol 2012; 50:334–346
39.
Raudenbush SW, Bryk AS: Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd ed. Thousand Oaks, CA, Sage Publications, 2002
40.
Singer JD, Willett JB: Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford, United Kingdom, Oxford University Press, 2009
41.
Muthén LK, Muthén BO: Mplus: Statistical Analysis With Latent Variables: User’s Guide. Los Angeles, Muthén & Muthén, 1998
42.
Campbell MK, Piaggio G, Elbourne DR, et al: Consort 2010 statement: extension to cluster randomised trials. BMJ 2012; 345:e5661
43.
Scoring Shortage Designations. Rockville, MD, Health Resources and Services Administration, Bureau of Health Workforce, 2020. https://bhw.hrsa.gov/shortage-designation/hpsa-criteria. Accessed Dec 4, 2019
44.
Chaple M, Sacks S: The impact of technical assistance and implementation support on program capacity to deliver integrated services. J Behav Health Serv Res 2016; 43:3–17
45.
McGovern MP, Lambert-Harris C, Gotham HJ, et al: Dual diagnosis capability in mental health and addiction treatment services: an assessment of programs across multiple state systems. Adm Policy Ment Health Ment Health Serv Res 2014; 41:205–214
46.
Ranganathan P, Pramesh CS, Aggarwal R: Common pitfalls in statistical analysis: intention-to-treat versus per-protocol analysis. Perspect Clin Res 2016; 7:144–146
47.
Lewis CC, Boyd MR, Walsh-Bailey C, et al: A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci 2020; 15:21
48.
Swain K, Whitley R, McHugo GJ, et al: The sustainability of evidence-based practices in routine mental health agencies. Community Ment Health J 2010; 46:119–129
Information & Authors
Information
Published In
History
Received: 28 April 2020
Revision received: 21 October 2020
Revision received: 26 February 2021
Revision received: 13 April 2021
Accepted: 24 May 2021
Published online: 4 August 2021
Published in print: March 01, 2022
Keywords
Authors
Competing Interests
The authors report no financial relationships with commercial interests.
Funding Information
This study was funded by grant R01DA037222 from the National Institute on Drug Abuse (NIDA) (Drs. McGovern and Ford, principal investigators) and by grant R01 DA044745 from NIDA (Dr. Saldana, principal investigator). ClinicalTrials.gov, NCT03007940. Registered January 2, 2017.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).