Costs to Community Mental Health Agencies to Sustain an Evidence-Based Practice
Abstract
Objective:
Dissemination of evidence-based practices (EBPs) has become a priority in children’s mental health services. Although implementation approaches and initiatives are proliferating, little is known about sustainment of EBPs, but evidence suggests that most EBPs are not sustained for more than a few years. Cost is the most frequently cited barrier to sustainment, yet very little is known about these costs. This study provides a method for quantifying incremental costs of an EBP compared with usual care and preliminary data on the costs in staff time, lost revenue, and other expenses of sustaining an EBP (trauma-focused cognitive-behavioral therapy [TF-CBT]) in community mental health settings.
Methods:
Fourteen community mental health agencies (CMHAs) completed a measure developed for this study to collect administrative data on implementation costs to sustain TF-CBT. Survey items captured activities that were related specifically to TF-CBT and that would not otherwise be conducted for usual care, such as TF-CBT training. Staff time in hours was converted to monetary estimates.
Results:
Costs varied widely across agencies. Preliminary results indicated that agencies spent on average $65,192 per year (2014 U.S.$) on incremental costs for TF-CBT sustainment (excluding costs of external trainers and other support); the average incremental cost per client was $1,896.
Conclusions:
The costs to sustain the EBP suggest that maintaining an EBP is a financial burden for CMHAs and that these costs can be a potential barrier to broader EBP uptake. Implications for public policy include providing reimbursement rates and financial incentives to offset potential implementation costs and promote sustainment of EBPs.
The growing interest in improving patient outcomes and the passage of the Affordable Care Act have mobilized efforts to increase access to high-quality, cost-efficient evidence-based practices (EBPs). Although research on strategies and efforts to disseminate and implement EBPs is increasing (1–3), research suggests that rates of long-term sustainment of EBPs are relatively low (4). Cost is one of the most significant barriers to sustainment of EBPs in community mental health agencies (CMHAs), yet EBPs are rarely reimbursed at higher rates than usual care despite evidence that they save money over the long term (5). This study was designed to describe the incremental implementation costs of providing an EBP compared with usual care.
Implementation costs have been cited as the most significant and least modifiable barrier to EBP implementation and sustainment (6). Costs are one of eight implementation outcomes relevant for evaluating EBP dissemination, yet there has been less research of implementation costs than of any other EBP implementation outcome (7). For CMHAs, EBP implementation costs include the time, effort, and monetary expenses associated with activities necessary to implement an EBP during all phases of implementation (7). Staff may need to attend training sessions (which may involve travel), engage in model-specific consultation and supervision, report and interpret data, modify data systems, provide quality assurance, review model materials, administer and score measures, modify agency practices and policies, and educate other agency staff or external partners about the EBP. Many of these implementation costs fall on the shoulders of the CMHA tasked with delivering the EBP (8). Implementation activities require additional staff time that may negatively affect the CMHA, possibly by reducing revenue.
Several studies have explored implementation costs during early stages of the implementation process (9,10). However, little research has been published on the sustainment phase of EBP implementation (11,12). Further, virtually no studies have examined implementation costs associated with the sustainment phase. The costs borne by CMHAs during sustainment may differ from costs associated with initial implementation, and understanding these sustainment costs is critical to fully integrate and maintain EBPs. For instance, one study reported that only 23 (47%) of 49 CMHAs sustained an EBP over a six-year period after initial implementation (4). Among the sites that discontinued the EBP, 95% cited costs as the primary reason. Thus sustainment costs are especially important to understand because initial implementation costs, although often quite high (11), are typically funded through a time-limited grant that does not support the sustainment costs of the EBP indefinitely.
Implementation costs are defined in this study as the additional costs in hours, effort, and dollars of implementing an EBP compared with usual care. These costs are critical to agencies’ decision to adopt, implement, and sustain an EBP. This study is the first to our knowledge that provides a method for quantifying the incremental costs of an EBP, and it provides preliminary data to answer the following research questions: First, what is the incremental cost to an agency to sustain an EBP? Second, what is the average annual incremental cost to an agency per child served and staff member trained of sustaining the EBP? Third, what activities account for the largest costs of sustaining an EBP? Fourth, to what extent do agencies provide productivity reductions to clinicians providing the EBP? Last, are the characteristics of an agency related to incremental costs?
Methods
Background
The Connecticut Department of Children and Families (DCF) contracted with the Child Health and Development Institute, an independent nonprofit organization with expertise in EBP dissemination, to disseminate trauma-focused cognitive-behavioral therapy (TF-CBT) to 16 community-based agencies in Connecticut from 2007 to 2010 (13). Agencies selected a subset of staff clinicians to be trained through the statewide dissemination efforts. TF-CBT is an outpatient treatment for traumatic stress reactions among children and adolescents ages three to 17 who have experienced a traumatic event (14). TF-CBT has demonstrated efficacy in randomized controlled trials (15) and has been successfully disseminated in community mental health settings (9). TF-CBT was the first widely implemented EBP at these CMHAs. The initial Connecticut dissemination was conducted from 2007 to 2010 by using the Breakthrough Series Collaborative model (16), after which all agencies continued providing TF-CBT for three to six years and received external training and consultation support through a state-funded TF-CBT Coordinating Center (17). Data were collected for quality improvement purposes, and the initiative was deemed exempt by the DCF Institutional Review Board.
Participants
Agency leaders from the 16 CMHAs that participated in the initial TF-CBT dissemination were invited to participate, and all agreed. Two agencies submitted incomplete data and were excluded from the analysis, resulting in valid data from 14 agencies. Each CMHA included at least one clinician, supervisor, and senior leader (for example, an agency administrator with oversight of children’s services) who provided or supported TF-CBT directly.
Measures
Survey development.
The Sustainment Cost Survey (SCS) is a 49-item measure developed for this study to collect administrative data on the costs to CMHAs—both in time and money—associated with sustaining an EBP following initial implementation. Items were initially generated through focus groups with CMHA senior leaders and were developed by the second author through an iterative process of revision across all participating agencies. Each item was generated to capture activities that pertained specifically to TF-CBT and that otherwise would not be conducted for usual care (such as TF-CBT training sessions). The survey was designed to be completed by the agency administrator with input from team members as needed.
The SCS includes information about agency characteristics (such as total staff size, EBP staff size, and agency budget), average salaries, various activities related specifically to TF-CBT sustainment, and questions about agency processes. These activities included training; consultation and supervision; data systems and data reporting requirements; clinical preparation, such as reviewing the TF-CBT treatment manual; and additional costs, such as costs for site coordinator managerial duties, not required for usual care. For example, a survey item related to training asked, “How many staff attended training by a TF-CBT trainer in the past 12 months?” A consultation and supervision item was used to note the amount of “staff time to attend TF-CBT consultation calls/Webinars with TF-CBT trainers.” Respondents reported the amount of time spent by staff on these activities on a weekly, monthly, or annual basis (all hours were converted to annual estimates). Each agency also reported the weekly productivity (billable hours) requirement for clinicians and noted any recurring (for example, weekly) productivity reductions related specifically to TF-CBT. For instance, agencies were asked, “Describe any standard productivity reductions related to TF-CBT.”
Incremental lost revenue (LR) and additional staff time (AST).
In the SCS, agency costs are categorized by three different types of costs. First, LR hours were defined as time spent by staff that resulted in a reduction in a billable session and that would not have been spent for usual care. LR time differed from the “standard productivity reductions” described above because LR hours did not represent ongoing reductions in productivity requirements for clinicians. Instead, they represented revenue lost for other reasons, such as revenue lost for several days while a clinician attended in-person training. Second, AST was defined as time spent on TF-CBT activities that would otherwise not be required for usual care and for which there was no loss of revenue (for example, time spent participating in consultation calls for which clinicians did not receive a productivity reduction). Finally, expenses were defined as direct monetary costs to the agency that were required or recommended to provide TF-CBT but not usual care; these included travel expenses, supplies, and other materials.
Data Collection Procedures
Surveys were distributed to 16 agencies in June 2013, and data collection ended in November 2013, by which time each agency had provided TF-CBT for four to five years. Agencies were requested to retrospectively report on the past year’s TF-CBT activities as part of the TF-CBT Coordinating Center’s quality improvement activities. To ensure accuracy of data collection, agency leaders were asked to work collaboratively with administrators and review records at their agency to complete the survey. Most agencies returned surveys within two to three months (mean±SD=2.8±1.4) of initial receipt.
Once the survey was completed, a rigorous review process was conducted to ensure accurate data. First, authors reviewed the surveys for missing data, discrepancies, and miscalculations. Common issues included whether an activity was categorized as LR or AST, duplication of costs in multiple categories, and miscalculation of hours. Next, agencies were contacted via e-mail if there were questions about their responses, and the agencies provided clarification; the review process was repeated as necessary. On average, surveys were sent back twice to agencies for further clarification; agency surveys were finalized through correspondence with the authors in approximately four months (3.9±.7). Throughout the review process, the authors created and used a decision tree to standardize guidance and decisions.
Conversion of Hours to Dollars
All time spent on TF-CBT activities was reported in hours. Agency reports of hours were standardized to obtain an annual figure assuming 48 working weeks per year, accounting for vacation and sick leave. These hours were summed and averaged across and within agencies. All monetary amounts were reported in U.S. dollars and were adjusted for the rate of inflation from 2013 to 2014.
Conversion of LR.
To convert hours of LR to a monetary cost, the sum of LR hours per agency was multiplied by the cost per hour for child outpatient psychotherapy. The standard rate for 13 of the CMHAs was $73.99 per session; one CMHA received a standard rate of $59.19, per state regulations. We then adjusted the LR by using an indicator of average rate of attendance (86%±6%), which was based on reported no-show rates for TF-CBT appointments at each agency.
Cost of AST.
The total monetary cost of AST was determined by multiplying the average number of hours of AST for each staff position. Because the SCS did not specifically query for hours by each staff position, we used the proportion of staff in each position to estimate how many hours that staff position spent on TF-CBT activities within each agency. Next, we multiplied AST hours for each position by the hourly agency salary for senior leaders, supervisors, and clinicians, respectively, and then summed the total costs for each agency. These costs represented the monetary cost of AST spent on sustaining TF-CBT.
Estimated full-time-equivalent staff.
Total time associated with LR and AST was converted into estimates of the full-time-equivalent (FTE) staff associated with providing TF-CBT. Total hours were divided equally across staff members to estimate the average FTE per staff member associated with providing TF-CBT. For example, agency 1 reported providing 2,975.75 total hours for TF-CBT activities annually across 19 TF-CBT staff members. Thus we estimated each TF-CBT staff member at agency 1 spent an average of 156.62 hours per year, or .08 FTE, on TF-CBT activities.
Results
Agency Characteristics
The 14 CMHAs varied in size and staffing. Agencies reported annual outpatient child mental health services budgets ranging from $756,350 to $3 million ($1,818,503±$652,753). A summary of agency characteristics, including number of staff trained in TF-CBT and number of children who received TF-CBT, is included in Table 1.
| Agency | Clinicians | Supervisors | Senior leaders | Total staff | Clients |
|---|---|---|---|---|---|
| 1 | 15 | 3 | 1 | 19 | 70 |
| 2 | 24 | 0 | 2 | 26 | 16 |
| 3 | 8 | 3 | 1 | 12 | 23 |
| 4 | 5 | 1 | 1 | 7 | 57 |
| 5 | 8 | 2 | 2 | 12 | 45 |
| 6 | 13 | 2 | 1 | 16 | 23 |
| 7 | 15 | 4 | 1 | 20 | 34 |
| 8 | 11 | 3 | 1 | 15 | 50 |
| 9 | 21 | 3 | 2 | 26 | 30 |
| 10 | 11 | 1 | 1 | 13 | 20 |
| 11 | 10 | 1 | 1 | 12 | 35 |
| 12 | 13 | 2 | 2 | 17 | 47 |
| 13 | 17 | 4 | 2 | 23 | 83 |
| 14 | 9 | 5 | 1 | 15 | 59 |
| Mean | 12.86 | 2.43 | 1.36 | 16.64 | 42.29 |
| SD | 5.23 | 1.40 | .50 | 5.61 | 20.03 |
a
TF-CBT, trauma-focused cognitive-behavioral therapy; CMHAs, community mental health agencies
Clinician salaries ranged from $53,799 to $78,042 ($70,968±$11,959). Supervisor salaries ranged from $72,421 to $125,448 ($92,519±$18,495). Senior administrator salaries ranged from $81,900 to $193,536 ($130,893±$31,939).
Incremental Implementation Costs
CMHAs varied widely on which activities they reported as LR or AST, meaning CMHAs varied in the extent to which they made occasional reductions in billable session requirements for staff (Table 2). The amount of time spent by the average staff member on TF-CBT activities was the equivalent of .04 FTE staff. Table 3 summarizes all monetary costs of LR, AST, and direct expenses and provides estimates of costs by child served and staff member trained. These incremental costs to sustain TF-CBT accounted for an average of 3.81%±1.44% of each CMHA’s annual budget.
| Agency | LR (hours) | AST (hours) | Total hours | Hours per staffb | FTE staff |
|---|---|---|---|---|---|
| 1 | 0 | 2,975.75 | 2,975.75 | 156.62 | .08 |
| 2 | 206.29 | 889.77 | 1,096.06 | 42.16 | .02 |
| 3 | 398.74 | 229.83 | 628.56 | 52.38 | .03 |
| 4 | 0 | 994.99 | 994.99 | 142.14 | .07 |
| 5 | 1,180.33 | 22.15 | 1,202.48 | 100.21 | .05 |
| 6 | 39.87 | 1,299.58 | 1,339.46 | 83.72 | .04 |
| 7 | 588.76 | 225.21 | 813.98 | 40.70 | .02 |
| 8 | 309.50 | 134.76 | 444.26 | 29.62 | .02 |
| 9 | 507.32 | 505.80 | 1,013.12 | 38.97 | .02 |
| 10 | 938.14 | 725.94 | 1,664.08 | 128.01 | .07 |
| 11 | 767.57 | 1,079.17 | 1,846.74 | 153.89 | .08 |
| 12 | 0 | 1,403.19 | 1,403.19 | 82.54 | .04 |
| 13 | 0 | 1,178.67 | 1,178.67 | 51.25 | .03 |
| 14 | 957.30 | 225.44 | 1,182.74 | 78.86 | .04 |
| Mean | 420.99 | 849.31 | 1,270.29 | 84.36 | .04 |
| SD | 411.43 | 769.91 | 599.46 | 45.11 | .02 |
a
FTE, full-time equivalent; TF-CBT, trauma-focused cognitive-behavioral therapy; CMHAs, community mental health agencies
b
The SCS does not specifically query for hours by each staff position; we used the proportion of staff in each position to estimate how many hours each staff position spent on TF-CBT activities within each agency.
| Agency | LR costsb | AST costsc | Expenses | Total costs | Cost per client served | Cost per staff member trained |
|---|---|---|---|---|---|---|
| 1 | 0 | 86,032 | 4,604 | 90,636 | 1,295 | 4,770 |
| 2 | 15,263 | 39,228 | 450 | 54,941 | 3,434 | 2,034 |
| 3 | 29,502 | 7,658 | 800 | 37,960 | 1650 | 3,163 |
| 4 | 0 | 54,267 | 744 | 55,011 | 965 | 7,858 |
| 5 | 87,333 | 1,127 | 2,585 | 91,045 | 2,023 | 7,587 |
| 6 | 2,950 | 47,747 | 185 | 50,882 | 2,212 | 3,180 |
| 7 | 43,563 | 8,492 | 1,454 | 53,508 | 1,574 | 2,675 |
| 8 | 18,320 | 3,625 | 3,696 | 25,641 | 513 | 1,709 |
| 9 | 37,536 | 18,615 | 517 | 56,668 | 1,889 | 2,179 |
| 10 | 69,413 | 21,423 | 2,080 | 92,916 | 4,646 | 7,147 |
| 11 | 56,792 | 45,083 | 1,100 | 102,976 | 2,942 | 8,581 |
| 12 | 0 | 64,313 | 40 | 64,353 | 1,369 | 3,785 |
| 13 | 0 | 54,625 | 1,151 | 55,776 | 672 | 2,425 |
| 14 | 70,831 | 8,949 | 600 | 80,379 | 1,362 | 5,358 |
| Mean | 30,822 | 32,942 | 1,429 | 65,192 | 1,896 | 4,461 |
| SD | 30,562 | 26,539 | 1,359 | 22,804 | 1,127 | 2,421 |
a
TF-CBT, trauma-focused cognitive-behavioral therapy; CMHAs, community mental health agencies
b
Lost revenue
c
Additional staff time
Implementation Costs by Activity
Table 4 summarizes the LR and AST hours and agency expenses by sustainment activity. The most time (sum of LR and AST) was spent on consultation (for example, internal TF-CBT coaching and supervision), followed by “other” activities, training, practice, and data requirements. Agencies reported the greatest expenses for “other” activities, primarily related to the time spent by site coordinators to manage the TF-CBT team. Overall, AST hours were greater than LR hours, suggesting that most of the additional time required for implementation did not result in associated productivity reductions for staff.
| Agency | Training | Consultation | Data requirementsb | Practice | Other | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| LR (hours) | AST (hours) | Expenses ($) | LR (hours) | AST (hours) | AST (hours) | Expenses ($) | (AST hours)c | LR (hours) | AST (hours) | Expenses ($) | |
| 1 | 0 | 492.88 | 1,019 | 0 | 846.39 | 127.37 | 0 | 487.34 | 0 | 1021.76 | 3,231 |
| 2 | 62.30 | 49.84 | 415 | 0 | 241.83 | 22.15 | 0 | 575.95 | 143.99 | 0 | 0 |
| 3 | 44.30 | 71.99 | 185 | 354.43 | 4.62 | 14.77 | 0 | 88.61 | 0 | 49.84 | 554 |
| 4 | 0 | 191.98 | 102 | 0 | 304.59 | 27.69 | 0 | 191.98 | 0 | 278.75 | 586 |
| 5 | 27.17 | 0 | 78 | 397.41 | 22.15 | .00 | 0 | 0 | 755.75 | 0 | 2,308 |
| 6 | 39.87 | 407.97 | 171 | 0 | 653.48 | 5.54 | 0 | 48.00 | 0 | 184.60 | 0 |
| 7 | 34.15 | 147.68 | 188 | 341.51 | .00 | 55.38 | 692 | 0 | 213.10 | 22.15 | 462 |
| 8 | 121.21 | 0 | 332 | 141.22 | 11.08 | 18.46 | 0 | 9.23 | 47.07 | 95.99 | 3,079 |
| 9 | 134.39 | 0 | 477 | 0 | 494.73 | 11.08 | 0 | 0 | 372.93 | 0 | 0 |
| 10 | 250.17 | 188.29 | 0 | 614.05 | 104.76 | 34.15 | 1,643 | 365.51 | 73.91 | 33.23 | 277 |
| 11 | 159.49 | 0 | 185 | 521.68 | 479.96 | 116.30 | 0 | 335.97 | 86.39 | 146.94 | 831 |
| 12 | 0 | 227.98 | 0 | .00 | 488.73 | 213.67 | 0 | 82.15 | 0 | 390.66 | 37 |
| 13 | 0 | 465.19 | 601 | .00 | 494.73 | 41.54 | 0 | .00 | 0 | 177.22 | 462 |
| 14 | 415.94 | 0 | 0 | 398.48 | .00 | 40.84 | 0 | 184.60 | 142.88 | 0 | 554 |
| Mean | 92.07 | 160.27 | 26 | 197.77 | 296.22 | 52.07 | 167 | 169.24 | 131.15 | 171.51 | 884 |
| SD | 36.95 | 179.73 | 285 | 264.52 | 282.38 | 60.16 | 463 | 196.63 | 209.61 | 272.15 | 1,125 |
a
TF-CBT, trauma-focused cognitive-behavioral therapy; CMHAs, community mental health agencies
b
Data requirements were not associated with LR.
c
Practice was not associated with LR or expenses.
Agency Productivity Reductions
Only three of 14 CMHAs reported any formal reduction in productivity requirements for TF-CBT staff. That is, although most agencies reported LR hours, this loss was limited to specific one-time activities and was not associated with a regular reduction in billable requirements. Of the three agencies that provided an ongoing reduction in productivity requirements, the reduction was equal to .5 to 1.0 billable hours, or sessions, per week (.83±.29).
Agency Characteristics and Implementation Costs
Exploratory analyses were conducted to examine how agency characteristics (budget, total staff size, and TF-CBT staff size) were related to costs. Table 5 presents bivariate two-tailed Pearson correlation analyses, which revealed that the agency’s budget for outpatient child services was positively correlated with total sustainment costs (sum of AST, LR, and expenses) (r=.61, p<.05). Agencies that provided a standard productivity reduction demonstrated greater total sustainment costs (t=5.87, df=10, p<.01) compared with agencies that did not provide a standard productivity reduction.
| # | Characteristic | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Child outpatient budget | 1.00 | .01 | .04 | .20 | .29 | .29 | .27 | .53 | –.11 | .30 | .18 | .61* |
| 2 | TF-CBT clients | 1.00 | .15 | –.27 | –.03 | –.28 | .36 | .26 | .36 | –.28 | .39 | .10 | |
| 3 | Total staff | 1.00 | .21 | .49 | –.51 | .38 | .14 | –.34 | –.50 | .50 | –.10 | ||
| 4 | Total full- and part-time clinical staff | 1.00 | .07 | .08 | .27 | .39 | –.17 | .08 | .32 | .48 | |||
| 5 | Total TF-CBT clinical staff | 1.00 | –.26 | .16 | .03 | –.11 | –.25 | .12 | –.20 | ||||
| 6 | Total LR hours for TF-CBT | 1.00 | –.61* | –.10 | .09 | .99** | –.71** | .52 | |||||
| 7 | Total AST hours for TF-CBT | 1.00 | .85** | .27 | –.59* | .94** | .32 | ||||||
| 8 | Total LR and AST hours for TF-CBT | 1.00 | .40 | –.07 | .70** | .74** | |||||||
| 9 | Total TF-CBT expenses ($) | 1.00 | .07 | .03 | .19 | ||||||||
| 10 | Total LR costs ($) for TF-CBT | 1.00 | –.70** | .55* | |||||||||
| 11 | Total AST costs for TF-CBT ($) | 1.00 | .23 | ||||||||||
| 12 | Total TF-CBT costs ($)b | 1.00 |
a
CMHAs, community mental health agencies; TF-CBT, trauma-focused cognitive-behavioral therapy; LR, lost revenue; AST, additional staff time
b
Includes total LR costs, total AST costs, and total expenses for TF-CBT
*
p<.05, **p<.01 (two-tailed)
Discussion
This study outlined a novel approach to quantifying incremental costs incurred by CMHAs to sustain an EBP and provided preliminary data to estimate costs. Our results suggest that there was great heterogeneity across agencies in implementation costs ($25,641–$102,976), costs per client ($513 to $4,646), and costs per staff ($1,709 to $8,581). Given the high variability in costs, findings should be interpreted cautiously pending further replication.
Nevertheless, the wide variability in costs provides insight into how agencies absorb costs and into agency efficiency. The positive correlation between agency budget and total costs suggests that larger agencies were better able to absorb costs and may be more likely to invest in sustaining TF-CBT, for example, by training more staff. Nonsignificant correlations between costs and variables such as clients served may be more reflective of agency efficiencies and inefficiencies. For example, agency 13 reported training 23 TF-CBT staff and providing TF-CBT to 83 children, whereas agency 2 reported training 26 staff and providing TF-CBT to far fewer children (N=16). Given the high staff turnover rates in many community mental health settings, it is possible that efforts to prevent having too few clinicians trained in TF-CBT resulted in agencies training too many staff at once or not identifying enough children to receive TF-CBT.
Our preliminary findings highlight another issue related to agency efficiency—variation in LR and AST costs. There was great variation in LR and AST costs within and across agencies, but there was less variability across agencies in total (combined LR and AST) costs. This indicates that there was some consistency in the amount of total time staff spent on the EBP across agencies but there was less consistency in how agencies accounted for this time. Indeed, some CMHAs provided regular or occasional reductions in productivity requirements for activities, whereas others required clinicians to absorb the additional time without such accommodations. The negative correlation between LR and AST is likely explained by this distinction.
Despite wide variability across agencies, all agencies reported significant costs to sustain the EBP, with few agencies reporting any formal reductions in productivity to reduce staff burden. The costliest activities included training, consultation, and “other” activities (primarily for a site coordinator). Although TF-CBT is just one EBP, it is a fairly representative EBP for outpatient children’s mental health treatment in terms of the general activities needed to sustain it. The implications of these variations in how agencies sustain EBPs and what activities are costliest are ripe for future research.
There were several important limitations inherent in the cost estimates in this study. First, retrospective estimates may be less reliable than prospective studies and may be subject to greater measurement error. Prospective cost studies requiring regular logging of activities may provide more accurate estimates, albeit with greater burden on CMHA staff. Second, all estimates were self-reported by staff, who used agency data when possible. Future studies might seek to validate the survey instrument against agency billing data. It is also important to note that agencies in this study received external sustainment support from the statewide TF-CBT Coordinating Center, including the external costs of TF-CBT trainers. Thus, sustaining an EBP without this support could create additional costs for CMHAs, but it could also reduce the number of additional responsibilities related to the EBP, for example, by requiring less use of standardized measures or data entry. Finally, it is possible that demand characteristics of this study led agency leaders to overestimate costs.
An important area for future research is the incorporation of implementation costs into comprehensive cost-effectiveness models. These future studies should incorporate potential cost savings not included in this study, such as reduced no-show rates, less staff turnover, and reduced length of treatment compared with usual care. Furthermore, more research is needed in the area of understanding usual care practices in order to help interpret incremental costs of the EBP. One of the challenges with community-based psychotherapy research is that usual care varies greatly and is difficult to define (18). Future replication studies could employ mixed-methods designs to quantitatively account for incremental costs and savings of sustaining an EBP and qualitatively explore usual care practices and cost variations across agencies.
Conclusions
Results suggest that reimbursement models for supporting EBPs must consider the costs of sustainment. When this study was conducted, there was no difference between the reimbursement rate for TF-CBT and usual care. The lack of commensurate staff productivity reductions to alleviate the burden of sustaining the EBP is concerning because it places agencies in a difficult position of choosing to implement and sustain an EBP while incurring staff and agency burden.
Policies to incentivize EBPs over usual care may increase the proliferation of EBPs and high-quality care, potentially saving money in the long term (17). Although costs to sustain an EBP may be substantial, there may be savings that accrue to other actors in the system, for example, fewer emergency department visits, that could be allocated to support the EBP. Another implication of these findings is the need to develop innovations in EBPs and implementation strategies that can reduce costs. For example, development of components-based EBPs that are appropriate for a range of clinical concerns is likely to create efficiencies (19). Alternatively, development of less complex EBPs that are simpler to implement and require less training and consultation time would reduce costs. Ultimately, a public health approach to improving the quality of care will require balancing the efficacy of EBPs with the costs required to implement and sustain them across a system of care.
Acknowledgments
The authors acknowledge the Connecticut Department of Children and Families and the many staff from the community provider agencies that participated.
References
1.
Hoagwood KE, Olin SS, Horwitz S, et al: Scaling up evidence-based practices for children and families in New York State: toward evidence-based policies on implementation for state mental health systems. Journal of Clinical Child and Adolescent Psychology 43:145–157, 2014
2.
Novins DK, Green AE, Legha RK, et al: Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. Journal of the American Academy of Child and Adolescent Psychiatry 52:1009–1025, 2013
3.
Proctor EK, Powell BJ, McMillen JC: Implementation strategies: recommendations for specifying and reporting. Implementation Science 8:139, 2013
4.
Bond GR, Drake RE, McHugo GJ, et al: Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research 41:228–236, 2014
5.
Lee S, Aos S, Pennucci A: What Works and What Does Not? Benefit-Cost Findings From WSIPP. Olympia, Washington State Institute for Public Policy, 2015
6.
Aarons GA, Wells RS, Zagursky K, et al: Implementing evidence-based practice in community mental health agencies: a multiple stakeholder analysis. American Journal of Public Health 99:2087–2095, 2009
7.
Proctor E, Silmere H, Raghavan R, et al: Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research 38:65–76, 2011
8.
Raghavan R, Bright CL, Shadoin AL: Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation Science 3:26, 2008
9.
Sigel BA, Benton AH, Lynch CE, et al: Characteristics of 17 statewide initiatives to disseminate trauma-focused cognitive-behavioral therapy (TF-CBT). Psychological Trauma: Theory, Research, Practice, and Policy 5:323–333, 2013
10.
Lang JM, Connell CM: Measuring costs to community-based agencies for implementation of an evidence-based practice. Journal of Behavioral Health Services and Research 44:122–134, 2017
11.
Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research 38:4–23, 2011
12.
Glasgow RE, Klesges LM, Dzewaltowski DA, et al: Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Education Research 21:688–694, 2006
13.
Lang JM, Franks RP, Epstein C, et al: Statewide dissemination of an evidence-based practice using Breakthrough Series Collaboratives. Children and Youth Services Review 55:201–209, 2015
14.
Cohen JA, Mannarino AP, Deblinger E: Treating Trauma and Traumatic Grief in Children and Adolescents. New York, Guilford, 2006
15.
Cary CE, McMillen JC: The data behind the dissemination: a systematic review of trauma-focused cognitive behavioral therapy for use with children and youth. Children and Youth Services Review 34:748–757, 2012
16.
Kilo CM: A framework for collaborative improvement: lessons from the Institute for Healthcare Improvement’s Breakthrough Series. Quality Management in Health Care 6:1–13, 1998
17.
Lang JM, Campbell K, Shanley P, et al: Building capacity for trauma-informed care in the child welfare system: initial results of a statewide implementation. Child Maltreatment 21:113–124, 2016
18.
Garland AF, Hurlburt MS, Brookman-Frazee L, et al: Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research 37:208–220, 2010
19.
Weisz JR, Chorpita BF, Palinkas LA, et al: Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: a randomized effectiveness trial. Archives of General Psychiatry 69:274–282, 2012
Information & Authors
Information
Published In
Cover: Glowing Night, by Oscar Bluemner, 1924. Watercolor and pencil on paper. Bequest of Charles F. Ikle, 1963. © The Metropolitan Museum of Art, New York City. Image source: Art Resource, New York City.
History
Received: 26 April 2016
Revision received: 14 October 2016
Accepted: 23 February 2017
Published online: 1 May 2017
Published in print: September 01, 2017
Keywords
Authors
Competing Interests
The authors report no financial relationships with commercial interests.
Funding Information
National Institute of Mental Health10.13039/100000025: 2T32MH018261-31
Ford Foundation Postdoctoral Fellowship:
Manuscript preparation by Dr. Roundfield was partially supported by a National Institute of Mental Health postdoctoral training grant (2T32MH018261) and a Ford Foundation postdoctoral fellowship.Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).