Factors Associated With Caregiver Attendance in Implementation of Multiple Evidence-Based Practices in Youth Mental Health Services
Abstract
Objective:
The implementation of evidence-based practices (EBPs) in community mental health settings for youths has consistently yielded weakened effects compared with controlled trials. There is a need to feasibly measure the quality of large-scale implementation efforts to inform improvement targets. This study used therapist-reported caregiver attendance in treatment sessions as a quality indicator in the community implementation of EBPs.
Methods:
Data were collected from therapists practicing in agencies contracted to provide publicly funded children’s mental health services following a system-driven implementation of multiple EBPs. Community therapists (N=101) provided information about youth clients (N=267) and psychotherapy sessions (N=685). Multivariable binomial logistic regressions were conducted to examine associations between caregiver attendance and therapist factors (e.g., licensure status, education), youth factors (e.g., gender, age), and the type of EBP delivered.
Results:
Caregiver attendance occurred in 42% of sessions. The following factors were associated with increased odds of caregiver attendance: younger client age, male sex of client, externalizing presenting problem, and delivery of an EBP that prescribes caregiver attendance at all sessions. Caregiver attendance at sessions targeting trauma or externalizing disorders appeared to explain the differences between boys and girls in levels of caregiver engagement.
Conclusions:
Overall, the patterns of actual caregiver attendance appeared consistent with empirically informed practice parameters for involvement of caregivers in treatment. Still, the rates of caregiver attendance in externalizing-focused sessions were suboptimal, and the gender difference in these rates—which clearly disfavored girls—suggests targeted areas for quality improvement. Potential reasons for these quality gaps are discussed.
HIGHLIGHTS
•
Caregiver attendance is a quality indicator for youth mental health treatment.
•
Caregiver attendance occurred in 288 (42%) sessions in which an evidence-based practice (EBP) was delivered within publicly funded youth mental health services.
•
Study findings indicate potential for quality improvement in EBP delivery for girls and for children being treated for externalizing problems.
Publicly funded mental health systems have enacted policies supporting the implementation of evidence-based practices (EBPs) as a strategy to improve the quality of care for the estimated 20% of youths with a mental disorder (1, 2). Despite these efforts, the effects of EBPs are consistently weaker when implemented in the community compared with their effects in controlled trials (3). This “implementation cliff” is explained chiefly by differences between experimental and usual care conditions, for example, the greater complexity of client profiles and the decreased intervention integrity in usual care (4, 5). Given the variability in success of EBP implementation, monitoring quality indicators to identify improvement targets is essential. To address the “quality chasm,” the Institute of Medicine has called for federal agencies to develop indicators grounded in the evidence base (6). Previously studied quality indicators include patient satisfaction and continuity of care following hospitalization (7). Yet the National Committee on Quality Assurance reported that neither the scope nor the design of extant indicators is sufficient to optimally measure quality of care (8).
Current quality monitoring tools focus largely on detailed measures of adherence to multicomponent therapy protocols. These measures index the extent to which providers deliver treatment elements as tested in controlled trials (9). Such assessments require laborious data collection and review, often involving observer ratings. Large-scale EBP implementation requires quality indicators that are relevant to a wide range of EBPs and that can be feasibly obtained at low cost and with minimal burden to the workforce (10, 11).
Caregiver attendance is an empirically informed, feasibly assessed quality indicator (12, 13) that can help gauge the quality of care within large-scale EBP delivery in youth mental health services. Indeed, caregiver attendance is a relevant quality indicator in publicly funded services, given that caregivers are often considered the primary agents of behavior change across many treatments for youths (14–16). Caregivers help reinforce skills and generalize desired youth behaviors to new settings (17). Moreover, participation by caregivers in treatment is linked to improved clinical outcomes (12).
There are three strong indications for caregiver attendance. First, systematic reviews list parent training as a frontline treatment for externalizing behaviors, including disruptive behavior and attention-deficit hyperactivity disorder (ADHD) (18–20). In contrast, meta-analyses and reviews show inconsistent evidence of incremental benefits of caregiver treatment participation for child internalizing disorders (21–24). Second, because early intervention generally produces the largest effects in preventing behavioral problems (25), children ages 12 and under are more likely than adolescents to benefit from caregiver treatment participation (19, 20). Third, caregiver attendance is a critical quality indicator in the delivery of EBPs that prescribe caregiver participation. Many EBPs for various disorders have been explicitly designed to include caregivers to drive therapeutic change or to improve caregiver-child relationships as the primary mechanism of change. Examples include attachment-based treatments for trauma (26).
Although caregiver attendance is not sufficient for high-quality implementation of an EBP, it is a necessary precondition to deliver EBPs that require caregiver involvement. Without caregiver attendance, a provider could not deliver caregiver-directed components, thereby compromising EBP integrity profoundly. Thus caregiver attendance should be considered a quality indicator in youth mental health treatment sessions in which a child ages 12 or under presents with an externalizing behavior problem or a therapist delivers an EBP that prescribes caregiver participation.
The complexity of translating research innovations into practice has spurred the examination of multilevel factors that are related to implementation outcomes. Provider-level factors—such as educational attainment, training background, and experience—have been linked to EBP implementation quality (27). Importantly, positive attitudes by therapists toward EBPs are associated with use of EBP posttraining and fewer self-reported modifications that reduce EBP content (28, 29). At the client level, studies demonstrate that compared with boys, girls are less likely to obtain needed treatment for mental health problems, especially externalizing problems (30, 31). The continued examination of therapist and youth factors in EBP implementation can identify quality improvement targets that could become areas of focus in efforts to scale up EBPs.
In this study, we examined caregiver attendance as a quality indicator in psychotherapy sessions delivered within youth mental health services in Los Angeles County, the nation’s largest county mental health system. The study addressed two research questions: In the large-scale implementation of multiple EBPs, does preadolescent age, externalizing problems, or type of EBP delivered predict the occurrence of caregiver attendance? Within sessions in which caregiver attendance is indicated, are therapist or youth factors associated with significant differences in caregiver attendance?
Methods
Current Study Context
Data were collected as part of the Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) study, which examined determinants of EBP sustainment following a system reform by the Los Angeles County Department of Mental Health (LACDMH) (32). In 2009, LACDMH enacted the Prevention and Early Intervention (PEI) transformation, which utilizes a state revenue stream funded by a voter-approved ballot initiative to promote EBP uptake. LACDMH offers reimbursement to community agencies for the delivery of EBPs from an approved list and facilitated the rapid scale-up of six EBPs (33). This study examined caregiver attendance in sessions in which one of the EBPs was delivered (child-parent psychotherapy [CPP], Positive Parenting Program [Triple P], trauma-focused cognitive-behavioral therapy [TF-CBT], Managing and Adapting Practices [MAP], and Seeking Safety; although LACDMH provided implementation support for Cognitive-Behavioral Intervention for Trauma in Schools, community therapists in this sample did not report on their delivery of this EBP.
CPP addresses trauma in children ages 6 and under, primarily with conjoint child-caregiver sessions, although some sessions are caregiver focused (26). Like CPP, Triple P also requires caregiver attendance in all sessions; it targets conduct problems among youths ages 17 and under (34). TF-CBT addresses trauma among youths ages 3 to 18. Optimal implementation of TF-CBT prescribes caregiver participation in every session, although it can be delivered effectively without a caregiver participant (35). Not considered a standalone EBP, MAP is a system of decision support tools that guide evidence-based treatment planning and delivery; therapists can select practice elements for anxiety, depression, conduct problems, and trauma for individuals ages 21 and under (36). Although most MAP practice elements that target conduct problems are caregiver directed, some strategies may be exclusively youth-directed (e.g., problem-solving skills training). Seeking Safety is youth directed and targets trauma and substance use in individuals ages 3 to 20 (37).
This study examined EBP implementation as usual and did not assess therapists’ specific training histories. The EBP training requirements established by LACDMH in collaboration with EBP developers are briefly described elsewhere (38–41) and are described in detail in the 2016 PEI Implementation Handbook; these guidelines were active during the time of data collection for this study (42). Generally speaking, after the initial training, Seeking Safety and Triple P do not require ongoing consultation, but CPP, MAP, and TF-CBT require consultation at varying lengths and formats. CPP requires 18 months of consultation calls with external trainers. MAP requires 6 months of ongoing consultation, which can be provided by a certified external or internal trainer; MAP therapists also submit case-based portfolios for certification. TF-CBT requires 16 consultation calls with external trainers along with audiotape review of two sessions.
Sample
Therapist participants worked across 14 LACDMH-contracted agencies. Eligible therapists delivered one of the five EBPs of interest to a youth client. The sample included 101 therapists who delivered care to 267 youth clients. Most therapists were female (N=89, 88%), had a master’s degree (N=86, 85%), and were not yet licensed (N=82, 81%); the mean±SD age was 33.9±8.7. The sample was racially and ethnically diverse, with 56 (55%) identifying as Latinx, 22 (22%) as white, and 23 (23%) as other (e.g., Asian, African American); 53 (52%) reported being able to deliver treatment in Spanish. Data were collected on 685 sessions, with each therapist contributing an average of 6.78 sessions and an average of 2.57 sessions per client.
Procedure
All procedures were approved by the institutional review boards at the University of California, San Diego; University of California, Los Angeles; and LACDMH. Data collection occurred between 2015 and 2017. Informed consent was obtained at recruitment meetings held during staff meetings at each community mental health agency. Therapists completed a baseline survey about their background and perspectives on EBPs and submitted questionnaires describing their delivery of EBPs for three sessions for up to three clients. Therapists obtained permission from caregivers for session recording, but no identifying information was collected about clients or caregivers beyond basic demographic characteristics and presenting problem. Therapists received $20 for completion of the survey and $5 for completion of each session questionnaire.
Measures
Therapist characteristics.
Therapists reported their personal characteristics (age, gender, and race-ethnicity) and professional history (licensure status, highest degree obtained, ability to deliver services in a language other than English, and caseload information) on the Therapist Background Questionnaire (43).
Perceived Characteristics of Intervention Scale (PCIS).
An adapted version of the PCIS was used to measure therapist perceptions about each EBP that they delivered (44). To mitigate participant burden, the 20 original PCIS items were pared to eight, including items measuring perceptions related to relative advantage (e.g., “[The EBP] is more effective than other therapies I have used”), compatibility (e.g., “[The EBP] is aligned with my clinical judgment”), complexity (e.g., “[The EBP] is easy to use”), and potential for reinvention (e.g., “[The EBP] can be adapted to meet the needs of my patients”). Items were rated on a 5-point Likert scale, from 1 (not at all), to 5 (a very great extent); a higher mean score reflected more positive views of the EBP. The eight-item scale had excellent internal consistency for all EBPs (Cronbach’s α=.92–.96).
Session Questionnaire.
For each session, therapists reported client demographic characteristics (gender, race-ethnicity, and age), targeted presenting problem, language used in session, and the EBP delivered.
EBP and expected caregiver attendance.
Based on a review of intervention materials, EBPs were categorized according to expected level of caregiver attendance. EBPs were classified into the following three-level ordinal scale: caregiver attendance not routinely expected (Seeking Safety and MAP targeting nonconduct problems [MAP-other]); caregiver attendance expected in at least some sessions (TF-CBT and MAP targeting conduct problems [MAP-conduct]); and caregiver attendance expected in every session (Triple P and CPP).
Sessions in which caregiver attendance was indicated.
On the basis of the evidence base, we identified sessions in which caregiver attendance was indicated as those in which the child was age 12 or under and the reported presenting problem was disruptive behavior or ADHD or those that involved the delivery of TF-CBT, MAP-conduct, Triple P, or CPP.
Caregiver attendance.
Caregiver attendance was a dichotomous, session-level variable based on therapists’ report of who attended the session. Therapists provided responses to, “Who was involved in the session on which you are responding?” Caregiver attendance was considered to occur if the session was attended by only a collateral caregiver or by both a caregiver and a youth.
Statistical Analysis
Multivariable binomial logistic regressions that accounted for a three-level data structure were conducted in Stata, version 15.1; sessions were nested within youth clients, who were nested within therapists (45). To address research question 1, analyses were conducted in the full sample to examine the relationship between caregiver attendance and the youth’s age, presenting problem, and caregiver attendance expected for the EBP (no sessions, some sessions, or all sessions); youth- and therapist-level covariates were included. To address research question 2, the model was conducted with the subsample of sessions involving an EBP in which caregiver attendance is indicated, with the goal of examining differences in caregiver attendance by youth and therapist attributes.
Results
Table 1 displays characteristics of the youths and sessions for the 685 sessions in which an EBP was provided. Table 2 shows overall rate of caregiver attendance and how caregiver attendance varied by client age, gender, presenting problem, and EBP type in the full sample of sessions and in the sessions in which caregiver attendance was indicated. Caregiver attendance occurred in 42% of sessions in the full sample and in 49% of sessions in which caregiver attendance was indicated. The rate of caregiver attendance was higher in sessions with children ages 12 and under compared with sessions with adolescents (13 and over). Caregiver attendance occurred more often in sessions with boys than with girls. In the full sample, caregiver attendance occurred in 53% of sessions targeting an externalizing problem, 45% of sessions targeting trauma, and 29% of sessions targeting internalizing problems. Caregiver attendance also varied by EBP, ranging from 28% of Seeking Safety and MAP-other sessions to 99% of Triple P and CPP sessions.
| All EBP sessions | Sessions in which caregiver attendance was expected | |||
|---|---|---|---|---|
| Characteristic | N | % | N | % |
| Clientb | ||||
| Age | ||||
| ≤12 | 201 | 75 | 161 | 84 |
| ≥13 | 66 | 25 | 30 | 16 |
| Gender | ||||
| Female | 136 | 51 | 85 | 45 |
| Male | 131 | 49 | 106 | 55 |
| Race-ethnicity | ||||
| Latinx | 186 | 70 | 131 | 69 |
| Otherc | 68 | 25 | 51 | 27 |
| Non-Hispanic white | 13 | 5 | 9 | 5 |
| Sessiond | ||||
| Language spoken | ||||
| English | 554 | 80 | 366 | 77 |
| Spanish | 120 | 18 | 102 | 21 |
| Other | 11 | 2 | 7 | 2 |
| Presenting problem | ||||
| Internalizing | 256 | 37 | 65 | 14 |
| Externalizing | 228 | 33 | 220 | 46 |
| Trauma | 188 | 27 | 178 | 37 |
| Other (e.g., autism, reactive attachment disorder) | 13 | 3 | 12 | 3 |
| Evidence-based practice delivered | ||||
| Child-parent psychotherapy | 49 | 7 | 49 | 10 |
| Positive Parenting Program | 38 | 6 | 38 | 8 |
| Trauma-focused cognitive behavioral therapy | 210 | 30 | 210 | 45 |
| Managing and Adaptive Practice (MAP)–conduct | 158 | 23 | 158 | 33 |
| MAP-other | 203 | 30 | 20 | 4 |
| Seeking Safety | 27 | 4 | 0 | — |
a
Caregiver attendance was expected in at least some sessions of child-parent psychotherapy and the Positive Parenting Program and in all sessions of trauma-focused cognitive-behavioral therapy and Managing and Adaptive Practice (MAP)–conduct and was not routinely expected in sessions of MAP-other and Seeking Safety.
b
Caregiver attendance was expected in sessions involving 191 of 267 clients.
c
Includes African Americans, Asian/Pacific Islanders, and persons of mixed race.
d
Caregiver attendance was expected in 475 of 685 sessions.
| All EBP sessions | EBP sessions in which caregiver attendance was expected | |||||
|---|---|---|---|---|---|---|
| Total | Attended by caregiver | Total | Attended by caregiver | |||
| Characteristic | N | N | % | N | N | % |
| Total | 685 | 288 | 42 | 475 | 233 | 49 |
| Client age | ||||||
| ≤12 | 519 | 255 | 49 | 400 | 215 | 54 |
| ≥13 | 166 | 33 | 20 | 75 | 18 | 24 |
| Client gender | ||||||
| Female | 355 | 110 | 31 | 224 | 79 | 35 |
| Male | 330 | 178 | 54 | 251 | 154 | 61 |
| Presenting problem | ||||||
| Internalizing | 256 | 73 | 29 | 65 | 21 | 32 |
| Externalizing | 228 | 120 | 53 | 220 | 119 | 54 |
| Trauma | 188 | 85 | 45 | 178 | 83 | 47 |
| Other (e.g., autism, reactive attachment disorder) | 13 | 10 | 77 | 12 | 10 | 83 |
| Expectation of caregiver attendance at EBP session | ||||||
| Not routinely expecteda | 230 | 64 | 28 | 20 | 9 | 45 |
| Expected at some sessionsb | 368 | 138 | 38 | 368 | 138 | 38 |
| Expected at all sessionsc | 87 | 86 | 99 | 87 | 86 | 99 |
a
Seeking Safety and Managing and Adaptive Practice (MAP)–other.
b
MAP-conduct and trauma-focused cognitive-behavioral therapy.
c
Child-parent psychotherapy and the Positive Parenting Program.
Predictors of Caregiver Attendance
Table 3 presents adjusted coefficients (b) or odds ratios (ORs) and 95% confidence intervals (CIs) for the full-sample model. Results showed significant associations between caregiver attendance and youths’ age, youths’ gender, and the type of EBP delivered. The odds of caregiver attendance were higher for sessions with children ages 12 and under compared with adolescents (OR=3.67, CI=1.53–8.84, p<.01); for sessions with boys compared with girls (OR=2.12, CI=1.08–4.16, p<.05); and for sessions with Triple P and CPP compared with Seeking Safety and MAP-other (OR=988.31, CI=45.52–21,459.11, p<.01). The other therapist and youth factors were nonsignificant.
| All EBP sessions | EBP sessions in which caregiver attendance was expected | |||||
|---|---|---|---|---|---|---|
| Predictor | OR | 95% CI | p | OR | 95% CI | p |
| Therapist race-ethnicity (reference: Latinx) | ||||||
| Non-Hispanic white | 1.15 | .33–4.03 | .83 | 1.24 | .29–5.31 | .78 |
| Othera | .51 | .12–2.19 | .37 | .76 | .13–4.37 | .76 |
| Therapist can treat in a non-English language (reference: English only) | ||||||
| Spanish | 1.73 | .52–5.75 | .37 | 2.54 | .59–11.03 | .21 |
| Other | 1.09 | .20–5.89 | .92 | .87 | .12–6.14 | .87 |
| Therapist educational background (reference: master’s degree) | ||||||
| Doctoral degree | 3.03 | .70–13.13 | .14 | 2.52 | .45–14.19 | .30 |
| Less than master’s degree | .46 | .02–11.37 | .64 | .19 | .003–11.34 | .43 |
| Licensed therapist (reference: not licensed) | 1.24 | .40–3.78 | .71 | 1.11 | .31–4.07 | .87 |
| N of therapist’s direct service hours (b) | –.02 | –.09 to .05 | .60 | –.005 | –.09 to .08 | .90 |
| N of therapist’s work travel hours (b) | .07 | –.09 to .24 | .40 | .12 | –.09 to .32 | .26 |
| Therapist attitudes toward EBP delivered during session (b) | .02 | –.50 to .53 | .95 | .28 | –.33 to .91 | .37 |
| Client age ≤12 (reference: ≥13) | 3.67 | 1.53–8.84 | .004 | 4.10 | 1.30–12.88 | .02 |
| Male client (reference: female)b | 2.12 | 1.08–4.16 | .03 | 3.06 | 1.39–6.75 | .006 |
| Client race-ethnicity (reference: Latinx) | ||||||
| Non-Hispanic white | .86 | .21–3.46 | .83 | .96 | .18–5.17 | .96 |
| Othera | 1.68 | .64–4.43 | .30 | 1.04 | .34–3.15 | .94 |
| Expectation of caregiver attendance at EBP session (reference: not routinely expectedc) | ||||||
| Expected at some sessionsd | 1.04 | .42–2.58 | .93 | 2.76 | .54–14.18 | .22 |
| Expected at all sessionse | 988.31 | 45.52–21,459.11 | <.001 | 2,545.52 | 51.68–125,392.40 | <.001 |
| Type of presenting problem (reference: internalizing) | ||||||
| Externalizing | 1.81 | .73–4.48 | .20 | 4.21 | 1.22–14.61 | .02 |
| Trauma | 1.13 | .43–2.95 | .81 | 2.01 | .64–6.36 | .23 |
| Other | 1.40 | .13–14.40 | .78 | 6.66 | .34–130.10 | .21 |
a
Includes African Americans, Asian/Pacific Islanders, and persons of mixed race.
b
Post-hoc chi-square tests of independence were conducted to examine gender differences in caregiver attendance by presenting problem type. The differences in caregiver attendance between boys and girls appeared to be driven by caregiver attendance at sessions targeting trauma or externalizing disorders.
c
Seeking Safety and Managing and Adaptive Practice (MAP)–other.
d
MAP-conduct and trauma-focused cognitive-behavioral therapy.
e
Child-parent psychotherapy and the Positive Parenting Program.
Determinants of Caregiver Attendance at Indicated Sessions
Youths’ age, youths’ gender, EBP type, and presenting problem showed a significant association with caregiver attendance at sessions involving an EBP with an expectation of at least some caregiver attendance (Table 3). Similar to full-sample results, the odds of caregiver attendance were higher in sessions with children compared with adolescents (OR=4.10, 95% CI=1.30–12.88, p<.05) and with boys compared with girls (OR=3.06, CI=1.39–6.75, p<.01). Once again, sessions in which Triple P or CPP was delivered had increased odds of caregiver attendance compared with sessions in which Seeking Safety or MAP-other was delivered (OR=2,545.52, 95% CI=51.68–125,392.40, p<.001). Sessions targeting externalizing problems had increased odds of caregiver attendance compared with those focusing on internalizing problems (OR=4.21, 95% CI=1.22–14.61, p<.05). Caregiver attendance was not significantly associated with other therapist factors and youths’ race-ethnicity.
Discussion
This study evaluated one quality indicator, caregiver attendance, in the delivery of multiple EBPs within a systemwide implementation effort. The analyses consistently demonstrated that sessions with younger children were associated with increased odds of caregiver attendance compared with sessions with adolescents. From a developmental perspective, this finding is not surprising, given that treatment with young children tends to involve caregivers (46). Moreover, adolescents may decline caregiver involvement because of feelings of discomfort when discussing emotional topics or because the nature of the parent-child relationship contributes to their presenting problem (47). Additionally, the delivery of Triple P and CPP, which prescribe caregiver attendance at every session, demonstrated increased odds of caregiver attendance compared with practices in which caregiver attendance is not routine. This finding is a promising indicator for the implementation of caregiver-mediated EBPs because it suggests some success in engaging caregivers.
Results show that few therapist and youth factors were associated with significant differences in the occurrence of caregiver attendance, with the exception of youths’ gender. Sessions with boys had increased odds of caregiver attendance in both models; this gender disparity was most pronounced in sessions targeting externalizing disorders and trauma. Haine-Schlagel and colleagues (48) previously found that community therapists were more likely to involve caregivers in the treatment of children with disruptive behavior if the patient was a boy. More generally, studies show that girls with externalizing problems are less likely to obtain necessary treatment (30, 31). The benefits of caregiver treatment participation for child trauma are unclear (49), and no previous studies document gender differences in caregiver attendance for trauma. However, girls may be less likely than boys to have caregivers who are available to support trauma treatment and who are themselves not a perpetrator, given that girls are more likely than boys to come from families where the parents are the perpetrators of abuse (50).
This study cannot narrow explanations for this observed gender disparity in caregiver attendance, which may be driven by therapist or youth and family factors. Given that most therapists were female, they may perceive a stronger alliance with girls, and view caregiver attendance as less necessary for treatment progress. Indeed, gender-matched therapist-youth dyads have reported stronger alliance than gender-mismatched dyads (51). Therapists may also expect productive individual therapy with girls based on gendered norms of boys being comparatively less expressive (52). Considering that the youths in our sample were predominantly Latinx, cultural factors may also shape this gender disparity. Traditionally, more restrictions are placed on girls in Latinx families; Latinx caregivers may be more motivated to participate in the treatment of sons because they are less equipped to set limits on boys’ problem behaviors (53). Although the explanation is unclear, the implication of the finding is that quality improvement efforts should focus on reducing the gender disparity in caregiver attendance. Clinical supervisors could monitor caregiver attendance, discuss how symptoms may manifest differently by gender and culture, and encourage caregiver inclusion through psychoeducation when it is indicated (54).
Overall, caregiver attendance occurred in 42% of all sessions and at only a slightly greater rate (49%) in sessions in which it was expected. This rate compares unfavorably to another community study, which found caregiver attendance in 70% of sessions with predominantly non-Hispanic white youths presenting with externalizing problems in usual care (55). The lower rate of caregiver attendance observed may be due to our majority Latinx sample, which is a population that faces increased barriers to treatment, such as a shortage of bilingual therapists (56). In addition, for EBPs in which at least some caregiver attendance was indicated, sessions involving externalizing problems had increased odds of caregiver attendance compared with sessions involving internalizing problems. However, the lack of association between caregiver attendance and externalizing problems in the full sample is concerning and requires further study.
It is unclear what accounts for suboptimal levels of caregiver attendance. Factors that may reduce caregiver attendance in community services are the inclusion of child welfare–involved youths who are not in the care of a long-term primary caregiver and low caregiver motivation (12, 16, 57). Another possibility is that system-driven EBP implementation has inadvertently shaped services. An unintended consequence of implementation is that the EBPs adopted by the agency and provider tend to dictate the content and form of services perhaps to a greater extent than the child’s presenting problem (58).
Altogether, the lower than optimal rates of caregiver attendance observed in this study are a reminder of the challenges of caregiver engagement in youth community treatment. These and other findings suggest the need for caregiver engagement interventions, which have gained increasing attention in the field (59). Tracking caregiver attendance as a quality indicator can help align community practice with evidence-based recommendations for caregiver treatment participation and can help to identify children who could benefit from caregiver engagement strategies. The results may have implications for the adoption of quality assurance procedures. For example, an agency can conduct a rapid quality assessment by collecting data on caregiver attendance through routine audits of billing for family or collateral services for youths; if rates of caregiver attendance are suboptimal for services with indications for caregiver engagement, focal quality improvement efforts could work to pinpoint reasons for low penetration of caregiver attendance. Clinical supervisors and providers can then attempt to resolve the problems that have circumvented efforts to involve caregivers.
This study adds to the literature examining caregiver engagement as a quality indicator; however, the findings should be considered in light of limitations (14, 16). First, caregiver engagement is only a proxy of treatment engagement, and assessment of the quality of in-session caregiver participation is needed (54). Second, it is unknown whether caregiver engagement was driven by youth, caregiver, or therapist preferences or decisions. For therapists, data were not collected on ongoing engagement in training activities, such as fidelity monitoring, which could have influenced caregiver engagement. Third, our sample included an overrepresentation of unlicensed therapists; thus it may not generalize to a more experienced workforce. Last, because of the client sample composition, we were unable to explore racial-ethnic disparities; further research is needed in this domain.
Conclusions
Measurement of caregiver engagement can be utilized as a pragmatic quality indicator to help identify quality gaps in EBP community implementation. Study findings suggest the need for improvement in the quality of treatment for girls and youths being treated for externalizing problems. Monitoring the quality of care remains essential, given that implementation of EBPs is likely to remain a priority of publicly funded behavioral health system reform.
Footnote
Preliminary results from the data were included in a poster presentation at the Association of Behavioral and Cognitive Therapies convention, San Diego, November 16–19, 2017.
References
1.
Rubin RM, Hurford MO, Hadley T, et al: Synchronizing watches: the challenge of aligning implementation science and public systems. Adm Policy Ment Health Ment Health Serv Res 2016; 43:1023–1028
2.
Gruttadaro D, Burns B, Duckworth K, et al: Choosing the Right Treatment: What Families Need to Know About Evidence-Based Practices. Arlington, VA, National Alliance on Mental Illness, 2007
3.
Weisz JR, Jensen-Doss A, Hawley KM: Evidence-based youth psychotherapies versus usual clinical care: a meta-analysis of direct comparisons. Am Psychol 2006; 61:671–689
4.
Weisz JR, Ugueto AM, Cheron DM, et al: Evidence-based youth psychotherapy in the mental health ecosystem. J Clin Child Adolesc Psychol 2013; 42:274–286
5.
McLeod BD, Southam-Gerow MA, Tully CB, et al: Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clin Psychol (New York) 2013; 20:14–32
6.
Institute of Medicine: Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC, National Academies Press, 2006
7.
Garland AF, Bickman L, Chorpita BF: Change what? Identifying quality improvement targets by investigating usual mental health care. Adm Policy Ment Health Ment Health Serv Res 2010; 37:15–26
8.
Schoenbaum SC, Holmgren AL: The National Committee for Quality Assurance’s The State of Health Care Quality 2006. New York, Commonwealth Fund, 2006
9.
Schoenwald SK, Garland AF, Chapman JE, et al: Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health Ment Health Serv Res 2011; 38:32–43
10.
Levin ME, Potts SA: Quality improvement and clinical psychological science; in Quality Improvement in Behavioral Health. Edited by O’Donohue W, Maragakis A. Heidelberg, Germany, Springer, 2016
11.
Sedlar G, Bruns EJ, Walker SC, et al: Developing a quality assurance system for multiple evidence based practices in a statewide service improvement initiative. Adm Policy Ment Health Ment Health Serv Res 2017; 44:29–41
12.
Haine-Schlagel R, Walsh NE: A review of parent participation engagement in child and family mental health treatment. Clin Child Fam Psychol Rev 2015; 18:133–150
13.
Becker M, Breuing J, Nothacker M, et al: Guideline-based quality indicators—a systematic comparison of German and international clinical practice guidelines: protocol for a systematic review. Syst Rev 2018; 7:5
14.
Zima BT, Hurlburt MS, Knapp P, et al: Quality of publicly-funded outpatient specialty mental health care for common childhood psychiatric disorders in California. J Am Acad Child Adolesc Psychiatry 2005; 44:130–144
15.
Noser K, Bickman L: Quality indicators of children’s mental health services: do they predict improved client outcomes? J Emot Behav Disord 2000; 8:9–18
16.
Fawley-King K, Haine-Schlagel R, Trask EV, et al: Caregiver participation in community-based mental health services for children receiving outpatient care. J Behav Health Serv Res 2013; 40:180–190
17.
Dowell KA, Ogles BM: The effects of parent participation on child psychotherapy outcome: a meta-analytic review. J Clin Child Adolesc Psychol 2010; 39:151–162
18.
McCart MR, Sheidow AJ: Evidence-based psychosocial treatments for adolescents with disruptive behavior. J Clin Child Adolesc Psychol 2016; 45:529–563
19.
Evans SW, Owens JS, Wymbs BT, et al: Evidence-based psychosocial treatments for children and adolescents with attention deficit/hyperactivity disorder. J Clin Child Adolesc Psychol 2018; 47:157–198
20.
Kaminski JW, Claussen AH: Evidence base update for psychosocial treatments for disruptive behaviors in children. J Clin Child Adolesc Psychol 2017; 46:477–499
21.
Sun M, Rith-Najarian LR, Williamson TJ, et al: Treatment features associated with youth cognitive behavioral therapy follow-up effects for internalizing disorders: a meta-analysis. J Clin Child Adolesc Psychol 2019; 48(suppl 1):S269–S283
22.
Ishikawa S, Okajima I, Matsuoka H, et al: Cognitive behavioural therapy for anxiety disorders in children and adolescents: a meta-analysis. Child Adolesc Ment Health 2007; 12:164–172
23.
Weersing VR, Jeffreys M, Do MT, et al: Evidence base update of psychosocial treatments for child and adolescent depression. J Clin Child Adolesc Psychol 2017; 46:11–43
24.
Breinholst S, Esbjørn BH, Reinholdt-Dunne ML, et al: CBT for the treatment of child anxiety disorders: a review of why parental involvement has not enhanced outcomes. J Anxiety Disord 2012; 26:416–424
25.
Dishion TJ, Shaw D, Connell A, et al: The family check-up with high-risk indigent families: preventing problem behavior by increasing parents’ positive behavior support in early childhood. Child Dev 2008; 79:1395–1414
26.
Lieberman AF: Child-parent psychotherapy: a relationship-based approach to the treatment of mental health disorders in infancy and early childhood, in Treating Parent-Infant Relationship Problems: Strategies for Intervention. Edited by Sameroff AJ, McDonough SC, Rosenblum KL. New York, Guilford Press, 2004
27.
Brookman-Frazee L, Haine RA, Baker-Ericzén M, et al: Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Adm Policy Ment Health Ment Health Serv Res 2010; 37:254–269
28.
Reding MEJ, Chorpita BF, Lau AS, et al: Providers’ attitudes toward evidence-based practices: is it just about providers, or do practices matter, too? Adm Policy Ment Health Ment Health Serv Res 2014; 41:767–776
29.
Lau A, Barnett M, Stadnick N, et al: Therapist report of adaptations to delivery of evidence-based practices within a system-driven reform of publicly funded children’s mental health services. J Consult Clin Psychol 2017; 85:664–675
30.
Zimmerman FJ: Social and economic determinants of disparities in professional help-seeking for child mental health problems: evidence from a national sample. Health Serv Res 2005; 40:1514–1533
31.
Bussing R, Zima BT, Gary FA, et al: Barriers to detection, help-seeking, and service use for children with ADHD symptoms. J Behav Health Serv Res 2003; 30:176–189
32.
Lau AS, Brookman-Frazee L: The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implement Sci 2016; 11:31
33.
Regan J, Lau AS, Barnett M, et al: Agency responses to a system-driven implementation of multiple evidence-based practices in children’s mental health services. BMC Health Serv Res 2017; 17:671
34.
Sanders MR: Triple P—Positive Parenting Program: towards an empirically validated multilevel parenting and family support strategy for the prevention of behavior and emotional problems in children. Clin Child Fam Psychol Rev 1999; 2:71–90
35.
Cohen JA, Mannarino AP: Trauma-focused cognitive behavioral therapy for traumatized children and families. Child Adolesc Psychiatr Clin N Am 2015; 24:557–570
36.
Chorpita BF, Daleiden EL, Collins KS: Managing and adapting practice: a system for applying evidence in clinical care with youth and families. Clin Soc Work J 2014; 42:134–142
37.
Najavits LM, Gallop RJ, Weiss RD: Seeking safety therapy for adolescent girls with PTSD and substance use disorder: a randomized controlled trial. J Behav Health Serv Res 2006; 33:453–463
38.
Managing & Adapting Practices (MAP) Training and Implementation. Los Angeles, Los Angeles County Department of Mental Health, 2010. http://file.lacounty.gov/SDSInter/dmh/159583_MAPKickoffMeeting_TrainingandImplementationPowerPoint.pdf
39.
Child-Parent Psychotherapy Introduction and Implementation Planning. Los Angeles, Los Angeles County Department of Mental Health, 2010. file:///C:/Users/kstearman/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.IE5/E75UELD3/159574_CPPIntroandImplementationPlanning.pdf
40.
TRIPLE P Positive Parenting Program: Introduction and Implementation Planning. Los Angeles, Los Angeles County Department of Mental Health, 2010. http://file.lacounty.gov/dmh/cms1_160228.pdf
41.
Trauma Focused–Cognitive Behavioral Therapy: Implementation and Planning Guide 1–8. Los Angeles, Los Angeles County Department of Mental Health, 2010. file:///C:/Users/kstearman/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.IE5/GWPQQKVY/160204_TF-CBTImplementationandPlanningGuide.pdf
42.
Training Protocols for Prevention and Early Intervention Practices. Los Angeles, Los Angeles County Department of Mental Health, 2016. file:///C:/Users/kstearman/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.IE5/HJSHGENA/201947_PEITrainingProtocolsrevised4-1-16.pdf
43.
Stadnick NA, Lau AS, Barnett M, et al: Comparing agency leader and therapist perspectives on evidence-based practices: associations with individual and organizational factors in a mental health system–driven implementation effort. Adm Policy Ment Health 2018; 45:447–461
44.
Cook JM, Thompson R, Schnurr PP: Perceived characteristics of intervention scale: development and psychometric properties. Assessment 2015; 22:704–714
45.
Stata Statistical Software: Release 15. College Station, TX, StataCorp, 2017
46.
Shirk SR, Karver M: Prediction of treatment outcome from relationship variables in child and adolescent therapy: a meta-analytic review. J Consult Clin Psychol 2003; 71:452–464
47.
Wisdom JP, Clarke GN, Green CA: What teens want: barriers to seeking care for depression. Adm Policy Ment Health Ment Health Serv Res 2006; 33:133–145
48.
Haine-Schlagel R, Brookman-Frazee L, Fettes DL, et al: Therapist focus on parent involvement in community-based youth psychotherapy. J Child Fam Stud 2012; 21:646–656
49.
Leenarts LEW, Diehle J, Doreleijers TAH, et al: Evidence-based treatments for children with trauma-related psychopathology as a result of childhood maltreatment: a systematic review. Eur Child Adolesc Psychiatry 2013; 22:269–283
50.
Connor DF, Doerfler LA, Toscano PF, et al: Characteristics of children and adolescents admitted to a residential treatment center. J Child Fam Stud 2004; 13:497–510
51.
Wintersteen MB, Mensinger JL, Diamond GS: Do gender and racial differences between patient and therapist affect therapeutic alliance and treatment retention in adolescents? Prof Psychol Res Pr 2005; 36:400–408
52.
Cauce AM, Domenech-Rodríguez M, Paradise M, et al: Cultural and contextual influences in mental health help seeking: a focus on ethnic minority youth. J Consult Clin Psychol 2002; 70:44–55
53.
Raffaelli M, Ontai LL: Gender socialization in Latino/a families: results from two retrospective studies. Sex Roles 2004; 50:287–299
54.
Becker KD, Boustani M, Gellatly R, et al: Forty years of engagement research in children’s mental health services: multidimensional measurement and practice elements. J Clin Child Adolesc Psychol 2018; 47:1–23
55.
Garland AF, Brookman-Frazee L, Hurlburt MS, et al: Mental health care for children with disruptive behavior problems: a view inside therapists’ offices. Psychiatr Serv 2010; 61:788–795
56.
Kapke TL, Gerdes AC: Latino family participation in youth mental health services: treatment retention, engagement, and response. Clin Child Fam Psychol Rev 2016; 19:329–351
57.
Hurlburt MS, Leslie LK, Landsverk J, et al: Contextual predictors of mental health service use among children open to child welfare. Arch Gen Psychiatry 2004; 61:1217–1224
58.
Park AL, Tsai KH, Guan K, et al: Unintended consequences of evidence-based treatment policy reform: is implementation the goal or the strategy for higher quality care? Adm Policy Ment Health Ment Health Serv Res 2018; 45:649–660
59.
Haine-Schlagel R, Martinez JI, Roesch SC, et al: Randomized trial of the Parent and Caregiver Active Participation Toolkit for child mental health treatment. J Clin Child Adolesc Psychol 2018; 47(suppl 1):S150–S160
Information & Authors
Information
Published In
History
Received: 27 September 2018
Revision received: 11 March 2019
Accepted: 11 April 2019
Published online: 4 June 2019
Published in print: September 01, 2019
Keywords
Authors
Competing Interests
The authors report no financial relationships with commercial interests.
Funding Information
Robert Wood Johnson Foundation Center for Health Policyhttp://dx.doi.org/10.13039/100006381: 74805
National Institute of Mental Healthhttp://dx.doi.org/10.13039/100000025: R01MH100134
This study was funded by the National Institute of Mental Health (R01MH100134) and the Robert Wood Johnson Foundation (74805).Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBGet Access
Login options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).