Sustainability of an Evidence-Based Practice in Community Mental Health Agencies Serving Children

The study population consisted of 89 clinicians who were employed in licensed NYS Office of Mental Health agencies serving children and adolescents and who were trained to proficiency in MAP from January 1 through December 31, 2016 (8). Nine to 18 months posttraining, clinicians were contacted via e-mail up to seven times and asked whether they were still using MAP and their reason for use or nonuse. Fifty-one (57%) clinicians completed the survey. The data collection part of the EBTDC program is considered to be a quality improvement activity and did not require institutional review board review.

Sociodemographic and professional practice characteristics—such as age, sex, race-ethnicity, education level, hours worked, service setting, and licensure—and reasons for enrolling in MAP training were all assessed prior to the training. The Texas Christian University (TCU) Survey of Organizational Functioning Efficacy and Director Leadership scales (11) were also administered prior to training. This scale and its components have excellent reliability (12). Clinicians’ familiarity and experience with Excel were also assessed prior to training.

The EBPAS was administered after the didactic training and again after the postconsultation Webinars (10). Three of the four subscales—appeal, openness, and divergence (12 questions)—were administered. EBPAS has moderate to good internal consistency and reliability (10, 13). The language was slightly edited to ascertain clinicians’ attitudes toward MAP specifically rather than toward EBPs in general. Responses (ranging from 1 to 5) to individual items in the subscales were summed and then averaged to create a mean score.

Categorical data were summarized with counts and percentages, whereas continuous data were summarized with means and standard deviations. Differences between the full sample of trained clinicians and the follow-up participants, between follow-up participants and nonparticipants, and between clinicians who continued to use (users) and did not continue to use (nonusers) MAP were evaluated with unpaired t tests for continuous data, chi-square tests for categorical data, and logistic regression analyses. All analyses used SPSS, version 23.0 (14).

Table 1 displays the characteristics of the 89 clinicians trained in 2016 and the 51 (57%) clinicians who responded to the follow-up survey. There were no statistically significant differences between the trained clinicians and the subgroup who completed the follow-up survey in sociodemographic or practice characteristics or in reasons for MAP training. Additionally, the two groups did not differ in their evaluation of organizational leadership and efficacy as measured by the TCU scales, in technology skills, or in EPBAS scores. Compared with responders (N=51), nonresponders (N=38) were less likely to be female (84% versus 100%, p=.003), more likely to report that MAP training was required by their agencies or supervisors (55% versus 33%, p=.05), and less likely to have a personal interest in using MAP (34% versus 63%, p=.01).

Sociodemographic and practice characteristics did not differ between MAP users (N=41, 80%) and nonusers (N=10, 20%). Similarly, there were no differences between MAP users and nonusers in the reason for seeking MAP training, the results of the TCU efficacy and director leadership subscales, the technology questions, or score on the EBPAS total scale at the end of the didactic training. The most common reason given for discontinued use was lack of time. Differences between MAP users and nonusers were observed in scores on the EBPAS administered after the consultation Webinars, both for total score (2.80±.44 versus 2.38±.44) and for the appeal subscale (2.69±.52 versus 2.19±.51). Logistic regression results (data not shown) suggested that clinicians who scored high on the appeal subscale were more likely to have continued using MAP compared with those who scored low on this scale (odds ratio=5.98, 95% confidence interval=1.31, 27.36).

Data from this follow-up study of a cohort of clinicians trained in an evidence-informed practice, MAP, suggest that 80% of clinicians who responded to the follow-up survey continued to use the EBP for 9 to 18 months after training. Given the investment that states are making in training agencies and providers, this is an encouraging finding. Brookman-Frazee and colleagues (15) examined the sustainability of using any of six EBPs over a 57-month period among clinicians practicing in the Los Angeles Department of Mental Health. They found that 88.9% of clinicians made claims for at least one of the six EBPs and submitted claims for one or more EBPs for an average of 21.71 months. Notably, MAP was one of two EBPs that showed a lower risk of discontinuation compared with the other EBPs examined. Thus MAP is well suited for use in community mental health settings that serve children. Southam-Gerow et al. (9), also examining MAP implementation in Los Angeles County, concluded that 75% of clinicians were trained to proficiency and that MAP could be implemented on a large scale.

Given the importance of sustained use of EBPs to improve the quality of mental health care for children in community settings, the ability to identify trainees who are unlikely to continue use of an EBP is critical. Our data suggest that the four-item appeal subscale of Aarons’ EBPAS (10) can differentiate between those who will and will not sustain use. This short set of questions, which in earlier work identified those who dropped out of training (2), can also be used to identify those who might benefit from additional training to improve sustained use of MAP. Interestingly, only the total subscale score—rather than any individual items—predicted sustained use. Clinicians who discontinued use were more likely to report a lack of time to use the MAP system. Future research is needed to document whether providing additional training for clinicians who score low on the EBPAS appeal subscale improves rates of sustainability.

These data had certain limitations. The study sample consisted of clinicians who practice in NYS licensed mental health agencies and who volunteered for MAP training. Thus they are a self-selected sample. Although the cohort of 89 clinicians trained in 2016 was contacted by e-mail up to seven times, the response rate, 57%, was suboptimal. Although there were no statistically significant differences between the trained cohort and the follow-up sample in sociodemographic or practice characteristics, reasons for seeking MAP training, or attitudes toward organizational efficacy and leadership, undetected differences between the samples could have biased the results. Further, although we know that there is considerable staff turnover in these community agencies and that turnover is likely responsible for some of the loss to follow-up, other possible reasons for nonresponse include lack of perceived relevance of MAP, lack of agency support for participation, and clinical demands. Comparisons between responders and nonresponders to the survey suggest that interest in using MAP and feeling required by agencies or supervisors to participate in training may be driving nonresponse. MAP use is self-reported, and no data were collected on fidelity to MAP or on whether it was related to improved client outcomes.

Data from NYS show that MAP was successfully adapted for use in a state system. Clinicians were trained to fidelity (2), and 80% reported having sustained use of the EBP for 9 to 18 months after training. Importantly, these data also show that the four-item appeal subscale of Aarons’ EBPAS (10) can be used to identify clinicians who are likely to discontinue use and who should be targeted for additional training.