Academic-Policy Partnerships in Evidence-Based Practice Implementation and Policy Maker Use of Child Mental Health Research

We used data from the 2015 National Association of State Mental Health Program Directors Research Institute (NRI) survey and a 2019 National Institute of Mental Health–funded web-based survey of state directors of CMH services (8, 11).

NRI conducts surveys on the characteristics of state mental health agencies (SMHAs), completed by SMHA representatives across the United States. For this study, we used the 2015 survey, which included data from 46 states and Washington, D.C. To evaluate the type of partnership, we used responses to the following NRI questions: “Does the SMHA work with universities and/or academia to support training of the mental health workforce?” and “Is your SMHA working with academic or university partners to implement or evaluate EBPs or promising practices?” State representatives who responded “yes” to either question were then asked to describe their partnership(s) in an open text field. Two authors (D.E.M.S, S.M.H) reviewed all of the open-text responses, found variation in scope and strength of partnerships, and agreed on the development of a dichotomous variable to categorize the strength of training and IE partnerships. States were categorized as having no or limited partnerships if they responded “no” to working within an academic partnership, responded “yes” but did not describe the partnership, or responded “yes” but described a partnership focused on implementing only one or two EBPs. States were coded as having strong partnerships if their responses described partnerships with multiple institutions or that were committed to a range of aims.

We also used responses to the NRI survey’s six questions on barriers to implementation of EBPs, which asked respondents on the NRI survey to rate (on a 7-point Likert scale) their state’s workforce shortages, financial barriers, provider readiness and capacity, mismatch between community needs and EBP implementation, difficulties maintaining fidelity to EBPs, and provider and/or clinician resistance. We dichotomized these responses; ratings of 5–7 indicated a significant barrier, and ratings of 1–4 indicated no significant barrier.

The 2019 SMHA survey was web based and covered respondent experiences using CMH research, preferences for receiving research findings, and perceptions of the extent to which specific CMH issues were priorities for their agency. The survey was sent to 253 SMHA personnel in leadership positions and was completed by 129 people across 47 states from December 2019 to February 2020. The Drexel University Institutional Review Board approved this study.

For the current analyses of the 2019 SMHA data, we retained responses from one respondent per state (N=47), according to the following hierarchy: SMHA heads (N=21, 45%); state CMH planners (N=15, 32%); respondents from the Children, Youth, and Families Division (N=7, 15%); and medical directors (N=3, 6%). One respondent did not indicate their role (2%). Nearly half (N=22) of the respondents had 10 or more years of experience working at the SMHA, and most (N=42, 89%) had at least a master’s degree. We analyzed data from three of the 2019 SMHA survey questions, which had respondents rate, on a 5-point Likert scale, their confidence using CMH research, the extent to which they used research in preliminary discussions about developing and/or changing a policy or program, and perceived barriers to using research. We then dichotomized the responses; ratings of 1–3 indicated not confident, rarely, and not a significant barrier, and ratings of 4–5 indicated confident, frequently, and a significant barrier.

Because of previous findings that collaboration acts as a facilitator to research use (12) and is associated with policy that supports EBPs (3), chi-square tests and, when assumptions were violated, one-tailed Fisher’s exact tests were used to examine relationships between type (i.e., training or IE) and strength (i.e., no/limited or strong) of partnerships on barriers to EBP implementation and research use and to evaluate the relationship between barriers to implementation and barriers to research use. After accounting for missing data across both surveys, the resulting sample included respondents from 44 states. Seven states did not respond about the presence of a training partnership. Therefore, sample size for training partnership analyses was 37. Because the sample was small, below we report relationships that were statistically significant as well as those that approached significance.

More than 75% of responding states (N=28 of 37) reported working with academic institutions to support workforce training. Ten (27%) responding states were categorized as having a strong partnership. Approximately 84% (N=37 of 44 states) reported partnering with academic institutions to implement and/or evaluate EBPs; 21 (48%) were categorized as having a strong partnership (see the online supplement to this report).

Having strong training partnerships was not associated with any barrier to implementation of EBPs. Having strong IE partnerships, however, was associated with more frequent endorsement of barriers to attaining or maintaining fidelity to EBPs (χ²=4.0, N=44, df=1, p=0.044). Furthermore, IE partnerships were marginally associated with fewer endorsements of provider readiness and/or capacity as a barrier.

Having strong training partnerships was associated with less frequent endorsement of lack of time (p=0.009) as a barrier to research use. Having strong IE partnerships was marginally associated with reporting confidence in finding CMH research to inform policy or program development and in interpreting research results. It was also marginally associated with less frequent reporting of using research because their organization required them to do so.

We then examined relationships between barriers. Having shortages in the trained workforce was associated with greater endorsement of several barriers to research use by policy makers, including lack of interaction or collaboration with researchers (p=0.022), lack of relevance of researchers’ questions to policy makers’ decisions (p=0.018), and unclear presentation or communication of findings (p=0.039). Mismatch between community needs and EBP implementation was associated with greater endorsement of limited agency resources as a barrier to research use by policy makers (p=0.006). Barriers to provider readiness and/or capacity were marginally associated with greater endorsement of limited agency resources as a barrier to research use by policy makers.

These preliminary findings suggest associations between the presence of academic-policy partnerships and EBP implementation and policy maker use of CMH research. Furthermore, having broadly integrated academic-policy partnerships may be more advantageous than having no or limited partnerships. More research is needed into effective models of collaboration across states for improvements in CMH service availability and quality.

Strong IE partnerships were associated with more issues with fidelity to EBPs, likely because states with strong partnerships were aware of fidelity issues because of monitoring of provider performance. States with no or limited partnerships may not collect such data, even though maintaining fidelity to EBP implementation is critical for improving outcomes (13, 14). Therefore, identification of problems with fidelity is consistent with the likely goals of these partnerships. Our results also indicated a marginal association between strong IE partnerships and fewer reported barriers related to provider readiness and capacity to implement EBP; more research is needed to confirm this finding. The data showed no relation between the strength of states’ training partnerships and any barriers to EBP implementation.

Academic-policy partnerships were important in policy makers’ use of CMH research. Strong training partnerships were associated with less frequent endorsement of lack of time as a barrier to the use of research, possibly because policy makers who champion training may be more effective in finding and applying research evidence to their work, which in turn may indicate increased organizational expertise. States with strong training partnerships may also have more resources, and, thus, their policy makers may have more time to allocate toward research use. However, significant differences were not found in endorsement of issues related to financing or agency resources, suggesting that greater resources may not fully explain this finding. Furthermore, the data indicated a marginal association of strong IE partnerships with greater confidence in finding and interpreting CMH research and with less frequent endorsement of using research because it was required. Strong IE collaborations may influence the culture of the agency to promote positive attitudes toward research and may increase policy maker competence. This is an area for further investigation.

Several trends in research use and endorsement of barriers to implementation were similar across states, regardless of academic-policy partnership status. The findings on policy makers’ use of research were encouraging; most respondents (>75%) reported frequent use of (N ranged from 33 to 36 of 42 across three items) and confidence in finding (N=33 of 43) and interpreting (N=33 of 43) CMH research. Furthermore, barriers to policy makers’ use of research were endorsed less frequently than barriers to EBP implementation, with only one item—limited agency resources— reported as a barrier by more than 50% (N=23 of 44) of respondents. In contrast, more than 70% endorsed trained workforce shortages (N=31 of 44), provider readiness and/or capacity (N=31 of 44), and financing issues (N=34 of 44) as barriers, indicating widespread systems issues that need to be addressed. Barriers to implementation of EBPs and policy makers’ use of research were also related. Endorsement of workforce shortages was associated with more barriers to research use, including lack of interaction or collaboration with researchers, limited relevance of research questions, and unclear presentation and/or communication of research findings. Endorsement of a mismatch between community needs and EBP implementation was associated with endorsement of limited agency resources as a barrier to research use; issues with provider readiness and/or capacity was also marginally associated with this research use barrier. These findings suggest that states endorsing more barriers to implementation also perceived more barriers to policy makers’ research use. These data also suggest stability in perceived barriers. States that reported implementation barriers in 2015 continued to endorse barriers in the related area of CMH research use in 2019.

This preliminary study had limitations. We relied solely on survey responses, and the NRI survey was completed 4 years prior to the 2019 SMHA survey. Therefore, the NRI survey may not have captured 2019 conditions. Furthermore, the NRI survey is not specific to CMH and may not reflect CMH conditions. Respondents to the SMHA survey were highly educated and experienced and may not represent all SMHA personnel. Our categorization of partnerships was crude and did not capture the degree of variation in partnerships across states. There were weaknesses in our approach to grouping together states with no partnerships and limited partnerships, because implementing partnerships even for a limited number of EBPs is a major undertaking. Future research should explore differences in partnership characteristics. Finally, other state-level factors (e.g., political party, per capita CMH spending, Medicaid expansion) and individual variables (e.g., respondent interpretation of questions) that were not assessed likely interacted in important ways with these outcomes and will require attention in future research.

These preliminary results suggest that broadly integrated academic-policy partnerships are associated with CMH research use and EBP implementation. Because these partnerships may improve CMH care, careful examination of varying models of academic-policy partnerships is warranted.