A Review of School Mental Health Programs in SAMHSA’s National Registry of Evidence-Based Programs and Practices

During summer of 2011, we conducted a review of programs in NREPP’s database that were classified as mental health promotion or treatment (or both) and were deliverable in school settings. Reference to this date is critical due to the registry’s reviewing processes; interventions are being continuously added as intervention developers elect to participate through a voluntary, self-nomination system for review. We note, however, that SAMHSA’s process does not review all programs and that NREPP is not an exhaustive registry.

Using information provided on the registry, we describe the program characteristics, including intervention tier (that is, whether the intervention was universal, selective, or indicated), outcomes (substance use, academic achievement, violence rate, externalizing behaviors, family functioning, and suicide rate), mechanism addressed to reach outcomes (such as social or emotional skills, parenting, system-level delivery, cognitive-behavioral approaches, psychoeducation, or a combination of these), and proprietary status (whether materials and services were available only for a fee or whether they were available at no cost, or some combination). Program costs were calculated on the basis of the first year of use with ten students and for continuing the program a second year, which was based on the expenses that the program developers listed (such as for workbooks, training, software, and fidelity monitoring tools). Costs reflected the sum of the items listed by the program developer; costs for two programs could not be determined from information provided. We acknowledge that this may not encompass costs for hiring staff to deliver programs unless specified by the developer.

We considered program ratings, as scored by SAMHSA reviewers, on quality of research and readiness for dissemination. Quality was measured on a 5-point scale ranging from 0, not acceptably addressed, to 4, acceptably addressed. Readiness for dissemination was measured on a 5-point scale ranging from 0, not available, to 4, adequate materials available. The SAMHSA reviewer assessing quality of research assigns ratings according to reliability and validity of measures, intervention fidelity, missing data and attrition, and potential confounds accounting for intervention effects. Readiness-for-dissemination scores are assigned on the basis of the reviewer’s evaluation of implementation material, training and support resources, and availability of quality assurance procedures.

At the time of the review, search results yielded 67 (34%) programs of the 200 interventions on the registry that met search criteria (specified by NREPP for being applicable in schools and focused on mental health promotion or treatment); however, four were not included in this review because the program’s target age did not include youths. In regard to program characteristics, almost half of the programs consisted of universal mental health promotion (N=29, 46%). Four (6%) programs were classified as selective, and three (5%) were indicated, whereas others addressed a combination of tiers. Only six (10%) programs provided support across all tiers, and five (8%) developers indicated that the program was not classifiable in this way.

The most common intervention outcomes addressed were youth substance use (N=15, 24%), followed by academic achievement (N=12, 19%), violence prevention (N=10, 16%), externalizing behaviors (such as attention-deficit hyperactivity disorder, conduct disorder, and aggression; N=8, 13%), family functioning (N=8, 13%), and suicide prevention (N=5, 8%). Programs used different mechanisms to address mental health; 19 programs (30%) targeted youth social or emotional skills or both, seven (11%) programs addressed parenting, four (6%) aimed efforts at system-level delivery, four (6%) addressed cognitive-behavioral approaches, three (5%) were primarily psychoeducation, and 26 (41%) programs targeted more than one of these areas.

Twenty-eight (44%) of the 63 programs provided all materials, training, and support at a cost—no program items were listed as available for free. Only one program (2%), TeenScreen (10), was publicly available at no cost. TeenScreen’s focus is to identify middle school and high school youths who need mental health services because of suicide risk or mental health concerns. TeenScreen reported free training, manuals, survey tools, consultation, and technical assistance, although costs for staff to administer the screening tool were not itemized in the estimate. This program was unique in that most programs identified expenses associated with program materials and manuals, training, measures, or technical support for adaptation or implementation. Program costs ranged from $0 to $50,000 for implementation for each year of the program. The mean±SD program cost was $4,338±$7,589 in the first year and an additional $1,856±6,805 for continuing the program a second year.

Program scores for quality of research (2.92±.56) and readiness for dissemination (3.34±.66) and their association with program costs are shown in Table 1. Notably, quality-of-research scores were not correlated with readiness-for-dissemination scores (r=.07, ns). Although quality of research was not significantly correlated with first-year costs, as quality scores increased, costs associated with second-year implementation (r=.34, p<.01) and total costs across the two years (r=.28, p<.05) increased.

ANOVAs examining differences in costs, quality-of-research scores, and readiness-for-dissemination scores as a function of program characteristics supported differences in program costs as a function of program tier (F=4.88, df=1 and 55, p<.05). Programs that focused on a single tier (universal, selective, or indicated) had significantly fewer costs the first year ($2,626±$3,284) compared with programs that addressed multiple tiers ($7,247±$11,699). Program readiness-for-dissemination scores also differed as a function of tier addressed (F=8.05, df=3 and 54, p<.01). Post hoc analyses indicated that programs that addressed only indicated supports yielded the poorest ratings for readiness for dissemination (1.93±.15), compared with universal interventions (3.48±.55) and interventions targeting multiple tiers (3.45±.59).

ANOVAs did not support significant differences in costs, quality-of-research scores, and readiness-for-dissemination scores as a function of program outcome or mechanism targeted to address outcome. Analyses supported differences in program costs as a function of proprietary status (first-year cost, F=4.78, df=1 and 58, p<.05; total costs across two years, F=4.07, df=1 and 58, p<.05). Programs that indicated that materials, training, or support were provided at cost and for free (that is, some items with cost and others not) had significantly fewer costs the first year (first year, $2,521±$2,903; total, $3,081±$4,202) compared with programs that provided all resources at cost (first year, $6,719±$10,570; total, $10,229±$19,863).

In summary, mental health treatment and promotion programs that can be delivered in schools accounted for 32% of NREPP, and only 10% of those support the provision of services across the continuum of prevention and selective and indicated intervention. Almost exclusively, programs required substantial expenses that generally did not include financial support and infrastructure for staffing and sustainability of programs, and almost half of them (42%) indicated they charged fees for all materials, training, and support. Although costs were higher for materials, training, and support for programs that addressed multiple tiers instead of just a single tier, it is important to consider the need to purchase additional programs to meet the needs of students in the other tiers. Readiness for dissemination was also low for programs addressing only tertiary or indicated supports in comparison with those addressing multiple tiers.

As found in the recent more general review of NREPP (9), this review, which focused on school-based programs for youths, suggests that navigation of evidence-based program registries to increase the likelihood of evidence-based practice is a challenging and multidimensional set of tasks. Effective integration of mental health intervention into the school environment often includes training school mental health professionals on how to work effectively in schools, including developing and maintaining relationships with school administrators, teachers, health staff, and others, and understanding relevant educational regulations and policies (11). Notably, the measurement and development of these relationships in school mental health continues to be explored empirically relative to interdisciplinary collaboration and resource sharing (12) and quality assessment and improvement (11).

While resource sharing and quality of interventions implemented with manuals are potentially captured by NREPP’s readiness-for-dissemination rating, additional assessment of the resources and infrastructure capacities for delivering these programs should be considered, given that they are essential to successful implementation (13). In addition, considering modular approaches for tertiary interventions may prove useful in advancing understanding of transportability and implementation in school settings (14). Further research is needed to understand organizational qualities (such as infrastructure, financing, implementation support, and emphasis on quality) that influence successful use of such registries and transportability of evidence-based programs and practices into common practice settings.

The authors report no competing interests.