Evaluation of a Web-Based Training Model for Family Peer Advocates in Children’s Mental Health

Data for this pre-post study were collected from individuals who enrolled in the PEP level 1 online training from February 2018 to April 2019. Of the 258 people who accessed the online modules, 144 completed the training (56%). Two participants who completed the training and three who did not were removed from the study because of missing or improperly coded data. Two were removed because their pre-post change scores on the general knowledge measures presented as extreme outliers (i.e., more than three interquartile ranges from the center of the data). Thus, data from 140 participants in the completers group were examined for knowledge gains and sociodemographic and employment factors related to knowledge gains and were compared against the results of the in-person training. The individuals completing the in-person training and those completing the online training (1) were FPAs. We also examined the sociodemographic characteristics common to both training groups and found no statistically significant age, sex, ethnicity, or educational differences (data not shown). The online training was a quality improvement effort and did not require institutional review board approval.

The traditional 5-day in-person PEP training covered the following topics: the FPA role, engagement and empowerment strategies, listening and communication skills, understanding the children’s behavioral health and education systems, partnership and negotiation skills, managing groups, boundaries, and self-care. The training approach included a mix of didactic instruction, small group discussion, and role-playing.

Because the traditional in-person training was considered effective, the new training addressed a similar set of competencies. The process of developing the online training began by reviewing the topics in the existing training, changes in the FPA scope of practice, and new competencies required by system changes. This information was gathered through discussion with current trainers, FPA supervisors, FPAs, clinical partners, and state and national leaders. The content of the existing training was then rereviewed for relevancy with any disagreements about content retention resolved through discussion. New topics for the online training were chosen through a similar process. Content experts, experienced FPAs, and a team of education technologists then reviewed each online module. Of the content in the earlier in-person training, 90% was retained in the new online level 1 modules.

The new online PEP modules are accessed through a learning management system and are completed independently by trainees. The modules are intended to introduce the role and fundamental principles of family peer support, teach basic skills, provide an overview of key child-serving systems, and explore other relevant topics. New content was added in the areas of crisis response, planning, documentation, and a structured approach to exploring family strengths and needs. A variety of activities were built into the updated training in lieu of in-person role-play and discussion. (see Table B in the online supplement for the areas addressed.)

Questions about demographic characteristics, completed prior to training, inquired about age, gender, ethnicity, language spoken, education level, and several work characteristics. The general knowledge pre-post tests consisted of 34 multiple-choice and true-false questions assessing knowledge related to the PEP curriculum content. Of note, some of the questions were designed to gauge how FPAs would respond in certain scenarios. Twenty-nine were identical to those used in the earlier evaluation (1). As in the earlier evaluation, scores for both the 34- and 29-item tests were calculated as the number of correct responses (1). Statistical analyses were performed with SPSS, version 25. Chi-square tests were conducted to examine potential differences between training completers and noncompleters.

Because data from the general knowledge test violated the assumption of normality, Wilcoxon signed-rank tests were used to evaluate change in knowledge. This test was first performed with the 34-item evaluation. Data from the 29 items that were identical across training sessions were then analyzed in the current sample and the previous sample. Because the previous study allowed participants with missing items on the pre- and posttests to be included in the comparisons, we retained participants missing no more than four item responses per test in this sample. Missing items were calculated as incorrect. A Mann-Whitney U test was then used to determine whether knowledge gains varied between the in-person and the Web-based training. Differences in knowledge gains across sociodemographic characteristics were evaluated with the Kruskal-Wallis and Mann-Whitney tests.

Completers and noncompleters did not differ on demographic factors (all p>0.050). However, significant differences in employment characteristics were found between groups, including work status as an FPA (χ²=28.85, df=1, p<0.001) and whether their employer requires credentialing as an FPA (χ²=20.53, df=2, p<0.001). Fewer noncompleters were employed as FPAs or worked for settings requiring FPA credentialing (see Table A in the online supplement).

In the sample of 140 completers, scores on the 34-item posttest (mean±SD=27.82±3.89, median=29) were higher, at a statistically significant level, than on the pretest (26.39±3.66, median=27) (z=–5.24, p<0.001, r=–0.31). Scores on the common 29 items were also significantly higher on the posttest (23.65±3.45, median=24.5) than on the pretest (22.57±3.34, median=23) (z=–4.29, p<0.001, r=–0.26). The 304 participants who received the in-person training (1) also scored significantly higher on the 29 common items on the posttest (23.73±3.29, median=24) than on the pretest (22.66±3.17, median=23) (z=–6.70, p<0.001, r=–0.27). Importantly, knowledge gains across training models did not differ (Table 1). No sociodemographic characteristic was related to differences in knowledge gains (data not shown).

Given the growing use of family peer support services, an efficient, effective, and accessible approach to training this emerging workforce is a critical need, particularly for states with large geographical areas. The cost and accessibility challenges of in-person didactic training prompted the NYSOMH to explore online training to teach the didactic portion of the effective in-person training models (1, 9). Data from the pre-post survey of the 140 completers of the online training showed a statistically significant gain in knowledge on both the battery of 34 items and the 29 items that were common to the pre-post tests administered in prior in-person training. Gains in knowledge were not significantly different between participants who completed the training in person and those who completed training via the Web-based modules. This finding is notable because online training is considerably more accessible, convenient, and cost-effective than in-person training. Importantly, most of the training content is not specific to New York State and, therefore, could be adopted by other states, thus potentially standardizing the educational training of FPAs nationwide.

The second important finding was that no sociodemographic characteristic was related to increases in knowledge. This is notable because data suggest that the use of technology in training is difficult for older individuals and those less comfortable with technology (13). Discomfort with technology has been identified as an important reason for dropping out of training (14).

These data have limitations. This study used a small sample of individuals from only New York State. The online program was accessed by 258 individuals, 56% of whom completed the training. Although no sociodemographic characteristics differentiated completers from noncompleters, the completers may have unmeasured characteristics that are partially responsible for the knowledge gain. However, a more likely explanation is that current FPAs for whom credentialing was not required or individuals interested in becoming credentialed as FPAs may have accessed the modules to understand the new responsibilities of an FPA. Importantly, these data are from only the online training. We have no information on whether improvements in knowledge translated into changes in the FPAs’ practice behaviors. These initial data on knowledge gains from an online training program for FPAs show promise. Having an accessible and affordable means of training this growing, important workforce is a critical step forward in improving care for vulnerable children.

The authors thank the NYSOMH for its support.