Skip to main content

Abstract

Objective:

The aim of this study was to compare knowledge gains from a new online training program with gains from an existing in-person training program for family peer advocates.

Methods:

Data were used from a pre-post study of individuals who enrolled in the Web-based Parent Empowerment Program training; 144 participants completed the training and pre-post tests, and 140 were admitted to the analyses. Knowledge was assessed with 34 questions, 29 of which were common to the online and in-person trainings. Pre-post knowledge scores were available from the in-person training.

Results:

Statistically significant gains in knowledge were found with both the 34 questions and the 29 questions common to both trainings. Knowledge gains across the two training models did not differ.

Conclusions:

Data on knowledge gains from this accessible, affordable online model show promise for training the growing and important workforce of family peer advocates.

HIGHLIGHTS

• An online training program for family peer advocates produced gains in knowledge scores that did not differ from those of in-person training.
• This accessible, affordable approach could assist in training the important growing workforce of family peer advocates.
To address workforce shortages in mental health and to improve parents’ engagement in their children’s mental health services, many states have funded family peer–delivered services (1). Studies demonstrate that embedding family peer advocates (FPAs) as deliverers of services can improve attendance rates and retention of families in continuing care (2). Nationally, FPAs help to address the well-documented and severe workforce shortages in children’s mental health (3). A growing number of states are developing certification standards, and a national certification process is under development. A recent survey indicated that 28 states and the District of Columbia pay for family peer support services through Medicaid (usually through a state plan amendment or a Medicaid waiver), with the training and credentialing processes varying by state (4).
The New York State Office of Mental Health (NYSOMH) has a 20-year history of supporting the FPA model in order to increase caregiver engagement in treatment through a network of FPAs embedded in children’s service provider agencies and in family-run organizations (5, 6). In New York State, FPAs receive training and credentialing through a structured process, using the Parent Empowerment Program (PEP) training that follows a health activation framework (1, 7). Furthermore, quality indicator standards are now available for use in the state in order to support clear specification of FPA roles (8). The overall goal of the FPA model has been to enhance parents’ capacity to parent, navigate care systems, and advocate for their children in part through assisting parents to become fully engaged as partners with provider teams who provide care to their children.
Educating and credentialing FPAs in New York State was a time-intensive process involving 40 hours of didactics followed by consultation calls every 2 weeks for 6 months (9). In 2010, to make training more accessible, PEP training moved to a train-the-trainer model, again showing statistically significant increases in knowledge acquisition and self-efficacy (1).
With the recent changes to children’s behavioral health services in the state, FPA services are now reimbursable with Medicaid. Thus, there is a need to increase the number of FPAs and to broaden their training so that they can support parents and caregivers across all child-serving systems (e.g., education, medical care, child welfare, substance use, juvenile justice). The NYSOMH acknowledged that 40 hours of in-person training was burdensome and costly and that FPAs needed immediate access to training to qualify for Medicaid reimbursement upon employment. Consequently, in 2014, the NYSOMH awarded New York University a contract to create a hybrid training program consisting of online learning modules, 2 days of in-person training, and 12 consultation calls over 3 months.
Web-based training programs have become increasingly popular as a cost-effective educational mechanism, particularly for training a workforce that is spread across a large geographical area. Research suggests that Web-based training is effective for health and mental health service providers (1012). Therefore, the purposes of this study were to examine the gains in knowledge from completing the Web-based PEP level 1 training program and to compare knowledge gains from the online training with gains from the in-person train-the-trainer model (1) by using the 29 knowledge questions that assessed knowledge in both training modalities.

Methods

Data for this pre-post study were collected from individuals who enrolled in the PEP level 1 online training from February 2018 to April 2019. Of the 258 people who accessed the online modules, 144 completed the training (56%). Two participants who completed the training and three who did not were removed from the study because of missing or improperly coded data. Two were removed because their pre-post change scores on the general knowledge measures presented as extreme outliers (i.e., more than three interquartile ranges from the center of the data). Thus, data from 140 participants in the completers group were examined for knowledge gains and sociodemographic and employment factors related to knowledge gains and were compared against the results of the in-person training. The individuals completing the in-person training and those completing the online training (1) were FPAs. We also examined the sociodemographic characteristics common to both training groups and found no statistically significant age, sex, ethnicity, or educational differences (data not shown). The online training was a quality improvement effort and did not require institutional review board approval.
The traditional 5-day in-person PEP training covered the following topics: the FPA role, engagement and empowerment strategies, listening and communication skills, understanding the children’s behavioral health and education systems, partnership and negotiation skills, managing groups, boundaries, and self-care. The training approach included a mix of didactic instruction, small group discussion, and role-playing.
Because the traditional in-person training was considered effective, the new training addressed a similar set of competencies. The process of developing the online training began by reviewing the topics in the existing training, changes in the FPA scope of practice, and new competencies required by system changes. This information was gathered through discussion with current trainers, FPA supervisors, FPAs, clinical partners, and state and national leaders. The content of the existing training was then rereviewed for relevancy with any disagreements about content retention resolved through discussion. New topics for the online training were chosen through a similar process. Content experts, experienced FPAs, and a team of education technologists then reviewed each online module. Of the content in the earlier in-person training, 90% was retained in the new online level 1 modules.
The new online PEP modules are accessed through a learning management system and are completed independently by trainees. The modules are intended to introduce the role and fundamental principles of family peer support, teach basic skills, provide an overview of key child-serving systems, and explore other relevant topics. New content was added in the areas of crisis response, planning, documentation, and a structured approach to exploring family strengths and needs. A variety of activities were built into the updated training in lieu of in-person role-play and discussion. (see Table B in the online supplement for the areas addressed.)
Questions about demographic characteristics, completed prior to training, inquired about age, gender, ethnicity, language spoken, education level, and several work characteristics. The general knowledge pre-post tests consisted of 34 multiple-choice and true-false questions assessing knowledge related to the PEP curriculum content. Of note, some of the questions were designed to gauge how FPAs would respond in certain scenarios. Twenty-nine were identical to those used in the earlier evaluation (1). As in the earlier evaluation, scores for both the 34- and 29-item tests were calculated as the number of correct responses (1). Statistical analyses were performed with SPSS, version 25. Chi-square tests were conducted to examine potential differences between training completers and noncompleters.
Because data from the general knowledge test violated the assumption of normality, Wilcoxon signed-rank tests were used to evaluate change in knowledge. This test was first performed with the 34-item evaluation. Data from the 29 items that were identical across training sessions were then analyzed in the current sample and the previous sample. Because the previous study allowed participants with missing items on the pre- and posttests to be included in the comparisons, we retained participants missing no more than four item responses per test in this sample. Missing items were calculated as incorrect. A Mann-Whitney U test was then used to determine whether knowledge gains varied between the in-person and the Web-based training. Differences in knowledge gains across sociodemographic characteristics were evaluated with the Kruskal-Wallis and Mann-Whitney tests.

Results

Completers and noncompleters did not differ on demographic factors (all p>0.050). However, significant differences in employment characteristics were found between groups, including work status as an FPA (χ2=28.85, df=1, p<0.001) and whether their employer requires credentialing as an FPA (χ2=20.53, df=2, p<0.001). Fewer noncompleters were employed as FPAs or worked for settings requiring FPA credentialing (see Table A in the online supplement).
In the sample of 140 completers, scores on the 34-item posttest (mean±SD=27.82±3.89, median=29) were higher, at a statistically significant level, than on the pretest (26.39±3.66, median=27) (z=–5.24, p<0.001, r=–0.31). Scores on the common 29 items were also significantly higher on the posttest (23.65±3.45, median=24.5) than on the pretest (22.57±3.34, median=23) (z=–4.29, p<0.001, r=–0.26). The 304 participants who received the in-person training (1) also scored significantly higher on the 29 common items on the posttest (23.73±3.29, median=24) than on the pretest (22.66±3.17, median=23) (z=–6.70, p<0.001, r=–0.27). Importantly, knowledge gains across training models did not differ (Table 1). No sociodemographic characteristic was related to differences in knowledge gains (data not shown).
TABLE 1. General knowledge scores among family peer advocates, by Web-based or in-person traininga
 Pretest scorePosttest score 
Evaluation typeMSDMedianMSDMedianEffect size (r)
Web-based training model (N=140)
34-item evaluation26.393.662727.823.8929−.31
29-item evaluation22.573.342323.653.4524.5−.26b
Train-the-trainer in-person training (N=304)
29-item evaluation22.663.172323.733.2924−.27b
a
General knowledge after the Web-based training was assessed with 34 questions, 29 of which were common to the online and in-person training. Scores were calculated as the number of correct responses. All p<0.001.
b
Mann-Whitney U test: U=21,082.00, z=–.16, p=.874.

Discussion and Conclusions

Given the growing use of family peer support services, an efficient, effective, and accessible approach to training this emerging workforce is a critical need, particularly for states with large geographical areas. The cost and accessibility challenges of in-person didactic training prompted the NYSOMH to explore online training to teach the didactic portion of the effective in-person training models (1, 9). Data from the pre-post survey of the 140 completers of the online training showed a statistically significant gain in knowledge on both the battery of 34 items and the 29 items that were common to the pre-post tests administered in prior in-person training. Gains in knowledge were not significantly different between participants who completed the training in person and those who completed training via the Web-based modules. This finding is notable because online training is considerably more accessible, convenient, and cost-effective than in-person training. Importantly, most of the training content is not specific to New York State and, therefore, could be adopted by other states, thus potentially standardizing the educational training of FPAs nationwide.
The second important finding was that no sociodemographic characteristic was related to increases in knowledge. This is notable because data suggest that the use of technology in training is difficult for older individuals and those less comfortable with technology (13). Discomfort with technology has been identified as an important reason for dropping out of training (14).
These data have limitations. This study used a small sample of individuals from only New York State. The online program was accessed by 258 individuals, 56% of whom completed the training. Although no sociodemographic characteristics differentiated completers from noncompleters, the completers may have unmeasured characteristics that are partially responsible for the knowledge gain. However, a more likely explanation is that current FPAs for whom credentialing was not required or individuals interested in becoming credentialed as FPAs may have accessed the modules to understand the new responsibilities of an FPA. Importantly, these data are from only the online training. We have no information on whether improvements in knowledge translated into changes in the FPAs’ practice behaviors. These initial data on knowledge gains from an online training program for FPAs show promise. Having an accessible and affordable means of training this growing, important workforce is a critical step forward in improving care for vulnerable children.

Acknowledgments

The authors thank the NYSOMH for its support.

Footnote

These findings and conclusions are those of the authors and do not necessarily reflect the opinions of NYSOMH.

Supplementary Material

File (appi.ps.201900365.ds001.pdf)

References

1.
Hoagwood KE, Olin SS, Storfer-Isser A, et al: Evaluation of a train-the-trainers model for family peer advocates in children’s mental health. J Child Fam Stud 2018; 27:1130–1136
2.
Gopalan G, Franco LM, Dean-Assael K, et al: Statewide implementation of the 4 Rs and 2 Ss for strengthening families. J Evidence-Based Soc Work 2014; 11:84–96
3.
Hoge M, Morris JA, Daniels A, et al: An Action Plan for Behavioral Health Workforce Development. Rockville, MD, US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, 2007
4.
Medicaid Financing for Family and Youth Peer Support: A Scan of State Programs. Hamilton, NJ, Center for Health Care Strategies, 2012. http://www.chcs.org/media/FYPS_Matrix.pdf
5.
Olin SS, Hoagwood KE, Rodriguez J, et al: The application of behavior change theory to family-based services: improving parent empowerment in children’s mental health. J Child Fam Stud 2010; 19:462–470
6.
Hoagwood KE, Olin SS, Horwitz S, et al: Scaling up evidence-based practices for children and families in New York State: toward evidence-based policies on implementation for state mental health systems. J Clin Child Adolesc Psychol 2014; 43:145–157
7.
Green CA, Perrin NA, Polen MR, et al: Development of the patient activation measure for mental health. Adm Policy Ment Health Ment Health Serv Res 2010; 37:327–333
8.
Olin SS, Kutash K, Pollock M, et al: Developing quality indicators for family support services in community team-based mental health care. Adm Policy Ment Health Ment Health Serv Res 2014; 41:7–20
9.
Rodriguez J, Olin SS, Hoagwood KE, et al: The development and evaluation of a parent empowerment program for family peer advocates. J Child Fam Stud 2011; 20:397–405
10.
Khatony A, Nayery ND, Ahmadi F, et al: The effectiveness of Web-based and face-to-face continuing education methods on nurses’ knowledge about AIDS: a comparative study. BMC Med Educ 2009; 9:41
11.
Brennan EM, Sellmaier C, Jivanjee P, et al: Is online training an effective workforce development strategy for transition service providers? Results of a comparative study. J Emot Behav Disord 2018; 27, 235–245
12.
Cook DA, Levinson AJ, Garside S, et al: Internet-based learning in the health professions: a meta-analysis. JAMA 2008; 300:1181–1196
13.
Atack L, Rankin J: A descriptive study of registered nurses’ experiences with Web-based learning. J Adv Nurs 2002; 40:457–465
14.
Vardanian M, Horwitz S, Storfer-Isser A, et al: A second look at dropout rates from state-sponsored MAP trainings: can targeted adaptations improve retention in evidence-based practice trainings? Behav Therapist 2017; 40:273–282

Information & Authors

Information

Published In

Go to Psychiatric Services
Go to Psychiatric Services
Psychiatric Services
Pages: 502 - 505
PubMed: 31910753

History

Received: 22 July 2019
Revision received: 1 November 2019
Accepted: 14 November 2019
Published online: 8 January 2020
Published in print: May 01, 2020

Keywords

  1. Research delivery
  2. service delivery
  3. state-university collaboration

Authors

Details

Sarah McCue Horwitz, Ph.D. [email protected]
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Paige Cervantes, Ph.D.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Anne D. Kuppinger, M.Ed.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Patricia L. Quintero, M.P.A.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Susan Burger, B.A.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Heather Lane, L.M.S.W.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Donna Bradbury, M.A., L.M.H.C.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Andrew F. Cleek, Psy.D.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).
Kimberly Eaton Hoagwood, Ph.D.
Department of Child and Adolescent Psychiatry, New York University School of Medicine, New York (Horwitz, Cervantes, Kuppinger, Hoagwood); McSilver Institute for Poverty Policy and Research, New York University Silver School of Social Work, New York (Quintero, Cleek); Families Together in New York State, Albany (Burger); Division of Integrated Community Services for Children and Families, New York State Office of Mental Health, Albany (Lane, Bradbury).

Notes

Send correspondence to Dr. Horwitz ([email protected]).

Competing Interests

The authors report no financial relationships with commercial interests.

Funding Information

New York State Office of Mental Healthhttp://dx.doi.org/10.13039/100004866: C007542/C008288
This research was supported by the New York State Office of Mental Health (NYSOMH) (C007542/C008288).

Metrics & Citations

Metrics

Citations

Export Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format
Citation style
Style
Copy to clipboard

View Options

View options

PDF/EPUB

View PDF/EPUB

Get Access

Login options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Personal login Institutional Login Open Athens login
Purchase Options

Purchase this article to access the full text.

PPV Articles - Psychiatric Services

PPV Articles - Psychiatric Services

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Media

Figures

Other

Tables

Share

Share

Share article link

Share