Evaluation of a Mentoring Program for PTSD Clinic Managers in the U.S. Department of Veterans Affairs
Abstract
Objective:
This evaluation study elicited feedback from participants in a novel program intended to help posttraumatic stress disorder (PTSD) clinical managers address organizational challenges in providing services and improving care.
Methods:
Program participants were invited to respond to an online survey developed for this study; 46% (N=121) responded.
Results:
Two-thirds of survey respondents had engaged in mentoring program activities ten or more times in the past six months. Roughly half the respondents reported that the program helped them be more connected to other clinics, learn about innovations in care, and feel more supported. Those who participated more often (β=.25, p<.01) and rated their mentors highly (β=.59, p<.01) reported greater benefits from the program.
Conclusions:
Mentees who were actively engaged with the mentoring program reported significant benefits. Efforts are under way to enhance the program by strengthening mentor selection and training.
The number of veterans diagnosed as having posttraumatic stress disorder (PTSD) who receive health care from the U.S. Department of Veterans Affairs (VA) has tripled since 1990 (1,2). With the combination of improved PTSD screening, veterans returning from conflicts in Iraq and Afghanistan, and more older Vietnam-era veterans enrolling as patients, the VA annually cares for over 300,000 veterans with PTSD diagnoses (2–4). To meet this demand, the VA has hired over 6,000 new mental health providers since 2004, including 350 providers in PTSD specialty clinics (5).
In 2008, the VA Uniform Mental Health Services Handbook set forth a vision to dramatically change mental health services in the VA and provide a more consistent quality of care (6). Criteria for improved quality included mandates to reduce wait times, make evidence-based psychotherapies for PTSD available at every VA medical center, tailor services to the needs of Iraq/Afghanistan veterans and female veterans, treat consequences of sexual trauma during military service, and improve coordination of care for co-occurring PTSD and substance use disorders. The VA supported these policy goals through national rollouts of training in evidence-based psychotherapies, appointing coordinators at each VA facility to work with Iraq/Afghanistan veterans, appointing coordinators to promote evidence-based psychotherapies, and hiring additional staff to provide care for co-occurring PTSD and substance misuse (7).
Implementing such mandates poses significant organizational challenges to clinic managers (8,9). For example, clinics are expected to provide more individual evidence-based psychotherapy and less group-based supportive counseling while continuing to serve the same number of clients or even more. Managers must maintain staff morale during a period of change. Yet clinic managers often lack formal training in management or systems thinking to help their clinics meet such challenges (10,11).
The VA PTSD Mentoring Program was launched in April 2008 at the behest of VA Mental Health Services (5). The mentoring program is intended to promote top-down and bottom-up sharing of information on new mandates and evidence-based practices, encourage peer-to-peer sharing of innovations and solutions, and facilitate mutual instrumental and emotional support.
Mental health leadership in each of the VA’s 21 regional networks invited two experienced and well-respected clinical managers of PTSD treatment programs to serve as mentors. All other heads of PTSD programs were invited to be mentees. Participation was voluntary. Mentors and mentees did not get any reduction in other job responsibilities to allow time for participation.
The 42 mentors were expected to champion best practices, offer advice from their own experience, and serve as conduits for sharing information (5). Mentors’ time involvement varied, but they typically chaired monthly regional telephone calls and provided additional individual consultation as needed. These regional efforts were supported by monthly national conference calls for the mentors, national calls open to all participants, site visits, an intranet site, a Webinar series on clinical issues, and a newsletter. Mentors received guidance and support through ongoing contact with the national mentoring program manager, who serves as an external facilitator (12).
We conducted the survey to elicit feedback from mentees to inform further program development. In addition to providing descriptive findings, the survey examined some assumptions of the program logic model (13). We anticipated that mentees who had fewer years of experience would engage more frequently in program activities. We expected that mentees who participated more frequently would derive greater personal benefit from the program. We predicted that mentees who perceived their mentor to be knowledgeable and helpful would participate more frequently in program activities and would derive more benefit from the program. Finally, we hypothesized that amount of benefit that mentees derived from the program would mediate how much they credited the program with helping them implement clinical innovations.
Methods
The Stanford Institutional Review Board (IRB) determined that this study was conducted for program evaluation and did not require IRB review. Our analysis focused on responses from mentees. All mentees received two e-mails inviting them to complete a 15-minute online survey during the month of June 2010. Participants were not compensated. Of 262 mentees contacted, 121 (46%) completed the survey. A majority of respondents were psychologists (72%, N=87) between 40 and 60 years of age (52%, N=63). Over half were managers of PTSD treatment programs (56%, N=68), and the rest were nonmanager clinicians. Mentees typically had been treating PTSD from six to 20 years (50%, N=61).
The Mentoring Program Feedback Questionnaire was developed by the authors for this evaluation. Background information included participants’ profession, age, gender, clinical role or roles, tenure with the VA, years of experience treating PTSD, and supervisory responsibilities. Participants indicated how many times they had engaged in 14 specific program activities in the past six months. Respondents also indicated to what extent lack of time, lack of leadership support, calls conflicting with other duties, and lack of benefit limited their participation in the mentoring program, using a 7-point Likert scale (from 1, “not a barrier,” to 7, “major reason for not participating more often”).
Participants indicated how much the program helped them achieve 15 specific benefits (for example, “Helping me learn about new innovations in PTSD care”) by using a Likert scale from 1, “not at all helpful,” to 7, “extremely helpful.” For descriptive purposes, scores of 5 or higher were considered endorsement of each benefit. Program benefit items formed an internally reliable scale (Cronbach’s α=.96).
Participants used a 7-point Likert scale (“strongly disagree” to “strongly agree”) to indicate how much their mentor “offers useful, practical solutions,” “is too busy to provide much help to me,” “is well-organized and uses time productively during our calls,” and “has too little power and authority to help me.” After negative items were reverse-coded, these formed a single scale (Cronbach’s α=.78).
Respondents were asked whether their clinic had made no changes, minor changes, or substantial changes in the past two years to achieve the following: better meet needs of Iraq/Afghanistan veterans, more efficiently provide support to older veterans, implement evidence-based treatments, use a new specialist to improve care for veterans with co-occurring PTSD and substance use disorders, and build staff cohesion and morale. Respondents who reported making any changes indicated how much they credited the program with facilitating that change by using a 7-point Likert scale from 1, “mentoring program did not help at all,” to 7, “mentoring program was essential.”
Descriptive results were presented as means, standard deviations, and frequencies. Bivariate associations were assessed with correlations. A structural equation model, using Amos 16.0 software, was used to test hypotheses regarding variables associated with perceived benefit from the program. Variables with significant bivariate correlations were included in the model, and paths were trimmed until the Akaike information criterion was minimized and the model chi square was nonsignificant.
Results
Participants reported engaging in program activities 12.1±6.7 times (mean±SD) in the past six months. The most commonly reported activities were attending regional conference calls (74%, N=90, participated one or more times), talking with or e-mailing their mentor (70%, N=84), and visiting the program Web site (68%, N=82). Contrary to our hypothesis, frequency of participation was not significantly correlated with years of experience treating PTSD or number of years in a supervisory role. As expected, frequency of participation was negatively associated with lack of time (r=–.25, df=120, p=.01) and positively associated with perceived mentor quality (r=.38, df=120, p=.01).
Over half of participants who responded to survey items about program benefits (N=117) reported that the mentoring program had helped them feel more connected to other PTSD program directors and clinicians (62%, N=72), better understand the goals of VA leadership (56%, N=65), learn about innovations in PTSD care (55%, N=65), and feel more supported (52%, N=61). From 40% to 50% reported that the program helped them improve implementation of VA Uniform Mental Health Services Handbook mandates in their programs (N=59), enhance quality of care (N=56), get access to training (N=54), implement VA policies (N=53), and solve practical problems (N=49). Only about one-quarter of respondents felt the program helped them achieve more distal goals, such as learning how to cut through bureaucratic red tape (N=29) or improve staff morale (N=28).
One hundred eight (89%) participants reported making substantial changes to implement evidence-based treatments (N=94, 78%), meet the needs of Afghanistan/Iraq veterans (N=78, 65%), more efficiently meet the needs of older veterans (N=38, 31%), improve coordination of care for PTSD and co-occurring substance use disorder (N=58, 48%), and improve staff morale (N=29, 24%). The program was credited with facilitating clinical improvements by about one-third (36%, N=41) of those who made changes to implement evidence-based treatments and one-quarter (25%, N=29) of those who modified services for Afghanistan/Iraq veterans. Only 14%–19% of participants whose clinics made changes to more efficiently meet the needs of older veterans (N=16), improve care for co-occurring PTSD and substance use disorders (N=12), or improve staff morale (N=13) credited the mentoring program with facilitating those changes.
The final structural equation model had a good fit to the observed data (χ2=3.6, df=5, p=.61, goodness-of-fit index=.988, root mean square error of approximation=.00) (Figure 1). The results confirmed that respondents who participated more frequently in program activities (β=.25, df=120, p<.01) and rated their mentor highly (β=.59, df=120, p<.01) reported greater benefit from the program. As hypothesized, the degree of benefit that participants perceived mediated the relationship between mentor quality, frequency of participation, and their crediting the program with facilitating clinical improvements.
Figure 1 Structural equation model of benefits of the VA Posttraumatic Stress Disorder Mentoring Program to clinical managers and their clinicsa
aVA, U.S. Department of Veterans Affairs. χ2=3.6, df=5, p<.61; goodness-of-fit index=.988; Akaike information criterion=23.6; root mean square error of approximation=.000
*p<.01
Discussion
The Uniform Mental Health Services Handbook mandates dramatic changes in mental health services in the VA. Local managers and clinicians must overcome administrative challenges to carry out these mandates successfully. The PTSD Mentoring Program was initiated with the hope that we could develop and share local solutions to such problems. Results of this evaluation suggest that the mentoring program has been somewhat successful in providing the right structures and supports for administrators to identify and work with others in similar positions, for mental health leadership to directly communicate policies to the field, and for providers to share their feedback with leadership.
The mentoring program was intended to help managers address organizational challenges which in turn could facilitate implementation of new VA policy directives. However, the program was initiated alongside several other initiatives to support these VA policies. These concurrent initiatives included nationwide training in evidence-based treatments, hiring of evidence-based practice coordinators, and setting of new performance standards. It is therefore hard to assess the incremental effect of the mentoring program on clinical improvement.
Our results confirm that the mentor-mentee relationship is critical to the program’s success. Program effectiveness can be enhanced by further strengthening the mentor-mentee relationship. First, it is important to select mentors who not only have the requisite expertise but can also dedicate sufficient time to mentoring. Second, mentors’ roles are being redefined to more explicitly support quality improvement. Each participant seeking mentoring is now expected to specify a quality improvement goal; mentors are tasked with helping mentees clarify the changes needed to achieve their goal, providing problem-solving support, linking them with peers who successfully instituted similar changes, and giving feedback. A potential future enhancement to improve effectiveness would be providing mentors with more formalized training in best coaching practices (12,14).
This study had important limitations. Although the 46% response rate is not unusual for a survey of this type, our sample likely overrepresents people most engaged with the program. Data were obtained by self-report. However, we obtained widely varying responses, suggesting that some participants were willing to report negative perceptions. The cross-sectional design allowed us only to show patterns of association, not prove causality.
Conclusions
The VA PTSD Mentoring Program is one of the first large-scale efforts to help mental health clinic managers address organizational challenges they face in implementing new policy mandates and local innovations to improve care. This evaluation provides initial indications that mentees who were actively engaged with the program derived important benefits. The quality of the mentoring relationship is critical to mentees’ experience. Efforts are under way to further enhance the program by strengthening mentor selection and training.
Acknowledgments and disclosures
This work was supported by the VA National Center for PTSD and the VA Palo Alto Health Care System. The authors thank the participants in the VA PTSD Mentor Program for their help in this evaluation and their ongoing efforts to improve care for veterans. The opinions are those of the authors and do not necessarily represent the position of the U.S. Department of Veterans Affairs.
The authors report no competing interests.
References
1.
Lund BC, Bernardy NC, Alexander B, et al.: Declining benzodiazepine use in veterans with posttraumatic stress disorder. Journal of Clinical Psychiatry 73:292–296, 2012
2.
Rosenheck RA, Fontana AF: Recent trends in VA treatment of post-traumatic stress disorder and other mental disorders. Health Affairs 26:1720–1727, 2007
3.
Analysis of VA Health Care Utilization Among Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) Veterans, Cumulative From 1st Quarter FY2002 Through 4th Quarter FY2010. Washington, DC, US Department of Veterans Affairs, Office of Public Health and Environmental Hazards, Dec 2010
4.
Desai R, Spencer H, Gray S, et al.: The Long Journey Home XVIII. Treatment of Posttraumatic Stress Disorder in the Department of Veterans Affairs: Fiscal Year 2009 Service Delivery and Performance. West Haven, Conn, VA Northeast Program Evaluation Center, 2010
5.
Bernardy NC, Hamblen JL, Friedman MJ, et al.: Implementation of a posttraumatic stress disorder mentoring program to improve treatment services. Psychological Trauma: Theory, Research, Practice, and Policy, 3:292–299, 2011
6.
VA Handbook 1160.01: Uniform Mental Health Services in VA Medical Centers and Clinics. Washington, DC, US Department of Veterans Affairs, 2008
7.
Karlin BE, Ruzek JI, Chard KM, et al.: Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the Veterans Health Administration. Journal of Traumatic Stress 23:663–673, 2010
8.
Forsner T, Hansson J, Brommels M, et al.: Implementing clinical guidelines in psychiatry: a qualitative study of perceived facilitators and barriers. BMC Psychiatry 10:8, 2010
9.
Ploeg J, Davies B, Edwards N, et al.: Factors influencing best-practice guideline implementation: lessons learned from administrators, nursing staff, and project leaders. Worldviews on Evidence-Based Nursing 4:210–219, 2007
10.
Ranz J, Stueve A: The role of the psychiatrist as program medical director. Psychiatric Services 49:1203–1207, 1998
11.
Varkey P, Karlapudi S, Rose S, et al.: A systems approach for implementing practice-based learning and improvement and systems-based practice in graduate medical education. Academic Medicine 84:335–339, 2009
12.
Stetler CB, Legro MW, Rycroft-Malone J, et al.: Role of “external facilitation” in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration. Implementation Science 1:23, 2006
13.
Weiss CH: How can theory-based evaluation make greater headway? Evaluation Review 21:501–524, 1997
14.
Boyle P, Boice B: Systematic mentoring for new faculty teachers and graduate teaching assistants. Innovative Higher Education 22:157–179, 1998
Information & Authors
Information
Published In
History
Published online: 1 October 2012
Published in print: October 2012
Authors
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).