Full access
Articles
Published Date: 30 August 2023

A Mixed-Methods Implementation Evaluation of Virtual Reality Job Interview Training in IPS Supported Employment

Shannon Blajeski, Ph.D., M.S.W., Matthew J. Smith, Ph.D., L.C.S.W. [email protected], Meghan Harrington, B.A., Jeffery Johnson, M.S.W., Brittany Ross, M.A., Addie Weaver, Ph.D., M.S.W., Lisa A. Razzano, Ph.D., C.P.R.P., … Show All … , Nicole Pashka, M.A., C.P.R.P., Adrienne Brown, M.S., John Prestipino, B.S., Karley Nelson, M.A., Tovah Lieberman, M.P.H., Neil Jordan, Ph.D., Eugene A. Oulvey, Ph.D., Kim T. Mueser, Ph.D., Susan R. McGurk, Ph.D., Morris D. Bell, Ph.D., and Justin D. Smith, Ph.D.Authors Info & Affiliations
Publication: Psychiatric Services
Volume 75, Number 3
https://doi.org/10.1176/appi.ps.20230023
Track CitationsAdd to favorites
PDF/EPUB

Abstract

Objective:

Employment rates among individuals with serious mental illness may be improved by engagement in the individual placement and support (IPS) model of supported employment. Results from a recent randomized controlled trial (RCT) indicate that virtual reality job interview training (VR-JIT) improves employment rates among individuals with serious mental illness who have been actively engaged in IPS for at least 90 days. This study reports on an initial implementation evaluation of VR-JIT during the RCT in a community mental health agency.

Methods:

A sequential, complementary mixed-methods design included use of qualitative data to improve understanding of quantitative findings. Thirteen IPS staff trained to lead VR-JIT implementation completed VR-JIT acceptability, appropriateness, and feasibility surveys. Participants randomly assigned to IPS with VR-JIT completed acceptability (N=42) and usability (N=28) surveys after implementation. The authors also conducted five focus groups with IPS staff (N=11) and VR-JIT recipients (N=13) and semistructured interviews with IPS staff (N=9) and VR-JIT recipients (N=4), followed by an integrated analysis process.

Results:

Quantitative results suggest that IPS staff found VR-JIT to be highly acceptable, appropriate for integration with IPS, and feasible for delivery. VR-JIT was highly acceptable to recipients. Qualitative results add important context to the quantitative findings, including benefits of VR-JIT for IPS staff as well as adaptations for delivering technology-based interventions to individuals with serious mental illness.

Conclusions:

These qualitative and quantitative findings are consistent with each other and were influenced by VR-JIT’s adaptability and perceived benefits. Tailoring VR-JIT instruction and delivery to individuals with serious mental illness may help optimize VR-JIT implementation within IPS.

HIGHLIGHTS

•
Individual placement and support (IPS) staff reported that virtual reality job interview training (VR-JIT) was appropriate and would be feasible to implement within the IPS model of supported employment.
•
IPS employment specialists, team leaders, and program directors, as well as VR-JIT implementers and recipients, reported that the intervention was highly acceptable.
•
IPS employment specialists reported that VR-JIT helped to create advantages for their clients, including more productive and efficient one-on-one meetings.
In the United States, approximately 80% of individuals with serious mental illness (such as schizophrenia) are unemployed (1, 2). Long-term unemployment and disability contribute to the prolonged poverty, housing instability, and food insecurity commonly experienced by these individuals (3). The gold standard for increasing employment in this population is the individual placement and support (IPS) model of supported employment (1), an evidence-based vocational rehabilitation service.
Specifically, IPS increases the employment rate for individuals with serious mental illness to approximately 55% via rapid job placement through job development with ongoing supports (4, 5). Technology-based adjunctive interventions have the potential to increase the employment rate of these individuals to an even greater extent (6, 7). For example, computer-based cognitive remediation and computerized job interview training increased employment among individuals with serious mental illness who had struggled to find employment through IPS as usual (8, 9). Although other technologies (e.g., smartphone apps or Minds@Work) have potential to improve employment (10, 11), they have not yet been evaluated in IPS.
Analysis of initial outcomes of implementing adjunctive interventions can elucidate whether the interventions may be appropriate for IPS, acceptable for IPS clients and staff, and feasible to deliver without reducing IPS fidelity. However, implementation of evidence-based practices typically occurs ≥10 years after an intervention is developed, which leads to substantial delay in the identification of effective implementation strategies (12).
To expedite implementation of an intervention, Curran and colleagues (12) formalized the use of hybrid effectiveness-implementation designs to concurrently evaluate an intervention’s effectiveness and implementation. The hybrid type 1 (HT1) design primarily studies an intervention’s effectiveness with a secondary evaluation of initial implementation process outcomes (12). An HT1 trial (Clinical Trials ID: NCT03049813) (9) recently evaluated whether virtual reality job interview training (VR-JIT), an Internet-delivered job interview simulator with automated feedback (13, 14), could be an effective adjunct to IPS. The results of the trial showed that individuals with serious mental illness who were still unemployed after the first 90 days of IPS and randomly assigned to receive IPS with VR-JIT had a significantly higher proportion of employment (52% vs. 19%) and shorter time to employment (hazard ratio=3.2) by 9 months after randomization than individuals with comparable unemployment after 90 days of IPS who were randomly assigned to receive IPS as usual (9). The aforementioned HT1 trial included the present multilevel, multimethod process evaluation of facilitators of and barriers to future VR-JIT implementation in high-fidelity IPS and of IPS staff’s and clients’ perceptions of VR-JIT appropriateness, acceptability, feasibility, and usability. Therefore, this initial implementation process evaluation could serve as a template for future studies evaluating technology-based interventions within IPS.

Methods

This initial process evaluation of VR-JIT implementation within IPS used a sequential, complementary mixed-methods design in which qualitative results provided additional depth of understanding of quantitative implementation results (15). Specifically, we collected quantitative data from IPS staff (i.e., employment specialists, team leaders, and program directors) on VR-JIT’s preimplementation acceptability, appropriateness, and expected feasibility in the parent RCT (9). We also used RCT data on the acceptability and usability of VR-JIT from recipients of the intervention. Next, we conducted qualitative postimplementation focus groups and semistructured interviews with research staff implementing VR-JIT, IPS employment specialists and team leaders, and VR-JIT recipients. The study protocol was approved by the institutional review boards at the University of Michigan and Northwestern University. A data safety and monitoring board supervised study procedures. All study participants provided informed consent.

Trial Setting and Interventions

The study was conducted at Thresholds, a nonprofit community mental health agency in Chicago, from June 2017 to October 2020. Thresholds provides comprehensive mental health services (http://thresholds.org).

IPS.

IPS is a vocational support model based on eight principles: consumer choice (no exclusion criteria—only interest in work is required), a focus on competitive employment (e.g., nonsheltered work in settings employing people without disabilities for competitive wages), integration of mental health and employment services, attention to client preferences, benefits planning, rapid job search, job development, and individualized job supports (16). Independent ratings by the state of Illinois assessed with the IPS Fidelity Scale (17) indicated mean fidelity ratings ranging from 111 to 117, reflecting good-to-exemplary fidelity to IPS on all scales (range 0–125, with higher scores indicating greater fidelity) during this study.

VR-JIT.

VR-JIT is an interactive, computerized job interview simulator commercially licensed by SIMmersion (www.simmersion.com) that uses a virtual hiring manager named “Molly Porter” who is governed by an algorithm that draws from >1,000 video clips of an actor who plays the role of Molly asking various job interview questions. Trainees verbally respond to Molly’s questions by repeating aloud a selection from 10 to 15 scripted choices offered via the virtual job interview interface. Trainees receive automated feedback in real time and via transcript and performance reviews, and their interview performance is scored between 0 and 100 after interview completion (18). Trainees apply for one of eight positions in the simulated interview, and an e-learning curriculum provides interview preparation advice. On the basis of the job interview literature and content expert review (19, 20), VR-JIT trains recipients in eight interview skills designed to convey an applicant’s positive attributes (e.g., being a hard worker and being easy to work with).

Delivery model.

Employment specialists were initially trained to implement VR-JIT but did not do so because of unforeseen implementation barriers. Instead, Thresholds’ internal research staff (called VR-JIT implementers) delivered VR-JIT at community agencies. (See the online supplement to this article for staff training methods to implement VR-JIT with fidelity and for the delivery structure of VR-JIT.)

Participants

IPS staff participants included program directors, team leaders, and employment specialists who were trained to implement VR-JIT during the main RCT or had VR-JIT recipients in their caseload. VR-JIT recipients were RCT participants who were actively engaged in IPS at the time of enrollment (i.e., they had at least one contact with an employment specialist in the past 30 days; were ≥18 years old; had a diagnosis of schizophrenia, schizoaffective disorder, major depressive disorder [any type], or bipolar disorder [types I and II, with or without psychotic features]; and were currently unemployed or underemployed and planning to interview for a job within the next 4 weeks). (Other inclusion criteria are listed in the online supplement.)

Quantitative Methods

IPS employment specialists (N=8), team leaders (N=3), and program directors (N=2) were invited to complete a paper survey after completing an orientation on how to teach RCT participants to use VR-JIT. Overall, 42 of 54 VR-JIT recipients completed a paper survey assessing VR-JIT acceptability. Six participants randomly assigned to VR-JIT never used the tool and did not complete the acceptability and usability surveys. Twenty-eight VR-JIT recipients completed the usability items, which were added during the second year of enrollment.

VR-JIT preimplementation acceptability, appropriateness, and feasibility among IPS staff.

VR-JIT orientation acceptability, VR-JIT appropriateness, and the expected feasibility of delivering VR-JIT within IPS were assessed by using items adapted from Proctor et al. (21) and Weiner et al. (22) and were used in previous evaluations of implementation of virtual interviewing in special education settings (23–25).
Orientation acceptability consisted of seven items (e.g., “How satisfied are you with the training you received on VR-JIT?”) that were summed for a total score. Possible scores ranged from 1 to 35, with higher scores indicating greater acceptability. Item responses were rated on a 5-point Likert scale (i.e., 1, not at all satisfied, to 5, very satisfied, or 1, not at all acceptable, to 5, very acceptable) (Cronbach’s α=0.74).
Appropriateness consisted of five items (e.g., “How well do you think VR-JIT fits with clients’ reasons and motivations for receiving IPS services?”) that were summed for a total score. Possible scores ranged from 1 to 25, with higher scores indicating greater appropriateness. Item responses were rated on a 5-point Likert scale (i.e., 1, not at all effective, to 5, very effective, or 1, not at all well, to 5, very well) (Cronbach’s α=0.82).
Expected feasibility consisted of nine items (e.g., “How confident are you that after training [participants], you will be able to support them as they implement VR-JIT?”) that were summed for a total score. Possible scores ranged from 1 to 45, with higher scores indicating greater feasibility. Item responses were rated on a 5-point Likert scale (i.e., 1, not at all prepared, to 5, very prepared, or 1, not at all confident, to 5, very confident) (Cronbach’s α=0.89).

Postimplementation VR-JIT acceptability and usability among recipients.

VR-JIT recipients completed acceptability (N=42) and usability (N=28) surveys after their final VR-JIT session. The Training Experience Questionnaire (18) was used to evaluate VR-JIT acceptability. This survey included five items (e.g., “How helpful was this training in preparing you for a job interview?”) that were summed for a total score. Possible scores ranged from 1 to 35, with higher scores indicating greater acceptability. Item responses were rated on a 7-point Likert scale (e.g., 1, extremely unhelpful, to 7, extremely helpful) (Cronbach’s α=0.73).
Two items were adapted from the Systems Usability Scale (26) to assess VR-JIT usability (i.e., “How comfortable are you using the VR-JIT training on your own?” and “How much do you need staff there to help you use VR-JIT?”). The items were rated on a 4-point Likert scale (i.e., 1, not at all, to 4, very much). Individual item means are reported.

Data analysis.

Quantitative data were analyzed with SPSS, version 26.0. Descriptive analyses (i.e., mean, SD, and range) were generated for the IPS staff–level and VR-JIT recipient–level initial implementation process outcomes.

Qualitative Methods

Qualitative data were collected to assess the initial process evaluation of VR-JIT implementation through focus groups and, later, through using purposive semistructured interviews, with a data analysis phase between the two stages.

Data collection.

The IPS program director at Thresholds shared a study recruitment e-mail with employment specialists and team leaders (who had clients in their caseload who received VR-JIT) to participate in the qualitative study. Two focus groups were held with IPS staff (N=11) and three with VR-JIT recipients (N=13). Thirteen semistructured interviews were held with VR-JIT implementers (N=3), employment specialists (N=3), team leaders (N=3), and VR-JIT recipients (N=4). Focus groups and interviews were video recorded. VR-JIT recipients were recruited by telephone and interviewed by research staff (recorded with digital audio). (See the online supplement for additional data collection methods.)

Analysis.

Interviews and focus group discussions were transcribed by research assistants and analyzed with dedoose software (27). Three research team members (S.B., M.H., J.J.) analyzed interview and focus group transcripts by using line-by-line coding and assigning codes on the basis of a priori and emerging themes (28). A priori themes included coding for content relevant to the experience of implementing VR-JIT, whereas emerging themes uncovered new information about using VR-JIT within IPS. Each coder worked with an unmarked transcript that was coded for themes. The three research team members then met to review codes and themes until categories were agreed on. Research team members (S.B., M.J.S., J.D.S.) organized the categorization of themes by using the Consolidated Framework for Implementation Research version 2.0 (CFIR 2.0) approach (29). CFIR 2.0 was selected because of its provision of evidence-based domains that help to frame the implementation needs of mental health interventions (e.g., outer setting and how aspects of the intervention itself influenced implementation at a specific site, with a particular population) (29, 30).

Integration of Quantitative and Qualitative Analyses

Following a sequential, complementary mixed-methods approach (15), we compared the qualitative results with the quantitative results to expand the depth of understanding of the outcomes of VR-JIT implementation. We then integrated the qualitative themes and applied them to the quantitative implementation categories of acceptability, appropriateness, and feasibility. Qualitative themes that did not fit the quantitative categories were included in the results as emerging data within CFIR 2.0 domains that enhanced our understanding of VR-JIT implementation in a community mental health agency.

Results

Table 1 displays the demographic characteristics of the study participants.
TABLE 1. Demographic characteristics of study participants, by group and type of contribution
CharacteristicIPS staffVR-JIT recipients
Surveys (N=13)Focus groups (N=11)Interviews (N=9)Surveys (N=42)Focus groups (N=13)Interviews (N=4)
N%N%N%N%N%N%
Age (M±SD years)34.8±7.8​33.5±9.9​33.1±8.6​47.5±13.1​50.7±13.2​51.8±12.6​
Female biological sex107776466721507544100
Race-ethnicity​​​​​​​​​​​​​
 African American/Black11843273332662969375
 Asian18191110—0—0—
 White1843655611263230—
 Latinx0—3270—4100—0—
 >1 race0—0—0—1218125
Highest level of education​​​​​​​​​​​​
 Less than high school diploma/GED0—0—0—1126323125
 High school diploma/GED180—0—10242150—
 Some college0—0—0—1843646250
 Associate’s degree0—0—0—120—0—
 Bachelor’s degree4317644440—180—
 Master’s degree8614365562518125
Clinical diagnosisa​​​​​​​​​​​​​
 Schizophrenia spectrum disorder0—0—0—2969969125
 Major depressive disorder0—0—0—921215375
 Bipolar disorder0—0—0—4102150—
 Substance use disorder (lifetime)0—0—0—6142150—
a
Participants could have multiple diagnoses.

Quantitative Analysis

IPS staff.

IPS staff (N=13) reported that VR-JIT orientation was acceptable (mean±SD acceptability score=32.1±2.7, range 28–35) and that VR-JIT was appropriate for interviewing practice within IPS (mean appropriateness score=22.2±2.4, range 19–25). Also, IPS staff expected VR-JIT implementation to be feasible within IPS (mean feasibility score=38.1±7.2, range 27–45).

VR-JIT recipients.

VR-JIT recipients (N=42) reported that VR-JIT was acceptable (mean acceptability score=30.4±4.0, range 16–35), that they were comfortable using VR-JIT independently (mean score assessing level of comfort with using VR-JIT independently=3.3±1.0, range 1–4), and that they did not need much staff support to use it (mean score assessing need for staff support in using VR-JIT=2.4±1.0, range 1–4).

Qualitative Analysis

Qualitative themes aligned with three broad CFIR 2.0 domains (Table 2): intervention characteristics (including relative advantages of the intervention and areas of adaptability), outer setting (including community characteristics), and individuals (including VR-JIT recipients). (See the online supplement for supporting quotations.)
TABLE 2. Qualitative implementation themes, by Consolidated Framework for Implementation Research version 2.0 (CFIR 2.0) domains
CFIR 2.0 domain and constructDefinitionStudy result
Intervention characteristic​​
 Relative advantageStakeholders see the advantage of implementing the intervention compared with an alternative solution or keeping things the same.The perceived relative advantage of virtual reality job interview training (VR-JIT) centered on Molly Porter’s (i.e., the virtual hiring manager’s) ability to be “blunt” and direct compared with the supportive, strengths-based clinical approach of individual placement and support (IPS) staff.
Staff at multiple implementation levels (e.g., VR-JIT implementer, employment specialist, team leader) discussed that being overly direct or blunt while practicing job interviews was beyond their typical scope of practice.
Employment specialists discussed how VR-JIT recipients had an improved understanding about job search processes and stronger motivation to find employment after completing VR-JIT.
Employment specialists shared that incorporating VR-JIT into IPS increased their efficiency. For example, IPS staff reported that VR-JIT enabled them to shift their attention to other important, yet time-consuming, IPS tasks such as job development.
 AdaptabilityStakeholders believe that the intervention can be sufficiently adapted, tailored, or reinvented to meet local needs.VR-JIT implementers reported that knowing how to troubleshoot technology before delivering VR-JIT reduced the chances of recipients’ disengaging from VR-JIT, particularly for those who had limited computer skills.
VR-JIT implementers reported that it was important to clearly explain the rationale for using the intervention because a few VR-JIT recipients thought that they were attending real job interviews. Thus, contextualizing VR-JIT more clearly may better prepare participants for staying engaged with the intervention.
Although IPS staff consistently discussed the need for more time to contextualize VR-JIT for recipients, IPS staff suggested that some recipients became less engaged with VR-JIT because some sessions were too long (>75 minutes). Thus, adapting both the duration of VR-JIT sessions and the frequency of scheduled sessions to meet recipient needs may enhance the delivery of VR-JIT within IPS.
Outer setting: community as a built environmentElements of the built environment create barriers to or facilitators of implementation.IPS staff described how transportation challenges affected recipients’ ability to attend weekly in-person VR-JIT sessions and suggested offering VR-JIT via remote services as a critical delivery adaptation.
Individuals: intervention recipientsReceiving the interventionVR-JIT recipients with serious mental illness and long periods of disability faced some barriers to staying engaged with VR-JIT sessions. IPS staff identified engagement barriers that they perceived as directly related to recipients’ psychiatric symptoms or to ancillary issues, such as low tolerance for frustration.

Intervention characteristics.

The first theme suggested that VR-JIT provided a relative advantage to IPS (i.e., VR-JIT provides direct, unapologetic feedback that IPS staff do not feel they can provide and enables staff to shift attention to job development). The second theme centered on potential VR-JIT adaptations to enhance uptake within IPS (i.e., IPS staff having a working knowledge of troubleshooting any technology issues, being able to translate the “virtual” aspect of VR-JIT during orientation (e.g., clearly indicating the simulated nature of the intervention to recipients and that it was not a remotely held interview for a real job), and reducing the duration and frequency of sessions) (Table 2).

Outer setting.

A single theme emerged on community characteristics that emphasized challenges to the use of public transportation as a barrier to attending multiple visits (Table 2).

Individuals.

A single theme emphasized psychiatric symptoms and long periods of disability as potential barriers to VR-JIT engagement for individuals with serious mental illness receiving this intervention (Table 2).

Mixed-Methods Analysis

Our mixed-methods approach helped complement the quantitative findings with the qualitative results obtained through the interviews and focus groups (31). Quantitative findings of acceptability, appropriateness, and feasibility among IPS staff were matched with quotations from the focus groups or interviews (Figure 1). For example, as reported above, IPS staff scored VR-JIT highly on appropriateness, and they noted that “[VR-JIT] has been a great tool for members who have been enrolled [in IPS] and were able to use it.” Quantitative findings on acceptability and usability of VR-JIT among its recipients matched the tenor of quotations from the focus groups and interviews with recipients (Figure 2). For example, VR-JIT recipients gave VR-JIT high scores on acceptability (see above), and one said, “I didn’t have a challenge, I enjoyed the Molly [VR-JIT] program. [I] made all of my appointments; of course I made them on time.” (See the online supplement results for additional supporting quotations.)
FIGURE 1. Quantitative and qualitative findings on the acceptability, appropriateness, and feasibility of virtual reality job interview training (VR-JIT) among individual placement and support (IPS) staffa
aA: acceptability (possible scores range from 1 to 35, with higher scores indicating greater acceptability), B: appropriateness (possible scores range from 1 to 25, with higher scores indicating greater appropriateness), and C: feasibility (possible scores range from 1 to 45, with higher scores indicating greater feasibility) of VR-JIT. Box plots represent the interquartile range (IQR); whiskers represent the range. The upper IQR value is also the maximum value in all plots.
FIGURE 2. Quantitative and qualitative findings on the acceptability and usability of virtual reality job interview training (VR-JIT) among its recipientsa
aA: acceptability (possible scores range from 1 to 35, with higher scores indicating greater acceptability), B: usability (level of comfort with using VR-JIT independently; possible scores range from 1 to 4, with higher scores indicating greater usability), and C: usability (need for staff support; possible scores range from 1 to 4, with higher scores indicating lower usability) of VR-JIT. Box plots represent the interquartile range (IQR); whiskers represent the range. The upper IQR value is also the maximum value in all plots; small circles in panels A and B denote potential outliers. The median in panel B is also the maximum of the IQR; the median in panel C is also the minimum of the IQR.

Discussion

This evaluation of the initial implementation of VR-JIT in IPS enhances our understanding of how to implement and potentially adapt this intervention in urban settings. Quantitative, qualitative, and mixed-methods results suggested that IPS staff found VR-JIT to be acceptable and appropriate and that the staff expected VR-JIT implementation to be mostly feasible. Notably, some employment specialists suggested that it would take time to strategize how VR-JIT would fit into their workflow. VR-JIT recipients reported that the intervention was highly acceptable and usable. The qualitative findings suggested that VR-JIT offered benefits for IPS employment specialists because of the addition of a focused and direct intervention for job interview training. Additionally, general technology skills were needed to ensure that the program would run smoothly in a community setting. Finally, a minority of recipients struggled to sustain VR-JIT engagement because of their psychiatric symptoms or challenges with focus and attention.
To overcome technology-related barriers, IPS staff and clients suggested adapting VR-JIT delivery for recipients with low computer literacy and training IPS staff in addressing recipients’ challenges with using technology. IPS staff also recommended that information to orient recipients on how to use VR-JIT be better at explaining that the intervention is a training activity. Furthermore, the results suggest that VR-JIT engagement and acceptability can be enhanced by aligning the duration and frequency of sessions with recipients’ preferences. Finally, barriers to use of public transportation were related to the need for multiple trips to the agency for VR-JIT sessions. Given that many low-income adults residing in an urban setting rely on public transportation, this finding raises the need to consider implementing VR-JIT via self-guided, fully remote, or hybrid (e.g., in-person VR-JIT orientation with remote support) delivery strategies. However, the feasibility of these implementation strategies is unknown and was beyond the scope of this study because we did not know the extent to which IPS clients had access to computing devices at home or in their neighborhood or whether they could borrow such devices from their agency. However, this scalability issue is agency specific and highlights the need for agency-level planning for VR-JIT implementation, which has associated labor costs (32).
Overall, our results suggest that technology-assisted interventions offer individuals with serious mental illness more opportunities to achieve their potential for recovery. Over the past decade, research has suggested that integrating technology-based interventions within the existing mental health service delivery system is practical, feasible, and acceptable (33–35). Consistent with these previous studies, the initial implementation outcomes in this study suggest that VR-JIT can add value and offer relative advantages to existing service, while requiring consideration of some potential adaptations to overcome initial barriers. Thus, this study provides findings on an intervention that focused on employment readiness, whereas most existing research on technology-assisted interventions within mental health services has focused on symptoms and treatment adherence (33).

Implications for Practice

We suggest some adaptations for future VR-JIT delivery. For instance, employment specialists could contextualize the importance of virtual interview practice before implementation, which may improve some issues with frustration or disengagement that were observed for a minority of VR-JIT recipients. In addition, mental health agencies could consider training peer support specialists as VR-JIT implementers, because a recent study found that peer support specialists reported that VR-JIT was acceptable and perceived themselves as being uniquely positioned to deliver the intervention (36).
Additional implications focus on how VR-JIT could enhance IPS. For instance, employment specialists reported that they appreciated the blunt responses from the virtual hiring manager (i.e., “Molly Porter”) during the virtual interview. This approach gave their clients direct feedback from a third party in addition to their own supportive responses that were often influenced by their intention to be socially desirable. Additionally, a recent qualitative study on IPS employment specialists and VR-JIT indicated that clients perceived that working with VR-JIT improved their confidence in job interviewing (37), consistent with earlier RCT results (9). Notably, employment specialists perceived their clients’ improved confidence after VR-JIT as helping with IPS engagement and job search activities.

Limitations and Future Directions

Most VR-JIT sessions were delivered by trained research staff (i.e., VR-JIT implementers) as opposed to employment specialists. Although employment specialists gave feedback on how VR-JIT assisted their clients and their own work, the results are limited for forecasting implementation by employment specialists. For VR-JIT recipients, we note a potential risk to internal validity due to maturation effects. Specifically, VR-JIT recipients completed their acceptability and usability measures and initial focus groups before enrollment was paused because of the COVID-19 pandemic. Individual interviews occurred after the termination of RCT study enrollment (but were held during the originally planned follow-up period) because of the COVID-19 pandemic. Last, the four VR-JIT recipients participating in the interviews had been conveniently sampled from the eight VR-JIT recipients who were still engaged with IPS services during the pandemic; therefore, these views represent those of recipients with greater VR-JIT engagement.

Conclusions

This sequential, complementary mixed-methods evaluation revealed initial outcomes of VR-JIT implementation within IPS. The survey-based quantitative findings suggested that IPS staff and VR-JIT recipients found VR-JIT to be acceptable, usable, and feasible for improving job interviewing skills. Qualitative results gained from focus groups and interviews with VR-JIT implementers, IPS staff, and VR-JIT recipients provided additional depth and context to VR-JIT implementation findings, such as participants’ noting the advantage of offering VR-JIT within IPS to assist clients with obtaining employment more efficiently and providing recommendations for how to improve VR-JIT delivery. Implications for future VR-JIT delivery within IPS include several adaptations, such as additional training for implementers and alternative delivery strategies, to help overcome initial implementation barriers.

Acknowledgments

The authors thank the members of Thresholds who participated in the study and acknowledge the more than 50 Thresholds administrators, research staff, and IPS staff who diligently supported various aspects of the study implementation.

Supplementary Material

File (appi.ps.20230023.ds001.pdf)

References

1.
Drake RE, Bond GR, Becker DR: IPS Supported Employment: An Evidence-Based Approach. New York, Oxford University Press, 2012
Crossref
Google Scholar
2.
McAlpine DD, Alang SM: Employment and economic outcomes of persons with mental illness and disability: the impact of the Great Recession in the United States. Psychiatr Rehabil J 2021; 44:132–141
Crossref
PubMed
Google Scholar
3.
Jester DJ, Thomas ML, Sturm ET, et al: Review of major social determinants of health in schizophrenia-spectrum psychotic disorders: I. Clinical outcomes. Schizophr Bull 2023; 49:837–850
Crossref
PubMed
Google Scholar
4.
Frederick DE, VanderWeele TJ: Supported employment: meta-analysis and review of randomized controlled trials of individual placement and support. PLoS One 2019; 14:e0212208
Crossref
PubMed
Google Scholar
5.
Bond GR, Drake RE, Becker DR: An update on individual placement and support. World Psychiatry 2020; 19:390–391
Crossref
PubMed
Google Scholar
6.
Lord SE, McGurk SR, Nicholson J, et al: The potential of technology for enhancing individual placement and support supported employment. Psychiatr Rehabil J 2014; 37:99–106
Crossref
PubMed
Google Scholar
7.
Ben-Zeev D, Buck B, Kopelovich S, et al: A technology-assisted life of recovery from psychosis. NPJ Schizophr 2019; 5:15
Crossref
PubMed
Google Scholar
8.
McGurk SR, Mueser KT, Xie H, et al: Cognitive enhancement treatment for people with mental illness who do not respond to supported employment: a randomized controlled trial. Am J Psychiatry 2015; 172:852–861
Crossref
PubMed
Google Scholar
9.
Smith MJ, Smith JD, Blajeski S, et al: An RCT of virtual reality job interview training for individuals with serious mental illness in IPS supported employment. Psychiatr Serv 2022; 73:1027–1038
Crossref
PubMed
Google Scholar
10.
Camacho E, Levin L, Torous J: Smartphone apps to support coordinated specialty care for prodromal and early course schizophrenia disorders: systematic review. J Med Internet Res 2019; 21:e16393
Crossref
PubMed
Google Scholar
11.
Sauvé G, Buck G, Lepage M, et al: Minds@Work: a new manualized intervention to improve job tenure in psychosis based on scoping review and logic model. J Occup Rehabil 2022; 32:515–528
Crossref
PubMed
Google Scholar
12.
Curran GM, Landes SJ, McBain SA, et al: Reflections on 10 years of effectiveness-implementation hybrid studies. Front Health Serv 2022; 2:1053496
Crossref
PubMed
Google Scholar
13.
Humm LB, Olsen D, Bell MD, et al: Simulated job interview improves skills for adults with serious mental illness. Stud Health Technol Inform 2014; 199:50–54
PubMed
Google Scholar
14.
Smith MJ, Smith JD, Fleming MF, et al: Enhancing individual placement and support (IPS)–supported employment: a type 1 hybrid design randomized controlled trial to evaluate virtual reality job interview training among adults with severe mental illness. Contemp Clin Trials 2019; 77:86–97
Crossref
PubMed
Google Scholar
15.
Palinkas LA, Cooper BR: Mixed methods evaluation in dissemination and implementation science; in Dissemination and Implementation Research in Health: Translating Science to Practice. Edited by Proctor EK, Colditz GA, Brownson RC. New York, Oxford University Press, 2017
Google Scholar
16.
Bond GR: Supported employment: evidence for an evidence-based practice. Psychiatr Rehabil J 2004; 27:345–359
Crossref
PubMed
Google Scholar
17.
Becker DR, Swanson SJ, Reese SL, et al: Supported Employment Fidelity Review Manual. A Companion Guide to the Evidence-Based IPS Supported Employment Fidelity Scale. Concord, NH, Dartmouth Psychiatric Research Center, 2015
Google Scholar
18.
Amith MJ, Ginger EJ, Wright M, et al: Virtual reality job interview training for individuals with psychiatric disabilities. J Nerv Ment Dis 2014; 202:659–667
Crossref
PubMed
Google Scholar
19.
Huffcutt AI: An empirical review of the employment interview construct literature. Int J Sel Assess 2011; 19:62–81
Crossref
Google Scholar
20.
Bell MD, Weinstein A: Simulated job interview skill training for people with psychiatric disability: feasibility and tolerability of virtual reality training. Schizophr Bull 2011; 37:S91–S97
Crossref
PubMed
Google Scholar
21.
Proctor E, Silmere H, Raghavan R, et al: Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38:65–76
Crossref
PubMed
Google Scholar
22.
Weiner BJ, Lewis CC, Stanick C, et al: Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12:108
Crossref
PubMed
Google Scholar
23.
Smith MJ, Sherwood K, Ross B, et al: Virtual interview training for autistic transition age youth: a randomized controlled feasibility and effectiveness trial. Autism 2021; 25:1536–1552
Crossref
PubMed
Google Scholar
24.
Smith MJ, Sherwood K, Ross B, et al: Scaling out virtual interview training for transition-age youth: a quasi-experimental hybrid effectiveness-implementation study. Career Dev Transit Except Individ 2022; 45:213–227
Crossref
Google Scholar
25.
Sherwood K, Smith MJ, Johnson J, et al: Mixed methods implementation evaluation of virtual interview training for transition-age autistic youth in pre-employment transition services. J Voc Rehabil 2023; 58:139–154
Crossref
Google Scholar
26.
Brooke J: SUS: a “quick and dirty” usability scale; in Usability Evaluation in Industry. Edited by Jordan PW, Thomas B, Weerdmeester BA, et al. London, Taylor & Francis, 1996
Google Scholar
27.
dedoose: Great Research Made Easy. Los Angeles, SocioCultural Research Consultants, 2021. www.dedoose.com
Google Scholar
28.
Miles MB, Huberman AM: Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. Thousand Oaks, CA, SAGE, 1994
Google Scholar
29.
Damschroder LJ, Reardon CM, Widerquist MAO, et al: The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci 2022; 17:75
Crossref
PubMed
Google Scholar
30.
Lodge AC, Kaufman L, Stevens Manser S: Barriers to implementing person-centered recovery planning in public mental health organizations in Texas: results from nine focus groups. Adm Policy Ment Health 2017; 44:413–429
Crossref
PubMed
Google Scholar
31.
Creswell JW, Plano Clark VL: Designing and Conducting Mixed Methods Research. Los Angeles, SAGE, 2018
Google Scholar
32.
Smith MJ, Graham AK, Sax RM, et al: Costs of preparing to implement a virtual reality job interview training programme in a community mental health agency: a budget impact analysis. J Eval Clin Pract 2020; 26:1188–1195
Crossref
PubMed
Google Scholar
33.
Berry N, Lobban F, Emsley R, et al: Acceptability of interventions delivered online and through mobile phones for people who experience severe mental health problems: a systematic review. J Med Internet Res 2016; 18:e121
Crossref
PubMed
Google Scholar
34.
Ben-Zeev D: Technology in mental health: creating new knowledge and inventing the future of services. Psychiatr Serv 2017; 68:107–108
Crossref
PubMed
Google Scholar
35.
Ben-Zeev D, Drake RE, Corrigan PW, et al: Using contemporary technologies in the assessment and treatment of serious mental illness. Am J Psychiatr Rehabil 2012; 15:357–376
Crossref
Google Scholar
36.
Üstel P, Smith MJ, Blajeski S, et al: Acceptability and feasibility of peer specialist–delivered virtual reality job interview training for individuals with serious mental illness: a qualitative study. J Technol Hum Serv 2012; 39:219–231
Crossref
Google Scholar
37.
Blajeski S, Smith MJ, Harrington M, et al: Critical elements in the experience of virtual reality job interview training for unemployed individuals with serious mental illness: implications for IPS supported employment. Psychiatr Rehabil J in press
Google Scholar

Information & Authors

Information

Published In

Go to Psychiatric Services
Go to Psychiatric Services
Psychiatric Services
Volume 75 • Number 3 • March 01, 2024
Pages: 228 - 236
PubMed: 37644829

History

Received: 17 January 2023
Revision received: 2 May 2023
Revision received: 19 June 2023
Accepted: 14 July 2023
Published online: 30 August 2023
Published in print: March 01, 2024

Keywords

  1. Schizophrenia
  2. Bipolar disorder
  3. Community mental health services
  4. Virtual job interview training
  5. Psychiatric rehabilitation

Authors

Details

Shannon Blajeski, Ph.D., M.S.W.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Matthew J. Smith, Ph.D., L.C.S.W. [email protected]
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Meghan Harrington, B.A.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Jeffery Johnson, M.S.W.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Brittany Ross, M.A.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Addie Weaver, Ph.D., M.S.W.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Lisa A. Razzano, Ph.D., C.P.R.P.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Nicole Pashka, M.A., C.P.R.P.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Adrienne Brown, M.S.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
John Prestipino, B.S.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Karley Nelson, M.A.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Tovah Lieberman, M.P.H.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Neil Jordan, Ph.D.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Eugene A. Oulvey, Ph.D.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Kim T. Mueser, Ph.D.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Susan R. McGurk, Ph.D.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Morris D. Bell, Ph.D.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author
Justin D. Smith, Ph.D.
School of Social Work, Portland State University School of Social Work, Portland, Oregon (Blajeski); School of Social Work, University of Michigan, Ann Arbor (M. J. Smith, Harrington, Ross, Weaver); United States Army, Nashville (Johnson); Department of Psychiatry, University of Illinois–Chicago, Chicago (Razzano); Thresholds, Chicago (Razzano, Pashka, Brown, Prestipino, Nelson, Lieberman); Department of Psychiatry and Behavioral Sciences, Northwestern University, Chicago (Jordan); Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr. Veterans Affairs Hospital, Hines, Illinois (Jordan); State of Illinois Department of Human Services, Chicago (Oulvey); Center for Psychiatric Rehabilitation, Boston University, Boston (Mueser, McGurk); Department of Psychiatry, Yale School of Medicine, New Haven (Bell); Department of Population Health Sciences, University of Utah Eccles School of Medicine, Salt Lake City (J. D. Smith).
View all articles by this author

Notes

Send correspondence to Dr. M. J. Smith ([email protected]).
A summary of the results from this study was included in a presentation at the 2023 Society for Social Work and Research annual meeting, Phoenix, January 11–15, 2023.

Competing Interests

The data reported in this manuscript were collected as part of a trial evaluating the original version of virtual reality job interview training.
Dr. M. J. Smith reports receiving royalty payments from SIMmersion L.L.C. for a youth-focused version of the virtual job interview intervention and receiving speaker honoraria for invited presentations from Hartford Healthcare, Ohio State University, Rush University, Virginia Commonwealth University, Boston University, University of Utah, and The Help Group. The other authors report no financial relationships with commercial interests.

Funding Information

This project was supported by funding from NIMH (R01 MH-110524; principal investigator: M. J. Smith).

Metrics & Citations

Metrics

Citations

Export Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format
Citation style
Style
Copy to clipboard
Tips for downloading citations

View Options

View options

PDF/EPUB

View PDF/EPUB

Login options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Personal login Institutional Login Open Athens login
Item saved, go to cart
Purchase Options

Purchase this article to access the full text.

PPV Articles - Psychiatric Services

PPV Articles - Psychiatric Services

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Figures

Tables

Media

Share

Share

Share article link

Share

×

As described within the American Psychiatric Association (APA)'s Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences. Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.