An RCT of Virtual Reality Job Interview Training for Individuals With Serious Mental Illness in IPS Supported Employment
Abstract
Objective:
Virtual Reality Job Interview Training (VR-JIT) is a computerized interview simulator with efficacy at enhancing interview skills and employment outcomes. A randomized controlled trial assessed VR-JIT effectiveness for participants in individual placement and support (IPS), in which approximately 55% of individuals with serious mental illness obtain employment.
Methods:
Ninety participants with serious mental illness were randomly assigned to IPS+VR-JIT (N=54) or IPS as usual (N=36), completing pretest-posttest assessments and an employment evaluation at 9 months. Intent-to-treat chi-square analysis, multivariable logistic regression, Cox proportional hazards models, and mixed-effects linear regressions were conducted. Fifty-one percent were IPS nonresponders (i.e., no employment within the first 90 days of IPS).
Results:
IPS+VR-JIT participants did not have significantly higher employment rates, compared with IPS-as-usual participants (43% versus 28%). IPS nonresponders (N=46) in the IPS+VR-JIT group had greater odds of obtaining employment (odds ratio [OR]=5.82, p=0.014) and shorter time to employment (hazard ratio=2.70, p=0.044) compared with IPS nonresponders in the IPS-as-usual group. Intent-to-treat mixed-effects linear analyses indicated that IPS+VR-JIT, compared with IPS as usual, significantly improved interview skills (p=0.006), interview confidence (p=0.013), and interview anxiety (p=0.019).
Conclusions:
VR-JIT’s potential benefits (increased employment in a shorter time) appeared to be specific to IPS nonresponders, whereas employment outcomes for recent IPS enrollees were not affected. VR-JIT could be a valuable resource for employment specialists to support IPS nonresponders, because 47% of participants engaged in mock interview training with their specialist. Future research should focus on evaluating the effectiveness and implementation of VR-JIT among IPS nonresponders.
HIGHLIGHTS
•
Virtual Reality Job Interview Training (VR-JIT) improved job interview skills, interview anxiety, and interview confidence among clients of an individual placement and support (IPS) supported employment program.
•
Among IPS clients who were not able to attain employment in 90 days (IPS nonresponders), the group that received IPS plus VR-JIT had a significantly higher employment rate (52% versus 19%) and reduced time to employment, compared with the IPS-as-usual group.
•
VR-JIT could offer employment specialists an effective tool to assess and guide interview skills development for IPS nonresponders.
Approximately 10%−20% of individuals with schizophrenia and other serious mental illnesses are competitively employed (1), which contributes to their improved physical and mental health, quality of life, social capital, and reduced poverty (2, 3). The individual placement and support (IPS) model of supported employment focuses on rapid job placement through job development with ongoing supports (4, 5) and is more effective than other supported employment programs at improving competitive work outcomes (6). Still, approximately 45% of IPS clients do not obtain competitive employment (7).
Job interviews are an established factor contributing to successful job attainment in both the general population (8–11) and among individuals with psychiatric or other disabilities (12–15). Job interviewing is anxiety provoking for most people (16) and more so for individuals with serious mental illness, who are prone to anxiety and often have impairments in their interpersonal skills (17, 18). Given that interviewing skills predict employment among individuals with serious mental illness (12, 13, 19, 20), improving such skills could be an important treatment target to enhance IPS outcomes. However, research focused on job interview skills training has not yet occurred within IPS. Although training in interviewing skills is not a core component of IPS (21, 22), it is recommended that employment specialists engage clients in practice interviews when needed (21–24). However, it is not known how frequently employment specialists engage their clients in practice interviews for jobs or whether job interview skills are improved by such practice, nor are the benefits of improving job interviewing skills in IPS known. Thus, tools to facilitate the training of interviewing skills among persons receiving IPS have the potential to improve work outcomes.
To address the need for improved tools to facilitate job interviewing training, a computerized job interview simulator, called Virtual Reality Job Interview Training (VR-JIT; www.simmersion.com), was developed to provide individuals with serious mental illness an opportunity to practice and hone their interview skills while receiving automated feedback. Prior evaluation of VR-JIT in a series of laboratory-based randomized controlled trials (RCTs) for job-seeking individuals with serious mental illness found that VR-JIT improved participants’ interview skills and self-confidence, led to more competitive job offers, and was associated with a shorter time to employment (25–31). Although approximately 30% of the participants in these efficacy studies were engaged in vocational rehabilitation, none of these studies were embedded within IPS (25–31).
The RCT reported here evaluated the effectiveness of VR-JIT when delivered within a high-fidelity IPS supported employment program. The primary hypotheses were that the provision of VR-JIT in addition to IPS would result in better employment outcomes, compared with IPS alone. Secondary hypotheses were that participants receiving VR-JIT would have greater improvements in interview skills, interview self-confidence, and interview anxiety, compared with those receiving IPS as usual.
Methods
Trial Site
The study was conducted at Thresholds (Chicago), a nonprofit community behavioral health agency that provides comprehensive mental health services, including IPS supported employment (http://thresholds.org).
Trial Design
We conducted an RCT with a type 1 hybrid effectiveness–implementation design (32) to test whether adding VR-JIT to IPS improves employment outcomes, compared with IPS as usual, describe the implementation process, and identify barriers to VR-JIT uptake. This article reports on the RCT data, and a report on the implementation processes and outcomes is forthcoming. The study protocol was approved by University of Michigan and Northwestern University institutional review boards, and all procedures were reviewed, approved, and monitored by a data and safety monitoring board (DSMB).
As detailed in our study protocol (33), employment specialists initially implemented VR-JIT at a centrally located computer lab. However, this approach resulted in low recruitment rates because of transportation and scheduling difficulties. Because VR-JIT is portable (e.g., Internet-based, delivered via computer), the location of VR-JIT delivery was changed to local neighborhood sites where participants received most of their IPS services (which improved recruitment rates). At this time, VR-JIT implementers transitioned from five employment specialists to three trained research staff to avoid significant delays related to the redesign of employment specialists’ workflow to deliver VR-JIT. Using research staff allowed for the development of an optimal process of integrating VR-JIT within IPS without affecting IPS services or fidelity. This process is being evaluated as part of a VR-JIT implementation study. Although the study was originally powered to randomly assign participants at a 1:1 ratio to receive IPS+VR-JIT or IPS as usual (33), recruitment challenges necessitated reducing the target from 160 to 133 participants by using a 3:2 randomization scheme (IPS+VR-JIT:IPS as usual). We initially planned to include only participants who were within 2 weeks of their enrollment in IPS. However, lower-than-expected enrollment necessitated expansion to all clients currently receiving IPS (i.e., with one or more contacts with an employment specialist in the past 30 days) (33).
Participants
Inclusion criteria were being age 18 or older; having a diagnosis of schizophrenia spectrum disorder, bipolar disorder, or major depression via the Structured Clinical Interview for the DSM-5 (SCID-5) (34); having a reading level greater or equal to fourth grade, as assessed with the Wide Range Achievement Test–V (WRAT) (35); being currently unemployed or underemployed; currently being a job seeker (via self-report, “Will you actively be looking for a job in the next 4 weeks?”); and willingness to be video recorded for assessments. Exclusion criteria were having a disability or medical illness that could significantly compromise cognition (e.g., traumatic brain injury), having a documented uncorrected vision or hearing problem that would prevent the participant from using VR-JIT, and being actively suicidal within the past 30 days, as assessed with the Columbia–Suicide Severity Rating Scale (36).
A total of 97 participants met inclusion criteria, provided informed consent, completed baseline study measures, and were randomly assigned to IPS+VR-JIT (N=59) or IPS as usual (N=38) (see CONSORT diagram in the online supplement to this article). Because IPS services were severely affected by COVID-19 pandemic restrictions during spring 2020, the DSMB retroactively stopped enrollment on February 1, 2020. The final intent-to-treat sample included 54 participants in IPS+VR-JIT and 36 participants in IPS as usual (N=90).
Study Measures
Baseline assessments gathered data on demographic, clinical, and cognitive characteristics; participants’ functional ability; and employment history. We assessed the following variables at pretest (baseline) and posttest (approximately 3 months later): interview skills, interview confidence, interview anxiety, and social competence. Comprehensive employment outcomes were obtained monthly throughout the 9-month study period. Because we expanded our inclusion criteria from recent IPS enrollees to also include unemployed clients who had been receiving IPS for a longer time, we ascertained post hoc the number of clients who met criteria for being an IPS nonresponder, as proposed by McGurk and colleagues (37): that is, they received IPS for at least 90 days without obtaining a job or obtained a job in IPS but lost it (were fired or quit) within 90 days.
Participant characteristics.
Participants completed a brief survey about their background (e.g., age and race-ethnicity), work history (e.g., employed full- or part-time within the past 2 years), whether they received vocational support prior to IPS (0, no; 1, yes), and total days in IPS at baseline.
Symptoms were assessed with the Brief Psychiatric Rating Scale (BPRS), with a four-factor solution used to create the following subscale domains: thought disturbance, anergia, depressed affect, and disorganization (38, 39). Participants completed the Alcohol Use Disorders Identification Test (AUDIT), with recoded total scores reflecting hazardous drinking, 1; and nonhazardous drinking, 0 (40). They also completed the Drug Abuse Screening Test (DAST), with recoded total scores reflecting moderate-to-severe use, 1; and low use, 0 (41). We computed a “problematic substance use” variable by combining the AUDIT and DAST, in which problematic substance use was a score of 1 on either the AUDIT or the DAST, and no problematic substance use was a score of 0 on both the AUDIT and the DAST.
Cognition was assessed with the MATRICS Consensus Cognitive Battery (MCCB) (42), which generates T scores for the following six domains: processing speed, attention-vigilance, verbal working memory, verbal learning, visual learning, and reasoning and problem solving. Mean scores were computed across these six indices to reflect overall neurocognition. The MCCB also included a separate T score for social cognition based on the Mayer-Salovey-Caruso Emotional Intelligence Test (43).
Community functioning was assessed by using the total score from the self-report version of the Specific Levels of Functioning (44). This instrument includes 30 items (rated, for example, as 1, highly untypical, to 5, highly typical; Cronbach’s α=0.87) across the domains of interpersonal relationships, social acceptability, activities of daily living, and work skills.
Primary outcomes.
Obtaining a competitive job (i.e., located in an integrated community setting that pays at least minimum wage and was not set aside for a person with a disability) during the 9-month follow-up was coded 1, yes, or 0, no. Competitive employment outcomes were verified by using a combination of self-reports, IPS employment records, and reports from employment specialists. Validation of the employment was required from at least two of these sources. Randomization date and employment start date were used to determine time to employment.
Secondary outcomes.
Interview skills were assessed. Participants completed two interview role plays at the pre- and posttest visits. The two interviews (at each time point) were video recorded and coded by raters masked to study condition and time point and with use of the Mock Interview Rating Scale (MIRS) (25–27, 29, 30). The MIRS uses an anchoring system to rate and score eight job interview skills (e.g., conveying oneself as a hard worker, sounding easy to work with, and sharing things in a positive way) on a 5-point scale. We used a total score across the eight ratings to reflect performance-based job interview skills for each role play and computed an average score for the two role plays at each assessment point to provide a single pretest score and a single posttest score. The MIRS demonstrated strong internal reliability and sensitivity to the effects of VR-JIT in prior studies (25–27, 29, 30) (for additional details on the MIRS, see online supplement).
Interview self-confidence was also rated. Participants self-reported their confidence in interview skills at pretest and posttest by using a nine-item survey, with items rated on a 7-point scale (1, extremely unskilled, to 7, extremely skilled); total scores were computed across all items. This survey demonstrated strong internal reliability and sensitivity to the effects of VR-JIT in prior studies (25–27, 29, 30).
Participants also rated their interview anxiety with an adapted version of the 34-item Personal Report of Public Speaking Apprehension (PRSPA) (45); the phrase “public speaking” was replaced with “job interviewing.” Items were coded from 1, strongly disagree, to 5, strongly agree. Total scores were computed by using the PRSPA’s validated two-step total score procedure, which is described elsewhere (45).
To assess social competence, participants completed the Social Skills Performance Assessment (SSPA) (46) at pretest and posttest. The SSPA includes two brief (3-minute) role plays in which participants engage in a conversation with an unknown confederate who plays the role of a new neighbor (NN) or a landlord (LL). The NN and LL role plays were video recorded at the pre- and posttest visits and were blindly rated by use of an anchoring system. The NN role play was scored on eight items via a 5-point scale (e.g., 5, very interested, to 1, very disinterested), and the LL role play was scored on nine items via a 5-point scale (e.g., 5, very focused, to 1, very unfocused). The item-level means were computed for each role play and then averaged for a single social competence score at pre- and posttest (for additional details on the SSPA, see online supplement).
Interventions
IPS.
All participants received IPS from five Thresholds teams. Independent ratings by the State of Illinois with the IPS Fidelity Scale (4) showed mean IPS fidelity ratings from 111 to 117, reflecting good-to-exemplary fidelity to IPS on all scales. To isolate the impact of VR-JIT on improving interviewing skills and employment outcomes, employment specialists were asked to refrain from conducting practice interview role plays with participants randomly assigned to IPS+VR-JIT, but they could continue discussing interview preparation with them. Employment specialists completed regular surveys reflecting specific interview preparation activities engaged in during each participant contact (for both study groups) over the past week: number of practice interviews completed, estimated time spent on practice interviews, and estimated time discussing job interview preparation.
VR-JIT.
VR-JIT is an interactive, computerized job interview simulator (see figure in online supplement) developed by SIMmersion LLC that uses a virtual hiring manager named Molly Porter, speech recognition, and nonbranching logic components to facilitate repeatedly practicing job interviews across three levels of difficulty (i.e., easy, medium, and hard) that portray different variations in Molly’s personality (e.g., friendly, professional, and inappropriate). Notably, Molly’s personality is selected at random and her mood can change depending on responses to her questions. On the basis of the job interview literature (47), VR-JIT highlights eight interview skills within an e-Learning curriculum designed to convey positive attributes of the applicant (e.g., being a hard worker and being easy to work with) and share job interview preparation tips (33). Prior to interviewing, trainees complete a job application for one of eight positions, which informs the questions asked during the virtual interview. Between Molly’s personality and mood and the unique questions for eight different positions, VR-JIT yields more than 20 hours (approximately 40 interviews) before content will begin to feel repetitive. Also, trainees receive automated feedback in real time and via transcript review and receive a score of 0 to 100 that is anchored to a summary performance assessment of the eight interview skills targeted by the intervention. Participants’ VR-JIT engagement was monitored for the total number of completed virtual interviews, total minutes engaged with the virtual interviewer, total minutes engaged with e-Learning, and mean performance score across all completed interviews.
Participants randomly assigned to IPS+VR-JIT were instructed to attend five, 2-hour VR-JIT sessions over approximately 4 weeks to complete a recommended progression across easy, medium, and hard interviews (see details in the online supplement). This approach was recommended based on the VR-JIT efficacy trials (25–31). However, the VR-JIT implementation in this study was pragmatic and allowed for clients and VR-JIT implementers to modify the number and length of sessions to fit within real-world services. Participants attended an initial face-to-face VR-JIT orientation on how to navigate the simulation (approximately 80 minutes) led by a VR-JIT implementer, who completed an adherence checklist to track whether the implementer had followed the orientation protocol.
The VR-JIT implementers were employment specialists for the first five participants, who used VR-JIT in the lab model, and research staff for the remaining 43 participants, who used VR-JIT in the community model. The checklist completion rate was 92.8%, reflecting exemplary fidelity. After the orientation visit, the remaining visits (∼70 minutes each) were coordinated by the VR-JIT implementer (including reminder calls) and consisted of face-to-face meetings in which VR-JIT implementers supported participants to log into the tool (if necessary), review e-Learning content (when requested), complete one or two virtual interviews (lasting approximately 25 minutes each), and then review transcripts and performance assessment feedback. The VR-JIT implementer monitored participant progress across easy, medium, and hard interviews by using a curriculum tracking form (for additional details on the VR-JIT implementation, see online supplement).
Study Procedures
We recruited from active caseloads from five IPS teams supervised by the same IPS administrator but located in geographically separate areas with different employment specialists. After referral, potential participants provided informed consent and then completed pretest visit 1, which included the SCID-V and WRAT (to verify inclusion), self-reported behavioral health status, interview self-confidence, and interview anxiety. At pretest visit 2, participants completed an employment history interview, the BPRS and MCCB, and the MIRS and SSPA role plays. Participants were then randomly assigned to IPS+VR-JIT or IPS as usual. Posttest visits were scheduled for approximately 3 months postrandomization. With use of established methods (48), posttest assessment dates for the IPS-as-usual group were yoked to the IPS+VR-JIT dates to avoid any confounding of group assignment with time elapsed between pre- and posttest assessments (48). The posttest visit was followed by monthly phone calls to check employment status and a 9-month follow-up visit as a final employment status check and review of behavioral health status (e.g., BPRS).
Data Analysis
Chi-square analyses and t tests were conducted to examine differences in baseline characteristics between randomly assigned study groups. To test hypotheses for the primary and secondary outcomes, intent-to-treat analyses were conducted by including all randomly assigned participants. Directional hypotheses regarding the effect of group on the primary and secondary outcomes were evaluated with one-sided tests, given prior research supporting the effects of VR-JIT on these outcomes (25–30, 49–51).
Chi-square, logistic regression, and Cox proportional hazards models evaluated whether IPS+VR-JIT resulted in a greater likelihood of employment and shorter time to employment during the 9-month study period. The proportional hazards assumption was evaluated by using a group-by-time interaction term and visual inspection of log minus log function plotted against time. The intent-to-treat analyses of the primary employment outcomes were conducted adjusting for covariates known to influence employment among individuals with serious mental illness, including prior employment (52), neurocognition (37, 53, 54), social cognition (55–57), negative symptoms (58, 59), community functioning (60–62), and problematic substance use (60, 63, 64). Because external historical events (e.g., changes in local economies in any given year) are associated with obtaining employment (65), we included baseline year of study participation as a covariate. Prior employment and neurocognition covariates were not significant predictors of employment or time to employment. Thus, they were dropped from the final models in the interest of parsimony. Baseline year of study participation, problematic substance use, social cognition, community functioning, and negative symptoms (i.e., anergia) were the final covariates in the employment outcome analyses. (A correlation matrix of the variables considered for the primary intent-to-treat analyses is included in the online supplement.)
Mixed-effects linear regression models with random intercepts and an AR[1] correlation structure (66) were used to test the secondary hypotheses that IPS+VR-JIT would lead to greater improvements in interview skills, interview self-confidence, interview anxiety, and social competence over time by modeling the effects of the group-by-time interaction. The intent-to-treat analyses of secondary outcomes were conducted, and we adjusted for the same covariates as in the primary outcome analyses.
Item-level and measure-level missing data were observed. Sixteen participants did not complete posttest measures. We did not impute measure-level data and analyzed only the raw available data. By using the interquartile range method (67), outliers were observed for interview self-confidence (N=2) and interview anxiety (N=3). Winsorized weighted outlier replacement yielded results similar to those of analyses with trimmed outliers; we thus opted for the latter approach, and the outliers were excluded from the analysis (see online supplement for power analysis details).
Results
Participant Characteristics
The 90 participants’ mean±SD age was 45.6±12.8, 57% (N=51) were male, 61% (N=55) had a schizophrenia spectrum disorder, and 39% (N=35) had a nonpsychotic mood disorder. Fifty-one percent (N=46) of participants met IPS nonresponder criteria. The study groups did not differ with respect to demographic, clinical, cognitive, or employment history variables or IPS nonresponder status (Table 1).
| IPS as usual (N=36) | IPS+VR-JIT (N=54) | Test | |||||
|---|---|---|---|---|---|---|---|
| Characteristic | N | % | N | % | statistic | df | p |
| Age (M±SD) | 43.8±12.8 | 46.9±12.8 | t=1.1 | 88 | .265 | ||
| Male biological sex | 22 | 61 | 29 | 54 | χ2=.5 | 1 | .487 |
| Race-ethnicity | |||||||
| Black, African American | 24 | 67 | 35 | 65 | |||
| White | 10 | 28 | 13 | 24 | χ2=1.2 | 3 | .759 |
| Latinx | 2 | 6 | 5 | 9 | |||
| More than one race | 0 | — | 1 | 2 | |||
| Years of education (M±SD) | 12.5±1.8 | 12.5±2.2 | t=.1 | 88 | .905 | ||
| Primary psychiatric diagnosis | |||||||
| Schizophrenia spectrum disorder | 19 | 53 | 36 | 67 | χ2=1.8 | 1 | .185 |
| Bipolar disorder (no psychotic features) | 9 | 25 | 6 | 11 | χ2=3.0 | 1 | .083 |
| Major depressive disorder (no psychotic features) | 8 | 22 | 12 | 22 | χ2=.0 | 1 | 1.00 |
| Brief Psychiatric Rating Scale (M±SD subscale score)b | |||||||
| Thought disturbance | 6.1±3.7 | 5.3±2.8 | t=–1.3 | 88 | .206 | ||
| Anergia | 5.0±2.4 | 5.1±2.0 | t=.1 | 88 | .905 | ||
| Depressed affect | 9.4±4.2 | 8.0±3.2 | t=–1.8 | 88 | .078 | ||
| Disorganization | 3.7±1.2 | 3.6±1.2 | t=-.2 | 88 | .759 | ||
| Problematic substance usec | 10 | 28 | 10 | 19 | χ2=1.1 | 1 | .301 |
| Cognitive ability (M±SD score) | |||||||
| Neurocognitiond | 33.8±9.9 | 34.1±7.9 | t=.2 | 88 | .858 | ||
| Social cognitione | 35.0±13.6 | 36.6±15.8 | t=.5 | 88 | .615 | ||
| Community functioning (M±SD score)f | 127.5±14.5 | 130.4±12.7 | t=1.0 | 88 | .318 | ||
| Employment history | |||||||
| Had competitive employment in past 2 years | 19 | 53 | 21 | 39 | χ2=1.7 | 1 | .194 |
| Ever employed full-time | 23 | 64 | 38 | 70 | χ2=.4 | 1 | .519 |
| Ever employed part-time | 29 | 81 | 48 | 89 | χ2=1.2 | 1 | .271 |
| Days in IPS at baseline (M±SD)g | 130.3±106.4 | 105.4±94.4 | t=–1.1 | 72 | .147 | ||
| IPS nonresponderh | 21 | 58 | 25 | 46 | χ2=1.3 | 1 | .263 |
| Criminal justice history: prior felony conviction | 10 | 29 | 18 | 33 | χ2=.2 | 1 | .637 |
a
IPS as usual, received standard individual placement and support (IPS) services; IPS+VR-JIT, received IPS standard services and Virtual Reality Job Interview Training (VR-JIT).
b
c
d
As measured by the MATRICS Consensus Cognitive Battery (42). Corrected T score mean±SD is 50±10, with higher scores indicating better neurocognition.
e
As measured by the Mayer-Salovey-Caruso Emotional Intelligence Test (43). Corrected T score mean±SD is 50±10, with higher scores indicating stronger social cognition.
f
As measured by the Specific Levels of Functioning scale (44). Possible scores range from 30 to 150, with higher scores indicating better functioning.
g
Two IPS-as-usual participants and one IPS+VR-JIT participant were enrolled in IPS for longer than 700 days; these individuals were outliers, and they were not included in the mean reporting of days in IPS at baseline.
h
A nonresponder was defined as a person who received IPS services for at least 90 days without obtaining a job or who obtained a job but lost it (was fired or quit) within 90 days.
IPS and VR-JIT Engagement
Table 2 presents data for the two groups on participant engagement in practice interview role plays and job interview discussions with their employment specialists. Per reports from employment specialists, 47% of the IPS-as-usual participants completed at least one practice job interview, and 69% discussed their interview skills with their specialist; 11% (N=4) of participants had missing data from their specialists on these variables. Although employment specialists were instructed to refrain from conducting practice interviews with IPS+VR-JIT participants, 17% (N=9) had such practice interviews. Approximately 65% of IPS+VR-JIT participants discussed their interview skills with their employment specialist, and 54% (N=28) discussed their progress in VR-JIT; 4% (N=2) of participants had missing data from their specialists on these variables.
| IPS as usual (N=36) | IPS+VR-JIT (N=54) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Variable | N | % | Median | Range | N | % | Median | Range | |
| IPS with employment specialistb | |||||||||
| Completed mock interview | 17 | 47 | 9 | 17 | |||||
| N of mock interviews completed (M±SD) | 1.8±1.2 | 1.0 | 1–5 | 1.4 ±.7 | 1.0 | 1–3 | |||
| Minutes of mock interviewing (M±SD)c | 24.3±14.3 | 22.5 | 5–60 | 19.4±15.5 | 15.0 | 5–45 | |||
| Discussed interview skills | 22 | 69 | 34 | 65 | |||||
| Minutes discussing interview skills (M±SD)d | 40.2±26.5 | 40.0 | 10–105 | 38.0±35.8 | 30 | 2–165 | |||
| Discussed VR-JIT interviewing progresse | 28 | 54 | |||||||
| Minutes discussing VR-JIT interviewing progress (M±SD) | 38.8±41.9 | 25.5 | 1–160 | ||||||
| VR-JIT engagement (M±SD)f | |||||||||
| Sessions completed | 4.7±2.1 | 4.5 | 1–10 | ||||||
| Virtual interviews completed | 8.8±4.2 | 10.0 | 1–15 | ||||||
| Minutes of virtual interviewing | 177.8±102.4 | 174.5 | 9–432 | ||||||
| Virtual interview scoreg | 78.7±11.6 | 80.9 | 46–96 | ||||||
| Minutes of e-Learning | 14.4±14.1 | 13.0 | 0–80 | ||||||
a
IPS as usual, received standard individual placement and support (IPS) services; IPS+VR-JIT, received IPS standard services plus Virtual Reality Job Interview Training (VR-JIT).
b
Employment specialist notes regarding job interview preparation were missing for four IPS-as-usual participants and two IPS+VR-JIT participants. Nine IPS+VR-JIT participants completed one to three mock interviews with their employment specialists, which was a minor violation of the protocol meant to isolate the effects of VR-JIT for secondary analyses.
c
Means do not include data from one IPS-as-usual participant who completed 210 minutes of mock interviewing with the employment specialist, because these data represented an outlier.
d
Means do not include data from one IPS-as-usual participant who completed 225 minutes discussing interview skills and two IPS+VR-JIT participants who completed 315 and 205 minutes discussing interview skills, because these data were outliers.
e
Two IPS_VR-JIT participants were missing specialist notes regarding discussing VR-JIT progress.
f
Six IPS+VR-JIT participants did not engage in VR-JIT (i.e., completed no VR-JIT sessions).
g
As measured by the VR-JIT algorithm. Possible scores range from 0 to 100, with higher scores indicating better interview performance.
Notably, the discussion of VR-JIT happened naturalistically and reflected general discussion about what participants learned from their virtual interview practice. Employment specialists were not instructed or trained to check in on VR-JIT progress, and results were not intentionally shared with them. The IPS-as-usual and IPS+VR-JIT groups did not differ with respect to the number of minutes in which interview skills were discussed with employment specialists. Engagement in VR-JIT showed that 48 (89%) individuals in this group participated in at least one VR-JIT session. Fifty percent (N=24) of these engaged participants completed at least five sessions, 25% (N=12) completed four sessions, and 10% (N=5) completed three sessions over 9.3±6.7 weeks. Notably, 14 (29%) of the 48 IPS+VR-JIT participants who engaged in VR-JIT completed more than five sessions (range six to 10 sessions) as a pragmatic adaptation that was necessary to progress from easy to hard interviews.
Primary Outcomes
Regarding employment, among the participants randomly assigned to IPS+VR-JIT, 43% (N=23) obtained a job during the 9-month study period, compared with 28% (N=10) in the IPS-as-usual group, which was not a statistically significant difference (see figure in online supplement). The multivariable logistic regression model adjusting for baseline year of study participation, problematic substance use, social cognition, community functioning, and negative symptoms (anergia) found that the IPS+VR-JIT group had significantly greater odds of obtaining employment within 9 months (odds ratio [OR]=3.03, p=0.027) (Table 3), compared with the IPS-as-usual group. The multivariable Cox proportional hazards model that adjusted for the same variables as the logistic regression indicated that the IPS+VR-JIT group had a shorter time to employment, compared with the IPS-as-usual group, but the difference was not statistically significant (Table 3) (see figure in online supplement).
| Predictor variable | OR | 95% CI | pb |
|---|---|---|---|
| IPS+VR-JIT (reference: IPS as usual) | 3.03 | 1.18c | .027 |
| Baseline year | 2.25 | 1.11–4.56 | .024 |
| Problematic substance use (reference: no) | 3.58 | 1.03–12.44 | .045 |
| Social cognition score | 1.06 | 1.02–1.10 | .007 |
| Community functioning score | .95 | .91–.99 | .010 |
| Negative symptoms score | .76 | .56–1.03 | .075 |
| HR | 95% CI | pb | |
| IPS+VR-JIT (reference: IPS as usual) | 1.84 | .96c | .062 |
| Baseline year | 1.64 | 1.03–2.61 | .039 |
| Problematic substance use (reference: no) | 2.37 | 1.05–5.32 | .037 |
| Social cognition score | 1.03 | 1.01–1.05 | .007 |
| Community functioning score | .97 | .94–.99 | .016 |
| Negative symptoms score | .83 | .65–1.06 | .136 |
a
Participants received standard individual placement and support (IPS) services (N=36) or standard IPS services plus Virtual Reality Job Interview Training (VR-JIT) (N=54). The odds ratios are for the multivariable logistic regression. The hazard ratios (HR) are for the multivariable Cox proportional hazards model. Logistic regression model fit statistics, χ2=28.84, df=6, p=.001; Nagelkerke R2=.375. Cox proportional hazards model fit statistic, χ2=24.37, df=6, p<.001.
b
One-sided p value for directional intervention hypothesis, two-sided p value for covariates and factors.
c
CI for directional intervention hypothesis uses only a lower-limit value.
Post Hoc Analyses of Primary Outcomes
Of the 90 participants, 46 (51%) met criteria for being IPS nonresponders—46% (N=25) of the IPS+VR-JIT group and 58% (N=21) of the IPS-as-usual group, which was not a statistically significant difference. By definition, IPS nonresponders have lower rates of obtaining employment when engaged in IPS, compared with new enrollees in IPS (37). As noted, the inclusion criteria for this study were modified from an initial focus on recent IPS enrollees to all unemployed IPS clients (i.e., including IPS nonresponders). This inclusion raises the question whether the effects of VR-JIT differed between the recent IPS enrollees and the nonresponders. Thus, we conducted a series of post hoc analyses to evaluate the impact of VR-JIT on employment outcomes separately for the IPS nonresponder subgroup and the recent IPS enrollee subgroup. The first set of analyses were for IPS nonresponders. A chi-square analysis among IPS nonresponders indicated that the IPS+VR-JIT group was significantly more likely than the IPS-as-usual group to obtain employment by 9-month follow-up (52% [N=13] versus 19.0% [N=4]; χ2=5.32, df=1, p=0.021). Subsequently, the addition of the same aforementioned covariates in a multivariable logistic regression within the IPS nonresponder subsample indicated that the IPS+VR-JIT group had greater odds of employment, compared with the IPS-as-usual group (OR=5.82, p=0.014; see table in online supplement). Finally, the adjusted Cox proportional hazards model among the IPS nonresponders (using the same covariates) indicated that the IPS+VR-JIT group had a significantly shorter time to employment, compared with the IPS-as-usual group (hazard ratio=2.70, p=0.044; see table in online supplement) (Figure 1).
FIGURE 1. Cumulative hazard plot of time to employment among IPS nonresponders, by study groupa
aTime to employment was measured as days from randomization to employment. Participants were randomly assigned to individual placement and support as usual (IPS as usual) or to IPS combined with Virtual Reality Job Interview Training (IPS+VR-JIT). Data were right-censored after 9 months (274 days), but no new employment occurred after day 235. A nonresponder was defined as a person who received IPS services for at least 90 days without obtaining a job or who obtained a job but lost it (was fired or quit) within 90 days.
The second set of analyses were for recent IPS enrollees. A chi-square analysis for recent IPS enrollees indicated that the IPS+VR-JIT group was not more likely to obtain employment by 9-month follow-up, compared with the IPS-as-usual group (34.5% [N=10] versus 40% [N=6]). Subsequently, the addition of the aforementioned covariates in a multivariable logistic regression for the recent IPS enrollee group indicated that the IPS+VR-JIT group did not differ regarding odds of employment, compared with the IPS-as-usual group (see table in online supplement). Finally, the adjusted Cox proportional hazards model for the recent IPS enrollees (using the aforementioned covariates) indicated that the IPS+VR-JIT group did not have a significantly shorter time to employment, compared with the IPS-as-usual group (see table and figure in online supplement). (The online supplement also includes a correlation matrix of the variables considered for the post hoc analyses among the IPS nonresponders and recent IPS enrollees.)
Secondary Outcomes
Longitudinal intent-to-treat analyses of secondary outcomes are shown in Table 4, including estimated marginal means for each group at pre- and posttest, results of the mixed-effects regression models (that included the same covariates from the primary analyses—baseline year of study participation, problematic substance use, social cognition, community functioning, and negative symptoms [anergia]), and longitudinal effect sizes (d) (66). Three models revealed significant group-by-time interactions that confirmed hypothesized greater improvements in interview skills (estimate±SE=1.62±0.62, p=0.006, d=0.33), interview self-confidence (estimate±SE=4.48±1.99, p=0.013, d=0.42), and decline in interview anxiety (estimate±SE=−8.21±3.90, p=0.019, d=−0.31) in the IPS+VR-JIT group, compared with the IPS-as-usual group. The model evaluating changes in general social competence did not reveal a significant group-by-time interaction.
| Estimated marginal score | Regression analysisa | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| IPS+VR-JIT (N=54) | IPS as usual (N=36) | Model | |||||||
| Variable and timeb | M | SE | M | SE | parameter | Estimate | SE | pc | dd |
| Interview skillse | .33 | ||||||||
| Pretest | 25.6 | .6 | 24.6 | .8 | Group | .98 | 1.00 | .331 | |
| Posttest | 27.4 | .6 | 24.8 | .8 | Time | .17 | .47 | .723 | |
| Group*time | 1.62 | .62 | .006 | ||||||
| Interview self-confidencef | .42 | ||||||||
| Pretest | 43.0 | 1.5 | 43.7 | 1.7 | Group | −.68 | 2.23 | .762 | |
| Posttest | 47.6 | 1.6 | 43.8 | 1.8 | Time | .17 | 1.50 | .908 | |
| Group*time | 4.48 | 1.99 | .013 | ||||||
| Interview anxietyg | –.31 | ||||||||
| Pretest | 100.2 | 3.6 | 90.5 | 4.5 | Group | 9.69 | 5.83 | .100 | |
| Posttest | 90.4 | 3.8 | 88.9 | 4.6 | Time | –1.63 | 2.94 | .582 | |
| Group*time | –8.21 | 3.90 | .019 | ||||||
| Social competenceh | –.08 | ||||||||
| Pretest | 3.71 | .1 | 3.81 | .1 | Group | −.09 | .13 | .463 | |
| Posttest | 3.76 | .1 | 3.90 | .1 | Time | .10 | .09 | .294 | |
| Group*time | −.05 | .12 | .692 | ||||||
a
Mixed-effects linear regression models (N=90). For all models, covariates included were the same as in the primary analyses: baseline year, problematic substance use, social cognition, community functioning, and negative symptoms (anergia).
b
Pretest, baseline (before randomization); posttest, approximately 3 months later.
c
One-sided p value for directional intervention by time hypothesis for interview outcomes; two-sided p value for remaining effects.
d
Longitudinal effect size.
e
f
g
As measured by the adapted Personal Report of Public Speaking Apprehension (45). Possible scores range from 34 to 170, with higher scores indicating greater interview anxiety.
h
As measured by the Social Skills Performance Assessment (46). Possible scores range from 1 to 4, with higher scores indicating better social competence.
Discussion
Job interview skills are a critical contributor to employment (47). IPS supported employment is the most effective psychiatric rehabilitation approach to promote competitive work among individuals with serious mental illness (7). However, the impact of training IPS participants in interviewing skills has not been evaluated. VR-JIT was developed as an efficient approach to interview training by providing clients with the opportunity to practice their interviewing skills and obtain real-time feedback through a computer-based virtual platform. Our primary results were not definitive, because the unadjusted results were nonsignificant and statistically underpowered because of COVID-19 pandemic–related recruitment shortfall. However, adjusting the model for covariates that were theoretically and empirically associated with employment indicated that participants randomly assigned to IPS+VR-JIT were significantly more likely than those in the IPS-as-usual group to obtain employment within the 9-month study period. A similar pattern was noted for the Cox proportional hazards model, in which the unadjusted analysis yielded a nonsignificant effect, whereas the adjusted results indicated a trend that IPS+VR-JIT participants had a shorter time to employment, compared with IPS-as-usual participants. Overall, these results are consistent with prior efficacy trials of VR-JIT among individuals with serious mental illness who were not engaged in vocational rehabilitation (26, 27, 29).
We know that approximately 45% of IPS clients are unable to obtain employment (i.e., IPS nonresponders) (7) and that additional supports or tools may improve employment outcomes for this group. Our post hoc analyses suggested that IPS nonresponders who received VR-JIT had significantly greater odds of employment and reduced time to employment (Figure 1), compared with nonresponders in the IPS-as-usual group. The results demonstrating that IPS nonresponders seem to benefit from VR-JIT are consistent with other lines of research also suggesting that individuals who do not initially benefit from IPS may be able to secure employment with adjunct services integrated within IPS (e.g., cognitive remediation) (37). Overall, our change in eligibility criteria enabled us to detect that VR-JIT may enhance employment outcomes among IPS nonresponders while concurrently providing insight that not all IPS clients may benefit from additional interview training. Notably, the effects of VR-JIT on IPS nonresponders was not hypothesized and requires replication.
We also found that VR-JIT facilitated improved interview skills, interview self-confidence, and interview anxiety for participants, which is consistent with prior trials evaluating VR-JIT among adults with serious mental illness (25–31) and an adapted VR-JIT among autistic transition-age youths (50). These findings have important implications, because self-confidence is known to improve interview performance (68), and high levels of anxiety can disrupt interview performance (16, 69, 70). Although IPS+VR-JIT participants improved their interviewing skills, no improvement was shown for general social competence, which is consistent with research and clinical recommendations that social skills training may improve specific targeted areas of social functioning but has limited generalizability to other areas (71).
Implications for Practice
Given the focus on rapid job search in IPS, employment specialists may or may not work with clients on interview skills preceding real-world job interviews. Further, there is limited information from prior studies about how many IPS clients practice job interview role plays. Notably, we observed that approximately 47% (N=17) of participants in the IPS-as-usual group completed 1.8±1.2 practice job interviews with their employment specialists. To the best of our knowledge, this is the first study to report on the frequency of mock interviews occurring within IPS. That said, employment specialists are not currently trained in established social skills methods for teaching interview skills. With VR-JIT’s use of behavioral learning principles, including scaffolding and performance feedback, the intervention’s pedagogy is consistent with evidence-based social skills training approaches (71).
As noted in this study, engagement with VR-JIT combined with high-fidelity IPS significantly reduced time to obtaining competitive employment for IPS nonresponders. Further, IPS employment specialists spent 38.8±41.9 minutes discussing virtual interviewing progress with IPS+VR-JIT participants, indicating that employment specialists could naturally integrate VR-JIT into the IPS service structure. Thus, the enhancement of IPS with VR-JIT addresses an area of need—namely, poor interview skills, which could be contributing to a diminished response to IPS for some clients. Improved interview skills could serve to bolster work outcomes, not only in IPS but also in a variety of highly utilized vocational rehabilitation services in the United States and internationally.
Because a large proportion of participants engaged in IPS as usual completed job interview role plays with their employment specialists, providing employment specialists with evidence-based tools to facilitate and standardize this training could bolster their practices. Thus, VR-JIT could potentially offer employment specialists a tool to assess and guide interview skills development for IPS nonresponders without affecting IPS fidelity, serving to enhance the impact of IPS in much the same way as motivational interviewing and cognitive remediation strategies. For example, employment specialists could train clients to use VR-JIT independently and then process transcripts and summary performance assessments with them. Alternatively, employment specialists could review clients’ VR-JIT performance assessments and transcripts prior to sessions with clients and then use the sessions to reinforce their clients’ interviewing strengths and focus in greater depth on interview skills that need improvement.
Although the data are preliminary and future studies are needed, this more naturalistic integration of VR-JIT with employment specialist support generated the best employment rates among all study participants: mock interviewing with the employment specialist plus VR-JIT, 56% (N=9); mock interviewing with the employment specialist alone, 40% (N=15); VR-JIT alone, 40% (N=43); and no mock interviewing with the employment specialist or VR-JIT, 24% (N=17). Finally, Internet-based programs, such as VR-JIT, provide IPS teams and individuals with serious mental illness flexible access and use of tools that can facilitate practice at their convenience and pace and could facilitate engagement in IPS for individuals with serious mental illness who face unexpected barriers to face-to-face meetings (e.g., inclement weather and transportation barriers), although future research is needed to examine potential barriers more carefully.
Limitations
The findings must be considered within the study limitations. First, the observed employment rates in both groups were lower than anticipated when considering other studies evaluating high-fidelity IPS (6). These lower-than-anticipated rates can potentially be explained by two factors. The first was that 51% (N=46) of the main sample included IPS nonresponders (i.e., individuals who did not benefit from IPS). The second was that 49% of the main sample were recent IPS enrollees, and one could estimate that 45% of these participants would become IPS nonresponders during the course of the study (based on historical IPS employment rates) (7). Second, overall recruitment challenges and the COVID-19 pandemic contributed to a smaller sample and significantly lower analytical power than anticipated. Notably, 24% of our sample (N=22) had at least 1 month of follow-up during the COVID-19–imposed quarantine period (March 2020 through October 2020). That said, 50% (N=11) of these participants obtained employment, which was consistent with the 45% employment rate at Thresholds during the quarantine. Thus, the pandemic appears to have had minimal impact on job attainment in this study.
A third limitation is that employment specialists were asked to refrain from mock interview training with IPS+VR-JIT participants and were not masked to study condition. This design may have facilitated bias in how the employment specialists treated participants in both groups. That said, we did not see a difference between groups with respect to the extent that employment specialists discussed job interview skills with the participants. This lack of a difference could reflect minimal bias or a potential increase in job interview skill discussion in the IPS-as-usual group. Fourth, findings regarding interview confidence and anxiety were limited to self-report. Finally, VR-JIT was primarily implemented by researchers, which may limit our understanding of its effectiveness when implemented by IPS staff. However, VR-JIT was delivered by using strategies and procedures recommended by IPS program administrators and in settings where participants typically met with their employment specialists to mirror real-world delivery.
Future Directions
Additional research is needed to help understand why VR-JIT may be effective in helping IPS nonresponders. For instance, research is needed to evaluate differential effectiveness of VR-JIT among IPS nonresponders and its mechanisms of enhanced interview skills and employment. Also, although VR-JIT was primarily implemented in the community by researchers, our hybrid type I design included an initial process evaluation of VR-JIT implementation feasibility, acceptability, and usability and potential barriers to future implementation—and that report is forthcoming. Meanwhile, a future multilevel implementation evaluation of VR-JIT is needed to assess how VR-JIT is delivered by IPS teams to IPS nonresponders while maintaining the core functions of high-fidelity IPS and how VR-JIT fits into the IPS workflow and progress note for services billing to further enhance and sustain implementation. Finally, agencies and interested implementers would benefit from understanding the cost-effectiveness of VR-JIT.
Conclusions
Vocational rehabilitation in general could benefit from an evidence-based practice targeting job interview skills for individuals with serious mental illness. VR-JIT appears to be a valid option to fill this service gap, given that the combination of high-fidelity IPS and VR-JIT was associated with significantly better employment outcomes among individuals with a history of being nonresponsive to IPS. In addition, VR-JIT trainees more broadly had significantly improved interview skills, interview confidence, and interview anxiety, which speaks to its effectiveness targeting these outcomes. Although more research is needed to understand why VR-JIT is effective and to identify optimal implementation strategies, these results may support the use of VR-JIT in current IPS programs in the United States and internationally.
Acknowledgments
The authors thank the members of Thresholds who participated in the study and acknowledge the more than 50 administrators, research staff, and employment staff at Thresholds in Chicago who diligently supported various aspects of the study implementation. They also thank the research staff at the University of Michigan Level Up: Employment Skills Simulation Lab (http://leveluplab.org/) for data collection, processing, and management and the research staff at the Mental Health Services and Policy Program, Department of Psychiatry and Behavioral Sciences, Northwestern University, for institutional review board regulatory support.
Footnote
Preliminary results from this trial were presented at the annual conference of the Society for Social Work and Research, Washington, D.C., January 15–19, 2020; the annual conference of the Schizophrenia International Research Society, Orlando, Florida, April 10–14, 2019, and the National Institute of Mental Health Mental Health Services Research Conference, Rockville, Maryland, August 1–2, 2018.
Supplementary Material
File (appi.ps.202100516.ds001.pdf)
- View/Download
- 984.42 KB
References
1.
Drake RE, Whitley R: Recovery and severe mental illness: description and analysis. Can J Psychiatry 2014; 59:236–242
2.
Marwaha S, Johnson S: Schizophrenia and employment—a review. Soc Psychiatry Psychiatr Epidemiol 2004; 39:337–349
3.
Rüesch P, Graf J, Meyer PC, et al: Occupation, social support and quality of life in persons with schizophrenic or affective disorders. Soc Psychiatry Psychiatr Epidemiol 2004; 39:686–694
4.
Becker DR, Swanson SJ, Reese SL, et al: Evidence-Based Supported Employment Fidelity Review Manual. Lebanon, NH, Westat, 2015
5.
Bond GR: Principles of the Individual Placement and Support model: empirical support. Psychiatr Rehabil J 1998; 22:11–23
6.
Frederick DE, VanderWeele TJ: Supported employment: meta-analysis and review of randomized controlled trials of individual placement and support. PloS One 2019; 14:e0212208-e
7.
Bond GR, Drake RE, Becker DR: An update on Individual Placement and Support. World Psychiatry 2020; 19:390–391
8.
Hollandsworth JG, Kazelskis R, Stevens J, et al: Relative contribution of verbal articulative and non-verbal communication to employment decisions in the job interview setting. Person Psychol 1979; 32:359–367
9.
Kacmar KM, Delery JE, Ferris GR: Differential effectiveness of applicant impression management tactics on employment interview decisions. J Appl Soc Psychol 1992; 22:1250–1272
10.
Macan T: The employment interview: a review of current studies and directions for future research. Person Psychol 2009; 55:1–81
11.
Posthuma RA, Morgeson FP, Campion MA: Beyond employment interview validity: a comprehensive narrative review of recent research and trends over time. Person Psychol 2002; 55:1–81
12.
Charisiou J, Jackson HJ, Boyle GJ, et al: Which employment interview skills best predict the employability of schizophrenic patients? Psychol Rep 1989; 64:683–694
13.
Solinski S, Jackson HJ, Bell RC: Prediction of employability in schizophrenic patients. Schizophr Res 1992; 7:141–148
14.
Venardos MG, Harris MB: Job interview training with rehabilitation clients: a comparison of videotape and role-playing procedures. J Appl Psychol 1973; 58:365–367
15.
Farley RC, Hinman S: Teaching rehabilitation clients effective job interview skills. Vocat Eval Work Adjust Bull 1988; 21:157–160
16.
McCarthy J, Goffin R: Measuring job interview anxiety: beyond weak knees and sweaty palms. Person Psychol 2004; 57:607–637
17.
Bellack AS, Morrison RL, Wixted JT, et al: An analysis of social competence in schizophrenia. Br J Psychiatry 1990; 156:809–818
18.
Achim AM, Maziade M, Raymond E, et al: How prevalent are anxiety disorders in schizophrenia? A meta-analysis and critical review on a significant association. Schizophr Bull 2011; 37:811–821
19.
Charisiou J, Jackson HJ, Boyle GJ, et al: Are employment-interview skills a correlate of subtypes of schizophrenia? Psychol Rep 1989; 65:951–960
20.
Jacobs HE, Collier R, Wissusik D: The job-finding module: training skills for seeking competitive community employment. New Dir Ment Health Serv 1992; 53:105–115
21.
Helping People With Criminal Histories Find Work: Tips for Employment Specialists. Lebanon, NH, Expert Employment Specialist Group, IPS Learning Community, 2010. https://ipsworks.org/wp-content/uploads/2017/08/Tips-For-Employment-Specialists.pdf
22.
Gaining Employment: A Collection of Resources. Lebanon, NH, Expert Employment Specialist Group, IPS Learning Community, 2014. https://ipsworks.org/wp-content/uploads/2017/07/Gaining-Employment-A-Collection-of-Resources.pdf
23.
Supported Employment: Training Frontline Staff. Rockville, MD, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, 2009
24.
Swanson SJ, Becker DR, Drake RE, et al: Supported Employment: A Practical Guide for Practitioners and Supervisors. Lebanon, NH, Dartmouth Psychiatric Research Center, 2008
25.
Smith MJ, Ginger EJ, Wright K, et al: Virtual reality job interview training in adults with autism spectrum disorder. J Autism Dev Disord 2014; 44:2450–2463
26.
Smith MJ, Ginger EJ, Wright K, et al: Virtual reality job interview training for individuals with psychiatric disabilities. J Nerv Ment Dis 2014; 202:659–667
27.
Smith MJ, Boteler Humm L, Fleming MF, et al: Virtual reality job interview training for veterans with posttraumatic stress disorder. J Vocat Rehabil 2015; 42:271–279
28.
Smith MJ, Fleming MF, Wright MA, et al: Job offers to individuals with severe mental illness after participation in virtual reality job interview training. Psychiatr Serv 2015; 66:1173–1179
29.
Smith MJ, Fleming MF, Wright MA, et al: Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment. Schizophr Res 2015; 166:86–91
30.
Smith MJ, Bell MD, Wright MA, et al: Virtual reality job interview training and 6-month employment outcomes for individuals with substance use disorders seeking employment. J Vocat Rehabil 2016; 44:323–332
31.
Smith MJ, Smith JD, Fleming MF, et al: Mechanism of action for obtaining job offers with virtual reality job interview training. Psychiatr Serv 2017; 68:747–750
32.
Curran GM, Bauer M, Mittman B, et al: Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012; 50:217–226
33.
Smith MJ, Smith JD, Fleming MF, et al: Enhancing Individual Placement and Support (IPS)–supported employment: a Type 1 hybrid design randomized controlled trial to evaluate virtual reality job interview training among adults with severe mental illness. Contemp Clin Trials 2019; 77:86–97
34.
First MB, Williams JBW, Karg RS, et al: Structured Clinical Interview for DSM-5 Disorders: Research Version (SCID-5-RV). Washington, DC, American Psychiatric Press, 2015
35.
Wilkinson GS, Robertson GJ: Wide Range Achievement Test, 5th ed. Bloomington, MN, Pearson, 2017
36.
Posner K, Brown GK, Stanley B, et al: The Columbia–Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry 2011; 168:1266–1277
37.
McGurk SR, Mueser KT, Xie H, et al: Cognitive enhancement treatment for people with mental illness who do not respond to supported employment: a randomized controlled trial. Am J Psychiatry 2015; 172:852–861
38.
Lukoff D, Nuechterlein K, Ventura J: Manual for the expanded Brief Psychiatric Rating Scale. Schizophr Bull 1986; 12:594–602
39.
Mueser KT, Curran PJ, McHugo GJ: Factor structure of the Brief Psychiatric Rating Scale in schizophrenia. Psychol Assess 1997; 9:196–204
40.
Saunders JB, Aasland OG, Babor TF, et al: Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons With Harmful Alcohol Consumption–II. Addiction 1993; 88:791–804
41.
Skinner HA: The Drug Abuse Screening Test. Addict Behav 1982; 7:363–371
42.
Nuechterlein KH, Green MF, Kern RS, et al: The MATRICS Consensus Cognitive Battery: part 1. test selection, reliability, and validity. Am J Psychiatry 2008; 165:203–213
43.
Mayer JD, Salovey P, Caruso DR: Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT) User’s Manual. Toronto, MHS, 2002
44.
Schneider LC, Struening EL: SLOF: a behavioral rating scale for assessing the mentally ill. Soc Work Res Abstr 1983; 19:9–21
45.
McCroskey JC: Measures of communication-bound anxiety. Speech Monogr 1970; 37:269–277
46.
Patterson TL, Moscona S, McKibbin CL, et al: Social skills performance assessment among older patients with schizophrenia. Schizophr Res 2001; 48:351–360
47.
Huffcutt AI: An empirical review of the employment interview construct literature. Int J Sel Assess 2011; 19:62–81
48.
Mueser KT, Gottlieb JD, Xie H, et al: Evaluation of cognitive restructuring for post-traumatic stress disorder in people with severe mental illness. Br J Psychiatry 2015; 206:501–508
49.
Smith MJ, Fleming MF, Wright MA, et al: Vocational outcomes for young adults with autism spectrum disorders at six months after virtual reality job interview training. J Autism Dev Disord 2015; 45:3364–3369
50.
Smith MJ, Sherwood K, Ross B, et al: Virtual interview training for autistic transition age youth: a randomized controlled feasibility and effectiveness trial. Autism 2021; 25:1536–1552
51.
Smith MJ, Smith JD, Jordan N, et al: Virtual reality job interview training in transition services: results of a single-arm, noncontrolled effectiveness-implementation hybrid trial. J Spec Educ Technol 2021; 36:3–17
52.
Campbell K, Bond GR, Drake RE, et al: Client predictors of employment outcomes in high-fidelity supported employment: a regression analysis. J Nerv Ment Dis 2010; 198:556–563
53.
McGurk SR, Drake RE, Xie H, et al: Cognitive predictors of work among Social Security Disability Insurance beneficiaries with psychiatric disorders enrolled in IPS supported employment. Schizophr Bull 2018; 44:32–37
54.
Depp CA, Mausbach BT, Harmell AL, et al: Meta-analysis of the association between cognitive abilities and everyday functioning in bipolar disorder. Bipolar Disord 2012; 14:217–226
55.
Bell M, Tsang HW, Greig TC, et al: Neurocognition, social cognition, perceived social discomfort, and vocational outcomes in schizophrenia. Schizophr Bull 2009; 35:738–747
56.
Couture SM, Penn DL, Roberts DL: The functional significance of social cognition in schizophrenia: a review. Schizophr Bull 2006; 32(suppl 1):S44–S63
57.
Eack SM, Hogarty SS, Greenwald DP, et al: Cognitive enhancement therapy for adult autism spectrum disorder: results of an 18-month randomized clinical trial. Autism Res 2018; 11:519–530
58.
Charette-Dussault É, Corbière M: An integrative review of the barriers to job acquisition for people with severe mental illnesses. J Nerv Ment Dis 2019; 207:523–537
59.
Llerena K, Reddy LF, Kern RS: The role of experiential and expressive negative symptoms on job obtainment and work outcome in individuals with schizophrenia. Schizophr Res 2018; 192:148–153
60.
Gilbert E, Marwaha S: Predictors of employment in bipolar disorder: a systematic review. J Affect Disord 2013; 145:156–164
61.
Razzano LA, Cook JA, Burke-Miller JK, et al: Clinical factors associated with employment among people with severe mental illness: findings from the employment intervention demonstration program. J Nerv Ment Dis 2005; 193:705–713
62.
Saavedra J, López M, González S, et al: Cognitive and social functioning correlates of employment among people with severe mental illness. Community Ment Health J 2016; 52:851–858
63.
Cook JA, Razzano LA, Burke-Miller JK, et al: Effects of co-occurring disorders on employment outcomes in a multisite randomized study of supported employment for people with severe mental illness. J Rehabil Res Dev 2007; 44:837–849
64.
Cook J, Blyler C, Burke-Miller J, et al: Effectiveness of supported employment for individuals with schizophrenia: results of a multi-site, randomized trial. Clin Schizophr Relat Psychoses 2008; 2:37–46
65.
Cook JA, Mulkern V, Grey DD, et al: Effects of local unemployment rate on vocational outcomes in a randomized trial of supported employment for individuals with psychiatric disabilities. J Vocat Rehabil 2006; 25:71–84
66.
Feingold A: A regression framework for effect size assessments in longitudinal modeling of group differences. Rev Gen Psychol 2013; 17:111–121
67.
Ghasemi A, Zahediasl S: Normality tests for statistical analysis: a guide for non-statisticians. Int J Endocrinol Metab 2012; 10:486–489
68.
Tay C, Ang S, Van Dyne L: Personality, biographical characteristics, and job interview success: a longitudinal study of the mediating effects of interviewing self-efficacy and the moderating effects of internal locus of causality. J Appl Psychol 2006; 91:446–454
69.
Feiler AR, Powell DM: Behavioral expression of job interview anxiety. J Bus Psychol 2016; 31:155–171
70.
Rockawin D: Using innovative technology to overcome job interview anxiety. Aust J Career Dev 2012; 21:46–52
71.
Bellack AS, Mueser KT, Gingerich S, et al: Social Skills Training for Schizophrenia: A Step-by-Step Guide, 2nd ed. New York, Guilford, 2004
Information & Authors
Information
Published In
History
Received: 23 August 2021
Revision received: 7 October 2021
Revision received: 10 November 2021
Accepted: 6 December 2021
Published online: 17 February 2022
Published in print: September 01, 2022
Keywords
Authors
Funding Information
This study was supported by research grant R01 MH110524 to Dr. M. J. Smith from the National Institute of Mental Health.No authors will receive royalties on the sales of the Virtual Reality Job Interview Training (VR-JIT) intervention that was the focus of this study. The University of Michigan will receive royalties from SIMmersion LLC on sales of an adapted version of VR-JIT that was tailored for transition-age youths with autism. These royalties will be shared with Dr. M. J. Smith and the University of Michigan School of Social Work. The other authors report no financial relationships with commercial interests.
Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).