Use of Software for Tablet Computers to Promote Engagement With Supported Employment: Results From an RCT

This randomized controlled trial compared the efficacy of a tablet-based engagement intervention and a brochure in promoting initial contact between clients and employment specialists to discuss enrollment in IPS.

Two sites within the Thresholds Rehabilitation and Recovery Centers in Chicago hosted the study. The Institutional Review Boards of Thresholds and Dartmouth College approved the study, and participants gave written informed consent. Participants were recruited from July through September 2011. The Bridge North site is located in an urban area, and all services are delivered via community outreach. The South Suburbs site serves clients who live in urban and suburban settings and offers both clinic-based and community-delivered services. Clients were potentially eligible if they had not previously received IPS services yet were actively engaged in mental health services at one of the two Thresholds locations. Final eligibility also required a verbal expression of interest in working.

The intervention included seven segments of video (12 minutes). These videos describe IPS and address topics such as whether to tell employers about mental health challenges, Social Security benefits, vocational preferences, and work history. Between video segments, brief questionnaires elicited values, preferences, and concerns about enrolling in IPS and seeking employment. The intervention provided encouraging feedback tailored to specific user responses. At the end of the approximately 30-minute intervention, users viewed a report summarizing their responses and were given an opportunity to request an appointment with an employment specialist to discuss enrolling in IPS.

We developed the intervention iteratively, involving people with lived experience of severe mental illness as consultants. The design also addressed principles from the fields of multimedia learning and persuasive systems (10,11), the literature on barriers to employment for people who have a severe mental illness (2), and prior research on the design of software for this population (12). The intervention was designed to be used by clients on their own, to require no previous experience with computers, and to accommodate persons who prefer to acquire information by listening rather than reading. The intervention software offers users the option to have all text read aloud as the associated text is highlighted on the screen.

For the comparison condition, we designed a trifold color brochure with both factual and testimonial content. Because we could not apply multimedia or persuasive systems principles in the design of this paper-only control condition, we took care to create a professional-looking, eye-catching document. [A digital version of the brochure is available online as a data supplement to this report.]

In real-world implementations of IPS, mental health clients who are not enrolled in IPS, especially those who receive their services in the community, may receive little or no information about IPS. We felt that a comparison condition consisting of a brochure was a fair representative of services as usual, given the lack of existing routinized engagement services for IPS and the practical constraints on clinical staff and employment specialists, who may have little or no time for engagement-related activities.

Each participant in the study attended two study sessions separated by three days. The first study session began immediately after clients completed the informed consent process and decided to participate. During the first study session, the clients participated in baseline assessments of cognitive functioning, demographic characteristics, substance use (CAGE-AID questionnaire), and criminal justice involvement. Participants were then randomly assigned to receive either the tablet-based intervention (including the brochure) or the brochure only. We stratified randomization by site and recent substance use (yes or no). The first study session concluded with the scheduling of a follow-up session three days later.

During the second study session, participants completed study-designed assessments of their knowledge about IPS and their vocational optimism and were paid $30 for their participation. The vocational optimism scale and IPS knowledge test were pilot tested and revised prior to this study, but their psychometric properties were not formally evaluated. The vocational optimism scale includes 11 questions presented on a visual analog scale, and the IPS knowledge test includes ten true-or-false questions

We used a binary measure of engagement for the main hypothesis and summary scores on posttest-only assessments of knowledge and optimism for the secondary hypotheses. Engagement was defined as requesting and attending an in-person meeting with an employment specialist within 30 days postintervention. Participants could self-refer to IPS during either study session or at any time during the 30-day follow-up period.

Pearson’s chi square analyses tested whether the two groups differed in rates of engagement, and independent-samples t tests examined differences between the groups in knowledge and optimism. We also examined univariate associations between each outcome variable and age, sex, race-ethnicity, work history, problematic substance use, criminal justice involvement, and possible cognitive impairment. The variables that were significantly related to outcomes (engagement, knowledge, and optimism) were entered into multivariate logistic models examining predictors of each outcome.

The primary and secondary study hypotheses were rejected or accepted on the basis of the results of the multivariate models (p≤.05). The effect size for engagement was the ratio of the odds of engagement for the experimental group to the odds of engagement for the control group. The effect size for knowledge and optimism was standardized mean differences in summary scores for each instrument.

In 2.5 months of recruiting, clinicians informed 214 chart-eligible clients about the study. The primary author interviewed 55 (26%) of these clients about the study. Others were uninterested, ineligible, or unavailable. Of the interviewed clients, 45 (82%) gave informed consent and entered the study. Twenty-two participants were assigned at random to the experimental group and 23 to the control group. [A description of the participant characteristics is available in the online data supplement.]

Engagement was reported for 11 (50%) members of the experimental group and one (4%) member of the control group (odds ratio [OR]=22.95, 95% confidence interval=2.51–193.38).

After testing candidate moderators, we included sex, race-ethnicity, and site in the final multivariate logistic model predicting engagement (Table 1). This model yielded a pseudo r² of .529. The ORs indicated that study condition and site were significant predictors of engagement.

Knowledge about IPS and vocational optimism were not significantly associated with study condition.

Confirming our primary hypothesis, the results indicated that the computerized engagement intervention was more effective than a brochure at promoting engagement with IPS services. The results for the IPS knowledge and vocational-optimism questionnaires did not support hypotheses regarding these outcomes.

The results indicate that a brief computerized intervention can be effective at fostering initial contact between clients and employment specialists. This intervention helps people to act from an informed position with respect to participating in supported employment and pursuing work, and it empowers them to directly self-refer to these services. Participants in the control condition received the same factual information about supported employment in a brochure, but the intervention failed to promote engagement.

How would this intervention be delivered as part of routine service delivery? Our experience suggests that with careful attention to usability during software design, clients may not need human support to successfully use this intervention or similar tools. As the ubiquity of mobile computing devices increases and user interface conventions evolve, the ability of clients to use these tools independently can be expected to improve. Our hope is that a tablet-based suite of engagement-related software can be delivered to clients by community support staff simply by loaning devices to them. The software itself provides an electronic link between the client and the providers of specific services. Hardware and software security and privacy concerns loom as barriers, but it is possible to remotely disable and remove information from these devices, and their cost continues to decline.

We cannot explain the absence of significant differences between the two groups in knowledge or optimism. A broader set of validated instruments should be used in future research to identify the important active ingredients in engagement interventions such as the one studied here.

The difference in outcome between sites was also unexpected. Although we cannot conclusively explain this difference, we observed important differences in the way employment specialists at the two sites responded to participant requests for follow-up appointments.

The limitations of the study included the small sample, the absence of tests for various active ingredients of the computerized intervention, and the lack of data on sustained engagement with supported employment or employment itself. If the sample had been larger, we may have seen significant associations with other covariates, and we may have been better able to explain site differences.

Mobile computer–based engagement interventions can promote initial contact between clients and employment specialists. Further research should identify the necessary ingredients for sustained engagement in psychosocial interventions, extend this work to people who are wholly unengaged with the mental health system, and integrate engagement tools with other computer-based services.

This study was funded by the Dartmouth Psychiatric Research Center, Lebanon, New Hampshire.

The authors report no competing interests.