Procedure
Project staff conducted a program evaluation over a 27-month project phase (January 2001 to March 2003) for five project sites. During this time, they systematically examined client background characteristics, competitive employment outcome rates, and client and stakeholder satisfaction. Satisfaction was measured by using an 11-item checklist—which included four items that were answered as yes, no, or partially and seven items that were measured with a 5-point Likert scale (1, excellent; 2, very good; 3, good; 4, fair, and 5, poor)—and by using an additional item regarding wait time. The stakeholders included occupational therapists, case managers, and team directors from the treatment teams and contract managers from the health authorities.
In addition, fidelity of implementation was assessed by using the 15-item Individual Placement and Support Fidelity Scale (
16). Each item is rated on a 5-point, behaviorally anchored scale, for a maximum possible score of 75. An item score of 5 indicates that the program has fully implemented the individual placement and support standard, a score of 4 indicates moderate implementation, and a score of 1 indicates the program has not met the standard at all. The first author conducted the fidelity assessment, using operational policies, employment data, and other information provided by the supported employment team. A total score of 66 or higher is considered good fidelity to individual placement and support.
Communities served
The communities served by the five teams differed in certain ways. The socioeconomic and educational levels of clients served by the West Side Vancouver team were relatively high. Although this environment fostered expectations of higher wages for work, motivation to work was sometimes undermined by financial support to clients from families.
The Northeast Vancouver team served a lower-income neighborhood of the city that had a high degree of transience. The vocational counselor had a number of clients whom she continued to follow even though they moved to distant neighborhoods outside the team's appointed area. This community also has a substantial Asian population. The individual placement and support program accommodated a large percentage of clients whose first language was not English by enlisting the help of mental health staff who spoke the client's native language.
The South Vancouver team served the area in the most southerly part of Vancouver. This community was culturally diverse with sizeable Indo-Canadian and Asian populations. These multicultural communities often have a strong work ethic. If this ethic is combined with a lack of understanding of mental illness, families can have difficulty understanding why the person is not already working. This team had a multicultural worker who provided a bridge to clients who did not speak English.
The Central Burnaby team served an urban area adjacent to Vancouver that had a mix of economic and multicultural communities.
Unlike the other sites, Maple Ridge is a small town that is surrounded by a semirural area and is located to the east of Vancouver and Burnaby. The vocational counselor of the Maple Ridge team reported that the challenge of identifying potential jobs was significant in a smaller economic market.
Sample
Over the project period, 249 clients were admitted to the individual placement and support program—129 during the first 15 months and 120 over the final 12 months. The 195 clients who were enrolled for six or more months in the individual placement and support program during the project constituted the primary focus of the outcome analysis. Forty-three clients were seen by the Westside Vancouver team, 54 by the Northeast Vancouver team, 26 by the South Vancouver team, 41 by the Central Burnaby team, and 31 by the Maple Ridge team. Among the 195 clients, 113 (58 percent) were women, 111 (57 percent) were 40 years or younger, and educational attainment varied from less than high school (48 clients, or 25 percent) to college graduates (62 clients, or 32 percent). Ninety-nine (51 percent) had a diagnosis of a schizophrenia spectrum disorder, 76 (39 percent) had never worked or had not worked in the past five years, and 67 (34 percent) did not speak English as a first language.
Project findings
Program fidelity. The individual placement and support fidelity score was 68, which suggested good implementation of the individual placement and support model. The single item that was scored below 4 was community-based services, which suggested that the amount of time that employment specialists spent outside of the office was suboptimal.
Competitive employment outcomes. Among the 195 clients in the sample, 27 (14 percent) were employed at the time of program entry. Among these 27 clients, 12 (44 percent) were working at the end of the project period, nine (33 percent) were working at the time they left the program, and the remaining six (22 percent) were still actively searching for a new job at the end of the project period.
Of the 168 clients in the sample who were unemployed at the time of program entry, 84 (50 percent) were competitively employed at some time during the study period. An additional 29 (17 percent) were still actively searching for a new job at the end of the project period, while 56 (33 percent) were terminated without obtaining competitive employment. The competitive employment rates were similar for the five treatment teams, ranging from 46 to 55 percent. At the end of the project period, 62 clients (37 percent) were competitively employed (ranging from 32 to 46 percent for the five teams). A total of 107 jobs were held by the 84 clients who entered the program unemployed and who were employed at any point during the project period. Of these 107 jobs, 79 (74 percent) were 15 hours or more a week and 33 (31 percent) were full-time jobs; 65 (61 percent) earned a monthly wage of at least $800 (Canadian). All the positions paid competitive wages.
Among the 84 clients who entered the program unemployed and who found work at some point during the project period, 62 (74 percent) did so within six months after program admission. Staff individualized their job search approaches: 19 clients (23 percent) received mostly hands-off consultative support, 35 (42 percent) received intensive coaching, and 30 (36 percent) received both intensive coaching and direct marketing to employers on the client's behalf.
Client satisfaction. In the summer of 2003, satisfaction surveys were sent to 251 clients involved in the pilot project, of which 53 (21 percent) were returned. For the item "How would you rate the overall service you received from this program?" 20 (40 percent) answered excellent, 14 (28 percent) answered very good, 11 (22 percent) answered good, three (6 percent) answered fair, and two (4 percent) answered poor. (Three responses were missing.) For the item "Were your employment goals met?" 21 (40 percent) indicated yes, 20 (38 percent) indicated partially, and 11 (21 percent) responded no. (One response was missing.) Thirty-three of 46 respondents (72 percent) indicated that they waited two weeks or less to first meet with their vocational counselor.
Stakeholder satisfaction. An independent facilitator was hired to gather stakeholder input during a workshop held in April 2003. Attending were 15 stakeholders—including case managers, occupational therapists, and program managers from the five mental health treatment teams—and contract managers from the health authority. Twelve stakeholders rated their overall satisfaction with individual placement and support programs as follows: one person (8 percent) was extremely satisfied, five (42 percent) were very satisfied, four (33 percent) were satisfied, one (8 percent) was somewhat satisfied, and one (8 percent) was not satisfied. The workshop focused on a range of specific issues related to employment specialists' integration onto treatment teams, the referral process, staff selection, documentation, and organizational structure. In each of these areas, specific recommendations were developed for incorporation into future quality improvements.