Relationship Between Self-Assessed Fidelity and Self-Reported Employment in the Individual Placement and Support Model of Supported Employment

Our work facilitating the implementation of IPS in programs across NYS by using learning collaboratives has been described previously (13). For the study presented here, 78 of the 98 community sites that joined the IPS learning collaborative initiative in 2016 provided self-reported employment data and self-assessed fidelity data. This IPS learning collaborative includes 34 outpatient mental health clinics operating in state psychiatric hospitals and 44 outpatient community programs that provide personalized, recovery-oriented services (PROS) to promote functional recovery for individuals with serious mental illness and are operated by local nonprofit agencies. Although both types of programs are found in the community, they differ in organizational structure (clinic appointments versus rehabilitation milieu) and governance (state government versus nonprofit agencies). The 78 community sites represent 67% (N=34) of 51 state clinic sites and 48% (N=44) of 92 PROS sites in the state in 2016. Of the 20 sites that were not included in this study, six sites were not providing IPS at the time and 14 did not provide fidelity assessments. It should be noted that there were no statistically significant differences in one-month employment rates and in geographic region of the state between the 78 programs in the study presented here and the 14 programs that were excluded for lack of fidelity assessments.

Site program staff—that is, employment staff and supervisors—provided self-reported fidelity assessments by using the IPS Supported Employment Fidelity Scale (https://ipsworks.org/wp-content/uploads/2017/08/IPS-Fidelity-Scale-Eng1.pdf). The IPS learning collaborative initiative emphasizes fidelity to the IPS model. In the learning collaborative, webinars and program-specific consultation calls and visits are used to teach program staff about IPS fidelity and how to self-assess fidelity. Each year, program staff receive one to three hours of training that is specifically focused on fidelity to the IPS model, along with general IPS training that also focus on fidelity (14). Fidelity self-assessments are completed annually by participating programs. In a recent study comparing self-assessed fidelity to independent ratings (14), we found agreement between these two rating methods when looking at programs’ total score on the IPS Supported Employment Fidelity Scale. However, there was variability in agreement for individual items.

Using an online survey via the Qualtrics software platform, participating programs provided monthly performance indicator data. This includes staffing allocations, caseload size, and competitive employment status of those receiving IPS. The employment data used in this study were reported for the month in which the fidelity assessment was conducted. During this month, all participating programs reported the number of people employed competitively. Competitive employment was defined as a community-based job that pays at least minimum wage, is available to any person (that is, not created specifically for people with disabilities), may be full-time or part-time, and does not have time limits determined by the mental health agency. Each program also submitted the number of people on their caseload, defined as the number of individuals receiving at least one individual IPS service during the referenced time (that is, one or three months). One program type (PROS, N=44) also reported competitive employment in a three-month period (the month in which the fidelity measure was submitted and the two months before it) and provided two separate fidelity estimates (with baseline fidelity collected between December 2015 and June 2016, and follow-up fidelity collected in December 2016). We calculated employment rate by dividing the number of people who were competitively employed (one or three months) by the caseload.

To replicate the analyses in the two recent IPS learning community studies by Bond and colleagues (10,11), we used the Pearson correlation to examine our primary hypothesis, that the total fidelity score would correlate positively with the one-month competitive employment rate (all sites) and the three-month competitive employment rate (PROS sites only). Second, in order to examine correlations of individual items on the IPS Supported Employment Fidelity Scale with the one-month competitive employment rate, we used the Bonferroni correction to adjust the p value for multiple comparisons (.05/25=.002). Within PROS programs, we conducted a mixed-effects model to examine the relationship between repeated fidelity and employment measures over time. Specifically, the mixed-effect model takes into account the correlation between repeated measures when estimating the relationship between fidelity and employment measures. Analyses were conducted by using IBM SPSS Statistics, version 23.

Scores on the IPS Supported Employment Fidelity Scale range from 0 to 125, with scores of 100 to 114 representing good fidelity and scores of 115 to 125 representing high fidelity. Across the 78 programs, total fidelity scores ranged from 58 to 122 (mean±SD=93.76±12.80), with almost one-third of the programs (N=25, or 32%) achieving scores that indicated good to high fidelity. The one-month employment rate ranged from 0% to 100% (mean±SD=43%±25%). We found statistically significant correlations between total fidelity scores and the one-month employment rate (r=.331, p=.003). In examining individual fidelity items by using the Bonferroni correction, one item was significantly correlated with the one-month employment rate: diversity of jobs developed (r=.364, p=.001) (Table 1).

In examining the 44 PROS programs with three-month outcomes, the baseline total fidelity score averaged 91.5±14.7 and the three-month employment rate averaged 38%±23%, whereas the follow-up total fidelity score averaged 92.2±14.5 and the three-month employment rate averaged 43%±23%. Similar to one-month employment outcomes, we found significant correlations between total fidelity scores and three-month employment rate at both baseline (r=.366, p=.015), and at follow-up (r=.378, p=.025). In the mixed-effects model, examining this relationship over the two time points, the total fidelity score remained a significant contributor (F=4.01, df=1 and 68, p=.049) to the three-month employment rate, although the results did not change significantly between the two time points.

This study demonstrated a significant positive relationship between self-reported fidelity to the IPS model and employment, consistent with prior studies assessing programs participating in the IPS learning community (10,11) and with programs contracted by the Massachusetts DMH (12). We also found significant positive correlations between one item on the IPS Supported Employment Fidelity Scale and employment outcomes. Our study extends the findings of previous studies by demonstrating that the relationship between fidelity and employment was sustained over time (up to one year). Additionally, our study shows that the relationship between fidelity and employment was observed through agency self-reported fidelity—rather than fidelity assessed by an external reviewer.

As the demand, particularly among federal and state payers, for evidence-based practices grows, so does the demand for fidelity assessments (15). The study presented here not only provides additional confirmation for the hypothesis that high-fidelity programs have better employment outcomes and shows that these effects are sustained over time, but it also shows that programs supported by the NYS learning collaborative have shown employment rates that are similar to the original IPS learning community. This demonstrates the efficacy of the IPS model and its robustness to implementation by experts other than the original IPS developers. This replicability is key for demonstrating the overall efficacy of the IPS model. Our results also show the robustness of the model in two different program types (outpatient clinics operated by state psychiatric facilities and recovery-oriented programs operated by independent, community-based nonprofit agencies).

As in the two studies of the IPS learning community (10,11) we found that an individual item or items in the IPS fidelity assessment were significantly associated with employment. However, the specific items differed among the three studies. This indicates the importance of using the full fidelity instrument when assessing program quality indicators; no subset of items has been shown to consistently deliver comparable results.

The fidelity assessments in our study were self-reported by program sites; this can be considered both an asset and a limitation. Although external, independent assessment is an important accountability measure, it can be expensive and burdensome for sponsoring agencies to provide on an ongoing basis (14). In a prior study, we showed that agency self-report can yield fidelity scores that are similar to the ones by independent assessors within the NYS learning collaborative (14). Also, in the study presented here, our results indicate that high levels of self-reported IPS fidelity were associated with greater employment rates, much like the findings of earlier studies that used independent assessments to measure IPS fidelity (10–12).

This was an observational study that does not demonstrate causality. It is possible that programs with high levels of IPS fidelity have other characteristics that contribute to high employment rates among their clients (for example, additional services may be offered that also contribute to employment rates).

Future research should look at whether site characteristics such as program size or longevity are related to fidelity, employment outcomes, or both; examine why programs choose to invest in providing IPS services; identify the most significant fidelity items contributing to positive employment outcomes; consider employment outcomes beyond employment status, such as job tenure and hours worked each week; and identify mediators and moderators at the client and site levels that may influence program outcomes.

The study presented here provides further evidence about the importance of implementation fidelity for IPS-supported employment. Self-assessment of IPS fidelity by program sites was shown to be associated with higher employment rates, which are sustained over time. Employment is a normative activity for individuals with serious mental illness and can be an important component of their recovery. Employment can also provide both an income and a quantifiable way to participate in the community. Programs that offer high-fidelity IPS services can help clients achieve their goals.