Validation of the Revised Individual Placement and Support Fidelity Scale (IPS-25)

Descriptive statistics of the IPS-25 items sorted by item correlations with employment are shown in Table 1. Item distributions were negatively skewed. The item scores indicate that some aspects of fidelity, such as caseload size, exclusively vocational services, focus on competitive employment, and rapid job search, are well understood and widely implemented in IPS programs. Other facets of fidelity, such as frequency of job development, community-based services, zero exclusion of clients, and contact with the treatment team, are less widely implemented. The internal consistency (Cronbach's alpha) for the total scale was .88. With two exceptions, all the correlations between items and employment were positive, and eight were statistically significant. Items related to vocational generalists, disclosure of disability, individualized and time-unlimited supports, and the vocational unit were the most strongly correlated with employment.

The mean±SD for the IPS-25 total score was 101±13, with a range of 56 to 123. Overall, 52 sites (66%) achieved a fidelity score of 100 or more, the cut-off for high fidelity. The mean±SD competitive employment rate was 37%±13%, with a range of 0% to 63%.

As shown in Table 2, IPS-25 fidelity was significantly correlated with employment rate, as hypothesized. Contrary to our hypothesis, the local unemployment rate was unrelated to the competitive employment rate. Finally, as hypothesized, program longevity was positively correlated with both fidelity and the competitive employment rate. Multiple regression analysis entering three predictor variables (IPS-25, unemployment rate, and program longevity) and the competitive employment rate as the criterion found that the IPS-25 was the only significant predictor (partial correlation=.24, p<.05).

We also examined the fidelity-outcome relationship by using the fidelity benchmark classification (Table 3). The mean competitive employment rate increased monotonically from “IPS not implemented” (29%) to fair fidelity (32%), high fidelity (39%), and exemplary high fidelity (44%) (F=6.80, df=1 and 75, p=.01).

The findings provide preliminary evidence for the psychometric adequacy of the IPS-25. The IPS-25 had satisfactory internal consistency. As hypothesized, fidelity was significantly correlated with the employment rate, even after the analysis controlled for local unemployment rate and program longevity. We also found that program longevity was associated with both fidelity and competitive employment outcome. The longer sites had participated in the learning collaborative, the better their fidelity and the better their outcomes, which are two goals of the collaborative. Given the intentional effort to improve fidelity through periodic fidelity reviews, the findings of higher fidelity and higher rates of competitive employment for more established programs are consistent with the hypothesis that fidelity is associated with better outcome.

Thus the study reported here adds to the literature suggesting that higher fidelity to the IPS model is associated with better employment outcomes (11). It reinforces the importance of measuring IPS fidelity for both research and quality improvement. The descriptive data also clarify the extent to which IPS components have been fully adopted within field settings and suggest directions for further model development.

Notably, all participating sites aspired to high fidelity to the IPS model, leading to a restriction of range in fidelity scores, as reflected in the preponderance of sites rated in the fair fidelity range and higher. Predictive validity would have been increased had we included more sites with both very low fidelity and poor employment outcomes, as the program longevity correlations suggest. Consistent with the restriction-of-range interpretation, one IPS-15 study with a very strong fidelity-outcome correlation (r=.76) used a sample that included sites with a variety of non-evidence-based models (28).

One pertinent question is whether the predictive validity for the IPS-25 is equal to or exceeds that for the IPS-15. One basis for comparison is a review of IPS-15 studies, which found a mean correlation of .39 (range −.07 to .76) between the IPS-15 and competitive employment in six studies that reported this statistic (11). Thus the evidence is currently stronger for the predictive validity of the IPS-15 than for the IPS-25, justifying its continued use. But fidelity assessors familiar with both scales agree that the IPS-25 is a far more useful tool for providing feedback to sites for quality improvement as well as for educating staff about model principles, especially in the area of job development. From a scientific perspective, an optimal solution is to use both scales until further evidence accumulates.

The IPS learning collaborative has made fidelity assessment the linchpin of its approach to quality improvement. Its two critical assumptions are that IPS is effective and that the fidelity scale measures the key components that make it effective. Regarding the first assumption, the evidence from randomized controlled trials is overwhelming (5). The study reported here provides evidence that the second assumption has merit. Improving program fidelity from poor fidelity (a score of less than 74 on the IPS-25) to good fidelity (100 or more) might be expected to result in an average increase in the competitive employment rate in the range of 10%. Moreover, several studies have provided direct evidence that programs receiving appropriate fidelity consultation can increase from poor to good IPS fidelity over a relatively brief time—within one year in one study (29). A direct test is needed of the hypothesis of improved employment as a result of improved fidelity as measured by the IPS-25.

Woolf and Johnson (30) have framed the question of quality improvement strategy in terms of the break-even point for investing resources in better fidelity or in further enhancements of an evidence-based practice. Although this break-even point for the IPS model has not been determined, the results of this study suggest that measuring fidelity and making improvements based on these assessments is worthwhile. Fidelity assessments do not conflict with other management practices and are sensible steps to take before augmenting IPS with complex and costly interventions.

The local economy—as reflected in the local unemployment rate—has been shown to be associated with competitive employment outcomes for IPS programs in some studies (6,31,32). The lack of a significant association in this study is puzzling. From other sources we know that people with disabilities were differentially affected by the widespread hiring freezes during the recessionary period in which this study was conducted (33). More specifically, longitudinal data from the IPS learning collaborative suggest that the overall employment rate for the total sample was suppressed approximately 3% to 5% during this period (26). We speculate that some IPS programs developed strategies that counteracted poor economic conditions.

Study limitations include the fact that the IPS-25 has been introduced into the field only recently, and fidelity assessors may suggest further item refinements as experience accumulates. In our view, content validity based on actual use should be the primary guide to scale revision. Some IPS-25 items may eventually be discarded as not useful. Other items, such as the role of collaboration with the state vocational rehabilitation agency, are culture bound and inapplicable in countries with different legislation regarding mental health and rehabilitation services. Because of the recent dissemination of the IPS-25, another limitation is that assessors may be less reliable in administering it (even with a comprehensive fidelity manual); with more experience, fidelity assessment may increase in validity. A further limitation concerns the potential variability in the collection of the employment data. Despite steps taken to enhance data quality, reliability across numerous sources reporting the data was not ensured. On the other hand, one strength of this study is that it was conducted under routine practice conditions. As a practical matter, we are interested in the validity of a fidelity scale in the hands of people who are on the front line seeking to implement evidence-based practices. Another limitation is external validity; states and sites participating in the learning collaborative are primarily early adopters (34), and findings from the sites included in this volunteer sample may not generalize.

Fidelity is not the only factor affecting outcome; at most it explains 25% of the variance in outcome (35). Factors such as practitioner skills (36), cultural and societal factors, systems factors, and community factors exert powerful influences on outcomes. Future psychometric studies should include such factors as covariates.

A revised fidelity scale, the IPS-25, is being widely disseminated as a quality improvement tool for implementing evidence-based supported employment programs. The preliminary evidence for its predictive validity is promising. This first report on the IPS-25 also provides information to program leaders regarding the relative attainment of IPS components in actual practice.

The authors thank the state leaders and program leaders in the IPS Learning Collaborative who provided the data for this study.

The authors have received financial support from the Johnson & Johnson Office of Corporate Contributions.