Modest Implementation Efforts, Modest Fidelity, and Modest Outcomes

It is terrific news that the Department of Veterans Affairs (VA) is disseminating a program model that arguably has the strongest evidence base of all the psychosocial practices for individuals with severe mental illness ( 1 ). It is not surprising that Dr. Rosenheck leads this effort; his contributions to implementing innovative practices in the VA system are enormous. However, I have concerns about the authors' conclusion that they have documented that "a sustained training program can be used to implement IPS in systems that have had little past experience with this approach [and the IPS program] was associated with improved employment outcomes."

This project falls short on three critical and interrelated facets. First, as the authors note, the training model for individual placement and support (IPS) was "modestly intensive." After a one-day training and weekly teleconferences in the first three months, the "sustained" training apparently consisted of one site-specific and one systemwide teleconference per month, which does not seem sufficient to ensure adequate implementation. In fairness, the training exceeded the intensity that my colleagues and I naively employed in a series of dissemination projects during the 1980s ( 2 ). As did others during this era, we had mixed results. We discovered that initial training typically had little impact (especially with staff turnover) and that written materials and occasional phone calls were inadequate for overcoming the many barriers to implementation. One provisional conclusion from recent dissemination studies is the importance of ongoing face-to-face contact by a trainer-consultant, because site visits usually lead to more accurate assessments and personalized interventions. In contrast to the VA project, the National Evidence-Based Practices Project used a far more intensive consultation approach, with up to two years of monthly face-to-face expert consultant contact. Even with this intensive consultant model, high fidelity was not uniformly achieved (McHugo GJ, Drake RE, Whitley R, et al, unpublished manuscript, 2007).

A second issue concerns the measurement of program fidelity. It is puzzling that the authors did not use the well-validated IPS-Fidelity Scale ( 3 ) but rather developed an abbreviated version for which there apparently are no norms, little psychometric data, and no second raters. It is impossible to judge the validity of the authors' ad hoc claim that 78% of programs had "acceptable" fidelity. My impression is that they are overly generous in this judgment. Clues that fidelity may have been problematic are details such as restriction of site implementation to a single employment specialist (not a vocational unit), sites' lack of "a structurally unified treatment team," reliance mostly on office-based client contacts, and continued use of compensated work therapy (CWT). It is difficult to implement IPS when a competing sheltered work program remains in place.

Third, the employment findings for this study were extremely modest. As the authors document, the absolute rates of competitive employment for IPS in this study were much smaller than those found in the literature, as were the differences between the IPS and control groups in this study. Although findings were statistically significant, the clinical significance of these differences is questionable.

Although I recognize this article's timely contributions to understanding the implementation process, my overall conclusion would be different. If we accept marginal improvements as what we can expect when we introduce evidence-based practices "in the real world," we do a disservice to the field. We set the bar too low. Readers may be misled to conclude that, under usual conditions, IPS may increase days of competitive work per month by perhaps only 9% (the improvement found for individuals with severe mental illness in this study). In addition, the site with lowest fidelity (an outlier) had the second-best employment outcomes. The discussion section speculates that other factors might compensate for low fidelity. Thus proponents of CWT and other traditional approaches might reason that their programs need not pay attention to evidence-based principles.

My interpretation of what this study shows is that modest implementation efforts lead to modest fidelity, which in turn leads to modest outcomes. We can do better.