Implementing Evidence-Based Interventions to Improve Vocational Recovery in Early Psychosis: A Quality-Improvement Report

For the model for change, we used the Breakthrough Series, a type of QIC that has been applied in many countries for various clinical problems, mostly in general medical care (24). Generally, this implementation approach has five central features: a focus on a specific aim, related to gaps between best and current practice; participation of multidisciplinary teams from multiple sites; a national advisory board with clinical experts; a model for improvement (i.e., setting targets, collecting data, and continuous feedback loops); and a quality improvement process with a series of structured activities in a given time frame (25). The specific mix of structured activities that was offered to the participating teams during the 2-year period of the Early Psychosis QIC is listed in Box 1. These implementation strategies had three levels of focus: professionals, patients, and organization.

Teams were selected after a general call to all the teams treating individuals with early psychosis that had participated in the study of Hendriksen-Favier and colleagues (4) and to teams participating in the Dutch Early Psychosis Network. Early psychosis in this project was defined to include the prodromal phase of psychosis, in which subclinical symptoms are present; FEP; and the 5-year period following FEP. Under our selection criteria, we sought multidisciplinary teams that had at least 25 patients with early psychosis in their caseload, were motivated to implement at least two of the three interventions proposed, had active leadership support, and could designate time for a local team coordinator.

Fidelity outcomes for each intervention were established by calculating the total mean, standard deviation, and range of all fidelity items for all participating teams and persons. For shared decision making, mean scores were established separately for patients and therapists. On the basis of prior studies, a cut-off score of ≥4 on the 5-point IPS Fidelity Scale was defined as good fidelity, ≥3 as moderate fidelity, and <3 as low fidelity (34). For cognitive remediation and shared decision making, mean scores were interpreted according to the normed proportions of the IPS Fidelity Scale. To facilitate this interpretation, all item scores were also standardized to the 1–5 scale range of the IPS Fidelity Scale.

Table 1 presents the baseline availability of interventions and the mean total fidelity scores for each intervention at month 10 and month 22 of the QIC. At month 22, teams reached moderate-to-high mean fidelity scores for each of the two or three interventions they selected, with a mean score of 3.2 on a 4-point scale for cognitive remediation, 3.7 on a 5-point scale for IPS, and 4.9 on a 6-point scale for shared decision making.

Item scores on the fidelity scales were analyzed to establish whether certain components of the interventions had been implemented with more fidelity than others. (Mean item scores for the three interventions at months 10 and 22 of the QIC are presented in the online supplement.)

For IPS, components that were implemented with high fidelity included integration within mental health care through team assignment (mean score IPS-Em=4.8 on a 5-point scale) and zero exclusion criteria (mean score IPS-Em=4.4/IPS-Ed=4.4). For cognitive remediation, components implemented with high fidelity included motivating patients (mean score=3.7 on a 4-point scale) and supporting patients with cognitive strategies (mean score=3.7). For shared decision making, therapists at month 10 reported a higher fidelity than did patients. Between the two assessment points, most therapist and patient scores converged.

Table 2 presents barriers and facilitators that were encountered during implementation of the three interventions and the implementation strategies used to address them. The barriers and facilitators identified concerned the interventions, the individuals (professionals), the inner setting (organization), the outer setting (society and patients’ needs), and the process. Most implementation strategies were included in the original mix of project activities. Some of the strategies were added in response to specific barriers.

Table 3 presents elements of the QIC that were rated as the most and least useful by means of a questionnaire with a 1–5 scale. Strategies perceived as most useful were the increased sense of urgency that occurred via presentation of a central action plan, interactive training sessions, strong local leadership, and monitoring and feedback on fidelity. Most teams (83%, N=10 of 12 teams) reported that implementation of the interventions had resulted in better treatment. Two-thirds (67%, N=8 of 12 teams) reported that they had accomplished a real breakthrough during the QIC, mainly by implementing IPS.

Results showed a large increase in the availability of IPS. This intervention was available in three of the participating teams at baseline and in all 14 teams at the end of the QIC. In addition, a high uptake was seen; 252 of the 332 patients in the QIC were engaged in IPS (76%). However, results showed a moderate (albeit a high-moderate) mean fidelity score for IPS of 3.7 on a 5-point scale at month 22. Implementing IPS with high adherence to the principles in the model can be challenging and time consuming, because it requires changes at the level of professionals and teams, as well as at the organizational level and even at the intersectoral level (8, 38). As found in earlier studies, our study found multiple barriers on all three of these levels. At the micro level of individuals and small groups, the therapists reported fear that competitive work might be an overambitious goal for patients and that it would have a negative impact on them (8, 39, 40). At the meso level of institutions, we saw a lack of leadership and commitment of upper management to create the required financial and practical conditions (such as low caseloads, an exclusive focus on IPS for IPS workers, and organization-wide IPS implementation) (17, 39). Lastly, at the macro level of society, we encountered a national employment policy that contradicted the IPS scheme (e.g., financial disincentives to employment because of the “benefit trap” and a risk for the employer because of a strict policy related to firing staff) and the requirement for funding from multisector finance systems (38, 39, 41).

Notably, the fidelity item scores for IPS-education were similar to the fidelity scores for IPS-employment. This is remarkable, because a specific focus on the educational part of IPS has not yet been fully developed. In fact, this QIC was one of the first initiatives undertaken with such a focus. The project showed that use of the IPS-education scale was feasible. However, despite the relatively high scores on fidelity, IPS workers reported that it was harder to attain educational goals, compared with employment goals; that it was more difficult to finance educational support; and that their team needed at least one educational specialist (26). For this reason, future research is needed on the content and predictive validity of the IPS-education items, and the practice of IPS-education will have to undergo substantial development.

The availability of cognitive remediation also increased substantially over the course of the study. Some type of cognitive remediation was available in three of the teams at baseline and in 13 teams at the end point. This increase in availability was partly attributable to the fact that the license for the digital program (CIRCuiTS) was provided during the QIC. The uptake was low, with 52 (16%) of the 332 patients in the QIC engaged in cognitive remediation. This limited uptake was not surprising, because the program for cognitive remediation was introduced in the context of a randomized trial. Up until month 20 of the QIC, a team could offer cognitive remediation to patients only if they matched inclusion criteria and were enrolled in the trial (see timeline in online supplement). Of the 64 patients included in both the QIC and the trial, 29 were randomly assigned to the experimental condition (IPS plus cognitive remediation) and 35 were randomly assigned to the active control condition (IPS plus computer games) (see diagram in online supplement). Therefore, the uptake of cognitive remediation was hindered by these inclusion criteria and by the chance of random assignment to the active control condition. Teams reported problems with the cognitive remediation software, but they also reported that patients’ cognitive skills could be improved easily. Teams achieved a mean proxy score indicating good fidelity—3.2 points on a 4-point scale. The good score on fidelity may be due to the fact that cognitive remediation can be implemented on an individual professional level.

The Deciding Together program was implemented to facilitate the process of shared decision making. The uptake of Deciding Together was low, with only 39 (12%) of the 332 patients in the QIC being engaged in this specific program. This low uptake appears to be correlated with poor usability and acceptability of the Deciding Together program. Teams reported that it had an unattractive interface and experienced multiple technical start-up problems related to matches with available Web browsers, log-in problems, inefficient reports produced by the application, incorrect translations, and problems with the switch to a new version of the program. These factors also resulted in a low availability; the Deciding Together program was available in six teams at the end point. However, on the basis of a modest number of fidelity scores, a mean score indicating good fidelity—4.9 on a 6-point scale—to general principles of shared decision making was achieved at the assessment at month 22. Thus therapists in teams that used the program appeared to conduct shared decision making with good adherence to the specific steps in the model.

Fidelity results for shared decision making indicated a gap between therapist and patient scores at month 10, but scores converged at month 22. Such a patient-therapist gap has previously been reported for shared decision making, indicating that professionals overestimate the extent of patient involvement, compared with the experience of patients (42, 43), and that the actual occurrence of shared decision making is limited when examined with an observer-based assessment (44, 45). Although professionals’ positive attitude toward shared decision making (46, 47) seemed to be a facilitator of implementation, two important barriers appeared to exist: insufficient knowledge of the exact definition and specific steps of shared decision making, together with a lack of the skills needed to perform these steps in daily practice. The converging fidelity scores in our study can thus be interpreted as progress, suggesting that therapists and patients had greater agreement on the definition of shared decision making, with therapists becoming more aware that there was room for improvement and patients experiencing increased sharing of decisions.

Apparently, this progress in fidelity to the shared decision–making model was accomplished despite the shortcomings of the Deciding Together program. Even though the uptake of this program was low because multiple teams did not use the program or stopped using it during the QIC, teams showed converging patient and therapist fidelity scores. Two other factors appear to have contributed to this progress: participation in the training on principles of shared decision making, including theory and role-plays concerning specific steps in shared decision making, as well as feedback from the fidelity scale assessing adherence to the specific steps in shared decision making.

Consistent with prior research (48), IPS scores indicated that within the 24 months of the QIC, sufficient implementation of any components of IPS that required structural changes at the team level was possible and attainable through a change in team policy or administrative mandate. For example, organizations could easily use team assignments to switch to integrating vocational rehabilitation and mental health care and to apply zero exclusion criteria. On the other hand, fidelity to the model was more challenging for components that required organizationwide commitment and changes. For example, it was highly challenging to organize IPS coaches from multiple teams into a single vocational unit and for executives to be supportive of IPS. In the category of “fast movers” were components of IPS and cognitive remediation in which individual professionals were able to increase fidelity scores by acquiring new knowledge and skills—such as by focusing more on competitive jobs for IPS and by teaching patients to apply cognitive strategies for cognitive remediation.

Our study had several limitations regarding the fidelity measures. Findings must be viewed as descriptive because of the lack of comparison sites, the low number of fidelity assessments (divided unevenly between interventions), and the risk of confounding (for example, by scales and raters). With regard to cognitive remediation, no specific validated fidelity scale was available for programs that use components such as massed practice, cognitive strategies, metacognition, and transfer to daily life—that is, components matching the cognitive remediation program that was implemented.

The method used to implement the three interventions had several strengths and limitations. Use of a multifaceted model was a strength, including learning within a collaborative, interactive training, and feedback on fidelity outcomes. However, conducting a randomized controlled trial of effectiveness that used an active control group within an implementation collaborative was more problematic than we had anticipated. Therefore, we recommend that future projects that combine these two ambitions consider research designs that interfere less with implementation goals. We should have taken more time and effort in the preparatory phases to address technical and usability problems in the software packages; inclusion of a thorough testing procedure would have helped avoid needless barriers to implementation. Given the general problems with engaging young adults in therapy after a first psychotic episode, the combination of IPS and cognitive remediation may have been too intensive for some patients. To address this issue, the goals of the separate practices could have been more strongly integrated, and a more critical estimation of the required intensity of the programs might have been helpful. Finally, the goal of implementing three interventions in the same period might have caused an initiative overload, making it too ambitious.