Effectiveness of Early Psychosis Intervention: Comparison of Service Users and Nonusers in Population-Based Health Administrative Data

This study benefited from the fact that the Prevention and Early Intervention Program for Psychoses in London, Ontario, is the only EPI service in a region with a population of approximately 425,000 people. This enabled us to ascertain cases of incident nonaffective psychosis in the program catchment area and identify persons who received EPI care. The Prevention and Early Intervention Program for Psychoses provides team-based, coordinated specialty care for people with early-stage psychosis, as described in detail elsewhere (19). Core features of the program include early case detection, an open-referral policy, rapid assessment of suspected cases, development of a flexible treatment plan, psychoeducation, and provision of comprehensive pharmacological and psychosocial interventions with an assertive case-management approach (19). The typical trajectory for a program client is intensive case management during the 2 years after admission, with care stepped down to medical management by a program psychiatrist between years 2 and 5. However, patients with greater needs or ongoing symptoms may continue to receive case management beyond 2 years (19). The program has evolved over the past two decades in accordance with emerging evidence pertaining to best practices and implementation of provincial standards for EPI services (20). It shares many core features with the National Institute of Mental Health NAVIGATE-RAISE (Recovery After an Initial Schizophrenia Episode [21]) program and the Early Psychosis Intervention Center program (22), with the main differences being the minimum age requirement for inclusion (16 years old) and an inpatient component of the Prevention and Early Intervention Program for Psychoses to provide continuity of care across settings.

We used linked health administrative data from the Institute for Clinical Evaluative Sciences, which captures all hospitalizations, emergency department visits, and outpatient physician visits covered under the universal Ontario Health Insurance Program. All medically necessary health services for nearly the entire population of Ontario are covered by the Ontario Health Insurance Program. Persons not covered are students from out-of-province jurisdictions and recent migrants (≤3 months of residency). The linked databases used and the services covered are listed below:

We constructed a retrospective cohort comprising incident case subjects with nonaffective psychotic disorder who resided in the EPI catchment area between 1997 and 2013. We mirrored the inclusion criteria for the Prevention and Early Intervention Program for Psychoses as closely as possible (19) and included case subjects who had an incident diagnosis of nonaffective psychosis as determined using a previously validated algorithm (23) and defined by the presence of either an inpatient admission with a primary discharge diagnosis of nonaffective psychotic disorder or two physician or emergency department visits with a diagnostic code for nonaffective psychotic disorder occurring within a 12-month period. The diagnostic codes varied by database and included ICD-9, ICD-10, and DSM-IV codes (for further details, see Table S1 in the data supplement accompanying the online version of this article). Additionally, we included case subjects who had no previous contacts for nonaffective psychotic disorder before 1997 (i.e., prevalent cases), with a look-back window of up to 20 years; who presented to services within the catchment area of the EPI program; and who were 16–50 years of age.

Primary data on EPI clients were linked deterministically to data from the Institute for Clinical Evaluative Sciences through Ontario Health Insurance Plan numbers (96% linkage rate), which enabled us to identify persons in our cohort who used EPI services. We linked data on EPI program psychiatrists using registration numbers from the College of Physicians and Surgeons of Ontario. All persons in the cohort who had a visit with a program psychiatrist but were not subsequently admitted were considered to have been screened for the program but found to be ineligible and were excluded from further analyses. The remaining members of the cohort (i.e., those who were not identified as EPI service users but were not excluded from further analyses) were classified as “nonusers.”

As a post hoc exclusion, we removed from further analysis people who had not had their diagnosis confirmed by a psychiatrist at any point in their medical record and had no subsequent contacts with services for psychosis, since these cases were likely “false positives.” We also excluded people with fewer than two contacts with any service in the catchment area in the 6 months after cohort entry, since they were likely to have moved from the area.

Data linkage procedures were approved by the Health Sciences Research Ethics Board at the University of Western Ontario, London. Patient-level data were linked using encoded identifiers and analyzed on-site at the Institute for Clinical Evaluative Sciences. We followed the RECORD ([REporting of studies Conducted using Observational Routinely-collected health Data] [24]) guidelines for observational studies. A description of the codes and algorithms used to create study variables is presented in Table S1 in the online data supplement, and the RECORD checklist is presented in Table S2.

The date of the first diagnosis of nonaffective psychosis was used as the index date. We extracted information on sociodemographic characteristics, including age, gender, neighborhood-level income quintile, migrant status, and rurality of residence.

We constructed several clinical indicators, including type of diagnosis (schizophrenia, delusional disorder, or other), source of index diagnosis (inpatient admission and outpatient visit), and history of contact with services for alcohol-related or substance-related disorders.

Finally, we constructed indicators of previous service use for mental disorders, including the number of prior mental health contacts with a family physician, a psychiatrist, and the emergency department, as well as previous psychiatric hospitalizations.

The primary outcome variables were obtained for two time periods: the period from EPI admission to 2 years postadmission, when clients typically receive intensive case management, and the period from 2 to 5 years postadmission, when care typically transitions from intensive case management to medical management by a psychiatrist. However, we do not have the exact dates for when this transition occurred.

We constructed outcome indicators for each observation period, including contact with a primary care physician, contact with a psychiatrist, emergency department visits, psychiatric hospitalizations, and involuntary admissions. We obtained data on any use of these services as an indicator of access (binary variable), intensity of service use (count variable), and time between EPI admission and the first encounter with each service.

We also obtained information on presentations to the emergency department for self-harm, and vital statistics and hospitalizations were used to identify deaths by suicide. All cases of unnatural death in Ontario are investigated by a coroner, and cases with clear evidence of suicidal intent are recorded as suicide. Given the rarity of suicide and the high likelihood of misclassification in coroner records (25), we made a post hoc decision to compare all-cause mortality between the groups.

We used propensity score matching to identify a comparison group of nonusers of EPI services. This method mimics some of the characteristics of a randomized trial when using observational data (26). A propensity score is defined as the probability that a person is “exposed,” conditional on a set of observed variables. In a randomized trial, the probability of exposure is 0.5, and this can be estimated in an observational study by regressing exposure status on observed baseline characteristics (26). Matching on the propensity score functions to achieve balance between exposed and unexposed groups on the variables included in the propensity score model (26) but does not achieve balance on unmeasured confounding factors as in a randomized trial.

We used logistic regression to model the exposure variable (EPI service user, EPI service nonuser) as a function of observed covariates to yield a probability of EPI admission for each person. The propensity score model included baseline covariates representing sociodemographic characteristics, clinical factors, and previous service use (for further details, see Table S1 in the online data supplement). We used a “greedy match” technique without replacement to match EPI service users with nonusers based on the year of index diagnosis and the caliper of the propensity score (27). Matching was conducted with a variable ratio of up to 1:4, and each EPI service nonuser was assigned the same admission date as each EPI service user in the matched set.

Baseline characteristics of EPI service users and nonusers were compared using standardized differences, and values >10% were indicative of significant differences (28). There were minimal missing data (<1%) for covariates of interest, and persons with missing data were excluded.

The Cox proportional hazards model with robust variance estimators was used to account for clustering within matched sets to model access to care, suicide, and mortality indicators (binary). Modified Poisson regression was used to model indicators of intensity of service use (counts). We adjusted for any covariates that were not well balanced by the propensity score matching. Observations were censored at the end of the follow-up period, at termination of Ontario Health Insurance Plan eligibility, or in the event of death.

Results were consistent across the indicators for access to care and intensity of service use. Therefore, only the results of the access (binary) indicators are presented. All results are presented as hazard ratios with corresponding 95% confidence intervals. Analyses were conducted with SAS Version 9.4 (SAS Institute, Cary, N.C.).

Details on creation of the study cohort are presented in Figure 1. An additional seven matched sets were removed from the analysis due to missing admission dates, which resulted in a final matched sample of 530 EPI service users (70% of the linked sample) and 992 EPI service nonusers.

The distribution of baseline characteristics of EPI service users and nonusers is summarized in Table 1. There were substantial differences across most indicators. After matching, the groups were well balanced on sociodemographic and clinical characteristics but not on several health service indicators, and we adjusted for these indicators in the analyses.

Our findings are strengthened by data from a well-established EPI program spanning nearly two decades, thus providing information on the effect of EPI services on important health care system indicators in a real-world context. We used propensity score matching to improve balance in the distribution of confounding factors between groups, and we used concurrent control subjects matched on the year of diagnosis to ensure equivalency in temporal and contextual factors.

It is noteworthy that propensity score matching allows for the estimation of the average treatment effect among those who are treated (26)—that is, the average treatment effect among persons who have a similar probability of admission to EPI services. Additionally, it is important to point out that a large proportion of individuals with nonaffective psychosis in the catchment area do not come into contact with EPI services, which we have discussed in detail elsewhere (40). Furthermore, propensity score matching enables the ability to achieve balance on observed covariates and cannot account for unmeasured confounding factors. We cannot rule out the possibility that our findings are due to baseline differences in factors such as illness severity and level of family involvement in treatment. Moreover, we assessed whether the protective effect of EPI services on all-cause mortality could be due to survivor treatment-selection bias (also referred to as immortal time bias) by modeling the use of EPI services as a time-dependent exposure (41) with negligible impact on our findings (data available on request from the authors).

Although we mirrored the inclusion criteria of the Prevention and Early Intervention Program for Psychoses as closely as possible, information on some criteria—specifically the duration of previous treatment with antipsychotic medication (<1 month) and outstanding legal issues—was not available in the administrative databases. In a previous study conducted in the United States, data from electronic medical records were used to confirm cases of first-episode psychosis identified from health administrative data (41). The proportion of putative cases that were true cases of first-episode psychosis varied by clinical setting and age, with confirmation rates ranging from 78% for those from inpatient or emergency department admissions to 34% for those from primary care settings (41). In our study, approximately 40% of EPI service nonusers did not have their diagnosis confirmed by a psychiatrist or during an inpatient admission at any point in their medical records. Consequently, a subset of our comparison group likely would not meet inclusion criteria for EPI programs. We attempted to reduce the effect of this by propensity score matching to select control subjects who had a similar probability of admission to an EPI program.

We did not have information pertaining to the date on which a patient was discharged or disengaged from the program, and thus misclassification of exposure was possible. However, this would underestimate the benefits of EPI services on the outcomes of interest. We were limited by information available in the health administrative data and were unable to capture use of nonmedical services, such as community-based mental health organizations, private psychologists, and other social services.

Finally, we lost approximately 30% of our sample due to an inability to find a suitable match, and this affects the generalizability of our findings. Unmatched EPI service users differed on several sociodemographic factors but did not differ on outcome measures relative to the matched EPI service users (for further details, see Table S3 in the online data supplement).