Given limitations of available treatments for schizophrenia, with most patients showing substantial deficits in social and occupational functioning throughout life, there is considerable interest in developing preventive approaches to psychotic disorders (
1). Ascertainment of individuals at greatest risk is crucial to these efforts. For the majority of patients, onset of fully psychotic symptoms is preceded by the emergence of subtler changes in belief, thought, and perception that appear to represent attenuated forms of delusions, formal thought disorder, and hallucinations, respectively. Among individuals 12–35 years old with a recent onset of such symptoms (termed clinical high-risk cases), approximately 20%−35% develop fully psychotic symptoms over a 2-year period, an incidence rate that is more than 100 times larger than in the same age band in the general population (
2). Furthermore, it appears that the clinical high-risk criteria are sensitive to an imminent risk for onset, as most of the conversions occur during the first year after ascertainment, with a decelerating conversion rate thereafter (
3).
Although clinical high-risk criteria have been validated in epidemiological studies as sensitive to conversion risk, their utility in individual decision making is currently limited, given that 65%–80% of cases ascertained by these methods do not convert to psychosis within a 2-year time frame. About a dozen studies have examined combinations of clinical and demographic variables to determine whether prediction of psychosis can be enhanced beyond the 20%−35% risk associated with clinical high-risk status (
4). Multivariate algorithms requiring particular combinations of symptoms and demographic factors achieve relatively high positive predictive values and specificity (e.g., in the 50%−70% range) but low sensitivity (e.g., in the 10%−30% range) (
3). There is consistency among studies in showing (unsurprisingly) that greater severity of the psychosis-risk symptoms at baseline is the best predictor of conversion; nevertheless, the most predictive multivariate profiles vary across studies (as summarized in reference
4). Although it should be noted that few studies have attempted direct replication of one another’s risk algorithms, this pattern suggests heterogeneity among profiles of clinical and demographic risk indicators among patients who convert to psychosis.
Here we present such an individualized risk calculator for psychosis, using data from the second phase of the North American Prodrome Longitudinal Study (NAPLS-2). Predictors were chosen a priori based on a review of the literature on psychosis risk prediction in clinical high-risk samples, blindly with respect to the empirical relationships between any of the nominated variables and psychosis outcome within the NAPLS-2 data set. We limited our scope to clinical, cognitive, and demographic measures that are readily obtainable in standard clinical settings. Using time-to-event proportional hazards regression, a risk calculator was generated that calculates risk according to an individual’s values on the included variables. We evaluated the performance of the risk calculator using the concordance index (the Harrell C-index, a measure of overall accuracy, analogous to area under the receiver operating characteristic curve) and assessed the relative importance of each of the included predictor variables.
Results
Baseline patient characteristics are summarized in
Table 1. There were no significant differences between those who were and were not followed up clinically on any of the predictor variables. Of the 596 participants for whom follow-up data were available, 84 converted to psychosis within the 2-year study period. The mean age of the sample was 18.5 years. Among converters, the mean time from baseline to conversion was 7.3 months, and among nonconverters, the mean follow-up time from baseline to the last contact was 19.1 months. A total of 280 cases were followed up at 24 months without converting, and the remaining “nonconverters” were lost to follow-up at various points between 6 months and 24 months.
The 2-year probability of conversion to psychosis was 0.16 (95% CI=0.13, 0.19).
Figure 1 provides frequency distributions of predicted risks for converters and nonconverters. Converters occur at a proportionally higher rate than nonconverters in each successive risk class, beginning at a predicted risk of 0.20. The output of the multivariate proportional hazards model is presented in
Table 2. Prodromal symptom severity (SIPS items P1 and P2, modified and summed), decline in social functioning, and verbal learning and memory (Hopkins Verbal Learning Test–Revised scores) were significant predictors, with nonsignificant effects for age at baseline and speed of processing (symbol coding score) (p values, <0.10), although all of these variables were significant in univariate analyses (p values, <0.01). Stressful life events, traumas, and family history of schizophrenia were not significant predictors in univariate or multivariate analyses.
Table 2 provides additional diagnostics of the performance of individual predictor variables. Predictors associated with the largest decreases in the C-index when removed from the model were symptom severity, decline in global social functioning, Hopkins Verbal Learning Test–Revised score, and symbol coding score. Predictors associated with the largest increases in the C-index (i.e., above that of the base model, which included only symptom severity) were symbol coding score, Hopkins Verbal Learning Test–Revised score, decline in social functioning, and age. Family history of psychosis, stressful life events, and traumas did not alter the C-index by more than 0.5% when added to or deleted from the model.
Based on the bootstrap internal validation, the multivariate model achieved a C-index of 0.71. As shown in
Figure 2, the calibration plot revealed a high degree of consistency between observed probabilities and model-predicted probabilities of conversion to psychosis within the range of 0.0–0.4, within which 95% of the cases fell (mean=0.18, SD=0.11, median=0.16).
Table 3 provides statistics for prediction of actual conversion to psychosis across several thresholds of model-predicted risk. There is a trade-off between the positive predictive value (proportion of cases at the threshold of predicted risk who actually converted) and sensitivity (proportion of actual converters who had predicted risks at that threshold). The positive predictive value is maximal (48.4%) at a threshold of 0.4 of model-predicted risk, but only 17.9% of converters had model-predicted risks at this threshold. Conversely, at a model-predicted risk of 0.2 or higher, the positive predictive value is 28.1%, but with a sensitivity of 66.7%.
An online version of the risk calculator was built to facilitate numeric calculation of the predicted probability of conversion to psychosis (
http://riskcalc.org:3838/napls/).
Discussion
The goal of this study was to develop a practical tool for the individualized prediction of psychosis in clinical high-risk patients. A well-performing risk calculator was generated from the NAPLS-2 cohort data using a small number of demographic (age, family history of psychosis), clinical (unusual thought content and suspiciousness), neurocognitive (verbal learning and memory, speed of processing), and psychosocial (traumas, stressful life events, decline in social functioning) predictor variables. The overall model achieved a C-index of 0.71, which is in the range of values for established calculators currently in use for cardiovascular disease and cancer recurrence risk, which range from 0.58 to 0.81 (
5–
9).
The risk calculator generates a number representing the probability of transition to psychosis given a particular profile of input variables. Technically, this is an observed likelihood of conversion within the NAPLS-2 cohort itself, but this framework uses the logic of predictive inference to extend that observed likelihood based on past cases to the predicted probability for a newly ascertained case with the same profile. This logic rests on the assumption that the new case is ascertained from the same population and in a manner similar to those in NAPLS-2.
In particular, given that this risk calculator assumes a SIPS-based diagnosis of a prodromal risk syndrome as a starting point, the risk prediction tool would not be usable if such a risk syndrome has not been diagnosed. The risk calculator also assumes particular pathways to ascertainment, in that clinical high-risk cases in NAPLS are distressed and treatment seeking. This tool would thus be most useful to clinicians with training in psychosis risk detection using the SIPS (which, in addition to risk status, ascertains severity of unusual thought content and suspiciousness and family history of psychosis), who could then use the calculator for patients who have screened positive for a prodromal risk syndrome. Critically, risk determinations should be communicated to clients by trained clinicians who can help clients understand the meaning of the risk estimates (i.e., calibrated to the sample from which they were generated) and provide commensurate treatment recommendations. Note that within the context of NAPLS-2, with a mean predicted risk of 0.18 (SD=0.11), predicted risks of 0.3 or higher are relatively rare (12.4% prevalence among those meeting clinical high-risk criteria) and potent (39.2% positive predictive value). Proper training in the administration and scoring of the other measures included in the risk calculator (symbol coding, Hopkins Verbal Learning Test–Revised, Global Functioning: Social scale, Research Interview Life Events Scale, Childhood Trauma and Abuse Scale) is also required.
A key advantage of the risk calculator is that it inherently accommodates heterogeneity in profiles of risk factors among clinical high-risk cases. Examining configurations that vary across the significant predictors—greater prodromal symptom severity, lower verbal learning and memory, slower speed of processing, greater decline in social functioning, and younger age—reveals that dozens of separate permutations yield predicted conversion risks of 0.3 or higher. Stressful life events, traumas, and family history of schizophrenia have a negligible impact on their own or in combinations with other variables in the prediction of psychosis, but they were present more frequently among clinical high-risk individuals compared with healthy comparison subjects. Perhaps these variables are more significant for determining presence of a clinical high-risk syndrome and thus are not as sensitive to outcomes within a group of subjects with a clinical high-risk syndrome.
A crucial test of robustness of a statistical model is validation on an independent external data set. In a companion article (
30), Carrión et al. report on a replication test of the NAPLS-2 risk calculator in an independent sample from the Early Detection, Intervention, and Prevention of Psychosis Program (EDIPPP) that included 176 clinical high-risk cases diagnosed using the SIPS and followed clinically to monitor conversion. Only the stress and trauma variables—found to be negligible in predicting conversion here—were not collected and were therefore omitted from the replication testing. The remaining six NAPLS-2 risk factors yielded a significant time-to-event proportional hazards regression model predicting conversion in the EDIPPP sample (p<0.003), with a C-index of 0.79, which is even better than in the NAPLS-2 sample (0.71). The predictive model was well calibrated, and the NAPLS-2 calculator provided a reasonable estimation of psychosis risk when considering the risk prediction generated by the validation model and the actual observed outcomes. In addition, when applied to the external EDIPPP sample, the NAPLS-2 calculator showed sensitivity and specificity values comparable to those observed in the NAPLS-2 sample across different levels of model-predicted risk (
30).
Our findings also show some degree of convergence with previous studies that reported multivariate models but that used their own samples for variable selection (i.e., model optimization) and did not present a web-based tool for extending individualized risk estimation to future patients (
3,
23,
31). For example, a recent study (
23) using a smaller (N=92) and nonoverlapping sample of clinical high-risk cases from one of the NAPLS sites developed a classifier that included three of the predictors included in the NAPLS-2 risk calculator (suspiciousness, verbal memory deficits, and decline in social functioning). Note that the sample in that study was less than one-fifth the size of the present sample, used a more restricted age range (ages 12–20), and was not ascertained using SIPS criteria, factors that make risk classifications based on it much less generalizable than that of the NAPLS-2 risk calculator.
The most immediate uses of the risk calculator are likely to be in the selection of subjects for participation in clinical (prevention) trials, given the desire to avoid exposing patients with lower conversion risks to the potential adverse consequences of any interventions and given the potential to evaluate whether interventions differ in effectiveness based on initial risk levels or profiles across predictors. In terms of clinical practice outside the context of a prevention trial, at this point the most likely use of the risk calculator is for the clinician to be able to communicate to the patient and family a scaling of risk that could help in recruiting their cooperation with a monitoring and/or intervention plan.
A current limitation of the psychosis risk calculator is that risk estimates are not bounded by a confidence interval, making it unclear how well the single value output as a conversion risk represents the individual’s actual likelihood of conversion. This issue is particularly problematic for computed risks of 0.5 or higher, for which there is sparse representation in the NAPLS-2 data set and for which calibration of the risk calculator could consequently not be adequately tested. Nevertheless, the use of confidence intervals is not likely to be of value in discussing risks with individual patients and their families, as the general concept of a confidence interval relates to likelihoods under future sampling rather than to an individual case, and the calculated risk is the best estimate for that individual (
32).
Because the replication study (EDIPPP) included several community behavioral health centers and intergovernmental managed mental health organizations, the risk calculator appears to be generalizable beyond academic medical centers, at least within the U.S. health care system. The degree to which the risk calculator generalizes to other health care system models remains an open question.
In addition to testing the calculator’s performance in independent data sets, future work could determine whether other variables, including biological tests, can improve prediction over and above the set of clinical, demographic, and cognitive measures evaluated here. Some promising leads on the use of biological assays to predict psychosis among clinical high-risk patients have emerged using empirically based discovery approaches, including machine learning algorithms for gray matter variations in structural brain images (
33) and “greedy” regression algorithms for proteomic/metabolic plasma parameters (
34). Studies employing discovery-oriented model-optimization methods, with parallel, independent samples, are needed to better inform future versions of this and other risk calculators. However, it is still critical to note that the data used in any risk calculator cannot be the same data that are used in the model optimization phase; as noted above, doing so would invalidate the risk predictions for new cases.
Given that in approximately one-third of clinical high-risk cases, the symptoms that determined their initial risk status remit within 6–12 months of ascertainment (
35,
36), it should be possible to develop a complementary tool to predict a new case’s likelihood of remission from a clinical high-risk syndrome. Such an estimate would not necessarily be merely the inverse of the conversion risk, as different predictors may be relevant.
It is also possible that risk calculators could eventually be used to select clients for different treatment regimens or to reclassify risk after completion of a particular intervention. At this stage, the knowledge base for doing so is limited, as only a small number of controlled prevention trials in clinical high-risk cases have appeared. Collectively, the results support the view that any targeted intervention, whether biological or psychological in approach, is associated with better outcomes than less targeted control conditions (
37). Results of two small trials with antipsychotic drugs do not support a prophylactic effect on conversion risk beyond the period of active treatment (
38,
39). In general, the use of such medications in individuals who are below the threshold of full psychosis is not recommended. Intriguing results have been obtained in an initial trial of omega-3 fatty acid supplementation (
40); this finding awaits confirmation by independent studies. Psychosocial interventions such as cognitive-behavior therapy and family-focused psychoeducation may be beneficial in deflecting the course of illness severity and chronicity (
41,
42); however, it remains unclear whether such approaches can prevent onset of illness. Future intervention studies are encouraged to use the risk calculator at end-stage analysis to determine whether treatment efficacy is moderated by initial risk level or profile.
Ultimately, the degree of risk estimated by the risk calculator may be useful for weighing the cost-benefit ratios of various treatment options that emerge from clinical intervention research in the clinical high-risk population. Treatments associated with greater risks to the patient (e.g., medication side effects) or greater costs to health care delivery systems (e.g., resource- and time-intensive psychotherapeutic inventions) may best be reserved for those with higher-than-median levels of predicted risk (i.e., ≥0.16), while cost-effective treatments with benign side effect profiles may be the best option for those whose predicted risk for psychosis is in the lower range.
Like a person at risk for cardiovascular disease or cancer, an individual with a prodromal risk syndrome is more interested in receiving information pertinent to his or her personal risk profile than information about the population at large. Publication of this risk calculator is intended to assist clinicians in providing such personalized risk estimates. It is of course possible for untrained individuals to access these tools and approximate their scores on the set of risk variables. If, in so doing, a high predicted risk of conversion were generated, this could lead to significant personal distress. To mitigate this possibility, we have built in a decision tree for the online calculator that requires confirmation of an interview-based SIPS diagnosis of a prodromal risk syndrome and confirmation that the ratings and test scores were obtained by a professional; if either one of these verifications are missing, the decision tree opts out of making a prediction. The risk for loss of privacy or stigmatization based on access of the prediction tool by untrained users is also mitigated for these reasons.
In summary, a well-performing risk calculator for psychosis is available for application to new patients who meet criteria for a psychosis risk syndrome. Challenges to be addressed in the next phase of research include incorporating biological assays into the risk calculations, extending the analysis to predict likelihood of remission, extending the framework to calculate reductions in risk based on particular interventions, and investigating how patients and family members feel about and use this information.